Subject: Re: [sloan-25mlog 2528] Re: 2.5m obslog 20150821 (SJD 57256) From: Kaike Pan Date: Tue, 25 Aug 2015 17:50:10 -0600 --Apple-Mail-9-530426268 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Thanks Russell and Conor for explanations. -kaike On Aug 25, 2015, at 5:30 PM, Russell Owen wrote: > I looked on tcc25m and I am convinced that =93setup tcc -t test=94 = will set up the new mirror controller code without adding any other = unwanted changes. >=20 > Only mirrorCtrl has been updated. The test version of the other = packages, such as tcc and twistedActor, are the same as the current = installed version. >=20 > =97 Russell >=20 >=20 > On Aug 25, 2015, at 4:22 PM, Kaike Pan wrote: >=20 >> Thanks, Conor! >>=20 >> I checked and confirmed that the primary has the version of 1.0.13b1, = and mirror >> are connected.=20 >>=20 >> Is there any default version of primary controller? if the mirror is = disconnected, will >> "tcc mirror connect" connect the correct version of pri controller?=20= >>=20 >> When I did a "tcc mirror status", i got several NaN (see below), will = these "NaN" worry >> you? >>=20 >> Thanks >>=20 >> -kaike >>=20 >>=20 >> mirror status output: >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> 23:15:49 APO.kpan 2 tcc i = mirrorConnState=3DConnected,Connected,NotAvailable >> 23:15:50 .tcc 0 tcc I = PrimEncMount=3D11500.0,-1200.0,5850.0,19600.0,24150.0,23900.0 >> 23:15:50 .tcc 0 tcc I PrimOrient=3D0.0,-24.31,6.14,356.65,1322.6,-0.01 >> 23:15:50 .tcc 0 tcc I = PrimActMount=3D11499.99,-1200.0,5850.01,19599.99,24150.0,23900.0 >> 23:15:50 .tcc 0 tcc I PrimCmdMount=3DNaN,NaN,NaN,NaN,NaN,NaN >> 23:15:50 .tcc 0 tcc I PrimStatus=3D4,4,4,4,4,4 >> 23:15:50 .tcc 0 tcc W Text=3D"number keys does not match number of = values" >> 23:15:50 .tcc 0 tcc I PrimMaxIter=3D1 >> 23:15:50 .tcc 0 tcc I PrimDesOrient=3DNaN,NaN,NaN,NaN,NaN,NaN >> 23:15:50 .tcc 0 tcc I PrimDesOrientAge=3DNaN >> 23:15:50 .tcc 0 tcc I PrimDesEncMount=3DNaN,NaN,NaN,NaN,NaN,NaN >> 23:15:50 .tcc 0 tcc I PrimState=3DDone,0,1,0.0,0.0 >>=20 >> On Aug 25, 2015, at 4:55 PM, Conor Sayres wrote: >>=20 >>> OK. >>>=20 >>> Primary mirror has been restarted. Version is 1.0.13b1. >>>=20 >>> I did not restart the TCC nor the secondary mirror. Should you need = to revert: >>>=20 >>> $ lsst >>> $ setup tcc # sets up current version >>> $ prim restart # restarts only the primary mirror controller >>>=20 >>> =46rom TUI you will have to issue tcc mirror connect (or init) after = any mirror controller restarts. >>>=20 >>>=20 >>> On Aug 25, 2015, at 7:42 PM, Kaike Pan wrote: >>>=20 >>>> Hi Conor, >>>>=20 >>>> Just checked, no one is using the telescope. Please go ahead. >>>>=20 >>>> Afterward, please send us an email with what to watch for, and = what version of tcc, pri controller, and how to restart >>>> them if we have to. Thanks, >>>>=20 >>>> -kaike >>>>=20 >>>> On Aug 25, 2015, at 4:30 PM, Conor Sayres wrote: >>>>=20 >>>>> I=92ve merged Russell=92s changes with mine, updated the code on = tcc25m, run unit tests, and I=92m ready to restart the primary mirror. = I=92m on STUI, waiting for permission. >>>>>=20 >>>>>=20 >>>>> On Aug 25, 2015, at 4:59 PM, Dan Long wrote: >>>>>=20 >>>>>> Hi Conor, >>>>>>=20 >>>>>> Russell sent something a while ago and I=92m not sure you were = on the distribution. Is this the same thing you saw? >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> On Aug 25, 2015, at 12:44 PM, Russell Owen wrote: >>>>>>=20 >>>>>>> I looked at some recent primary logs. The one named = prim25m.log-20150822 seems to contain most of the interesting data. >>>>>>>=20 >>>>>>> I think I have found the problem, but I cannot explain why the = system worked before this shutdown. >>>>>>>=20 >>>>>>> What I observe is that the usual pattern is as follows: >>>>>>> - A change in orientation is commanded. >>>>>>> - The mirror controller does the right thing -- if the move is = pure piston, then that is implemented as such, otherwise not >>>>>>> - BUT then the same orientation is usually commanded again. That = induces a tilt because the mirror controller treats a null move as NOT a = "pure piston" move and thus feels free to move A, B or C by different = amounts. >>>>>>>=20 >>>>>>> Thus I think the change I suggested in my previous email will = actually fix this problem. If others agree, I am happy to implement it. >>>>>>>=20 >>>>>>> -- Russell >>>>>>>=20 >>>>>>> On Aug 25, 2015, at 10:43 AM, Russell Owen wrote: >>>>>>>=20 >>>>>>>> I looked at the history of the tcc software and I only spooted = one change in the mirrorCtrl package that I thought might change the = behavior of the tertiary's handling of pure piston moves, and I'm not at = all sure it breaks anything. This is commit 3dfca64 on May 1 (post = 1.0.10, pre 1.0.11b1). This is the change (with a minor tweak to = compensate for a refactoring): >>>>>>>>=20 >>>>>>>> The old definition is: >>>>>>>>=20 >>>>>>>> orientChanged =3D = numpy.abs(numpy.asarray(orient[:5])-numpy.asarray(self.status.desOrient[:5= ])) > 1e-7 >>>>>>>> pistonOnly =3D orientChanged[0]=3D=3DTrue and = numpy.all(orientChanged[1:]=3D=3DFalse) >>>>>>>>=20 >>>>>>>> New definition adds a line to the middle: >>>>>>>>=20 >>>>>>>> orientChanged =3D = numpy.abs(numpy.asarray(orient[:5])-numpy.asarray(self.status.desOrient[:5= ])) > 1e-7 >>>>>>>> = orientChanged[numpy.nonzero(self.status.desOrient[:5]=3D=3Dnumpy.nan)] =3D= True >>>>>>>> pistonOnly =3D orientChanged[0]=3D=3DTrue and = numpy.all(orientChanged[1:]=3D=3DFalse) >>>>>>>>=20 >>>>>>>> The new line appears to set orientChanged true for any axis = whose desired orientation is unknown. I'm not sure why desired = orientation would ever be NaN, except right after startup. But if this = is occurring on a semi-routine basis, that could explain it. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> One obvious question is whether that date and version makes = sense with regards to the observed changed behavior. >>>>>>>>=20 >>>>>>>> Conor also introduced a useful change to the logging; for each = move it now logs whether the move is being done as pure piston or not. I = will have a look at the logs. >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> One suggestion, but I suggest tracking down the problem before = applying this: I think the code would be more robust if it ignored = piston changes. What we really want to know is if this move introduces a = change in tilt or translation? If so, then do a full move, otherwise = only move equal amounts in piston. Thus a tiny move should take the = "piston only" path to avoid introducing an unwanted tilt. A further = simplification is to test for orientUnchanged, as that test will = automatically handle NaN correctly. The results are: >>>>>>>>=20 >>>>>>>> # note that an orientation of NaN results in orientUnchanged = False, as desired >>>>>>>> orientUnchanged =3D = numpy.abs(numpy.asarray(orient[1:5])-numpy.asarray(self.status.desOrient[1= :5])) >>>>>>> pistonOnly =3D numpy.all(orientUnchanged[1:]) >>>>>>>>=20 >>>>>>>> -- Russell >>>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> On Aug 25, 2015, at 1:41 PM, Conor Sayres wrote: >>>>>>=20 >>>>>>> Hey Dan and Kaike, >>>>>>>=20 >>>>>>> This was a legitimate bug that I fixed last night somewhere over = Peru :), just getting internet back now. Would you like to run the new = code tonight in test mode, and maybe tag/make current tomorrow? >>>>>>>=20 >>>>>>> Conor >>>>>>>=20 >>>>>>> On Aug 24, 2015, at 4:45 PM, John K. Parejko = wrote: >>>>>>>=20 >>>>>>>> On 24Aug 2015, at 14:42, ucolick wrote: >>>>>>>>=20 >>>>>>>>> For starters, we have known forever that we don't need to = change the scale nearly as often as it gets updated in the current = guiding scheme. =20 >>>>>>>>=20 >>>>>>>> I'm also about to jump on a plane, but this can be solve easily = enough with the observers just unchecking the "scale" box while guiding. = Might be worth a shot, for a quick-and-dirty fix. >>>>>>>>=20 >>>>>>>> We can discuss making such a change more permanent later. The = guider code is about to get a major working over in Chile, so such = changes may well become easier to manage after this week. >>>>>>>>=20 >>>>>>>> John >>>>>>>>=20 >>>>>>>> -- >>>>>>>> ************************* >>>>>>>> John Parejko >>>>>>>> john.parejko@yale.edu >>>>>>>> http://www.astro.yale.edu/~jp727/ >>>>>>>> 203 432-9759 >>>>>>>> JWG 465 >>>>>>>> Department of Physics >>>>>>>> Yale University >>>>>>>> New Haven, CT >>>>>>>> ************************** >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>=20 >>>>=20 >>>=20 >>=20 >> ____________________________________________________________ >> Logs of Nightly APO Activities >> = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html >=20 --Apple-Mail-9-530426268 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Thanks Russell and Conor for explanations. = -kaikeOn Aug 25, 2015, at 5:30 PM, Russell Owen = wrote:I = looked on tcc25m and I am convinced that =93setup tcc -t test=94 will = set up the new mirror controller code without adding any other unwanted = changes.Only mirrorCtrl has been updated. The test = version of the other packages, such as tcc and twistedActor, are the = same as the current installed version.=97 = RussellOn Aug 25, 2015, at 4:22 PM, = Kaike Pan = wrote:Thanks, = Conor!I checked and confirmed that the = primary has the version of 1.0.13b1, and mirrorare = connected. Is there any default version = of primary controller? if the mirror is disconnected, = will"tcc mirror connect" connect the correct version of pri = controller? When I did a "tcc mirror = status", i got several NaN (see below), will these "NaN" = worryyou?Thanks-kaikemirror status = output:=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= 23:15:49 APO.kpan 2 tcc i = mirrorConnState=3DConnected,Connected,NotAvailable23:15:50 = .tcc 0 tcc I = PrimEncMount=3D11500.0,-1200.0,5850.0,19600.0,24150.0,23900.023= :15:50 .tcc 0 tcc I = PrimOrient=3D0.0,-24.31,6.14,356.65,1322.6,-0.0123:15:50 .tcc = 0 tcc I = PrimActMount=3D11499.99,-1200.0,5850.01,19599.99,24150.0,23900.023:15:50 .tcc 0 tcc I = PrimCmdMount=3DNaN,NaN,NaN,NaN,NaN,NaN23:15:50 .tcc 0 tcc I = PrimStatus=3D4,4,4,4,4,423:15:50 .tcc 0 tcc W Text=3D"number = keys does not match number of values"23:15:50 .tcc 0 tcc I = PrimMaxIter=3D123:15:50 .tcc 0 tcc I = PrimDesOrient=3DNaN,NaN,NaN,NaN,NaN,NaN23:15:50 .tcc 0 tcc I = PrimDesOrientAge=3DNaN23:15:50 .tcc 0 tcc I = PrimDesEncMount=3DNaN,NaN,NaN,NaN,NaN,NaN23:15:50 .tcc 0 tcc = I PrimState=3DDone,0,1,0.0,0.0On Aug 25, 2015, = at 4:55 PM, Conor Sayres wrote:OK.Primary mirror has been = restarted. Version is 1.0.13b1.I did not = restart the TCC nor the secondary mirror. Should you need to = revert:$ lsst$ setup tcc # sets up = current version$ prim restart # restarts only the primary = mirror controller=46rom TUI you will have to = issue tcc mirror connect (or init) after any mirror controller = restarts.On Aug 25, 2015, at 7:42 PM, Kaike Pan = wrote:Hi = Conor, Just checked, no one is using the = telescope. Please go ahead. = Afterward, please send us an email with what to watch for, and = what version of tcc, pri controller, and how to restartthem = if we have to. Thanks, = -kaikeOn Aug 25, 2015, at 4:30 PM, Conor Sayres = wrote:I=92ve = merged Russell=92s changes with mine, updated the code on tcc25m, run = unit tests, and I=92m ready to restart the primary mirror. I=92m = on STUI, waiting for permission.On Aug = 25, 2015, at 4:59 PM, Dan Long = wrote:Hi = Conor, Russell sent something a while = ago and I=92m not sure you were on the distribution. Is this the = same thing you = saw?On Aug = 25, 2015, at 12:44 PM, Russell Owen wrote:I looked = at some recent primary logs. The one named prim25m.log-20150822 = seems to contain most of the interesting data.I = think I have found the problem, but I cannot explain why the system = worked before this shutdown.What I observe is = that the usual pattern is as follows:- A change in = orientation is commanded.- The mirror controller does the = right thing -- if the move is pure piston, then that is implemented as = such, otherwise not- BUT then the same orientation is usually = commanded again. That induces a tilt because the mirror controller = treats a null move as NOT a "pure piston" move and thus feels free to = move A, B or C by different amounts.Thus I = think the change I suggested in my previous email will actually fix this = problem. If others agree, I am happy to implement = it.-- RussellOn Aug 25, = 2015, at 10:43 AM, Russell Owen wrote:I looked at the history of the = tcc software and I only spooted one change in the mirrorCtrl package = that I thought might change the behavior of the tertiary's handling of = pure piston moves, and I'm not at all sure it breaks anything. This is = commit 3dfca64 on May 1 (post 1.0.10, pre 1.0.11b1). This is the change = (with a minor tweak to compensate for a = refactoring):The old definition = is:orientChanged =3D = numpy.abs(numpy.asarray(orient[:5])-numpy.asarray(self.status.desOrient[:5= ])) > 1e-7pistonOnly =3D orientChanged[0]=3D=3DTrue and = numpy.all(orientChanged[1:]=3D=3DFalse)New= definition adds a line to the = middle:orientChanged =3D = numpy.abs(numpy.asarray(orient[:5])-numpy.asarray(self.status.desOrient[:5= ])) > = 1e-7orientChanged[numpy.nonzero(self.status.desOrient[:5]=3D=3D= numpy.nan)] =3D TruepistonOnly =3D orientChanged[0]=3D=3DTrue = and = numpy.all(orientChanged[1:]=3D=3DFalse)The new line appears to set orientChanged true for any axis whose = desired orientation is unknown. I'm not sure why desired orientation = would ever be NaN, except right after startup. But if this is occurring = on a semi-routine basis, that could explain = it.One obvious question is whether that date and version = makes sense with regards to the observed changed = behavior.Conor also introduced a useful change = to the logging; for each move it now logs whether the move is being done = as pure piston or not. I will have a look at the = logs.One suggestion, but I = suggest tracking down the problem before applying this: I think the code = would be more robust if it ignored piston changes. What we really want = to know is if this move introduces a change in tilt or translation? If = so, then do a full move, otherwise only move equal amounts in piston. = Thus a tiny move should take the "piston only" path to avoid introducing = an unwanted tilt. A further simplification is to test for = orientUnchanged, as that test will automatically handle NaN correctly. = The results are:# note that an orientation of = NaN results in orientUnchanged False, as = desiredorientUnchanged =3D = numpy.abs(numpy.asarray(orient[1:5])-numpy.asarray(self.status.desOrient[1= :5])) < 1e-7pistonOnly =3D = numpy.all(orientUnchanged[1:])-- = RussellOn = Aug 25, 2015, at 1:41 PM, Conor Sayres wrote:Hey Dan = and Kaike,This was a legitimate bug that I fixed last night = somewhere over Peru :), just getting internet back now. Would you = like to run the new code tonight in test mode, and maybe tag/make = current tomorrow?ConorOn Aug 24, 2015, at 4:45 PM, John = K. Parejko = wrote:On 24Aug 2015, at 14:42, ucolick = = wrote:For starters, we have known = forever that we don't need to change the scale nearly as often as it = gets updated in the current guiding scheme. = I'm also about to jump on a plane, but this = can be solve easily enough with the observers just unchecking the = "scale" box while guiding. Might be worth a shot, for a quick-and-dirty = fix.We can discuss making such a change more permanent later. = The guider code is about to get a major working over in Chile, so such = changes may well become easier to manage after this = week.John--*************************John = Parejkojohn.parejko@yale.eduhttp://www.astro.yale.edu/~jp72= 7/203 432-9759JWG 465Department of PhysicsYale = UniversityNew Haven, = CT**************************______________________________________= ______________________Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/in= dex.html= --Apple-Mail-9-530426268--