From: Craig Loomis Subject: Re: Telescope Runaway inside Enclosure Date: Mon, 8 Nov 2004 13:55:46 -0700 On Nov 8, 2004, at 10:47 AM, Russell E Owen wrote: > At 6:48 PM -0600 11/6/04, Dan Long wrote: >> >> The question is why didn't the TCC complain earlier when the >> telescope was clearly deviating from the requested track? Another >> question is even if the TCC had lost control, what process was >> telling the telescope where to go? It wasn't as if the TCC lost >> control an the axes continued "drifting" at the last commanded >> velocity. They were continuing to accelerate. > > The TCC performs many error checks, such as verifying that the command > echo is correct and that the command completes in reasonable time > without errors. However, the TCC does not try to sanity-check the > reported positions. This decision was made for good reasons, and I > think is still the best choice, but it does leave us open to > possibility that a controller can go nuts and not realize it has done > so. That is what happened here. At that point we rely on the > mechanical design, which is supposed to protect the telescope against > runaways, or an operator pressing a stop button. (Note: even if the > TCC were able to detect a runaway, it could not do anything reliable > to stop it; the TCC does not have an interface to the stop buttons.) > > Unfortunately the logs of MCP output (in pod's /log/from.MCP) are > empty. This makes it very difficult to tell what the MCP thinks was > going on, since we can't see the output of the DRIFT command or the > STATUS commands. My guess is they would not have shown anything > useful, just an indication that the axes were running away for no > obvious reason. Still, it'd be comforting to be able to confirm that, > and the from.MCP logs should be fixed. > Gak. Bad. COM2 was plugged into COM1. The to.MCP log shows that the TCC was requesting just what it should have requested (DRIFT, MOVE 121.0, etc.). The little one can glean from the murmur logs shows that the MCP probably wanted to do the right thing with the DRIFT. Some oddities. - According the the TCC's status queries, both the azimuth and the rotator were continuously accelerating. The az was accelerating really slowly -- after 16s it was just over 0.5 deg/s. But the rotator was flying: it made it to at least 6 deg/s. - The slew crossed the 86400s clock rollover, and the azimuth and rotator slews "jumped" over it. 121.0 now should be the same as 121.0 any other time, and it it hard to believe this has not been done before, but it is slightly intriguing. 2004-11-05 23:58:40.372Z AZ DRIFT 2004-11-05 23:58:40.532Z ALT DRIFT 2004-11-05 23:58:40.692Z ROT DRIFT 2004-11-05 23:58:40.852Z ALT STATUS 2004-11-05 23:58:41.012Z AZ MS.ON 2004-11-05 23:58:41.092Z ALT MS.ON 2004-11-05 23:58:41.172Z ROT MS.ON 2004-11-05 23:58:41.252Z ROT STATUS 2004-11-05 23:58:41.461Z AZ MOVE 121.0009860 0.0000000 86355.52000 2004-11-05 23:58:41.651Z ALT MOVE 30.2017090 0.0000000 86355.52000 2004-11-05 23:58:41.761Z ROT MOVE 44.9915210 0.0000000 86355.52000 2004-11-05 23:58:41.931Z AZ MOVE 121.0000000 0.0000000 86357.52000 2004-11-05 23:58:42.138Z ALT MOVE 30.2150423 0.0400000 86356.52000 2004-11-05 23:58:42.241Z ROT MOVE 45.0000000 0.0000000 86357.52000 2004-11-05 23:58:42.401Z AZ MOVE 121.0000000 0.0000000 15.13553 2004-11-05 23:58:42.561Z ALT MOVE 43.5275423 1.4600000 86374.27000 2004-11-05 23:58:42.721Z ROT MOVE 45.0000000 0.0000000 15.13553 2004-11-05 23:58:42.841Z 2004-11-05 23:58:42.961Z ALT MOVE 45.0142090 1.5000000 86375.27000 2004-11-05 23:58:43.080Z 2004-11-05 23:58:43.160Z 2004-11-05 23:58:43.280Z ALT MOVE 75.1875000 1.5000000 86395.38553 2004-11-05 23:58:43.400Z 2004-11-05 23:58:43.480Z 2004-11-05 23:58:43.600Z ALT MOVE 76.6741667 1.4600000 86396.38553 2004-11-05 23:58:43.720Z 2004-11-05 23:58:43.800Z 2004-11-05 23:58:43.920Z ALT MOVE 89.9866667 0.0400000 14.13553 2004-11-05 23:58:44.040Z 2004-11-05 23:58:44.120Z 2004-11-05 23:58:44.240Z ALT MOVE 90.0000000 0.0000000 15.13553 2004-11-05 23:58:44.360Z 2004-11-05 23:58:44.460Z AZ MS.OFF 14.13553 2004-11-05 23:58:44.540Z ALT MS.OFF 14.13553 2004-11-05 23:58:44.620Z ROT MS.OFF 14.13553 - craig > The few things I can reliably say are: > - User # 10 issued the track command: > 2004-11-05 23:58:40.233Z 0 10 > Started; Cmd="track 121,90 mount" > so it is a red herring that user *8 had stopped listening to output > and was cut off. > - The clocks were correctly synchronized (based on the numerous > reports of AxisDTime ~= 0). Not that clock errors would be likely to > introduce such a runaway in any case. > - The TCC *thought* it was doing what the obs specs asked it to do. > That rules out a nasty delay in the TCC causing an older slew to be > run, not to mention the unlikely possibility of operator error. > > I think this is entirely internal to the MCP. I recommend looking in > any MCP logs that are available (presumably TPM data). > > Congratulations to the obs specs for catching the problem quickly and > handling it so gracefully. And thanks to Dan for the clear problem > report. I'm really sorry I can't offer more. > > Regards, > > -- Russell > > DSS DSS DSS DSS DSS Digital Sky Survey DSS DSS DSS DSS DSS DSS > DSS *** The information posted to SDSS mailing lists is proprietary *** > DSS This is message 2199 in the sdss-25mlog archive, URL > DSS http://www.astro.princeton.edu:81/sdss-25mlog/msg.2199.html > DSS The index is at > http://www.astro.princeton.edu:81/sdss-25mlog/INDEX.html > DSS To join/leave the list, go to > DSS > http://www.astro.princeton.edu:81/cgi-bin/mailinglists.pl/ > show_subscription?list=sdss-25mlog > DSS To post a message, mail it to sdss-25mlog@astro.princeton.edu > DSS > DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS