Date: Thu, 12 Sep 2019 09:37:28 -0600
Subject: Re: [sloan-25mlog 4635] 2.5m obslog 20190911 (SJD 58738)
From: "Daniel Oravetz"
It's interesting that TCCPos was 'None' during the slew misbehaviors. The
slew command that gets sent as part of the 'sop gotoGangChange' sequence
is a 'tcc track' command that should compute the path and then update
TCCPos with the target destination. The fact that the telescope is moving
without the TCC having specified a target mount position seems to indicate
that the "crazy" destinations aren't being supplied by the TCC. Similarly,
the unexpected rotator slew on the night of 9/5 also occurred when TCCPos
wasn't set, although it was 'nan' that time instead of 'None'.
The faster speeds noted during the misbhaving slews also make sense in
this context: the MaxVel, MaxAcc, etc. limits in axeLim.dat only apply to
TCC-requested motion. Since the observers use the TCC for the overwhelming
majority of our slews we only usually see motion within these limits.
Dan
>
>                        APACHE POINT OBSERVATORY SDSS 2.5M
>
>                                  OBSERVING LOG
>                           Wednesday September 11, 2019
>
>                                   (SJD 58738)
> 1.Axes misbehavior: time stamp: 07:55, 10:31
> The telescope slews crazy with "goto Gang Change" after science data.
> After apogee science data, "goto Gang Change" sop command supposes to
> close Flat
> Field screen, stop the guiding, and take a dome flat when the telescope is
> tracking , then slew to the gang change position: 121, 45.
>
> After the dome flat, axis status goes to halting on AZ and Alt, and halted
> on
> rot for ~ 5 sec, then slew craze to where that differs from tcc track
> command:
> track 121, 45 with high speed. In the first incidence: the telescope
> suppose to
> slew to gangChange position (121, 45, 153) from altitude of 48 deg. The
> altitude
> moved at a speed faster than normal, and rotator oscillation. An estop is
> pressed. The altitude past the destination, 45 deg, reached 41 degrees
> when the
> eStop is pressed. The error messages are in problem section.  The 2nd
> incidence,
>  the telescope moved at high speed that we never seen in altitude and
> rotator.
> For more details, see log and problem section.
>
> Work around solution: Take dome flat, tcc axis stop, goto GangChange with
> only slew.
> ---- Problems in Detail ----
> 2. Axes Misbehavior:
> 07:55:00 OBSERVER.aoravetz 89 sop w text="Not waiting for response from:
> FF on"
> 07:55:01 .tcc 0 tcc W Text="number keys does not match number of values"
> 07:55:02 OBSERVER.aoravetz.sop 799 guider w text="FF petals aren't all
> open; aborting guiding"
> 07:55:04 .hub 0 cmds d
> CmdQueued=1360832,1568274904.19,"OBSERVER.aoravetz.sop",825,"mcp",121,"ff.off"
> 07:55:19 .boss 1524 boss w SP1BlueIonPumpLatched=1568231236.42;
> SP2R1HeaterVRead=-0.586
> 07:55:19 .boss 1524 boss w camCheck=SP1BlueIonPumpLatched,SP2R1HeaterVRead
> 07:55:22 .alerts 0 alerts w alert=boss.SP1BlueIonPump,warn,"Reported by
> camCheck",enabled,noack,""
> 07:55:33 .hub 0 cmds d
> CmdQueued=1360835,1568274933.26,"OBSERVER.aoravetz.sop",826,"apogee",348,"shutter
> close"
> 07:55:43 .hub 0 cmds d
> CmdQueued=1360836,1568274943.40,"OBSERVER.aoravetz.sop",827,"tcc",5752,"axis
> status"
> 07:55:43 .hub 0 cmds d
> CmdQueued=1360837,1568274943.54,"OBSERVER.aoravetz.sop",828,"tcc",5753,"axis
> init"
> 07:55:43 .tcc 0 tcc w AxisCmdState=Halting,Halting,Halting;
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 07:55:50 .hub 0 cmds d
> CmdQueued=1360839,1568274950.95,"OBSERVER.aoravetz.sop",829,"tcc",5754,"track
> 121.000000, 45.000000 mount/rottype=mount/rotangle=153.344742"
> 07:55:50 .tcc 0 tcc w text='STATUS failed for axis: az'
> 07:55:50 .tcc 0 tcc w text='STATUS failed for axis: az'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: az'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 07:55:51 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 07:55:51 .tcc 0 tcc w AxisErrCode=OK,OK,HaltRequested; AxisNoTrack=FFT
> 07:55:51 .tcc 0 tcc w AxisErrCode=OK,HaltRequested,HaltRequested;
> AxisNoTrack=FTT
> 07:55:51 .tcc 0 tcc w
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 07:55:56 .mcp 0 mcp w rotBadFiducial=-5,155.24
> 07:55:56 .tcc 0 tcc w
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 07:55:56 .hub 0 cmds d
> CmdQueued=1360841,1568274956.57,"OBSERVER.aoravetz.sop",830,"tcc",5755,"axis
> stop"
> 07:56:01 .tcc 0 tcc W Text="number keys does not match number of values"
> 07:56:02 .mcp 0 mcp w rotBadFiducial=-5,150.50
> 07:56:15 .tcc 0 tcc w BadAzStatus;
> AzStat=10.525328,0.000000,5074991775.01740,0x00003800; AzDTime=0.04
> 07:56:15 .tcc 0 tcc w BadRotStatus;
> RotStat=150.397473,0.000000,5074991775.05811,0x00003800; RotDTime=0.04
> 07:56:15 .tcc 0 tcc w BadAltStatus;
> AltStat=41.373320,0.000000,5074991775.09910,0x00003800; AltDTime=0.04
> 07:56:16 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:56:16 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:56:16 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:56:19 .boss 1526 boss w SP2R1HeaterVRead=-0.586; SP1BlueIonPump=-4.94;
> SP1BlueIonPumpLatched=1568231236.42
> 07:56:19 .boss 1526 boss w
> camCheck=SP2R1HeaterVRead,SP1BlueIonPump,SP1BlueIonPumpLatched
> 07:56:19 .boss 0 boss w text="Clearing sticky errors on sp1cam"
> 07:56:19 .alerts 0 alerts w text="sent MAIL WARN
> boss.SP1BlueIonPumpLatched == Reported by camCheck"
> 07:56:26 .tcc 0 tcc w BadAzStatus;
> AzStat=10.525324,0.000000,5074991786.26925,0x00001c00; AzDTime=0.05
> 07:56:26 .tcc 0 tcc w BadRotStatus;
> RotStat=150.397479,0.000000,5074991786.32557,0x00001c00; RotDTime=0.04
> 07:56:26 .tcc 0 tcc w BadAltStatus;
> AltStat=41.373304,0.000000,5074991786.36678,0x00001c00; AltDTime=0.04
> 07:56:26 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:56:36 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:56:46 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:56:56 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:57:01 .tcc 0 tcc W Text="number keys does not match number of values"
> 07:57:05 .hub 0 cmds d
> CmdQueued=1360852,1568275025.79,"OBSERVER.telescope",293,"tcc",5758,"axis
> init"
> 07:57:10 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:57:10 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:57:10 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:57:19 .boss 1528 boss w SP1BlueIonPumpLatched=1568231236.42;
> SP2R1HeaterVRead=-0.586
> 07:57:19 .boss 1528 boss w camCheck=SP1BlueIonPumpLatched,SP2R1HeaterVRead
> 07:57:20 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:57:21 .alerts 0 alerts w text="sent MAIL WARN       boss.SP2R0VSc1Read
> == Reported by camCheck"
> 07:57:21 .alerts 0 alerts w text="sent MAIL WARN       boss.SP2R1VSc2Read
> == Reported by camCheck"
> 07:57:21 .alerts 0 alerts w alert=boss.SP1BlueIonPump,warn,"Reported by
> camCheck",enabled,noack,""
> 07:57:31 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 07:57:37 .hub 0 cmds d
>
> 10:30:22 .alerts 0 alerts w alert=boss.SP1BlueIonPump,warn,"Reported by
> camCheck",enabled,noack,""
> 10:30:26 OBSERVER.aoravetz.sop 941 guider w text="FF petals aren't all
> open; aborting guiding"
> 10:30:27 .hub 0 cmds d
> CmdQueued=1363151,1568284227.24,"OBSERVER.aoravetz.sop",975,"mcp",152,"ff.on"
> 10:30:27 OBSERVER.aoravetz 142 sop w text="Not waiting for response from:
> FF on"
> 10:30:31 .hub 0 cmds d
> CmdQueued=1363153,1568284231.28,"OBSERVER.aoravetz.sop",976,"mcp",153,"ff.off"
> 10:30:50 .tcc 0 tcc W Text="number keys does not match number of values"
> 10:31:00 .hub 0 cmds d
> CmdQueued=1363159,1568284260.31,"OBSERVER.aoravetz.sop",977,"apogee",385,"shutter
> close"
> 10:31:02 .tcc 0 tcc W Text="number keys does not match number of values"
> 10:31:10 .hub 0 cmds d
> CmdQueued=1363164,1568284270.37,"OBSERVER.aoravetz.sop",978,"tcc",6683,"axis
> status"
> 10:31:10 .hub 0 cmds d
> CmdQueued=1363165,1568284270.51,"OBSERVER.aoravetz.sop",979,"tcc",6684,"axis
> init"
> 10:31:10 .tcc 0 tcc w AxisCmdState=Halting,Halting,Halting;
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 10:31:20 .boss 1846 boss w SP1BlueIonPumpLatched=1568231236.42
> 10:31:20 .boss 1846 boss w camCheck=SP1BlueIonPumpLatched
> 10:31:21 .alerts 0 alerts w alert=boss.SP2B2VSc1Read,warn,"Reported by
> camCheck",enabled,noack,""
> 10:31:21 .hub 0 cmds d
> CmdQueued=1363167,1568284281.69,"OBSERVER.aoravetz.sop",980,"tcc",6685,"track
> 121.000000, 45.000000 mount/rottype=mount/rotangle=0.000000"
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: az'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: az'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: az'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: az'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: az'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: alt'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 10:31:21 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 10:31:22 .tcc 0 tcc w AxisErrCode=OK,OK,HaltRequested; AxisNoTrack=FFT
> 10:31:22 .tcc 0 tcc w AxisErrCode=OK,HaltRequested,HaltRequested;
> AxisNoTrack=FTT
> 10:31:22 .tcc 0 tcc w
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 10:31:26 .mcp 0 mcp w rotBadFiducial=-5,8.40
> 10:31:27 .tcc 0 tcc w
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 10:31:27 .tcc 0 tcc w text='STATUS failed for axis: rot1'
> 10:31:27 .hub 0 cmds d
> CmdQueued=1363168,1568284287.02,"OBSERVER.aoravetz.sop",981,"tcc",6686,"axis
> stop"
> 10:31:27 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:31:43 .tcc 0 tcc w BadAzStatus;
> AzStat=179.924928,0.000000,5075001103.88939,0x00003800; AzDTime=0.04
> 10:31:43 .tcc 0 tcc w BadRotStatus;
> RotStat=5.127252,0.000000,5075001103.93332,0x00003800; RotDTime=0.04
> 10:31:43 .tcc 0 tcc w BadAltStatus;
> AltStat=59.578486,0.000000,5075001103.97410,0x00003800; AltDTime=0.04
> 10:31:44 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:31:54 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:32:02 .tcc 0 tcc W Text="number keys does not match number of values"
> 10:32:04 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:32:14 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:32:20 .boss 1848 boss w SP1BlueIonPump=-4.93;
> SP1BlueIonPumpLatched=1568231236.42
> 10:32:20 .boss 1848 boss w camCheck=SP1BlueIonPump,SP1BlueIonPumpLatched
> 10:32:20 .boss 0 boss w text="Clearing sticky errors on sp1cam"
> 10:32:25 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:32:35 .tcc 0 tcc w BadAzStatus;
> AzStat=179.924912,0.000000,5075001155.20935,0x00001c00; AzDTime=0.05
> 10:32:35 .tcc 0 tcc w BadRotStatus;
> RotStat=5.127240,0.000000,5075001155.25856,0x00001c00; RotDTime=0.04
> 10:32:35 .tcc 0 tcc w BadAltStatus;
> AltStat=59.578471,0.000000,5075001155.32531,0x00001c00; AltDTime=0.04
> 10:32:35 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:32:45 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:32:55 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:02 .tcc 0 tcc W Text="number keys does not match number of values"
> 10:33:06 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:16 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:20 .boss 1850 boss w SP1BlueIonPumpLatched=1568231236.42
> 10:33:20 .boss 1850 boss w camCheck=SP1BlueIonPumpLatched
> 10:33:20 .alerts 0 alerts w alert=boss.SP1BlueIonPump,warn,"Reported by
> camCheck",enabled,noack,""
> 10:33:23 .alerts 0 alerts w alert=boss.SP2B3VSc2Read,warn,"Reported by
> camCheck",enabled,noack,""
> 10:33:26 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:36 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:37 .hub 0 cmds d
> CmdQueued=1363182,1568284417.81,"OBSERVER.telescope",297,"tcc",6689,"axis
> init"
> 10:33:42 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:42 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:52 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:33:56 .hub 0 cmds d
> CmdQueued=1363184,1568284436.32,"OBSERVER.aoravetz",143,"sop",144,"gotoGangChange
> nodomeFlat  alt=45.0"
> 10:33:56 .hub 0 cmds d
> CmdQueued=1363185,1568284436.42,"OBSERVER.aoravetz.sop",982,"tcc",6690,"axis
> status"
> 10:33:56 .hub 0 cmds d
> CmdQueued=1363186,1568284436.52,"OBSERVER.aoravetz.sop",983,"tcc",6691,"axis
> init"
> 10:33:58 .hub 0 cmds d
> CmdQueued=1363187,1568284438.43,"OBSERVER.aoravetz.sop",984,"tcc",6692,"track
> 121.000000, 45.000000 mount/rottype=mount/rotangle=0.000000"
> 10:34:02 .tcc 0 tcc W Text="number keys does not match number of values"
> 10:34:20 .boss 1852 boss w SP1BlueIonPump=-4.94;
> SP1BlueIonPumpLatched=1568231236.42
> 10:34:20 .boss 1852 boss w camCheck=SP1BlueIonPump,SP1BlueIonPumpLatched
> 10:34:20 .boss 0 boss w text="Clearing sticky errors on sp1cam"
> 10:34:42 .hub 0 cmds d
> CmdQueued=1363195,1568284482.49,"OBSERVER.aoravetz.sop",985,"tcc",6694,"axis
> stop"
> 10:34:42 .tcc 0 tcc w AxisCmdState=Halting,Halting,Halting;
> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT
> 10:34:42 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:34:44 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:34:54 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:35:02 .tcc 0 tcc W Text="number keys does not match number of values"
> 10:35:05 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:35:15 .alerts 0 alerts w text="Error processing rules for tcc.tccPos:
> KeyVar('tcc', tccPos=(None, None, None)) is unknown"
> 10:35:19 .hub 0 cmds d
> CmdQueued=1363201,1568284519.91,"OBSERVER.aoravetz",144,"sop",145,"gotoInstrumentChange
> "
> 10:35:20 .hub 0 cmds d
> CmdQueued=1363203,1568284520.01,"OBSERVER.aoravetz.sop",986,"tcc",6696,"axis
> status"
>
> ____________________________________________________________
> Logs of Nightly APO Activities
> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>