Subject: 2.5m obslog 20150225 (SJD 57079) From: viktorm@apo.nmsu.edu Date: Thu, 26 Feb 2015 13:13:59 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday February 25, 2015 (SJD 57079) ---- Observing Team ---- Karen Kinemuchi Victor Malanushenko Katie Grabowski Olena Malanushenko John Parejko Connor Sayers Joe Huehnerhoff Fritz Stauffer (on line) Dan Long (by phone) Russell Owen (by phone) Jim Gunn Connie Rockosi Renbin Yan Kaike Pan (afternoon) ---- Observing Summary ---- Closed at 2:52Z due to weather. ct=4, pl=8120, APOGEE-2; ORPHAN-2; APOGEE lead; 1xABBA+2xC; missing fibers: 20,39,195,278,288 ct=3, pl=8317, APOGEE-2&MaNGA; MJ193.52957+43.56598; MaNGA dither; E+NSE; 2xABBA; missing fibers: 195 ct=1, pl=8329, APOGEE-2&MaNGA; MJ213.06078+44.99336; MaNGA dither; 1xENS; 2xABBA missing fibers: 195, 258, 259 ct=3, pl=8317, APOGEE-2&MaNGA; MJ193.52957+43.56598; MaNGA dither; 1N; 1xAB missing fibers: 195 ct=1, pl=8329, APOGEE-2&MaNGA; MJ213.06078+44.99336; MaNGA dither; S+NES; 2xABBA missing fibers: 195, 258, 259 -- We had a problem with the mcp/tcc this evening. Resolved by 05:18Z, lost 4 hours, but we closed due to radar and clouds. See Problems in Details for information. -- the 'landru' computer working slow: the STUI loaded ~5 min. ---- Observing Plan ---- Wed 25 FEB 2015 57079.0 0.50 01:32 12:26 04:52 15:47 ---- APOGEE-2 until 08:28 ( 6.9 h), then MaNGA ( 4.0 h) ---- Observing Log ---- Afternoon: ========== Collimation was checked with cart 11. B1 focus ring was adjusted by 1.5 degrees. 00:04:40 Hartmann collimate output on cart #11 sp1: offset: r=-329 (In Focus); b=-1.0 (In Focus) sp2: offset: r=0 (In Focus); b=1.0 (In Focus) sp1: pred. move: spAverageMove= -4 sp2: pred. move: spAverageMove= -157 sp1: pred. spResiduals: r=-325, b=-1.0, txt=OK, spTemp = 4.5 sp2: pred. spResiduals: r=157, b=0.5, txt=OK, spTemp = 3.7 Guider cart= 11, pl= 8296, side= A (loaded); motorPosition sp1: 523, 624, 559; sp2: 16687, 15833, 17057; Median spectrographs temperatures: Tsp1= 4.6, Tsp2= 3.7; Outside air temperature= 1.7; APOGEE evening calibration was taken and checked. 00:10Z Mounted cart 9, started ventilation. 02:52:21 encl25m : closed; ---------------------------------------- 06:34:21 encl25m : open; 06:36:47 apogee.expose=apRaw-15170011, 10, A, Dark 06:38:35 apogee.expose=apRaw-15170012, 10, A, Dark 06:29:57 loadCart: ct=4, pl=8120, APOGEE-2; ORPHAN-2 06:59:50 Hartmann collimate output on cart #4 sp1: offset: r=-329 (In Focus); b=3.0 (In Focus) sp2: offset: r=-315 (In Focus); b=-3.0 (In Focus) sp1: pred. move: spAverageMove= -646 sp2: pred. move: spAverageMove= 315 sp1: pred. spResiduals: r=317, b=1.0, txt=OK, spTemp = 1.1 sp2: pred. spResiduals: r=-630, b=-2.0, txt=OK, spTemp = 0.4 07:13:13 apogee.expose=apRaw-15170013, 47, A, Object 07:13:19 boss exposure 900.0, file=195469, C 07:21:36 apogee.expose=apRaw-15170014, 47, B, Object 07:29:56 apogee.expose=apRaw-15170015, 47, B, Object 07:30:03 boss exposure 900.0, file=195470, C 07:38:24 apogee.expose=apRaw-15170016, 47, A, Object 07:47:38 apogee.expose=apRaw-15170017, 5, A, DomeFlat master= /home/observer/bin/apRaw-03720068.fits nfibers= 300 average ratio= 0.765387587738 missing fibers (flat/master -------------------------------------------------------------------------------- 07:12 4-8120A 1.000168 -1656 360 603 -5.4 72.1 -2.1 12 2 1.5 apog 07:12 4-8120A 1.000168 -1656 360 603 -5.4 72.1 -2.1 12 2 1.5 boss 07:21 4-8120A 1.000162 -1602 418 617 -11.9 71.9 -2.1 18 15 1.8 apog 07:29 4-8120A 1.000161 -1589 439 625 -18.1 71.4 -2.1 16 4 1.6 apog 07:29 4-8120A 1.000161 -1589 439 625 -18.2 71.4 -2.1 16 4 1.6 boss 07:37 4-8120A 1.000152 -1497 518 627 -24.1 70.8 -2.2 16 10 1.7 apog 08:08 3-8317A 1.000050 -496 1334 614 128.5 70.0 -2.4 15 32 1.8 boss 08:26 3-8317A 1.000054 -538 1285 603 133.6 72.9 -2.6 23 7 1.5 boss 08:44 3-8317A 1.000044 -436 1363 600 140.8 75.4 -2.8 10 17 1.5 boss 09:01 3-8317A 1.000030 -299 1470 596 151.5 77.5 -2.9 16 2 1.5 boss 09:38 1-8329A 1.000041 -407 1394 603 135.6 71.2 -3.2 27 26 1.6 boss 09:54 1-8329A 1.000027 -264 1515 609 141.5 73.5 -3.3 27 25 1.9 boss 10:11 1-8329A 1.000022 -213 1548 603 149.6 75.5 -3.3 19 3 1.6 boss 10:28 1-8329A 1.000012 -121 1615 593 161.1 77.0 -3.2 19 35 1.6 boss 11:01 3-8317A 0.999991 89 1775 569 231.5 69.9 -3.4 13 28 1.4 boss 11:37 1-8329A 0.999967 331 1962 565 212.5 75.1 -3.2 29 22 1.3 boss 11:53 1-8329A 0.999959 401 2009 549 219.9 73.0 -3.0 21 27 1.4 boss 12:10 1-8329A 0.999962 372 1984 545 225.5 70.7 -3.2 18 22 1.3 boss 12:27 1-8329A 0.999953 458 2054 539 229.7 67.9 -3.2 12 7 1.3 boss ---- Weather Log ---- --- Weather --- Time Inst Temp DP Dif Humid Wind Dir Dust,1um IRSC IRSCm -------------------------------------------------------------------------------- 07:12 4-8120A -2.1 -8.1 6.0 63 12 2 302 44.0 805 1.5 apog 07:12 4-8120A -2.1 -8.1 6.0 63 12 2 302 44.0 805 1.5 boss 07:21 4-8120A -2.1 -8.4 6.3 61 18 15 274 11.8 805 1.8 apog 07:29 4-8120A -2.1 -9.1 7.0 58 16 4 263 11.9 806 1.6 apog 07:29 4-8120A -2.1 -9.1 7.0 58 16 4 263 11.9 806 1.6 boss 07:37 4-8120A -2.2 -9.6 7.4 56 16 10 274 11.9 806 1.7 apog 08:08 3-8317A -2.4 -11.5 9.1 49 15 32 259 73.3 808 1.8 boss 08:26 3-8317A -2.6 -11.3 8.7 50 23 7 268 11.1 807 1.5 boss 08:44 3-8317A -2.8 -11.4 8.6 51 10 17 275 12.8 806 1.5 boss 09:01 3-8317A -2.9 -10.9 8.0 53 16 2 337 11.9 806 1.5 boss 09:38 1-8329A -3.2 -10.4 7.2 57 27 26 305 15.7 804 1.6 boss 09:54 1-8329A -3.3 -10.3 7.0 58 27 25 312 86.9 813 1.9 boss 10:11 1-8329A -3.3 -10.8 7.5 55 19 3 311 20.9 803 1.6 boss 10:28 1-8329A -3.2 -10.1 6.9 58 19 35 339 101.3 802 1.6 boss 11:01 3-8317A -3.4 -9.7 6.3 62 13 28 306 65.5 814 1.4 boss 11:37 1-8329A -3.2 -11.6 8.4 52 29 22 324 149.8 831 1.3 boss 11:53 1-8329A -3.0 -11.5 8.5 51 21 27 290 132.1 817 1.4 boss 12:10 1-8329A -3.2 -11.5 8.3 52 18 22 301 116.3 826 1.3 boss 12:27 1-8329A -3.2 -11.8 8.6 51 12 7 287 266.4 886 1.3 boss Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma ---- Telescope Status ---- 00:10Z started ventilation 00:58:21 encl25m : open; 02:52:21 encl25m : closed; 06:34:21 encl25m : open; 13:01:21 encl25m : closed; No interlocks bypassed Status at: 13:11:05 Telescope stowed at: 121.0, 30.0, -0.0 mount Instrument mounted: cartridge = 11 Counterweights at: 439, 439, 439, 438 LN2 autofill systems: Connected and turned on -180L LN2 dewar scales: SP1 152.1 lbs, 19.7 psi SP2 161.1 lbs, 22.4 psi -Temperatures: PMSSAvg = -4.0 (avg temp. of Pri) TelairAvg = -4.3 (avg temp. in enclosure) ---- Software Used ---- STUI : 1.6.0b1 alerts : trunk+svn149374 apo : trunk+svn158476 boss : v3_5 -boss.daq : 1.3.2 -boss.specMec : 3.0.4 gcamera : v2_0 guider : branch_guider_highalt_tracking+svn160865M hartmann : branch_updating_constants+svn159329 hub : Tag: sdss4_v3_0 mcp : v6_6_18 -mcp.plc : 31 -mcp.azFiduci : v1_114 -mcp.altFiduc : v1_114 -mcp.rotFiduc : v1_114 platedb : v2_3 sop : v2_18_2 tcc : 1.0.6 apogee : not availble apogeecal : not availble apogeeql : v1_5_7 -- Other software sdssProcedure : v3_06 APO-local : not availble idlspec2d (sos3) --- v5_7_7 plate-mapper3 ------ NOCVS:v5_1_9 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: 0.03 0.032 Axial B: -0.009 -0.026 Axial C: -0.021 -0.024 Trans D: -0.022 -0.014 Lateral E: 10.6934 10.7061 Lateral F: 12.319 12.3063 GALILS Commanded: 10850 -450 5750 13050 14750 14300 Actual: 10850 -450 5750 13050 14750 14300 SETMIR VALUES PrimDesOrient: 0.0 0.0 0.0 651.0 937.4 PrimOrient: -0.0 -0.0 0.0 651.08 936.55 SECONDARY: ---------- Focus: 0.0000 Air temp: 2 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -7.061 -7.065 Axial B: -7.115 -7.121 Axial C: -7.023 -7.028 Trans D: -0.187 -0.188 GALILS Commanded: 1581850 1579700 1593950 1400 -2000 Actual: 1581274 1578887 1590628 1400 -2000 SETMIR VALUES SecDesOrient: 1257.0 -3.8 0.0 0.0 9.58 SecOrient: 1256.99 -3.78 0.02 0.02 9.42 ---- Miscellaneous ---- --- Hartmann --- Time Inst r1 b1 move1 r1pred b1pred Temp r2 b2 move2 r2pred b2pred Temp -------------- ------------------------------------ ------------------------------------ 06:49 4-8120A -329 -1.0 -4 -325 -1.0 4.6 0 1.0 -157 157 0.5 3.7 06:49 4-8120A -329 -1.0 -4 -325 -1.0 4.6 0 1.0 -157 157 0.5 3.7 06:59 4-8120A -329 3.0 -646 317 1.0 1.1 -315 -3.0 315 -630 -2.0 0.4 07:56 3-8317A 2968 -3.0 1966 1002 3.1 0.0 1262 -8.9 2049 -787 -2.5 -0.5 09:29 1-8329A -4617 15.1 -4718 101 0.3 -1.1 -5049 11.9 -4415 -633 -2.0 -1.3 10:54 3-8317A 989 -5.0 1297 -308 -1.0 -1.6 946 -8.9 1891 -945 -3.0 -1.8 11:28 1-8329A -2968 10.1 -3090 122 0.4 -1.7 -5680 14.8 -5204 -476 -1.5 -2.0 ---- Problems in Detail ---- 03:35Z We have had trouble moving the telescope when we went to the initial start slew (60,60,60) at the start of the night. We have contacted Conor, Russell, DanL, and JohnP. We believe there may be a communication problem between tcc and mcp. We have restarted MCP twice (soft restart) and reverted to tcc version 1.0.5. We are seeing a large dtime. We have tried reconnecting to each axis through tcc. (command: tcc axis init rot). We are seeing in the STUI status window that the axes are "clock not set". We can move the telescope via MCPMenu window. We had to close the enclosure due to weather at 02:52Z. Telescope was moved via MCPMenu. Some errors from the log: 01:37:19 APO.telescope 380 tcc f text="'22329 axis stop' failed for az, alt, rot: Cancelled on queue by immediate priority command 'INIT'" 01:37:19 CmdDone: APO.telescope 380 tcc axis stop 01:37:28 .alerts 0 alerts w alert=mcp.heartbeat,serious,"The MCP's been dead for at least 274s",enabled,noack,"" 01:37:31 APO.telescope 381 tcc f text="Sub-command(s) failed: : 'UserCmd for az INIT, alt INIT, rot INIT' failed for az, alt, rot: Timed out" 01:37:31 CmdDone: APO.telescope 381 tcc axis init 01:37:38 .alerts 0 alerts w alert=mcp.heartbeat,serious,"The MCP's been dead for at least We have cleared the alerts actor since we were getting critical alerts for MCP and TCC heartbeat. More errors from the log (both tcc and mcp): 02:41:02 .hub 314 tcc : 02:41:02 CmdDone: 0 314 tcc mir status 02:41:11 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:11 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=64.999308,80.009566,nan; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 02:41:11 .tcc 0 tcc i AxisCmdState=Halted,Halted,NotAvailable; AxisErrCode=HaltRequested,HaltRequested,NotAvailable; AxisNoTrack=TTT 02:41:11 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:21 CmdStarted: apo.apo 707 tcc set weather humid=0.621, wdir=334.8, pressure=72051.982, wspeed=16.4, airtemp=0.3, sectrusstemp=-1.69 02:41:21 apo.apo 707 tcc i AirTemp=0.30 02:41:21 apo.apo 707 tcc i SecTrussTemp=-1.69 02:41:21 apo.apo 707 tcc i PrimF_BFTemp=0.31,-0.04 02:41:21 apo.apo 707 tcc i SecF_BFTemp=0.11,0.00 02:41:21 apo.apo 707 tcc i Pressure=72052.0 02:41:21 apo.apo 707 tcc i Humidity=0.62 02:41:21 apo.apo 707 tcc i TLapse=6.50 02:41:21 apo.apo 707 tcc i WSpeed=16.4 02:41:21 apo.apo 707 tcc i WDir=334.8 02:41:21 apo.apo 707 tcc i TimeStamp=4931635281.14 02:41:21 apo.apo 707 tcc : 02:41:21 CmdDone: apo.apo 707 tcc set weather humid=0.621, wdir=334.8, pressure=72051.982, wspeed=16.4, airtemp=0.3, sectrusstemp=-1.69 02:41:21 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:21 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=64.999308,80.009566,nan; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 02:41:21 .tcc 0 tcc i AxisCmdState=Halted,Halted,NotAvailable; AxisErrCode=HaltRequested,HaltRequested,NotAvailable; AxisNoTrack=TTT 02:41:21 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:31 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:31 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=64.999308,80.009566,nan; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 02:41:31 .tcc 0 tcc i AxisCmdState=Halted,Halted,NotAvailable; AxisErrCode=HaltRequested,HaltRequested,NotAvailable; AxisNoTrack=TTT 02:41:31 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:41 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:41 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=64.999308,80.009566,nan; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 02:41:41 .tcc 0 tcc i AxisCmdState=Halted,Halted,NotAvailable; AxisErrCode=HaltRequested,HaltRequested,NotAvailable; AxisNoTrack=TTT 02:41:41 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:49 CmdStarted: TU02.conor 26 tcc axis status 02:41:49 TU02.conor 26 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:41:50 TU02.conor 26 tcc f text="'12 axis status' failed for az, alt, rot: not connected" 02:41:50 CmdDone: TU02.conor 26 tcc axis status 02:42:00 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 02:42:00 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=64.999308,80.009566,nan; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 02:42:00 .tcc 0 tcc i AxisCmdState=Halted,Halted,NotAvailable; AxisErrCode=HaltRequested,HaltRequested,NotAvailable; AxisNoTrack=TTT 02:42:00 .tcc 0 tcc w axisConnState=Disconnected,Disconnected,Disconnected 03:29:01 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999391,29.995902,0.000024; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 03:29:01 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 03:29:02 .tcc 0 tcc i UserInfo=1,10.25.1.15 03:29:02 .tcc 0 tcc i UserInfo=2,10.25.1.250 03:29:02 .tcc 0 tcc i AzLim=-195.00,435.00,2.00,0.20,0.20 03:29:02 .tcc 0 tcc i AltLim=18.20,90.00,1.50,0.08,0.08 03:29:02 .tcc 0 tcc i RotLim=NaN,NaN,NaN,NaN,NaN 03:29:02 .tcc 0 tcc i AxisBadStatusMask=0x45800; AxisWarnStatusMask=0x610320FF; AxisMaxDTime=0.850 03:29:02 .tcc 0 tcc i DisabledProc 03:29:02 .tcc 0 tcc i StInterval=10.00,2.00,10.00; MaxUsers=25 03:29:02 .tcc 0 tcc i ScaleFac=1.000000; ScaleFacRange=1.000000,1.000000 03:29:02 .tcc 0 tcc i AirTemp=0.00 03:29:02 .tcc 0 tcc i SecTrussTemp=-0.04 03:29:02 .tcc 0 tcc i PrimF_BFTemp=0.28,0.00 03:29:02 .tcc 0 tcc i SecF_BFTemp=0.11,0.00 03:29:02 .tcc 0 tcc i Pressure=72072.0 03:29:02 .tcc 0 tcc i Humidity=0.58 03:29:02 .tcc 0 tcc i TLapse=6.50 03:29:02 .tcc 0 tcc i WSpeed=12.6 03:29:02 .tcc 0 tcc i WDir=314.3 03:29:02 .tcc 0 tcc i TimeStamp=4931638142.16 03:29:02 .tcc 0 tcc : 03:29:02 .tcc 0 tcc i UserInfo=1,10.25.1.15 03:29:11 .tcc 0 tcc e BadAzStatus; AzStat=120.999395,0.000000,4931625600.35534,0x00043800; AzDTime=12551.05 03:29:11 .tcc 0 tcc e BadAltStatus; AltStat=29.995906,0.000000,4931625600.43261,0x00043800; AltDTime=12551.06 03:29:11 .tcc 0 tcc e BadRotStatus; RotStat=0.000030,0.000000,4931625600.51338,0x00043800; RotDTime=12551.06 03:29:11 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999395,29.995906,0.000030; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 03:29:11 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 03:29:21 CmdStarted: apo.apo 755 tcc set weather humid=0.621, wdir=306.2, pressure=72072.300, wspeed=13.8, airtemp=0.3, sectrusstemp=-1.59 03:29:21 apo.apo 755 tcc i AirTemp=0.30 03:29:21 apo.apo 755 tcc i SecTrussTemp=-1.59 03:29:21 apo.apo 755 tcc i PrimF_BFTemp=0.28,0.00 03:29:21 apo.apo 755 tcc i SecF_BFTemp=0.11,0.00 03:29:21 apo.apo 755 tcc i Pressure=72072.3 03:29:21 apo.apo 755 tcc i Humidity=0.62 03:29:21 apo.apo 755 tcc i TLapse=6.50 03:29:21 apo.apo 755 tcc i WSpeed=13.8 03:29:21 apo.apo 755 tcc i WDir=306.2 03:29:21 apo.apo 755 tcc i TimeStamp=4931638161.16 03:29:21 apo.apo 755 tcc : 03:29:21 CmdDone: apo.apo 755 tcc set weather humid=0.621, wdir=306.2, pressure=72072.300, wspeed=13.8, airtemp=0.3, sectrusstemp=-1.59 03:29:21 .tcc 0 tcc e BadAzStatus; AzStat=120.999395,0.000000,4931625600.59565,0x00043800; AzDTime=12561.13 03:29:21 .tcc 0 tcc e BadAltStatus; AltStat=29.995906,0.000000,4931625600.74963,0x00043800; AltDTime=12561.06 03:29:21 .tcc 0 tcc e BadRotStatus; RotStat=0.000030,0.000000,4931625600.83256,0x00043800; RotDTime=12561.06 03:29:21 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999395,29.995906,0.000030; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 03:29:21 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 03:29:32 .tcc 0 tcc e BadAzStatus; AzStat=120.999395,0.000000,4931625600.91718,0x00043800; AzDTime=12571.13 03:29:32 .tcc 0 tcc e BadAltStatus; AltStat=29.995902,0.000000,4931625600.06935,0x00043800; AltDTime=12572.06 03:29:32 .tcc 0 tcc e BadRotStatus; RotStat=0.000030,0.000000,4931625600.15270,0x00043800; RotDTime=12572.06 03:29:32 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999395,29.995902,0.000030; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 03:29:32 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 03:29:42 .tcc 0 tcc e BadAzStatus; AzStat=120.999391,0.000000,4931625600.23466,0x00043800; AzDTime=12582.06 03:29:42 .tcc 0 tcc e BadAltStatus; AltStat=29.995909,0.000000,4931625600.31317,0x00043800; AltDTime=12582.06 03:29:42 .tcc 0 tcc e BadRotStatus; RotStat=0.000024,0.000000,4931625600.39413,0x00043800; RotDTime=12582.06 03:29:42 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999391,29.995909,0.000024; ObjInstAng=nan,nan,4931634276.7422695; SpiderInstAng=nan,nan,4931634276.7422695 03:29:42 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 1:24:25 .mcp 0 mcp i trace="MGT: bad rot state ABORT_EVENT: ID_ERROR_LIMIT" 01:24:26 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:24:26 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:24:31 .mcp 0 mcp i ab_status=0,1,0,14 01:24:31 .mcp 3145519 mcp i trace="ERR: init cannot take semLatch" 01:24:31 .mcp 3145519 mcp ! command="init" 01:24:31 .mcp 3145519 mcp d trace="set PID coeffs. axis=1, block=1" 01:24:32 .mcp 3145520 mcp d trace="INIT axis rotator: not running: ABORT_EVENT" 01:24:32 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:24:33 .mcp 3145520 mcp d trace="set PID coeffs. axis=2, block=1" 01:24:34 .mcp 0 mcp i ab_status=0,1,0,0 01:24:34 .mcp 3145521 mcp i trace="MOVE CMD: requested time=0.000000" 01:24:34 .mcp 3145521 mcp i trace=" current time=5074.707820" 01:24:34 .mcp 3145521 mcp ! command="move" 01:24:35 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:24:35 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:24:35 .mcp 0 mcp i ab_status=0,0,0,0 01:24:35 .mcp 0 mcp i cwPositions=439,438,439,440 01:24:35 .mcp 3145522 mcp d trace="set PID coeffs. axis=0, block=1" 01:24:37 .mcp 3145523 mcp i trace="MOVE CMD: requested time=0.000000" 01:24:37 .mcp 3145523 mcp i trace=" current time=5077.145715" 01:24:37 .mcp 3145523 mcp ! command="move" 01:24:37 .mcp 3145524 mcp i trace="MOVE CMD: requested time=5074.259700" 01:24:37 .mcp 3145524 mcp i trace=" current time=5077.554286" 01:24:37 .mcp 3145524 mcp ! command="move" 01:24:38 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:24:38 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:24:38 .mcp 3145526 mcp d trace="set PID coeffs. axis=0, block=1" 01:24:40 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:24:40 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:24:40 .mcp 3145527 mcp d trace="set PID coeffs. axis=1, block=1" 01:24:42 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:24:42 .mcp 3145528 mcp d trace="set PID coeffs. axis=2, block=1" 01:24:43 .mcp 0 mcp i aliveAt=1424913866 01:24:44 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:24:44 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:24:44 .mcp 3145530 mcp d trace="set PID coeffs. axis=0, block=1" 01:24:46 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:24:46 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:24:46 .mcp 3145534 mcp d trace="set PID coeffs. axis=0, block=1" 01:24:48 .mcp 0 mcp i trace="MGT: bad alt state ABORT_EVENT: ID_ERROR_LIMIT" 01:24:49 .mcp 0 mcp i ab_status=0,0,14,0 01:24:49 .mcp 0 mcp i cwPositions=438,439,438,439 01:24:49 .mcp 0 mcp i altMaxAccRequested=1.584628 01:24:49 .mcp 0 mcp i altMaxVelRequested=8.567793 01:24:49 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:24:49 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:24:49 .mcp 0 mcp i altMaxVelRequested=8.554279 01:24:50 .mcp 3145538 mcp d trace="set PID coeffs. axis=0, block=1" 01:24:50 .mcp 0 mcp i altMaxVelRequested=8.553942 01:24:50 .mcp 0 mcp i altMaxVelRequested=8.553604 01:24:50 .mcp 0 mcp i altMaxVelRequested=8.553267 01:24:50 .mcp 0 mcp i altMaxVelRequested=8.552929 01:24:50 .mcp 0 mcp i altMaxVelRequested=8.552591 01:24:51 .mcp 0 mcp i ab_I9_L0=0x00340000 01:24:51 .mcp 0 mcp i ab_O12_L0=0x02090000 01:24:51 .mcp 0 mcp i altMaxVelRequested=8.552254 01:24:51 .mcp 0 mcp i altMaxVelRequested=8.551915 01:24:51 .mcp 0 mcp i altMaxVelRequested=8.551578 01:24:52 .mcp 3145539 mcp d trace="INIT axis altitude: not running: ABORT_EVENT" 01:24:52 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:24:52 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:24:52 .mcp 0 mcp i altMaxVelRequested=8.551240 01:24:52 .mcp 0 mcp i altMaxVelRequested=8.550902 01:24:52 .mcp 0 mcp i altMaxVelRequested=8.550565 01:24:53 .mcp 3145539 mcp d trace="set PID coeffs. axis=1, block=1" 01:24:53 .mcp 0 mcp i altMaxVelRequested=8.550227 01:24:53 .mcp 0 mcp i altMaxVelRequested=8.549890 01:24:53 .mcp 0 mcp i altMaxVelRequested=8.549552 01:24:53 .mcp 0 mcp i altMaxVelRequested=8.549214 01:24:54 .mcp 0 mcp i altMaxVelRequested=8.548877 01:24:54 .mcp 0 mcp i altMaxVelRequested=8.548539 01:24:54 .mcp 0 mcp i ab_status=0,0,1,0 01:24:54 .mcp 0 mcp i ab_I9_L0=0x00540000 01:24:54 .mcp 0 mcp i ab_O12_L0=0x02050000 01:24:54 .mcp 0 mcp i altMaxVelRequested=8.567793 01:24:54 .mcp 0 mcp i altMaxVelRequested=8.554279 01:24:55 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:24:55 .mcp 0 mcp i altMaxVelRequested=8.553942 01:24:55 .mcp 3145540 mcp d trace="set PID coeffs. axis=2, block=1" 01:24:55 .mcp 0 mcp i altMaxVelRequested=8.553604 01:24:55 .mcp 0 mcp i altMaxVelRequested=8.553267 01:24:55 .mcp 0 mcp i altMaxVelRequested=8.552929 01:24:56 .mcp 0 mcp i altMaxVelRequested=8.552591 01:24:56 .mcp 0 mcp i altMaxVelRequested=8.552254 01:24:56 .mcp 0 mcp i ab_status=0,0,1,14 01:24:56 .mcp 0 mcp i altMaxVelRequested=8.551915 01:24:57 .mcp 0 mcp i trace="MGT: bad rot state ABORT_EVENT: ID_ERROR_LIMIT" 01:24:57 .mcp 0 mcp i rotMaxAccRequested=-2.302623 01:24:57 .mcp 0 mcp i altMaxVelRequested=8.551578 01:24:57 .mcp 0 mcp i rotMaxVelRequested=17.528719 01:24:57 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:24:57 .mcp 0 mcp i altMaxVelRequested=8.551240 01:24:57 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:24:57 .mcp 0 mcp i rotMaxVelRequested=17.487926 01:24:57 .mcp 3145541 mcp d trace="set PID coeffs. axis=1, block=1" 01:24:58 .mcp 0 mcp i altMaxVelRequested=8.550902 01:24:58 .mcp 0 mcp i rotMaxVelRequested=17.487236 01:24:58 .mcp 0 mcp i altMaxVelRequested=8.550565 01:24:58 .mcp 0 mcp i rotMaxVelRequested=17.486546 01:24:58 .mcp 0 mcp i altMaxVelRequested=8.550227 01:24:58 .mcp 0 mcp i rotMaxVelRequested=17.485853 01:24:58 .mcp 0 mcp i altMaxVelRequested=8.549890 01:24:58 .mcp 0 mcp i rotMaxVelRequested=17.485163 01:24:58 .mcp 0 mcp i altMaxVelRequested=8.549552 01:24:58 .mcp 0 mcp i rotMaxVelRequested=17.484472 01:24:58 .mcp 0 mcp i altMaxVelRequested=8.549214 01:24:59 .mcp 0 mcp i ab_status=0,0,0,14 01:24:59 .mcp 0 mcp i rotMaxVelRequested=17.483782 01:24:59 .mcp 0 mcp i altMaxVelRequested=8.548877 01:24:59 .mcp 0 mcp i rotMaxVelRequested=17.483089 01:24:59 .mcp 0 mcp i altMaxVelRequested=8.548539 01:24:59 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:24:59 .mcp 0 mcp i rotMaxVelRequested=17.482399 01:24:59 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:24:59 .mcp 0 mcp i rotMaxVelRequested=17.481709 01:24:59 .mcp 3145544 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:00 .mcp 0 mcp i rotMaxVelRequested=17.481016 01:25:00 .mcp 0 mcp i rotMaxVelRequested=17.480326 01:25:00 .mcp 0 mcp i rotMaxVelRequested=17.479635 01:25:00 .mcp 0 mcp i rotMaxVelRequested=17.478945 01:25:00 .mcp 0 mcp i rotMaxVelRequested=17.478252 01:25:01 .mcp 0 mcp i rotMaxVelRequested=17.477562 01:25:01 .mcp 3145546 mcp d trace="INIT axis rotator: not running: ABORT_EVENT" 01:25:01 .mcp 0 mcp i rotMaxVelRequested=17.476871 01:25:01 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:01 .mcp 0 mcp i rotMaxVelRequested=17.476181 01:25:01 .mcp 0 mcp i rotMaxVelRequested=17.528719 01:25:02 .mcp 3145546 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:02 .mcp 0 mcp i rotMaxVelRequested=17.487926 01:25:02 .mcp 0 mcp i rotMaxVelRequested=17.487236 01:25:02 .mcp 0 mcp i rotMaxVelRequested=17.486546 01:25:02 .mcp 0 mcp i rotMaxVelRequested=17.485853 01:25:03 .mcp 0 mcp i rotMaxVelRequested=17.485163 01:25:03 .mcp 0 mcp i rotMaxVelRequested=17.484472 01:25:03 .mcp 0 mcp i ab_status=0,0,0,0 01:25:03 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:25:03 .mcp 0 mcp i rotMaxVelRequested=17.483782 01:25:03 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:25:03 .mcp 0 mcp i rotMaxVelRequested=17.483089 01:25:04 .mcp 3145547 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:04 .mcp 0 mcp i rotMaxVelRequested=17.482399 01:25:04 .mcp 0 mcp i rotMaxVelRequested=17.481709 01:25:04 .mcp 0 mcp i rotMaxVelRequested=17.481016 01:25:04 .mcp 0 mcp i rotMaxVelRequested=17.480326 01:25:04 .mcp 0 mcp i rotMaxVelRequested=17.479635 01:25:05 .mcp 0 mcp i rotMaxVelRequested=17.478945 01:25:05 .mcp 0 mcp i rotMaxVelRequested=17.478252 01:25:05 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:05 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:06 .mcp 0 mcp i rotMaxVelRequested=17.477562 01:25:06 .mcp 3145548 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:06 .mcp 0 mcp i rotMaxVelRequested=17.476871 01:25:06 .mcp 0 mcp i rotMaxVelRequested=17.476181 01:25:07 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:07 .mcp 3145549 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:09 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:09 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:09 .mcp 3145552 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:11 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:25:11 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:25:11 .mcp 3145554 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:13 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:13 .mcp 3145556 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:14 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:25:14 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:25:15 .mcp 3145557 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:16 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:17 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:17 .mcp 3145558 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:18 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:18 .mcp 3145559 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:20 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:20 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:20 .mcp 3145561 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:23 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:25:23 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:25:23 .mcp 3145565 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:24 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:25 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:25 .mcp 3145566 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:26 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:26 .mcp 3145567 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:27 .mcp 0 mcp i trace="MGT: bad alt state ABORT_EVENT: ID_ERROR_LIMIT" 01:25:27 .mcp 0 mcp i altMaxAccRequested=-1.152456 01:25:27 .mcp 0 mcp i altMaxVelRequested=8.773074 01:25:27 .mcp 0 mcp i altMaxVelRequested=8.752658 01:25:27 .mcp 0 mcp i altMaxVelRequested=8.752312 01:25:28 .mcp 3145568 mcp d trace="INIT axis altitude: not running: ABORT_EVENT" 01:25:28 .mcp 0 mcp i altMaxVelRequested=8.751966 01:25:28 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:28 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:28 .mcp 0 mcp i altMaxVelRequested=8.751620 01:25:28 .mcp 0 mcp i altMaxVelRequested=8.751274 01:25:29 .mcp 0 mcp i altMaxVelRequested=8.750928 01:25:29 .mcp 3145568 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:29 .mcp 0 mcp i altMaxVelRequested=8.750583 01:25:29 .mcp 0 mcp i altMaxVelRequested=8.750237 01:25:29 .mcp 0 mcp i altMaxVelRequested=8.749890 01:25:29 .mcp 0 mcp i altMaxVelRequested=8.749545 01:25:30 .mcp 0 mcp i altMaxVelRequested=8.749199 01:25:30 .mcp 0 mcp i altMaxVelRequested=8.748853 01:25:30 .mcp 0 mcp i altMaxVelRequested=8.748507 01:25:30 .mcp 0 mcp i altMaxVelRequested=8.748161 01:25:30 .mcp 0 mcp i altMaxVelRequested=8.747815 01:25:31 .mcp 0 mcp i altMaxVelRequested=8.747470 01:25:31 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:31 .mcp 0 mcp i altMaxVelRequested=8.747124 01:25:31 .mcp 3145572 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:31 .mcp 0 mcp i altMaxVelRequested=8.746778 01:25:31 .mcp 0 mcp i altMaxVelRequested=8.773074 01:25:31 .mcp 0 mcp i altMaxVelRequested=8.752658 01:25:32 .mcp 0 mcp i altMaxVelRequested=8.752312 01:25:32 .mcp 0 mcp i altMaxVelRequested=8.751966 01:25:32 .mcp 0 mcp i altMaxVelRequested=8.751620 01:25:32 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:32 .mcp 0 mcp i altMaxVelRequested=8.751274 01:25:33 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:33 .mcp 0 mcp i altMaxVelRequested=8.750928 01:25:33 .mcp 3145574 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:33 .mcp 0 mcp i altMaxVelRequested=8.750583 01:25:33 .mcp 0 mcp i altMaxVelRequested=8.750237 01:25:33 .mcp 0 mcp i altMaxVelRequested=8.749890 01:25:34 .mcp 0 mcp i altMaxVelRequested=8.749545 01:25:34 .mcp 0 mcp i trace="MGT: bad rot state ABORT_EVENT: ID_ERROR_LIMIT" 01:25:34 .mcp 0 mcp i altMaxVelRequested=8.749199 01:25:34 .mcp 0 mcp i rotMaxAccRequested=-7.771354 01:25:34 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:25:34 .mcp 0 mcp i altMaxVelRequested=8.748853 01:25:34 .mcp 0 mcp i rotMaxVelRequested=12.743581 01:25:34 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:25:35 .mcp 0 mcp i rotMaxVelRequested=12.704364 01:25:35 .mcp 3145575 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:35 .mcp 0 mcp i altMaxVelRequested=8.748507 01:25:35 .mcp 0 mcp i rotMaxVelRequested=12.703861 01:25:35 .mcp 0 mcp i altMaxVelRequested=8.748161 01:25:35 .mcp 0 mcp i altMaxVelRequested=8.747815 01:25:35 .mcp 0 mcp i rotMaxVelRequested=12.703360 01:25:35 .mcp 0 mcp i altMaxVelRequested=8.747470 01:25:35 .mcp 0 mcp i rotMaxVelRequested=12.702857 01:25:35 .mcp 0 mcp i altMaxVelRequested=8.747124 01:25:35 .mcp 0 mcp i rotMaxVelRequested=12.702354 01:25:36 .mcp 0 mcp i altMaxVelRequested=8.746778 01:25:36 .mcp 0 mcp i rotMaxVelRequested=12.701852 01:25:36 .mcp 0 mcp i rotMaxVelRequested=12.701349 01:25:36 .mcp 0 mcp i rotMaxVelRequested=12.700847 01:25:36 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:36 .mcp 0 mcp i rotMaxVelRequested=12.700344 01:25:36 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:37 .mcp 3145579 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:37 .mcp 0 mcp i rotMaxVelRequested=12.699842 01:25:37 .mcp 0 mcp i rotMaxVelRequested=12.699340 01:25:37 .mcp 0 mcp i rotMaxVelRequested=12.698837 01:25:37 .mcp 0 mcp i rotMaxVelRequested=12.698335 01:25:38 .mcp 0 mcp i rotMaxVelRequested=12.697832 01:25:38 .mcp 0 mcp i rotMaxVelRequested=12.697330 01:25:38 .mcp 0 mcp i rotMaxVelRequested=12.696828 01:25:38 .mcp 3145580 mcp d trace="INIT axis rotator: not running: ABORT_EVENT" 01:25:38 .mcp 0 mcp i trace="Resetting amp for axis rotator: 4" 01:25:38 .mcp 0 mcp i rotMaxVelRequested=12.696325 01:25:39 .mcp 0 mcp i rotMaxVelRequested=12.695823 01:25:39 .mcp 0 mcp i rotMaxVelRequested=12.743581 01:25:39 .mcp 3145580 mcp d trace="set PID coeffs. axis=2, block=1" 01:25:39 .mcp 0 mcp i rotMaxVelRequested=12.704364 01:25:39 .mcp 0 mcp i rotMaxVelRequested=12.703861 01:25:39 .mcp 0 mcp i rotMaxVelRequested=12.703360 01:25:40 .mcp 0 mcp i rotMaxVelRequested=12.702857 01:25:40 .mcp 0 mcp i rotMaxVelRequested=12.702354 01:25:40 .mcp 0 mcp i rotMaxVelRequested=12.701852 01:25:40 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 0" 01:25:40 .mcp 0 mcp i rotMaxVelRequested=12.701349 01:25:40 .mcp 0 mcp i trace="Resetting amp for axis azimuth: 1" 01:25:41 .mcp 3145581 mcp d trace="set PID coeffs. axis=0, block=1" 01:25:41 .mcp 0 mcp i rotMaxVelRequested=12.700847 01:25:41 .mcp 0 mcp i rotMaxVelRequested=12.700344 01:25:41 .mcp 0 mcp i rotMaxVelRequested=12.699842 01:25:41 .mcp 0 mcp i rotMaxVelRequested=12.699340 01:25:41 .mcp 0 mcp i rotMaxVelRequested=12.698837 01:25:42 .mcp 0 mcp i rotMaxVelRequested=12.698335 01:25:42 .mcp 0 mcp i rotMaxVelRequested=12.697832 01:25:42 .mcp 0 mcp i trace="Resetting amp for axis altitude: 2" 01:25:42 .mcp 0 mcp i trace="Resetting amp for axis altitude: 3" 01:25:42 .mcp 0 mcp i rotMaxVelRequested=12.697330 01:25:42 .mcp 3145582 mcp d trace="set PID coeffs. axis=1, block=1" 01:25:43 .mcp 0 mcp i rotMaxVelRequested=12.696828 01:25:43 .mcp 0 mcp i rotMaxVelRequested=12.696325 01:25:43 .mcp 0 mcp i rotMaxVelRequested=12.695823 01:25:44 .mcp 3145462 mcp i azMsOn=false 01:25:45 .mcp 0 mcp i aliveAt=1424913928 01:25:59 .mcp 0 mcp i trace="MGT: bad alt state ABORT_EVENT: ID_ERROR_LIMIT" 04:08:03 APO.telescope 425 tcc f text="Sub-command(s) failed: : 'UserCmd for az INIT, alt INIT, rot INIT' failed for az, alt, rot: Timed out" 04:08:03 CmdDone: APO.telescope 425 tcc axis init 04:08:22 APO.telescope 427 tcc f text="Sub-command(s) failed: : 'UserCmd for az INIT, alt INIT, rot INIT' failed for az, alt, rot: Timed out" 04:08:22 CmdDone: APO.telescope 427 tcc axis init 04:09:11 APO.telescope 429 tcc f text="Sub-command(s) failed: : 'UserCmd for az INIT, alt INIT, rot INIT' failed for az, alt, rot: Timed out" 04:09:11 CmdDone: APO.telescope 429 tcc axis init 04:10:01 APO.telescope 430 tcc f text="'120 axis status' failed for az, alt, rot: Cancelled on queue by immediate priority command 'INIT'" 04:10:01 CmdDone: APO.telescope 430 tcc axis status 04:11:38 .tcc 0 tcc w axisConnState=Connecting,Connecting,NotAvailable 04:11:38 .hub 319 tcc w axisConnState=Connecting,Connecting,NotAvailable 04:11:38 .hub 324 tcc w axisConnState=Connecting,Connecting,NotAvailable 04:11:39 .hub 324 tcc e BadAzStatus; AzStat=120.999388,0.000000,4931625600.19919,0x00043000; AzDTime=15099.06 04:11:39 .tcc 0 tcc w axisConnState=Connected,Connecting,NotAvailable 04:11:39 .hub 324 tcc e BadAzStatus; AzStat=120.999407,0.000000,4931625600.28335,0x00043000; AzDTime=15099.10 04:11:39 .hub 324 tcc e BadAltStatus; AltStat=29.995509,0.000000,4931625600.54473,0x00043000; AltDTime=15099.12 04:11:43 APO.telescope 432 tcc e BadAzStatus; AzStat=120.999407,0.000000,4931625600.82697,0x00043800; AzDTime=15103.08 04:11:44 APO.telescope 432 tcc e BadAltStatus; 04:23:54 .tcc 0 tcc w axisConnState=Disconnecting,Connected,NotAvailable 04:23:54 .tcc 0 tcc w axisConnState=Disconnecting,Disconnecting,NotAvailable 04:23:58 .tcc 0 tcc w axisConnState=Disconnecting,Disconnecting,NotAvailable 04:23:58 .tcc 0 tcc w axisConnState=Disconnecting,Disconnecting,NotAvailable 04:24:08 .tcc 0 tcc w axisConnState=Disconnecting,Disconnecting,NotAvailable 04:24:08 .tcc 0 tcc w axisConnState=Disconnecting,Disconnecting,NotAvailable 04:47:33 CmdDone: APO.fs 1003 keys getFor=tcc windSpeed yourUserNum 04:47:33 APO.fs 1017 keys f keysTxt="actor tcc is not connected" 04:47:33 CmdDone: APO.fs 1017 keys getFor=tcc airTemp axePos altDTime altLim altMSStat altStat axisBadStatusMask axisCmdState axisErrCode axisWarnStatusMask azDTime azLim azMSStat azStat boresight calibOff currArcOff currUsers gBinFactor gBoxSize 04:47:33 APO.fs 1019 keys f keysTxt="actor tcc is not connected" 04:47:33 CmdDone: APO.fs 1019 keys getFor=tcc instFocus instPos ipConfig lst maxUsers objArcOff objDist objInstAng objMag objName objNetPos objOff objObs objPM objPos objSys objWavelength objZPMPos pressure primActMount 04:47:33 APO.fs 1021 keys f keysTxt="actor alerts is not connected" 04:47:33 CmdDone: APO.fs 1021 keys getFor=alerts version activeAlerts disabledAlertRules downInstruments instrumentNames text 04:47:33 APO.fs 1022 keys f keysTxt="actor tcc is not connected" 04:47:33 CmdDone: APO.fs 1022 keys getFor=tcc gCamCmd gCamID gCamInfo gCamReply gcFilt gcFocus gcFocusLim gcNFilt gcNomFocus gcPist gcView gFitChiSq gImCtr gImLim gImScale gPCentroid gPCtr gPDisable gPFindStar gPLim 04:47:33 APO.fs 1034 keys f keysTxt="actor tcc is not connected" 04:47:33 CmdDone: APO.fs 1034 keys getFor=tcc gPNoStar gPRotGImAng gPRotXY gProbe gResid gSDataPix gSMag gSName gSPM gSPos gSSys gSWavelength gScanSize gStats guideOff humidity iimCtr iimLim iimScale inst 04:47:33 APO.fs 1049 hub f NoTarget="the target named tcc is not connected" 04:47:33 APO.fs 1050 keys f keysTxt="actor tcc is not connected" 04:47:33 CmdDone: APO.fs 1050 keys getFor=tcc secConstRMS secDesOrient secF_BFTemp secFocus secOrient secStatus secTrussTemp slewAdvTime spiderInstAng tLapse tccPos tccStatus tdiCorr useCheby userInfo userNum ut1 utc_TAI version windDir 04:47:33 APO.fs 1051 keys f keysTxt="actor tcc is not connected" 04:47:33 CmdDone: APO.fs 1051 keys getFor=tcc primCmdMount primConstRMS primDesOrient primF_BFTemp primOrient primStatus ptErrProbe rotDTime rotID rotInstXYAng rotLim rotMSStat rotOffsetScale rotPos rotStat rotType scaleFac scaleFacRange secActMount secCmdMount 04:48:21 apo.apo 834 hub f NoTarget="the target named tcc is not connected" 04:48:21 .apo 160211 apo w text='Failed to set TCC weather' 04:48:29 APO.kgrabow 138 hub f NoTarget="the target named tcc is not connected" 04:49:13 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:13 .hub 329 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:13 .hub 334 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:13 .hub 334 tcc f text="'9 axis status' failed for az, alt: not connected" 04:49:13 CmdDone: 0 334 tcc axis status 04:49:21 APO.kgrabow 140 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:21 APO.kgrabow 140 tcc f text="'12 axis status' failed for az, alt: not connected" 04:49:21 CmdDone: APO.kgrabow 140 tcc axis status 04:49:30 .tcc 0 tcc w axisConnState=Failed,Connecting,NotAvailable 04:49:30 .tcc 0 tcc w axisConnState=Connecting,Connecting,NotAvailable 04:49:30 .tcc 0 tcc w axisConnState=Connecting,Failed,NotAvailable 04:49:30 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:30 APO.kgrabow 141 tcc f text="Sub-command(s) failed: : AxisDevice(alt) failed to connect: connection failed; : AxisDevice(az) failed to connect: connection failed" 04:49:30 CmdDone: APO.kgrabow 141 tcc axis connect 04:49:31 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:31 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:41 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:41 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:51 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:49:51 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:01 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:02 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:11 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:11 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:21 .tcc 0 tcc w axisConnState=Connecting,Failed,NotAvailable 04:50:21 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:21 APO.kgrabow 142 tcc f text="Sub-command(s) failed: : AxisDevice(az) failed to connect: connection failed" 04:50:21 CmdDone: APO.kgrabow 142 tcc axis connect az 04:50:21 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:21 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:30 APO.parejkoj 1143 keys f keysTxt="actor alerts is not connected" 04:50:30 CmdDone: APO.parejkoj 1143 keys getFor=alerts version activeAlerts disabledAlertRules downInstruments instrumentNames text 04:50:31 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:31 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:41 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:41 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:51 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:50:51 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:51:01 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:51:01 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:51:11 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:51:11 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:51:21 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 04:51:22 .tcc 0 tcc w axisConnState=Failed,Failed,NotAvailable 05:00Z Problem resolved. Summary: not sure how the time got out of sync between tcc and mcp. Mcp was not responsive - unknown. The was dumping a long sequence of MURMUR MESSAGES to the console: "DID48 interrupt with no set time." This was indicative of the mcp not having the time set each time it received a 1PPS signal. We searched for potential problems with the 1PPS, but everything appeared fine in the lower enclosure. The root of the problem was the fact that the mcp had not yet received set.time post-boot. See below for details. Every time we issue tcc axis init, the tcc and mcp should sync their times. The old tcc would set the time via "set.time" command when tcc axis init was issued. The new tcc does not do this. On start up, the MCP doesn't have the correct time. MCP does not have an NTP client, but calls "sntpcTimeGet" (vxworks function) to set the time from the APO NTP server (see setTimeFromNTP in mcp/src/timers.c, called from set_time_cmd). The old tcc spare machine died. The solution to the problem is on axis init, issue a set time command. Conor opened ticket #2292 to deal with this problem. Interestingly, mcp set.time is not "set your time to the time I specify," although it takes an optional numerical argument. It tells the mcp to "go find the time and set yourself appropriately." As with other mcp commands, it is not documented... MCP time got out of sync, and could have been fixed by issuing the following command: mcp set.time so sayeth JohnP.