Subject: 2.5m obslog 20150531 (SJD 57174) From: kgrabowski@apo.nmsu.edu Date: Mon, 1 Jun 2015 11:22:57 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday May 31, 2015 (SJD 57174) ---- Observing Team ---- Dan Oravetz Katie Grabowski Support: John Parejko (ecam pointing model, guider) Russell Owen (tcc) Elena Malanushenko APOGEE-S Training: Garrett Ebelke Christian Nitschelm Andres Vargas Jamie Almeida ---- Observing Summary ---- Opened late due to radar and clouds. Switched between test and previous tcc version when we couldn't set inst=engcam. Russell rebuilt the file without changing anything and then it was successful. JohnP updated the guider so the engineering camera images were shown in the window. Took 20-star pointing model in engcam frames 104-132, with up to 136 including the rotator at the end. Minor hiccups along the way detailed in log and problems section. APOGEE: ======== Cart 2; plate 8579, field=K10_079+12 2xABBA Missing fibers: [157, 163, 195, 227] Faint fibers: [217] Cart 4; plate 7537, field=094+00 2xABBA Missing fibers: [20, 39, 195, 278] Faint fibers: [129, 173, 197, 253, 280, 284] Mounted Cart 9 and started morning calibrations. ---- Observing Plan ---- Sun 31 MAY 2015 57174.0 Night Engineering & APOGEE-2 ( 8.5 h) ---- Observing Log ---- Afternoon Checkout ================== All systems alive and responsive. Mounted engineering camera for 20-point pointing model. Homed the stage, took a dark stack (20 sec, stack=9), and experimented with the flat exposure time. The documentation says to try a 5s exposure and adjust in order to get ~40k counts. The last time this twilight pointing-model test was attempted the flat was 35 sec. resulting in ~22k counts. I tried 5s and then 30s and was getting nowhere near 40k counts. A 90s flat resulted in ~14k counts. It seems that the ND filter is in place and we will amend the documentation to account for exposure-time variations when the ND filter is in place. Ventilation delayed as we have nearby storms. Night ===== Restarted tcc to test version 1.2.0b1. In testing the movement of the axes after restarting, the rotator began oscillating before slewing from stow to instrument change, and the axis for the rotator halted. In subsequent tests, the rotator and azimuth axes slewed properly. 02:45Z Started ventilation 04:12:42 encl25m : open; Reverted to current version of tcc 04:52:37 CmdStarted: APO.kgrabow 10 tcc show version 04:52:37 APO.kgrabow 10 tcc : Version="1.1.0" 04:52:37 CmdDone: APO.kgrabow 10 tcc show version Altitude and azimuth brakes came on during slew from (60,60,60) to the first pointing model field. More details in the log. 5:04 JohnP had to fix and restart the guider 5:20 restarted guider again 5:28 restarted tcc 05:30:08 CmdStarted: APO.doravetz 244 tcc set inst=engcam Success! Could have been a corrupted build, because Russell didn't change anything except add a few print statements and rebuild. 6:30 Processed guider images had the dark subtracted, but were not normalized. Once the flats were normalized, the camera was able to find the star more consistently. focus sweeps: /data/ecam/57174/proc-gimg-0040.fits first star to appear in the guider window ecam focus sweep window: Failed: 'Model' object has no attribute 'ecam_star' because observers don't have updated actorkeys. JohnP got the script running, but it still had trouble finding the star. Best focus=-175 06:35Z seeing approximately 1" fwhm Check Optical Axis Position on CCD with eCamRotTest: Alt axis halted during rotator slew to 0 object. Assuming the engcam is well centered in the rotator so we can take data. Skipping straight to the pointing model JohnP started 20-star pointing model grid with the Pointing Data script that will be included in the next STUI. We got fiducial errors after a slew to an azimuth of 430 degrees. 06:55:15 Az: ind=40, deg=407.3, err= 3857674 error is too large to be corrected ( > 600) 06:55:24 Az: ind=40, deg=422.3, err= 7707327 error is too large to be corrected ( > 600) 06:57:52 Az: ind=40, deg=422.3, err= 7707739 error is too large to be corrected ( > 600) 06:58:00 Az: ind=40, deg=407.3, err= 3857946 error is too large to be corrected ( > 600) 07:05:49 .tcc 0 tcc w BadAzStatus; AzStat=170.445789,0.000000,4939859149.39183,0x00000800; AzDTime=0.14 07:05:49 APO.parejkoj_nontest 425 tcc : 07:05:49 CmdDone: APO.parejkoj_nontest 425 tcc axis init After this, we lost the stars in the guider window.Paused the Pointing Data script to go to 119.3,55.2, 271.8 to take a frame in a known position. We thought that possibly the altitude axis halting would have caused the petals to close and decrease counts, but as the FITS headers state the petals were open until gimg-00129.fits (the most recent image), so it's more likely that the petals automatically closed as the telescope approached the DIMM at a low altitude. 2015-06-01 05:17:11.9206Z -------------------------------------------------------------------------------- 06:51 19-0A 1.000000 0 1216 -175 341.6 22.3 14.1 7 262 99.9 08:29 2-8579A 1.000150 -1481 167 259 133.5 67.9 13.5 6 199 1.2 apog 08:38 2-8579A 1.000131 -1290 326 262 135.6 69.2 13.5 7 194 1.2 apog 08:46 2-8579A 1.000137 -1350 279 266 137.9 70.4 13.2 5 188 1.2 apog 08:54 2-8579A 1.000137 -1350 278 267 140.7 71.5 13.1 7 195 1.3 apog 09:03 2-8579A 1.000128 -1264 334 255 143.8 72.6 13.5 10 215 1.3 apog 09:11 2-8579A 1.000128 -1261 349 269 147.5 73.6 13.2 6 185 1.4 apog 09:19 2-8579A 1.000129 -1274 356 287 151.6 74.5 12.4 8 187 1.4 apog 09:28 2-8579A 1.000136 -1344 312 301 156.4 75.3 12.8 11 182 1.3 apog 09:52 4-7537A 1.000135 -1328 371 321 138.8 61.1 12.8 10 182 1.2 apog 10:01 4-7537A 1.000124 -1220 457 321 140.1 62.3 12.9 10 186 1.2 apog 10:09 4-7537A 1.000124 -1220 444 317 141.6 63.4 12.8 10 184 1.3 apog 10:18 4-7537A 1.000119 -1175 485 316 143.4 64.5 12.9 9 195 1.2 apog 10:26 4-7537A 1.000117 -1153 495 311 145.3 65.5 12.8 10 186 1.4 apog 10:34 4-7537A 1.000111 -1099 532 304 147.5 66.5 12.9 8 182 1.3 apog 10:43 4-7537A 1.000109 -1076 533 295 149.9 67.4 13.1 8 181 1.2 apog 10:51 4-7537A 1.000099 -977 618 292 152.5 68.2 12.9 8 178 1.3 apog ---- Weather Log ---- --- Weather --- Time Inst Temp DP Dif Humid Wind Dir Dust,1um IRSC IRSCm -------------------------------------------------------------------------------- 06:51 19-0A 14.1 3.6 10.5 49 7 262 812 13.4 806 99.9 08:29 2-8579A 13.5 4.2 9.3 53 6 199 789 6.5 805 1.2 apog 08:38 2-8579A 13.5 4.2 9.3 53 7 194 835 7.1 805 1.2 apog 08:46 2-8579A 13.2 4.1 9.1 54 5 188 736 7.0 808 1.2 apog 08:54 2-8579A 13.1 3.9 9.2 53 7 195 771 7.0 805 1.3 apog 09:03 2-8579A 13.5 3.8 9.7 51 10 215 783 7.8 806 1.3 apog 09:11 2-8579A 13.2 3.8 9.4 52 6 185 756 7.3 806 1.4 apog 09:19 2-8579A 12.4 3.5 8.9 54 8 187 761 7.2 806 1.4 apog 09:28 2-8579A 12.8 3.0 9.8 51 11 182 585 9.7 806 1.3 apog 09:52 4-7537A 12.8 1.6 11.2 46 10 182 503 7.0 806 1.2 apog 10:01 4-7537A 12.9 1.7 11.2 46 10 186 466 7.4 805 1.2 apog 10:09 4-7537A 12.8 2.0 10.8 47 10 184 464 7.2 806 1.3 apog 10:18 4-7537A 12.9 2.0 10.9 47 9 195 475 7.5 806 1.2 apog 10:26 4-7537A 12.8 2.0 10.8 47 10 186 453 8.2 806 1.4 apog 10:34 4-7537A 12.9 1.8 11.1 46 8 182 446 6.3 805 1.3 apog 10:43 4-7537A 13.1 1.8 11.3 46 8 181 524 7.6 806 1.2 apog 10:51 4-7537A 12.9 1.8 11.1 46 8 178 521 8.5 807 1.3 apog ---- Telescope Status ---- Status at:11:21Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 9 Counterweights at: 443 444 437 449 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 104.00 lbs, 20.48 psi SP2 93.10 lbs, 21.15 psi ---- Software Used ---- --- STUI software --- STUI : 1.6.0 alerts : trunk+svn149374 apo : trunk+svn158476 boss : v3_6 -boss.daq : FAILED -boss.specMec : 3.0.4 gcamera : v2_1 guider : v3_1 guider : trunk+svn162567 - for pointing model test hartmann : v1_4_3 hub : Tag: sdss4_v3_0 mcp : v6_6_18 -mcp.plc : 31 -mcp.azFiduci : v1_114 -mcp.altFiduc : v1_114 -mcp.rotFiduc : v1_114 platedb : v2_3 sop : v3_1_3 tcc : 1.1.0 apogee : not availble apogeecal : not availble apogeeql : v1_5_9 --- Other software --- sdssProcedure : v3_06 APO-local : trunk idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_2 petunia : v2_3 autoscheduler : v5_30 sdss_python_module: v2_0 mangadrp : trunk exported Restarted tcc to test new 1.2.0b1 version ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.034 0.028 Axial B: -0.007 -0.007 Axial C: -0.018 -0.027 Trans D: -0.018 -0.019 Lateral E: 10.6934 10.7061 Lateral F: 12.3063 12.319 GALILS Commanded: 10850 -450 5750 13050 14750 14300 Actual: 10850 -450 5750 13050 14750 14300 SETMIR VALUES PrimDesOrient: 0.0 -17.47 11.84 635.9 916.8 PrimOrient: 0.0 -17.47 11.84 636.03 916.38 SECONDARY: ---------- Focus: 0.0000 Air temp: 16.7 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -7.062 -7.063 Axial B: -7.117 -7.117 Axial C: -0.048 -0.046 Trans D: -0.012 -0.016 GALILS Commanded: 1602200 1596000 1627750 1400 -2000 Actual: 1581293 1578862 1590635 1400 -2000 SETMIR VALUES SecDesOrient: 1257.0 -3.8 0.0 0.0 9.58 SecOrient: 1256.99 -3.8 0.0 0.0 9.43 ---- Problems in Detail ---- We issued tcc set inst=spectro when we restarted the tcc, but were unable to set the instrument back to the engineering camera. 4:22:11 APO.elenam 1125 alerts : text="Now you know all I know" 04:22:11 CmdDone: APO.elenam 1125 alerts status 04:22:15 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=121.000135,29.726584,0.000148; ObjInstAng=nan,nan,4939848806.8062429; SpiderInstAng=nan,nan,4939848806.8062429 04:22:15 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 04:22:17 .mcp 0 mcp i cwPositions=363,363,363,362 04:22:23 CmdStarted: APO.doravetz 151 tcc set inst=engcam 04:22:23 APO.doravetz 151 tcc f Exception=RuntimeError; text="Failed to set GProbe: rot_gp_xy" 04:22:23 CmdDone: APO.doravetz 151 tcc set inst=engcam John thought it might be due to the minus sign in the i_ca1_engcam.dat file, but it's not causing any issues when Russell uses the emulator. The instrument can be set to spectro, but not engcam. Traceback in the log: tcc/base/loadInst.py\", line 70 04:45:39 APO.parejkoj_nontest: Actually, wait, it's mixed with other things. /home/tcc/tccdata/inst/i_ca1_engcam.dat line 39, 83, and/or 94 the probe 2 data is incorrect. It is missing a minus sign. But that's not what is causing the problem. Details on altitude and azimuth halt before heading to first pointing model stars: 04:57:00 CmdDone: APO.telescope.sop 581 tcc track 60.000000, 60.000000 mount/rottype=mount/rotangle=60.000000 04:57:00 CmdStarted: APO.telescope.sop 582 tcc axis stop 04:57:00 .tcc 0 tcc w AxisCmdState=Tracking,Tracking,Halted; AxisErrCode=OK,OK,HaltRequested; AxisNoTrack=FFT 04:57:00 .tcc 0 tcc I SecState=Moving,2,3,0.9,0.9 04:57:00 .tcc 0 tcc w AxisCmdState=Tracking,Halted,Halted; AxisErrCode=OK,HaltRequested,HaltRequested; AxisNoTrack=FTT 04:57:01 .tcc 0 tcc w AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 04:57:01 APO.telescope.sop 582 tcc : 04:57:01 CmdDone: APO.telescope.sop 582 tcc axis stop 04:57:01 APO.telescope 267 sop i gotoAll60State="running","OK","done" 04:57:01 APO.telescope 267 sop i gotoAll60Stages="slew" 04:57:01 APO.telescope 267 sop i gotoAll60State="done","OK","done" 04:57:01 APO.telescope 267 sop : text="at stow position" 04:57:01 CmdDone: APO.telescope 267 sop gotoAll60 04:57:01 .mcp 0 mcp i semaphoreOwner="None" Guider errors: 05:01:02 .sop 0 sop e text="Do not understand plateType: ecamera." 05:01:02 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:01:02 .mcp 0 mcp w rotBadFiducial=-5,271.02 05:01:11 APO.doravetz 191 guider w text="Guider flat is for cartridge 0 but 19 is loaded" 05:01:12 APO.doravetz 191 guider w text="Bad guider read! This exposure is mangled and will not be used." 05:01:12 APO.doravetz 191 guider w text="Skipping badly formatted image." 05:02:56 APO.doravetz 191 guider e text="Unexpected exception expected string or buffer in guider master thread" 05:04:15 .sop 0 sop e text="Do not understand plateType: (invalid)." 05:04:15 .sop 0 sop e text="Do not understand surveyMode: (invalid)." 05:04:15 .sop 0 sop w text="loadCartridge fired cart=(invalid) survey=sopActor.UNKNOWN surveyMode=None" 05:04:27 .sop 0 sop e text="Do not understand plateType: ecamera." 05:04:27 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:04:31 .sop 0 sop e text="Do not understand plateType: ecamera." 05:04:31 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:04:40 APO.doravetz 196 guider w text="Guider flat is for cartridge 0 but 19 is loaded" 05:04:41 APO.doravetz 196 guider w text="Much more signal in the dark than expected: 21.28" 05:04:41 APO.doravetz 196 guider f guideState="failed"; text="Error processing ecamera image: " 05:04:41 CmdDone: APO.doravetz 196 guider on 05:04:43 .sop 0 sop e text="Do not understand plateType: ecamera." 05:04:43 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:05:40 .sop 0 sop e text="Do not understand plateType: (invalid)." 05:05:40 .sop 0 sop e text="Do not understand surveyMode: (invalid)." 05:05:40 .sop 0 sop w text="loadCartridge fired cart=(invalid) survey=sopActor.UNKNOWN surveyMode=None" 05:05:57 .sop 0 sop e text="Do not understand plateType: ecamera." 05:05:57 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:06:05 .sop 0 sop e text="Do not understand plateType: ecamera." 05:06:05 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:06:14 APO.doravetz 198 guider w text="Guider flat is for cartridge 0 but 19 is loaded" 05:06:14 APO.doravetz 198 guider w text="Guider raw exposure has 1 saturated pixels." 05:06:14 APO.doravetz 198 guider w text="ecam_star=None in exposure 20!" 05:06:14 APO.doravetz 198 guider e text="failed to write FITS file /data/ecam/57174/proc-gimg-0020.fits: KeyError('int8',)" 05:06:14 APO.doravetz 198 guider e text="Unexpected exception 'int8' in guider master thread" 05:10:16 .sop 0 sop e text="Do not understand plateType: ecamera." 05:10:16 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:10:26 APO.doravetz 204 guider w text="Guider flat is for cartridge 0 but 19 is loaded" 05:10:26 APO.doravetz 204 guider w text="ecam_star=None in exposure 21!" 05:10:26 APO.doravetz 204 guider e text="failed to write FITS file /data/ecam/57174/proc-gimg-0021.fits: KeyError('int8',)" 05:10:26 APO.doravetz 204 guider e text="Unexpected exception 'int8' in guider master thread" 05:10:53 .sop 0 sop e text="Do not understand plateType: ecamera." 05:10:53 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:11:02 APO.doravetz 206 guider w text="Guider flat is for cartridge 0 but 19 is loaded" 05:11:02 APO.doravetz 206 guider w text="ecam_star=None in exposure 22!" 05:11:02 APO.doravetz 206 guider e text="failed to write FITS file /data/ecam/57174/proc-gimg-0022.fits: KeyError('int8',)" 05:11:02 APO.doravetz 206 guider e text="Unexpected exception 'int8' in guider master thread" 05:12:01 .sop 0 sop e text="Do not understand plateType: ecamera." 05:12:01 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:12:11 APO.doravetz 209 guider w text="Guider flat is for cartridge 0 but 19 is loaded" 05:12:11 APO.doravetz 209 guider e text="failed to write FITS file /data/ecam/57174/proc-gimg-0023.fits: KeyError('int8',)" 05:12:11 APO.doravetz 209 guider e text="Unexpected exception 'int8' in guider master thread" 05:13:01 .sop 0 sop e text="Do not understand plateType: ecamera." 05:45:13 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:45:13 .sop 0 sop w bypassedNames=axes 05:45:17 APO.parejkoj_nontest 244 guider f text="The guider is already off" 05:45:17 CmdDone: APO.parejkoj_nontest 244 guider off 05:45:18 .sop 0 sop e text="Do not understand plateType: ecamera." 05:45:18 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:45:27 APO.parejkoj_nontest 245 guider w text="ecam_star=None in exposure 37!" 05:46:13 .sop 0 sop e text="Do not understand plateType: (invalid)." 05:46:13 .sop 0 sop e text="Do not understand surveyMode: (invalid)." 05:46:13 .sop 0 sop w text="loadCartridge fired cart=(invalid) survey=sopActor.UNKNOWN surveyMode=None" 05:46:14 .sop 0 sop e text="Do not understand plateType: ecamera." 05:46:14 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:47:13 .sop 0 sop e text="Do not understand plateType: ecamera." 05:47:13 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:47:22 APO.parejkoj_nontest 248 guider w text="ecam_star=None in exposure 38!" 05:47:27 APO.parejkoj_nontest 248 guider e text="failed to write FITS file /data/ecam/57174/proc-gimg-0038.fits: TypeError("; UNPARSEDTEXT="__init__() got an unexpected keyword argument 'unit'\",)\"" 05:47:27 APO.parejkoj_nontest 248 guider e text="Unexpected exception __init__() got an unexpected keyword argument 'unit' in guider master thread" 05:48:06 .sop 0 sop e text="Do not understand plateType: (invalid)." 05:48:06 .sop 0 sop e text="Do not understand surveyMode: (invalid)." 05:48:06 .sop 0 sop w text="loadCartridge fired cart=(invalid) survey=sopActor.UNKNOWN surveyMode=None" 05:48:14 .sop 0 sop e text="Do not understand plateType: ecamera." 05:48:14 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:48:15 .sop 0 sop e text="Do not understand plateType: ecamera." 05:48:15 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 05:48:25 APO.parejkoj_nontest 250 guider w text="ecam_star=None in exposure 39!" /data/ecam/57174/proc-gimg-0040.fits first star to appear in the guider window ecam focus sweep window: Failed: 'Model' object has no attribute 'ecam_star' because observers don't have updated actorkeys. JohnP got the script running, but it still had trouble finding the star. 06:17:55 .sop 0 sop e text="Do not understand plateType: ecamera." 06:17:55 .sop 0 sop w text="loadCartridge fired cart=19 survey=sopActor.UNKNOWN surveyMode=None" 06:18:05 APO.parejkoj 29 guider w text="CCD temp signifcantly different (>3.0) from dark temp: -34.9, expected -31.1" 06:18:05 APO.parejkoj 29 guider w text="ecam_star=None in exposure 88!" 06:18:05 APO.parejkoj 29 guider w text="guider cartridgeKey is not well defined: KeyVar('guider', cartridgeLoaded=(Int(19), Int(0), String('A'), (invalid), (invalid)))" 06:18:05 APO.parejkoj 29 guider w text="failed to make MAPID card from (invalid)" 06:18:05 APO.parejkoj 29 guider w text="failed to index KeyVar('apo', version=(None,)) by None from version for v_apo: KeyVar('apo', version=(None,)) is unknown" Processed guider images had the dark subtracted, but were not normalized. Once the flats were normalized, the camera was able to find the star more consistently. Altitude brake came on during rotator slew: 6:40:21 APO.parejkoj_nontest 348 guider i text="guideStep GuiderImageAnalysis.findStars()..." 06:40:21 .tcc 0 tcc w BadAltStatus; AltStat=67.036882,0.000000,4939857621.76605,0x00000800; AltDTime=0.14 06:40:22 .tcc 0 tcc i TCCPos=125.117923,67.055653,-71.439321; AxePos=125.117798,67.036882,-71.439187; ObjInstAng=0.000000,0.000000,4939857428.8053341; SpiderInstAng=108.741572,0.005189,4939857627.7550859 06:40:22 .tcc 0 tcc i AxisCmdState=Tracking,Tracking,Tracking; AxisErrCode=OK,OK,OK; AxisNoTrack=FFF 06:40:22 .mcp 0 mcp i trace="Saw PLC version 31; expected "; UNPARSEDTEXT="Version 29 : NOTAG\"\"" 06:40:22 .mcp 0 mcp i cwPositions=362,363,358,365 06:40:24 APO.parejkoj_nontest 350 msg : msg="2015-06-01T06:40:24","That's what elena had in her script" 06:40:26 APO.doravetz 344 msg : msg="2015-06-01T06:40:26","no that's correct" 06:40:26 .mcp 0 mcp i ab_I6_L0=0xfff37f7c 06:40:26 .mcp 0 mcp i ab_O2_L0=0x00000003 06:40:27 .mcp 0 mcp i ab_O11_L0=0x07fc0040 06:40:27 .tcc 0 tcc i TCCPos=125.130384,67.069887,-71.413603; AxePos=125.130446,67.036871,-71.413348; ObjInstAng=0.000000,0.000000,4939857428.8053341; SpiderInstAng=108.767527,0.005194,4939857632.7550859 06:40:27 .tcc 0 tcc i AxisCmdState=Tracking,Tracking,Tracking; AxisErrCode=OK,OK,OK; AxisNoTrack=FFF 06:40:28 .alerts 0 alerts i alert=tcc.heartbeat,critical,"The TCC's been dead for at least 687s",disabled,noack,"" 06:40:31 APO.parejkoj_nontest 351 msg : msg="2015-06-01T06:40:31","And alt went bonk" 06:40:31 .tcc 0 tcc i Rot1TrackAdvTime=20,0.04,186.00,1.05,64.18 06:40:31 .tcc 0 tcc w BadAltStatus; AltStat=67.036871,0.000000,4939857631.76655,0x00001800; AltDTime=0.14 06:40:32 .tcc 0 tcc i TCCPos=125.143073,67.084349,-71.387443; Altitude Axis Halt: 07:05:47 .mcp 0 mcp i cwPositions=362,363,360,361 07:05:47 .tcc 0 tcc I SecEncMount=1309477.0,1306673.0,1318239.0,13850.0,10450.0 07:05:47 .tcc 0 tcc I SecOrient=1040.53,-3.67,-0.14,-0.09,-275.68,-0.85 07:05:47 .tcc 0 tcc I SecActMount=1310598.97,1305918.95,1317891.07,13850.36,10450.36 07:05:47 .tcc 0 tcc I SecCmdMount=1299548.35,1299848.82,1312574.37,13873.23,10473.23 07:05:47 .tcc 0 tcc I SecMountErr=-155.27,114.2,448.95,48.56,48.56 07:05:47 .mcp 0 mcp i trace="MGT: bad az state ABORT_EVENT: ID_ERROR_LIMIT" 07:05:48 APO.doravetz 391 msg : msg="2015-06-01T07:05:48","will it retry that star or go to the next?" 07:05:48 .mcp 0 mcp i ab_I9_L0=0x00440000 07:05:48 .mcp 0 mcp i ab_O12_L0=0x02060000 07:05:48 .boss 0 boss i aliveAt=1433142348 07:05:48 .tcc 0 tcc I SecEncMount=1309477.0,1306699.0,1318258.0,13850.0,10450.0 07:05:48 .tcc 0 tcc I SecOrient=1040.55,-3.67,-0.13,-0.08,-275.68,-0.85 07:05:48 .tcc 0 tcc I SecActMount=1310598.51,1305943.01,1317912.47,13850.36,10450.36 07:05:48 .tcc 0 tcc I SecCmdMount=1299548.35,1299848.82,1312574.37,13873.23,10473.23 07:05:48 .tcc 0 tcc I SecStatus=4,4,4,4,4 07:05:48 .tcc 0 tcc I SecState=Moving,3,3,-1.36,0.8 07:05:48 .tcc 0 tcc I SecState=Done,0,3,0.0,0.0 07:05:48 .alerts 0 alerts i alert=tcc.heartbeat,critical,"The TCC's been dead for at least 1212s",disabled,noack,"" 07:05:49 .boss 90725 boss i text="Parsing CAMCHECK from sp1cam" 07:05:49 .boss 90725 boss i text="Parsed CAMCHECK on sp1cam" 07:05:49 .boss 90725 boss i text="Parsing CAMCHECK from sp2cam" 07:05:49 .boss 90725 boss i text="Parsed CAMCHECK on sp2cam" 07:05:49 .boss 90725 boss i camCheck 07:05:49 .boss 90725 boss : 07:05:49 .mcp 0 mcp i ab_I9_L0=0x004c0000 07:05:49 .tcc 0 tcc w BadAzStatus; AzStat=170.445789,0.000000,4939859149.39183,0x00000800; AzDTime=0.14 07:05:49 APO.parejkoj_nontest 425 tcc : 07:05:49 CmdDone: APO.parejkoj_nontest 425 tcc axis init 07:05:49 .mcp 0 mcp i semaphoreOwner="None" 07:05:49 .mcp 0 mcp i semaphoreOwner="TCC:0:0" 07:05:49 .mcp 1238734 mcp i trace="MOVE CMD: requested time=25549.626390" 07:05:49 .mcp 1238734 mcp i trace=" current time=25549.670513" 07:05:49 .mcp 1238734 mcp ! command="move" 07:05:49 .tcc 0 tcc i AltTrackAdvTime=20,-0.15,54.61,1.37,18.07