Subject: 2.5m obslog 20200130 (SJD 58879) From: asimmons@apo.nmsu.edu Date: Fri, 31 Jan 2020 11:25:18 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday January 30, 2020 (SJD 58879) ---- Observing Team ---- Karen Kinemuchi Audrey Oravetz Juan Trujillo (shadowing first part of night) John Downey (support) ---- Observing Plan ---- Thu 30 JAN 2020 58879.0 0.29 01:45 12:49 03:21 14:27 ---- MaNGA (11.1 h) ---- Observing Summary ---- The night started out clear, then became cloudy enough to close for 3.5hrs (MANGA loss). We were able to reopen after the clouds moved away, but closed yet again at 10:04Z and stayed closed the rest of the night (an additional 2.75hrs MANGA loss). Cart 5, M/Ap deep plate 8078, 173-51_MGA; 2XNSE + 6XAB Missing fibers: [154, 211, 273] Faint fibers: [37, 52, '121 - 153', '155 - 210', '212 - 213', '215 - 216', '222 - 227', '229 - 231', '234 - 249', '251 - 270'] * postcals showed RED XSIGMA/WSIGA values ----------------- Cart 3, M/Ap plate 10515, 232+42_MGA; 3XNSE + 1XN + 13XAB Missing fibers: [211, 273] Faint fibers: ['121 - 180', 185, '187 - 188', 192, 194, 197, '271 - 272', '274 - 276', '278 - 300'] ----------------- NOTES: ====== * We received an apogeeLN2 critical alert at 09:13:33 .alerts 0 alerts e alert=apogee.ln2Alarm,critical,"True",enabled,noack, APOGEE Telemetry: BAD, LN2 75 Current, 75% threshold. Called JohnD, and will hope to get into contact with someone from the APOGEE team soon. No audible alarms were going off within the instrument room. The vacuum seemed ok. * We had multiple instances of altitude failures during our second opening. Slewing down to 5, several times moving up to stow from 5, and again to 60,60,60 for our test slew. Each time, activating the stop button and then releasing it at the podium allowed the slew for a few more degrees before failing again. These STUI messages are listed within the problem section. A screen shot of the interlocks window can be found still open on the podium laptop. We attribute this problem to the interlock grounding issue we have been seeing lately, except with greater frequency. * Reposition frame break error a few times tonight, once for azimuth and once for altitude. * The cart bay lift was acting finicky when we opened the second time. It would intermittently fail to move up or down at all, sometimes only up but not down, and one time moved down to the bottom-most position, but was not level. * One secondary latch during a cart change was slow to release, but moving the lift slightly higher to 1350-1400 on the pendent helped. * One instance of a motor time-out during the hartmann sequence on cart 3. No time lost. * Disk space is getting full: /dev/vdd 688G 536G 152G 78% /home /dev/vdc1 11T 9.7T 1.2T 90% /data ---- Observing Log ---- 22:26Z Clear skies AFTERNOON CHECKOUT: =================== 22:26Z No APOGEE cals have been taken due to UPS issue. We are waiting for instructions from the day staff. 23:19Z We power cycled the iBootBar near the apogee cal box. Status window shows apogee reporting information. Test 3-read dark looked fine. Had to restart the apogeeql actor and tried to restart the scripts. Holtz and Nathan were on STUI to help get things restarted. Apogeeql actor now reports responsive, and taking evening cals. Monitoring the cals. 23:21Z Grabbed mirror numbers - ok; gcamera/guider prep - done. MaNGA cals: 23:26:37 boss exposure 0.0, file=307347, C 23:27:57 boss exposure 20.0, file=307348, C DoS is alive. BOSS COLLIMATION: 23:30:15 boss exposure 4.0, file=307349, C 23:31:26 boss exposure 4.0, file=307350, C 23:33:09 sp1.motor.move= 1972, 1970, 1973 23:33:14 sp2.motor.move= 1577, 1577, 1577 23:33:14 Hartmann collimate output on cart #5 sp1: offset: r= 989 (In Focus); b= -9.3 (Out of focus) sp2: offset: r= 1578 (Out of focus); b= -4.9 (Out of focus) sp1: pred. move: spAverageMove= 1972 sp2: pred. move: spAverageMove= 1577 sp1: pred. resid: r= -983, b= -3.1, txt= OK, spTemp = 4.3 sp2: pred. resid: r= 1, b= 0.0, txt= OK, spTemp = 5.1 Blue rings were not adjusted. ---------- motorPosition sp1: 21367, 20542, 18913; sp2: 19164, 18922, 18777; ---------- Median spectrographs temperatures: Tsp1= 4.3, Tsp2= 5.1; Outside air temperature= 0.7; MaNGA CALS: 23:34:27 boss exposure 25.0, file=307351, C 23:37:14 boss exposure 4.0, file=307352, C 23:30Z Ventilation begins. 23:57Z Encountered the frame break error message. Az would not move. Followed protocol with tcc axis init and power cycled upper and lower enclosure stop buttons. Had to power cycle the wind baffle toaster switch. wind-az interlock finally turned green in the interlocks window. Telescope is moving fine now. See problems in detail for the time stamps of the Az frame break messages from the mcp log. 00:18Z Cart bay ready. Telescope ready. Disk check df -kh Filesystem Size Used Avail Use% Mounted on /dev/vda2 7.7G 6.0G 1.4G 82% / tmpfs 3.4G 0 3.4G 0% /dev/shm /dev/vda1 194M 128M 57M 70% /boot /dev/vdb 19G 8.8G 9.1G 49% /exports/hub-local /dev/vdd 688G 536G 152G 78% /home /dev/vdc1 11T 9.7T 1.2T 90% /data Night: ====== 00:30Z - Opened enclosure and started a fill. Cart 5, M/Ap deep plate 8078, 173-51_MGA 01:16:04 boss exposure 4.0, file=307357, C 01:16:43 boss exposure 4.0, file=307358, C 01:17:55 sp1.motor.move= 46, 40, 39 01:18:01 sp2.motor.move= -316, -316, -315 01:18:01 Hartmann collimate output on cart #5 sp1: offset: r= 989 (In Focus); b= 2.8 (In Focus) sp2: offset: r= -946 (In Focus); b= -1.0 (In Focus) sp1: pred. move: spAverageMove= 45 sp2: pred. move: spAverageMove= -315 sp1: pred. resid: r= 944, b= 3.0, txt= OK, spTemp = 1.5 sp2: pred. resid: r= -630, b= -2.0, txt= OK, spTemp = 1.2 01:18:20 boss exposure 4.0, file=307359, C 01:19:49 boss exposure 25.0, file=307360, C 01:33:32 apogee.expose=apRaw-33170011, 42, A, Object 01:33:39 boss exposure 900.0, file=307361, N 01:41:03 apogee.expose=apRaw-33170012, 42, B, Object 01:49:57 apogee.expose=apRaw-33170013, 42, B, Object 01:50:04 boss exposure 900.0, file=307362, S 01:57:32 apogee.expose=apRaw-33170014, 42, A, Object 02:06:22 apogee.expose=apRaw-33170015, 42, A, Object 02:06:28 boss exposure 900.0, file=307363, E 02:13:52 apogee.expose=apRaw-33170016, 42, B, Object 02:22:49 apogee.expose=apRaw-33170017, 42, B, Object 02:22:56 boss exposure 900.0, file=307364, N 02:30:24 apogee.expose=apRaw-33170018, 42, A, Object 02:39:13 apogee.expose=apRaw-33170019, 42, A, Object 02:39:20 boss exposure 900.0, file=307365, S. Lots of clouds now. 02:46:43 apogee.expose=apRaw-33170020, 42, B, Object 02:55:38 apogee.expose=apRaw-33170021, 42, B, Object 02:55:45 boss exposure 900.0, file=307366, E 03:03:13 apogee.expose=apRaw-33170022, 42, A, Object 03:12:53 boss exposure 25.0, file=307367, C *** RED XSIGMA 03:15:58 boss exposure 4.0, file=307368, C *** RED WSIGMA 03:16:47 apogee.expose=apRaw-33170023, 5, A, DomeFlat Missing fibers: [154, 211, 273] Faint fibers: [37, 52, '121 - 153', '155 - 210', '212 - 213', '215 - 216', '222 - 227', '229 - 231', '234 - 249', '251 - 270'] ----------------- 03:30Z - Too many clouds, and oncoming radar. We closed but remained ventilating. 06:48Z - Reopened. Clouds parted. Cart 3, M/Ap plate 10515, 232+42_MGA 06:41:53 boss exposure 4.0, file=307369, C 06:42:31 boss exposure 4.0, file=307370, C 06:44:13 Hartmann collimate output on cart #3 Hartmann failed sp1: offset: r= -6596 (Out of focus); b= 19.9 (Out of focus) sp2: offset: r= -5049 (Out of focus); b= 11.9 (Out of focus) sp1: pred. move: spAverageMove= -6479 sp2: pred. move: spAverageMove= -4415 sp1: pred. resid: r= -116, b= -0.4, txt= OK, spTemp = -0.0 sp2: pred. resid: r= -633, b= -2.0, txt= OK, spTemp = -0.9 06:44:13 sop.gotoField = failed; 06:44:15 sp1.motor.move= -6478, -6479, -6480 Failed on the motor movement on SP1 (time-out). 06:44:12 client.v70.hartmann 488 boss w text="Failure from sp1mech: Failed motor command 129 \"1MN,1MR-483,1WE50,WA100,0MF\r\" timed out" 06:44:12 client.v70.hartmann 488 boss w text='move failed; ignoring' 06:44:13 OBSERVER.telescope.sop 512 hartmann e text="Failed to move collimator pistons for sp1. Command timed out" 06:44:13 OBSERVER.telescope.sop 512 hartmann f text="collimation process failed" 06:44:13 CmdDone: OBSERVER.telescope.sop 512 hartmann collimate ignoreResiduals 06:44:13 OBSERVER.telescope 182 sop e text='Failed to collimate BOSS spectrographs.' 06:44:13 OBSERVER.telescope 182 sop f text="Failed to take hartmann sequence. lamps are on." 06:45:49 boss exposure 4.0, file=307371, C 06:46:27 boss exposure 4.0, file=307372, C 06:47:42 sp1.motor.move= -779, -774, -777 06:47:54 sp2.motor.move= -4415, -4410, -4415 06:47:54 Hartmann collimate output on cart #3 sp1: offset: r= -659 (In Focus); b= 2.8 (In Focus) sp2: offset: r= -5049 (Out of focus); b= 11.9 (Out of focus) sp1: pred. move: spAverageMove= -779 sp2: pred. move: spAverageMove= -4415 sp1: pred. resid: r= 120, b= 0.4, txt= OK, spTemp = -0.1 sp2: pred. resid: r= -633, b= -2.0, txt= OK, spTemp = -1.0 06:48:13 boss exposure 4.0, file=307373, C 06:49:41 boss exposure 25.0, file=307374, C 06:55:00 apogee.expose=apRaw-33170028, 42, A, Object 06:55:07 boss exposure 900.0, file=307375, N 07:02:31 apogee.expose=apRaw-33170029, 42, B, Object 07:11:24 apogee.expose=apRaw-33170030, 42, B, Object 07:11:31 boss exposure 900.0, file=307376, S 07:18:58 apogee.expose=apRaw-33170031, 42, A, Object 07:27:47 apogee.expose=apRaw-33170032, 42, A, Object 07:27:55 boss exposure 900.0, file=307377, E 07:35:17 apogee.expose=apRaw-33170033, 42, B, Object 07:44:11 apogee.expose=apRaw-33170034, 42, B, Object 07:44:18 boss exposure 900.0, file=307378, N 07:51:46 apogee.expose=apRaw-33170035, 42, A, Object 08:00:35 apogee.expose=apRaw-33170036, 42, A, Object 08:00:41 boss exposure 900.0, file=307379, S 08:08:05 apogee.expose=apRaw-33170037, 42, B, Object 08:17:00 apogee.expose=apRaw-33170038, 42, B, Object 08:17:06 boss exposure 900.0, file=307380, E 08:24:34 apogee.expose=apRaw-33170039, 42, A, Object 08:33:25 apogee.expose=apRaw-33170040, 42, A, Object 08:33:31 boss exposure 900.0, file=307381, N 08:40:54 apogee.expose=apRaw-33170041, 42, B, Object 08:49:49 apogee.expose=apRaw-33170042, 42, B, Object 08:49:56 boss exposure 900.0, file=307382, S 08:57:24 apogee.expose=apRaw-33170043, 42, A, Object 09:06:13 apogee.expose=apRaw-33170044, 42, A, Object 09:06:20 boss exposure 900.0, file=307383, E 09:13:43 apogee.expose=apRaw-33170045, 42, B, Object 09:13:33 .alerts 0 alerts e alert=apogee.ln2Alarm,critical,"True",enabled,noack,"" APOGEE Telemetry: BAD 09:23:50 boss exposure 25.0, file=307384, C 09:27:01 boss exposure 4.0, file=307385, C 09:32:27 apogee.expose=apRaw-33170046, 42, B, Object 09:32:34 boss exposure 900.0, file=307386, N 09:40:02 apogee.expose=apRaw-33170047, 42, A, Object 09:48:51 apogee.expose=apRaw-33170048, 42, A, Object (ABORTED) 09:48:58 boss exposure 900.0, file=307387, S (ABORTED) 09:52:11 apogee.expose=apRaw-33170049, 5, A, DomeFlat Missing fibers: [211, 273] Faint fibers: ['121 - 180', 185, '187 - 188', 192, 194, 197, '271 - 272', '274 - 276', '278 - 300'] ----------------- 09:57Z - Closed in a hurry because the humidity spiked high, the wind got very strong, our seeing was terrible, and orographic clouds were forming on top of us. Started cals. ---- BOSS Spectrographs Collimation Summary ---- --- Hartmann --- Time Cart R1 B1 Move1 B1Resid TSP1 R2 B2 Move2 B2Resid TSP2 ================================================================================ 01:02 5-8078A 3958 -15.3 4421 -1.5 1.8 2209 -9.9 2680 -1.5 1.5 01:17 5-8078A 989 +2.8 45 3.0 1.5 -946 -1.0 -315 -2.0 1.2 06:43 3-10515A -6596 +19.9 -6479 -0.4 -0.0 -5049 11.9 -4415 -2.0 -0.9 06:47 3-10515A -659 +2.8 -779 0.4 -0.1 -5049 11.9 -4415 -2.0 -1.0 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: -5.331 -5.332 Axial B: -5.035 -5.031 Axial C: -5.271 -5.27 Trans D: -0.579 -0.579 Lateral E: 9.3599 9.3472 Lateral F: 14.5161 14.4907 GALILS Commanded: 11200 -1850 6850 29650 21250 21050 Actual: 11200 -1850 6850 29650 21250 21050 SETMIR VALUES PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 PrimOrient: 0.0 -11.4 6.58 375.38 620.77 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.3 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -6.661 -6.661 Axial B: -0.029 -0.023 Axial C: -6.659 -6.661 Trans D: -0.066 -0.07 GALILS Commanded: 1094650 1134450 1131900 -7250 -6550 Actual: 1058718 1101465 1079487 -7250 -6550 SETMIR VALUES SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 SecOrient: 857.0 2.71 12.76 -38.42 156.71 ---- Telescope Offsets and Scale ---- --- Offsets --- (arcsec) Time Cart Az Alt Rot objOff guideRot calibOff guideRMS ================================================================================ 01:31 5-8078A -9.8 57.0 +171.7 (-0.1,-1.9) -13.6 ( 0, 0, 0) 0.309 01:48 5-8078A -17.0 56.2 +165.8 (-0.5,-1.4) -5.5 ( 0, 0, 0) 0.957 02:04 5-8078A -23.9 55.0 +160.0 (-0.8,-3.9) -4.0 ( 0, 0, 0) 0.827 02:20 5-8078A -30.5 53.4 +154.7 (0.1,-4.2) +0.6 ( 0, 0, 0) 0.871 02:37 5-8078A -36.4 51.5 +150.0 (-1.5,-4.4) +3.0 ( 0, 0, 0) 0.863 02:53 5-8078A -41.9 49.3 +145.8 (-2.0,-5.9) +5.8 ( 0, 0, 0) 0.794 03:10 5-8078A -46.9 46.9 +142.1 (-1.2,-5.3) +11.0 ( 0, 0, 0) 0.957 06:26 3-10515A +121.0 90.0 +0.0 (0.0,0.0) +0.0 ( 0, 0, 0) 0.391 06:53 3-10515A +39.3 56.1 +212.3 (0.6,-8.6) -37.3 ( 0, 0, 0) 0.454 07:09 3-10515A +33.0 58.1 +207.3 (0.4,-7.1) -33.4 ( 0, 0, 0) 0.817 07:25 3-10515A +25.8 59.7 +201.5 (0.8,-8.8) -23.4 ( 0, 0, 0) 0.871 07:42 3-10515A +17.9 61.0 +195.0 (1.8,-8.9) -21.1 ( 0, 0, 0) 0.777 07:58 3-10515A +9.5 61.9 +188.0 (0.7,-8.0) -16.3 ( 0, 0, 0) 0.993 08:15 3-10515A +0.8 62.2 +180.6 (0.6,-9.5) -10.2 ( 0, 0, 0) 0.921 08:31 3-10515A -7.9 62.0 +173.3 (1.5,-9.3) -11.6 ( 0, 0, 0) 1.050 08:47 3-10515A -16.4 61.2 +166.2 (-0.2,-8.5) -8.7 ( 0, 0, 0) 0.675 09:04 3-10515A -24.4 60.0 +159.6 (-0.6,-10.2) -3.5 ( 0, 0, 0) 0.945 09:20 3-10515A -31.7 58.4 +153.6 (0.3,-9.9) -4.3 ( 0, 0, 0) 1.193 09:30 3-10515A -35.9 57.2 +150.3 (-0.6,-10.0) -2.4 ( 0, 0, 0) 0.351 09:47 3-10515A -42.0 55.1 +145.6 (-1.1,-8.8) +2.0 ( 0, 0, 0) 0.810 ---- Focus Log ---- --- Focus --- Time Cart Scale M1 M2 Focus Az Alt Temp Wind Dir FWHM ================================================================================ 01:31 5-8078A +243.0 -2398 -722 +503 -9.8 57.0 -2.9 7 330 1.4 01:48 5-8078A +243.0 -2398 -682 +540 -17.0 56.2 -2.8 17 324 1.3 02:04 5-8078A +218.0 -2147 -528 +484 -23.9 55.0 -2.8 6 311 1.3 02:20 5-8078A +179.0 -1761 -286 +403 -30.5 53.4 -2.8 7 343 1.5 02:37 5-8078A +144.0 -1420 -68 +331 -36.4 51.5 -2.7 6 336 1.3 02:53 5-8078A +124.0 -1229 +30 +264 -41.9 49.3 -2.6 3 295 1.4 03:10 5-8078A +110.0 -1086 +108 +216 -46.9 46.9 -2.4 2 7 1.3 06:26 3-10515A +111.0 -1099 +10 +206 +121.0 90.0 -3.9 15 7 1.4 06:53 3-10515A +141.0 -1392 -270 +118 +39.3 56.1 -4.2 15 30 1.7 07:09 3-10515A +159.0 -1570 -369 +173 +33.0 58.1 -4.3 18 37 1.5 07:25 3-10515A +144.0 -1417 -231 +190 +25.8 59.7 -4.4 18 17 1.4 07:42 3-10515A +122.0 -1204 -59 +188 +17.9 61.0 -4.4 17 32 1.8 07:58 3-10515A +118.0 -1169 -48 +171 +9.5 61.9 -4.3 17 55 1.9 08:15 3-10515A +101.0 -997 +75 +154 +0.8 62.2 -4.3 15 53 2.3 08:31 3-10515A +81.0 -796 +228 +140 -7.9 62.0 -4.6 16 59 2.2 08:47 3-10515A +89.0 -879 +151 +131 -16.4 61.2 -4.4 19 56 2.6 09:04 3-10515A +93.0 -920 +142 +147 -24.4 60.0 -5.0 16 65 2.3 09:20 3-10515A +81.0 -802 +244 +152 -31.7 58.4 -4.9 18 69 3.2 09:30 3-10515A +89.0 -882 +172 +142 -35.9 57.2 -5.0 21 71 3.0 09:47 3-10515A +133.0 -1315 -120 +202 -42.0 55.1 -5.7 32 71 3.1 ---- Telescope Status ---- 23:30Z Started ventilation 00:30Z encl25m : open 09:57Z encl25m : closed No interlocks bypassed Status at: 11:24Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 3 LN2 autofill systems: Connected and turned on -180L LN2 dewar scales: SP1 208.8 lbs, 20.3 psi SP2 169.6 lbs, 23.5 psi ---- Software Used ---- Get Versions, 01/30/20, 22:30:00 -- STUI software: STUI : 1.7.8 alerts : trunk+svn66561 apo : trunk boss : v4_1_2 -boss.specMec : 3.0.4 gcamera : v2_2 guider : 3.8.1 hartmann : 1.7.1 hub : v4.2.3 mcp : 1.10-1-gfdd1 -mcp.plc : 34 -mcp.azFiduci : v1_141 -mcp.altFiduc : v1_141 -mcp.rotFiduc : v1_141 platedb : 2.8.0 sop : 3.12.7 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : FAILED APO-local : trunk -- done -- Home Disk Hogs (GB): manga 89.4416 eboss 86.258 sdss3 27.1813 apogee 18.3769 logging 13.7424 observer 12.2519 Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 536G 152G 78% /home /dev/vdc1 11T 9.7T 1.2T 90% /data Setting up: idlspec2d Flavor: Linux64 Version: v5_7_12 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: trunk Setting up: |specflat Flavor: Linux64 Version: v1_20 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |sdss_python_module Flavor: Linux64 Version: v1_6_7 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.9.7 Setting up: |psycopg2 Flavor: Linux64 Version: 2.5.3 Setting up: |idlutils Flavor: Linux64 Version: v5_5_16 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: |photolog Flavor: Linux64 Version: trunk Setting up: actorcore Flavor: Linux64 Version: v4_1_8 Setting up: |opscore Flavor: Linux64 Version: v2_5 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v30_2 Setting up: |sdsstools Flavor: Linux64 Version: v4_33 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_13 Setting up: |RO Flavor: Linux64 Version: 3.6.9 -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_12 petunia : 1.5.0 autoscheduler : 1.4.1 sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- APOGEE data list, mjd=58879, all raw_data: /data/apogee/utr_cdr/58879/apRaw-*.fits archive: /data/apogee/archive/58879/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/58879/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 58879 1 no file found: /data/apogee/utr_cdr/58879/apRaw-33170001.fits 58879 2 23:16 33170002 Dark 60 A 13.0 5-8078 a-b-c - - 58879 3 23:27 33170003 QuarFlat 10 A 13.0 5-8078 a-b-c - 33 58879 4 23:29 33170004 Arc-Thar 12 A 13.0 5-8078 a-b-c -6.37 - 58879 5 23:31 33170005 Arc-Une 40 A 13.0 5-8078 a-b-c - - 58879 6 23:38 33170006 Arc-Thar 12 B 13.5 5-8078 a-b-c -5.92 - 58879 7 23:40 33170007 Arc-Une 40 B 13.5 5-8078 a-b-c - - 58879 8 23:48 33170008 Dark 60 A 13.0 5-8078 a-b-c - - 58879 9 00:48 33170009 Dark 10 A 13.0 5-8078 a-b-c - - 58879 10 00:50 33170010 Dark 10 A 13.0 5-8078 a-b-c - - 58879 11 01:33 33170011 Object 42 A 13.0 5-8078 a-b-c - - 58879 12 01:40 33170012 Object 42 B 13.5 5-8078 a-b-c - - 58879 13 01:49 33170013 Object 42 B 13.5 5-8078 a-b-c - - 58879 14 01:57 33170014 Object 42 A 13.0 5-8078 a-b-c - - 58879 15 02:06 33170015 Object 42 A 13.0 5-8078 a-b-c - - 58879 16 02:13 33170016 Object 42 B 13.5 5-8078 a-b-c - - 58879 17 02:22 33170017 Object 42 B 13.5 5-8078 a-b-c - - 58879 18 02:30 33170018 Object 42 A 13.0 5-8078 a-b-c - - 58879 19 02:39 33170019 Object 42 A 13.0 5-8078 a-b-c - - 58879 20 02:46 33170020 Object 42 B 13.5 5-8078 a-b-c - - 58879 21 02:55 33170021 Object 42 B 13.5 5-8078 a-b-c - - 58879 22 03:03 33170022 Object 42 A 13.0 5-8078 a-b-c - - 58879 23 03:16 33170023 DomeFlat 5 A 13.0 5-8078 a-b-c - 26 58879 24 03:18 33170024 Dark 10 A 13.0 5-8078 a-b-c - - 58879 25 03:20 33170025 Dark 10 A 13.0 5-8078 a-b-c - - 58879 26 06:32 33170026 Dark 10 A 13.0 3-10515 a-b-c - - 58879 27 06:34 33170027 Dark 10 A 13.0 3-10515 a-b-c - - 58879 28 06:54 33170028 Object 42 A 13.0 3-10515 a-b-c - - 58879 29 07:02 33170029 Object 42 B 13.5 3-10515 a-b-c - - 58879 30 07:11 33170030 Object 42 B 13.5 3-10515 a-b-c - - 58879 31 07:18 33170031 Object 42 A 13.0 3-10515 a-b-c - - 58879 32 07:27 33170032 Object 42 A 13.0 3-10515 a-b-c - - 58879 33 07:35 33170033 Object 42 B 13.5 3-10515 a-b-c - - 58879 34 07:43 33170034 Object 42 B 13.5 3-10515 a-b-c - - 58879 35 07:51 33170035 Object 42 A 13.0 3-10515 a-b-c - - 58879 36 08:00 33170036 Object 42 A 13.0 3-10515 a-b-c - - 58879 37 08:07 33170037 Object 42 B 13.5 3-10515 a-b-c - - 58879 38 08:16 33170038 Object 42 B 13.5 3-10515 a-b-c - - 58879 39 08:24 33170039 Object 42 A 13.0 3-10515 a-b-c - - 58879 40 08:33 33170040 Object 42 A 13.0 3-10515 a-b-c - - 58879 41 08:40 33170041 Object 42 B 13.5 3-10515 a-b-c - - 58879 42 08:49 33170042 Object 42 B 13.5 3-10515 a-b-c - - 58879 43 08:57 33170043 Object 42 A 13.0 3-10515 a-b-c - - 58879 44 09:06 33170044 Object 42 A 13.0 3-10515 a-b-c - - 58879 45 09:13 33170045 Object 42 B 13.5 3-10515 a-b-c - - 58879 46 09:32 33170046 Object 42 B 13.5 3-10515 a-b-c - - 58879 47 09:39 33170047 Object 42 A 13.0 3-10515 a-b-c - - 58879 48 09:48 33170048 Object 42 A 13.0 3-10515 a-b-c - - 58879 49 09:51 33170049 DomeFlat 5 A 13.0 3-10515 a-b-c - 27 58879 50 09:53 33170050 Dark 10 A 13.0 3-10515 a-b-c - - 58879 51 09:55 33170051 Dark 10 A 13.0 3-10515 a-b-c - - -------------------------------------------------------------------------------- ---- MaNGA Data Summary ---- Reading FITS from /data/spectro/58879/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n time filename crt plt flav Shu Dth exp colA colB colC Hrt b1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:26 b1-00307347 5 8078 bia Clo C 0.0 19395 18572 16940 23:26 b2-00307347 5 8078 bia Clo C 0.0 17587 17345 17200 23:26 r1-00307347 5 8078 bia Clo C 0.0 19395 18572 16940 23:26 r2-00307347 5 8078 bia Clo C 0.0 17587 17345 17200 2 23:27 b1-00307348 5 8078 dar Clo C 20.0 19395 18572 16940 23:27 b2-00307348 5 8078 dar Clo C 20.0 17587 17345 17200 23:27 r1-00307348 5 8078 dar Clo C 20.0 19395 18572 16940 23:27 r2-00307348 5 8078 dar Clo C 20.0 17587 17345 17200 3 23:30 b1-00307349 5 8078 arc Lef C 4.1 19395 18572 16940 HART 23:30 b2-00307349 5 8078 arc Lef C 4.2 17587 17345 17200 HART 23:30 r1-00307349 5 8078 arc Lef C 4.1 19395 18572 16940 HART 23:30 r2-00307349 5 8078 arc Lef C 4.2 17587 17345 17200 HART 4 23:31 b1-00307350 5 8078 arc Rig C 4.1 19395 18572 16940 HART 23:31 b2-00307350 5 8078 arc Rig C 4.2 17587 17345 17200 HART 23:31 r1-00307350 5 8078 arc Rig C 4.1 19395 18572 16940 HART 23:31 r2-00307350 5 8078 arc Rig C 4.2 17587 17345 17200 HART 5 23:34 b1-00307351 5 8078 fla Out C 25.1 21367 20542 18913 23:34 b2-00307351 5 8078 fla Out C 25.2 19164 18922 18777 23:34 r1-00307351 5 8078 fla Out C 25.1 21367 20542 18913 23:34 r2-00307351 5 8078 fla Out C 25.2 19164 18922 18777 6 23:37 b1-00307352 5 8078 arc Out C 4.1 21367 20542 18913 23:37 b2-00307352 5 8078 arc Out C 4.2 19164 18922 18777 23:37 r1-00307352 5 8078 arc Out C 4.1 21367 20542 18913 23:37 r2-00307352 5 8078 arc Out C 4.2 19164 18922 18777 7 01:00 b1-00307353 5 8078 arc Lef C 4.1 21367 20542 18913 HART 01:00 b2-00307353 5 8078 arc Lef C 4.2 19164 18922 18777 HART 01:00 r1-00307353 5 8078 arc Lef C 4.1 21367 20542 18913 HART 01:00 r2-00307353 5 8078 arc Lef C 4.2 19164 18922 18777 HART 8 01:01 b1-00307354 5 8078 arc Rig C 4.1 21367 20542 18913 HART 01:01 b2-00307354 5 8078 arc Rig C 4.2 19164 18922 18777 HART 01:01 r1-00307354 5 8078 arc Rig C 4.1 21367 20542 18913 HART 01:01 r2-00307354 5 8078 arc Rig C 4.2 19164 18922 18777 HART 9 01:03 b1-00307355 5 8078 arc Out C 4.1 25789 24963 23335 01:03 b2-00307355 5 8078 arc Out C 4.2 21845 21603 21457 01:03 r1-00307355 5 8078 arc Out C 4.1 25789 24963 23335 01:03 r2-00307355 5 8078 arc Out C 4.2 21845 21603 21457 10 01:04 b1-00307356 5 8078 fla Out C 25.1 25789 24963 23335 01:04 b2-00307356 5 8078 fla Out C 25.2 21845 21603 21457 01:04 r1-00307356 5 8078 fla Out C 25.1 25789 24963 23335 01:04 r2-00307356 5 8078 fla Out C 25.2 21845 21603 21457 11 01:16 b1-00307357 5 8078 arc Lef C 4.1 25789 24963 23335 HART 01:16 b2-00307357 5 8078 arc Lef C 4.2 21845 21603 21457 HART 01:16 r1-00307357 5 8078 arc Lef C 4.1 25789 24963 23335 HART 01:16 r2-00307357 5 8078 arc Lef C 4.2 21845 21603 21457 HART 12 01:16 b1-00307358 5 8078 arc Rig C 4.1 25789 24963 23335 HART 01:16 b2-00307358 5 8078 arc Rig C 4.2 21845 21603 21457 HART 01:16 r1-00307358 5 8078 arc Rig C 4.1 25789 24963 23335 HART 01:16 r2-00307358 5 8078 arc Rig C 4.2 21845 21603 21457 HART 13 01:18 b1-00307359 5 8078 arc Out C 4.1 25835 25003 23374 01:18 b2-00307359 5 8078 arc Out C 4.2 21529 21287 21142 01:18 r1-00307359 5 8078 arc Out C 4.1 25835 25003 23374 01:18 r2-00307359 5 8078 arc Out C 4.2 21529 21287 21142 14 01:19 b1-00307360 5 8078 fla Out C 25.1 25835 25003 23374 01:19 b2-00307360 5 8078 fla Out C 25.2 21529 21287 21142 01:19 r1-00307360 5 8078 fla Out C 25.1 25835 25003 23374 01:19 r2-00307360 5 8078 fla Out C 25.2 21529 21287 21142 15 01:33 b1-00307361 5 8078 sci Out N 900.1 25835 25003 23374 01:33 b2-00307361 5 8078 sci Out N 900.2 21529 21287 21142 01:33 r1-00307361 5 8078 sci Out N 900.1 25835 25003 23374 01:33 r2-00307361 5 8078 sci Out N 900.2 21529 21287 21142 16 01:50 b1-00307362 5 8078 sci Out S 900.1 25835 25003 23374 01:50 b2-00307362 5 8078 sci Out S 900.2 21529 21287 21142 01:50 r1-00307362 5 8078 sci Out S 900.1 25835 25003 23374 01:50 r2-00307362 5 8078 sci Out S 900.2 21529 21287 21142 17 02:06 b1-00307363 5 8078 sci Out E 900.1 25835 25003 23374 02:06 b2-00307363 5 8078 sci Out E 900.2 21529 21287 21142 02:06 r1-00307363 5 8078 sci Out E 900.1 25835 25003 23374 02:06 r2-00307363 5 8078 sci Out E 900.2 21529 21287 21142 18 02:22 b1-00307364 5 8078 sci Out N 900.1 25835 25003 23374 02:22 b2-00307364 5 8078 sci Out N 900.2 21529 21287 21142 02:22 r1-00307364 5 8078 sci Out N 900.1 25835 25003 23374 02:22 r2-00307364 5 8078 sci Out N 900.2 21529 21287 21142 19 02:39 b1-00307365 5 8078 sci Out S 900.1 25835 25003 23374 02:39 b2-00307365 5 8078 sci Out S 900.2 21529 21287 21142 02:39 r1-00307365 5 8078 sci Out S 900.1 25835 25003 23374 02:39 r2-00307365 5 8078 sci Out S 900.2 21529 21287 21142 20 02:55 b1-00307366 5 8078 sci Out E 900.1 25835 25003 23374 02:55 b2-00307366 5 8078 sci Out E 900.2 21529 21287 21142 02:55 r1-00307366 5 8078 sci Out E 900.1 25835 25003 23374 02:55 r2-00307366 5 8078 sci Out E 900.2 21529 21287 21142 21 03:12 b1-00307367 5 8078 fla Out C 25.1 25835 25003 23374 03:12 b2-00307367 5 8078 fla Out C 25.2 21529 21287 21142 03:12 r1-00307367 5 8078 fla Out C 25.1 25835 25003 23374 03:12 r2-00307367 5 8078 fla Out C 25.2 21529 21287 21142 22 03:15 b1-00307368 5 8078 arc Out C 4.1 25835 25003 23374 03:15 b2-00307368 5 8078 arc Out C 4.2 21529 21287 21142 03:15 r1-00307368 5 8078 arc Out C 4.1 25835 25003 23374 03:15 r2-00307368 5 8078 arc Out C 4.2 21529 21287 21142 23 06:41 b1-00307369 3 10515 arc Lef C 4.1 25835 25003 23374 HART 06:41 b2-00307369 3 10515 arc Lef C 4.2 21529 21287 21142 HART 06:41 r1-00307369 3 10515 arc Lef C 4.1 25835 25003 23374 HART 06:41 r2-00307369 3 10515 arc Lef C 4.2 21529 21287 21142 HART 24 06:42 b1-00307370 3 10515 arc Rig C 4.1 25835 25003 23374 HART 06:42 b2-00307370 3 10515 arc Rig C 4.2 21529 21287 21142 HART 06:42 r1-00307370 3 10515 arc Rig C 4.1 25835 25003 23374 HART 06:42 r2-00307370 3 10515 arc Rig C 4.2 21529 21287 21142 HART 25 06:45 b1-00307371 3 10515 arc Lef C 4.1 19357 18524 16894 HART 06:45 b2-00307371 3 10515 arc Lef C 4.2 21529 21287 21142 HART 06:45 r1-00307371 3 10515 arc Lef C 4.1 19357 18524 16894 HART 06:45 r2-00307371 3 10515 arc Lef C 4.2 21529 21287 21142 HART 26 06:46 b1-00307372 3 10515 arc Rig C 4.1 19357 18524 16894 HART 06:46 b2-00307372 3 10515 arc Rig C 4.2 21529 21287 21142 HART 06:46 r1-00307372 3 10515 arc Rig C 4.1 19357 18524 16894 HART 06:46 r2-00307372 3 10515 arc Rig C 4.2 21529 21287 21142 HART 27 06:48 b1-00307373 3 10515 arc Out C 4.1 18578 17750 16117 06:48 b2-00307373 3 10515 arc Out C 4.2 17114 16877 16727 06:48 r1-00307373 3 10515 arc Out C 4.1 18578 17750 16117 06:48 r2-00307373 3 10515 arc Out C 4.2 17114 16877 16727 28 06:49 b1-00307374 3 10515 fla Out C 25.1 18578 17750 16117 06:49 b2-00307374 3 10515 fla Out C 25.2 17114 16877 16727 06:49 r1-00307374 3 10515 fla Out C 25.1 18578 17750 16117 06:49 r2-00307374 3 10515 fla Out C 25.2 17114 16877 16727 29 06:55 b1-00307375 3 10515 sci Out N 900.1 18578 17750 16117 06:55 b2-00307375 3 10515 sci Out N 900.2 17114 16877 16727 06:55 r1-00307375 3 10515 sci Out N 900.1 18578 17750 16117 06:55 r2-00307375 3 10515 sci Out N 900.2 17114 16877 16727 30 07:11 b1-00307376 3 10515 sci Out S 900.1 18578 17750 16117 07:11 b2-00307376 3 10515 sci Out S 900.2 17114 16877 16727 07:11 r1-00307376 3 10515 sci Out S 900.1 18578 17750 16117 07:11 r2-00307376 3 10515 sci Out S 900.2 17114 16877 16727 31 07:27 b1-00307377 3 10515 sci Out E 900.1 18578 17750 16117 07:27 b2-00307377 3 10515 sci Out E 900.2 17114 16877 16727 07:27 r1-00307377 3 10515 sci Out E 900.1 18578 17750 16117 07:27 r2-00307377 3 10515 sci Out E 900.2 17114 16877 16727 32 07:44 b1-00307378 3 10515 sci Out N 900.1 18578 17750 16117 07:44 b2-00307378 3 10515 sci Out N 900.2 17114 16877 16727 07:44 r1-00307378 3 10515 sci Out N 900.1 18578 17750 16117 07:44 r2-00307378 3 10515 sci Out N 900.2 17114 16877 16727 33 08:00 b1-00307379 3 10515 sci Out S 900.1 18578 17750 16117 08:00 b2-00307379 3 10515 sci Out S 900.2 17114 16877 16727 08:00 r1-00307379 3 10515 sci Out S 900.1 18578 17750 16117 08:00 r2-00307379 3 10515 sci Out S 900.2 17114 16877 16727 34 08:17 b1-00307380 3 10515 sci Out E 900.1 18578 17750 16117 08:17 b2-00307380 3 10515 sci Out E 900.2 17114 16877 16727 08:17 r1-00307380 3 10515 sci Out E 900.1 18578 17750 16117 08:17 r2-00307380 3 10515 sci Out E 900.2 17114 16877 16727 35 08:33 b1-00307381 3 10515 sci Out N 900.1 18578 17750 16117 08:33 b2-00307381 3 10515 sci Out N 900.2 17114 16877 16727 08:33 r1-00307381 3 10515 sci Out N 900.1 18578 17750 16117 08:33 r2-00307381 3 10515 sci Out N 900.2 17114 16877 16727 36 08:49 b1-00307382 3 10515 sci Out S 900.1 18578 17750 16117 08:49 b2-00307382 3 10515 sci Out S 900.2 17114 16877 16727 08:49 r1-00307382 3 10515 sci Out S 900.1 18578 17750 16117 08:49 r2-00307382 3 10515 sci Out S 900.2 17114 16877 16727 37 09:06 b1-00307383 3 10515 sci Out E 900.1 18578 17750 16117 09:06 b2-00307383 3 10515 sci Out E 900.2 17114 16877 16727 09:06 r1-00307383 3 10515 sci Out E 900.1 18578 17750 16117 09:06 r2-00307383 3 10515 sci Out E 900.2 17114 16877 16727 38 09:23 b1-00307384 3 10515 fla Out C 25.1 18578 17750 16117 09:23 b2-00307384 3 10515 fla Out C 25.2 17114 16877 16727 09:23 r1-00307384 3 10515 fla Out C 25.1 18578 17750 16117 09:23 r2-00307384 3 10515 fla Out C 25.2 17114 16877 16727 39 09:26 b1-00307385 3 10515 arc Out C 4.1 18578 17750 16117 09:26 b2-00307385 3 10515 arc Out C 4.2 17114 16877 16727 09:26 r1-00307385 3 10515 arc Out C 4.1 18578 17750 16117 09:26 r2-00307385 3 10515 arc Out C 4.2 17114 16877 16727 40 09:32 b1-00307386 3 10515 sci Out N 900.1 18578 17750 16117 09:32 b2-00307386 3 10515 sci Out N 900.2 17114 16877 16727 09:32 r1-00307386 3 10515 sci Out N 900.1 18578 17750 16117 09:32 r2-00307386 3 10515 sci Out N 900.2 17114 16877 16727 41 09:48 b1-00307388 3 10515 sci Out S 74.1 18578 17750 16117 09:48 b2-00307388 3 10515 sci Out S 74.4 17114 16877 16727 09:48 r1-00307388 3 10515 sci Out S 74.1 18578 17750 16117 09:48 r2-00307388 3 10515 sci Out S 74.4 17114 16877 16727 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- Weather Log ---- --- Weather --- Time Cart Temp DP Diff Humid Wind Dir 1umDst IRSCSig IRSCm ================================================================================ 01:31 5-8078A -2.9 -8.6 5.7 65 7 330 262 4.7 332.0 01:48 5-8078A -2.8 -9.0 6.2 62 17 324 248 3.8 332.0 02:04 5-8078A -2.8 -9.6 6.8 59 6 311 232 11.6 334.0 02:20 5-8078A -2.8 -9.4 6.6 60 7 343 257 41.0 339.0 02:37 5-8078A -2.7 -10.0 7.3 57 6 336 208 18.7 337.0 02:53 5-8078A -2.6 -9.8 7.2 57 3 295 171 33.6 346.0 03:10 5-8078A -2.4 -9.6 7.2 58 2 7 198 30.5 336.0 06:26 3-10515A -3.9 -8.1 4.2 73 15 7 224 8.8 331.0 06:53 3-10515A -4.2 -9.4 5.2 66 15 30 202 32.1 331.0 07:09 3-10515A -4.3 -9.6 5.3 66 18 37 100 3.6 332.0 07:25 3-10515A -4.4 -9.3 4.9 68 18 17 100 45.5 339.0 07:42 3-10515A -4.4 -8.9 4.5 70 17 32 119 96.6 396.0 07:58 3-10515A -4.3 -8.9 4.6 70 17 55 145 13.9 332.0 08:15 3-10515A -4.3 -9.3 5.0 67 15 53 109 6.9 332.0 08:31 3-10515A -4.6 -9.2 4.6 70 16 59 113 3.0 332.0 08:47 3-10515A -4.4 -9.4 5.0 68 19 56 630 3.7 332.0 09:04 3-10515A -5.0 -9.0 4.0 73 16 65 205 3.6 331.0 09:20 3-10515A -4.9 -9.0 4.1 73 18 69 179 3.5 332.0 09:30 3-10515A -5.0 -8.9 3.9 73 21 71 242 3.4 332.0 09:47 3-10515A -5.7 -8.1 2.4 82 32 71 521 3.1 332.0 ---- Problems in Detail ---- Here are the time stamps of the frame_break errors for Az (from the current.log file in /data/logs/tron/mcp): 2020-01-30 23:50:40,255Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:50:45,261Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:50:50,243Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:50:55,243Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:51:00,243Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:51:05,242Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:51:10,246Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:51:15,267Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:51:20,242Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:51:25,281Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:52:37,809Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:52:42,889Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:52:47,865Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:52:52,868Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:52:57,881Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:02,904Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:07,784Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:12,889Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:17,865Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:22,783Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:27,783Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:32,783Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:37,839Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:42,874Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:47,783Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:52,783Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:53:57,783Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:54:52,794Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:54:52,825Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:54:57,788Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:54:57,825Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:55:02,787Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:55:02,831Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:55:07,821Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:55:07,864Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:55:12,787Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:55:12,825Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:55:17,787Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:55:17,826Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:56:44,811Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:56:44,835Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:56:49,798Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:56:49,870Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:56:54,798Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:56:54,835Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:56:59,798Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:56:59,835Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:57:04,797Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:57:04,841Z < 0 0 i trace="rotator reposition caused frame break" 2020-01-30 23:57:43,472Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:57:48,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:57:53,378Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:57:58,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:03,378Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:08,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:13,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:18,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:23,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:28,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:33,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:38,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:43,370Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:48,420Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:53,414Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:58:58,370Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:03,370Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:08,370Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:13,370Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:18,376Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:23,488Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:28,370Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:33,432Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:38,396Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:43,441Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:48,453Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:53,509Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-30 23:59:58,375Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:03,375Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:08,405Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:13,406Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:18,375Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:23,375Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:28,375Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:33,374Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:38,374Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:43,412Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:48,418Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:53,374Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:00:58,374Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:03,386Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:08,374Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:13,374Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:18,373Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:23,373Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:28,373Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:33,373Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:38,457Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:43,404Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:48,385Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:53,373Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:01:58,423Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:03,373Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:08,412Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:13,425Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:18,416Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:23,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:28,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:33,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:38,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:43,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:48,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:53,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:02:58,372Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:03:03,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:03:08,371Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:03:13,460Z < 0 0 i trace="azimuth reposition caused frame break" 2020-01-31 00:03:18,408Z < 0 0 i trace="azimuth reposition caused frame break" altitude reposition caused frame break: 06:22:00 .mcp 0 mcp ! command="move" === 06:33:12 .hub 0 cmds d CmdQueued=367486,1580452392.04,"OBSERVER.telescope.sop",484,"tcc",74966,"track 121.000000, 30.000000 mount/rottype=mount/rotangle=0.000000" 06:33:51 .mcp 0 mcp ! command="move" 06:33:52 .hub 0 cmds d CmdQueued=367495,1580452432.24,"OBSERVER.telescope.sop",485,"tcc",74968,"axis stop" 06:33:52 OBSERVER.telescope 160 sop e text="tcc track command ended with some bad bits set: 0x1800,0x1800,0x0" 06:34:18 OBSERVER.telescope 163 sop e text="Trying to start slew with axis states: Halted,Halted,Halted, error codes: HaltRequested,HaltRequested,HaltRequested, and status bits: 0x1800,0x800,0x0" 06:34:18 OBSERVER.telescope 163 sop f text="Failed to slew to position az=121, alt=30, rot=0" 06:34:18 CmdDone: OBSERVER.telescope 163 sop gotoStow === 06:36:09 OBSERVER.telescope 169 sop e text="tcc track command ended with some bad bits set: 0x1800,0x1800,0x0" 06:36:09 .hub 0 cmds d CmdQueued=367535,1580452569.66,"OBSERVER.telescope.sop",495,"tcc",74987,"axis stop" 06:36:09 OBSERVER.telescope 169 sop e text="Failed to complete slew: see TCC messages for details." === 06:38:29 .mcp 0 mcp ! command="move" 06:38:29 OBSERVER.telescope 177 sop e text="tcc track command ended with some bad bits set: 0x0,0x1800,0x0" 06:38:29 .hub 0 cmds d CmdQueued=367575,1580452709.57,"OBSERVER.telescope.sop",503,"tcc",75000,"axis stop" 06:38:29 OBSERVER.telescope 177 sop e text="Failed to complete slew: see TCC messages for details." 06:38:29 .tcc 0 tcc w AxisCmdState=Halting,Halting,Halting; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 06:38:29 OBSERVER.telescope 177 sop f text="Failed to slew to position az=60, alt=60, rot=60" 06:38:29 CmdDone: OBSERVER.telescope 177 sop gotoAll60