Subject: 2.5m obslog 20190906 (SJD 58733) From: asimmons@apo.nmsu.edu Date: Sat, 7 Sep 2019 11:27:46 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday September 06, 2019 (SJD 58733) ---- Observing Team ---- Elena Malanushenko Audrey Oravetz Dylan Gatlin (training) Kaike Pan (support) ---- Observing Plan ---- Fri 6 SEP 2019 58733.0 0.57 01:57 11:33 17:58 03:34 ---- APOGEE-2 until 06:55 ( 5.0 h), then MaNGA ( 4.6 h) ---- Observing Summary ---- Clear night. Good seeing. Cart 9, AP plate 11268, K19_076+07_btx; 2XDAB Missing fibers: [36, 211, 245, 273] Faint fibers: [123, '126 - 127'] ----------------- Cart 8, Ap plate 11463, NGC6791_btx; 4XAB Missing fibers: [35, 184, 211, 227, 273] Faint fibers: [123, 127] Cart 6, Ap plate 11421, N6819-RV_btx; 4XAB Missing fibers: [181, 211, 273] Faint fibers: [15, 34, 105, 121, 123, 125, 127, 289] Cart 3, Ap plate 11466, NGC7789_btx; 4XAB Missing fibers: [211, 273] Faint fibers: [123, 125, 127, '133 - 134'] Cart 7, Ap plate 11213, 135-24_btx; 4XAB, done Missing fibers: [211, 223, 273] Faint fibers: [123, 127, 148, 173, 202] Cart 1, Ap plate 11602, W34_btx; 2XAB Missing fibers: [95, 211, 273] Faint fibers: ['1 - 30', 148, '151 - 180', 227] Notes: ====== We operated the crane for cart changes. APOGEE lost 38 minutes due to a software disconnect. One APOGEE exposure hung mid-way through integration, which severed the connection with any apogee or apogeecal actor. Kaike helped remotely by restarting the apogee-ics machine. Same error as yesterday with gotoGangChange: 'INIT'; STATUS: Cancelled on queue 05:54Z - M2 hung once and we followed Conor's steps to restart the mirror controller. Cart 1 seems to have many faint fibers, please check. 1.1 hrs APOGEE lost due to weather (high humidity) ---- Observing Log ---- Afternoon ---------- Telescope has been checked, all system are working as expected. eBoss cameras are not mounted on telescope. Cartridge buy manipulator is not operated. gcamera temperature was low, set it to -35 degrees. Mirror numbers are not available. Apogee evening calibrations has been done. Telescope has been checked, movement, fiducial reading, all ok. 01:00 started ventilation. Night ====== 02:08:58 encl25m : open; **************** 01:44:44 loadCart: Cart 9, Plate 11268; K19_076+07_btx 02:13:42 sop.gotoField = running; 02:20:59 apogee.expose=apRaw-31710010, 94, A, Object 02:37:41 apogee.expose=apRaw-31710011, 94, B, Object This exposure hung on 81 reads of 93, and with this STUI message, we were unable to start the next exposure sequence. The apogee actor/ics machine failed to respond to any restart/initialization. We do not have permission to access apogee-ics. We called Kaike. We was able to log into the apogee-ics machine to restart the process. Afterwards, hub startNubs worked and the actors began to respond again. 02:54:02 .tcc 0 tcc w Text="Collimation Failure: Cancelled while queued by new command '949 move -835.9427,8.7039,10.8271,-40.2729,-133.1170'" 02:54:09 .boss 3893 boss w text="camcheck found missing or unresponsive cameras: set(['sp1cam'])" 02:54:16 .hub 0 cmds d CmdQueued=1299083,1567824855.82,"OBSERVER.elenam.guider",376,"gcamera",3343,"expose time=5.0 stack=3" 02:54:24 OBSERVER.elenam 57 sop e text="failed to start object exposure" 02:54:24 OBSERVER.elenam 57 sop f text="Failed to take an object exposure" 02:54:24 CmdDone: OBSERVER.elenam 57 sop doApogeeScience ditherPairs=2 expTime=\"1000.0\" 03:16:42 apogee.expose=apRaw-31710012, 3, A, Dark. After a reset of the machine. 03:21:48 apogee.expose=apRaw-31710013, 94, B, Object 03:54 apogee.expose=apRaw-31710014, 94, A, Object 04:03:41 apogee.expose=apRaw-31710015, 94, B, Object 04:14:06 apogee.expose=apRaw-31710016, 94, A, Object 04:15:27 apogee.expose=apRaw-31710017, 5, A, DomeFlat Missing fibers: [36, 211, 245, 273] Faint fibers: [123, '126 - 127'] **************** Cart 8, Ap plate 11463, NGC6791_btx 04:29:14 apogee.expose=apRaw-31710020, 47, A, Object 04:37:38 apogee.expose=apRaw-31710021, 47, B, Object 04:45:58 apogee.expose=apRaw-31710022, 47, B, Object 04:54:25 apogee.expose=apRaw-31710023, 47, A, Object 05:02:45 apogee.expose=apRaw-31710024, 47, A, Object 05:11:08 apogee.expose=apRaw-31710025, 47, B, Object 05:19:28 apogee.expose=apRaw-31710026, 47, B, Object 05:27:56 apogee.expose=apRaw-31710027, 47, A, Object 05:36:48 apogee.expose=apRaw-31710028, 5, A, DomeFlat 05:40:58 apogee.expose=apRaw-31710029, 10, A, Dark 05:42:45 apogee.expose=apRaw-31710030, 10, A, Dark Missing fibers: [35, 184, 211, 227, 273] Faint fibers: [123, 127] *************** Cart 6, Ap plate 11421, N6819-RV_btx 06:11:50 apogee.expose=apRaw-31710031, 47, A, Object 06:20:14 apogee.expose=apRaw-31710032, 47, B, Object 06:28:33 apogee.expose=apRaw-31710033, 47, B, Object 06:37:01 apogee.expose=apRaw-31710034, 47, A, Object 06:45:21 apogee.expose=apRaw-31710035, 47, A, Object 06:53:44 apogee.expose=apRaw-31710036, 47, B, Object 07:02:04 apogee.expose=apRaw-31710037, 47, B, Object 07:10:32 apogee.expose=apRaw-31710038, 47, A, Object 07:19:29 apogee.expose=apRaw-31710039, 5, A, DomeFlat 07:23:36 apogee.expose=apRaw-31710040, 10, A, Dark 07:25:24 apogee.expose=apRaw-31710041, 10, A, Dark Missing fibers: [181, 211, 273] Faint fibers: [15, 34, 105, 121, 123, 125, 127, 289] ************** Cart 3, Ap plate 11466, NGC7789_btx 07:37:06 apogee.expose=apRaw-31710042, 47, A, Object 07:45:29 apogee.expose=apRaw-31710043, 47, B, Object 07:53:49 apogee.expose=apRaw-31710044, 47, B, Object 08:02:16 apogee.expose=apRaw-31710045, 47, A, Object 08:10:36 apogee.expose=apRaw-31710046, 47, A, Object 08:18:59 apogee.expose=apRaw-31710047, 47, B, Object 08:27:19 apogee.expose=apRaw-31710048, 47, B, Object 08:35:47 apogee.expose=apRaw-31710049, 47, A, Object 08:44:35 apogee.expose=apRaw-31710050, 5, A, DomeFlat 08:48:18 apogee.expose=apRaw-31710051, 10, A, Dark 08:50:06 apogee.expose=apRaw-31710052, 10, A, Dark Missing fibers: [211, 273] Faint fibers: [123, 125, 127, '133 - 134'] *************** Cart 7, Ap plate 11213, 135-24_btx 09:05:01 apogee.expose=apRaw-31710053, 47, A, Object 09:13:25 apogee.expose=apRaw-31710054, 47, B, Object 09:21:45 apogee.expose=apRaw-31710055, 47, B, Object 09:30:13 apogee.expose=apRaw-31710056, 47, A, Object 09:38:32 apogee.expose=apRaw-31710057, 47, A, Object 09:46:56 apogee.expose=apRaw-31710058, 47, B, Object 09:55:16 apogee.expose=apRaw-31710059, 47, B, Object 10:03:43 apogee.expose=apRaw-31710060, 47, A, Object 10:12:23 apogee.expose=apRaw-31710061, 5, A, DomeFlat 10:15:06 apogee.expose=apRaw-31710062, 10, A, Dark 10:16:54 apogee.expose=apRaw-31710063, 10, A, Dark 10:20:45 apogee.expose=apRaw-31710064, 5, A, DomeFlat -- Please ignore Missing fibers: [211, 223, 273] Faint fibers: [123, 127, 148, 173, 202] ************** Cart 1, Ap plate 11602, W34_btx 10:27:55 apogee.expose=apRaw-31710065, 47, A, Object 10:36:19 apogee.expose=apRaw-31710066, 47, B, Object 10:44:38 apogee.expose=apRaw-31710067, 47, B, Object 10:53:06 apogee.expose=apRaw-31710068, 47, A, Object 11:01:42 apogee.expose=apRaw-31710069, 5, A, DomeFlat 11:12:21 apogee.expose=apRaw-31710070, 60, A, Dark Missing fibers: [95, 211, 273] Faint fibers: ['1 - 30', 148, '151 - 180', 227] 11:22Z Closed enclosure, stopped ventilating, closed early due to high humidity, started APOGEE morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: tpm_MIG1A --- Invalid channel name tpm_MIG1A --- Invalid channel name Axial B: tpm_MIG1B --- Invalid channel name tpm_MIG1B --- Invalid channel name Axial C: tpm_MIG1C --- Invalid channel name tpm_MIG1C --- Invalid channel name Trans D: tpm_MIG1DT --- Invalid channel name tpm_MIG1DT --- Invalid channel name Lateral E: tpm_MIG1EL --- Invalid channel name 9.3345 Lateral F: tpm_MIG1FL --- Invalid channel name 14.3637 GALILS Commanded: 11200 -1850 6850 29650 21250 21050 Actual: 11200 -1850 6850 29650 21250 21050 SETMIR VALUES PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 PrimOrient: 0.0 -11.4 6.58 375.38 620.77 SECONDARY: ---------- Focus: 0.0000 Air temp: 17.3 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -6.665 -6.668 Axial B: -0.036 tpm_MIG2B --- Invalid channel name Axial C: -6.67 tpm_MIG2C --- Invalid channel name Trans D: -0.08 -0.074 GALILS Commanded: 1058750 1101250 1079500 -7227 -6550 Actual: 1058687 1101628 1080123 -7250 -6550 SETMIR VALUES SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 SecOrient: 857.14 2.61 12.85 -38.93 157.36 PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: -5.33 tpm_MIG1A --- Invalid channel name Axial B: -5.052 tpm_MIG1B --- Invalid channel name Axial C: tpm_MIG1C --- Invalid channel name tpm_MIG1C --- Invalid channel name Trans D: -0.576 tpm_MIG1DT --- Invalid channel name Lateral E: 9.3472 9.3345 Lateral F: 14.4399 14.3637 GALILS Commanded: 11200 -1850 6850 29650 21250 21050 Actual: 11200 -1850 6850 29650 21250 21050 SETMIR VALUES PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 PrimOrient: 0.0 -11.4 6.58 375.38 620.77 SECONDARY: ---------- Focus: 0.0000 Air temp: 17.2 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -6.665 -6.668 Axial B: -0.036 tpm_MIG2B --- Invalid channel name Axial C: -6.67 tpm_MIG2C --- Invalid channel name Trans D: -0.08 -0.074 GALILS Commanded: 1058750 1101400 1079700 -7250 -6550 Actual: 1058762 1101477 1079814 -7250 -6550 SETMIR VALUES SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 SecOrient: 857.06 2.66 12.78 -38.41 156.75 ---- Telescope Offsets and Scale ---- --- Offsets --- (arcsec) Time Inst Az Alt Rot objArcOff guideRot calibOff -------------------------------------------------------------------------------- 04:03 9-11268A 194.2 81.5 15.6 ( -4.1, -8.0) (-371.3) ( 0.0, 0.0, 0.0) apog 04:03 9-11268A 194.3 81.4 15.8 ( -4.1, -8.0) (-371.3) ( 0.0, 0.0, 0.0) 04:13 9-11268A 205.9 80.7 28.9 ( -1.4, -7.3) (-372.0) ( 0.0, 0.0, 0.0) apog 04:28 8-11463A 246.4 75.6 75.8 ( 2.3, -3.9) (-285.0) ( 0.0, 0.0, 0.0) apog 04:37 8-11463A 247.8 74.0 78.5 ( 2.7, -3.6) (-287.5) ( 0.0, 0.0, 0.0) apog 04:45 8-11463A 248.8 72.3 80.7 ( 2.1, -3.2) (-281.7) ( 0.0, 0.0, 0.0) apog 04:53 8-11463A 249.5 70.7 82.7 ( 2.2, -2.6) (-280.8) ( 0.0, 0.0, 0.0) apog 05:02 8-11463A 250.0 69.0 84.4 ( 3.2, -1.9) (-282.6) ( 0.0, 0.0, 0.0) apog 05:10 8-11463A 250.4 67.4 86.1 ( 3.1, -1.8) (-281.9) ( 0.0, 0.0, 0.0) apog 05:18 8-11463A 250.6 65.7 87.5 ( 3.0, -1.5) (-281.1) ( 0.0, 0.0, 0.0) apog 05:27 8-11463A 250.7 64.1 88.9 ( 3.1, -1.1) (-282.4) ( 0.0, 0.0, 0.0) apog 06:11 6-11421A 246.8 58.9 89.6 ( 4.8, -1.9) (-274.9) ( 0.0, 0.0, 0.0) apog 06:19 6-11421A 246.8 57.3 90.9 ( 3.4, -1.8) (-280.3) ( 0.0, 0.0, 0.0) apog 06:27 6-11421A 246.7 55.6 92.2 ( 3.6, -1.6) (-278.9) ( 0.0, 0.0, 0.0) apog 06:36 6-11421A 246.6 54.0 93.4 ( 3.7, -1.6) (-276.9) ( 0.0, 0.0, 0.0) apog 06:44 6-11421A 246.4 52.4 94.6 ( 3.7, -1.6) (-278.5) ( 0.0, 0.0, 0.0) apog 06:53 6-11421A 246.1 50.8 95.7 ( 4.0, -1.7) (-277.7) ( 0.0, 0.0, 0.0) apog 07:01 6-11421A 245.9 49.2 96.8 ( 3.9, -1.8) (-277.9) ( 0.0, 0.0, 0.0) apog 07:09 6-11421A 245.5 47.5 97.9 ( 4.1, -1.6) (-279.9) ( 0.0, 0.0, 0.0) apog 07:36 3-11466A 173.0 65.7 -10.8 ( -7.1, -6.4) (-381.3) ( 0.0, 0.0, 0.0) apog 07:44 3-11466A 175.8 65.9 -6.5 ( -6.9, -6.8) (-353.1) ( 0.0, 0.0, 0.0) apog 07:53 3-11466A 178.6 66.0 -2.2 ( -6.2, -7.2) (-349.6) ( 0.0, 0.0, 0.0) apog 08:01 3-11466A 181.4 66.0 2.1 ( -5.6, -7.5) (-348.0) ( 0.0, 0.0, 0.0) apog 08:09 3-11466A 184.2 65.9 6.4 ( -4.9, -7.3) (-349.9) ( 0.0, 0.0, 0.0) apog 08:18 3-11466A 187.0 65.7 10.7 ( -4.3, -7.7) (-349.5) ( 0.0, 0.0, 0.0) apog 08:26 3-11466A 189.7 65.5 14.9 ( -3.8, -7.6) (-343.0) ( 0.0, 0.0, 0.0) apog 08:35 3-11466A 192.3 65.1 19.1 ( -3.1, -7.5) (-340.7) ( 0.0, 0.0, 0.0) apog 09:04 7-11213A 117.8 80.5 -68.3 ( -3.6, -3.4) (-338.4) ( 0.0, 0.0, 0.0) apog 09:12 7-11213A 122.4 82.1 -62.4 ( -2.8, -3.4) (-326.7) ( 0.0, 0.0, 0.0) apog 09:21 7-11213A 129.4 83.5 -54.1 ( -1.6, -3.4) (-327.3) ( 0.0, 0.0, 0.0) apog 09:29 7-11213A 140.5 84.8 -41.9 ( 0.0, -3.0) (-327.5) ( 0.0, 0.0, 0.0) apog 09:37 7-11213A 157.5 85.7 -23.7 ( 1.7, -1.5) (-324.6) ( 0.0, 0.0, 0.0) apog 09:46 7-11213A 180.5 86.0 0.5 ( 2.5, 0.4) (-316.2) ( 0.0, 0.0, 0.0) apog 09:54 7-11213A 203.1 85.7 24.3 ( 3.2, 2.3) (-312.7) ( 0.0, 0.0, 0.0) apog 10:03 7-11213A 220.0 84.7 42.4 ( 3.1, 4.2) (-313.3) ( 0.0, 0.0, 0.0) apog 10:27 1-11602A 179.6 61.3 -0.6 ( -7.6, -7.2) (-352.1) ( 0.0, 0.0, 0.0) apog 10:35 1-11602A 181.7 61.3 3.1 ( -7.5, -8.3) (-340.5) ( 0.0, 0.0, 0.0) apog 10:44 1-11602A 183.8 61.2 6.8 ( -7.1, -8.7) (-342.8) ( 0.0, 0.0, 0.0) apog 10:52 1-11602A 185.9 61.1 10.4 ( -6.6, -8.9) (-343.3) ( 0.0, 0.0, 0.0) apog ---- Focus Log ---- --- Focus --- Time Inst Scale M1 M2 Focus Az Alt Temp Wind Dir -------------------------------------------------------------------------------- 04:03 9-11268A 1.000170 -1678 -790 -115 194.2 81.5 15.2 14 113 nan apog 04:03 9-11268A 1.000170 -1678 -788 -115 194.3 81.4 15.2 14 113 nan 04:13 9-11268A 1.000170 -1678 -736 -65 205.9 80.7 14.9 14 112 nan apog 04:28 8-11463A 1.000170 -1678 -682 -15 246.4 75.6 14.4 15 108 nan apog 04:37 8-11463A 1.000170 -1678 -662 2 247.8 74.0 14.6 18 109 nan apog 04:45 8-11463A 1.000170 -1678 -651 10 248.8 72.3 14.5 15 112 nan apog 04:53 8-11463A 1.000170 -1678 -637 22 249.5 70.7 14.3 16 112 nan apog 05:02 8-11463A 1.000200 -1975 -871 30 250.0 69.0 14.1 17 121 nan apog 05:10 8-11463A 1.000200 -1975 -886 11 250.4 67.4 14.2 17 115 nan apog 05:18 8-11463A 1.000200 -1975 -908 -12 250.6 65.7 14.6 15 125 nan apog 05:27 8-11463A 1.000200 -1975 -911 -40 250.7 64.1 14.4 13 116 nan apog 06:11 6-11421A 1.000218 -2153 -1142 -115 246.8 58.9 14.2 17 123 nan apog 06:19 6-11421A 1.000128 -1264 -402 -115 246.8 57.3 14.2 14 124 nan apog 06:27 6-11421A 1.000128 -1264 -402 -119 246.7 55.6 14.1 12 112 nan apog 06:36 6-11421A 1.000128 -1264 -400 -122 246.6 54.0 14.0 17 126 nan apog 06:44 6-11421A 1.000128 -1264 -395 -122 246.4 52.4 14.0 16 123 nan apog 06:53 6-11421A 1.000128 -1264 -392 -124 246.1 50.8 13.9 18 124 nan apog 07:01 6-11421A 1.000128 -1264 -389 -129 245.9 49.2 13.9 11 123 nan apog 07:09 6-11421A 1.000158 -1264 -392 -135 245.5 47.5 13.8 17 127 nan apog 07:36 3-11466A 1.000158 -1560 -706 -153 173.0 65.7 13.8 15 128 nan apog 07:44 3-11466A 1.000158 -1560 -717 -165 175.8 65.9 13.9 8 111 nan apog 07:53 3-11466A 1.000158 -1560 -715 -162 178.6 66.0 13.9 12 126 nan apog 08:01 3-11466A 1.000158 -1560 -724 -172 181.4 66.0 13.9 15 132 nan apog 08:09 3-11466A 1.000158 -1560 -736 -184 184.2 65.9 13.9 11 134 nan apog 08:18 3-11466A 1.000158 -1560 -742 -190 187.0 65.7 13.9 14 132 nan apog 08:26 3-11466A 1.000158 -1560 -745 -191 189.7 65.5 13.8 11 126 nan apog 08:35 3-11466A 1.000158 -1560 -737 -187 192.3 65.1 13.7 8 119 nan apog 09:04 7-11213A 1.000158 -1560 -765 -191 117.8 80.5 13.3 6 142 nan apog 09:12 7-11213A 1.000188 -1857 -1015 -196 122.4 82.1 13.4 7 114 nan apog 09:21 7-11213A 1.000188 -1857 -1012 -192 129.4 83.5 13.5 14 138 nan apog 09:29 7-11213A 1.000188 -1857 -1020 -199 140.5 84.8 13.5 15 134 nan apog 09:37 7-11213A 1.000188 -1857 -1025 -203 157.5 85.7 13.4 14 135 nan apog 09:46 7-11213A 1.000188 -1857 -1028 -206 180.5 86.0 13.5 14 135 nan apog 09:54 7-11213A 1.000218 -2153 -1265 -198 203.1 85.7 13.3 13 137 nan apog 10:03 7-11213A 1.000218 -2153 -1251 -184 220.0 84.7 13.2 10 134 nan apog 10:27 1-11602A 1.000218 -2153 -1215 -182 179.6 61.3 12.8 9 152 nan apog 10:35 1-11602A 1.000218 -2153 -1232 -199 181.7 61.3 12.8 6 166 nan apog 10:44 1-11602A 1.000218 -2153 -1247 -214 183.8 61.2 12.6 5 179 nan apog 10:52 1-11602A 1.000218 -2153 -1242 -207 185.9 61.1 12.4 3 134 nan apog ---- Telescope Status ---- 00:30Z Started ventilation 01:30Z encl25m : open 11:22Z encl25m : closed Status at:11:23Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 1 Counterweights at: 270 LN2 autofill systems: Connected and turned on ---- Software Used ---- Get Versions, 09/07/19, 00:56:53 -- STUI software: STUI : 1.7.8 alerts : trunk apo : trunk boss : v4_0 -boss.specMec : 3.0.4 gcamera : v2_2 guider : 3.8.1 hartmann : 1.6.0 hub : Tag: v4.2 mcp : Linux -mcp.plc : 34 -mcp.azFiduci : v1_140 -mcp.altFiduc : v1_140 -mcp.rotFiduc : v1_140 platedb : 2.8.0 sop : 3.12.3 tcc : 1.3.1 apogee : not availble apogeecal : not availble apogeeql : v1_6_1 APO-local : not availble -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_12 petunia : 1.4.2 autoscheduler : 1.3.0 sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- APOGEE data list, mjd=58733, all raw_data: /data/apogee/utr_cdr/58733/apRaw-*.fits archive: /data/apogee/archive/58733/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/58733/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 58733 1 23:04 31710001 Dark 60 A 13.0 6-10351 a-b-c - - 58733 2 23:15 31710002 QuarFlat 10 A 13.0 6-10351 a-b-c - 50 58733 3 23:17 31710003 Arc-Thar 12 A 13.0 6-10351 a-b-c -3.30 - 58733 4 23:19 31710004 Arc-Une 40 A 13.0 6-10351 a-b-c - - 58733 5 23:26 31710005 Arc-Thar 12 B 13.5 6-10351 a-b-c -2.80 - 58733 6 23:28 31710006 Arc-Une 40 B 13.5 6-10351 a-b-c - - 58733 7 23:36 31710007 Dark 60 A 13.0 6-10351 a-b-c - - 58733 8 02:08 31710008 Dark 10 A 13.0 9-11268 a-b-c - - 58733 9 02:09 31710009 Dark 10 A 13.0 9-11268 a-b-c - - 58733 10 02:20 31710010 Object 94 A 13.0 9-11268 a-b-c - - 58733 11 02:51 31710011 Object 94 B 13.5 ------- x-x-x - - 58733 12 03:16 31710012 Dark 3 A 13.0 9-11268 a-b-c - - 58733 13 03:21 31710013 Object 94 B 13.5 9-11268 a-b-c - - 58733 14 03:38 31710014 Object 94 A 13.0 9-11268 a-b-c - - 58733 15 03:57 31710015 Object 94 B 13.5 9-11268 a-b-c - - 58733 16 04:13 31710016 Object 94 A 13.0 9-11268 a-b-c - - 58733 17 04:15 31710017 DomeFlat 5 A 13.0 9-11268 a-b-c - 91 58733 18 04:18 31710018 Dark 10 A 13.0 9-11268 a-b-c - - 58733 19 04:20 31710019 Dark 10 A 13.0 9-11268 a-b-c - - 58733 20 04:29 31710020 Object 47 A 13.0 8-11463 a-b-c - - 58733 21 04:37 31710021 Object 47 B 13.5 8-11463 a-b-c - - 58733 22 04:45 31710022 Object 47 B 13.5 8-11463 a-b-c - - 58733 23 04:54 31710023 Object 47 A 13.0 8-11463 a-b-c - - 58733 24 05:02 31710024 Object 47 A 13.0 8-11463 a-b-c - - 58733 25 05:10 31710025 Object 47 B 13.5 8-11463 a-b-c - - 58733 26 05:19 31710026 Object 47 B 13.5 8-11463 a-b-c - - 58733 27 05:27 31710027 Object 47 A 13.0 8-11463 a-b-c - - 58733 28 05:36 31710028 DomeFlat 5 A 13.0 8-11463 a-b-c - 80 58733 29 05:40 31710029 Dark 10 A 13.0 8-11463 a-b-c - - 58733 30 05:42 31710030 Dark 10 A 13.0 8-11463 a-b-c - - 58733 31 06:11 31710031 Object 47 A 13.0 6-11421 a-b-c - - 58733 32 06:20 31710032 Object 47 B 13.5 6-11421 a-b-c - - 58733 33 06:28 31710033 Object 47 B 13.5 6-11421 a-b-c - - 58733 34 06:36 31710034 Object 47 A 13.0 6-11421 a-b-c - - 58733 35 06:45 31710035 Object 47 A 13.0 6-11421 a-b-c - - 58733 36 06:53 31710036 Object 47 B 13.5 6-11421 a-b-c - - 58733 37 07:01 31710037 Object 47 B 13.5 6-11421 a-b-c - - 58733 38 07:10 31710038 Object 47 A 13.0 6-11421 a-b-c - - 58733 39 07:19 31710039 DomeFlat 5 A 13.0 6-11421 a-b-c - 77 58733 40 07:23 31710040 Dark 10 A 13.0 6-11421 a-b-c - - 58733 41 07:25 31710041 Dark 10 A 13.0 6-11421 a-b-c - - 58733 42 07:36 31710042 Object 47 A 13.0 3-11466 a-b-c - - 58733 43 07:45 31710043 Object 47 B 13.5 3-11466 a-b-c - - 58733 44 07:53 31710044 Object 47 B 13.5 3-11466 a-b-c - - 58733 45 08:02 31710045 Object 47 A 13.0 3-11466 a-b-c - - 58733 46 08:10 31710046 Object 47 A 13.0 3-11466 a-b-c - - 58733 47 08:18 31710047 Object 47 B 13.5 3-11466 a-b-c - - 58733 48 08:27 31710048 Object 47 B 13.5 3-11466 a-b-c - - 58733 49 08:35 31710049 Object 47 A 13.0 3-11466 a-b-c - - 58733 50 08:44 31710050 DomeFlat 5 A 13.0 3-11466 a-b-c - 88 58733 51 08:48 31710051 Dark 10 A 13.0 3-11466 a-b-c - - 58733 52 08:49 31710052 Dark 10 A 13.0 3-11466 a-b-c - - 58733 53 09:04 31710053 Object 47 A 13.0 7-11213 a-b-c - - 58733 54 09:13 31710054 Object 47 B 13.5 7-11213 a-b-c - - 58733 55 09:21 31710055 Object 47 B 13.5 7-11213 a-b-c - - 58733 56 09:30 31710056 Object 47 A 13.0 7-11213 a-b-c - - 58733 57 09:38 31710057 Object 47 A 13.0 7-11213 a-b-c - - 58733 58 09:46 31710058 Object 47 B 13.5 7-11213 a-b-c - - 58733 59 09:55 31710059 Object 47 B 13.5 7-11213 a-b-c - - 58733 60 10:03 31710060 Object 47 A 13.0 7-11213 a-b-c - - 58733 61 10:12 31710061 DomeFlat 5 A 13.0 7-11213 a-b-c - 80 58733 62 10:14 31710062 Dark 10 A 13.0 7-11213 a-b-c - - 58733 63 10:16 31710063 Dark 10 A 13.0 7-11213 a-b-c - - 58733 64 10:20 31710064 DomeFlat 5 A 13.0 7-11213 a-b-c - 0 58733 65 10:27 31710065 Object 47 A 13.0 1-11602 a-b-c - - 58733 66 10:36 31710066 Object 47 B 13.5 1-11602 a-b-c - - 58733 67 10:44 31710067 Object 47 B 13.5 1-11602 a-b-c - - 58733 68 10:52 31710068 Object 47 A 13.0 1-11602 a-b-c - - 58733 69 11:01 31710069 DomeFlat 5 A 13.0 1-11602 a-b-c - 64 58733 70 11:22 31710070 Dark 60 A 13.0 ------- x-x-x - - 58733 71 11:25 31710071 Dark 60 A 13.0 ------- x-x-x - - -------------------------------------------------------------------------------- ---- Weather Log ---- --- Weather --- Time Inst Temp DP Dif Humid Wind Dir Dust,1um IRSC IRSCm -------------------------------------------------------------------------------- 04:03 9-11268A 15.2 9.3 5.9 67 14 113 347 3.3 332 nan apog 04:03 9-11268A 15.2 9.3 5.9 67 14 113 347 3.3 332 nan 04:13 9-11268A 14.9 9.9 5.0 71 14 112 486 2.6 331 nan apog 04:28 8-11463A 14.4 9.9 4.5 74 15 108 477 2.8 332 nan apog 04:37 8-11463A 14.6 10.0 4.6 73 18 109 487 2.4 332 nan apog 04:45 8-11463A 14.5 9.9 4.6 74 15 112 479 2.8 331 nan apog 04:53 8-11463A 14.3 9.9 4.4 75 16 112 468 3.6 322 nan apog 05:02 8-11463A 14.1 9.9 4.2 75 17 121 445 2.1 327 nan apog 05:10 8-11463A 14.2 9.9 4.3 75 17 115 437 2.8 332 nan apog 05:18 8-11463A 14.6 9.9 4.7 73 15 125 428 2.6 332 nan apog 05:27 8-11463A 14.4 9.9 4.5 74 13 116 432 2.3 332 nan apog 06:11 6-11421A 14.2 10.0 4.2 75 17 123 463 3.2 332 nan apog 06:19 6-11421A 14.2 9.9 4.3 75 14 124 365 4.1 332 nan apog 06:27 6-11421A 14.1 10.0 4.1 76 12 112 424 3.6 332 nan apog 06:36 6-11421A 14.0 9.9 4.1 76 17 126 371 3.0 318 nan apog 06:44 6-11421A 14.0 10.0 4.0 76 16 123 401 2.4 332 nan apog 06:53 6-11421A 13.9 10.0 3.9 77 18 124 420 2.6 332 nan apog 07:01 6-11421A 13.9 9.9 4.0 76 11 123 409 2.2 332 nan apog 07:09 6-11421A 13.8 9.9 3.9 77 17 127 408 2.4 332 nan apog 07:36 3-11466A 13.8 9.9 3.9 77 15 128 422 2.2 332 nan apog 07:44 3-11466A 13.9 9.8 4.1 76 8 111 344 2.0 332 nan apog 07:53 3-11466A 13.9 9.8 4.1 76 12 126 357 2.4 332 nan apog 08:01 3-11466A 13.9 9.8 4.1 76 15 132 359 2.6 332 nan apog 08:09 3-11466A 13.9 9.8 4.1 76 11 134 321 2.3 332 nan apog 08:18 3-11466A 13.9 9.8 4.1 76 14 132 359 2.1 332 nan apog 08:26 3-11466A 13.8 9.8 4.0 77 11 126 350 1.9 332 nan apog 08:35 3-11466A 13.7 9.8 3.9 77 8 119 344 2.9 331 nan apog 09:04 7-11213A 13.3 9.6 3.7 78 6 142 365 2.8 332 nan apog 09:12 7-11213A 13.4 9.6 3.8 77 7 114 332 2.9 332 nan apog 09:21 7-11213A 13.5 9.7 3.8 77 14 138 321 2.0 332 nan apog 09:29 7-11213A 13.5 9.8 3.7 78 15 134 361 2.9 332 nan apog 09:37 7-11213A 13.4 9.8 3.6 78 14 135 347 3.7 318 nan apog 09:46 7-11213A 13.5 9.9 3.6 78 14 135 322 3.0 332 nan apog 09:54 7-11213A 13.3 9.9 3.4 79 13 137 343 2.8 332 nan apog 10:03 7-11213A 13.2 9.9 3.3 80 10 134 320 2.7 332 nan apog 10:27 1-11602A 12.8 9.9 2.9 82 9 152 325 2.0 332 nan apog 10:35 1-11602A 12.8 9.9 2.9 82 6 166 303 3.4 332 nan apog 10:44 1-11602A 12.6 10.1 2.5 84 5 179 309 2.2 332 nan apog 10:52 1-11602A 12.4 10.3 2.1 87 3 134 264 1.9 332 nan apog ---- Problems in Detail ---- Same TCC/MCP error after every gotoGangChange command tonight: 04:16:36 .tcc 0 tcc w text='STATUS failed for axis: rot1' 04:16:36 .alerts 0 alerts w text="Error processing rules for tcc.tccPos: KeyVar('tcc', tccPos=(None, None, None)) is unknown" 04:16:42 .tcc 0 tcc w text='STATUS failed for axis: az' 04:16:42 .tcc 0 tcc w text='STATUS failed for axis: alt' 04:16:42 .tcc 0 tcc w text='STATUS failed for axis: rot1' 04:16:42 .alerts 0 alerts w text="Error processing rules for tcc.tccPos: KeyVar('tcc', tccPos=(None, None, None)) is unknown" 04:16:44 .tcc 0 tcc W Text="number keys does not match number of values" 04:16:45 .hub 0 cmds d CmdQueued=1300277,1567829805.64,"OBSERVER.elenam",140,"sop",168,"gotoGangChange nodomeFlat alt=30.0" 04:16:45 .hub 0 cmds d CmdQueued=1300278,1567829805.68,"OBSERVER.elenam.sop",884,"tcc",14562,"axis status" 04:16:48 .tcc 0 tcc w text='STATUS failed for axis: az' 04:16:48 .tcc 0 tcc w text='STATUS failed for axis: az' 04:16:48 .tcc 0 tcc w text='STATUS failed for axis: alt' 04:16:48 .tcc 0 tcc w text='STATUS failed for axis: alt' 04:16:48 .tcc 0 tcc w text='STATUS failed for axis: rot1' 04:16:48 .tcc 0 tcc w text='STATUS failed for axis: rot1' 04:16:48 OBSERVER.elenam.sop 884 tcc f text="Sub-command(s) failed: STATUS: Cancelled on queue by immediate priority command 'INIT'; STATUS: Cancelled on queue by immediate priority command 'INIT'; STATUS: Cancelled on queue by immediate priority command 'INIT'" 04:16:48 CmdDone: OBSERVER.elenam.sop 884 tcc axis status 04:16:48 .alerts 0 alerts w text="Error processing rules for tcc.tccPos: KeyVar('tcc', tccPos=(None, None, None)) is unknown" 04:16:48 OBSERVER.elenam 140 sop e text="Cannot check axis status. Something is very wrong!" 04:16:48 OBSERVER.elenam 140 sop e text="Is the TCC in a responsive state?" 04:16:48 OBSERVER.elenam 140 sop f text="Failed to slew to gang change" 04:16:48 CmdDone: OBSERVER.elenam 140 sop gotoGangChange nodomeFlat alt=30.0 04:16:48 .mcp 0 mcp ! command="init" 04:16:54 .tcc 0 tcc w text='STATUS failed for axis: alt' 04:16:54 .alerts 0 alerts w text="Error processing rules for tcc.tccPos: KeyVar('tcc', tccPos=(None, None, None)) is unknown" 04:17:00 .tcc 0 tcc w text='STATUS failed for axis: alt' 04:17:00 .tcc 0 tcc w text='STATUS failed for axis: rot1' 04:17:00 .alerts 0 alerts w text="Error processing rules for tcc.tccPos: KeyVar('tcc', tccPos=(None, None, None)) is unknown" 04:17:02 .hub 0 cmds d CmdQueued=1300282,1567829822.91,"OBSERVER.elenam",141,"sop",169,"gotoGangChange nodomeFlat alt=30.0" 04:17:02 .hub 0 cmds d CmdQueued=1300283,1567829822.94,"OBSERVER.elenam.sop",885,"tcc",14564,"axis status" 04:17:06 .alerts 0 alerts w text="Error processing rules for tcc.tccPos: KeyVar('tcc', tccPos=(None, None, None)) is unknown" 04:17:10 .boss 4064 boss w text="camcheck found missing or unresponsive cameras: set(['sp1cam'])"