Subject: 2.5m obslog 20210122 (SJD 59237) From: kinemuchi@apo.nmsu.edu Date: Sat, 23 Jan 2021 11:38:25 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday January 22, 2021 (SJD 59237) ---- Observing Team ---- Audrey Oravetz Karen Kinemuchi Dylan Gatlin (online testing scripts) ---- Observing Plan ---- Fri 22 JAN 2021 59237.0 0.72 01:25 12:53 02:32 14:02 ---- KK MWM until 09:43 ( 8.3 h), then BHM ( 3.2 h) ---- Observing Summary ---- Entire night lost to weather. MWM lost 8.3 hours and BHM lost 3.2 hours. MTP test: The number of faint fibers increased the most at the very beginning of the test when cycling the connector in and out of the cartridge, but was more-or-less stable after that. There was a very slight variation in faint fibers during the last part of the test, which can point to a slight variation in the intensity of the FF lamps. Our guess is that the signal is being diminished the most by the male connector MTPs not mating consistently with the female end within the cartridge. Notes: ====== The transD MIG measurements for M1/M2 are slightly out of tolerance. A home/relax may be needed soon during an engineering night. * Dylan's test script to move the telescope to 5 caused the TCC to disconnect. See log for details and email traffic from the early part of the night. Resending the log at 11:38Z ---- Observing Log ---- Afternoon: ========== Checked actors, disk space, gathered guider dark, mirror numbers, BHM cals, and MWM cals. The TRANSD mirror numbers are slightly out of tolerance. A home and relax may be needed soon during engineering time. /dev/vdd 688G 608G 81G 89% /home /dev/vdc1 11T 9.2T 1.7T 85% /data The blue camera may need adjusting, but because we have not been open the past few nights, it is hard to tell where b1 needs to be (temperature-wise). Ran sop collimateBoss with cart 7. motorPosition sp1: 3173, 3171, 3218; ---------- Median spectrographs temperatures: Tsp1= 5.3 Outside air temperature= 3.0; 22:35:34 boss exposure 4.0, file=323855 22:36:44 boss exposure 4.0, file=323856 22:38:05 sp1.motor.move= 24, 24, 17 22:38:05 Hartmann collimate output on cart #3 sp1: offset: r= -659 (In Focus); b= -2.2 (In Focus) sp1: pred. move: spAverageMove= 24 sp1: pred. resid: r= -683, b= -2.1, txt= OK, spTemp = 5.3 22:39:21 boss exposure 25.0, file=323857 22:42:08 boss exposure 4.0, file=323858 APOGEE morning cals from last night and evening cals today were visually checked. Everything is ok. Telescope check is done. There was another instance of the azimuth not wanting to move off of 121 (instrumentChange) position. Cycling the amplifiers, and the locking pin did not help. However, cycling the wind baffle switch worked. Unfortunately I was not able to look at the error message inside of the box. Next time. Mounted cart 2 for the beginning of the night. Delayed ventilating because of low clouds and the dew point difference would jump from 2.6 to 3.8. Very unstable. Night: ====== 00:22Z Dylan is online to test some scripts. Something has triggered the tcc.heartbeat critical alert. Critical tcc.heartbeat at 2021-01-23 00:22:58; last seen 2 sec ago Tried to restart the alerts and TCC actors. We've had to restart our STUI sessions to clear out the alerts window of the warnings. As far as we can tell, Dylan's test script somehow disconnected the TCC. Digging into system logs, it looks like Dylan's script called for 80 "tcc axis status" calls within 1.5 minutes, which overwhelmed the TCC. We don't know what Dylan's script was doing, but here is what he said he was expecting: Even after the alerts actor stopped receiving pings, my command reliably returned results in 0.2s the entire time, so I didn't think anything was going wrong. That was actually the third time I ran the script. The previous two times I stopped it after 10 and 33 pings, neither of which raised an alert, and it always returned within 0.2s. That's why I said 130-160s. tcc must have stopped responding to alerts after around 24 pings, and then it took 120s to timeout. It failed after the 121st cycle, and ~1.2s per cycle 02:48Z Skies are clear, but humidity is variable. Analysis for the MTP test: 59237 8 03:50 36750008 Dark 3 A 13.0 2-15217 a-b-c - - ================= Cycling the gang connector in and out of the cartridge: 59237 9 03:56 36750009 DomeFlat 5 A 13.0 2-15217 a-b-c - 33 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 162, '176 - 177', 179] ----------------- 59237 10 03:58 36750010 DomeFlat 5 A 13.0 2-15217 a-b-c - 30 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '160 - 162', 171, '175 - 177', 179, '212 - 240'] ----------------- 59237 11 04:00 36750011 DomeFlat 5 A 13.0 2-15217 a-b-c - 31 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 156, 158, 162, 166, 171, 173, '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 12 04:01 36750012 DomeFlat 5 A 13.0 2-15217 a-b-c - 31 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 154, 156, 158, 160, 162, '166 - 169', 171, '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 13 04:03 36750013 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 154, 156, 158, 160, 162, '166 - 169', 171, '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 14 04:04 36750014 DomeFlat 5 A 13.0 2-15217 a-b-c - 30 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 154, 156, 158, 160, '162 - 163', '166 - 171', '175 - 177', '179 - 180', '212 - 240'] ----------------- ================== Cycling the gang connector in between the cartridge and the podium, waiting 5 minutes and taking darks in between: 59237 15 04:06 36750015 Dark 10 A 13.0 2-15217 a-b-c - - 59237 16 04:08 36750016 Dark 10 A 13.0 2-15217 a-b-c - - 59237 17 04:12 36750017 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 153, 156, 158, 160, '162 - 163', '166 - 171', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 18 04:14 36750018 Dark 10 A 13.0 2-15217 a-b-c - - 59237 19 04:15 36750019 Dark 10 A 13.0 2-15217 a-b-c - - 59237 20 04:19 36750020 DomeFlat 5 A 13.0 2-15217 a-b-c - 30 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', 154, '160 - 163', '166 - 167', '170 - 171', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 21 04:21 36750021 Dark 10 A 13.0 2-15217 a-b-c - - 59237 22 04:23 36750022 Dark 10 A 13.0 2-15217 a-b-c - - 59237 23 04:27 36750023 DomeFlat 5 A 13.0 ------- a-b-c - 29 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '153 - 154', 156, '158 - 163', '166 - 173', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 24 04:29 36750024 Dark 10 A 13.0 2-15217 a-b-c - - 59237 25 04:31 36750025 Dark 10 A 13.0 2-15217 a-b-c - - 59237 26 04:34 36750026 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '153 - 154', 156, '158 - 160', '162 - 163', '166 - 173', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 27 04:35 36750027 Dark 10 A 13.0 2-15217 a-b-c - - 59237 28 04:37 36750028 Dark 10 A 13.0 2-15217 a-b-c - - 59237 29 04:40 36750029 DomeFlat 5 A 13.0 2-15217 a-b-c - 27 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '153 - 154', 156, '158 - 163', '166 - 173', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 30 04:42 36750030 Dark 10 A 13.0 2-15217 a-b-c - - 59237 31 04:43 36750031 Dark 10 A 13.0 2-15217 a-b-c - - 59237 32 04:47 36750032 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 154', 156, '158 - 163', '166 - 173', '175 - 180', '212 - 240'] ----------------- ========= Taking back-to-back dome flats without touching the connector: 59237 33 04:48 36750033 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 151', '153 - 154', 156, '158 - 163', '166 - 173', '175 - 180', '212 - 240'] ----------------- 59237 34 04:50 36750034 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '152 - 154', 156, '158 - 163', '166 - 173', '175 - 180', '212 - 240'] ----------------- 59237 35 04:51 36750035 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '152 - 154', 156, '158 - 164', '166 - 173', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 36 04:52 36750036 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 154', '156 - 164', '166 - 173', '175 - 180', '212 - 240'] ----------------- 59237 37 04:54 36750037 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '152 - 154', 156, '158 - 164', '166 - 173', '175 - 177', '179 - 180', '212 - 240'] ----------------- 59237 38 04:55 36750038 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 Missing fibers: [38, 57, 59, 137, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 136', '138 - 150', '152 - 154', 156, '158 - 163', '166 - 173', '175 - 180', '212 - 240'] ----------------- Conclusions: The number of faint fibers increased the most at the very beginning of the test when cycling the connector in and out of the cartridge, but was more-or-less stable after that. There was a very slight variation in faint fibers during the last part of the test, which can point to a slight variation in the intensity of the FF lamps. Our guess is that the signal is being diminished the most by the male connector MTP's not mating as well with the female end within the cartridge. 11:26Z Giving up. 11:28Z Started morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: -4.9290 -4.9310 Axial B: -4.6640 -4.6660 Axial C: -5.7740 -5.7830 Trans D: -4.3790 -4.4340 Lateral E: 0.3750 0.3745 Lateral F: 0.5720 0.5715 GALILS Commanded: 5850 -4500 14850 28200 25200 24900 Actual: 5850 -4500 14850 28200 25200 24900 SETMIR VALUES PrimDesOrient: 0.0 69.89 100.0 375.3 622.11 PrimOrient: 0.04 69.89 100.0 375.3 622.11 SECONDARY: ---------- Focus: 0.0000 Air temp: 2.72 Altitude: 30.000143956518016 MIGS TONIGHT NOMINAL Axial A: -9.4420 -9.4550 Axial B: -9.4060 -9.4200 Axial C: -9.4960 -9.4970 Trans D: 0.0430 0.0380 GALILS Commanded: 4622750 4606450 4675250 -2000 -6600 Actual: 4604050 4585637 4632314 -2000 -6600 SETMIR VALUES SecDesOrient: 3657.0 -14.01 -2.9 12.5 114.22 SecOrient: 3657.0 -14.02 -2.94 12.66 114.21 ---- Telescope Status ---- Never opened. No interlocks bypassed Status at: 11:29Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 2 Counterweights at: LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 170 lbs, 23.4 psi ---- Software Used ---- Get Versions, 01/22/21, 22:50:21 -- STUI software: STUI : 1.7.12 alerts : 2.0.1 apo : v1_11 boss : 5.0.1 -boss.specMec : 3.0.4 gcamera : v2_2 guider : 3.9.4 hartmann : 1.8.0 hub : v4.2.4 mcp : 1.12-12-g222b -mcp.plc : 34 -mcp.azFiduci : v1_144 -mcp.altFiduc : v1_144 -mcp.rotFiduc : v1_144 platedb : 2.8.0 sop : 4.0.8 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.4 -- done -- -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v7_0_8 petunia : 2.1.3 autoscheduler : 1.5.4 sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- APOGEE data list, mjd=59237, all raw_data: /data/apogee/utr_cdr/59237/apRaw-*.fits archive: /data/apogee/archive/59237/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59237/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59237 1 22:28 36750001 Dark 60 A 13.0 3-15112 a-b-c - - 59237 2 22:39 36750002 QuarFlat 10 A 13.0 3-15112 a-b-c - 51 59237 3 22:41 36750003 Arc-Thar 12 A 13.0 3-15112 a-b-c -1.53 - 59237 4 22:43 36750004 Arc-Thar 12 A 13.0 3-15112 a-b-c -1.53 - 59237 5 22:45 36750005 Arc-Une 40 A 13.0 3-15112 a-b-c - - 59237 6 22:52 36750006 Arc-Une 40 A 13.0 3-15112 a-b-c - - 59237 7 23:00 36750007 Dark 60 A 13.0 3-15112 a-b-c - - 59237 8 03:50 36750008 Dark 3 A 13.0 2-15217 a-b-c - - 59237 9 03:56 36750009 DomeFlat 5 A 13.0 2-15217 a-b-c - 33 59237 10 03:58 36750010 DomeFlat 5 A 13.0 2-15217 a-b-c - 30 59237 11 04:00 36750011 DomeFlat 5 A 13.0 2-15217 a-b-c - 31 59237 12 04:01 36750012 DomeFlat 5 A 13.0 2-15217 a-b-c - 31 59237 13 04:03 36750013 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 59237 14 04:04 36750014 DomeFlat 5 A 13.0 2-15217 a-b-c - 30 59237 15 04:06 36750015 Dark 10 A 13.0 2-15217 a-b-c - - 59237 16 04:08 36750016 Dark 10 A 13.0 2-15217 a-b-c - - 59237 17 04:12 36750017 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 59237 18 04:14 36750018 Dark 10 A 13.0 2-15217 a-b-c - - 59237 19 04:15 36750019 Dark 10 A 13.0 2-15217 a-b-c - - 59237 20 04:19 36750020 DomeFlat 5 A 13.0 2-15217 a-b-c - 30 59237 21 04:21 36750021 Dark 10 A 13.0 2-15217 a-b-c - - 59237 22 04:23 36750022 Dark 10 A 13.0 2-15217 a-b-c - - 59237 23 04:26 36750023 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 59237 24 04:29 36750024 Dark 10 A 13.0 2-15217 a-b-c - - 59237 25 04:31 36750025 Dark 10 A 13.0 2-15217 a-b-c - - 59237 26 04:34 36750026 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 59237 27 04:35 36750027 Dark 10 A 13.0 2-15217 a-b-c - - 59237 28 04:37 36750028 Dark 10 A 13.0 2-15217 a-b-c - - 59237 29 04:40 36750029 DomeFlat 5 A 13.0 2-15217 a-b-c - 27 59237 30 04:42 36750030 Dark 10 A 13.0 2-15217 a-b-c - - 59237 31 04:43 36750031 Dark 10 A 13.0 2-15217 a-b-c - - 59237 32 04:47 36750032 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 59237 33 04:48 36750033 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 59237 34 04:50 36750034 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 59237 35 04:51 36750035 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 59237 36 04:52 36750036 DomeFlat 5 A 13.0 2-15217 a-b-c - 29 59237 37 04:54 36750037 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 59237 38 04:55 36750038 DomeFlat 5 A 13.0 2-15217 a-b-c - 28 59237 39 04:57 36750039 Dark 10 A 13.0 2-15217 a-b-c - - 59237 40 04:59 36750040 Dark 10 A 13.0 2-15217 a-b-c - - -------------------------------------------------------------------------------- ---- Miscellaneous ---- [observer@sdss4-hub 59234]$ list_ap -m 59236 -morning APOGEE data list, mjd=59236, morning raw_data: /data/apogee/utr_cdr/59236/apRaw-*.fits archive: /data/apogee/archive/59236/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59236/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59236 1 11:25 36740009 Dark 60 A 13.0 3-15112 a-b-c - - 59236 2 11:36 36740010 Dark 60 A 13.0 3-15112 a-b-c - - 59236 3 11:47 36740011 Dark 60 A 13.0 3-15112 a-b-c - - 59236 4 11:57 36740012 QuarFlat 10 A 13.0 3-15112 a-b-c - 50 59236 5 11:59 36740013 QuarFlat 10 A 13.0 3-15112 a-b-c - 51 59236 6 12:01 36740014 QuarFlat 10 A 13.0 3-15112 a-b-c - 50 59236 7 12:03 36740015 Arc-Thar 12 A 13.0 3-15112 a-b-c -1.49 - 59236 8 12:05 36740016 Arc-Thar 12 A 13.0 3-15112 a-b-c -1.50 - 59236 9 12:07 36740017 Arc-Une 40 A 13.0 3-15112 a-b-c - - 59236 10 12:14 36740018 Arc-Une 40 A 13.0 3-15112 a-b-c - - 59236 11 12:22 36740019 Dark 30 A 13.0 3-15112 a-b-c - - 59236 12 12:27 36740020 InteFlat 30 A 13.0 3-15112 a-b-c - 0 59236 13 12:32 36740021 InteFlat 30 A 13.0 3-15112 a-b-c - 0 59236 14 12:38 36740022 InteFlat 30 A 13.0 3-15112 a-b-c - 0 59236 15 12:43 36740023 Dark 30 A 13.0 3-15112 a-b-c - -