Subject: 2.5m obslog 20211017 (SJD 59505) From: asimmons@apo.nmsu.edu Date: Mon, 18 Oct 2021 10:02:21 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday October 17, 2021 (SJD 59505) ---- Observing Team ---- Audrey Oravetz Viktor Malanushenko (R) ---- Observing Plan ---- BOSS camera tests ---- Observing Summary ---- We spent this evening checking the health of the BOSS cameras. The collimators were exercised and many calibrations were taken: biases, darks, and arcs and flats (without lights). There were no SP1mech disconnections, nor netburner errors. At 02:53Z we received a critical alert for R1 warming. Current deductions may point to an intermittent hardware failure of a mechanism that allows LN2 to flow to the red camera (not the blue). ===== Correlations from the previous occurrence: => Not a problem with empty 10L dewars: *Oct 8th: 20lbs of LN2 was transferred to from 180L to 10L dewar at 10:54 (local time). Alert was triggered at 15:20 (local time), ~6.5 hrs apart. *Tonight: 20lbs of LN2 was transferred to from 180L to 10L dewar at 10:00 (local time). Alert was triggered at 21:50 (local time), ~12 hrs apart. *Oct 8th: Likely a primary fill was spawned. The 10L pressure slight pressure drop (1psi) corresponded with the predicted fill time and there was no indication of a fill fault. However, somehow only nitrogen passed into the blue camera, not the red. *Tonight: The blue camera was never affected. Same as before. => IS a problem with LN2 not getting to the red CCDs: *Oct 8th: LN2Temp jumped from 83k - 145k within a few minutes and stayed at the top of the range until personnel issued a force fill. RED_ION_PUMP_LOGP also went from -6.07 to -5.70 during the warm up. *Tonight: LN2Temp jumped from 85k - 117k, RED_ION_PUMP_LOGP went from -6.41 to -6.35. => The occurrence is intermittent: *Oct 8th: Initial startFillSequences did not fix the problem. A manual force fill was needed. *Tonight: Only one initial startFillSequence was needed to fix the problem. ---- Observing Log ---- Afternoon: ========== Software check ok. All actors pinged, except MCP needed an IACK. STUI on the telescope latop needed to be restarted. 23:00Z - Started ventilating. sop script run eveningcals_nodither, FPI cals [observer@sdss4-hub ~]$ list_ap APOGEE data list, mjd=59505, all raw_data: /data/apogee/utr_cdr/59505/apRaw-*.fits archive: /data/apogee/archive/59505/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59505/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59505 1 23:26 39430001 Dark 60 A 13.0 ------- a-b-c - - 59505 2 23:28 39430002 QuarFlat 10 A 13.0 ------- a-b-c - 99 59505 3 23:30 39430003 Arc---- 12 A 13.0 ------- a-b-c - - 59505 4 23:32 39430004 Arc---- 12 A 13.0 ------- a-b-c - - 59505 5 23:39 39430005 Arc---- 40 A 13.0 ------- a-b-c - - 59505 6 23:47 39430006 Arc---- 40 A 13.0 ------- a-b-c - - 59505 7 23:58 39430007 Dark 60 A 13.0 ------- a-b-c - - 59505 8 00:07 39430008 Arc---- 30 A 13.0 ------- a-b-c - - FPI 59505 9 00:15 39430009 Arc---- 30 A 13.0 ------- x-x-x - - FPI -------------------------------------------------------------------------------- Took a BOSS bias and dark. Mirror numbers gathered. Night: ====== 00:17Z - After speaking with Kaike, the plan is now to take roughly 7 hours worth of BOSS cals to verify that we do not encounter any sp1mech disconnects anymore. It would be nice to see how many instances (if any) of netburner errors we run into as well. Stopped ventilating. Started the first loop of commands to make sure that they work and that the collimator motors return to their original values after one loop. Looks good. Started an additional 19 loops of the following commands: boss exposure arc itime=4 boss moveColl spec=sp1 a=1000 b=1000 c=1000 boss exposure flat itime=25 boss moveColl spec=sp1 a=-1000 b=-1000 c=-1000 boss exposure science itime=900 00:13Z - Exposure numbers start from 334583. At 02:53Z, we received a critical alert for the R1 camera warming up. The alert was acknowledged, and I immediately went down to the telescope to observe the subsequent forced primary fill with 'startFillSeq'. Below are some notes... Before the fill, an ln2stat returned: 02:59:05 .alerts 0 alerts e alert=camCheck.SP1R0CCDTempRead,critical,"at 2021-10-18 02:58:13 UT found 'Reported by camCheck'",enabled,ack,apo.aoravetz 02:59:08 CmdStarted: APO.telescope 3 boss sp1cam raw=ln2stat 02:59:08 APO.telescope 3 boss i text="Sending raw command LN2STAT to sp1cam" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'LN2_FILL ON'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'FILLTIME 120_MINUTES'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'NEXT_FILL 108_MINUTES'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b''" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'2NDARY_DEWAR_PRESS 10 # PSI GAUGE'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'NORM_RETRIG 3 '" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'FILL_MODE COLD'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'WARM_FILLTIME 15_MINUTES'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'WARM_FILLS 2 '" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'WARM_RETRIG 10 '" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'FILLFAULT NO'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'EMPTY_TRIGGER NO'" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'AUTOFILL_ON 1 2 '" 02:59:08 APO.telescope 3 boss i sp1camRawText="b'LN2_EMPTY '" 02:59:08 APO.telescope 3 boss i sp1camRawText="b' ok'" A primary fill should have automatically been triggered around ~02:47Z. Normal pressure drop for a primary fill should be around 1-3psi and there was a 1psi drop at 02:44Z. The 10L dewar pressure value never dropped below 9psi. Fill_Fault returned 'No', which means the fill must have occurred. There were no software disconnections. The last secondary fill was ~10 hours ago (at 10:00 local time), 20lbs were transferred. So, the 10L could be low... There was a steady elevated trend in the SP1R0LN2temp since time stamp of 02:19Z, starting from ~91K. Current value is 118K. The RED_ION_PUMP_LOGP value has also changed, from -6.41 to current -6.35. Both values indicate lack of nitrogren reaching the CCDs. 02:59:22 CmdDone: APO.telescope 4 boss sp1cam raw=startfillseq Visually watched the fill. Everything behaved normally. 03:01:21 .alerts 0 alerts w alert=camCheck.SP1R1HeaterVRead,warn,"at 2021-10-18 03:01:13 UT found 'Reported by camCheck'",enabled,ack,apo.aoravetz 03:01:21 .alerts 0 alerts e alert=camCheck.SP1R0CCDTempRead,critical,"at 2021-10-18 03:01:13 UT found 'Reported by camCheck'",enabled,ack,apo.telescope 03:01:22 CmdStarted: APO.telescope 7 boss sp1cam raw=ln2stat 03:01:22 APO.telescope 7 boss i text="Sending raw command LN2STAT to sp1cam" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'LN2_FILL ON'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'FILLTIME 120_MINUTES'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'NEXT_FILL 1_MINUTES'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b''" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'2NDARY_DEWAR_PRESS 10 # PSI GAUGE'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'NORM_RETRIG 3 '" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'FILL_MODE COLD'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'WARM_FILLTIME 15_MINUTES'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'WARM_FILLS 2 '" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'WARM_RETRIG 10 '" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'FILLFAULT NO'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'EMPTY_TRIGGER NO'" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'AUTOFILL_ON 1 2 '" 03:01:23 APO.telescope 7 boss i sp1camRawText="b'LN2_EMPTY '" 03:01:23 APO.telescope 7 boss i sp1camRawText="b' ok'" The temperature immediately stopped dropping and steadily began to cool back down. The lowest it reached was -120C. Only one startFillSeq was necessary. As a precaution, I forced a secondary fill a few minutes later to make sure that there was enough LN2 in the 10L dewar. 06:32Z - Last image, exposure 334642. No disconnections nor netburner errors. 09:04Z - Started morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0090 0.0080 Axial B: 0.0360 0.0330 Axial C: -0.1140 -0.1140 Trans D: -0.0365 -0.0395 Lateral E: 0.3315 0.3310 Lateral F: 0.5340 0.5335 GALILS Commanded: 12500 -1600 5350 47050 -8800 -8400 Actual: 12500 -1600 5350 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.3 -8.75 -512.73 1906.51 PrimOrient: 0.01 -59.26 -8.77 -512.68 1906.6 SECONDARY: ---------- Focus: 0.0000 Air temp: 13.50 Altitude: 30.000038956303893 MIGS TONIGHT NOMINAL Axial A: -9.4550 -9.4610 Axial B: -9.4660 -9.4760 Axial C: -9.5000 -9.5050 Trans D: 0.0920 0.0880 GALILS Commanded: 4630950 4624900 4648350 -50 -3450 Actual: 4607287 4607250 4607306 -50 -3450 SETMIR VALUES SecDesOrient: 3657.0 0.0 0.0 0.0 42.22 SecOrient: 3657.01 -0.01 -0.01 -0.01 42.69 ---- Telescope Status ---- Status at: 09:04Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 12 Counterweights at: 510 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 104 lbs, 21 psi ---- Software Used ---- Get Versions, 10/18/21, 00:00:33 -- STUI software: STUI : 1.7.13 alerts : 2.1.6 apo : 3.0.0 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : 0.1.1-alpha.0 hartmann : 2.0.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_146 -mcp.altFiduc : v1_146 -mcp.rotFiduc : v1_146 sop : 0.1.0-alpha.0 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.8 -- done -- -- Other versions ---- idlspec2d : plate-mapper3 : petunia : autoscheduler : sdss_python_module: /bin/sh: /home/manga/products/Linux64/mangadrp/trunk/bin/mangadrp_version: No such file or directory mangadrp : ---- Miscellaneous ---- [observer@sdss4-hub ~]$ list_ap APOGEE data list, mjd=59505, all raw_data: /data/apogee/utr_cdr/59505/apRaw-*.fits archive: /data/apogee/archive/59505/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59505/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59505 1 23:26 39430001 Dark 60 A 13.0 ------- a-b-c - - 59505 2 23:28 39430002 QuarFlat 10 A 13.0 ------- a-b-c - 99 59505 3 23:30 39430003 Arc---- 12 A 13.0 ------- a-b-c - - 59505 4 23:32 39430004 Arc---- 12 A 13.0 ------- a-b-c - - 59505 5 23:39 39430005 Arc---- 40 A 13.0 ------- a-b-c - - 59505 6 23:47 39430006 Arc---- 40 A 13.0 ------- a-b-c - - 59505 7 23:58 39430007 Dark 60 A 13.0 ------- a-b-c - - 59505 8 00:07 39430008 Arc---- 30 A 13.0 ------- a-b-c - - FPI 59505 9 00:15 39430009 Arc---- 30 A 13.0 ------- x-x-x - - FPI --------------------------------------------------------------------------------