Subject: 2.5m obslog 20250527 (SJD 60823) From: dmbiz@apo.nmsu.edu Date: Wed, 28 May 2025 10:36:41 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday May 27, 2025 (SJD 60823) ---- Observing Team ---- Dmitry Bizyaev Jose Sanchez-Gallego (re-compiling mcp and debugging) ---- Observing Plan ---- Tue 27 May 2025 60823 0.02 03:05 03:23 10:37 12:44 19:58 --- dark ( 7.2 h) plus 1.2 h for 8 deg design ---- Observing Summary ---- The night was lost primarily to the weather: a massive cloud front passed APO with dripping no any rain. At the same time, the problem with the MCP continues. Tonight we started long APOGEE calibrations by request of APOGEE team. They should be finished by upcoming late morning. Notes ===== - the apogeeql crashed 2 times during the long Internal Flats 12 and 14. Although apRaw files were created for them, archive files were not: ls /data/apogee/archive/60823/ -rw-rw-r-- 1 2005 3005 129M May 28 02:18 apR-c-52610010.apz -rw-rw-r-- 1 2005 3005 4.4G May 28 05:58 apR-c-52610011.fits -rw-rw-r-- 1 2005 3005 4.4G May 28 08:37 apR-c-52610013.fits ... Somebody needs to re-bundle flats 12 and 14, if archive files are required. - in the morning I noticed some pollen on car's windshield. Watch for pollen! ---- Observing Log ---- Afternoon ========= Checked what can be checked with the mcp problm continuing. APOGEE calibrations started. The command "jaeger fvc loop --no-apply" doesn't work as well, but it is connected to the MCP problem: the fvc code requires knowledge about the Rot position, but in the FITS headers it is now reported as NaN. ======================================================= =====>>>>> 01:14:13 Config Loaded: -999--999-20987 ======================================================= 01:19:51 apogee exposure file= apRaw-52610001 3 B ArcLamp 01:22:38 apogee exposure file= apRaw-52610002 60 B Dark 01:33:38 apogee exposure file= apRaw-52610003 10 A QuartzFlat 01:35:32 apogee exposure file= apRaw-52610004 12 B ArcLamp 01:37:45 apogee exposure file= apRaw-52610005 40 B ArcLamp 01:45:02 apogee exposure file= apRaw-52610006 12 A ArcLamp 01:47:14 apogee exposure file= apRaw-52610007 40 A ArcLamp 01:54:40 apogee exposure file= apRaw-52610008 60 A Dark 02:05:39 apogee exposure file= apRaw-52610009 30 A ArcLamp 02:11:02 apogee exposure file= apRaw-52610010 30 B ArcLamp We cannot open because of mcp problem, but the sky is very cloudy as well and we have nearby radars. This lost time should be claimed to the weather. By request from Andrew Saydjari, long APOGEE calibrations started: Nonlinearity Ramps: - 5x 450 reads INTERNALFLAT - 20x 60 reads DARK - 10x 30 reads INTERNALFLAT Yes, 450 reads confirmed. 06:06:47 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2025-05-28 06:04:58; last seen 708 sec ago",enabled,ack,apo.dmbiz 06:06:47 .alerts 0 alerts e alert=mcp.heartbeat,critical,"at 2025-05-28 05:18:48; last seen 2998 sec ago",disabled,noack,0 The process was fould dead on sdss5-mwm machine. Restarted via systemctl. All files apRaw-52610012-xxx.fits that could be affected are found in the /data-ics/ directory. Calling it a night. Morning APOGEE calibrations are not started because of on-going long calibrations. ---- Telescope Status ---- Never opened. MCP doesn't work; repair setup in the lower enclosure, at the MCP rack. Status at: 10:22Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 651 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 214 lbs, 23 psi FPS Status ========== CP10 Loop = 7.48 L/min GFA Loop = 2.21 L/min Supply pressure = 43.4 psi Return pressure = 29.8 psi Chiller Status ============== Set temperarature = 15.0 C Flow = 4.9 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.9.2a0 kaiju : 1.4.0 coordio : 1.13.0 fps_calibrati : 2025.02.24 cherno : 1.1.0 astrometrynet : 0.93 hal : 1.4.1 hartmann : 2.0.2 hub : 5.0.1a0 mcp : FAILED -mcp.plc : FAILED -mcp.azFiduci : FAILED -mcp.altFiduc : FAILED -mcp.rotFiduc : FAILED tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.4.9 --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : unavailable idlspec2d : v6_2_1 Kronos : 2.0.0 roboscheduler : 2.1.0 /dev/sdb1 2.0T 462G 1.6T 23% /home /dev/sdd1 22T 13T 8.3T 61% /data ---- APOGEE Data Summary ---- Last morning's calibrations look OK: 60822 11 09:29 52600011 Dark 60 B 13.5 ---------- a-b-c - - 60822 12 09:40 52600012 Dark 60 B 13.5 ---------- a-b-c - - 60822 13 09:50 52600013 Dark 60 B 13.5 ---------- a-b-c - - 60822 14 09:52 52600014 QuarFlat 10 B 13.5 ---------- a-b-c - - 60822 15 09:54 52600015 QuarFlat 10 B 13.5 ---------- a-b-c - - 60822 16 09:56 52600016 QuarFlat 10 B 13.5 ---------- a-b-c - - 60822 17 09:58 52600017 Arc-Thar 12 B 13.5 ---------- a-b-c 3.02 - 60822 18 10:05 52600018 Arc-Une 40 B 13.5 ---------- a-b-c - - 60822 19 10:07 52600019 Arc-Thar 12 A 13.0 ---------- a-b-c 2.45 - 60822 20 10:15 52600020 Arc-Une 40 A 13.0 ---------- a-b-c - - 60822 21 10:20 52600021 Dark 30 A 13.0 ---------- a-b-c - - 60822 22 10:25 52600022 InteFlat 30 A 13.0 ---------- a-b-c - - 60822 23 10:31 52600023 InteFlat 30 A 13.0 ---------- a-b-c - - 60822 24 10:36 52600024 InteFlat 30 A 13.0 ---------- a-b-c - - 60822 25 10:42 52600025 Dark 30 A 13.0 ---------- a-b-c - - 60822 26 10:47 52600026 Arc---- 30 A 13.0 ---------- a-b-c - 111 FPI A 60822 27 10:53 52600027 Arc---- 30 B 13.5 ---------- a-b-c - 110 FPI B APOGEE data list, mjd=60823, all raw_data: /data/apogee/utr_cdr/60823/apRaw-*.fits archive: /data/apogee/archive/60823/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60823/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60823 1 01:20 52610001 Arc---- 3 B 13.5 ---------- a-b-c - 0 test 60823 2 01:33 52610002 Dark 60 B 13.5 ---------- a-b-c - - 60823 3 01:35 52610003 QuarFlat 10 A 13.0 ---------- a-b-c - - 60823 4 01:37 52610004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.01 - 60823 5 01:44 52610005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60823 6 01:46 52610006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.46 - 60823 7 01:53 52610007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60823 8 02:05 52610008 Dark 60 A 13.0 ---------- a-b-c - - 60823 9 02:10 52610009 Arc---- 30 A 13.0 ---------- a-b-c - 110 FPI A 60823 10 02:15 52610010 Arc---- 30 B 13.5 ---------- a-b-c - 111 FPI B ... ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60823 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 10:14 b1-00385929 ----------- ---- bia Clo 0.0 5933 6073 6055 2 10:14 r1-00385929 ----------- ---- bia Clo 0.0 5933 6073 6055 3 10:16 r1-00385930 ----------- ---- dar Clo 20.0 5933 6073 6055 4 10:16 b1-00385930 ----------- ---- dar Clo 20.0 5933 6073 6055 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 18:00 16.7 1.4 9.0 93.0 3068 18:15 19.3 2.1 9.4 56.9 2927 18:30 17.6 1.3 15.2 58.0 2738 18:45 18.6 3.2 12.8 31.5 2609 19:00 19.0 3.4 13.0 02.2 2606 19:15 18.3 3.1 18.7 41.4 2781 19:30 18.1 2.6 11.4 60.3 2694 19:45 17.2 2.3 14.6 62.8 2689 20:00 18.5 1.8 15.7 06.5 2588 20:15 20.5 1.6 13.7 25.8 2434 20:30 21.4 1.3 15.8 63.5 2485 20:45 20.4 1.1 14.6 11.7 2369 21:00 18.9 0.2 10.4 60.0 2331 21:15 17.5 -2.6 7.2 68.2 2282 21:30 23.1 1.0 20.7 33.2 2209 21:45 20.5 -0.6 14.6 50.8 2239 22:00 19.3 -0.9 12.2 40.6 2277 22:15 18.0 -0.8 8.3 01.9 2188 22:30 17.4 -3.3 14.5 03.4 2144 22:45 16.0 -1.8 6.8 61.8 2228 23:00 16.2 -2.3 7.0 13.8 2211 23:15 15.6 -1.4 10.4 10.5 2325 23:30 14.0 -4.3 19.7 58.2 3415 23:45 12.9 -2.2 19.9 16.9 3068 00:00 12.2 -4.1 4.3 58.6 2276 00:15 12.9 -5.0 1.3 44.2 2451 00:30 13.1 -4.6 13.4 42.1 2481 00:45 12.1 -3.8 16.9 18.8 2720 01:00 14.4 -7.5 32.9 32.2 5314 01:15 14.8 -8.2 6.4 00.1 2627 01:30 14.4 -8.2 1.5 08.3 2440 01:45 14.7 -6.4 6.6 60.8 2669 02:00 14.7 -5.5 6.4 66.5 3177 02:15 14.6 -5.5 7.9 64.7 3037 02:30 14.6 -4.9 6.3 58.9 3026 02:45 14.6 -4.9 7.6 60.6 3007 03:00 14.6 -5.4 7.8 50.0 3428 03:15 14.6 -5.7 7.8 60.8 3377 03:30 14.1 -5.1 7.6 61.8 3759 03:45 12.9 -5.6 8.7 44.8 3398 04:00 13.3 -5.7 8.1 24.3 3398 04:15 13.4 -5.7 6.6 02.0 3436 04:30 13.6 -5.6 6.1 08.7 3262 04:45 13.8 -5.7 2.0 10.3 3332 05:00 13.7 -5.4 4.2 99.4 3511 05:15 14.2 -6.1 6.4 41.0 4065 05:30 14.0 -5.6 3.8 16.8 3947 05:45 13.9 -4.5 1.4 26.6 3506 06:00 13.9 -4.4 3.3 97.7 3537 06:15 10.9 -5.4 13.1 69.5 3359 06:30 10.9 -4.2 11.4 83.0 2605 06:45 10.7 -4.3 12.0 81.7 2461 07:00 10.9 -4.2 9.0 81.0 2246 07:15 11.4 -4.3 5.5 84.0 2396 07:30 11.3 -3.9 6.8 89.7 2487 07:45 11.9 -3.9 5.2 96.8 2469 08:00 11.6 -4.2 3.8 99.5 2389 08:15 11.4 -4.1 8.4 89.3 2411 08:30 11.8 -3.8 6.1 34.9 2551 08:45 10.9 -4.1 3.6 29.0 2381 09:00 11.3 -4.0 8.2 36.2 2585 09:15 11.1 -4.2 7.0 34.5 2410 09:30 11.1 -4.3 8.1 73.7 3349 09:45 11.2 -4.2 8.2 76.3 6894 10:00 11.2 -4.0 0.9 99.7 8707 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60823 Fill night time in Hours below *=*=*= Total bright time : 1.2 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 1.2 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 7.2 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 7.2 *=*=*= Dark time lost to technical issues : 0