Subject: 3.5m Night Log 20210421 From: decolib@apo.nmsu.edu Date: Thu, 22 Apr 2021 12:04:31 +0000 (UTC) Apache Point Observatory 3.5m Telescope Night Log Wednesday, April 21, 2021 PROG PI ASTRONOMER OBSERVER(S) -------------------------------------------------------------------- AP01 S.Merkowitz R.McMillan GS02 S.Lepine N.Hejazi WY01 M.Brotherton J.McLane NM03 A.Hedlund P.Gaulme ACTUAL PROG INSTRUMENT START FINISH -------------------------------------------------------------------- AP01 APOLLO 19:20 20:44 GS02 ECHELLE 20:47 01:05 WY01 TSPEC 01:09 02:33 NM03 ECHELLE 02:37 05:37 PROG SCHEDULED WEATH EQUIP OBS NOT USED I/ROW/DTN START FINISH D B D B D B D B ----------------------------------------------------------------- AP01/R 19:47 20:47 0 0 0 0 0.0 1.0 0 0 GS02/R 20:47 01:10 0 0 0 0.5 0.0 3.9 0 0 WY01/R 01:10 02:40 0 0 0 0 0.0 1.5 0 0 NM03/R 02:40 06:17 0 0 0 0 2.3 1.3 0 0 note: scheduled times listed include instrument change time ------------- ACTIVITY LOG -------------- Observing Specialist: D.DeColibus The afternoon instrument checkout script gave a 'target named nicfps is not connected' error. Upon restarting the ICS, then trying to issue a >hub startNubs nicfps, the ICS threw another error. Upon Tracey's advice, I reset the NICFPS port on the terminal server on 35milts02 (the same steps as for having trouble with the filter wheel), then restarted the NICFPS ICS. It worked fine after that. We tried to start APOLLO early, but the seeing was pretty bad. It improved after sunset. ARCSAT was having consistent and frequent issues with freezing during plate-solve, pointing, and other scripts, as ACP seemed unwilling or unable to command the camera to take or read out exposures without hanging. After uncountable times 'nudging' it by taking exposures manually through Maxim and several full software restarts, I went for the nuclear option and rebooted the entire arcsat-user computer. It worked fine after that for the rest of the night, at least until it hung a few times when attempting to take dome flats in the morning. At around 23:30, we ran into a weird issue with the echelle that I haven't seen before. The user was taking a ThAr exposure, but the exposure failed for no apparent reason: 2021-04-22 05:33:43 GS02.Neda.echelleExpose 2239 echelle f text="exceptions.RuntimeError: No data read" 2021-04-22 05:33:43 GS02.Neda 50 echelleExpose f exposeTxt="object time=30.00 failed" I issued a startNubs echelle and startNubs echelleExpose, then tried again with this result: 2021-04-22 05:39:22 GS02.Neda.echelleExpose 2 echelle f text="an exposure is active." 2021-04-22 05:39:22 CmdDone: client-1084 2 echelle expose object time=30.00 diskname=/export/images/forTron/echelle/20210422T053922Z01.fits 2021-04-22 05:39:22 GS02.Neda 65 echelleExpose i echelleSeqState="GS02.Neda","object",30.0,1,1,"failed" 2021-04-22 05:39:22 GS02.Neda 65 echelleExpose i comment="None" 2021-04-22 05:39:22 GS02.Neda 65 echelleExpose i echelleNextPath="GS02.Neda","","EchelleThAr.","0027",".fits" 2021-04-22 05:39:22 GS02.Neda 65 echelleExpose f exposeTxt="object time=30.00 failed" I restarted the echelle ICC, then tried another ThAr. It started okay, but with about 20-30 seconds left in the readout, it failed with a different error message: 2021-04-22 05:47:07 GS02.Neda.echelleExpose 1 echelle f text="exceptions.RuntimeError: Failed to resize data" 2021-04-22 05:47:07 CmdDone: client-1085 1 echelle expose object time=30.00 diskname=/export/images/forTron/echelle/20210422T054429Z01.fits 2021-04-22 05:47:07 GS02.Neda 78 echelleExpose i echelleSeqState="GS02.Neda","object",30.0,1,1,"failed" 2021-04-22 05:47:07 GS02.Neda 78 echelleExpose i comment="None" 2021-04-22 05:47:07 GS02.Neda 78 echelleExpose i echelleNextPath="GS02.Neda","","EchelleThAr.","0028",".fits" 2021-04-22 05:47:07 GS02.Neda 78 echelleExpose f exposeTxt="object time=30.00 failed" I restarted the echelle ICC a second time, then tried a bias frame and finally another ThAr, which both worked fine. We lost about 30 minutes to troubleshooting, but got back on track at midnight. The rest of the night was uneventful. ------- FAILURE LOG ------- FAILURE MODE TIME PROG INSTR TI/SHU START FINISH DESCRIPTION ---------------------------------------------------------------------- GS02 E I/S 23:33 00:01 Exposure failures, restarted ICC twice ---------------- FOCUS LOG --------------- Time Instrument Focus Az El Temp Strc Prim Sec Air filt FWHM -------------------------------------------------------------------------- 19:24 APOLLO 500( 1563) 56 58 7.6 9.8 9.7 9.5 8.5 530 1.80 20:55 ecamera 428( -347) 87 48 7.8 7.8 8.3 7.7 7.6 Clear 1.17 01:15 tcamera 482( 930) -172 69 6.5 6.4 7.0 6.4 6.3 Ks 0.99 02:41 ecamera 501( -395) 80 60 4.6 5.4 6.0 5.4 5.1 Clear 1.40 ------------- WEATHER LOG ---------------- Wind Precip.or Time Temp F Dewp F MPH Direction Dust Sky Conditions ------------------------------------------------------------------ 16:00 53 19 11 255 (WSW) 1461 clear none 17:00 54 16 14 192 (SSW) 1507 clear none 18:00 51 14 14 214 (SW) 1570 clear none 19:00 47 14 4 190 (S) 1777 clear none 20:00 45 16 13 190 (S) 2027 clear none 21:00 46 17 15 204 (SSW) 2593 clear none 22:00 46 18 18 205 (SSW) 2591 clear none 23:00 45 20 19 200 (SSW) 2225 mstly clr none 00:00 45 21 15 190 (S) 1855 ptly cldy none 01:00 44 23 23 207 (SSW) 1581 mstly clr none 02:00 43 24 21 213 (SSW) 1805 mstly clr none 03:00 39 22 16 215 (SW) 1843 clear none 04:00 40 22 19 235 (SW) 1770 clear none 05:00 39 22 11 237 (WSW) 1604 clear none 06:00 38 21 14 260 (W) 1694 clear none Note: the wind was coming from the azimuth listed. The convention used is north=0 degrees, east=90 degrees. The dust count is particles > 1u per 0.1 cubic feet. ---------- TELESCOPE SOFTWARE ---------- Current TCC version: 1.4.2 Current HUB version: v20171231_01 Current TUI version: 2.6.0