Subject: 3.5m Night Log 20200726 From: decolib@apo.nmsu.edu Date: Mon, 27 Jul 2020 10:38:52 +0000 (UTC) Apache Point Observatory 3.5m Telescope Night Log Sunday, July 26, 2020 PROG PI ASTRONOMER OBSERVER(S) -------------------------------------------------------------------- AP01 T.Murphy R.McMillan,T.Murphy CU08 J.Bally J.Bally OU02 J.DerKacy J.DerKacy ACTUAL PROG INSTRUMENT START FINISH -------------------------------------------------------------------- AP01 APOLLO 20:52 21:08 CU08 NICFPS 21:22 01:04 OU02 DIS 01:29 03:48 PROG SCHEDULED WEATH EQUIP OBS NOT USED I/ROW/DTN START FINISH D B D B D B D B ----------------------------------------------------------------- AP01/R 20:14 21:14 0 0.7 0 0 0.0 0.3 0 0 CU08/R 21:14 01:10 0 0 0 0 0.7 3.2 0 0 OU02/R 01:10 06:05 2.3 0 0.3 0 2.3 0.0 0 0 note: scheduled times listed include instrument change time ------------- ACTIVITY LOG -------------- Observing Specialist: D.DeColibus Last night was pretty uneventful. Tonight was VERY eventful, including but not limited to three hours of frantic, simultaneous multi-instrument troubleshooting: during checkout and A half, we had major issues with DIS and APOLLO, and minor issues with NICFPS and Dataq. Later on, we found that the TelEx fans weren't working. When checking central wavelengths/grating tilts on DIS during checkout, the red channel (lowres) wavelengths were significantly off. The blue lowres side was fine. Troubleshooting identified that the red grating motor was not moving when commanded. As the B-half observer required low-resolution for their science, and highres gratings are not needed for a few days, we opted to swap the lowres gratings into the grating set 1 position, where (we thought) the motors are functional. We left the highres gratings out of the instrument (stored in filter cabinet) to allow easier troubleshooting of the grating set 2 motors. During testing of the grating swap at switchover, we also figured out that the bay 1 blue motors are not working either! See PR #1954. While I was troubleshooting DIS and during their scheduled time, AP01 had significant issues with their equipment. Turning on the laser triggered software crashes twice, and once apparently turned off Houston, in addition to an unexplained software restart during the day. While we did open the dome once the weather looked safe, these problems precluded any observations. We got on sky with NICFPS, and CU08 did successfully acquire science data despite being plagued by multiple instances of the instrument recording bad (all zeros) images. Each occurrence was fixed with a nicfps expose init. It seemed to calm down a bit after the first hour or so. See PR #1955. gcam needed to be re-initialized twice during A half (hub startnubs gcam followed by gcam init). The dataq recording software froze again, and I had to log into hub35m to power cycle the dataq box. See PR #1956. We were slightly late in getting on sky for the B-half observer because we were double-checking DIS wavelength centers and grating zerosteps after having swapped the gratings in the turret. We did not have time to do this earlier due to obligations for the A-half observer, and this time is recorded as EQUIP due to being connected to an instrument failure. It took a while to focus, and around 01:45 we figured out why the focus value was so high and the mirror was still 2 degrees warmer than the outside air...the TelEx fans aren't working! See PR #1957. OU02 did get a couple hours of good science data despite fate's best efforts to stymie us, but we had to close around 03:50 due to high humidity and called it a night as the humidity was expected to stay above limits. Our hypothesis for the multiple hardware and software incidents is a possible lightning strike on the dome during the day, although the lack of recorded events on the surge detectors makes that hard to confirm. See separate email discussions for details. ------- FAILURE LOG ------- FAILURE MODE TIME PROG INSTR TI/SHU START FINISH DESCRIPTION ---------------------------------------------------------------------- OU02 DIS I/H 01:16 01:29 testing grating swap ---------------- FOCUS LOG --------------- Time Instrument Focus Az El Temp Strc Prim Sec Air filt FWHM -------------------------------------------------------------------------- 21:39 NICFPS 650( 1879) 31 47 14.4 16.0 17.1 15.6 14.8 Ks 0.85 22:53 NICFPS 650( 1812) -11 77 14.6 15.2 16.8 14.8 14.6 H2-2.12 0.61 01:42 dcamera 893( 1284) 264 65 13.1 13.3 15.4 13.1 12.9 -- 1.32 Wind Precip.or Time Temp F Dewp F MPH Direction Dust Sky Conditions ------------------------------------------------------------------ 16:00 65 37 8 177 (S) 512 mstly cldy none 17:00 64 35 3 131 (SE) 490 mstly cldy none 18:00 64 35 16 119 (ESE) 456 ptly cldy none 19:00 62 35 5 89 (E) 485 ptly cldy none 20:00 59 33 10 88 (E) 535 mstly clr none 21:00 59 33 8 110 (ESE) 540 mstly clr none 22:00 58 30 10 128 (SE) 567 ptly cldy none 23:00 58 28 5 114 (ESE) 569 ptly cldy none 00:00 57 30 11 112 (ESE) 638 ptly cldy none 01:00 56 32 9 114 (ESE) 826 mstly clr none 02:00 55 28 10 112 (ESE) 810 mstly clr none 03:00 53 28 9 107 (ESE) clear humid 04:00 54 29 8 156 (SSE) clear humid Note: the wind was coming from the azimuth listed. The convention used is north=0 degrees, east=90 degrees. The dust count is particles > 1u per 0.1 cubic feet. ---------- TELESCOPE SOFTWARE ---------- Current TCC version: 1.2.4 Current HUB version: v20171231_01 Current TUI version: 2.6.0