Subject: 3.5m Night Log 20210831 From: cgray@apo.nmsu.edu Date: Wed, 1 Sep 2021 11:48:45 +0000 (UTC) Apache Point Observatory 3.5m Telescope Night Log Tuesday, August 31, 2021 PROG PI ASTRONOMER OBSERVER(S) -------------------------------------------------------------------- WY02 J.McLane J.McLane OU02 A.Kosakowski A.Kosakowski NM04 E.Nielsen E.Nielsen AP01 S.Merkowitz R.McMillan ACTUAL PROG INSTRUMENT START FINISH -------------------------------------------------------------------- WY02 TSPEC --:-- --:-- OU02 AGILE 23:26 23:42 NM04 ECHELLE --:-- --:-- AP01 APOLLO --:-- --:-- PROG SCHEDULED WEATH EQUIP OBS NOT USED I/ROW/DTN START FINISH D B D B D B D B ----------------------------------------------------------------- WY02/R 19:37 20:47 1.2 0.0 0 0 0 0 0 0 OU02/R 20:47 01:10 4.1 0 0 0 0.3 0.0 0 0 NM04/R 01:10 05:30 0.0 4.3 0 0 0 0 0 0 AP01/R 05:30 06:30 0.0 1.0 0 0 0 0 0 0 note: scheduled times listed include instrument change time ------------- ACTIVITY LOG -------------- Observing Specialist: C.Gray We had a FUN* check out today! When I went to run the checkout script this afternoon, none of the guiders would take exposures. They were throwing errors I had not seen before (see PR 2039) and it looked like they could not communicate with the hub. When Russet tried to connect to TUI, she could not connect. I called Shane and I was then thrown out of TUI and could not connect myself. I believe Shane said he rebooted the Hub3.5m earlier in the day and, given the problems, rebooted it again, after which we were able to connect. However, once we were able to log back into TUI, issues in addition to the guiders were: ARCTIC ICC was down, nicfips and APOLLO were not responding and while, TUI was working, the Enclosure window was not and telmech was not communicating (hub could not connect to the enclosure controller). 2021-09-01 00:46:24 MN01.telmech-mon 18 telmech f errtxt="failed to call Enclosure controller" We tried a hub startNubs telmech to no avail. We were able to get the guiders and all instruments except arctic back but not the enclosure. Logging into talk2boxes showed that we could not login to "tcc_tmech" or "tcc_encl" but we could log into the other controllers (e.g. rotators). All disks were mounted and there were no disk space issues. Bill had me power cycle the enclosure controller, Shane ran a code via talk2boxes (see PR 2030) and we were able to connect to tcc_telmech and tcc_encl again. After waiting several minutes, we ran a hub startNubs telmech again and everything came back. Going back to gcam, we were getting the error: - 2021-09-01 00:45:45 .gcamera.0 0 gcamera f text="gcamera locked out by process 10003" - Shane rebooted gcam via talk2boxes and 35m_gcamera. - gcam hub was restarted and, while the quad was dropped, arctic came back. - We did not attempt to recover the quad as we expect more problems throughout the week. But, none of this insanity actually lost us time on sky. That was the weather's fault. Thunderstorms in the afternoon and evening, radar around us, high humidity, and dust. At one point things did look good enough to open, we had amazing seeing....and then the dust spiked and we closed again, not to reopen. Only made it on sky 15 min :/ *FUN = freaking unrealistic nonsense ------- FAILURE LOG ------- FAILURE MODE TIME PROG INSTR TI/SHU START FINISH DESCRIPTION ---------------------------------------------------------------------- ---------------- FOCUS LOG --------------- Time Instrument Focus Az El Temp Strc Prim Sec Air filt FWHM -------------------------------------------------------------------------- 22:32 AGILE 440( -828) 37 65 15.2 15.2 15.4 15.2 14.9 BG40 0.76 ------------- WEATHER LOG ---------------- Wind Precip.or Time Temp F Dewp F MPH Direction Dust Sky Conditions ------------------------------------------------------------------ 16:00 57 51 7 110 (ESE) 2641 mstly cldy rain 17:00 57 48 16 66 (ENE) 1415 mstly cldy lt rain 18:00 60 49 7 88 (E) 815 ptly cldy none 19:00 61 51 8 165 (SSE) 2253 ptly cldy none 20:00 60 50 15 129 (SE) 2663 ptly cldy none 21:00 59 50 18 128 (SE) 2258 mstly cldy none 22:00 60 52 15 137 (SE) 2863 mstly cldy dust 23:00 60 51 13 152 (SSE) 4031 mstly clr dust 00:00 58 52 12 180 (S) 6260 mstly clr dust 01:00 57 53 16 152 (SSE) mstly cldy humid 02:00 57 53 16 180 (S) cloudy rain 03:00 54 54 10 200 (SSW) cloudy lt rain 04:00 54 53 10 183 (S) cloudy humid 05:00 55 53 10 189 (S) cloudy humid Note: the wind was coming from the azimuth listed. The convention used is north=0 degrees, east=90 degrees. The dust count is particles > 1u per 0.1 cubic feet. ---------- TELESCOPE SOFTWARE ---------- Current TCC version: 1.4.2 Current HUB version: v20171231_01 Current TUI version: 2.6.0