Subject: 2.5m obslog 20231126 (SJD 60275) From: asimmons@apo.nmsu.edu Date: Mon, 27 Nov 2023 12:03:52 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday November 26, 2023 (SJD 60275) ---- Observing Team ---- Audrey Oravetz Dan Oravetz (support) John Wilson (support) Shane Thomas (support) ---- Observing Plan ---- Sun 26 Nov 2023 60275 1.00 00:54 01:09 12:32 22:29 09:53 --- bright (11.6 h, w/ 0.2 h for 12 deg design) ---- Observing Summary ---- The night was lost to weather (high humidity and clouds), which is fortunate in a way because we also had possible network connection issues that caused us lose connection with the APOGEE-ICS machine. The critical alert was triggered around 00:40Z and at the time of this nightlog, we are still unable to connect with APOGEE. The apogee, apogeeql, and apogeebpr actors have been silenced in the STUI alerts until site personnel can look at the physical connections tomorrow. Morning cals were not taken tonight. ---- Observing Log ---- Afternoon: ========== BOSS: 21:56:25 boss exposure 0.0, file=361813 21:57:52 boss exposure 20.0, file=361814 22:00:02 boss exposure 4.0, file=361815 22:01:34 boss exposure 4.0, file=361816 22:02:15 boss exposure 4.0, file=361817 22:02:57 sp1.motor.move= -948, -950, -952 22:02:57 Hartmann collimate output sp1: offset: r= -1319 (Out of focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -948 sp1: pred. resid: r= -370, b= -1.2, txt= OK, spTemp = 3.1 22:06:31 boss exposure 25.0, file=361818 22:05:41 APO.aoravetz 42 boss e text="sp1daq readout failed: timed out (>)." boss reloadControllers APOGEE: (observers39) [observer@sdss5-hub: ~]$ list_ap APOGEE data list, mjd=60275, all raw_data: /data/apogee/utr_cdr/60275/apRaw-*.fits archive: /data/apogee/archive/60275/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60275/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60275 1 22:06 47130001 Dark 60 A 13.0 ---------- a-b-c - - 60275 2 22:08 47130002 QuarFlat 10 A 13.0 ---------- a-b-c - - 60275 3 22:10 47130003 Arc-Thar 12 B 13.5 ---------- a-b-c 1.47 - 60275 4 22:17 47130004 Arc-Une 40 B 13.5 ---------- a-b-c - - 60275 5 22:19 47130005 Arc-Thar 12 A 13.0 ---------- a-b-c 1.01 - 60275 6 22:27 47130006 Arc-Une 40 A 13.0 ---------- a-b-c - - 60275 7 22:38 47130007 Dark 60 A 13.0 ---------- a-b-c - - 60275 8 22:43 47130008 Arc---- 30 A 13.0 ---------- a-b-c - 82 FPI ------------------------------------------------------------------------------ FPS: 22:00:33 Config Loaded: -999-11728 22:07:40 jaeger.jaeger 328 fliswarm i filename_bundle=/data/fcam/60275/fimg-fvc1n-0001.fits 22:07:40 jaeger.jaeger 328 fliswarm : 22:07:40 CmdDone: jaeger.jaeger 328 fliswarm talk -c fvc expose 5.0 22:09:29 APO.aoravetz 50 fliswarm i filename_bundle=/data/fcam/60275/fimg-fvc1n-0002.fits 22:09:29 APO.aoravetz 50 fliswarm : 22:09:29 CmdDone: APO.aoravetz 50 fliswarm talk -n fvc expose 50 fliswarm talk -c gfa expose --bias --count 10 fliswarm talk -c gfa expose --dark --count 5 5 Did not ventilate because the humidity was high. Night: ====== 00:40:19 .alerts 0 alerts e alert=apogee.heartbeat,critical,"at 2023-11-27 00:40:19; last seen 599 sec ago",enabled,noack,-1 (3.10.9) [sdss5@sdss5-hub ~]$ ssh observer@apogee-ics ssh: connect to host apogee-ics port 22: No route to host Troubleshooting apogee's heartbeat critical alert... Upon first glance, the APOGEE instrument/network rack in the APOGEE room did not point to anything out of the ordinary (the APOGEE-ICS machine was turned on and had flashing lights). I restarted it anyway. I also found the ibootbar in the back of that rack and rebooted it as well (going through the same procedure that Dmitry did back on the night of Oct 8th thinking that the issues are related). I am still receiving the message though: (3.10.9) [sdss5@sdss5-hub ~]$ ssh -A observer@apogee-ics ssh: connect to host apogee-ics port 22: No route to host With DanO's help, I checked the network switches in the rack, which looked normal for the instrument network but there were suspiciously no lights flashing for the 2.5m network. The network switches within the plug lab: Port 23 is lit and seems connected, which leads me to believe that we have an issue with the ssh network switch, port 22, within the APOGEE rack instead of the plug room. I called John Wilson for assistance. Also, John Donor, Jose, and Kaike all responded to the critical alerts email. With John's recommendation, I power-cycled the network board in the plug lab but still cannot talk with the apogee-ics machine. I called Shane and left a message on his phone. He called back and said that he would have a look and see what can be done from offsite. There is nothing else that we can do tonight, especially since APOGEE is down during bright time. Turned everything off. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.5010 0.5050 Axial B: -0.2000 -0.1950 Axial C: -0.1170 -0.1100 Trans D: 1.9030 1.9140 Lateral E: 0.3395 0.3395 Lateral F: 0.5465 0.5460 GALILS Commanded: 5650 1750 8800 -4500 -3400 -3100 Actual: 5650 1750 8800 -4500 -3400 -3100 SETMIR VALUES PrimDesOrient: 0.0 44.5 -7.8 -427.0 -2433.9 PrimOrient: 1.07 44.57 -7.9 -426.41 -2433.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -1.0490 -1.0360 Axial B: -1.0590 -1.0720 Axial C: -1.0980 -1.1070 Trans D: -0.0390 -0.0470 GALILS Commanded: 5909150 5829550 5969600 -1500 -7300 Actual: 5883668 5811706 5906213 -1500 -7300 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4657.02 -24.51 -17.51 18.31 125.89 ---- Telescope Status ---- No interlocks bypassed Status at: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 690/690/715/715 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 214 lbs, 22 psi ---- Software Used ---- Get Versions, 11/26/23, 05:20:13 -- STUI software: STUI : 1.7.18b1 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 4.0.0 fliswarm : 0.4.1a0 jaeger : 1.5.1a0 kaiju : 1.3.1 coordio : 1.9.0 fps_calibrati : 2023.09.29 cherno : 0.6.4a0 astrometrynet : 0.93 hal : 0.6.0b5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1-1-ge461 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.3.2 -- done -- --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_149 idlspec2d : v6_1_0 Kronos : 1.6.2a roboscheduler : 1.4.0 /dev/sdb1 2.0T 300G 1.8T 15% /home /dev/sdd1 22T 16T 5.3T 75% /data ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60275 Fill night time in Hours below *=*=*= Total bright time : 11.6 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 11.6 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 0 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 0 *=*=*= Dark time lost to technical issues : 0 Sun 26 Nov 2023 60275 1.00 00:54 01:09 12:32 22:29 09:53 --- bright (11.6 h, w/ 0.2 h for 12 deg design)