Subject: 2.5m obslog 20210924 (SJD 59482) From: kinemuchi@apo.nmsu.edu Date: Sat, 25 Sep 2021 10:40:43 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday September 24, 2021 (SJD 59482) ---- Observing Team ---- Katie Grabowski Karen Kinemuchi Kaike Pan (remote for M1 galils) Jose Sanchez-Gallego ---- Observing Plan ---- Focus vs altitude test if weather clears. ---- Observing Summary ---- Lost about 6.5 hours to weather and 3 hours to software. NOTES TO DAY STAFF AND DEVELOPERS: ================================== * Please check the ethernet cable on the podium laptop. Connectivity may have been flaky? Was ok for the rest of the night. See log for details. * Recorded mirror numbers were old ones after sdss5 server switchover. Recovered current mirror numbers, but recorded focus is 48? See log for more details. * The humidity for 2.5m on the weather status page does not match with 3.5m information. Temperatures are the same, however. Can someone please check if the sensor is reading ok or if this is a software issue? * Unable to run focus sweep script or guider gui to take exposures. Dependencies with SOP prevented us from completing the test. However, clouds increased later in the night. Called Jose and left a message. * getDust.py are not returning dust counts. Estimating from the Dust counter gui, ~4000 counts. * Unable to run APOGEE cals from the podium laptop. The scripts are missing. ---- Observing Log ---- Afternoon: =========== Tracy reported that the MCP at the podium was unavailable. Rebooted the laptop and checked no other MCP processed running. STUI running from SDSS-5 hub. Test actors currently shows sop and platedb pings fail, as well as the expected boss and gcamera actors. list_ap doesn't work because there's no pyfits module. ls of apogee data directory shows short dark files are there, but nothing about bundling Can't run APOGEE evening cals from SOP Used eveningcals_nodith.inp in Run Commands window 23:15Z- Started APOGEE cals Completed FPI cals too. Podium laptop STUI connection was 9GB Unable to reconnect STUI on podium laptop Looks like MCP reported that it became unavailable around 20:24Z MCP at podium says "TPM not available" Other connected MCP windows (on obs1/2) look ok Jose checked the network on the laptop, and though it seems to be inconsistent, it's working ok. MCP running fine now. Some rain on site. M1 mirror numbers are off. It looks like the galils were hard-coded for some other pre-set value, and not the galils after this summer's shutdown. The SET MIR values for both mirrors are zero. Kaike logged in and said the default.dat file on SDSS5 server need an update. He did that, and we restarted the mirror controllers. Humidity levels seem incorrect. It's actively raining on site, yet the humidity is purportedly ~50% and the dewpoint is approx zero. The temperature seems consistent. Restarted optidew processes, then tpm. When the MCP became unavailable, we restarted the MCP then TPM. All systems alive and well. Will double check fiducials. Trying to update the weather made absolutely no difference. It seems the weather information isn't correct. The trouble might be in the webapp, since the APO actor and TPM are the same. 01:25:14 OBSERVER.karen 21 apo : pressure=21.599; windd=91.0; winds=14.0; gustd=78.3; gusts=17.4; airtemp=10.5; dewpoint=7.3; dpErr=P; humidity=80.8; dusta=dpDep; dustb=dpDep; dustc=dpDep; dustd=dpDep; windd25m=-999.0; winds25m=- 999.0; encl25m=0; truss25m=-19.8; airTempPT=9.9; dpTempPT=-0.1; humidPT=50.0 That means there are two places the weather info comes from, and rather than seeing 80.8% humidity given a dewpoint of 7.3, the weather page pulls the PT humidity with a dewpoint of -0.1. Started engineering camera, took flat and dark. Restarted mirror controllers on SDSS-V servers. logged into TCC computer, systemctl --user restart prim25m Mirror numbers for M1 look ok now. Still no SET MIR values, focus reported as 48 and scale is reported as 4 mirror orient values show up in the mirror status window in STUI no difference whether using the new or old LUI After restarting tcc, checked fiducials. All ok. Many of the Telescope Controls commands that use stop fail. tcc track still works 04:31- STUI memory 3GB NIGHT: ====== 04:40Z Delaying ventilation and opening due to nearby radar and clouds. 06:05Z Ventilation begins. 06:13:39 encl25m : open; We will wait about an hour for thermal equilibrium with the telescope outside. Clouds are forming in the south, so we will attempt to finish one set of the focus sweep vs altitude test tonight. 07:00Z- tcc track/pterr 0, 65 obs /rotty=obj /rota=0 /magrange=(8.5,9.5) guider GUI hangs with "guider on oneExp time=3" command Error message: guider 0 guider e text="Unexpected exception 'sop' in guider master thread" no trouble with guider darks 07:24Z Attempted to contact Jose regarding the guider/sop failures. We don't know if there is a way to revert back to SDSS-4 servers, where SOP has been running fine. 07:45Z Clouds are reappearing in the south and west. 09:45Z Still unable to command the ecamera to take exposures or do a focus sweep. Getting very cloudy, but no radar. 10:00Z Giving up. Running APOGEE morning cals from control room. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 4 MIGS TONIGHT NOMINAL Axial A: 0.1440 0.0080 Axial B: -0.0270 0.0330 Axial C: -0.0430 -0.1140 Trans D: -0.0290 -0.0395 Lateral E: 0.3315 0.3310 Lateral F: 0.5345 0.5335 GALILS Commanded: 10650 -650 6250 43400 -8800 -8400 Actual: 10650 -650 6250 43400 -8800 -8400 SETMIR VALUES PrimDesOrient: 0 PrimOrient: 0 SECONDARY: ---------- Focus: 48 Air temp: 9.94 Altitude: 30.000587290755433 MIGS TONIGHT NOMINAL Axial A: -9.4570 -9.4610 Axial B: -9.4740 -9.4760 Axial C: -9.5030 -9.5050 Trans D: 0.0900 0.0880 GALILS Commanded: 4634000 4630700 4653050 -50 -3450 Actual: 4608654 4608339 4608119 -50 -3450 SETMIR VALUES SecDesOrient: 0 SecOrient: 0 PRIMARY: -------- Scale: 4 MIGS TONIGHT NOMINAL Axial A: 0.0070 0.0080 Axial B: 0.0320 0.0330 Axial C: -0.1150 -0.1140 Trans D: -0.0375 -0.0395 Lateral E: 0.3310 0.3310 Lateral F: 0.5345 0.5335 GALILS Commanded: 12500 -1600 5350 47050 -8800 -8400 Actual: 12500 -1600 5350 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0 PrimOrient: 0 SECONDARY: ---------- Focus: 48 Air temp: 11.17 Altitude: 30.000587290755433 MIGS TONIGHT NOMINAL Axial A: -9.4570 -9.4610 Axial B: -9.4750 -9.4760 Axial C: -9.5030 -9.5050 Trans D: 0.0910 0.0880 GALILS Commanded: 4633200 4630550 4651600 -50 -3450 Actual: 4606985 4607193 4607187 -50 -3450 SETMIR VALUES SecDesOrient: 0 SecOrient: 0 ---- Telescope Status ---- 06:05Z Started ventilation 06:13Z encl25m : open 10:11Z encl25m : closed No interlocks bypassed ---- Software Used ---- Get Versions, 09/25/21, 10:31:17 -- STUI software: STUI : 1.7.13 alerts : 2.1.4 apo : 3.0.0 boss : 6.0.0-alpha.0 -boss.specMec : Offline gcamera : FAILED guider : 4.0.0 hartmann : 2.0.0 hub : 5.0.0a0 mcp : 2.0-7-g8799 -mcp.plc : 34 -mcp.azFiduci : v1_146 -mcp.altFiduc : v1_146 -mcp.rotFiduc : v1_146 platedb : FAILED sop : FAILED tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.7 -- done -- df -kh Filesystem Size Used Avail Use% Mounted on /dev/vda2 7.7G 7.0G 332M 96% / /dev/vdd 688G 660G 28G 96% /home /dev/loop1 194M 133M 52M 73% /boot tmpfs 3.4G 0 3.4G 0% /dev/shm /dev/vdb 19G 8.8G 9.1G 49% /exports/hub-local /dev/vdc1 21T 14T 6.5T 68% /data 10.25.1.230:/data 21T 14T 6.5T 68% /data -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v7_0_9 petunia : 2.2.3 autoscheduler : git sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- APOGEE data list, mjd=59482, all raw_data: /data/apogee/utr_cdr/59482/apRaw-*.fits archive: /data/apogee/archive/59482/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59482/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59482 1 no file found: /data/apogee/utr_cdr/59482/apRaw-39200001.fits 59482 2 22:38 39200002 Dark 3 A 13.0 ------- x-x-x - - 59482 3 22:41 39200003 Dark 3 A 13.0 ------- a-b-c - - 59482 4 23:10 39200004 Dark 3 A 13.0 ------- a-b-c - - 59482 5 23:26 39200005 Dark 60 A 13.0 ------- a-b-c - - 59482 6 23:28 39200006 QuarFlat 10 A 13.0 ------- a-b-c - 95 59482 7 23:30 39200007 Arc---- 12 A 13.0 ------- a-b-c - - 59482 8 23:32 39200008 Arc---- 12 A 13.0 ------- a-b-c - - 59482 9 23:39 39200009 Arc---- 40 A 13.0 ------- a-b-c - - 59482 10 23:46 39200010 Arc---- 40 A 13.0 ------- a-b-c - - 59482 11 23:57 39200011 Dark 60 A 13.0 ------- a-b-c - - 59482 12 00:35 39200012 Arc---- 20 A 13.0 ------- a-b-c - - FPI 59482 13 00:44 39200013 Arc---- 30 A 13.0 ------- a-b-c - - FPI -------------------------------------------------------------------------------- APOGEE data list, mjd=59482, all raw_data: /data/apogee/utr_cdr/59482/apRaw-*.fits archive: /data/apogee/archive/59482/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59482/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59482 1 no file found: /data/apogee/utr_cdr/59482/apRaw-39200001.fits 59482 2 22:38 39200002 Dark 3 A 13.0 ------- x-x-x - - 59482 3 22:41 39200003 Dark 3 A 13.0 ------- a-b-c - - 59482 4 23:10 39200004 Dark 3 A 13.0 ------- a-b-c - - 59482 5 23:26 39200005 Dark 60 A 13.0 ------- a-b-c - - 59482 6 23:28 39200006 QuarFlat 10 A 13.0 ------- a-b-c - 95 59482 7 23:30 39200007 Arc---- 12 A 13.0 ------- a-b-c - - 59482 8 23:32 39200008 Arc---- 12 A 13.0 ------- a-b-c - - 59482 9 23:39 39200009 Arc---- 40 A 13.0 ------- a-b-c - - 59482 10 23:46 39200010 Arc---- 40 A 13.0 ------- a-b-c - - 59482 11 23:57 39200011 Dark 60 A 13.0 ------- a-b-c - - 59482 12 00:35 39200012 Arc---- 20 A 13.0 ------- a-b-c - - FPI 59482 13 00:44 39200013 Arc---- 30 A 13.0 ------- a-b-c - - FPI 59482 14 10:27 39200014 Dark 60 A 13.0 ------- x-x-x - - --------------------------------------------------------------------------------