Subject: 2.5m obslog 20211123 (SJD 59542) From: kinemuchi@apo.nmsu.edu Date: Wed, 24 Nov 2021 11:20:21 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday November 23, 2021 (SJD 59542) ---- Observing Team ---- Karen Kinemuchi Audrey Oravetz (R) Kaike Pan (online) ---- Observing Plan ---- Sparse plug tests with Cart 2. ---- Observing Summary ---- Lost entire night to weather. Rained on site, with one dust event. Completed the sparse plug tests. NOTES TO DAY STAFF AND DEVELOPERS: ================================== * We had a couple of instances where R1 CCD Temp started to rise, but LN2 fills cooled it back down to -126. Another event happened at 3:40Z, where an auto-fill may not have been successful, and the temp rose by 5 degrees. * We had to issue several boss iacks and boss mechIack to get the spectrograph to respond. Not sure what caused the disconnects. * We were unable to ventilate due to rain on site/nearby radar. We did not collect alt and rot fiducial data. Temps were above 5 degrees for most of the night. ---- Observing Log ---- AFTERNOON CHECKOUT: =================== 23:40Z No APOGEE cals taken. Grabbed mirror numbers - ok. 23:40Z Boss actor was found non-responsive. Error message: 23:43:51 OBSERVER.karen 54 boss f text="command failed: ConnectionResetError(104, 'Connection reset by peer') at /home/sdss5/software/bossICC/main/python/bossICC/DAQInterface.py:1603" Issued a boss iack to clear some phase boot warnings. Immediately after the iack, we got the CCD Temp Read critical alert: 23:48:10 .alerts 0 alerts e alert=camCheck.SP1R0CCDTempRead,critical,"at 2021-11-23 23:48:10 UT found 'Reported by camCheck'",enabled,noack,-1 Since no afternoon force fill happened, went down to the telescope to do a force fill. Temperatures were only slightly out of spec: -119 for R1. With the fill, the temps have returned to normal. Alerts have ceased. 00:14Z Taking some BOSS test cals: 00:14:39 boss exposure 0.0, file=334799 00:16:31 boss exposure 20.0, file=334800 NIGHT: ====== Tried taking BOSS flat and arc, but SoS is reporting the flat is too faint and is being rejected. This cart is sparse plugged, so the flat is faint. Everything is ok. 02:00:58 boss exposure 25.0, file=334801 02:03:31 boss exposure 4.0, file=334802 02:31:51 boss exposure 30.0, file=334803 02:40:52 boss exposure 30.0, file=334804 02:43:10 boss exposure 4.0, file=334805 02:44:50 boss exposure 4.0, file=334806 02:45:31 boss exposure 4.0, file=334807 02:46:11 sp1.motor.move= 996, 996, 996 02:46:11 Hartmann collimate output on cart #2 sp1: offset: r= 0 (In Focus); b= -6.2 (Out of focus) sp1: pred. move: spAverageMove= 996 sp1: pred. resid: r= -996, b= -3.1, txt= OK, spTemp = 7.8 02:50Z checking the flat field lamp at the telescope and moving b1 blue ring by -5 degrees. During the phase boot error, need to do both : --> boss iack --> boss mechIack to ensure the shutters will be working. Sparse Plugging Test for Xmid Range Focused sp1: hartmann collimate 03:02:53 boss exposure 4.0, file=334808 03:03:34 boss exposure 4.0, file=334809 03:04:16 sp1.motor.move= -1105, -1105, -1104 03:04:16 Hartmann collimate output on cart #2 sp1: offset: r= -989 (In Focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -1105 sp1: pred. resid: r= 116, b= 0.4, txt= OK, spTemp = 7.8 Piston relative positions are: A = -109 B = -109 C = -108 03:45Z Had to boss iack again to get the cameras to work. Item #2: mcp ff.on boss exposure flat itime=30 mcp ff.off mcp ne.on mcp hgcd.on boss exposure arc itime=30 boss exposure arc itime=60 03:40Z R1 camera warmed up slightly to -121.5. Did two startfillseqs, and now the temp is hovering between -124.2 and -124.6. R1 temp is set to -126.6, but does not seem to cool down to this value. Went down to the telescope, and LN2 is venting during the fills. Monitoring R1 temps for the rest of the night. Corresponding exposures: 03:46:51 boss exposure 30.0, file=334811 03:48:48 boss exposure 30.0, file=334812 04:21:07 boss exposure 60.0, file=334813 04:32Z Checked images, they look like arcs and flat. Moved collimator 30 pixels: boss moveColl spec=sp1 a=-1890 b=-1890 c=1890 boss exposure arc itime=30 boss exposure arc itime=60 Corresponding exposure numbers: 04:32:09 sp1.motor.move= -1890, -1890, 1891 04:32:48 boss exposure 30.0, file=334814 04:34:46 boss exposure 60.0, file=334815 Move collimator -10 pixels: boss moveColl spec=sp1 a=630 b=630 c=-630 boss exposure arc itime=30 boss exposure arc itime=60 04:37:00 sp1.motor.move= 630, 630, -633 04:48:27 boss exposure 30.0, file=334816 04:50:29 boss exposure 60.0, file=334817 Move collimator -2 pixels: boss moveColl spec=sp1 a=126 b=126 c=-126 boss exposure arc itime=30 boss exposure arc itime=60 Corresponding exposure numbers: 04:52:49 sp1.motor.move= 125, 126, -125 04:53:19 boss exposure 30.0, file=334818 04:55:18 boss exposure 60.0, file=334819 05:13Z rain sensor went off. we remain closed. Repeat this step 19 times: Output: 04:57:56 sp1.motor.move= 126, 125, -126 04:58:15 boss exposure 30.0, file=334820 05:00:08 boss exposure 60.0, file=334821 05:02:15 sp1.motor.move= 126, 126, -126 05:02:34 boss exposure 30.0, file=334822 05:04:27 boss exposure 60.0, file=334823 05:06:35 sp1.motor.move= 126, 126, -126 05:06:54 boss exposure 30.0, file=334824 05:08:47 boss exposure 60.0, file=334825 05:10:54 sp1.motor.move= 126, 127, -126 05:11:13 boss exposure 30.0, file=334826 05:13:06 boss exposure 60.0, file=334827 05:15:13 sp1.motor.move= 126, 126, -126 05:15:32 boss exposure 30.0, file=334828 05:17:25 boss exposure 60.0, file=334829 05:19:32 sp1.motor.move= 126, 125, -126 05:19:51 boss exposure 30.0, file=334830 05:21:46 boss exposure 60.0, file=334831 05:23:53 sp1.motor.move= 127, 127, -126 05:24:11 boss exposure 30.0, file=334832 05:26:04 boss exposure 60.0, file=334833 05:28:12 sp1.motor.move= 126, 125, -126 05:28:31 boss exposure 30.0, file=334834 05:30:24 boss exposure 60.0, file=334835 05:32:31 sp1.motor.move= 126, 127, -126 05:32:50 boss exposure 30.0, file=334836 05:34:46 boss exposure 60.0, file=334837 05:36:53 sp1.motor.move= 125, 126, -126 05:37:12 boss exposure 30.0, file=334838 05:39:05 boss exposure 60.0, file=334839 05:41:12 sp1.motor.move= 126, 126, -126 05:41:31 boss exposure 30.0, file=334840 05:43:24 boss exposure 60.0, file=334841 05:45:31 sp1.motor.move= 127, 125, -126 05:45:50 boss exposure 30.0, file=334842 05:47:46 boss exposure 60.0, file=334843 05:49:53 sp1.motor.move= 126, 126, -126 05:50:12 boss exposure 30.0, file=334844 05:52:05 boss exposure 60.0, file=334845 05:54:12 sp1.motor.move= 126, 126, -126 05:54:31 boss exposure 30.0, file=334846 05:56:24 boss exposure 60.0, file=334847 05:58:31 sp1.motor.move= 126, 126, -126 05:58:50 boss exposure 30.0, file=334848 06:00:46 boss exposure 60.0, file=334849 06:02:53 sp1.motor.move= 125, 126, -126 06:03:12 boss exposure 30.0, file=334850 06:05:04 boss exposure 60.0, file=334851 06:07:11 sp1.motor.move= 127, 126, -126 06:07:30 boss exposure 30.0, file=334852 06:09:24 boss exposure 60.0, file=334853 06:11:31 sp1.motor.move= 125, 126, -126 06:11:50 boss exposure 30.0, file=334854 06:13:46 boss exposure 60.0, file=334855 06:15:53 sp1.motor.move= 127, 126, -126 06:16:12 boss exposure 30.0, file=334856 06:18:05 boss exposure 60.0, file=334857 Moving collimator by -10 pixels: boss moveColl spec=sp1 a=630 b=630 c=-630 boss exposure arc itime=30 boss exposure arc itime=60 Corresponding exposure numbers: 06:21:06 sp1.motor.move= 629, 631, -630 06:21:40 boss exposure 30.0, file=334858 06:23:36 boss exposure 60.0, file=334859 Move collimator back: 06:26:23 sp1.motor.move= -1890, -1890, 1891 Piston positions are: A = -110 B= -109 C=-106 More or less in the same position as beginning. Taking a normal arc and flat: 06:28:12 boss exposure 4.0, file=334860 06:29:46 boss exposure 30.0, file=334861 This sparse plug test is complete. On to the next one... 06:15Z Another fill occurred. R1 temps remain stable at -124.6. Sparse Plugging Test for WAVEMID Ranges As requested, checking focus again, but this time issuing hartmann collimate noSubframe --> need to turn on lamps! try again: 06:38:15 boss exposure 4.0, file=334864 06:39:29 boss exposure 4.0, file=334865 06:40:47 sp1.motor.move= -128, -128, -128 06:40:47 Hartmann collimate output on cart #2 sp1: offset: r= 0 (In Focus); b= 0.8 (In Focus) sp1: pred. move: spAverageMove= -128 sp1: pred. resid: r= 128, b= 0.4, txt= OK, spTemp = 7.5 Still in focus. Piston positions: A = -238 B = -237 C = -234 Taking normal flat and two arcs: mcp ff.on boss exposure flat itime=30 mcp ff.off mcp ne.on mcp hgcd.on boss exposure arc itime=30 boss exposure arc itime=60 Corresponding exposure numbers: 06:42:57 boss exposure 30.0, file=334866 06:44:53 boss exposure 30.0, file=334867 06:46:46 boss exposure 60.0, file=334868 Visually inspected and ok. Moving collimator for 50A: boss moveColl spec=sp1 a=1550 b=-1550 c=0 boss exposure arc itime=30 boss exposure arc itime=60 Corresponding exposure numbers: 06:49:57 sp1.motor.move= 1550, -1550, 0 06:50:16 boss exposure 30.0, file=334869 06:52:09 boss exposure 60.0, file=334870 Move collimator for -5A: boss moveColl spec=sp1 a=-155 b=155 c=0 boss exposure arc itime=30 boss exposure arc itime=60 Corresponding exposure numbers: 06:55:18 sp1.motor.move= -155, 155, 0 06:55:36 boss exposure 30.0, file=334871 06:57:29 boss exposure 60.0, file=334872 07:50Z Dust counts have increased. Humidity at 60%. We remain closed. Repeat this 19 times. Output: 06:59:59 sp1.motor.move= -155, 155, 0 07:00:18 boss exposure 30.0, file=334873 07:02:12 boss exposure 60.0, file=334874 07:04:18 sp1.motor.move= -155, 155, 0 07:04:37 boss exposure 30.0, file=334875 07:06:30 boss exposure 60.0, file=334876 07:08:38 sp1.motor.move= -155, 155, 0 07:08:56 boss exposure 30.0, file=334877 07:10:49 boss exposure 60.0, file=334878 07:12:56 sp1.motor.move= -155, 154, 0 07:13:15 boss exposure 30.0, file=334879 07:15:08 boss exposure 60.0, file=334880 07:17:15 sp1.motor.move= -155, 156, 0 07:17:34 boss exposure 30.0, file=334881 07:19:27 boss exposure 60.0, file=334882 07:21:34 sp1.motor.move= -155, 155, 0 07:21:53 boss exposure 30.0, file=334883 07:23:47 boss exposure 60.0, file=334884 07:25:54 sp1.motor.move= -155, 155, 0 07:26:12 boss exposure 30.0, file=334885 07:28:05 boss exposure 60.0, file=334886 07:30:12 sp1.motor.move= -155, 155, 0 07:30:31 boss exposure 30.0, file=334887 07:32:24 boss exposure 60.0, file=334888 07:34:32 sp1.motor.move= -155, 155, 0 07:34:51 boss exposure 30.0, file=334889 07:36:46 boss exposure 60.0, file=334890 07:38:53 sp1.motor.move= -155, 155, 0 07:39:12 boss exposure 30.0, file=334891 07:41:05 boss exposure 60.0, file=334892 07:43:12 sp1.motor.move= -155, 155, 0 07:43:31 boss exposure 30.0, file=334893 07:45:24 boss exposure 60.0, file=334894 07:47:31 sp1.motor.move= -155, 155, 0 07:47:50 boss exposure 30.0, file=334895 07:49:46 boss exposure 60.0, file=334896 07:51:53 sp1.motor.move= -155, 155, 0 07:52:12 boss exposure 30.0, file=334897 07:54:05 boss exposure 60.0, file=334898 07:56:13 sp1.motor.move= -155, 154, 0 07:56:31 boss exposure 30.0, file=334899 07:58:24 boss exposure 60.0, file=334900 08:00:31 sp1.motor.move= -155, 155, 0 08:00:50 boss exposure 30.0, file=334901 08:02:46 boss exposure 60.0, file=334902 08:04:53 sp1.motor.move= -155, 156, 0 08:05:12 boss exposure 30.0, file=334903 08:07:04 boss exposure 60.0, file=334904 08:09:11 sp1.motor.move= -155, 154, 0 08:09:30 boss exposure 30.0, file=334905 08:11:23 boss exposure 60.0, file=334906 08:13:30 sp1.motor.move= -155, 155, 0 08:13:50 boss exposure 30.0, file=334907 08:15:46 boss exposure 60.0, file=334908 08:17:53 sp1.motor.move= -155, 155, 0 08:18:12 boss exposure 30.0, file=334909 08:20:05 boss exposure 60.0, file=334910 08:17Z another fill happened. R1 temps are still ok. 08:28Z Moving collimator back: boss moveColl spec=sp1 a=1550 b=-1550 c=0 Piston positions: A = -238 B = -238 C = -234 Returned to the same position. Taking a normal arc and flat 08:31:06 boss exposure 4.0, file=334911 08:32:37 boss exposure 30.0, file=334912 08:34Z Lots of radar nearby. Dust levels have risen. We remain closed. 11:05Z Giving up. Lots of radar, some sprinkles. ---- BOSS Spectrographs Collimation Summary ---- --- Hartmann --- Time Cart TSP1 R1 B1 Move1 B1Resid ================================================================================ 02:46 2-15315A 7.8 0 -6.2 996 -3.1 03:04 2-15315A 7.8 -989 3.8 -1105 0.4 06:38 2-15315A 7.5 nan nan nan nan 06:41 2-15315A 7.5 0 0.8 -128 0.4 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0060 0.0080 Axial B: 0.0060 0.0070 Axial C: -0.1170 -0.1230 Trans D: -0.0348 -0.0358 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5340 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 7.28 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.4570 -9.4520 Axial B: -9.4650 -9.4620 Axial C: -9.4970 -9.4990 Trans D: 0.0980 0.0910 GALILS Commanded: 4635300 4630500 4652900 -50 -3450 Actual: 4607224 4607161 4607250 -50 -3450 SETMIR VALUES SecDesOrient: 3657.0 0.0 0.0 0.0 42.22 SecOrient: 3656.97 -0.01 -0.02 -0.01 42.69 ---- Telescope Status ---- Never opened. No interlocks bypassed Status at:11:09Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 2 Counterweights at: 541 542 560 561 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 214 lbs, 22.2 psi ---- Software Used ---- Get Versions, 11/23/21, 23:32:56 -- STUI software: STUI : 1.7.13 alerts : 2.1.6 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 gcamera : FAILED guider : 4.0.0 hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_147 -mcp.altFiduc : v1_147 -mcp.rotFiduc : v1_147 platedb : 2.8.0 sop : 0.1.0-alpha.0 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : master -- done -- Home Disk Hogs (GB): muna 132.068 eboss 83.1765 manga 66.812 sdss3 27.1813 apogee 17.4802 logging 13.6955 Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 661G 27G 97% /home 10.25.1.230:/data 21T 17T 3.9T 81% /data Setting up: idlspec2d Flavor: Linux64 Version: v5_7_12 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: trunk Setting up: |specflat Flavor: Linux64 Version: v1_20 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |sdss_python_module Flavor: Linux64 Version: v1_6_7 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.9.7 Setting up: |psycopg2 Flavor: Linux64 Version: 2.5.3 Setting up: |idlutils Flavor: Linux64 Version: v5_5_16 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: |photolog Flavor: Linux64 Version: trunk Setting up: actorcore Flavor: Linux64 Version: v4_1_6 Setting up: |opscore Flavor: Linux64 Version: v2_5 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v30_2 Setting up: |sdsstools Flavor: Linux64 Version: v4_33 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_13 Setting up: |RO Flavor: Linux64 Version: 3.6.9 -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v7_0_9 petunia : 2.2.3 autoscheduler : git sdss_python_module: trunk mangadrp : trunk exported