Subject: 2.5m obslog 20211203 (SJD 59552) From: dmbiz@apo.nmsu.edu Date: Sat, 4 Dec 2021 14:23:38 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday December 03, 2021 (SJD 59552) ---- Observing Team ---- Kaike Pan Dmitry Bizyaev Commissioning team: Richard Pogge Conor Sayres José Sánchez-Gallego Zoom-in observers: ALL Ryan: on the phone in the evening Jamey: at 7 am in the morning on th ephone ---- Observing Plan ---- FPS commission ---- Observing Summary ---- Clear night, photometric, with variable seeing. Activities performed: Checked the APOGEE cable in motion when th e gang connector is plugged to the FPS, Kaike created a new instrumental block for the "fps", we counted stars in the fields sent by Jon Holtzman earlier, We ran Conor's test of applying small, random offsets at a field, we ran focus sweeps at different angles of rotator. Notes ----- - dust accumulated 0.9K - attention to day time crew: we cannot either ping the sp1daq or reload the sp1daq. A long investigation (see log) points at a hardware problem. James, please check wire connection to sp1 DAQ. - BAD THING happened in the morning. We found that we cannot do start fill seq. Attempts to reload the sp1cam controller failed. No any alerts came in, but we suspect that the camera may be warm. We decided not to use the emergency fill pendant because of a danger of condensations, and to contact engineers first. Note that alert actor did not report any problems. We left a voice mail to Ryan and then to James. Later we contacted Jamey, and he asked Ryan to drive up at APO. ---- Observing Log ---- Afternoon: ========= Checked with Ryan and commission team members for new changes, plans. Take apogee evening calibration. The data is in /data/apogee/u Night ----- 00:22Z started ventilation , opened enclosure First we found that BOSS doesn't take exposures. Reloaded controllers, but the sp1daq bright up an error: 1:41:05 .boss 0 boss w text="failed to start DAQ sp1daq, disconnecting myself, [Failure instance: Traceback: : [Errno 113] No route to host" ... [see the Problem in Detail for the full error message] We tried to reset Daqs, then to track down the cables. No signs of problems, and it did not fix the connection. We called Ryan. He confirmed that James worked with wires today, so it is good to ask James. We gave him a call, no James, left a voice message. Also, thre are no ports to log out for the daq since it goes directly to the telescope and not through the xyplex. Jose joined the search and we traced all copper and glass wires. Nothing actually helped. We have to, but we cannot ping 192.168.1.4 (the daq ip) from the sdss5-boss-icc computer. Need James for the further investigation. using BOSS camera was possible but not necessary for the activities tonight. ** Checking the APOGEE cable in motion when the gang connector is plugged to the FPS. Rick showed us how to connect/disconnect the gang connector with those new 3 FPI fibers. A short procedure t move the gang connector from the Dense port to the FPS is 1. Remove caps from 2 pink fibers and from the 2 small ports at the bottom of FPS - those two at the edges. Do not touch the middle one. (usually is it covered with the red cap) 2. Connect the gang connector, in a regular way. 3. Connect the 2 fibers to the ports in the FPS. To move the gang connector to the podium Dense port, do the 3-2-1. We moved telescope between the zenith and the stow altitudes, and move rotator. The APOGEE cable doesn't show any unusual tension. ** New Instrumental block type Kaike created a new instrumental block for the "fps". (Use lower case!) The difference from the spectro is (3.9.7) [observer@sdss5-boss-icc: inst]$ diff i_ca1_fps.dat i_ca1_spectro.dat 17c17 Inst_Foc 3800 !Changed by KPAN 06132021 20c20 Rot_Inst_xy 0.000 0.0010 23c23 Rot_Inst_ang -89.89 ** Conor adjusted TCC to use the new wavelength: 6231A , which corresponds tot he r filter. tcc was rebooted to make changes in effect. ** Counting stars in fields sent by Jon Holtzman exposures: RA Dec 15,60 sec exposure # ---------------- 330 20 11,12 345 20 13,14 0 20 15,16 15 20 17,18 30 20 19,20 45 20 21,22 60 20 23,24 75 20 25,26 90 20 27,28 120 20 29,30 Please note: since we've taken images for focusing with only one camera, gfa5, now gfa5 #31 corresponds to gfa2,3,4,6 #11. The table above shows the numbers for gfas 2,3,4,6 Counting stars with peaks > 2000 above the background in the files: /data/gcam/59552/gimg-gfa2n-0011.fits 1 /data/gcam/59552/gimg-gfa2n-0012.fits 13 /data/gcam/59552/gimg-gfa2n-0013.fits 2 /data/gcam/59552/gimg-gfa2n-0014.fits 8 /data/gcam/59552/gimg-gfa2n-0015.fits 3 /data/gcam/59552/gimg-gfa2n-0016.fits 7 /data/gcam/59552/gimg-gfa2n-0017.fits 15 /data/gcam/59552/gimg-gfa2n-0018.fits 17 /data/gcam/59552/gimg-gfa2n-0019.fits 2 /data/gcam/59552/gimg-gfa2n-0020.fits 16 /data/gcam/59552/gimg-gfa2n-0021.fits 2 /data/gcam/59552/gimg-gfa2n-0022.fits 8 /data/gcam/59552/gimg-gfa2n-0023.fits 7 /data/gcam/59552/gimg-gfa2n-0024.fits 10 /data/gcam/59552/gimg-gfa2n-0025.fits 18 /data/gcam/59552/gimg-gfa2n-0026.fits 42 /data/gcam/59552/gimg-gfa2n-0027.fits 8 /data/gcam/59552/gimg-gfa2n-0028.fits 39 /data/gcam/59552/gimg-gfa2n-0029.fits 1 /data/gcam/59552/gimg-gfa2n-0030.fits 5 /data/gcam/59552/gimg-gfa3n-0011.fits 0 /data/gcam/59552/gimg-gfa3n-0012.fits 12 /data/gcam/59552/gimg-gfa3n-0013.fits 2 /data/gcam/59552/gimg-gfa3n-0014.fits 7 /data/gcam/59552/gimg-gfa3n-0015.fits 1 /data/gcam/59552/gimg-gfa3n-0016.fits 4 /data/gcam/59552/gimg-gfa3n-0017.fits 1 /data/gcam/59552/gimg-gfa3n-0018.fits 1 /data/gcam/59552/gimg-gfa3n-0019.fits 0 /data/gcam/59552/gimg-gfa3n-0020.fits 2 /data/gcam/59552/gimg-gfa3n-0021.fits 3 /data/gcam/59552/gimg-gfa3n-0022.fits 5 /data/gcam/59552/gimg-gfa3n-0023.fits 3 /data/gcam/59552/gimg-gfa3n-0024.fits 12 /data/gcam/59552/gimg-gfa3n-0025.fits 4 /data/gcam/59552/gimg-gfa3n-0026.fits 11 /data/gcam/59552/gimg-gfa3n-0027.fits 10 /data/gcam/59552/gimg-gfa3n-0028.fits 31 /data/gcam/59552/gimg-gfa3n-0029.fits 5 /data/gcam/59552/gimg-gfa3n-0030.fits 17 /data/gcam/59552/gimg-gfa4n-0011.fits 7 /data/gcam/59552/gimg-gfa4n-0012.fits 14 /data/gcam/59552/gimg-gfa4n-0013.fits 3 /data/gcam/59552/gimg-gfa4n-0014.fits 3 /data/gcam/59552/gimg-gfa4n-0015.fits 3 /data/gcam/59552/gimg-gfa4n-0016.fits 8 /data/gcam/59552/gimg-gfa4n-0017.fits 3 /data/gcam/59552/gimg-gfa4n-0018.fits 14 /data/gcam/59552/gimg-gfa4n-0019.fits 0 /data/gcam/59552/gimg-gfa4n-0020.fits 1 /data/gcam/59552/gimg-gfa4n-0021.fits 6 /data/gcam/59552/gimg-gfa4n-0022.fits 9 /data/gcam/59552/gimg-gfa4n-0023.fits 1 /data/gcam/59552/gimg-gfa4n-0024.fits 6 /data/gcam/59552/gimg-gfa4n-0025.fits 25 /data/gcam/59552/gimg-gfa4n-0026.fits 46 /data/gcam/59552/gimg-gfa4n-0027.fits 10 /data/gcam/59552/gimg-gfa4n-0028.fits 45 /data/gcam/59552/gimg-gfa4n-0029.fits 1 /data/gcam/59552/gimg-gfa4n-0030.fits 14 /data/gcam/59552/gimg-gfa6n-0011.fits 5 /data/gcam/59552/gimg-gfa6n-0012.fits 9 /data/gcam/59552/gimg-gfa6n-0013.fits 3 /data/gcam/59552/gimg-gfa6n-0014.fits 7 /data/gcam/59552/gimg-gfa6n-0015.fits 1 /data/gcam/59552/gimg-gfa6n-0016.fits 2 /data/gcam/59552/gimg-gfa6n-0017.fits 5 /data/gcam/59552/gimg-gfa6n-0018.fits 9 /data/gcam/59552/gimg-gfa6n-0019.fits 4 /data/gcam/59552/gimg-gfa6n-0020.fits 5 /data/gcam/59552/gimg-gfa6n-0021.fits 1 /data/gcam/59552/gimg-gfa6n-0022.fits 2 /data/gcam/59552/gimg-gfa6n-0023.fits 1 /data/gcam/59552/gimg-gfa6n-0024.fits 11 /data/gcam/59552/gimg-gfa6n-0025.fits 2 /data/gcam/59552/gimg-gfa6n-0026.fits 11 /data/gcam/59552/gimg-gfa6n-0027.fits 14 /data/gcam/59552/gimg-gfa6n-0028.fits 47 /data/gcam/59552/gimg-gfa6n-0029.fits 4 /data/gcam/59552/gimg-gfa6n-0030.fits 10 ** We run Conor's test to apply small, random offsets at a field (offset, then go back, and so on). RA=60 dec = +20 starting frame # 31 - 32 Wrong attempt. Adjusting script... Checking seeing at high altitude: /data/gcam/59552/gimg-gfa2n-0033.fits 590.741 1612.407 1.298 Starting over: RA=60 dec = +20 starting frame # 34 - 50 - Sorry, did not copy the whole script file. Starting over RA=60 dec = +20 starting frame # 51 - 115 RA=75 dec = +20 starting frame # 116 - 180 ** Focus sweep at different angles of rotator Note: for the gfa5 the corresponding frame numbers are +20 (e.g. 190 for gfa4 is 210 for gfa5) ROT = 0 obj exp time = 30 sec rot foc frame 0 25 190 0 50 191 0 75 192 0 100 193 0 125 194 0 150 195 0 175 196 0 200 197 0 225 198 0 250 199 0 275 200 ROT = 30 obj exp time = 30 sec rot foc frame 30 25 201 30 50 202 30 75 203 30 100 204 30 125 205 30 150 206 30 175 207 30 200 208 30 225 209 30 250 210 30 275 211 ROT = 60 obj exp time = 30 sec rot foc frame 60 25 213 60 50 214 60 75 215 60 100 216 60 125 217 60 150 218 60 175 219 60 200 220 60 225 221 60 250 222 60 275 223 ** One more dithering test for Conor RA = 120 Dec = +20 The exposure range is 228 - 292 ** Taking images at different Rot angle Rot Frame 0 293 60 294 120 295 180 296 240 297 300 298 360 299 ** One more Conor's dither test for the field RA = 135 , Dec = +20 First exposure 305 (still the same +20 difference in GFA 5) Last exposure 369 The script that we used, as an example. is shown in the Miscellaneous secion 12:50Z enclosure closed - BAD THING happened in the morning. We found that we cannot do start fill seq. Attempts to reload the sp1cam controller failed. No any alerts came in, but we suspect that the camera may be warm. We decided not to use the emergency fill pendant because of a danger of condensations, and to contact engineers first. 13:12:53 .alerts 0 alerts i activeAlerts=camCheck.SP1MicroNotTalking,camCheck.SP1ExecBoot,camCheck.SP1PhaseBoot,camCheck.SP1R0CCDTempRead We tried to restart the bossICC, but it did not help. We cleared ports on xyplex and restarted bossICC. sp1cam reported as connected, but then it fails to report 13:27:57 .boss 0 boss i text="Controller sp1cam connected." 13:27:51 .boss 34524 boss w text="camcheck found missing or unresponsive cameras: {'sp1cam'}" 13:27:51 .boss 34524 boss w 13:27:51 .boss 34524 boss w camCheck=SP1MicroNotTalking Note that alert actor did not report any problems. According to Grafana, last temperature and pressure for the camera were reported at 02:06Z today. We contacted Jamey. He was able to sign Ryan out of Morpheus's realm. Ryan heads up to the APO. This is what we have at 14:21Z. ---- Mirror Numbers ---- Not taken in the evening because we are going to restart TCC and change it - Conor needs to update the wavelength to that close to which the guider uses. PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0080 Axial B: 0.0030 0.0070 Axial C: -0.1150 -0.1230 Trans D: -0.0338 -0.0358 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5340 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 5.06 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6570 -9.4520 Axial B: -9.6710 -9.4620 Axial C: -9.6970 -9.4990 Trans D: 0.1140 0.0910 GALILS Commanded: 4891800 4890400 4905100 -50 -3450 Actual: 4859028 4859148 4859211 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.93 -0.04 0.05 0.03 42.89 We saved these mirror numbers as Nominals taken with the FPS instrumental block. Note the difference for the M2 MIGs. ---- Telescope Status ---- 00:22Z Started ventilation 00:22Z encl25m : open 12:50Z encl25m : closed No interlocks bypassed Status at: 13:45Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 686 / 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 180 lbs, 19 psi ---- Software Used ---- TUI : 1.7.13 alerts : 2.1.6 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : FAILED hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 sop : 0.1.0-alpha.0 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.9 /dev/sdb1 2.0T 202G 1.9T 10% /home 10.25.1.230:/data 21T 17T 3.5T 83% /data ---- APOGEE Data Summary ---- APOGEE data list, mjd=59552, all raw_data: /data/apogee/utr_cdr/59552/apRaw-*.fits archive: /data/apogee/archive/59552/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59552/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59552 1 23:15 39900001 Dark 3 A 13.0 ------- a-b-c - - 59552 2 23:42 39900002 Dark 3 A 13.0 ------- a-b-c - - 59552 3 23:52 39900003 Arc---- 3 A 13.0 19- 0 a-b-c - - 59552 4 00:43 39900004 Dark 60 A 13.0 ------- a-b-c - - 59552 5 00:45 39900005 QuarFlat 10 A 13.0 19- 0 a-b-c - 107 59552 6 00:47 39900006 Arc-Thar 12 A 13.0 19- 0 a-b-c 4.78 - 59552 7 00:55 39900007 Arc-Une 40 A 13.0 19- 0 a-b-c - - 59552 8 00:57 39900008 Arc-Thar 12 B 13.5 19- 0 a-b-c 5.40 - 59552 9 01:04 39900009 Arc-Une 40 B 13.5 19- 0 a-b-c - - 59552 10 01:15 39900010 Dark 60 A 13.0 ------- a-b-c - - 59552 11 09:24 39900011 Arc---- 30 A 13.0 19- 0 a-b-c - - FPI ------------------------------------------------------------------------------ ---- Miscellaneous ---- import numpy import sys ArcSecPerDeg = 3600 exptime = 15 # fliswarm talk -c gfa expose 60 # offset guide az,alt,rot /computed def slewCmd(raDeg, decDeg, f): f.write("tcc track %.2f, %.2f ICRS/RotAngle=0/RotType=object\n"%(raDeg, decDeg)) def exposeCmd(f): f.write("fliswarm talk -c gfa expose %.2f\n"%exptime) def offsetCmd(f, ra=False, dec=False, rot=False): dra = 0 ddec = 0 drot = 0 if ra: dra = numpy.random.uniform(-15, 15) / ArcSecPerDeg if dec: ddec = numpy.random.uniform(-15, 15) / ArcSecPerDeg if rot: drot = numpy.random.uniform(-0.2, 0.2) if ra or dec: f.write("tcc offset arc %.6f, %.6f /computed\n"%(dra, ddec)) if rot: f.write("tcc offset guide 0,0,%.6f /computed\n"%(drot)) def genScript(raDeg, decDeg, nIter): scriptName = "small_dither_%.4f_%.4f_%i.txt"%(raDeg, decDeg, nIter) f = open(scriptName, "w") slewCmd(raDeg, decDeg, f) exposeCmd(f) for ii in range(nIter): offsetCmd(f, ra=True) exposeCmd(f) slewCmd(raDeg, decDeg,f) exposeCmd(f) offsetCmd(f, dec=True) exposeCmd(f) slewCmd(raDeg, decDeg,f) exposeCmd(f) offsetCmd(f, rot=True) exposeCmd(f) slewCmd(raDeg, decDeg,f) exposeCmd(f) offsetCmd(f, ra=True, dec=True, rot=True) exposeCmd(f) slewCmd(raDeg, decDeg,f) exposeCmd(f) f.close() if __name__ == "__main__": ra = float(sys.argv[1]) dec = float(sys.argv[2]) nIter = int(sys.argv[3]) genScript(ra, dec, nIter) ---- Problems in Detail ---- 1:41:05 .boss 0 boss w text="failed to start DAQ sp1daq, disconnecting myself, [Failure instance: Traceback: : [Errno 113] No route to host" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/bossICC/main/python/bossICC/Commands/TopCmd.py:233:connectControllers" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/actorcore/main/python/actorcore/ICC.py:106:attachController" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/bossICC/main/python/bossICC/Controllers/DAQController.py:36:start" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/bossICC/main/python/bossICC/Controllers/DAQController.py:80:reset" 01:41:05 .boss 0 boss w RawLine="--- ---" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/bossICC/main/python/bossICC/Controllers/DAQController.py:67:reset" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/bossICC/main/python/bossICC/DAQInterface.py:204:__init__" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/software/bossICC/main/python/bossICC/DAQInterface.py:237:reconnect" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/opt/pyenv/versions/3.9.7/lib/python3.9/socket.py:844:create_connection" 01:41:05 .boss 0 boss w RawLine="/home/sdss5/opt/pyenv/versions/3.9.7/lib/python3.9/socket.py:832:create_connection" 01:41:05 .boss 0 boss w RawLine="]\""