Subject: 2.5m obslog 20211206 (SJD 59555) From: kinemuchi@apo.nmsu.edu Date: Tue, 7 Dec 2021 12:06:50 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday December 06, 2021 (SJD 59555) ---- Observing Team ---- Katie Grabowski Karen Kinemuchi On Zoom: Viktor Malanushenko Kaike Pan Audrey and Dan Oravetz Dylan Gatlin Dmitry Bizyaev FPS COMMISSIONING: Rick Pogge Conor Sayres Jose Sanchez-Gallego Juna Kollmeier ---- Observing Plan ---- FPS Commissioning ---- Observing Summary ---- Took FPS BOSS and APOGEE cals Test jaeger configurations GFA field tests Night ended early due to thick cloud cover after 07:00Z NOTES FOR DAY STAFF AND DEVELOPERS: =================================== * Dust counts ~9000 * A fuse was blown on the IEB and the FPS lost power briefly this evening. See log for details. ---- Observing Log ---- Afternoon: ========== All systems alive and responsive. Restarted podium laptop ~ 10GB 22:42:31 boss exposure 0.0, file=335239 22:47:23 boss exposure 20.0, file=335240 Noticed the FPS cooling loop was about 3Lpm instead of 6Lpm. Optimal rate around 7Lpm. Rick moved it up to about 9, then to 7Lpm Pressure still ok,although slightly higher than normal. About 45psi versus 35psi. 22:51Z- Started APOGEE cals Critical alert: apogeeql heartbeat while taking the first two arcs. Doesn't ping back. JohnD et al. were contacted and this alert has been with us for the past few days. This alert will trigger during data collection with APOGEE and will end when the data are done. 23:30:49 boss exposure 25.0, file=335241 23:32:58 boss exposure 4.0, file=335242 (FF lamps on, not arclamps) Without the cart and plate numbers, the BOSS calibrations aren't on the SOS page. 23:30Z- started ventilation NIGHT: ====== Started up FPS and began cooling at 00:15Z. 00:22:24 encl25m : open; Taking an APOGEE dome flat through the FPS: 00:45:47 apogee.expose=apRaw-39930011, 5, A, DomeFlat We are not seeing any light through the fibers. 01:13:52 apogee.expose=apRaw-39930012, 5, A, DomeFlat 01:17:48 apogee.expose=apRaw-39930013, 5, A, DomeFlat 01:19:24 apogee.expose=apRaw-39930014, 5, A, DomeFlat Forgot to open the cold shutter to APOGEE. To take a dome flat with APOGEE: ================================ close FF petals turn on FF lamp open cold shutter take a dome flat through the APOGEE instrument gui: set to 5 reads and manually turn on lamp on read #3. check on ds9. 1:31:43 apogee.expose=apRaw-39930016, 5, A, DomeFlat 01:49Z Taking 20s darks, some flats, and a bias with the GFAs file type time comments 0001 darks 15sx10 0002 dark 30sx10 0003 dark 60x5 0004 flat 1s 16k counts 0005 flat 5x4s 65K too bright 0006 flat 5x3s good flat 0007 dark 5x3s dark/bias for flats 02:11Z Conor modified the FPS instrument block for rot_inst_xy_angle = -89.586000 and we have updated the instrument block. 02:25Z Taking a BOSS science exposure exptime = 900s. Telescope position is (60,60,60). Just to get some blank sky data, sky lines. Some accidental stars may appear in the exposure. 02:24:03 boss exposure 900.0, file=335244 Jose's test: 02:48:11 jaeger configuration load 50 02:49:28 jaeger configuration execute Going to field: 02:48:58 tcc track 30,20 icrs /rota=0 /rotty=obj Had a collision. Error message: 02:49:31 OBSERVER.kgrabow 110 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: START_TRAJECTORY failed" 02:49:31 CmdDone: OBSERVER.kgrabow 110 jaeger configuration execute 02:49:31 .jaeger 0 jaeger f text="A collision was detected in positioner 189. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 02:51:00 jaeger unlock 02:51:12 jaeger configuration execute 02:52:38 jaeger fvc loop error message: 2:52:57 OBSERVER.kgrabow 113 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'KeyError: \"None of ['fibre_type'] are in the columns\"'. See traceback in the log for more information." 02:52:57 CmdDone: OBSERVER.kgrabow 113 jaeger fvc loop 03:02:39 jaeger unwind 03:11:07 jaeger configuration load --from-positions 03:11:51 jaeger fvc loop error message: 03:12:10 OBSERVER.kgrabow 118 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'ValueError: attempt to get argmin of an empty sequence'. See traceback in the log for more information." 03:14:22 jaeger fvc loop --exposure-time 5 --fbi-level 5 --no-apply to adjust the level of the flux to be more visible in the metrology fibers. setting exposure time to 5s Output from jaeger: 03:14:43 OBSERVER.kgrabow 119 jaeger i text="[FVC]: Max counts in image: 48586" 03:14:44 OBSERVER.kgrabow 119 jaeger i text="[FVC]: Found 559 centroids" 03:14:44 OBSERVER.kgrabow 119 jaeger i text="[FVC]: Expected 560 centroids" 03:14:45 OBSERVER.kgrabow 119 jaeger i text="[FVC]: Max fiducial rough distance: 54.734" 03:18:55 jaeger fvc loop --exposure-time 5 --fbi-level 10 --no-apply Output from jaeger: 03:19:17 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Processing raw image /data/fcam/59555/fimg-fvc1n-0005.fits" 03:19:17 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Max counts in image: 65535" 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Found 560 centroids" 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Expected 560 centroids" 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Max fiducial rough distance: 70.025" 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Full transform 1. Bisased RMS=27725.546, Unbiased RMS=37519.244." 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Max fiducial fit 2 distance: 56.633" 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Full transform 2. Bisased RMS=16099.771, Unbiased RMS=18281.713." 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Max metrology distance: 37.308" 03:19:19 OBSERVER.kgrabow 120 jaeger i text="[FVC]: RMS full fit 10385.960 um." 03:19:19 OBSERVER.kgrabow 120 jaeger i fvc_rms=10385.9600002 03:19:20 OBSERVER.kgrabow 120 jaeger i text="[FVC]: Calculating offset from FVC image and fit." RMS values are bad here. Checking the field for the correct sense of the rotation. Take an image of a Holtz field. 03:21:55 tcc track 30,20 icrs /rota=0 /rotty=obj 03:23:47 tcc track 30,20 icrs /rota=135.4 03:25:56 tcc track 30,20 icrs /rota=135.4 /rotty=mount 03:27:42 jaeger fvc loop --exposure-time 5 --fbi-level 10 --no-apply Change fbi to 8% 03:29:03 jaeger fvc loop --exposure-time 5 --fbi-level 8 --no-apply Scale fbi down to 6% 03:30:37 jaeger fvc loop --exposure-time 5 --fbi-level 6 --no-apply 03:35:43 jaeger fvc loop --exposure-time 5 --fbi-level 6 Slewing back to field: 03:37:57 tcc track 30,20 icrs /rota=0 /rotty=obj Check if the computer is connected: 03:38:19 fliswarm status 03:38:38 fliswarm -c gfa talk status Taking an image of the field with the GFAs for Conor: 03:40:20 fliswarm talk -c gfa expose 60 Under image #8 Focusing telescope: 03:46:00 tcc set focus=300 taking more data with telescope focused: 03:48:26 fliswarm talk -c gfa expose 60 To find the filename, look for key word filename_bundle: filename_bundle=/data/gcam/59555/gimg-gfa6n-0009.fits,/data/gcam/59555/gimg-gfa3n-0009.fits,/data/gcam/59555/gimg-gfa2n-0009.fits,/data/gcam/59555/gimg-gfa4n-0009.fits,/data/gcam/59555/gimg-gfa5n-0009.fits 03:52:58 tcc set focus=350 03:53:16 fliswarm talk -c gfa expose 60 filename_bundle=/data/gcam/59555/gimg-gfa4n-0010.fits,/data/gcam/59555/gimg-gfa5n-0010.fits,/data/gcam/59555/gimg-gfa6n-0010.fits,/data/gcam/59555/gimg-gfa2n-0010.fits,/data/gcam/59555/gimg-gfa3n-0010.fits Best focus seems to be at 300. 03:56:47 tcc set focus=300 Going back to rotator position 135.4: 03:57:14 tcc track 30,20 icrs /rota=135.4 /rotty=mount Jaeger is being restarted. hub startNubs jaeger Loading a configuration: 03:58:33 jaeger configuration load --from-positions 03:59:55 jaeger fvc loop --fbi-level 5 --exposure-time 5 04:26Z Jose is debugging. 04:49:45 jaeger unwind 04:50:00 jaeger configuration random --danger 04:53:43 jaeger fvc loop --fbi-level 5 --exposure-time 5 Error message: 04:54:08 .jaeger 0 jaeger w text="[FVC]: Failed to calculate corrected positioner coordinates for 5 positioners." 04:54:08 OBSERVER.kgrabow 147 jaeger w text="[FVC]: Failed to calculate corrected positioner coordinates for 5 positioners." 04:54:12 .hub 0 cmds d CmdQueued=281096,1638852851.94,"cerebro.tron",4745,"fliswarm",4689,"talk status" 04:54:13 OBSERVER.kgrabow 147 jaeger f error="Failed processing image: Cannot apply corrections. 2 robots are collided." 04:56:10 jaeger configuration load --from-positions Error message: 04:56:14 OBSERVER.kgrabow 148 jaeger f error="Failed generating paths: field_center must be passed to Field" 05:00:00 jaeger explode 5 05:00:21 jaeger configuration load --from-positions 05:02:41 jaeger unwind 05:03:19 jaeger explode 5 05:03:38 jaeger unwind 04:58:47 OBSERVER.kgrabow 150 jaeger f error="Failed generating paths: Grid is Kaiju-collided, cannot generate paths" In this case, we force an unwind: 05:04:32 jaeger unwind --force Check with the snapshot Nudge one of the robots, but first find where the offending robot is: 05:09:16 CmdStarted: OBSERVER.kgrabow 157 jaeger status 944 05:09:16 CmdStarted: OBSERVER.kgrabow 157 jaeger status 944 05:09:16 OBSERVER.kgrabow 157 jaeger > 05:09:16 OBSERVER.kgrabow 157 jaeger i locked=F 05:09:16 OBSERVER.kgrabow 157 jaeger i n_positioners=500 05:09:16 OBSERVER.kgrabow 157 jaeger i fps_status=0x11 05:09:17 OBSERVER.kgrabow 157 jaeger i positioner_status=944,1.4999,152.9653,0x8db067e1,T,F,F,F,04.01.21,6,3,16 05:09:17 OBSERVER.kgrabow 157 jaeger i fbi_led=0.0,0.0,0.0,0.0; flow=7.311,2.165; humidity=22.584,41.969; power_can=T,T,T,T,T,T; power_gfa=F,T,T,T,T,T; power_nuc=T,T,T,T,T,T; power_sextant=T,T,T,T,T,T; power_sync=F; pressure=45.617,22.586; temperature=13.678,3.937,14.75,18.92,18.72,10.72,5.72,5.52,24.92,9.602,9.82,10.01,12.169; voltage=24.112,24.323,24.19,23.987,24.151,24.058 To nudge the offending robot: 05:10:46 jaeger goto 944 1.4999 120 05:11:37 jaeger unwind 05:12:14 jaeger configuration random --danger 05:14:16 OBSERVER.kgrabow 161 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: Failed sending SEND_NEW_TRAJECTORY" 05:14:16 CmdDone: OBSERVER.kgrabow 161 jaeger configuration random --danger 05:14Z FPS lost power and we have gone down to investigate. 05:25Z Power cycled the IEB, nothing came back. 05:34Z Called Ryan to go check if we can flip the breaker circuit that looked like it tripped and also check southeast rack. 06:06Z A fuse was blown in the IEB. Rick replaced it and restored power to the FPS. 06:12Z Jose restarted jaeger and everything is coming back. Turning on the FPS and GFAs. fliswarm reconnect is not working. pinging the nucs. Issued fliswarm status command fliswarm talk status command failed fliswarm reconnect --force Error: 06:16:29 OBSERVER.kgrabow 178 fliswarm f error="Command 'reconnect --force' failed because of an uncaught error 'OSError: [Errno 113] No route to host'. See traceback in the log for more information." 06:16:29 CmdDone: OBSERVER.kgrabow 178 fliswarm reconnect --force 06:17:17 fliswarm reconnect 06:17:33 fliswarm talk status 06:17:44 fliswarm disable gfa1 06:17:46 fliswarm reconnect 06:18:08 fliswarm talk status 06:18:26 fliswarm -c gfa temperature -20 06:20Z Waiting for the GFAs to cool down, then do Conor's test. 06:21:33 tcc track 60,20 icrs /rota=0 /rotty=mount Take an image of one of Holtzman's fields: 06:26:12 fliswarm talk -c gfa expose 60 filename_bundle=/data/gcam/59555/gimg-gfa3n-0011.fits,/data/gcam/59555/gimg-gfa6n-0011.fits,/data/gcam/59555/gimg-gfa5n-0011.fits,/data/gcam/59555/gimg-gfa4n-0011.fits,/data/gcam/59555/gimg-gfa2n-0011.fits Don't see any stars. Trying again 06:33:59 fliswarm talk -c gfa expose 60 filename_bundle=/data/gcam/59555/gimg-gfa3n-0012.fits,/data/gcam/59555/gimg-gfa6n-0012.fits,/data/gcam/59555/gimg-gfa5n-0012.fits,/data/gcam/59555/gimg-gfa4n-0012.fits,/data/gcam/59555/gimg-gfa2n-0012.fits Stars appear to be smeared. Selected a field with a lower altitude. Still seeing streaked stars. 06:46:36 tcc track 45,20 icrs /rota=0 /rotty=obj 06:48:08 fliswarm talk -c gfa expose 60 filename_bundle=/data/gcam/59555/gimg-gfa3n-0015.fits,/data/gcam/59555/gimg-gfa6n-0015.fits,/data/gcam/59555/gimg-gfa5n-0015.fits,/data/gcam/59555/gimg-gfa4n-0015.fits,/data/gcam/59555/gimg-gfa2n-0015.fits ok, smeared stars are gone. However, the clouds are coming in. 07:00:58 tcc track 30,20 icrs /rota=0 /rotty=obj 07:01:39 fliswarm talk -c gfa expose 60 filename_bundle=/data/gcam/59555/gimg-gfa6n-0016.fits,/data/gcam/59555/gimg-gfa3n-0016.fits,/data/gcam/59555/gimg-gfa4n-0016.fits,/data/gcam/59555/gimg-gfa5n-0016.fits,/data/gcam/59555/gimg-gfa2n-0016.fits 07:02:44 fliswarm talk -c gfa expose 60 07:04:46 tcc track 75,20 icrs /rota=0 /rotty=mount Trying to find stars. 07:05:59 fliswarm talk -c gfa expose 60 filename_bundle=/data/gcam/59555/gimg-gfa3n-0017.fits,/data/gcam/59555/gimg-gfa6n-0017.fits,/data/gcam/59555/gimg-gfa5n-0017.fits,/data/gcam/59555/gimg-gfa4n-0017.fits,/data/gcam/59555/gimg-gfa2n-0017.fits 07:09Z Giving up. Getting too cloudy. 07:16:24 encl25m : closed; 11:49:44 jaeger power off 12:05Z Started APOGEE non-dithered morning cals. APOGEE instrument team - please let us know if you want dithered or non-dithered cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0050 0.0040 Axial B: 0.0040 0.0030 Axial C: -0.1170 -0.1150 Trans D: -0.0341 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5345 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 5.94 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6540 -9.6570 Axial B: -9.6660 -9.6710 Axial C: -9.6970 -9.6970 Trans D: 0.1140 0.1140 GALILS Commanded: 4887750 4884350 4903050 0 -3400 Actual: 4859167 4859060 4859243 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.95 -0.05 -0.03 -0.02 41.67 ---- Telescope Status ---- 23:30Z Started ventilation 00:22Z encl25m : open 07:16Z encl25m : closed No interlocks bypassed Status at: 12:04Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 191 lbs, 23.4 psi ---- Software Used ---- Get Versions, 12/06/21, 23:00:45 -- STUI software: STUI : 1.7.13 alerts : 2.1.6 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : FAILED hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 sop : 0.1.0-alpha.0 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.9 Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 0 16G 0% /dev tmpfs 16G 8.0K 16G 1% /dev/shm tmpfs 16G 1.4M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 40G 11G 79% / /dev/sda1 1014M 238M 777M 24% /boot /dev/sdb1 2.0T 203G 1.9T 10% /home tmpfs 3.2G 0 3.2G 0% /run/user/2005 10.25.1.230:/data 21T 17T 3.3T 84% /data overlay 50G 40G 11G 79% /var/lib/docker/overlay2/2fc82a212d7b05767e9dcf9122c0af446f9144f66730227739ab2d409681017f/merged tmpfs 3.2G 4.0K 3.2G 1% /run/user/3021 tmpfs 3.2G 0 3.2G 0% /run/user/1002 ---- APOGEE Data Summary ---- APOGEE data list, mjd=59555, all raw_data: /data/apogee/utr_cdr/59555/apRaw-*.fits archive: /data/apogee/archive/59555/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59555/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59555 1 22:40 39930001 Dark 3 A 13.0 ------- a-b-c - - 59555 2 23:02 39930002 Dark 60 A 13.0 ------- a-b-c - - 59555 3 23:04 39930003 QuarFlat 10 A 13.0 19- 0 a-b-c - 103 59555 4 23:06 39930004 Arc-Thar 12 A 13.0 19- 0 a-b-c 4.76 - 59555 5 23:13 39930005 Arc-Une 40 A 13.0 19- 0 a-b-c - - 59555 6 23:15 39930006 Arc-Thar 12 B 13.5 19- 0 a-b-c 5.38 - 59555 7 23:23 39930007 Arc-Une 40 B 13.5 19- 0 a-b-c - - 59555 8 23:34 39930008 Dark 60 A 13.0 ------- a-b-c - - 59555 9 23:45 39930009 Arc---- 20 A 13.0 19- 0 a-b-c - - FPI 59555 10 00:45 39930010 DomeFlat 5 A 13.0 19- 0 a-b-c - 0 59555 11 00:46 39930011 DomeFlat 5 A 13.0 19- 0 a-b-c - -1 59555 12 01:14 39930012 DomeFlat 5 A 13.0 19- 0 a-b-c - -1 59555 13 01:18 39930013 DomeFlat 5 A 13.0 19- 0 a-b-c - 0 59555 14 01:19 39930014 DomeFlat 5 A 13.0 19- 0 a-b-c - 0 59555 15 01:23 39930015 DomeFlat 5 A 13.0 19- 0 a-b-c - 0 59555 16 01:32 39930016 DomeFlat 5 A 13.0 19- 0 a-b-c - 239 ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59555 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename crt plt flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:42 b1-00335239 FPS-N None bia Clo 0.0 -2242 -2242 -2256 2 22:42 r1-00335239 FPS-N None bia Clo 0.0 -2242 -2242 -2256 3 22:47 b1-00335240 FPS-N None dar Clo 20.0 -2242 -2242 -2256 4 22:47 r1-00335240 FPS-N None dar Clo 20.0 -2242 -2242 -2256 5 23:30 b1-00335241 FPS-N None fla Out 25.1 -2242 -2242 -2256 6 23:30 r1-00335241 FPS-N None fla Out 25.1 -2242 -2242 -2256 7 23:32 b1-00335242 FPS-N None arc Out 4.1 -2242 -2242 -2256 8 23:32 r1-00335242 FPS-N None arc Out 4.1 -2242 -2242 -2256 9 23:38 b1-00335243 FPS-N None arc Out 4.1 -2242 -2242 -2256 10 23:38 r1-00335243 FPS-N None arc Out 4.1 -2242 -2242 -2256 11 02:24 b1-00335244 FPS-N None sci Out 900.1 -2242 -2242 -2256 12 02:24 r1-00335244 FPS-N None sci Out 900.1 -2242 -2242 -2256