Subject: 2.5m obslog 20211230 (SJD 59579) From: dgatlin@apo.nmsu.edu Date: Fri, 31 Dec 2021 11:35:13 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday December 30, 2021 (SJD 59579) ---- Observing Team ---- Katie Grabowski Dylan Gatlin ---- Observing Plan ---- Because we're closed, we'll run a couple fvc loop --no-apply's on dithers at different fps designs ---- Observing Summary ---- We were closed all night and received about 2" of snow. Roads are slushy and foggy. We did some dither tests at various dither sizes. Notes: - Most fvc loops taken tonight exited before they hit an rms of 5. Repeated tries usually turned out better. - Design 35922 seems to be unreachable for fiber 608, it seems to get stuck in the same spot each time at 348.1447, 88.6302. It looks like the alpha arm can't quite make it to its 360deg limit, and this is a rare case where the design asks it to go too far. - Many of the jaeger unwinds had deadlocks in them, more than I previous remember - There's a significant amount of snow in the enclosure by the west wall. Not sure where it comes in from. Fortunately it was very sticky, so I was able to make some snow balls and carry them outside, but there's still some left, all snow is on an absorbant pad. ---- Observing Log ---- Afternoon: =========== All systems alive and responsive. 22:45- Started APOGEE evening cals 22:45:35 boss exposure 0.0, file=335848 22:47:19 boss exposure 20.0, file=335849 First set of hartmanns didn't have the lights turned on: 22:50:14 boss exposure 4.0, file=335850 22:50:54 boss exposure 4.0, file=335851 22:52:49 boss exposure 4.0, file=335852 22:53:30 boss exposure 4.0, file=335853 22:54:17 sp1.motor.move= 2142, 2142, 2142 22:54:17 Hartmann collimate output on cart #19 sp1: offset: r= 1649 (Out of focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 2142 sp1: pred. resid: r= -493, b= -1.5, txt= OK, spTemp = 2.0 22:55:12 boss exposure 30.0, file=335854 23:01:10 boss exposure 4.0, file=335855 Delayed ventilation due to radar, snow flurries, and thick clouds. Absorbent pads are down in the enclosure. Night: ====== Started a set of dithered exposures on an fps configuration 00:57:55 CmdStarted: APO.dgatlin 3 jaeger configuration load 35915 00:59:16 CmdStarted: APO.dgatlin 6 jaeger configuration execute 01:04:59 CmdStarted: APO.dgatlin 7 jaeger fvc loop Final RMS: 3.2 @ snapshot 4 Config 908 Snapshot 5, fvc 4 First dither happened at snapshot 5 1 (5) --- 01:08:49 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (5) --- 01:10:54 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (5) --- 01:12:59 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (5) --- 01:15:04 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (5) --- 01:17:08 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply # New field 01:29:37 CmdStarted: APO.dgatlin 33 jaeger configuration load 35918 01:31:10 CmdStarted: APO.dgatlin 34 jaeger configuration execute 01:32:08 CmdStarted: APO.dgatlin 35 jaeger fvc loop Final RMS: 6.3 @ snapshot 21 # The FVC loop took 3 tries, first it was in the 30s, then it was 5.89, then # up to 6.3 Config 914 Snapshot 22, fvc 22 1 (5) --- 01:41:19 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (5) --- 01:43:23 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (5) --- 01:45:28 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (5) --- 01:47:31 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (5) --- 01:49:35 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply # While trying to unwind from this field, I had some difficulty with the # following error 01:54:08 APO.dgatlin 67 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 6 deadlocks were found" 01:54:08 CmdDone: APO.dgatlin 67 jaeger unwind # I tried exploding up to 20 and still had the same error, the issue with it is # that I had a hard time finding the deadlocks. It looks like the same thing # happened twice in the same unwind, fibers P1109 and P1331 were pinched between # a properly homed fiber above and a stuck fiber below. The solution that worked # here was a jaeger goto 90 90 in both cases, then an explode --one on the bottom # then an unwind # New Field 02:12:07 CmdStarted: APO.dgatlin 82 jaeger configuration load 35920 02:14:37 CmdStarted: APO.dgatlin 84 jaeger configuration execute 02:16:17 CmdStarted: APO.dgatlin 85 jaeger fvc loop Final RMS: 6.1 # Got a weird fvc loop error 02:19:51 CmdDone: cerebro.tron 182 fliswarm talk status 02:19:51 APO.dgatlin 86 jaeger f error="Failed processing image: The exposure succeeded but did not output the filename." 02:19:51 CmdDone: APO.dgatlin 86 jaeger fvc loop 1 (5) --- 02:22:31 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (5) --- 02:24:35 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (5) --- 02:26:40 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (5) --- 02:28:46 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (5) --- 02:30:52 jaeger configuration dither 0.1 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply # Same field, higher dither size 02:36:49 CmdStarted: APO.dgatlin 109 jaeger fvc loop Final RMS: 5.48 1 (5) --- 02:38:32 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (5) --- 02:40:36 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (5) --- 02:42:42 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (5) --- 02:44:47 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (5) --- 02:46:52 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply # New Field 03:01:28 CmdStarted: APO.dgatlin 131 jaeger configuration load 35922 # Had a collision for P0608. Unwound and re-executed multiple times to no avail. # The "collision" kept happening at position 348.1447, 88.6302, giving up and # trying a new design # Had some trouble unwinding due to a fiber getting pinched again 03:31:51 CmdStarted: APO.dgatlin 163 jaeger configuration load 35924 03:33:43 CmdStarted: APO.dgatlin 165 jaeger configuration execute 03:38:13 CmdStarted: APO.dgatlin 168 jaeger fvc loop Final RMS: 4.51 snapshot 85, fvc 67 1 (5) --- 03:40:22 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (5) --- 03:42:27 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (5) --- 03:44:32 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (5) --- 03:46:36 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (5) --- 03:48:42 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply We decided to revisit the deadlock with design 35922 and fiber 608. We started by moving a bunch of fibers out of the way, then moving it around with jaeger goto commands. After some debugging, it looks like the beta arm can move comfortably between 170 and 10, but alpha gets stuck around 349, which is where the design was sending it. # New Field 05:19:36 CmdStarted: APO.dgatlin 273 jaeger configuration load 35926 05:22:50 CmdStarted: APO.dgatlin 274 jaeger configuration execute 05:24:31 CmdStarted: APO.dgatlin 275 jaeger fvc loop Final RMS: 3.31 Config 943 Snapshot 152, fvc 81 1 (10) --- 05:31:58 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (10) --- 05:34:03 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (10) --- 05:36:07 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (10) --- 05:38:13 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (10) --- 05:40:18 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 6 (10) --- 05:42:24 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 7 (10) --- 05:44:29 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 8 (10) --- 05:46:35 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 9 (10) --- 05:48:41 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 10 (10) --- 05:50:46 jaeger configuration dither 0.15 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 06:17:17 CmdStarted: APO.dgatlin 318 jaeger configuration load 35928 06:18:09 CmdStarted: APO.dgatlin 319 jaeger configuration execute It took 3 fvc loops, but the fit stopped improving at 9.14 Config 954 Snapshot 179, fvc 110 1 (10) --- 06:49:08 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (10) --- 06:51:19 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (10) --- 06:53:24 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (10) --- 06:55:29 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (10) --- 06:57:34 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 6 (10) --- 06:59:42 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 7 (10) --- 07:01:48 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 8 (10) --- 07:03:53 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 9 (10) --- 07:05:59 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 10 (10) --- 07:08:05 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply # New Field 07:49:46 CmdStarted: APO.dgatlin 373 jaeger configuration load 35930 07:50:59 CmdStarted: APO.dgatlin 374 jaeger configuration execute 07:52:24 CmdStarted: APO.dgatlin 375 jaeger fvc loop Final RMS: 6.73, Config 964 Started at snapshot 208, fvc 136 1 (10) --- 07:58:55 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (10) --- 08:01:00 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (10) --- 08:03:06 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (10) --- 08:05:12 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (10) --- 08:07:18 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 6 (10) --- 08:09:23 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 7 (10) --- 08:11:30 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 8 (10) --- 08:13:35 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 9 (10) --- 08:15:41 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 10 (10) --- 08:17:46 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply # New Field 08:29:14 CmdStarted: APO.dgatlin 426 jaeger configuration load 35932 08:31:52 CmdStarted: APO.dgatlin 427 jaeger configuration execute Final RMS: , Configuration 975 snapshot 234, fvc 161 1 (10) --- 08:38:05 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 2 (10) --- 08:40:12 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 3 (10) --- 08:42:18 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 4 (10) --- 08:44:23 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 5 (10) --- 08:46:29 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 6 (10) --- 08:50:05 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 7 (10) --- 08:52:11 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 8 (10) --- 08:54:17 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 9 (10) --- 08:56:23 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 10 (10) --- 08:58:29 jaeger configuration dither 0.2 jaeger fvc loop --no-apply jaeger configuration reverse tui wait 5 jaeger fvc loop --no-apply 09:09:02 CmdStarted: APO.dgatlin 471 jaeger configuration load 35934 09:19:21 CmdStarted: APO.dgatlin 472 jaeger configuration execute Kept getting errors for fiber 664. Despite my best efforts, I couldn't find anything wrong with the fibers motion using goto commands. We could reliably repeat the error. Weirdly, the snapshot doesn't show anything wrong at fiber 664 10:25:42 .jaeger 0 jaeger f text="A collision was detected in positioner 664. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 11:00Z At this point we decided to conclude tests for the night and pack up, morning cals started ---- BOSS Spectrographs Collimation Summary ---- 22:52:49 boss exposure 4.0, file=335852 22:53:30 boss exposure 4.0, file=335853 22:54:17 sp1.motor.move= 2142, 2142, 2142 22:54:17 Hartmann collimate output on cart #19 sp1: offset: r= 1649 (Out of focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 2142 sp1: pred. resid: r= -493, b= -1.5, txt= OK, spTemp = 2.0 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0020 0.0040 Axial B: 0.0020 0.0030 Axial C: -0.1110 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: -1.67 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6570 -9.6570 Axial B: -9.6710 -9.6710 Axial C: -9.6950 -9.6970 Trans D: 0.1160 0.1140 GALILS Commanded: 4898450 4900250 4913850 0 -3400 Actual: 4859041 4859324 4859091 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.95 0.05 0.07 0.05 41.61 ---- Telescope Status ---- Started ventilation encl25m : open encl25m : closed No interlocks bypassed Status at: 1:00:39Z Telescope stowed at: 121.0, 30.0, -0.0 mount Instrument mounted: ECam Counterweights at: 685.5, 687.3, 705.2, 704.6 LN2 autofill systems: Connected and turned on 180L LN2 dewar scale: SP1 210.0 lbs, 13.6 psi ---- Software Used ---- /dev/sdb1 2.0T 154G 1.9T 8% /home 10.25.1.230:/data 21T 17T 3.1T 85% /data -- STUI software: STUI : 1.7.13 alerts : 2.2.0 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : FAILED hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 sop : FAILED tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.9 --------------Other Versions-------------- sdss-obstools : 3.7.6 TPM : 1.2-31-gd7eb MCP : 2.1 idlspec2d : v6_0_4