Subject: 2.5m obslog 20220104 (SJD 59584) From: kgrabowski@apo.nmsu.edu Date: Wed, 5 Jan 2022 13:01:28 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday January 04, 2022 (SJD 59584) ---- Observing Team ---- Kaike Pan Katie Grabowski Online: Jose Sanchez-Gallego Zoom: Dylan, Karen, Dmitry, Dan, Audrey ---- Observing Plan ---- If clear, by priority: 1. when Jose is present, work on guiding investigation at non-zero PA. Jose will provide more detailed instructions 2. choose an APOGEE field which will be at low airmass for several hours from https://data.sdss5.org/sas/sdsswork/target/commissioning_design/fps_position/fps_position_fields_10/designs.html a. after an FVC loop, on the undithered robot position, take a detector AB dither pair with APOGEE, 42 reads b. at this undithered position, take a series of exposures with focus offsets -200, -100, 0, 100, 200 from the best focus position, single apogee A exposure, 30 reads c. do a robot dither 0.0, FVC no-apply, then an APOGEE 30 read exposure d. do 12 sets of dither 0.1, FVC no-apply, APOGEE 30 read exposure 3. choose a BOSS field which will be at low airmass for several hours from https://data.sdss5.org/sas/sdsswork/target/commissioning_design/fps_position/fps_position_fields_10/designs.html a. after an FVC loop, on the undithered robot position, take a series of exposures with focus offsets -200, -100, 0, 100, 200 from the best focus position, 300 sec b. do 24 sets of dither 0.2, BOSS 300s exposures (no FVC loop) 4. At some point during the night, when array us fully folded, take an FVC loop with --no-apply. It will fail but that's fine; please add the FVC image number to the log. ---- Observing Summary ---- Variations of APOGEE and BOSS robot dithers and focus sweeps, science exposures with focus offsets Variable wind, coming out of the north second half, seeing ~1-2.3" array is fully folded, take an FVC loop with --no-apply: Finished processing /data/fcam/59584/fimg-fvc1n-0023.fits" 06:59:17 jaeger i fvc_rms=707.8 Time, robot failures, snapshot # =============================== 00:58:41 - 372, image #1 06:33:21 - 187,1163 #31 07:55:30 - 184 image #43 #43 also shows 1174 and 663 plus 673 and 201 kaiju-collided after unwind --force, 1290,1251 collided with 1281 at home, snapshot #49 For Jose's test: PA=0, gcam images=13-22, best RMS=0.07 PA=60, gcam images=23-41, best RMS=0.85-1 PA=90, gcam images=43-56, best RMS=1-1.6 PA=120 gcam images=57-67, best RMS=1.3 PA=180 gcam images=68-74, best RMS=1.5 with cherno --no-apply: PA=180 gcam images=75-100, best RMS=1.8 Seeing ranged from 1.1-1.2" With 3" offset in Dec: PA=180, gcam images=106-123, best RMS=1.8 see log body for more details Notes: ======= For BOSS goTo-field, the petals don't open after the flat is taken and reading out. The error we see is "Timed out waiting for shutter to close on sp1mech." Log Function doesn't work in STUI versions 1.17.14 or 15 Dylan loaded a new Log Function. The BOSS widget shows all the information in version 15 ---- Observing Log ---- Afternoon: ========== Jose is working on Jaeger actor, others are working fine. Took apogee evening cal, Took a set of boss bias, dark, flat and arc Boss collimation info: 3:06:06 Hartmann collimate output on cart #19 sp1: offset: r= 989 (In Focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 1812 sp1: pred. resid: r= -823, b= -2.6, txt= OK, spTemp = 5.1 take a pair of boss flat and arc. 22:20Z: power on fps -gfas, set gfa temperature to -20C. 23:22:10 OBSERVER.kpan 84 jaeger : text="All positioners are folded." 23:22:10 CmdDone: OBSERVER.kpan 84 jaeger unwind. 23:25Z: Started ventilation. Night: ======== Opened to clear skies at 00:20Z. jaeger configuration load 35883 "Configuration 1278 loaded." configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59584/configuration_snapshot_59584_1278.pdf 00:58:41 OBSERVER.kpan.jaeger 677 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: START_TRAJECTORY failed" 00:58:41 CmdDone: OBSERVER.kpan.jaeger 677 jaeger configuration execute 00:58:41 .jaeger 0 jaeger f text="A collision was detected in positioner 372. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 00:58:45 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59584/fps_snapshot_59584_0001.pdf 00:58:45 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59584/fps_snapshot_59584_0001.pdf (JaegerUserWarning)" Positioner is pink, position is currently ok, but possible collision anticipated. fliswarm reconnect before taking gfa images to make sure the nightly directory is created Using the GFA script requires Volumes/data to be connected. Best focus 385, observing at 350 Jose's test: tcc track 180.5,24.7 /rota=135.4 /rotty=mount tcc axis stop jaeger fvc loop --no-apply FCAM image #1: fvc_filename=/data/fcam/59584/fimg-fvc1n-0001.fits 01:32:27 APO.Jose: For the guider test, I added a new command \"jaeger testing fake-configuration RA DEC PA\" It will output the configuration_loaded keyword with the input RA DEC PA and you can slew there. It won't move the robots but we don't need that for this test. 01:33:27 APO.Jose: What I want is to go to a field at low airmass and start with PA=0 and try to guide. Then at the same RA/Dec change PA to 60, 90, 120, and 180 and try to guide there, and see how the RMS behaves. jaeger testing fake-configuration 15 20 0 jaeger configuration slew 1:42:09 OBSERVER.kpan 160 cherno d guide_rms=8,218.043,6608.596,6612.192 01:42:51 OBSERVER.kpan 160 cherno d guide_rms=9,246.56,6411.33,6416.07 I think it was missing one element in the keyword. Very large fitting offsets, so we used cherno stop Jose thinks it was missing one element in the keyword, and the GFAs weren't seeing the fake configuration. Altitude approximately 70 degrees PA=0, Rotator angle 143.2 /mount 01:50:20 OBSERVER.kpan 170 cherno d guide_rms=13,1.09,0.759,1.328 01:51:08 OBSERVER.kpan 170 cherno d guide_rms=14,0.48,0.501,0.694 01:51:56 OBSERVER.kpan 170 cherno d guide_rms=15,0.362,0.33,0.49 01:52:43 OBSERVER.kpan 170 cherno d guide_rms=16,0.287,0.273,0.396 01:53:30 OBSERVER.kpan 170 cherno d guide_rms=17,0.039,0.061,0.072 01:54:22 OBSERVER.kpan 170 cherno d guide_rms=18,0.034,0.065,0.073 Changed exp time to 15 seconds from 30: cherno set exposure-time 15 01:55:10 OBSERVER.kpan 170 cherno d guide_rms=19,0.037,0.061,0.071 01:55:39 OBSERVER.kpan 170 cherno d guide_rms=20,0.04,0.06,0.072 01:56:12 OBSERVER.kpan 170 cherno d guide_rms=21,0.041,0.055,0.069 01:56:41 OBSERVER.kpan 170 cherno d guide_rms=22,0.042,0.057,0.071 PA=60, Rotator angle 199.8 /mount jaeger testing fake-configuration 15 20 60, slew manually, and start guiding tcc track 15,20 icrs /rota=60 /rotty=obj 01:59:38 OBSERVER.kpan 190 cherno d guide_rms=23,1.374,1.064,1.738 02:00:08 OBSERVER.kpan 190 cherno d guide_rms=24,0.791,0.308,0.849 02:00:44 OBSERVER.kpan 190 cherno d guide_rms=25,0.637,0.563,0.85 02:01:18 OBSERVER.kpan 190 cherno d guide_rms=26,0.207,0.837,0.862 02:01:52 OBSERVER.kpan 190 cherno d guide_rms=27,0.856,0.254,0.893 02:02:21 OBSERVER.kpan 190 cherno d guide_rms=28,0.646,0.625,0.899 2:02:54 OBSERVER.kpan 190 cherno d guide_rms=29,0.264,0.874,0.913 02:03:28 OBSERVER.kpan 190 cherno d guide_rms=30,0.919,0.216,0.944 02:03:57 OBSERVER.kpan 190 cherno d guide_rms=31,0.63,0.736,0.969 Tried reducing the PID from 1 to 0.7 02:04:31 OBSERVER.kpan 190 cherno d guide_rms=32,0.348,0.943,1.005 02:05:05 OBSERVER.kpan 190 cherno d guide_rms=33,1.03,0.124,1.037 02:05:34 OBSERVER.kpan 190 cherno d guide_rms=34,1.017,0.094,1.022 02:06:03 OBSERVER.kpan 190 cherno d guide_rms=35,1.029,0.14,1.039 02:06:36 OBSERVER.kpan 190 cherno d guide_rms=36,0.608,0.817,1.019 02:07:09 OBSERVER.kpan 190 cherno d guide_rms=37,0.419,0.96,1.047 02:07:43 OBSERVER.kpan 190 cherno d guide_rms=38,1.07,0.074,1.072 02:08:12 OBSERVER.kpan 190 cherno d guide_rms=39,1.047,0.07,1.05 02:08:41 OBSERVER.kpan 190 cherno d guide_rms=40,1.068,0.071,1.07 02:09:09 OBSERVER.kpan 190 cherno d guide_rms=41,1.064,0.029,1.065 02:01:49 OBSERVER.kpan 190 cherno w text="Camera gfa2: not enough sources." PA=90, rotator angle=223.5/mount Cherno stop and restart on new field Jose mentioned if you start guiding with --full it ignores the PID loop, it always uses k=1 2:10:50 OBSERVER.kpan 190 cherno d guide_rms=45,1.39,0.975,1.698 02:11:19 OBSERVER.kpan 205 cherno d guide_rms=46,1.348,0.828,1.582 02:11:50 OBSERVER.kpan 205 cherno d guide_rms=47,0.975,0.478,1.086 02:12:19 OBSERVER.kpan 205 cherno d guide_rms=48,1.464,0.563,1.569 02:12:49 OBSERVER.kpan 205 cherno d guide_rms=49,0.883,2.075,2.255 02:13:19 OBSERVER.kpan 205 cherno d guide_rms=50,1.265,3.001,3.256 02:13:54 OBSERVER.kpan 205 cherno d guide_rms=51,4.369,1.709,4.692 02:14:24 OBSERVER.kpan 205 cherno d guide_rms=52,6.13,2.787,6.734 02:15:00 OBSERVER.kpan 205 cherno d guide_rms=53,3.24,9.133,9.691 02:15:35 OBSERVER.kpan 205 cherno d guide_rms=54,6.139,12.503,13.928 02:16:11 OBSERVER.kpan 205 cherno d guide_rms=55,19.115,6.213,20.1 02:16:39 OBSERVER.kpan 205 cherno d guide_rms=56,19.184,6.231,20.17 02:16:08 OBSERVER.kpan 205 cherno w text="Camera gfa2: not enough sources." PA=120, Rotator angle=251.6 /mount 02:18:33 OBSERVER.kpan 215 cherno d guide_rms=57,1.227,1.223,1.732 02:19:04 OBSERVER.kpan 215 cherno d guide_rms=58,1.08,0.678,1.275 02:19:39 OBSERVER.kpan 215 cherno d guide_rms=59,2.265,0.097,2.267 02:20:09 OBSERVER.kpan 215 cherno d guide_rms=60,3.506,1.888,3.982 02:20:40 OBSERVER.kpan 215 cherno d guide_rms=61,3.626,5.983,6.996 02:21:16 OBSERVER.kpan 215 cherno d guide_rms=62,0.182,12.376,12.378 02:21:45 OBSERVER.kpan 215 cherno d guide_rms=63,0.152,12.435,12.435 02:22:15 OBSERVER.kpan 215 cherno d guide_rms=64,10.868,18.844,21.753 02:22:46 OBSERVER.kpan 215 cherno d guide_rms=65,10.851,18.898,21.792 02:23:22 OBSERVER.kpan 215 cherno d guide_rms=66,33.398,18.95,38.399 02:23:52 OBSERVER.kpan 215 cherno d guide_rms=67,33.703,18.957,38.669 PA=180, Rotator angle=-50.4 /mount 02:27:24 OBSERVER.kpan 225 cherno d guide_rms=68,1.466,0.971,1.758 02:27:55 OBSERVER.kpan 225 cherno d guide_rms=69,0.867,1.165,1.452 02:28:29 OBSERVER.kpan 225 cherno d guide_rms=70,1.692,2.374,2.915 02:29:04 OBSERVER.kpan 225 cherno d guide_rms=71,3.404,4.854,5.928 02:29:39 OBSERVER.kpan 225 cherno d guide_rms=72,7.022,9.869,12.112 02:30:14 OBSERVER.kpan 225 cherno d guide_rms=73,14.283,20.176,24.72 02:30:50 OBSERVER.kpan 225 cherno d guide_rms=74,29.028,41.003,50.238 Tried cherno acquire --no-apply 02:32:05 OBSERVER.kpan 231 cherno d guide_rms=75,1.48,0.974,1.772 02:32:30 OBSERVER.kpan 231 cherno d guide_rms=76,1.475,0.971,1.766 02:32:54 OBSERVER.kpan 231 cherno d guide_rms=77,1.475,0.979,1.771 02:32:30 OBSERVER.kpan 231 cherno d guide_rms=76,1.475,0.971,1.766 02:32:54 OBSERVER.kpan 231 cherno d guide_rms=77,1.475,0.979,1.771 02:33:19 OBSERVER.kpan 231 cherno d guide_rms=78,1.477,0.974,1.769 02:33:43 OBSERVER.kpan 231 cherno d guide_rms=79,1.476,0.98,1.771 02:34:08 OBSERVER.kpan 231 cherno d guide_rms=80,1.479,0.98,1.774 02:34:32 OBSERVER.kpan 231 cherno d guide_rms=81,1.455,1.193,1.881 02:34:57 OBSERVER.kpan 231 cherno d guide_rms=82,1.468,0.977,1.763 02:35:21 OBSERVER.kpan 231 cherno d guide_rms=83,1.467,0.977,1.762 02:35:46 OBSERVER.kpan 231 cherno d guide_rms=84,1.468,0.977,1.764 02:36:11 OBSERVER.kpan 231 cherno d guide_rms=85,1.475,0.985,1.773 02:36:35 OBSERVER.kpan 231 cherno d guide_rms=86,1.281,1.361,1.869 02:37:00 OBSERVER.kpan 231 cherno d guide_rms=87,1.469,0.978,1.765 02:37:24 OBSERVER.kpan 231 cherno d guide_rms=88,1.468,0.978,1.764 02:37:49 OBSERVER.kpan 231 cherno d guide_rms=89,1.469,0.983,1.768 2:38:14 OBSERVER.kpan 231 cherno d guide_rms=90,1.466,0.977,1.762 02:38:38 OBSERVER.kpan 231 cherno d guide_rms=91,1.474,0.979,1.77 02:39:02 OBSERVER.kpan 231 cherno d guide_rms=92,1.459,1.193,1.884 02:39:27 OBSERVER.kpan 231 cherno d guide_rms=93,1.473,0.985,1.772 2:39:51 OBSERVER.kpan 231 cherno d guide_rms=94,1.457,1.196,1.885 02:40:16 OBSERVER.kpan 231 cherno d guide_rms=95,1.455,1.201,1.886 02:40:40 OBSERVER.kpan 231 cherno d guide_rms=96,1.454,1.198,1.884 02:41:05 OBSERVER.kpan 231 cherno d guide_rms=97,1.284,1.372,1.88 02:41:29 OBSERVER.kpan 231 cherno d guide_rms=98,1.476,0.981,1.772 02:41:54 OBSERVER.kpan 231 cherno d guide_rms=99,1.455,1.199,1.885 02:42:19 OBSERVER.kpan 231 cherno d guide_rms=100,1.46,1.199,1.889 Jose thinks the problem may be with astrometry.net or the fit, and should be solvable. It's possible that the corrections have the opposite sign. It's kind of removing the measured error, at least in rotation. It's finding a reasonable astrometric solution. Maybe astrometry.net doesn't constrain rotation that well. It's a problem with the coordinate transformations or the camera rotation. Altitude approx 65 degrees. Manual offset of 3 arcsec in declination and see if it measures it. 02:43:09 OBSERVER.kpan 231 cherno d guide_rms=102,114.623,547.357,559.23 02:43:34 OBSERVER.kpan 231 cherno d guide_rms=103,99.677,862.991,868.728 02:43:59 OBSERVER.kpan 231 cherno d guide_rms=104,125.538,773.875,783.991 Applied 3 degrees instead 3 arcseconds: tcc offset guide 0.0, 0.0, 0.0023888888888888887 /computed 02:44:47 OBSERVER.kpan 231 cherno d guide_rms=106,1.475,1.199,1.901 02:45:11 OBSERVER.kpan 231 cherno d guide_rms=107,1.488,0.987,1.786 02:45:36 OBSERVER.kpan 231 cherno d guide_rms=108,1.33,1.382,1.918 02:46:00 OBSERVER.kpan 231 cherno d guide_rms=109,1.531,0.991,1.824 02:46:25 OBSERVER.kpan 231 cherno d guide_rms=110,1.468,1.168,1.876 02:46:49 OBSERVER.kpan 231 cherno d guide_rms=111,1.433,1.005,1.75 02:47:13 OBSERVER.kpan 231 cherno d guide_rms=112,1.431,1.004,1.748 02:47:37 OBSERVER.kpan 231 cherno d guide_rms=113,1.439,1.012,1.759 02:48:02 OBSERVER.kpan 231 cherno d guide_rms=114,1.186,1.416,1.847 02:48:26 OBSERVER.kpan 231 cherno d guide_rms=115,1.439,1.006,1.756 02:48:51 OBSERVER.kpan 231 cherno d guide_rms=116,1.439,0.999,1.752 02:49:15 OBSERVER.kpan 231 cherno d guide_rms=117,1.444,1.003,1.758 02:49:39 OBSERVER.kpan 231 cherno d guide_rms=118,1.443,1.003,1.757 02:50:04 OBSERVER.kpan 231 cherno d guide_rms=119,1.442,1.011,1.761 02:50:28 OBSERVER.kpan 231 cherno d guide_rms=120,1.442,1.011,1.762 02:50:53 OBSERVER.kpan 231 cherno d guide_rms=121,1.448,1.002,1.761 02:51:18 OBSERVER.kpan 231 cherno d guide_rms=122,1.45,1.011,1.767 02:51:43 OBSERVER.kpan 231 cherno d guide_rms=123,1.452,1.009,1.768 02:47:10 OBSERVER.kpan 231 cherno w text="Camera gfa6: not enough sources." arc offset 3" in dec It went from 14.059,19.262 to 13.709,16.521 in RA,Dec The rotation also changed a bit, but the fit didn't fully isolate the Dec offset. Configuration load 35913, Design ID FVC image=#2 snapshot #2 Config Loaded: 35913-1279 ========================== configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59584/configuration_snapshot_59584_1279.pdf 02:57:40 OBSERVER.kpan 259 jaeger : text="Configuration 1279 loaded." jaeger configuration execute tcc track 75.0, 20.0 icrs /rottype=object/rotang=0/rotwrap=mid hal goto-field -s slew,reconfigure,fvc,acquire,guide,cleanup 02:59:56 OBSERVER.kpan 261 hal i text="Reconfiguring FPS array." 02:59:56 OBSERVER.kpan 261 hal w text="FPS is not folded. Unwinding." Command jaeger configuration explode 5 better to use goto-field without reconfigure, after executing robot moving No issues starting FVC loop 03:03:23 OBSERVER.kpan.jaeger 694 jaeger i text="[FVC]: RMS full fit 52.500 um." 03:03:23 OBSERVER.kpan.jaeger 694 jaeger i fvc_rms=52.5 03:04:08 OBSERVER.kpan.jaeger 694 jaeger i fvc_rms=25.48 03:04:08 OBSERVER.kpan.jaeger 694 jaeger i fvc_deltarms=27.02 RMS full fit 671.290 um Iteration 3: 03:04:51 OBSERVER.kpan.jaeger 694 jaeger i fvc_rms=22.64 03:04:51 OBSERVER.kpan.jaeger 694 jaeger i fvc_deltarms=2.84 Iteration 4: 03:05:33 OBSERVER.kpan.jaeger 694 jaeger i fvc_rms=23.04 03:05:33 OBSERVER.kpan.jaeger 694 jaeger i fvc_deltarms=-0.4 03:05:33 OBSERVER.kpan.jaeger 694 jaeger i text="Delta RMS reached. RMS target has not been reached." Reslewed, started another FVC loop: Image #6 03:07:00 FVC Loop Finished. RMS: 4.89 Delta RMS: -0.40 03:07:00 OBSERVER.kpan 265 jaeger i text="[FVC]: Max. valid metrology distance: 0.022" 03:07:00 OBSERVER.kpan 265 jaeger i text="[FVC]: RMS full fit 4.890 um." SDSS-Obs2 monitors died, logged user out. Left one crashed, the right one flickered in pixel chunks, and then died. Focus sweep Best focus:204, guiding at 190 03:25:10 apogee.expose=apRaw-40220010, 42, A, Object 03:34:12 apogee.expose=apRaw-40220011, 42, B, Object Focus sweep: cherno stop loop tcc focus, fliswarm talk -c gfa expose 30 Best focus=170 tcc set focus=-30 apogee expose Object=object nreads=30 03:54:30 apogee.expose=apRaw-40220012, 30, A, Object tcc set focus=70 apogee expose Object=object nreads=30 03:59:58 apogee.expose=apRaw-40220013, 30, A, Object Wasn't guiding. Restarting with cherno running Also had the apogee shutter closed ---------------- tcc set focus=-30 apogee expose Object=object nreads=30 04:11:23 apogee.expose=apRaw-40220014, 30, A, Object tcc set focus=70 apogee expose Object=object nreads=30 04:16:51 apogee.expose=apRaw-40220015, 30, A, Object tcc set focus=170 apogee expose Object=object nreads=30 04:22:18 apogee.expose=apRaw-40220016, 30, A, Object tcc set focus=270 apogee expose Object=object nreads=30 04:27:46 apogee.expose=apRaw-40220017, 30, A, Object tcc set focus=370 apogee expose Object=object nreads=30 04:33:14 apogee.expose=apRaw-40220018, 30, A, Object Do robot dither 0.0, FVC no-apply, then an APOGEE 30 read exposure jaeger configuration dither 0.0 Creating dithered configuration from 1279. 04:40:20 Config Loaded: 35913-1280 fps_snapshot_59584_0006.pdf fimg-fvc1n-0007.fits fvc_rms=55.04 set focus=170 04:42:46 apogee.expose=apRaw-40220019, 30, A, Object Do 12 sets of dither 0.1, FVC no-apply, APOGEE 30 read exposure 04:52:06 Config Loaded: 35913-1281 fvc_rms=55.24, fimg-fvc1n-0008.fits 04:55:58 apogee.expose=apRaw-40220020, 30, A, Object fvc_rms=54.94, fimg-fvc1n-0009.fits 05:01:49 Config Loaded: 35913-1282 05:04:35 apogee.expose=apRaw-40220021, 30, A, Object 05:10:37 Config Loaded: 35913-1283 fvc_rms=54.42, fimg-fvc1n-0010.fits 05:14:03 apogee.expose=apRaw-40220022, 30, A, Object 05:19:51 Config Loaded: 35913-1284 fvc_rms=53.37, fimg-fvc1n-0011.fits 05:22:04 apogee.expose=apRaw-40220023, 30, A, Object 05:27:55 Config Loaded: 35913-1285 fvc_rms=52.67, fimg-fvc1n-0012.fits 05:29:50 apogee.expose=apRaw-40220024, 30, A, Object 5:35:25 Config Loaded: 35913-1286 fvc_rms=51.58, fimg-fvc1n-0013.fits 05:37:19 apogee.expose=apRaw-40220025, 30, A, Object 05:43:02 Config Loaded: 35913-1287 fvc_rms=50.5, fimg-fvc1n-0014.fits missed an exposure, still tracking 05:51:10 OBSERVER.kpan 377 jaeger f error="Failed processing image: The exposure succeeded but did not output the filename." 05:51:10 CmdDone: OBSERVER.kpan 377 jaeger fvc loop --no-apply 05:50:36 Config Loaded: 35913-1289 fvc_rms=50.91, fimg-fvc1n-0016.fits 05:55:41 apogee.expose=apRaw-40220026, 30, A, Object 06:01:15 Config Loaded: 35913-1291 fvc_rms=53.81, fimg-fvc1n-0017.fits 6:03:03 apogee.expose=apRaw-40220027, 30, A, Object 06:08:40 Config Loaded: 35913-1292 fvc_rms=54.31, fimg-fvc1n-0018.fits 06:11:02 apogee.expose=apRaw-40220028, 30, A, Object 06:16:49 Config Loaded: 35913-1293 fvc_rms=54.77, fimg-fvc1n-0019.fits 06:18:37 apogee.expose=apRaw-40220029, 30, A, Object 06:24:17 Config Loaded: 35913-1294 fvc_rms=53.41, fimg-fvc1n-0020.fits 06:27:08 apogee.expose=apRaw-40220030, 30, A, Object jaeger unwind 06:33:21 OBSERVER.kpan 430 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." /fps_snapshot_59584_0031.pdf jaeger unwind failed 187 and 1163 are kaiju-collided in snapshot 31, possibly also 1025 in the way dithered one last time 06:35:43 Config Loaded: 35913-1295 fvc_rms=52.07, fimg-fvc1n-0021.fits 06:38:41 apogee.expose=apRaw-40220031, 30, A, Object Shutter was closed prematurely. 06:46:14 Config Loaded: 35913-1296 fvc_rms=51.59, fimg-fvc1n-0022.fits 06:48:41 apogee.expose=apRaw-40220032, 30, A, Object unwound with no collisions array is fully folded, take an FVC loop with --no-apply [FVC]: Finished processing /data/fcam/59584/fimg-fvc1n-0023.fits" 06:59:17 OBSERVER.kpan 460 jaeger i fvc_rms=707.8 Took APOGEE dome flat at correct telescope position, robots at home position 07:07:10 apogee.expose=apRaw-40220033, 5, A, DomeFlat jaeger configuration load 35954 07:03:23 Config Loaded: 35954-1297 hal goto-field -s prepare,slew,reconfigure,boss_hartmann,fvc,boss_arcs,boss_flat,acquire,guide,cleanup After 4 iterations, /data/fcam/59584/fimg-fvc1n-0027.fits 07:17:26 OBSERVER.kpan.jaeger 717 jaeger i fvc_rms=5.67 07:17:26 OBSERVER.kpan.jaeger 717 jaeger i fvc_deltarms=-0.06 07:17:26 OBSERVER.kpan.jaeger 717 jaeger i text="Delta RMS reached. RMS target has not been reached." 7:13:32 boss exposure 4.0, file=335918 07:14:13 boss exposure 4.0, file=335919 07:14:54 sp1.motor.move= -770, -773, -774 07:14:54 Hartmann collimate output on cart #19 sp1: offset: r= 0 (In Focus); b= 4.8 (Out of focus) sp1: pred. move: spAverageMove= -771 sp1: pred. resid: r= 771, b= 2.4, txt= OK, spTemp = 2.0 07:18:18 boss exposure 4.0, file=335920 07:19:51 boss exposure 25.0, file=335921 07:18:27 OBSERVER.kpan.boss 721 boss w text="Timed out waiting for shutter to close on sp1mech." Goto-Field stopped here, didn't acquire or open petals. focus sweep - best focus=300, seeing ~2.1" a. after an FVC loop, on the undithered robot position, take a series of exposures with focus offsets -200, -100, 0, 100, 200 from the best focus position, 300 sec tcc set focus=100 07:40:01 boss exposure 300.0, file=335922 Started guiding 07:42:56 OBSERVER.kpan 504 cherno d guide_rms=454,0.525,0.225,0.571 07:44:20 OBSERVER.kpan 504 cherno w text="Camera gfa3: not enough sources." 7:47:52 OBSERVER.kpan 504 cherno d guide_rms=461,2.403,0.202,2.411 07:48:44 OBSERVER.kpan 504 cherno d guide_rms=462,0.032,0.498,0.499 camera_solution=gfa4,462,F,-999.0,-999.0,-999.0,-999.0,-999.0 Changing fields Visually, there should be enough round stars, but it's having trouble finding solutions 07:54:30 Config Loaded: 35948-1298 07:55:30 Jaeger collision: 184, snapshot #43 tried multiple explode 10s, but subsequent snapshots (44-48) don't identify which positioners are troublesome. 08:01:42 OBSERVER.kpan 520 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 3 deadlocks were found" unwind --force 1290 is perpendicular to 1281, which is homed. Also 1251 is at an angle (beta 180) 08:10:47 InvalidKeywordData=jaeger.positioner_status, ['1290', '-0.0049', '86.9469', '0x8db067e1', 'T', 'F', 'F', 'F', '04.01.21', '1', '2', '48'] 08:10:54 cerebro.tron 7720 fliswarm w text=gfa1,"no cameras connected." 08:11:13 InvalidKeywordData=jaeger.positioner_status, ['1251', '326.295', '169.9995', '0x8db067e1', 'T', 'F', 'F', 'F', '04.01.21', '1', '3', '48'] jaeger explode --one 1251 20 jaeger explode -- one 1290 20 jaeger unwind 08:19:00 Config Loaded: 35948-1299 08:22:46 boss exposure 4.0, file=335923 08:23:27 boss exposure 4.0, file=335924 8:24:08 sp1.motor.move= -775, -777, -777 08:24:08 Hartmann collimate output on cart #19 sp1: offset: r= -329 (In Focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -775 sp1: pred. resid: r= 446, b= 1.4, txt= OK, spTemp = 2.2 08:27:32 boss exposure 4.0, file=335925 08:29:06 boss exposure 25.0, file=335926 fimg-fvc1n-0028.fits, fvc_rms=53.4 Iteration 2: fvc_rms=11.74 Iteration 3: fvc_rms=10.38 Iteration 4: 08:26:41 OBSERVER.kpan.jaeger 757 jaeger i fvc_rms=10.14 08:26:41 OBSERVER.kpan.jaeger 757 jaeger i fvc_deltarms=0.24 08:26:41 OBSERVER.kpan.jaeger 757 jaeger i text="Delta RMS reached. RMS target has not been reached." Repeat fvc-loop: fimg-fvc1n-0032.fits. fvc_rms=25.62 Iteration 2: fvc_rms=8.01 Iteration 3: fvc_rms=6.62 Iteration 4: fvc_rms=6.95 08:34:25 OBSERVER.kpan 538 jaeger i fvc_deltarms=-0.33 08:34:25 OBSERVER.kpan 538 jaeger i text="Delta RMS reached. RMS target has not been reached." focus sweep, best=225 tcc set focus=25 08:45:51 boss exposure 300.0, file=335927 tcc set focus=125 08:52:19 boss exposure 300.0, file=335928 tcc set focus=225 08:58:55 boss exposure 300.0, file=335929 tcc set focus=325 09:05:27 boss exposure 300.0, file=335930 tcc set focus=425 09:12:30 boss exposure 300.0, file=335931 Do 24 sets of dither 0.2, BOSS 300s exposures (no FVC loop): - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 09:19:46 Config Loaded: 35948-1300 09:20:18 boss exposure 300.0, file=335932 09:26:45 Config Loaded: 35948-1301 09:27:18 boss exposure 300.0, file=335933 09:33:44 Config Loaded: 35948-1302 09:34:18 boss exposure 300.0, file=335934 09:40:44 Config Loaded: 35948-1303 09:41:19 boss exposure 300.0, file=335935 09:47:45 Config Loaded: 35948-1304 09:48:18 boss exposure 300.0, file=335936 09:54:45 Config Loaded: 35948-1305 09:55:18 boss exposure 300.0, file=335937 10:01:45 Config Loaded: 35948-1306 10:02:18 boss exposure 300.0, file=335938 10:08:45 Config Loaded: 35948-1307 10:09:18 boss exposure 300.0, file=335939 10:15:44 Config Loaded: 35948-1308 10:16:18 boss exposure 300.0, file=335940 10:22:45 Config Loaded: 35948-1309 10:23:19 boss exposure 300.0, file=335941 10:29:44 Config Loaded: 35948-1310 10:30:19 boss exposure 300.0, file=335942 10:36:44 Config Loaded: 35948-1311 10:37:18 boss exposure 300.0, file=335943 10:43:45 Config Loaded: 35948-1312 10:44:19 boss exposure 300.0, file=335944 10:50:45 Config Loaded: 35948-1313 10:51:18 boss exposure 300.0, file=335945 10:57:45 Config Loaded: 35948-1314 10:58:18 boss exposure 300.0, file=335946 11:04:45 Config Loaded: 35948-1315 11:05:19 boss exposure 300.0, file=335947 11:11:44 Config Loaded: 35948-1316 11:12:19 boss exposure 300.0, file=335948 11:18:44 Config Loaded: 35948-1317 11:19:18 boss exposure 300.0, file=335949 11:25:46 Config Loaded: 35948-1318 11:26:19 boss exposure 300.0, file=335950 11:32:44 Config Loaded: 35948-1319 11:33:19 boss exposure 300.0, file=335951 11:39:45 Config Loaded: 35948-1320 11:40:18 boss exposure 300.0, file=335952 11:46:45 Config Loaded: 35948-1321 11:47:19 boss exposure 300.0, file=335953 11:53:45 Config Loaded: 35948-1322 11:54:19 boss exposure 300.0, file=335954 12:00:45 Config Loaded: 35948-1323 12:01:19 boss exposure 300.0, file=335955 Those we already have the requisite number, we have more time before the night is over, but not enough time to start on a different field, so we will continue with this one. 12:09:39 boss exposure 300.0, file=335956 12:16:06 Config Loaded: 35948-1325 12:16:37 boss exposure 300.0, file=335957 12:23:03 Config Loaded: 35948-1326 12:23:35 boss exposure 300.0, file=335958 2:30:01 Config Loaded: 35948-1327 12:30:33 boss exposure 300.0, file=335959 12:36:59 Config Loaded: 35948-1328 12:37:33 boss exposure 300.0, file=335960 12:53:02 encl25m : closed; Started APOGEE and BOSS calibrations. 12:56:17 apogee.expose=apRaw-40220034, 60, A, Dark 12:56:55 boss exposure 0.0, file=335961 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0030 0.0040 Axial B: 0.0020 0.0030 Axial C: -0.1150 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 3.50 Altitude: 29.9 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6670 -9.6710 Axial C: -9.6960 -9.6970 Trans D: 0.1150 0.1140 GALILS Commanded: 4891200 4889000 4907900 0 -3400 Actual: 4859318 4859255 4859148 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.01 0.04 -0.05 -0.03 41.62 ---- Telescope Status ---- 23:25Z Started ventilation 00:20Z encl25m : open 12:53Z encl25m : closed No interlocks bypassed Status at: 12:58Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 685 686 704 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 149 lbs, 23.1 psi ---- Software Used ---- get Versions, 01/04/22, 23:13:08 -- STUI software: STUI : 1.7.14 alerts : 2.2.0 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : FAILED guider : 4.0.0 ecamera : FAILED fliswarm : 0.1.1-alpha.0 jaeger : 0.13.1-alpha.0 kaiju : 1.2.2-alpha.0 coordio : 1.2.1-alpha.0 fps_calibrati : APO.2021.12.25 cherno : 0.1.0-alpha.0 hal : 0.1.0-alpha.0 hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : nosop -- done -- ---- APOGEE Data Summary ---- APOGEE data list, mjd=59584, all raw_data: /data/apogee/utr_cdr/59584/apRaw-*.fits archive: /data/apogee/archive/59584/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59584/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59584 1 22:56 40220001 Dark 3 A 13.0 -------- a-b-c - - 59584 2 23:07 40220002 Dark 60 A 13.0 -------- a-b-c - - 59584 3 23:09 40220003 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 100 59584 4 23:11 40220004 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.50 - 59584 5 23:19 40220005 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59584 6 23:21 40220006 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.12 - 59584 7 23:28 40220007 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59584 8 23:39 40220008 Dark 60 A 13.0 -------- a-b-c - - 59584 9 23:53 40220009 Arc---- 30 A 13.0 FPS- 0 a-b-c - - FPI 59584 10 03:32 40220010 Object 42 A 13.0 FPS- 0 a-b-c -0.11 - 59584 11 03:41 40220011 Object 42 B 13.5 FPS- 0 a-b-c 0.42 - 59584 12 03:59 40220012 Object 30 A 13.0 FPS- 0 a-b-c 0.27 - 59584 13 04:04 40220013 Object 30 A 13.0 FPS- 0 a-b-c -0.41 - 59584 14 04:16 40220014 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 15 04:21 40220015 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 16 04:27 40220016 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 17 04:32 40220017 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 18 04:38 40220018 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 19 04:47 40220019 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 20 05:00 40220020 Object 30 A 13.0 FPS- 0 a-b-c -0.11 - 59584 21 05:09 40220021 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 22 05:18 40220022 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 23 05:27 40220023 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 24 05:34 40220024 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 25 05:42 40220025 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 26 06:00 40220026 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 27 06:07 40220027 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 28 06:15 40220028 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 29 06:23 40220029 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 30 06:32 40220030 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 31 06:43 40220031 Object 30 A 13.0 FPS- 0 a-b-c -0.46 - 59584 32 06:53 40220032 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59584 33 07:07 40220033 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 127 ------------------------------------------------------------------------------