Subject: 2.5m obslog 20220120 (SJD 59600) From: asimmons@apo.nmsu.edu Date: Fri, 21 Jan 2022 12:57:53 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday January 20, 2022 (SJD 59600) ---- Observing Team ---- Kaike Pan Audrey Oravatz José Sánchez-Gallego (support) Conor Sayres (support) ---- Observing Plan ---- Highest priority is to test revised robot software from Conor if he is ready. This can be done if it is cloudy by going to designs as you've done the past few nights, and noting success/failure/etc. If it is clear, you should try to execute some designs and collect data, but as discussed on Tuesday, to improve efficiency, other try a design configuration 3 times before moving on to trying another one. If Conor or Jose is online, they can of course direct as needed. If it is clear, try to get dither data on several APOGEE and several BOSS fields. If possible, repeat one APOGEE and one BOSS from previously observed (APOGEE 16186, 20888, 16196, 20916; BOSS 16187, 20935, 20909, 20941) then attempt to get different designs. Beware of Moon at RA=10 52 12, DEC=12 27.5 If possible, observe one science design, 12831, which has both APOGEE and BOSS targets. You should be able to see visibility in Kronos Plan Observing (if not, enter the design in the new Viz Preview menu feature). It's got a pretty long visibility window, but not quite to the end of the night. ---- Observing Summary ---- We remained closed most of the night due to high humidity and blowing snow, but opened the last couple of hours. While we were closed and with Jose's and Conor's help, we: (1) executed many different configurations and tallied any error messages observed. Out of 50 total configurations, we encountered 7 fields that reported robot collisions (1 additional collisions during FVC loops), 1 failed generating paths during loading, 1 'move failed to start during FVC loop, and 2 VALUE_OUT_OF_RANGE errors. (2) downloaded the newest BOSON version was on both obs1 and obs2 (0.0.4-alpha 18), which has new guiding features within the guide window (PID coefficients, offsets). Looking forwarding to testing it on sky in the coming nights. (3) hard-coded the --scale additive (default is 0.999882), so load configuration XXXX --scale is not needed anymore. The humidity and blowing snow got much better around 11:00Z. We opened and centered on Design 4583, Field 16963 and Design 8110, Field 17818 - science fields with BOSS and APOGEE targets. (1) Tested the HAL window within BOSON. Should the 'tcc track' command be a part of 'Acquire'? The brakes do not always clear after the FVC loop if the entire hal go-field sequence is not run. (2) Oddly, we were unable to guide with more than 1 camera on both fields (see notes in log) (3) We ran into a collision with robot 1136 on the second design during execution. (4) The FVC RMS values consistently seem to be large. The routine fails after this step. (5) The new features within HAL and GUIDER BOSON windows are really helpful! NOTES: ======= * Consistent warning messages during every FVC loop: "Failed to calculate corrected positioner coordinates for 5(+) positioners." RMS settles around 10-50 microns often, an increase from what was seen during previous nights. * BOSON crashed on us once at around 10:06Z. Nothing out of the ordinary. Snapshot window was open at the time. Crashed on both Kaike's and my machine. Needed to be restarted. * jaeger configuration reverse seems to work well. We did not need to use unwind --force like in the past. * Data (in chronological order): -> Config # 35903-2482, failed 2 times - collision with robot 1136 during the forward execution time, and collision robot 608 during the reverse after the FVC loop. -> Config # 35905-2488, failed 3 times altogether: (1) collision with robot 160 during execution 2489, (2) collision with robot 565 during execution, 608/565 during unwind 2490, (3) collision with robot 565 during execution and unwind -> Config # 35906-2492, failed once - collision with robot 608 during the reverse after the FVC loop -> Config # 35925, failed once - 'Failure upon loading, failed generating paths' during the loading phase -> Config # 35934-2507, failed twice DURING the FVC loop - both attempts involving collision with robot 664. Collision with robot 608 during the reverse. -> Config # 35936-2509, failed DURING the first attempt of the FVC loop, without any collisions - 'Failed executing the correction trajectory: Something went wrong during the trajectory: Move failed to start' -> Config # 35944-2517, failed during only the first execution, 'Trajectory failed with error: Something went wrong sending the trajectory: SEND_TRAJECTORY_DATA UID=8734 with 'VALUE_OUT_OF_RANGE'." (time stamp: 07:54Z) -> Config # 35946-2520 through 35946-2522, failed during execution on three attempts - 'Trajectory failed with error: Something went wrong sending the trajectory: positioner 302 replied to SEND_TRAJECTORY_DATA UID=9197 with 'VALUE_OUT_OF_RANGE." (time stamp: 08:09Z) -> Config # 35951-2527, failed during execution on first attempt - collision with robot 664 -> Config # 35961-2538, failed during execution (and once during reverse) all 3 attempts - collision with robot 608 -> Config # 35962-2542, failed twice with robot collision 608 upon execution -> Config # 35963-2544, failed with robot 1136 collision ---- Observing Log ---- Afternoon: ========= Apogee evening cals are taken and checked along with morning cals taken this morning. They are good. BOSS cals are taken and checked. Collimation is checked and confirmed good. 23:32:30 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= -5.2 (Out of focus) sp1: pred. move: spAverageMove= 835 sp1: pred. resid: r= -835, b= -2.6, txt= OK, spTemp = 5.8 MotorPosition sp1: -2872, -2833, -2892; ---------- Median spectrographs temperatures: Tsp1= 5.9 Outside air temperature= -3.4; ---------- All systems and actors are alive. /data disk space is at 95% full. High humidity, delayed evening ventilation. Night: ====== 02:27Z - The humidity is dropping, but we have lots of blowing snow. We will sit tight while Jose and Conor are logged on to run tests with new trajectory code with new random safe configurations. The new code works with a different way how the trajectories are computed. 03:53Z - Filled the cameras and moved the telescope to 70 degrees for an FVC loop test (to minimize flexure). 03:22:00 CmdDone: APO.Jose 64 jaeger configuration load 35903 03:22:45 APO.Jose 66 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 32.33 seconds after start" 03:22:41 Jaeger collision: 1136 03:22:46 CmdDone: APO.Jose 66 jaeger configuration execute 03:23:27 CmdStarted: APO.Jose 70 jaeger unwind 03:23:27 APO.Jose 70 jaeger > 03:23:28 APO.Jose 70 jaeger i folded=F 03:23:28 APO.Jose 70 jaeger d text="Calculating unwind trajectory." 03:23:28 APO.Jose 70 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." 03:23:45 CmdDone: APO.Jose 71 jaeger explode 5 03:23:59 APO.Jose 72 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 3 deadlocks were found ([215, 245, 444])" 03:25:14 CmdStarted: APO.Jose 76 jaeger explode --one 215 15 03:27:05 CmdStarted: APO.Jose 78 jaeger unwind 03:50:35 CmdStarted: APO.Jose 100 jaeger fvc loop 3:51:00 APO.Jose 100 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0001.fits 03:51:42 APO.Jose 100 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0002.fits 03:52:25 APO.Jose 100 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0003.fits 03:53:08 APO.Jose 100 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0004.fits 03:53:50 APO.Jose 100 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0005.fits 03:51:03 APO.Jose 100 jaeger i text="[FVC]: RMS full fit 71.210 um." 03:51:45 APO.Jose 100 jaeger i text="[FVC]: RMS full fit 14.680 um." 03:52:28 APO.Jose 100 jaeger i text="[FVC]: RMS full fit 8.830 um." 03:53:11 APO.Jose 100 jaeger i text="[FVC]: RMS full fit 7.480 um." 03:53:52 APO.Jose 100 jaeger i text="[FVC]: RMS full fit 8.030 um." 03:55:52 APO.Jose 101 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 8.38 seconds after start" 03:55:52 CmdDone: APO.Jose 101 jaeger configuration reverse 03:55:46 Jaeger collision: 608 03:56:33 CmdDone: APO.Jose 104 jaeger explode 5 03:56:34 CmdStarted: APO.Jose 105 jaeger unwind 03:58:00 CmdStarted: APO.Jose 107 jaeger configuration load 35904 04:00:20 APO.Jose 107 jaeger : text="Configuration 2487 loaded." 04:00:55 CmdStarted: APO.Jose 108 jaeger configuration execute 04:01:40 CmdStarted: APO.Jose 109 jaeger fvc loop 4:02:05 APO.Jose 109 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0006.fits 04:02:49 APO.Jose 109 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0007.fits 04:03:32 APO.Jose 109 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0008.fits 04:04:31 APO.Jose 110 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0009.fits 04:05:14 APO.Jose 110 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0010.fits 04:02:08 APO.Jose 109 jaeger i text="[FVC]: RMS full fit 50.010 um." 04:02:51 APO.Jose 109 jaeger i text="[FVC]: RMS full fit 13.100 um." 04:03:35 APO.Jose 109 jaeger i text="[FVC]: RMS full fit 26.490 um." 04:04:34 APO.Jose 110 jaeger i text="[FVC]: RMS full fit 27.530 um." 04:05:17 APO.Jose 110 jaeger i text="[FVC]: RMS full fit 38.530 um." Perhaps the RMS didn't converge because the field is currently at a low altitude (~30 degrees)? Trying again with another field. 04:17:28 CmdDone: OBSERVER.kpan 126 jaeger configuration load 35905 --scale 0.999882 04:20:59 OBSERVER.kpan 131 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 7.25 seconds after start" .jaeger 0 jaeger f text="A collision was detected in positioner 160. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 04:20:59 CmdDone: OBSERVER.kpan 131 jaeger configuration reverse 04:21:23 CmdStarted: OBSERVER.kpan 134 jaeger unwind 04:21:23 OBSERVER.kpan 134 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." 04:22:04 .jaeger 0 jaeger f text="A collision was detected in positioner 160. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 04:22:11 OBSERVER.kpan 135 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 8.38 seconds after start" 04:22:11 CmdDone: OBSERVER.kpan 135 jaeger explode 10 *** Note: We do NOT need to issue the --scale additive to the command anymore since the default value (0.999882 is coded in). *** 04:24:40 OBSERVER.kpan 138 jaeger : text="Configuration 2489 loaded." 04:24:40 CmdDone: OBSERVER.kpan 138 jaeger configuration load 35905 --scale 0.999882 04:24:40 CmdStarted: OBSERVER.kpan 139 jaeger configuration execute 04:25:13 .jaeger 0 jaeger f text="A collision was detected in positioner 565. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 04:25:17 OBSERVER.kpan 139 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 32.22 seconds after start" jaeger unwind 04:40:35 Config Loaded: 35906-2491 04:43:12 FVC Loop - RMS:12.67 Delta RMS: -0.65 Image #: 0013 04:41:48 OBSERVER.kpan 184 jaeger i text="[FVC]: RMS full fit 54.230 um." 04:42:29 OBSERVER.kpan 184 jaeger i text="[FVC]: RMS full fit 12.020 um." 04:43:12 OBSERVER.kpan 184 jaeger i text="[FVC]: RMS full fit 12.670 um." 04:48:25 Config Loaded: 35911-2492 04:51:47 FVC Loop - RMS:28.61 Delta RMS: 0.61 Image #: 0017 4:49:39 OBSERVER.kpan 192 jaeger i text="[FVC]: RMS full fit 60.340 um." 04:50:22 OBSERVER.kpan 192 jaeger i text="[FVC]: RMS full fit 30.280 um." 04:51:05 OBSERVER.kpan 192 jaeger i text="[FVC]: RMS full fit 29.220 um." 04:51:47 OBSERVER.kpan 192 jaeger i text="[FVC]: RMS full fit 28.610 um." 04:52:10 Jaeger collision: 608 04:55:44 Config Loaded: 35912-2493 04:58:37 OBSERVER.kpan 200 jaeger f error="Failed processing image: The exposure succeeded but did not output the filename." 04:58:37 CmdDone: OBSERVER.kpan 200 jaeger fvc loop 04:56:56 OBSERVER.kpan 200 jaeger i text="[FVC]: RMS full fit 72.890 um." 04:57:39 OBSERVER.kpan 200 jaeger i text="[FVC]: RMS full fit 16.950 um." 04:58:21 OBSERVER.kpan 200 jaeger i text="[FVC]: RMS full fit 13.780 um." 05:01:44 OBSERVER.kpan 204 jaeger i text="[FVC]: RMS full fit 14.150 um." 05:02:25 OBSERVER.kpan 204 jaeger i text="[FVC]: RMS full fit 14.720 um." 05:02:25 FVC Loop - RMS:14.72 Delta RMS: -0.57 Image #: 0023 05:05:16 Config Loaded: 35913-2494 05:08:35 FVC Loop - RMS: 6.10 Delta RMS: -0.50 Image #: 0027 05:06:28 OBSERVER.kpan 209 jaeger i text="[FVC]: RMS full fit 50.300 um." 05:07:10 OBSERVER.kpan 209 jaeger i text="[FVC]: RMS full fit 9.340 um." 05:07:51 OBSERVER.kpan 209 jaeger i text="[FVC]: RMS full fit 5.600 um." 05:08:35 OBSERVER.kpan 209 jaeger i text="[FVC]: RMS full fit 6.100 um." 05:10:45 Config Loaded: 35914-2495 05:13:22 FVC Loop - RMS:13.62 Delta RMS: 0.23 Image #: 0030 05:11:59 OBSERVER.kpan 217 jaeger i text="[FVC]: RMS full fit 54.100 um." 05:12:41 OBSERVER.kpan 217 jaeger i text="[FVC]: RMS full fit 13.850 um." 05:13:22 OBSERVER.kpan 217 jaeger i text="[FVC]: RMS full fit 13.620 um." 05:16:04 Config Loaded: 35917-2496 05:17:16 OBSERVER.kpan 222 jaeger i text="[FVC]: RMS full fit 71.670 um." 05:18:00 OBSERVER.kpan 222 jaeger i text="[FVC]: RMS full fit 47.670 um." 05:18:41 OBSERVER.kpan 222 jaeger i text="[FVC]: RMS full fit 45.530 um." 05:19:23 OBSERVER.kpan 222 jaeger i text="[FVC]: RMS full fit 44.110 um." 05:20:04 OBSERVER.kpan 222 jaeger i text="[FVC]: RMS full fit 42.530 um." 05:22:09 Config Loaded: 35918-2497 05:25:25 FVC Loop - RMS:19.94 Delta RMS: -0.01 Image #: 0039 05:23:19 OBSERVER.kpan 227 jaeger i text="[FVC]: RMS full fit 72.180 um." 05:24:02 OBSERVER.kpan 227 jaeger i text="[FVC]: RMS full fit 21.500 um." 05:24:44 OBSERVER.kpan 227 jaeger i text="[FVC]: RMS full fit 19.930 um." 05:25:25 OBSERVER.kpan 227 jaeger i text="[FVC]: RMS full fit 19.940 um." 05:28:17 Config Loaded: 35919-2498 05:32:16 FVC Loop - RMS: 5.26 Delta RMS: 0.76 Image #: 0044 5:29:27 OBSERVER.kpan 232 jaeger i text="[FVC]: RMS full fit 50.520 um." 05:30:12 OBSERVER.kpan 232 jaeger i text="[FVC]: RMS full fit 9.750 um." 05:30:54 OBSERVER.kpan 232 jaeger i text="[FVC]: RMS full fit 7.180 um." 05:31:35 OBSERVER.kpan 232 jaeger i text="[FVC]: RMS full fit 6.020 um." 05:32:16 OBSERVER.kpan 232 jaeger i text="[FVC]: RMS full fit 5.260 um." 05:34:03 Config Loaded: 35920-2499 05:38:04 FVC Loop - RMS: 6.72 Delta RMS: 0.24 Image #: 0049 5:35:16 OBSERVER.kpan 237 jaeger i text="[FVC]: RMS full fit 49.030 um." 05:35:59 OBSERVER.kpan 237 jaeger i text="[FVC]: RMS full fit 10.140 um." 05:36:41 OBSERVER.kpan 237 jaeger i text="[FVC]: RMS full fit 8.290 um." 05:37:22 OBSERVER.kpan 237 jaeger i text="[FVC]: RMS full fit 6.960 um." 05:38:04 OBSERVER.kpan 237 jaeger i text="[FVC]: RMS full fit 6.720 um." 05:43:20 Config Loaded: 35925-2500 05:47:17 FVC Loop - RMS:12.08 Delta RMS: 0.14 Image #: 0054 05:44:30 OBSERVER.kpan 243 jaeger i text="[FVC]: RMS full fit 55.120 um." 05:45:12 OBSERVER.kpan 243 jaeger i text="[FVC]: RMS full fit 16.260 um." 05:45:54 OBSERVER.kpan 243 jaeger i text="[FVC]: RMS full fit 13.420 um." 05:46:35 OBSERVER.kpan 243 jaeger i text="[FVC]: RMS full fit 12.220 um." 05:47:16 OBSERVER.kpan 243 jaeger i text="[FVC]: RMS full fit 12.080 um." 05:39:15 CmdStarted: OBSERVER.kpan 240 jaeger configuration load 35925 05:41:29 OBSERVER.kpan 240 jaeger f error="Failed generating paths: Attempt 6: 2 deadlocks remain but the number of retries has been exhausted" 5:49:04 Config Loaded: 35926-2501 05:51:40 FVC Loop - RMS: 7.36 Delta RMS: 0.75 Image #: 0057 05:50:16 OBSERVER.kpan 249 jaeger i text="[FVC]: RMS full fit 49.020 um." 05:50:58 OBSERVER.kpan 249 jaeger i text="[FVC]: RMS full fit 8.110 um." 05:51:40 OBSERVER.kpan 249 jaeger i text="[FVC]: RMS full fit 7.360 um." 05:53:33 Config Loaded: 35929-2502 05:56:53 FVC Loop - RMS: 5.75 Delta RMS: 0.65 Image #: 0061 5:54:46 OBSERVER.kpan 256 jaeger i text="[FVC]: RMS full fit 52.860 um." 05:55:30 OBSERVER.kpan 256 jaeger i text="[FVC]: RMS full fit 10.600 um." 05:56:12 OBSERVER.kpan 256 jaeger i text="[FVC]: RMS full fit 6.400 um." 05:56:53 OBSERVER.kpan 256 jaeger i text="[FVC]: RMS full fit 5.750 um." 06:06:25 Config Loaded: 35930-2503 06:09:02 FVC Loop - RMS: 8.44 Delta RMS: 0.64 Image #: 0064 06:11:52 Config Loaded: 35931-2504 06:15:10 FVC Loop - RMS:12.21 Delta RMS: -0.55 Image #: 0068 06:17:32 Config Loaded: 35932-2505 06:21:31 FVC Loop - RMS: 4.54 Delta RMS: 0.97 Image #: 0073 06:24:43 Config Loaded: 35933-2506 06:28:00 FVC Loop - RMS: 9.23 Delta RMS: 0.98 Image #: 0077 06:30:19 Config Loaded: 35934-2507 06:31:03 CmdStarted: OBSERVER.kpan 296 jaeger fvc loop 06:31:44 OBSERVER.kpan 296 jaeger f error="Failed processing image: Failed executing the correction trajectory: Something went wrong during the trajectory: Some positioners did not reach their destinations. Trajectory failed 6.95 seconds after start" 06:33:25 .jaeger 0 jaeger f text="A collision was detected in positioner 664. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 06:38:15 CmdDone: OBSERVER.kpan 301 jaeger explode --one 664 0.5 Second attempt: 06:38:28 CmdStarted: OBSERVER.kpan 302 jaeger fvc loop 06:39:49 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0147.pdf 06:39:49 OBSERVER.kpan 302 jaeger f error="Failed processing image: Failed executing the correction trajectory: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 6.79 seconds after start" 06:39:49 CmdDone: OBSERVER.kpan 302 jaeger fvc loop 06:31:31 OBSERVER.kpan 296 jaeger i text="[FVC]: RMS full fit 61.790 um." 06:33:18 OBSERVER.kpan 297 jaeger i text="[FVC]: RMS full fit 16.960 um." 06:38:56 OBSERVER.kpan 302 jaeger i text="[FVC]: RMS full fit 16.410 um." 06:39:37 OBSERVER.kpan 302 jaeger i text="[FVC]: RMS full fit 11.290 um." 06:40:35 CmdStarted: OBSERVER.kpan 305 jaeger configuration reverse 06:40:41 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 06:40:41 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 06:41:35 CmdStarted: OBSERVER.kpan 307 jaeger explode --one 664 0.5 06:41:46 CmdStarted: OBSERVER.kpan 308 jaeger configuration reverse 07:00:42 Config Loaded: 35935-2508 07:01:54 OBSERVER.kpan 312 jaeger i text="[FVC]: RMS full fit 69.580 um." 07:02:36 OBSERVER.kpan 312 jaeger i text="[FVC]: RMS full fit 15.950 um." 07:03:18 OBSERVER.kpan 312 jaeger i text="[FVC]: RMS full fit 14.560 um." 07:04:00 OBSERVER.kpan 312 jaeger i text="[FVC]: RMS full fit 11.930 um." 07:04:42 OBSERVER.kpan 312 jaeger i text="[FVC]: RMS full fit 10.310 um." 7:01:51 OBSERVER.kpan 312 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0082.fits 07:02:33 OBSERVER.kpan 312 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0083.fits 07:03:16 OBSERVER.kpan 312 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0084.fits 07:03:57 OBSERVER.kpan 312 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0085.fits 07:04:39 OBSERVER.kpan 312 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0086.fits 07:06:56 Config Loaded: 35936-2509 Failed during FVC loop: 07:08:20 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0159.pdf 07:08:20 OBSERVER.kpan 317 jaeger f error="Failed processing image: Failed executing the correction trajectory: Something went wrong during the trajectory: Move failed to start. Trajectory failed 6.05 seconds after start" Trying again: 07:08:20 CmdDone: OBSERVER.kpan 317 jaeger fvc loop 07:11:56 FVC Loop - RMS: 6.40 Delta RMS: 0.20 Image #: 0090 07:08:08 OBSERVER.kpan 317 jaeger i text="[FVC]: RMS full fit 49.200 um." 07:10:31 OBSERVER.kpan 318 jaeger i text="[FVC]: RMS full fit 11.210 um." 07:11:14 OBSERVER.kpan 318 jaeger i text="[FVC]: RMS full fit 6.600 um." 07:11:56 OBSERVER.kpan 318 jaeger i text="[FVC]: RMS full fit 6.400 um." 07:14:17 Config Loaded: 35937-2510 07:17:38 FVC Loop - RMS: 9.81 Delta RMS: -1.43 Image #: 0094 07:19:34 Config Loaded: 35938-2511 07:23:37 FVC Loop - RMS: 8.43 Delta RMS: -0.08 Image #: 0099 07:25:57 Config Loaded: 35939-2512 07:29:14 FVC Loop - RMS: 7.80 Delta RMS: 0.89 Image #: 0103 07:32:10 Config Loaded: 35940-2513 07:35:27 FVC Loop - RMS:10.99 Delta RMS: -0.70 Image #: 0107 07:37:46 Config Loaded: 35941-2514 07:41:07 FVC Loop - RMS: 7.78 Delta RMS: -0.28 Image #: 0111 07:43:04 Config Loaded: 35942-2515 07:46:21 FVC Loop - RMS:12.80 Delta RMS: -0.21 Image #: 0115 07:48:39 Config Loaded: 35943-2516 07:51:57 FVC Loop - RMS:10.32 Delta RMS: -0.12 Image #: 0119 **** OUT OF RANGE ERROR HERE **** 07:53:54 Config Loaded: 35944-2517 07:52:56 CmdStarted: OBSERVER.kpan 356 jaeger configuration load 35944 07:53:57 OBSERVER.kpan 356 jaeger : text="Configuration 2517 loaded." configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2517.pdf 07:53:57 CmdStarted: OBSERVER.kpan 357 jaeger configuration execute 07:54:01 .jaeger 0 jaeger w text="Positioner 437 replied to SEND_TRAJECTORY_DATA UID=8734 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 07:54:01 OBSERVER.kpan 357 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 07:54:40 CmdStarted: OBSERVER.kpan 358 jaeger status 437 07:54:40 OBSERVER.kpan 358 jaeger > 07:54:41 OBSERVER.kpan 358 jaeger i positioner_status=437,10.0039,169.9968,0x8db067b1,T,F,F,F,04.01.21,2,2,196 07:54:41 InvalidKeywordData=jaeger.positioner_status, ['437', '10.0039', '169.9968', '0x8db067b1', 'T', 'F', 'F', 'F', '04.01.21', '2', '2', '196'] 07:54:41 OBSERVER.kpan 358 jaeger : Trying again: 07:56:02 CmdStarted: OBSERVER.kpan 359 jaeger configuration load 35944 07:57:00 OBSERVER.kpan 359 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2518.pdf 07:57:00 OBSERVER.kpan 359 jaeger : text="Configuration 2518 loaded." 07:57:13 CmdStarted: OBSERVER.kpan 360 jaeger configuration execute Worked. 07:56:57 Config Loaded: 35944-2518 07:58:33 OBSERVER.kpan 361 jaeger i text="[FVC]: RMS full fit 63.350 um." 07:59:17 OBSERVER.kpan 361 jaeger i text="[FVC]: RMS full fit 26.600 um." 07:59:59 OBSERVER.kpan 361 jaeger i text="[FVC]: RMS full fit 21.680 um." 08:00:40 OBSERVER.kpan 361 jaeger i text="[FVC]: RMS full fit 20.050 um." 08:01:22 OBSERVER.kpan 361 jaeger i text="[FVC]: RMS full fit 17.190 um." 7:58:31 OBSERVER.kpan 361 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0120.fits 07:59:14 OBSERVER.kpan 361 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0121.fits 07:59:56 OBSERVER.kpan 361 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0122.fits 08:00:37 OBSERVER.kpan 361 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0123.fits 08:01:19 OBSERVER.kpan 361 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0124.fits 08:04:11 Config Loaded: 35945-2519 08:08:12 FVC Loop - RMS: 8.03 Delta RMS: 0.40 Image #: 0129 *** OUT OF RANGE HERE *** 08:09:50 Config Loaded: 35946-2520 configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2520.pdf 08:09:52 OBSERVER.kpan 369 jaeger : text="Configuration 2520 loaded." 08:09:52 CmdDone: OBSERVER.kpan 369 jaeger configuration load 35946 08:09:52 CmdStarted: OBSERVER.kpan 370 jaeger configuration execute 08:09:57 .jaeger 0 jaeger w text="Positioner 302 replied to SEND_TRAJECTORY_DATA UID=9174 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 08:09:57 OBSERVER.kpan 370 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 08:10:53 CmdStarted: OBSERVER.kpan 372 jaeger status 302 08:10:53 OBSERVER.kpan 372 jaeger > 08:10:53 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999787,69.999731,0.008301; ObjInstAng=nan,nan,5149440339.2290611; SpiderInstAng=nan,nan,5149440339.2290611 08:10:53 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 08:10:53 .alerts 0 alerts i activeAlerts 08:10:53 .alerts 0 alerts i disabledAlertRules 08:10:53 OBSERVER.kpan 372 jaeger i positioner_status=302,10.0134,169.9953,0x8db067b1,T,F,F,F,04.01.21,4,1,208 08:10:53 InvalidKeywordData=jaeger.positioner_status, ['302', '10.0134', '169.9953', '0x8db067b1', 'T', 'F', 'F', 'F', '04.01.21', '4', '1', '208'] 08:10:53 OBSERVER.kpan 372 jaeger : 08:10:53 CmdDone: OBSERVER.kpan 372 jaeger status 302 Trying again: configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2521.pdf 08:12:08 OBSERVER.kpan 373 jaeger : text="Configuration 2521 loaded." 08:12:08 CmdDone: OBSERVER.kpan 373 jaeger configuration load 35946 08:12:08 CmdStarted: OBSERVER.kpan 374 jaeger configuration execute 08:12:08 OBSERVER.kpan 374 jaeger > 08:12:08 OBSERVER.kpan 374 jaeger i text="Using stored trajectory (from destination)." 08:12:08 OBSERVER.kpan 374 jaeger i text="Sending and executing forward trajectory." 08:12:12 .jaeger 0 jaeger w text="Positioner 302 replied to SEND_TRAJECTORY_DATA UID=9197 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 08:12:12 OBSERVER.kpan 374 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 08:12:12 CmdDone: OBSERVER.kpan 374 jaeger configuration execute 08:13:32 OBSERVER.kpan 376 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2522.pdf 08:13:32 OBSERVER.kpan 376 jaeger : text="Configuration 2522 loaded." 08:13:32 CmdDone: OBSERVER.kpan 376 jaeger configuration load 35946 08:13:32 CmdStarted: OBSERVER.kpan 377 jaeger configuration execute 08:13:32 OBSERVER.kpan 377 jaeger > 08:13:32 OBSERVER.kpan 377 jaeger i text="Using stored trajectory (from destination)." 08:13:32 OBSERVER.kpan 377 jaeger i text="Sending and executing forward trajectory." 08:13:34 .tcc 0 tcc i TCCPos=nan,nan,nan; AxePos=120.999787,69.999734,0.008295; ObjInstAng=nan,nan,5149440339.2290611; SpiderInstAng=nan,nan,5149440339.2290611 08:13:34 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT 08:13:34 .alerts 0 alerts i activeAlerts 08:13:34 .alerts 0 alerts i disabledAlertRules 08:13:37 .jaeger 0 jaeger w text="Positioner 302 replied to SEND_TRAJECTORY_DATA UID=9213 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 08:13:37 OBSERVER.kpan 377 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 08:13:37 CmdDone: OBSERVER.kpan 377 jaeger configuration execute 08:14:34 Config Loaded: 35947-2523 08:18:33 FVC Loop - RMS: 6.80 Delta RMS: -1.65 Image #: 0134 08:20:28 Config Loaded: 35948-2524 08:23:04 FVC Loop - RMS:16.56 Delta RMS: -1.70 Image #: 0137 08:25:45 Config Loaded: 35949-2525 08:29:07 FVC Loop - RMS:11.00 Delta RMS: -0.03 Image #: 0141 08:31:57 Config Loaded: 35950-2526 08:33:09 OBSERVER.kpan 395 jaeger i text="[FVC]: RMS full fit 62.770 um." 08:33:51 OBSERVER.kpan 395 jaeger i text="[FVC]: RMS full fit 33.930 um." 08:34:33 OBSERVER.kpan 395 jaeger i text="[FVC]: RMS full fit 29.100 um." 08:35:15 OBSERVER.kpan 395 jaeger i text="[FVC]: RMS full fit 27.490 um." 08:35:56 OBSERVER.kpan 395 jaeger i text="[FVC]: RMS full fit 26.480 um." 8:33:49 OBSERVER.kpan 395 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0143.fits 08:34:31 OBSERVER.kpan 395 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0144.fits 08:35:12 OBSERVER.kpan 395 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0145.fits 08:35:54 OBSERVER.kpan 395 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0146.fits 08:43:55 OBSERVER.kpan 405 jaeger d fvc_filename=/data/fcam/59600/fimg-fvc1n-0147.fits 08:38:15 Config Loaded: 35951-2527 configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2527.pdf 08:38:18 OBSERVER.kpan 398 jaeger : text="Configuration 2527 loaded." 08:38:18 CmdDone: OBSERVER.kpan 398 jaeger configuration load 35951 08:38:18 CmdStarted: OBSERVER.kpan 399 jaeger configuration execute 08:38:54 .jaeger 0 jaeger f text="A collision was detected in positioner 664. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 08:38:59 OBSERVER.kpan 399 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 35.92 seconds after start" 08:38:59 CmdDone: OBSERVER.kpan 399 jaeger configuration execute 08:38:59 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0232.pdf 08:40:02 CmdStarted: OBSERVER.kpan 401 jaeger explode --one 664 0.5 jaeger unwind Try again: 8:42:46 Config Loaded: 35951-2528 08:46:07 FVC Loop - RMS:19.35 Delta RMS: -0.77 Image #: 0150 08:48:18 Config Loaded: 35952-2529 08:51:36 FVC Loop - RMS:10.82 Delta RMS: 0.17 Image #: 0154 08:53:16 Config Loaded: 35953-2530 08:56:34 FVC Loop - RMS: 6.24 Delta RMS: -0.04 Image #: 0158 08:58:29 Config Loaded: 35954-2531 09:02:30 FVC Loop - RMS: 5.41 Delta RMS: 0.80 Image #: 0163 09:04:56 Config Loaded: 35955-2532 09:07:34 FVC Loop - RMS:57.22 Delta RMS: -26.88 Image #: 0166 09:09:10 Config Loaded: 35956-2533 09:11:47 FVC Loop - RMS:31.81 Delta RMS: -10.60 Image #: 0169 09:13:49 Config Loaded: 35957-2534 09:16:28 FVC Loop - RMS:45.59 Delta RMS: -0.46 Image #: 0172 09:18:24 Config Loaded: 35958-2535 09:21:45 FVC Loop - RMS: 6.29 Delta RMS: 0.56 Image #: 0176 09:23:28 Config Loaded: 35959-2536 09:26:49 FVC Loop - RMS: 7.94 Delta RMS: -0.39 Image #: 0180 09:29:29 Config Loaded: 35960-2537 09:32:50 FVC Loop - RMS: 9.04 Delta RMS: 0.17 Image #: 0184 09:34:51 Config Loaded: 35961-2538 09:34:54 CmdStarted: OBSERVER.kpan 454 jaeger configuration execute 09:35:26 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 9:41:20 OBSERVER.kpan 462 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." Try again: configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2540.pdf 09:43:54 OBSERVER.kpan 466 jaeger : text="Configuration 2540 loaded." 09:43:54 CmdDone: OBSERVER.kpan 466 jaeger configuration load 35961 09:43:54 CmdStarted: OBSERVER.kpan 467 jaeger configuration execute 09:34:51 Config Loaded: 35961-2538 09:35:26 Jaeger collision: 608 09:39:00 Config Loaded: 35961-2539 09:43:51 Config Loaded: 35961-2540 09:44:31 Jaeger collision: 608 09:49:12 FVC Loop - RMS:42.18 Delta RMS: 0.35 Image #: 0189 09:49:43 Jaeger collision: 608 09:53:46 Config Loaded: 35961-2541 09:57:06 FVC Loop - RMS:12.40 Delta RMS: -2.03 Image #: 0193 09:59:21 Config Loaded: 35962-2542 10:00:02 Jaeger collision: 608 10:05:08 Config Loaded: 35962-2543 10:05:49 Jaeger collision: 608 10:09Z - BOSON crashed. CPU at 100%. Nothing out of the ordinary. The snapshot window was open at the time that both Kaike's and my BOSON crashed and needed to be restarted. 10:12:04 Config Loaded: 35963-2544 configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2544.pdf 10:12:07 OBSERVER.kpan 493 jaeger : text="Configuration 2544 loaded." 10:12:07 CmdDone: OBSERVER.kpan 493 jaeger configuration load 35963 10:12:07 CmdStarted: OBSERVER.kpan 494 jaeger configuration execute 10:12:45 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0303.pdf 10:12:45 OBSERVER.kpan 494 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 33.57 seconds after start" 10:12:45 CmdDone: OBSERVER.kpan 494 jaeger configuration execute 10:12:46 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0303.pdf 10:12:46 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0303.pdf (JaegerUserWarning)" Try again: 0:16:03 Config Loaded: 35963-2545 10:19:24 FVC Loop - RMS:38.09 Delta RMS: 0.55 Image #: 0197 10:21:00 Config Loaded: 35964-2546 10:24:21 FVC Loop - RMS:29.86 Delta RMS: -20.30 Image #: 0201 10:43Z - The humidity is improving. Not a lot of evidence of blowing snow anymore. Started ventilating. 11:08Z - Opened the enclosure. Tested the HAL window within BOSON: 11:24:49 CmdStarted: APO.apo_2 21 hal goto-field --guider-time 15 11:30:06 APO.apo_2 21 hal e error="FVC loop failed. RMS=12.07." 1:25:56 boss exposure 4.0, file=336396 11:26:37 boss exposure 4.0, file=336397 11:27:21 sp1.motor.move= 1482, 1482, 1481 11:27:21 Hartmann collimate output sp1: offset: r= 329 (In Focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 1482 sp1: pred. resid: r= -1153, b= -3.6, txt= OK, spTemp = 2.3 11:29:51 FVC Loop - RMS:12.07 Delta RMS: 0.33 Image #: 0205 11:33:56 boss exposure 4.0, file=336398 11:35:29 boss exposure 25.0, file=336399 Focused. Best focus at 700, (guide exposure #0017) Started cherno. All guide camera corrections are enabled, but the guiding is only finding a solution with 1-2 cameras. We are not quite sure why... A small clip... filename_bundle=/data/gcam/59600/gimg-gfa2n-0036.fits,/data/gcam/59600/gimg-gfa6n-0036.fits,/data/gcam/59600/gimg-gfa3n-0036.fits,/data/gcam/59600/gimg-gfa4n-0036.fits,/data/gcam/59600/gimg-gfa5n-0036.fits 12:11:56 cherno.cherno 97 fliswarm : 12:11:56 CmdDone: cherno.cherno 97 fliswarm talk -n gfa2,gfa3,gfa4,gfa5,gfa6 -- expose -n 36 15.0 12:11:56 .cherno 0 cherno i guider_status=0x20 12:11:56 .cherno 0 cherno i guider_status=0x24 12:12:00 APO.apo_2 25 cherno i camera_solution=gfa2,36,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:12:00 APO.apo_2 25 cherno i camera_solution=gfa3,36,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:12:00 APO.apo_2 25 cherno i camera_solution=gfa6,36,T,197.885391,71.254796,347.087,257.104,17.095 12:12:00 APO.apo_2 25 cherno i camera_solution=gfa4,36,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:12:00 APO.apo_2 25 cherno i camera_solution=gfa5,36,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:12:00 APO.apo_2 25 cherno w offset=0,0,-432 12:12:00 APO.apo_2 25 cherno i astrometry_fit=36,1,-999.0,-999.0,1.54,0.11,17.095,-1.952,6.34,-51.3,0.999758 12:12:00 APO.apo_2 25 cherno i text="Applying corrections." The 'RMS'/'FWHM'/'Acquisition failed' flags within the guide window are very nice! Trying a different field: Field 17818 RA: 205.68, Dec: 29.06 Cadence: bright_15x1_v1 Design 8110 12:18:10 CmdStarted: OBSERVER.kpan 552 jaeger configuration load 8110 configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59600/configuration_snapshot_59600_2548.pdf 12:20:16 OBSERVER.kpan 552 jaeger : text="Configuration 2548 loaded." 12:20:16 CmdDone: OBSERVER.kpan 552 jaeger configuration load 8110 12:23:38 APO.apo_2.jaeger 527 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 31.44 seconds after start" 12:23:38 CmdDone: APO.apo_2.jaeger 527 jaeger configuration execute 12:23:38 APO.apo_2 28 hal e error="Command jaeger configuration execute failed." 12:23:38 CmdStarted: APO.apo_2.mcp 528 mcp ff_off 12:23:38 CmdStarted: APO.apo_2.mcp 529 mcp hgcd_off 12:23:38 CmdStarted: APO.apo_2.mcp 530 mcp ne_off 12:23:38 APO.apo_2 28 hal w text="Running cleanup tasks." 12:23:38 APO.apo_2 28 hal f 12:23:38 CmdDone: APO.apo_2 28 hal goto-field --guider-time 15 12:23:38 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0323.pdf 12:23:38 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59600/fps_snapshot_59600_0323.pdf (JaegerUserWarning)" Trying again: 12:29:23 boss exposure 4.0, file=336400 12:30:03 boss exposure 4.0, file=336401 12:30:45 sp1.motor.move= -948, -948, -947 12:30:45 Hartmann collimate output sp1: offset: r= -1319 (Out of focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -948 sp1: pred. resid: r= -370, b= -1.2, txt= OK, spTemp = 0.7 12:32:36 FVC Loop - RMS:44.94 Delta RMS: -4.51 Image #: 0208 12:32:51 APO.apo_2 29 hal e error="FVC loop failed. RMS=44.94." Started cherno. We are still only seeing 1 camera being used for convergence. filename_bundle=/data/gcam/59600/gimg-gfa2n-0049.fits,/data/gcam/59600/gimg-gfa6n-0049.fits,/data/gcam/59600/gimg-gfa3n-0049.fits,/data/gcam/59600/gimg-gfa5n-0049.fits,/data/gcam/59600/gimg-gfa4n-0049.fits 12:34:46 cherno.cherno 131 fliswarm : 12:34:46 CmdDone: cherno.cherno 131 fliswarm talk -n gfa2,gfa3,gfa4,gfa5,gfa6 -- expose -n 49 15.0 12:34:46 .cherno 0 cherno i guider_status=0x20 12:34:46 .cherno 0 cherno i guider_status=0x24 12:34:49 APO.apo_2 30 cherno i camera_solution=gfa2,49,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:34:50 APO.apo_2 30 cherno i camera_solution=gfa3,49,T,207.238628,29.645277,157.125,67.089,7.107 12:34:50 APO.apo_2 30 cherno i camera_solution=gfa6,49,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:34:50 APO.apo_2 30 cherno i camera_solution=gfa5,49,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:34:50 APO.apo_2 30 cherno i camera_solution=gfa4,49,F,-999.0,-999.0,-999.0,-999.0,-999.0 12:34:50 APO.apo_2 30 cherno w offset=0,0,-432 12:34:50 APO.apo_2 30 cherno i astrometry_fit=49,1,-999.0,-999.0,1.35,0.17,7.107,8.814,4.943,-657.7,1.000238 12:34:50 APO.apo_2 30 cherno i text="Applying corrections." There is nothing else we can do for tonight. 12:57Z - Closed enclosure and stopped ventilating. Shut down FPS. Started cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0030 0.0030 Axial C: -0.1170 -0.1150 Trans D: -0.0340 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: -2.89 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6670 -9.6710 Axial C: -9.6970 -9.6970 Trans D: 0.1030 0.1140 GALILS Commanded: 4890900 4888150 4910100 0 -3400 Actual: 4859167 4859199 4859293 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.0 -0.03 0.01 0.01 41.66 ---- Telescope Status ---- No interlocks bypassed Status at: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 680/710 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 207 lbs, 22 psi ---- Software Used ---- Get Versions, 01/20/22, 23:39:17 -- STUI software: STUI : 1.7.16 alerts : 2.2.0 apo : FAILED boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED fliswarm : 0.2.1-alpha.0 jaeger : 0.13.2-alpha.0 kaiju : 1.2.2-alpha.0 coordio : 1.2.1-alpha.2 fps_calibrati : APO.2021.12.25 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.11 ---- Miscellaneous ---- (observers39) [observer@sdss5-hub: ~]$ list_ap -m 59599 - morning 59599 10 12:42 40370010 Dark 60 A 13.0 FPS- 0 a-b-c - - 59599 11 12:53 40370011 Dark 60 A 13.0 FPS- 0 a-b-c - - 59599 12 13:04 40370012 Dark 60 A 13.0 FPS- 0 a-b-c - - 59599 13 13:06 40370013 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59599 14 13:07 40370014 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59599 15 13:09 40370015 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59599 16 13:11 40370016 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.50 - 59599 17 13:19 40370017 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59599 18 13:21 40370018 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.10 - 59599 19 13:28 40370019 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59599 20 13:34 40370020 Dark 30 A 13.0 FPS- 0 a-b-c - - 59599 21 13:39 40370021 InteFlat 30 A 13.0 FPS- 0 a-b-c - 279 59599 22 13:44 40370022 InteFlat 30 A 13.0 FPS- 0 a-b-c - 279 59599 23 13:50 40370023 InteFlat 30 A 13.0 FPS- 0 a-b-c - 279 59599 24 13:55 40370024 Dark 30 A 13.0 FPS- 0 a-b-c - - 59599 25 14:01 40370025 Arc---- 30 A 13.0 FPS- 0 a-b-c - - FPI ------------------------------------------------------------------------------ (observers39) [observer@sdss5-hub: ~]$