Subject: 2.5m obslog 20220110 (SJD 59590) From: kinemuchi@apo.nmsu.edu Date: Tue, 11 Jan 2022 11:30:43 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday January 10, 2022 (SJD 59590) ---- Observing Team ---- Viktor Malanushenko Karen Kinemuchi (R) Jose Sanchez-Gallego (online) Conor Sayres (online) ---- Observing Plan ---- Monday January 10 If cloudy, please do the following in-dome tests with the telescope stationary at ~70 deg altitude. Test 1 Restart jaeger and cherno. Load an APOGEE design and execute the configuration. Run an FVC loop and make sure the RMS is ~ --scale XXX, and execute Run an FVC loop and make sure the RMS is ~ 00:42:54 OBSERVER.viktorm 586 jaeger i text="Loading design 35899." 00:42:55 OBSERVER.viktorm 586 jaeger f error="Failed retrieving design: 'Configuration' object has no attribute 'assignment_data'" 00:42:55 CmdDone: OBSERVER.viktorm 586 jaeger configuration load 35899 Contacted Jose for help and to debug new changes in jaeger. 01:09:32 jaeger configuration load 35899 01:10:57 jaeger configuration execute 01:12:48 jaeger fvc loop 01:13:16 OBSERVER.viktorm 615 jaeger i text="[FVC]: RMS full fit 72.960 um." 01:14:01 OBSERVER.viktorm 615 jaeger i text="[FVC]: RMS full fit 10.950 um." 01:14:43 OBSERVER.viktorm 615 jaeger i text="[FVC]: RMS full fit 9.020 um." 01:15:27 OBSERVER.viktorm 615 jaeger i text="[FVC]: RMS full fit 8.960 um." Running FVC loop again to get the RMS value below 5: 01:16:59 OBSERVER.viktorm 620 jaeger i text="[FVC]: RMS full fit 7.680 um." Still not converging to for some reason Log Function did not output this set of information. 04:07:12 Config Loaded: 35899-1561 04:07:26 Config Loaded: 35899-1573 04:08:05 RMS full fit 59.920 um." 04:09:30 FVC Loop - RMS:20.40 Delta RMS: 0.95 Image #: 0055 04:10:04 Config Loaded: 35899-1561 04:10:18 Config Loaded: 35899-1574 04:10:57 RMS full fit 70.170 um." 04:13:05 FVC Loop - RMS:25.65 Delta RMS: 0.09 Image #: 0059 04:13:38 Config Loaded: 35899-1561 04:13:52 Config Loaded: 35899-1575 04:14:31 RMS full fit 75.610 um." 04:16:40 FVC Loop - RMS:43.30 Delta RMS: 0.20 Image #: 0063 04:17:15 Config Loaded: 35899-1561 04:17:29 Config Loaded: 35899-1576 04:18:08 RMS full fit 58.380 um." 04:20:18 FVC Loop - RMS:20.97 Delta RMS: 0.16 Image #: 0067 04:20:52 Config Loaded: 35899-1561 04:21:06 Config Loaded: 35899-1577 04:21:45 RMS full fit 57.460 um." 04:23:55 FVC Loop - RMS:14.80 Delta RMS: 0.09 Image #: 0071 04:24:29 Config Loaded: 35899-1561 04:24:43 Config Loaded: 35899-1578 04:25:22 RMS full fit 60.840 um." 04:26:47 FVC Loop - RMS:30.17 Delta RMS: 0.18 Image #: 0074 04:27:20 Config Loaded: 35899-1561 04:27:34 Config Loaded: 35899-1579 04:28:12 RMS full fit 64.830 um." 04:29:38 FVC Loop - RMS:37.33 Delta RMS: 0.63 Image #: 0077 04:30:12 Config Loaded: 35899-1561 04:30:26 Config Loaded: 35899-1580 04:31:05 RMS full fit 64.550 um." 04:33:15 FVC Loop - RMS:10.98 Delta RMS: 0.36 Image #: 0081 04:33:48 Config Loaded: 35899-1561 04:34:03 Config Loaded: 35899-1581 04:34:41 RMS full fit 69.880 um." 04:36:49 FVC Loop - RMS:40.22 Delta RMS: 0.99 Image #: 0085 04:37:21 Config Loaded: 35899-1561 04:37:35 Config Loaded: 35899-1582 04:38:15 RMS full fit 53.790 um." 04:40:23 FVC Loop - RMS:10.81 Delta RMS: 0.02 Image #: 0089 04:40:56 Config Loaded: 35899-1561 04:41:10 Config Loaded: 35899-1583 The FVC loop failed at the 3rd iteration and output this message: 04:43:08 OBSERVER.viktorm 784 jaeger f error="Failed processing image: The exposure succeeded but did not output the filename." 04:43:08 CmdDone: OBSERVER.viktorm 784 jaeger fvc loop We will run a final jaeger fvc loop manually to finish this off. We are not sure why the file was not produced. 04:41:49 OBSERVER.viktorm 784 jaeger i text="[FVC]: RMS full fit 70.590 um." 04:42:31 OBSERVER.viktorm 784 jaeger i text="[FVC]: RMS full fit 28.310 um." 04:54:46 OBSERVER.viktorm 794 jaeger i text="[FVC]: RMS full fit 12.330 um." 04:55:28 OBSERVER.viktorm 794 jaeger i text="[FVC]: RMS full fit 5.420 um." 04:56:10 OBSERVER.viktorm 794 jaeger i text="[FVC]: RMS full fit 4.690 um." 04:40:56 Config Loaded: 35899-1561 04:41:10 Config Loaded: 35899-1583 04:56:10 FVC Loop - RMS: 4.69 Delta RMS: 0.73 Image #: 0095 04:59:00 Config Loaded: 35899-1561 05:02:25 Config Loaded: 35899-1584 05:03:04 RMS full fit 66.730 um." 05:04:29 FVC Loop - RMS:37.21 Delta RMS: -0.01 Image #: 0098 05:05:02 Config Loaded: 35899-1561 05:05:16 Config Loaded: 35899-1585 05:05:54 RMS full fit 63.080 um." 05:08:44 FVC Loop - RMS:25.27 Delta RMS: 0.50 Image #: 0103 05:09:18 Config Loaded: 35899-1561 Done with Test #1. Tried to do a jaeger unwind, but failed. Took a quick snapshot: 5:15:52 OBSERVER.viktorm 821 jaeger : snapshot=/data/logs/jaeger/snapshots/59590/fps_snapshot_59590_0130.pdf We see robot collisions, so issued an explode: 05:17:45 jaeger explode 20 05:18:52 jaeger unwind 05:20:06 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59590/fps_snapshot_59590_0132.pdf 05:20:06 OBSERVER.viktorm 828 jaeger i folded=T 05:20:06 OBSERVER.viktorm 828 jaeger i text="All positioners have reached their destinations." All good. Checked back with Jose, and we will skip ahead to Test #2 to check the functionality of the --scale flag with the jaeger configuration command. ============================================================================= 06:12Z Starting Test #2 since airmass is getting better and the field is rising. Field 35932 has RA=105.00 DEC=+40.00 06:13:07 jaeger configuration load 35932 --scale 0.999 06:14:50 jaeger configuration execute 06:15:28 .jaeger 0 jaeger f text="A collision was detected in positioner 184. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 06:15:28 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 06:15:28 .jaeger 0 jaeger i fps_status=0x14 06:15:30 .jaeger 0 jaeger e locked=T 06:15:33 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59590/fps_snapshot_59590_0133.pdf 06:15:33 OBSERVER.viktorm 868 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 37.66 seconds after start" 06:15:33 CmdDone: OBSERVER.viktorm 868 jaeger configuration execute 06:20:14 jaeger unlock 06:20:40 jaeger explode 20 06:20:54 jaeger configuration execute gave errors 09:58:16 .jaeger 0 jaeger w text="Positioner 971 replied to SEND_TRAJECTORY_DATA UID=10082 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:58:16 .jaeger 0 jaeger w text="Positioner 971 replied to SEND_TRAJECTORY_DATA UID=10082 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:58:16 .jaeger 0 jaeger w text="Positioner 971 replied to SEND_TRAJECTORY_DATA UID=10082 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:58:16 .jaeger 0 jaeger w text="Positioner 971 replied to SEND_TRAJECTORY_DATA UID=10082 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:58:16 OBSERVER.viktorm 126 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" try again: 10:00:14 jaeger unwind 10:00:31 jaeger configuration load 35964 --scale 0.999882 10:01:13 Config Loaded: 35964-1622 10:01:25 jaeger configuration execute 10:02:21 jaeger fvc loop Start RMS: 10:02:47 RMS full fit 49.370 um." 10:04:58 FVC Loop - RMS: 6.14 Delta RMS: 0.58 Image #: 0221 10:05:29 jaeger configuration reverse 10:07:00 jaeger configuration load 35964 --scale 1 10:07:43 Config Loaded: 35964-1623 10:08:07 jaeger configuration execute 10:09:12 jaeger fvc loop Start RMS: 10:09:39 RMS full fit 49.520 um." 10:11:50 FVC Loop - RMS: 6.16 Delta RMS: 0.63 Image #: 0225 10:12:32 jaeger configuration reverse 10:14:17 jaeger configuration load 35964 --scale 1.000118 10:14:59 Config Loaded: 35964-1624 10:15:24 jaeger configuration execute 10:16:29 jaeger fvc loop Start RMS: 10:16:55 RMS full fit 49.650 um." 10:19:03 FVC Loop - RMS: 6.04 Delta RMS: 0.74 Image #: 0229 10:19:43 jaeger configuration reverse 10:21:25 jaeger configuration load 35964 --scale 1.001 10:22:08 Config Loaded: 35964-1625 10:22:32 jaeger configuration execute 10:23:41 jaeger fvc loop Start RMS: 10:24:07 RMS full fit 49.080 um." 10:26:17 FVC Loop - RMS: 6.34 Delta RMS: 0.35 Image #: 0233 10:26:54 jaeger configuration reverse Test #2 done for this BOSS field. ============================================================================= 10:43Z Checked site weather since skies are beginning to clear up. There is blowing snow on site, and humidity is still high. 10:30:11 jaeger unwind 11:07:48 jaeger power off 11:07Z Giving up. Humidity is still high, but skies are clearing up. There's a lot of blowing snow on site. 11:16Z Started morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0020 0.0040 Axial B: 0.0080 0.0030 Axial C: -0.1100 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: -1.44 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6700 -9.6710 Axial C: -9.6950 -9.6970 Trans D: 0.1210 0.1140 GALILS Commanded: 4895950 4897900 4912300 -50 -3450 Actual: 4858285 4858934 4859060 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.64 -0.12 0.22 0.14 42.94 ---- Telescope Status ---- Never opened No interlocks bypassed Status at: 11:23Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 160 lbs, 21 psi ---- Software Used ---- Get Versions, 01/10/22, 23:21:35 -- STUI software: STUI : 1.7.16 alerts : 2.2.0 apo : 3.0.1 boss : 6.0.1 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : 0.2.1-alpha.0 jaeger : 0.13.2-alpha.0 kaiju : 1.2.2-alpha.0 coordio : 1.2.1-alpha.0 fps_calibrati : APO.2021.12.25 cherno : 0.1.1-alpha.0 hal : 0.1.0-alpha.0 hartmann : 2.0.2 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.10 --------------Other Versions-------------- sdss-obstools : 3.7.6 TPM : 1.2-31-gd7eb MCP : 2.1 idlspec2d : v6_0_4 ---- APOGEE Data Summary ---- APOGEE data list, mjd=59590, all raw_data: /data/apogee/utr_cdr/59590/apRaw-*.fits archive: /data/apogee/archive/59590/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59590/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59590 1 21:19 40280001 Object 3 A 13.0 FPS- 0 a-b-c err - 59590 2 21:22 40280002 Object 5 A 13.0 FPS- 0 a-b-c err - 59590 3 21:25 40280003 QuarFlat 5 A 13.0 FPS- 0 a-b-c - 75 59590 4 21:31 40280004 QuarFlat 5 A 13.0 FPS- 0 a-b-c - 75 59590 5 21:35 40280005 QuarFlat 5 A 13.0 FPS- 0 a-b-c - 75 59590 6 21:37 40280006 QuarFlat 3 A 13.0 FPS- 0 a-b-c - 41 apogeeql 59590 7 22:53 40280007 Dark 3 A 13.0 -------- a-b-c - - 59590 8 23:08 40280008 Dark 60 A 13.0 -------- a-b-c - - 59590 9 23:10 40280009 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59590 10 23:12 40280010 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.48 - 59590 11 23:19 40280011 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59590 12 23:21 40280012 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.11 - 59590 13 23:28 40280013 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59590 14 23:39 40280014 Dark 60 A 13.0 -------- a-b-c - - 59590 15 23:45 40280015 Arc---- 30 A 13.0 FPS- 0 a-b-c - - FPI 59590 16 11:17 40280016 Dark 60 A 13.0 -------- x-x-x - - ------------------------------------------------------------------------------ ---- Miscellaneous ---- Looped Commands for Test #1 (for both APOGEE and BOSS field designs): jaeger configuration dither 0.1 jaeger fvc loop tui wait 3 jaeger configuration reverse tui wait 5