Subject: 2.5m obslog 20220208 (SJD 59619) From: kgrabowski@apo.nmsu.edu Date: Wed, 9 Feb 2022 12:52:37 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday February 08, 2022 (SJD 59619) ---- Observing Team ---- Audrey Oravetz Katie Grabowski Zoom: Arlin Cortes, Pablo Vera, Paulo Assuncao Lago, Natalie Ulloa, Consuelo Conzalez Avila, Dan Oravetz, Karen Kinemuchi, Dylan Gatlin, ---- Observing Plan ---- https://wiki.sdss.org/pages/viewpage.action?spaceKey=COM&title=Commissioning+Nightly+Plan#CommissioningNightlyPlan-TuesdayFebruary8 ---- Observing Summary ---- Closed for about 2 hours due to wind gusts over 45 mph, and clouds. Observed 4 configurations on design 18492 Observed 3 configurations on design 12217 Observed 1 exposure each on the BOSS fields 35946, 35948, and 35950. Observed 1 exposure each on the APOGEE fields 35945, 35947, and 35949, although the first image with 35947 was taken without opening the shutter. Observed 15 dither APOGEE exposures on design 35975, with fvc loop --no apply Manually exploded and unwound to get to the next robot position. Using explode and reverse usually worked too. Had to put the designs back on top of the queue in Kronos, otherwise it moved on to the next one, off the current design. A boss exposure failed 26 read into an APOGEE exposure. May be some kind of miscommunication issue. No usual netburner errors. 2 out-of-closed-loop events slewing to the next field. One started with the altitude axis, the other with azimuth. The 180L dewar pressure went from 20 psi to 16 psi while we were filling after sunset. The spectro fill pressure decreased to 2psi, but did go up to 10 psi and hold steady there when we restarted the fill. 180L dewar pressure 18.6 at the end of the night. ---- Observing Log ---- Night: ====== (observers39) [observer@sdss5-hub: ~]$ df -h Filesystem Size Used Avail Use% Mounted on 10.25.1.230:/data 21T 16T 4.3T 79% /data 10.25.1.230:/archive 20T 24K 19T 1% /archive Checked actors, gathered mirror numbers, APOGEE and BOSS cals, turned on FPS. 22:41:56 boss exposure 0.0, file=337140 22:43:33 boss exposure 20.0, file=337141 22:47:23 boss exposure 4.0, file=337142 22:48:03 boss exposure 4.0, file=337143 22:48:47 sp1.motor.move= 1977, 1978, 1976 22:48:47 Hartmann collimate output sp1: offset: r= 1319 (Out of focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 1977 sp1: pred. resid: r= -658, b= -2.1, txt= OK, spTemp = 3.9 22:49:22 boss exposure 4.0, file=337144 22:52:54 boss exposure 30.0, file=337145 22:41:40 apogee.expose=apRaw-40570001, 60, A, Dark 22:52:32 apogee.expose=apRaw-40570002, 10, A, QuartzFlat 22:54:26 apogee.expose=apRaw-40570003, 12, B, ArcLamp 22:56:39 apogee.expose=apRaw-40570004, 40, B, ArcLamp 23:03:56 apogee.expose=apRaw-40570005, 12, A, ArcLamp 23:06:09 apogee.expose=apRaw-40570006, 40, A, ArcLamp 23:13:35 apogee.expose=apRaw-40570007, 60, A, Dark 23:24:27 apogee.expose=apRaw-40570008, 30, A, ArcLamp, FPI 23:10Z - Started ventilating. Night: ====== 00:49:10 encl25m : open; clear sky The fill took longer than expected because the 180L dewar only has 16 psi. It was at 20psi when we started the fill. The 10L pressure dropped down to 2psi, but we checked to make sure it would go back up to 6 and stay while we finished the fill. It went up to 10 when we started the fill again and is normal now that the fill completed. No further issues. Design 18492 Configuration 3196 01:33:08 boss exposure 4.0, file=337146 01:33:51 boss exposure 4.0, file=337147 01:34:40 sp1.motor.move= 4915, 4914, 4913 01:34:40 Hartmann collimate output sp1: offset: r= 4947 (Out of focus); b= -15.3 (Out of focus) sp1: pred. move: spAverageMove= 4915 sp1: pred. resid: r= 31, b= 0.1, txt= OK, spTemp = 3.0 01:35:29 APO.apo.jaeger 985 jaeger i text="[FVC]: RMS full fit 42.450 um." 01:36:13 FVC Loop - RMS:50.98 Delta RMS: -8.53 Image #: 0002 01:36:13 APO.apo.jaeger 985 jaeger i fvc_rms=50.98 01:36:13 APO.apo.jaeger 985 jaeger i fvc_deltarms=-8.53 01:36:13 APO.apo.jaeger 985 jaeger i text="Delta RMS reached. RMS target has not been reached." 01:36:30 APO.apo 24 hal e error="FVC loop failed. RMS=50.98." 01:36:12 .jaeger 0 jaeger w text="[FVC]: Found 7 positioners with dubious centroid matches." 01:36:13 APO.apo.jaeger 985 jaeger w text="[FVC]: Found 7 positioners with dubious centroid matches." 01:36:14 .jaeger 0 jaeger w text="[FVC]: Failed to calculate corrected positioner coordinates for 3 positioners." 01:36:14 APO.apo.jaeger 985 jaeger w text="[FVC]: Failed to calculate corrected positioner coordinates for 3 positioners." Retried FVC loop: 01:37:40 APO.apo.jaeger 993 jaeger i fvc_rms=50.47 01:37:43 .jaeger 0 jaeger w text="Found 12 deadlocks but applying correction anyway." 01:38:24 FVC Loop - RMS:58.46 Delta RMS: -7.99 Image #: 0004 01:38:24 APO.apo.jaeger 993 jaeger i fvc_rms=58.46 01:38:24 APO.apo.jaeger 993 jaeger i fvc_deltarms=-7.99 Reloaded design for a new configuration. Had to put the design back at the top of the queue in Kronos, otherwise it tried to set up for the next design, 16714. Design 18492, Configuration 3198 cannot unwind back to home position. 01:44:07 APO.apo.jaeger 1002 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 2 deadlocks were found ([865, 1150])" 01:44:07 CmdDone: APO.apo.jaeger 1002 jaeger unwind 01:44:07 APO.apo 28 hal e error="Command jaeger unwind failed." Manually explode 5, unwound fine. 01:46:56 boss exposure 4.0, file=337148 01:47:37 boss exposure 4.0, file=337149 01:48:18 sp1.motor.move= -293, -292, -293 01:48:18 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= 0.8 (In Focus) sp1: pred. move: spAverageMove= -293 sp1: pred. resid: r= -36, b= -0.1, txt= OK, spTemp = 2.6 01:48:50 APO.apo.jaeger 1013 jaeger i fvc_rms=41.9 01:49:33 .jaeger 0 jaeger w text="[FVC]: Found 7 positioners with dubious centroid matches." 01:49:33 APO.apo.jaeger 1013 jaeger w text="[FVC]: Found 7 positioners with dubious centroid matches." 01:49:33 APO.apo.jaeger 1013 jaeger i text="[FVC]: RMS full fit 50.640 um." 01:49:33 FVC Loop - RMS:50.64 Delta RMS: -8.74 Image #: 0006 Jose wanted us to restart cherno to test some code he added to the median scale calculation. for jaeger. 01:53:31 boss exposure 4.0, file=337150 01:55:04 boss exposure 25.0, file=337151 01:57:12 APO.apo 30 hal e error="Command cherno acquire -t 15.0 --full timed out." 01:57:41 APO.apo.cherno 1031 cherno d guide_rms=1,0.646,0.597,0.88 01:57:41 APO.apo.cherno 1031 cherno i astrometry_fit=1,4,-999.0,-999.0,4.77,0.1,18.072,9.969,5.324,-342.2,0.999965 01:57:41 APO.apo.cherno 1031 cherno i text="Applying corrections." 01:57:41 APO.apo.cherno 1031 cherno d text="Applying only large rotator correction." 01:57:12 APO.apo 30 hal e error="Command cherno acquire -t 15.0 --full timed out." Restarted cherno manually 01:59:19 APO.aoravetz 84 cherno d guide_rms=2,0.511,0.513,0.724 01:59:19 APO.aoravetz 84 cherno i astrometry_fit=2,4,-999.0,-999.0,4.56,0.15,18.087,10.326,5.333,-17.5,0.999993 Set focus to 450 um. Focus sweep ~400. Seeing ~1.6" 02:13:03 apogee.expose=apRaw-40570009, 43, A, Object 02:13:10 boss exposure 900.0, file=337152 02:20:44 apogee.expose=apRaw-40570010, 43, B, Object ~100 APOGEE targets observed. Reload design for new configuration. Cloned to be configuration 3199 02:30:27 boss exposure 900.0, file=337153 02:30:28 apogee.expose=apRaw-40570011, 43, A, Object 02:38:09 apogee.expose=apRaw-40570012, 43, B, Object Haven't loaded 18492 back to the top of the queue, so we'll reconfigure again. Not labeled a cloned design in HAL. The robots didn't move to the next queued position. Design 18492, Configuration 3201 (same robot positions as 3198 and 3199) 02:53:25 boss exposure 900.0, file=337154 02:53:27 apogee.expose=apRaw-40570013, 43, A, Object 02:53:25 boss exposure 900.0, file=337154 02:53:27 apogee.expose=apRaw-40570013, 43, A, Object 03:01:08 apogee.expose=apRaw-40570014, 43, B, Object Design 18492, Configuration 3202. Cloned. 03:10:24 boss exposure 900.0, file=337155 03:10:24 apogee.expose=apRaw-40570015, 43, A, Object 03:14:30 APO.apo 44 hal e error="Command 'boss exposure science itime=900.0' failed." 03:14:35 APO.apo 44 hal w text="Cancelling current APOGEE exposure." 03:14:35 APO.apo 44 hal w text="Cancelling current BOSS exposure." 03:14:35 APO.apo.boss 1057 boss f text="exposure aborted" 03:14:35 CmdDone: APO.apo.boss 1057 boss exposure science itime=900.0 Restarted. Not sure why it failed. Only 26 reads for APOGEE. This is not recorded in the log function, but the magnitude slope in the quick-red plot is much steeper. Jose thinks it may be a netburner issue. None of the usual netburner errors were shown. May have been a miscommunication between hal and cameras. 03:20:12 apogee.expose=apRaw-40570016, 43, A, Object 03:20:18 boss exposure 900.0, file=337155 Image was cleared and rewritten. Unclear as to how. It shouldn't be an errant click in Boson that stopped it--the Run button was grayed out. Image taken at this time is recorded on the SOS page as 156. There is no 155. 03:14:35 CmdDone: APO.apo 44 hal expose --pairs -b 900 --count 1 03:27:52 apogee.expose=apRaw-40570017, 43, B, Object At some point during this sequence before 19 reads into APOGEE, HAL stopped counting down the exposure times, and the "Run" button for expose procedure is blue and not grayed out. Focus sweep before moving on to the next field. 375um Started a bias so the Log Function window has the right exposure numbers. 03:41:52 boss exposure 0.0, file=337157 03:43:27 apogee.expose=apRaw-40570018, 5, B, DomeFlat Had to manually explode and unwind --force. 3 deadlocks were found ([148, 865, 1150]) jaeger explode 20 --one 0148 jaeger goto 1150 10 170 unwound 03:47:44 APO.aoravetz 164 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59619/trajectory-59619-0010.json 03:47:44 APO.aoravetz 164 jaeger i folded=F 03:47:44 APO.aoravetz 164 jaeger i text="All positioners have reached their destinations." 03:47:46 APO.aoravetz 165 jaeger d text="Calculating unwind trajectory." 03:47:51 .jaeger 0 jaeger d alive_at=1644378471.6813264 03:48:04 .jaeger 0 jaeger d text="Setting chiller to 0.0 C" 03:48:06 APO.aoravetz 165 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 3 deadlocks were found ([148, 865, 1150])" 03:48:06 CmdDone: APO.aoravetz 165 jaeger unwind Designed 12217 Configuration 3203 03:55:12 boss exposure 4.0, file=337158 03:55:53 boss exposure 4.0, file=337159 03:56:42 sp1.motor.move= -3227, -3228, -3227 03:56:42 Hartmann collimate output sp1: offset: r= -3628 (Out of focus); b= 8.9 (Out of focus) sp1: pred. move: spAverageMove= -3228 sp1: pred. resid: r= -400, b= -1.3, txt= OK, spTemp = 1.4 03:57:13 APO.apo.jaeger 1093 jaeger i fvc_rms=44.55 03:57:58 APO.apo.jaeger 1093 jaeger i fvc_rms=22.79 03:57:58 APO.apo.jaeger 1093 jaeger i fvc_deltarms=21.76 03:58:43 APO.apo.jaeger 1093 jaeger i fvc_rms=18.49 03:58:43 APO.apo.jaeger 1093 jaeger i fvc_deltarms=4.3 03:59:26 APO.apo.jaeger 1093 jaeger i fvc_rms=19.72 03:59:26 APO.apo.jaeger 1093 jaeger i fvc_deltarms=-1.23 03:59:26 APO.apo.jaeger 1093 jaeger i text="Delta RMS reached. RMS target has not been reached." 03:59:26 FVC Loop - RMS:19.72 Delta RMS: -1.23 Image #: 0010 04:00:03 boss exposure 4.0, file=337160 04:01:38 boss exposure 25.0, file=337161 Seeing has deteriorated to 2" due to high wind speeds. 04:07:20 boss exposure 900.0, file=337162 04:07:22 apogee.expose=apRaw-40570019, 43, A, Object 04:15:02 apogee.expose=apRaw-40570020, 43, B, Object jaeger configuration load pulls the next field from Kronos. --clone or specifying the design number will create a new configuration number for the same design Design 12217, configuration 3205 was created so we would go back to the design the robots are configured for. Confirmed jaeger configuration load 12217 becomes configuration 3206 04:29:15 boss exposure 900.0, file=337163 04:29:16 apogee.expose=apRaw-40570021, 43, A, Object 04:36:57 apogee.expose=apRaw-40570022, 43, B, Object Design 12217, Configuration 3207 04:45:53 boss exposure 900.0, file=337164 04:45:55 apogee.expose=apRaw-40570023, 43, A, Object 04:53:35 apogee.expose=apRaw-40570024, 43, B, Object 04:46:27 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-02-09 04:46:27; last seen 600 sec ago",enabled,noack,-1 No ping back from the actor.Restarted. 05:02:06 apogee.expose=apRaw-40570025, 5, B, DomeFlat 05:07:11 encl25m : closed; Closed due to wind gusts over 45 mph. Seeing ~2.5" unwinding robots 07:14:11 encl25m : open; Design 35947, Configuration 3208 07:24:53 APO.apo.jaeger 1145 jaeger i fvc_rms=55.28 07:25:35 APO.apo.jaeger 1145 jaeger i fvc_rms=17.92 07:25:35 APO.apo.jaeger 1145 jaeger i fvc_deltarms=37.36 07:26:17 APO.apo.jaeger 1145 jaeger i fvc_rms=20.08 07:26:17 APO.apo.jaeger 1145 jaeger i fvc_deltarms=-2.16 07:26:17 APO.apo.jaeger 1145 jaeger i text="Delta RMS reached. RMS target has not been reached." 07:26:17 FVC Loop - RMS:20.08 Delta RMS: -2.16 Image #: 0013 07:26:34 APO.apo 24 hal e error="FVC loop failed. RMS=20.08. 07:27:27 APO.apo.jaeger 1151 jaeger i text="[FVC]: RMS full fit 16.430 um." 07:28:09 .jaeger 0 jaeger w text="[FVC]: Found 6 positioners with dubious centroid matches." 07:28:09 APO.apo.jaeger 1151 jaeger w text="[FVC]: Found 6 positioners with dubious centroid matches." 07:28:09 APO.apo.jaeger 1151 jaeger i fvc_rms=17.49 07:28:09 APO.apo.jaeger 1151 jaeger i fvc_deltarms=-1.06 07:28:09 APO.apo.jaeger 1151 jaeger i text="Delta RMS reached. RMS target has not been reached." 07:28:09 FVC Loop - RMS:17.49 Delta RMS: -1.06 Image #: 0015 Focus sweep. ~3C colder than earlier. Seeing ~2-3" Took a while for the guider to converge, due to clouds and seeing. 08:05:28 apogee.expose=apRaw-40570026, 30, A, Object 08:12:05 apogee.expose=apRaw-40570027, 5, A, DomeFlat Quick-red doesn't show any science targets or sources, although it is definitely an APOGEE field. Guider continued, but field may have been impacted by clouds. Exposure was initiated from the command line, but the shutter was closed. Altitude and then rotator out of closed loop while slewing to the next field. 08:15:12 APO.apo 28 hal e error="TCC slew command ended with some bad bits set: 0x1800,0x1800,0x800" 08:16:18 .jaeger 0 jaeger w text="Positioner 437 replied to SEND_TRAJECTORY_DATA UID=14292 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" Design 35945 Configuration 3211 08:24:59 APO.apo.jaeger 1193 jaeger i fvc_rms=46.2 08:25:42 APO.apo.jaeger 1193 jaeger i fvc_rms=17.35 08:25:42 APO.apo.jaeger 1193 jaeger i fvc_deltarms=28.85 08:26:28 APO.apo.jaeger 1193 jaeger i fvc_rms=11.44 08:26:28 APO.apo.jaeger 1193 jaeger i fvc_deltarms=5.91 08:27:12 APO.apo.jaeger 1193 jaeger i fvc_rms=13.11 08:27:12 APO.apo.jaeger 1193 jaeger i fvc_deltarms=-1.67 08:27:12 FVC Loop - RMS:13.11 Delta RMS: -1.67 Image #: 0021 08:31:07 apogee.expose=apRaw-40570028, 30, A, Object 08:36:45 apogee.expose=apRaw-40570029, 5, A, DomeFlat Design 35946, Configuation 3212 08:40:49 boss exposure 4.0, file=337165 08:41:30 boss exposure 4.0, file=337166 08:42:17 sp1.motor.move= -771, -772, -774 08:42:17 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= 4.8 (Out of focus) sp1: pred. move: spAverageMove= -771 sp1: pred. resid: r= 771, b= 2.4, txt= OK, spTemp = 0.3 08:42:47 APO.apo.jaeger 1219 jaeger i text="[FVC]: RMS full fit 48.170 um." 08:43:30 APO.apo.jaeger 1219 jaeger i fvc_rms=14.56 08:43:30 APO.apo.jaeger 1219 jaeger i fvc_deltarms=33.61 08:44:11 APO.apo.jaeger 1219 jaeger i fvc_rms=11.71 08:44:11 APO.apo.jaeger 1219 jaeger i fvc_deltarms=2.85 08:44:52 APO.apo.jaeger 1219 jaeger i fvc_rms=8.92 08:44:52 APO.apo.jaeger 1219 jaeger i fvc_deltarms=2.79 08:45:33 APO.apo.jaeger 1219 jaeger i fvc_rms=12.67 08:45:33 APO.apo.jaeger 1219 jaeger i fvc_deltarms=-3.75 08:45:33 APO.apo.jaeger 1219 jaeger i text="Delta RMS reached. RMS target has not been reached." 08:45:33 FVC Loop - RMS:12.67 Delta RMS: -3.75 Image #: 0026 08:46:10 boss exposure 4.0, file=337167 08:47:45 boss exposure 25.0, file=337168 08:51:40 boss exposure 300.0, file=337169 Unwind failed. Explode twice. Reverse didn't help. Unwind --force Design 34947, Configuration 3213 886, 552, and 733-- explode 20 unable to send 552 directly to home position (snapshots 49-51) 09:07:09 APO.apo.jaeger 1249 jaeger i fvc_rms=54.78 09:07:53 APO.apo.jaeger 1249 jaeger i fvc_rms=15.97 09:07:53 APO.apo.jaeger 1249 jaeger i fvc_deltarms=38.81 09:08:34 APO.apo.jaeger 1249 jaeger i fvc_rms=15.48 09:08:34 APO.apo.jaeger 1249 jaeger i fvc_deltarms=0.49 09:08:34 APO.apo.jaeger 1249 jaeger i text="Delta RMS reached. RMS target has not been reached." 09:08:34 FVC Loop - RMS:15.48 Delta RMS: 0.49 Image #: 0029 09:11:10 apogee.expose=apRaw-40570030, 30, A, Object 09:16:45 apogee.expose=apRaw-40570031, 5, A, DomeFlat Design 35948, Configuration 3214 09:20:23 boss exposure 4.0, file=337170 09:21:05 boss exposure 4.0, file=337171 09:21:46 sp1.motor.move= -449, -449, -450 09:21:46 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= 2.8 (In Focus) sp1: pred. move: spAverageMove= -449 sp1: pred. resid: r= 449, b= 1.4, txt= OK, spTemp = -0.0 09:22:17 APO.apo.jaeger 1276 jaeger i fvc_rms=49.19 09:23:00 APO.apo.jaeger 1276 jaeger i fvc_rms=26.2 09:23:00 APO.apo.jaeger 1276 jaeger i fvc_deltarms=22.99 09:23:41 APO.apo.jaeger 1276 jaeger i fvc_rms=26.12 09:23:41 APO.apo.jaeger 1276 jaeger i fvc_deltarms=0.08 09:23:41 APO.apo.jaeger 1276 jaeger i text="Delta RMS reached. RMS target has not been reached." 09:23:41 FVC Loop - RMS:26.12 Delta RMS: 0.08 Image #: 0032 09:26:39 boss exposure 4.0, file=337172 09:28:13 boss exposure 25.0, file=337173 09:31:31 boss exposure 300.0, file=337174 Design 35950 Configuration 3215 09:40:07 boss exposure 4.0, file=337175 09:40:47 boss exposure 4.0, file=337176 09:41:29 sp1.motor.move= -453, -453, -454 09:41:29 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -453 sp1: pred. resid: r= 124, b= 0.4, txt= OK, spTemp = 0.0 09:44:44 APO.apo.jaeger 1316 jaeger i fvc_rms=211.92 09:45:27 APO.apo.jaeger 1316 jaeger i fvc_rms=11.93 09:45:27 APO.apo.jaeger 1316 jaeger i fvc_deltarms=199.99 09:46:07 APO.apo.jaeger 1316 jaeger i fvc_rms=10.66 09:46:07 APO.apo.jaeger 1316 jaeger i fvc_deltarms=1.27 09:46:49 APO.apo.jaeger 1316 jaeger i fvc_rms=12.31 09:46:49 APO.apo.jaeger 1316 jaeger i fvc_deltarms=-1.65 09:46:49 APO.apo.jaeger 1316 jaeger i text="Delta RMS reached. RMS target has not been reached." 09:42:43 FVC Loop - RMS:211.61 Delta RMS: -162.51 Image #: 0034 09:47:26 boss exposure 4.0, file=337177 09:48:59 boss exposure 25.0, file=337178 09:52:13 boss exposure 300.0, file=337179 Explode and reverse in between unwinding and fields seemed to avoid collisions. 10:01:26 APO.apo.jaeger 1339 jaeger i fvc_rms=45.5 10:02:09 APO.apo.jaeger 1339 jaeger i fvc_rms=39.08 10:02:09 APO.apo.jaeger 1339 jaeger i fvc_deltarms=6.42 10:02:51 APO.apo.jaeger 1339 jaeger i fvc_rms=15.13 10:02:51 APO.apo.jaeger 1339 jaeger i fvc_deltarms=23.95 10:03:33 APO.apo.jaeger 1339 jaeger i fvc_rms=12.4 10:03:33 APO.apo.jaeger 1339 jaeger i fvc_deltarms=2.73 10:04:15 APO.apo.jaeger 1339 jaeger i fvc_rms=10.56 10:04:15 APO.apo.jaeger 1339 jaeger i fvc_deltarms=1.84 Design 35949 Configuration 3216 10:07:14 apogee.expose=apRaw-40570032, 30, A, Object 10:12:50 apogee.expose=apRaw-40570033, 5, A, DomeFlat Azimuth altitude out of closed loop during slew, which caused other axes to halt. 10:15:53 APO.apo.mcp 1359 mcp : command="ffs_close" 10:15:53 CmdDone: APO.apo.mcp 1359 mcp ffs.close 10:15:57 .tcc 0 tcc w BadRotStatus; RotStat=74.924656,0.000000,5151118557.87896,0x00000800; RotDTime=0.04 10:15:59 .tcc 0 tcc w BadAzStatus; Design 35975 Configuration 3217 10:20:25 APO.apo.jaeger 1371 jaeger i text="[FVC]: RMS full fit 50.770 um." 10:21:09 APO.apo.jaeger 1371 jaeger i fvc_rms=24.69 10:21:09 APO.apo.jaeger 1371 jaeger i fvc_deltarms=26.08 10:21:50 APO.apo.jaeger 1371 jaeger i fvc_rms=16.21 10:21:50 APO.apo.jaeger 1371 jaeger i fvc_deltarms=8.48 10:22:32 APO.apo.jaeger 1371 jaeger i fvc_rms=15.91 10:22:32 APO.apo.jaeger 1371 jaeger i fvc_deltarms=0.3 10:22:32 APO.apo.jaeger 1371 jaeger i text="Delta RMS reached. RMS target has not been reached." 10:22:32 FVC Loop - RMS:15.91 Delta RMS: 0.30 Image #: 0047 10:25:56 apogee.expose=apRaw-40570034, 30, A, Object 10:32:56 apogee.expose=apRaw-40570035, 30, A, Object 10:31:54 APO.aoravetz 344 jaeger i fvc_rms=49.97 10:39:08 APO.aoravetz 355 jaeger i fvc_rms=51.0 10:40:11 apogee.expose=apRaw-40570036, 30, A, Object 10:46:21 APO.aoravetz 364 jaeger i fvc_rms=49.69 10:47:23 apogee.expose=apRaw-40570037, 30, A, Object 10:53:35 APO.aoravetz 373 jaeger i fvc_rms=49.17 10:54:37 apogee.expose=apRaw-40570038, 30, A, Object 11:00:48 APO.aoravetz 383 jaeger i fvc_rms=50.34 11:01:51 apogee.expose=apRaw-40570039, 30, A, Object 11:08:04 APO.aoravetz 392 jaeger i fvc_rms=54.71 11:09:06 apogee.expose=apRaw-40570040, 30, A, Object 11:15:18 APO.aoravetz 401 jaeger i fvc_rms=50.15 11:16:20 apogee.expose=apRaw-40570041, 30, A, Object 11:22:30 APO.aoravetz 410 jaeger i fvc_rms=50.03 11:23:32 apogee.expose=apRaw-40570042, 30, A, Object 11:29:42 APO.aoravetz 419 jaeger i fvc_rms=51.59 11:30:44 apogee.expose=apRaw-40570043, 30, A, Object 11:36:56 APO.aoravetz 428 jaeger i fvc_rms=53.96 11:37:58 apogee.expose=apRaw-40570044, 30, A, Object 11:44:09 APO.aoravetz 437 jaeger i fvc_rms=50.12 11:45:10 apogee.expose=apRaw-40570045, 30, A, Object 11:51:21 APO.aoravetz 446 jaeger i fvc_rms=50.35 11:52:23 apogee.expose=apRaw-40570046, 30, A, Object 12:00:16 APO.aoravetz 455 jaeger i fvc_rms=51.09 12:01:25 apogee.expose=apRaw-40570047, 30, A, Object 12:07:48 APO.aoravetz 464 jaeger i fvc_rms=50.62 12:08:50 apogee.expose=apRaw-40570048, 30, A, Object 12:20:58 APO.aoravetz 473 jaeger i fvc_rms=49.82 12:21:59 apogee.expose=apRaw-40570049, 30, A, Object 12:29:34 apogee.expose=apRaw-40570050, 5, A, DomeFlat 12:37:11 encl25m : closed; 12:37:21 apogee.expose=apRaw-40570051, 60, A, Dark 12:38:22 boss exposure 0.0, file=337180 12:39:51 boss exposure 300.0, file=337181 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0010 0.0030 Axial C: -0.1130 -0.1150 Trans D: -0.0343 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 8.11 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6670 -9.6710 Axial C: -9.6940 -9.6970 Trans D: 0.1160 0.1140 GALILS Commanded: 4894250 4893350 4912200 0 -3400 Actual: 4859236 4859186 4859161 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.98 0.01 -0.02 -0.01 41.64 ---- Telescope Status ---- 23:15Z Started ventilation 00:49Z encl25m : open 12:37Z encl25m : closed No interlocks bypassed Status at:12:43Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 685 687 705 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 220 lbs, 18.6 psi ---- Software Used ---- Get Versions, 02/08/22, 22:46:38 -- STUI software: STUI : 1.7.16 alerts : 2.2.0 apo : 3.0.1 boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED fliswarm : 0.2.1-alpha.0 jaeger : 0.13.2-alpha.0 kaiju : 1.2.2-alpha.1 coordio : 1.2.1-alpha.2 fps_calibrati : APO.2021.12.25+960cb05 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.11 -- done -- ---- APOGEE Data Summary ---- APOGEE data list, mjd=59619, all raw_data: /data/apogee/utr_cdr/59619/apRaw-*.fits archive: /data/apogee/archive/59619/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59619/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59619 1 22:51 40570001 Dark 60 A 13.0 ---------- a-b-c - - 59619 2 22:53 40570002 QuarFlat 10 A 13.0 ---------- a-b-c - 99 59619 3 22:56 40570003 Arc-Thar 12 B 13.5 ---------- a-b-c 0.50 - 59619 4 23:03 40570004 Arc-Une 40 B 13.5 ---------- a-b-c - - 59619 5 23:05 40570005 Arc-Thar 12 A 13.0 ---------- a-b-c -0.13 - 59619 6 23:12 40570006 Arc-Une 40 A 13.0 ---------- a-b-c - - 59619 7 23:23 40570007 Dark 60 A 13.0 ---------- a-b-c - - 59619 8 23:29 40570008 Arc---- 30 A 13.0 ---------- a-b-c - - FPI 59619 9 02:20 40570009 Object 43 A 13.0 18492-3198 a-b-c -0.10 - 59619 10 02:27 40570010 Object 43 B 13.5 18492-3198 a-b-c 0.40 - 59619 11 02:37 40570011 Object 43 A 13.0 18492-3199 a-b-c -0.09 - 59619 12 02:45 40570012 Object 43 B 13.5 18492-3199 a-b-c 0.40 - 59619 13 03:00 40570013 Object 43 A 13.0 18492-3201 a-b-c -0.10 - 59619 14 03:08 40570014 Object 43 B 13.5 18492-3201 a-b-c 0.41 - 59619 15 03:14 40570015 Object 26 A 13.0 18492-3202 a-b-c -0.09 - 59619 16 03:27 40570016 Object 43 A 13.0 18492-3202 a-b-c -0.10 - 59619 17 03:35 40570017 Object 43 B 13.5 18492-3202 a-b-c 0.47 - 59619 18 03:43 40570018 DomeFlat 5 B 13.5 18492-3202 a-b-c - 148 59619 19 04:14 40570019 Object 43 A 13.0 12217-3203 a-b-c -0.10 - 59619 20 04:22 40570020 Object 43 B 13.5 12217-3203 a-b-c 0.41 - 59619 21 04:36 40570021 Object 43 A 13.0 12217-3206 a-b-c -0.10 - 59619 22 04:44 40570022 Object 43 B 13.5 12217-3206 a-b-c 0.39 - 59619 23 04:53 40570023 Object 43 A 13.0 12217-3207 a-b-c -0.10 - 59619 24 05:00 40570024 Object 43 B 13.5 12217-3207 a-b-c 0.40 - 59619 25 05:02 40570025 DomeFlat 5 B 13.5 12217-3207 a-b-c - 142 59619 26 08:10 40570026 Object 30 A 13.0 35947-3208 a-b-c err - 59619 27 08:12 40570027 DomeFlat 5 A 13.0 35947-3208 a-b-c - 144 59619 28 08:36 40570028 Object 30 A 13.0 35945-3211 a-b-c -0.10 - 59619 29 08:37 40570029 DomeFlat 5 A 13.0 35945-3211 a-b-c - 147 59619 30 09:16 40570030 Object 30 A 13.0 35947-3213 a-b-c -0.10 - 59619 31 09:17 40570031 DomeFlat 5 A 13.0 35947-3213 a-b-c - 151 59619 32 10:12 40570032 Object 30 A 13.0 35949-3216 a-b-c -0.10 - 59619 33 10:13 40570033 DomeFlat 5 A 13.0 35949-3216 a-b-c - 149 59619 34 10:30 40570034 Object 30 A 13.0 35975-3217 a-b-c -0.10 - 59619 35 10:37 40570035 Object 30 A 13.0 35975-3218 a-b-c -0.10 - 59619 36 10:45 40570036 Object 30 A 13.0 35975-3219 a-b-c -0.11 - 59619 37 10:52 40570037 Object 30 A 13.0 35975-3220 a-b-c -0.10 - 59619 38 10:59 40570038 Object 30 A 13.0 35975-3221 a-b-c -0.10 - 59619 39 11:06 40570039 Object 30 A 13.0 35975-3222 a-b-c -0.10 - 59619 40 11:14 40570040 Object 30 A 13.0 35975-3223 a-b-c -0.10 - 59619 41 11:21 40570041 Object 30 A 13.0 35975-3224 a-b-c -0.09 - 59619 42 11:28 40570042 Object 30 A 13.0 35975-3225 a-b-c -0.10 - 59619 43 11:35 40570043 Object 30 A 13.0 35975-3226 a-b-c -0.09 - 59619 44 11:42 40570044 Object 30 A 13.0 35975-3227 a-b-c -0.09 - 59619 45 11:50 40570045 Object 30 A 13.0 35975-3228 a-b-c -0.09 - 59619 46 11:57 40570046 Object 30 A 13.0 35975-3229 a-b-c -0.10 - 59619 47 12:06 40570047 Object 30 A 13.0 35975-3230 a-b-c -0.10 - 59619 48 12:13 40570048 Object 30 A 13.0 35975-3231 a-b-c -0.10 - 59619 49 12:26 40570049 Object 30 A 13.0 35975-3232 a-b-c -0.09 - 59619 50 12:30 40570050 DomeFlat 5 A 13.0 35975-3217 a-b-c - 145 cannot read file : /data/apogee/utr_cdr/59619/apRaw-40570051.fits ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59619 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:41 b1-00337140 ----------- ---- bia Clo 0.0 -7432 -7290 -8665 2 22:41 r1-00337140 ----------- ---- bia Clo 0.0 -7432 -7290 -8665 3 22:43 b1-00337141 ----------- ---- dar Clo 20.0 -7432 -7290 -8665 4 22:43 r1-00337141 ----------- ---- dar Clo 20.0 -7432 -7290 -8665 5 22:47 b1-00337142 35969-20938 3179 arc Lef 4.1 -7432 -7290 -8665 HART 6 22:47 r1-00337142 35969-20938 3179 arc Lef 4.1 -7432 -7290 -8665 HART 7 22:48 r1-00337143 35969-20938 3179 arc Rig 4.1 -7432 -7290 -8665 HART 8 22:48 b1-00337143 35969-20938 3179 arc Rig 4.1 -7432 -7290 -8665 HART 9 22:49 r1-00337144 35969-20938 3179 arc Out 4.1 -5455 -5312 -6689 10 22:49 b1-00337144 35969-20938 3179 arc Out 4.1 -5455 -5312 -6689 11 22:52 b1-00337145 35969-20938 3179 fla Out 30.1 -5455 -5312 -6689 12 22:52 r1-00337145 35969-20938 3179 fla Out 30.1 -5455 -5312 -6689 13 01:33 b1-00337146 18492-20549 3196 arc Lef 4.1 -5455 -5312 -6689 HART 14 01:33 r1-00337146 18492-20549 3196 arc Lef 4.1 -5455 -5312 -6689 HART 15 01:33 r1-00337147 18492-20549 3196 arc Rig 4.1 -5455 -5312 -6689 HART 16 01:33 b1-00337147 18492-20549 3196 arc Rig 4.1 -5455 -5312 -6689 HART 17 01:46 b1-00337148 18492-20549 3198 arc Lef 4.1 -540 -398 -1776 HART 18 01:46 r1-00337148 18492-20549 3198 arc Lef 4.1 -540 -398 -1776 HART 19 01:47 r1-00337149 18492-20549 3198 arc Rig 4.1 -540 -398 -1776 HART 20 01:47 b1-00337149 18492-20549 3198 arc Rig 4.1 -540 -398 -1776 HART 21 01:53 b1-00337150 18492-20549 3198 arc Out 4.1 -833 -690 -2069 22 01:53 r1-00337150 18492-20549 3198 arc Out 4.1 -833 -690 -2069 23 01:55 b1-00337151 18492-20549 3198 fla Out 25.1 -833 -690 -2069 24 01:55 r1-00337151 18492-20549 3198 fla Out 25.1 -833 -690 -2069 25 02:13 r1-00337152 18492-20549 3198 sci Out 900.1 -833 -690 -2069 26 02:13 b1-00337152 18492-20549 3198 sci Out 900.1 -833 -690 -2069 27 02:30 b1-00337153 18492-20549 3199 sci Out 900.1 -833 -690 -2069 28 02:30 r1-00337153 18492-20549 3199 sci Out 900.1 -833 -690 -2069 29 02:53 b1-00337154 18492-20549 3201 sci Out 900.1 -833 -690 -2069 30 02:53 r1-00337154 18492-20549 3201 sci Out 900.1 -833 -690 -2069 31 03:20 r1-00337156 18492-20549 3202 sci Out 900.1 -833 -690 -2069 32 03:20 b1-00337156 18492-20549 3202 sci Out 900.1 -833 -690 -2069 33 03:41 r1-00337157 ----------- ---- bia Clo 0.0 -833 -690 -2069 34 03:41 b1-00337157 ----------- ---- bia Clo 0.0 -833 -690 -2069 35 03:55 r1-00337158 12217-18956 3203 arc Lef 4.1 -833 -690 -2069 HART 36 03:55 b1-00337158 12217-18956 3203 arc Lef 4.1 -833 -690 -2069 HART 37 03:55 b1-00337159 12217-18956 3203 arc Rig 4.1 -833 -690 -2069 HART 38 03:55 r1-00337159 12217-18956 3203 arc Rig 4.1 -833 -690 -2069 HART 39 04:00 b1-00337160 12217-18956 3203 arc Out 4.1 -4060 -3918 -5296 40 04:00 r1-00337160 12217-18956 3203 arc Out 4.1 -4060 -3918 -5296 41 04:01 b1-00337161 12217-18956 3203 fla Out 25.1 -4060 -3918 -5296 42 04:01 r1-00337161 12217-18956 3203 fla Out 25.1 -4060 -3918 -5296 43 04:07 r1-00337162 12217-18956 3203 sci Out 900.1 -4060 -3918 -5296 44 04:07 b1-00337162 12217-18956 3203 sci Out 900.1 -4060 -3918 -5296 45 04:29 r1-00337163 12217-18956 3206 sci Out 900.1 -4060 -3918 -5296 46 04:29 b1-00337163 12217-18956 3206 sci Out 900.1 -4060 -3918 -5296 47 04:45 r1-00337164 12217-18956 3207 sci Out 900.1 -4060 -3918 -5296 48 04:45 b1-00337164 12217-18956 3207 sci Out 900.1 -4060 -3918 -5296 49 08:40 r1-00337165 35946-20915 3212 arc Lef 4.1 -4060 -3918 -5296 HART 50 08:40 b1-00337165 35946-20915 3212 arc Lef 4.1 -4060 -3918 -5296 HART 51 08:41 r1-00337166 35946-20915 3212 arc Rig 4.1 -4060 -3918 -5296 HART 52 08:41 b1-00337166 35946-20915 3212 arc Rig 4.1 -4060 -3918 -5296 HART 53 08:46 b1-00337167 35946-20915 3212 arc Out 4.1 -4831 -4690 -6070 54 08:46 r1-00337167 35946-20915 3212 arc Out 4.1 -4831 -4690 -6070 55 08:47 b1-00337168 35946-20915 3212 fla Out 25.1 -4831 -4690 -6070 56 08:47 r1-00337168 35946-20915 3212 fla Out 25.1 -4831 -4690 -6070 57 08:51 b1-00337169 35946-20915 3212 sci Out 300.1 -4831 -4690 -6070 58 08:51 r1-00337169 35946-20915 3212 sci Out 300.1 -4831 -4690 -6070 59 09:20 b1-00337170 35948-20917 3214 arc Lef 4.1 -4831 -4690 -6070 HART 60 09:20 r1-00337170 35948-20917 3214 arc Lef 4.1 -4831 -4690 -6070 HART 61 09:21 b1-00337171 35948-20917 3214 arc Rig 4.1 -4831 -4690 -6070 HART 62 09:21 r1-00337171 35948-20917 3214 arc Rig 4.1 -4831 -4690 -6070 HART 63 09:26 r1-00337172 35948-20917 3214 arc Out 4.1 -5280 -5139 -6520 64 09:26 b1-00337172 35948-20917 3214 arc Out 4.1 -5280 -5139 -6520 65 09:28 r1-00337173 35948-20917 3214 fla Out 25.1 -5280 -5139 -6520 66 09:28 b1-00337173 35948-20917 3214 fla Out 25.1 -5280 -5139 -6520 67 09:31 b1-00337174 35948-20917 3214 sci Out 300.1 -5280 -5139 -6520 68 09:31 r1-00337174 35948-20917 3214 sci Out 300.1 -5280 -5139 -6520 69 09:40 b1-00337175 35950-20919 3215 arc Lef 4.1 -5280 -5139 -6520 HART 70 09:40 r1-00337175 35950-20919 3215 arc Lef 4.1 -5280 -5139 -6520 HART 71 09:40 b1-00337176 35950-20919 3215 arc Rig 4.1 -5280 -5139 -6520 HART 72 09:40 r1-00337176 35950-20919 3215 arc Rig 4.1 -5280 -5139 -6520 HART 73 09:47 b1-00337177 35950-20919 3215 arc Out 4.1 -5733 -5592 -6974 74 09:47 r1-00337177 35950-20919 3215 arc Out 4.1 -5733 -5592 -6974 75 09:48 r1-00337178 35950-20919 3215 fla Out 25.1 -5733 -5592 -6974 76 09:48 b1-00337178 35950-20919 3215 fla Out 25.1 -5733 -5592 -6974 77 09:52 r1-00337179 35950-20919 3215 sci Out 300.1 -5733 -5592 -6974 78 09:52 b1-00337179 35950-20919 3215 sci Out 300.1 -5733 -5592 -6974 79 12:38 r1-00337180 ----------- ---- bia Clo 0.0 -5733 -5592 -6974 80 12:38 b1-00337180 ----------- ---- bia Clo 0.0 -5733 -5592 -6974