Subject: 2.5m obslog 20220224 (SJD 59635) From: asimmons@apo.nmsu.edu Date: Fri, 25 Feb 2022 11:47:20 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday February 24, 2022 (SJD 59635) ---- Observing Team ---- Karen Kinemuchi Audrey Oravetz ---- Observing Plan ---- https://wiki.sdss.org/pages/viewpage.action?spaceKey=COM&title=Commissioning+Nightly+Plan ---- Observing Summary ---- Clear for the first half of the night, with not the greatest seeing (mostly around 2.0"). Second half was partly to mostly cloudy with very high seeing (2-6"). The last two hours were lost completely due to weather. *Design 16716, Configuration 3706, Field 19942 B: 1x900s, A: 1xAB(43) gotoField total overhead: 34.0 mins [incl. 27 mins of robot collision triage of 208 and 184, fps_snapshot_59635_0001.pdf - fps_snapshot_59635_0012.pdf, separate occasions] Slew/prepare/reconfigure - 0.8 mins [not incl. the robot problem and the slew] Hartmanns - 1.8 mins Cals (combined arc/flat) - 3.5 mins FVC loop - 3.8 mins *Design 4791, Configuration 3707, Field 17049 B: 1x900s, A: 1xAB(43) gotoField total overhead: 10.0 mins [no problems] Slew/prepare/reconfigure - 1.4 mins Hartmanns, arc, flat - SAME AS BEFORE FVC loop - 2.9 mins *Design 36183, Configurations (cloned) 3708-3714, 3716-3717, Field 21152 B: 9x900s, A: 9xAB(43) (one bonus exposure due to the netburner issue) gotoField total overhead: 9.6 mins [no problems] Slew/prepare/reconfigure - 2.5 mins Hartmanns, arc, flat - SAME AS BEFORE FVC loop - 3.1 mins * Designs 26127-30, Configurations 3718-21, Field 16163 (POOR SIGNAL DUE TO WEATHER) B: 4x900s, A: 4xAB(43) gotoField total overhead: 11.5 mins [minor unwinding problem during transition] Slew/prepare/reconfigure - 3.25 mins Hartmanns, arc, flat - SAME AS BEFORE FVC loop - 3.25 mins * Designs 27519-22, Configurations 3722-25, Field 16164 B: 4x900s, A: 4xAB(43) gotoField total overhead: 8.5 mins [minor unwinding problem during transition] Slew/prepare/reconfigure - 2.0 mins Hartmanns, arc, flat - SAME AS BEFORE FVC loop - 2.8 mins * Design 34479, Configuration 3726, Field 16169 No observations. Too cloudy. Notes: ====== * Robots 208 and 184 gave us trouble for design 16716. We disabled them and kept them disabled the rest of the night (without any additional trouble). We will need to establish a protocol going forward how to handle problematic robots on fields. Should the robots be re-enabled for any subsequent fields with the risk that we may need to spend time triaging them again? * One netburner problem during a BOSS readout: 04:26:32 "sp1daq readout failed: timed out. We followed the procedure back from SDSSIV to successfully recover the image 337963 to sdss5-hub:/home/sdss5 and updated the procedure on the wiki. 12 minutes was lost to this problem. The image was checked after we took a bonus exposure, and everything looked normal. * The configurations for design 36183 were all cloned, except the sequence skipped configuration 3715 for some odd reason. There was no "design 36183 matches previously loaded design 36183. Cloning configuration" message given around 06:11:11, and instead the routine starting calculating new trajectories for 3716. The robots were never moved. Same thing happened for config 3721 with design 26127, and config 3724 for design 27521. * Frequent BOSON crashes. ---- Observing Log ---- 22:43Z Clear on site AFTERNOON CHECKOUT: =================== 23:10Z Evening cals stated. Grabbed mirror numbers - ok BOSS Spectrograph cals: 23:17:47 boss exposure 0.0, file=337938 23:19:20 boss exposure 20.0, file=337939 BOSS Collimation: 23:21:53 boss exposure 4.0, file=337940 23:22:34 boss exposure 4.0, file=337941 23:23:18 sp1.motor.move= -1270, -1269, -1267 23:23:18 Hartmann collimate output sp1: offset: r= -1319 (Out of focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -1270 sp1: pred. resid: r= -49, b= -0.2, txt= OK, spTemp = 2.4 ---------- motorPosition sp1: -6723, -6579, -7962; ---------- Median spectrographs temperatures: Tsp1= 2.4 Outside air temperature= -1.9; Blue ring was not adjusted - it is in spec. 23:24:11 boss exposure 4.0, file=337942 23:26:00 boss exposure 25.0, file=337943 00:11Z Telescope ready. Night: ====== 01:13Z - Opened enclosure and filled the spectrograph. ====== Design 16716, Configuration 3702, Field 19942 02:00:25 CmdStarted: APO.apo 27 hal goto-field --guider-time 15 02:00:56 .jaeger 0 jaeger f text="A collision was detected in positioner 208. Sending SEND_TRAJECTORY_ABORT and locking the FPS." Variety of explode, jaeger unwind --force, etc was used to get back to home. 02:05:54 CmdDone: OBSERVER.karen 123 jaeger unwind Execute. 02:06:37 .jaeger 0 jaeger f text="A collision was detected in positioner 208. Sending SEND_TRAJECTORY_ABORT and locking the FPS." Variety of explode, jaeger unwind --force, etc was used to get back to home. 02:10:07 CmdDone: OBSERVER.karen 133 jaeger unwind 02:15:38 .jaeger 0 jaeger w text="Mismatch between number of positioners and trajectory. (JaegerUserWarning)" 02:15:38 APO.apo.jaeger 1953 jaeger f error="Trajectory validation failed." Trying a different configuration: 02:10:37 CmdStarted: OBSERVER.karen 134 jaeger configuration load --no-clone 16716 Execute. 02:12:43 .jaeger 0 jaeger f text="A collision was detected in positioner 208. Sending SEND_TRAJECTORY_ABORT and locking the FPS." Nope. Didn't work. Unwinding again and disabling the offending robot. 02:15:01 CmdDone: OBSERVER.karen 136 jaeger unwind 02:15:18 CmdStarted: OBSERVER.karen 137 jaeger disable 208 Trying again with another configuration: 02:17:10 CmdDone: OBSERVER.karen 138 jaeger configuration load --no-clone 16716 Execute. 02:17:55 .jaeger 0 jaeger f text="A collision was detected in positioner 184. Sending SEND_TRAJECTORY_ABORT and locking the FPS." With 208 disabled, now the notorious 184 collision has reappeared. Trying the procedure again. 02:20:47 CmdDone: OBSERVER.karen 142 jaeger unwind Trying again with a different configuration: 02:20:54 CmdStarted: OBSERVER.karen 143 jaeger configuration load --no-clone 16716 Execute. 02:23:00 .jaeger 0 jaeger f text="A collision was detected in positioner 184. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 02:24:46 CmdDone: OBSERVER.karen 146 jaeger unwind 02:25:41 CmdStarted: OBSERVER.karen 147 jaeger disable 184 02:25:46 CmdStarted: OBSERVER.karen 148 jaeger configuration load --no-clone 16716 configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59635/configuration_snapshot_59635_3706.pdf 02:26:44 OBSERVER.karen 148 jaeger : text="Configuration 3706 loaded." *HAL WORKING FROM THIS POINT: 02:26:55 CmdStarted: APO.apo 37 hal goto-field -s reconfigure,boss_hartmann,fvc,boss_arcs,boss_flat,acquire,guide --guider-time 15 Slew/Prepare/Reconfigure 02:26:58 CmdStarted: APO.apo.jaeger 1977 jaeger configuration execute 02:27:41 CmdDone: APO.apo.jaeger 1977 jaeger configuration execute Hartmanns 02:27:41 CmdStarted: APO.apo.hartmann 1978 hartmann collimate ignoreResiduals 02:29:28 CmdDone: client.v1451.hartmann 624 boss moveColl spec=sp1 piston=2307 Cals 02:29:29 CmdStarted: APO.apo.boss 1980 boss exposure arc itime=4 noreadout 02:31:02 APO.apo.boss 1982 boss i text="Readout complete."; exposureId=337945 02:33:24 CmdStarted: APO.apo.boss 1986 boss exposure flat itime=25 noreadout 02:35:19 CmdDone: APO.apo.boss 1987 boss exposure readout FVC loop: 02:29:31 CmdStarted: APO.apo.jaeger 1981 jaeger fvc loop 02:30:12 APO.apo.jaeger 1981 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0001.fits 02:30:54 APO.apo.jaeger 1981 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0002.fits 02:31:35 APO.apo.jaeger 1981 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0003.fits 02:32:16 APO.apo.jaeger 1981 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0004.fits 02:32:57 APO.apo.jaeger 1981 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0005.fits 02:30:16 APO.apo.jaeger 1981 jaeger i text="[FVC]: RMS full fit 48.740 um." 02:30:58 APO.apo.jaeger 1981 jaeger i text="[FVC]: RMS full fit 16.130 um." 02:31:39 APO.apo.jaeger 1981 jaeger i text="[FVC]: RMS full fit 12.530 um." 02:32:20 APO.apo.jaeger 1981 jaeger i text="[FVC]: RMS full fit 10.820 um." 02:33:01 APO.apo.jaeger 1981 jaeger i text="[FVC]: RMS full fit 12.650 um." 02:33:17 CmdDone: APO.apo.jaeger 1981 jaeger fvc loop 02:26:42 Config Loaded: 16716-3706 02:27:52 sp1.motor.move= None, None, None 02:28:00 boss exposure 4.0, file=337944 02:28:41 boss exposure 4.0, file=337945 02:29:28 sp1.motor.move= 2306, 2307, 2307 02:29:28 Hartmann collimate output sp1: offset: r= 1979 (Out of focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 2307 sp1: pred. resid: r= -328, b= -1.0, txt= OK, spTemp = 0.6 02:29:48 boss exposure 4.0, file=337946 02:33:01 FVC Loop - RMS: 12.65 Delta RMS: -1.83 Image #: 0005 02:33:43 boss exposure 25.0, file=337947 02:42:35 apogee.expose=apRaw-40730010, 43, A, Object 02:42:42 boss exposure 900.0, file=337948 02:50:16 apogee.expose=apRaw-40730011, 43, B, Object 03:01:39 apogee.expose=apRaw-40730012, 5, B, DomeFlat ======= Design 4791, Configuration 3707, Field 17049 03:12:37 APO.apo.jaeger 2020 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0006.fits 03:13:21 APO.apo.jaeger 2020 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0007.fits 03:14:04 APO.apo.jaeger 2020 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0008.fits 03:14:44 APO.apo.jaeger 2020 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0009.fits 03:12:42 APO.apo.jaeger 2020 jaeger i text="[FVC]: RMS full fit 46.840 um." 03:13:26 APO.apo.jaeger 2020 jaeger i text="[FVC]: RMS full fit 15.950 um." 03:14:08 APO.apo.jaeger 2020 jaeger i text="[FVC]: RMS full fit 12.450 um." 03:14:49 APO.apo.jaeger 2020 jaeger i text="[FVC]: RMS full fit 12.530 um." 03:07:56 Config Loaded: 4791-3707 03:10:47 boss exposure 4.0, file=337949 03:11:28 boss exposure 4.0, file=337950 03:12:12 sp1.motor.move= -1265, -1263, -1264 03:12:12 Hartmann collimate output sp1: offset: r= -989 (In Focus); b= 4.8 (Out of focus) sp1: pred. move: spAverageMove= -1265 sp1: pred. resid: r= 276, b= 0.9, txt= OK, spTemp = 0.5 03:12:31 boss exposure 4.0, file=337951 03:14:49 FVC Loop - RMS: 12.53 Delta RMS: -0.08 Image #: 0009 03:15:29 boss exposure 25.0, file=337952 03:18:57 boss exposure 900.0, file=337953 03:18:59 apogee.expose=apRaw-40730013, 43, A, Object 03:26:39 apogee.expose=apRaw-40730014, 43, B, Object 03:35:41 apogee.expose=apRaw-40730015, 5, B, DomeFlat ======== Design 36183, Configuration 3708, Field 21152 03:45:48 APO.apo.jaeger 2060 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0010.fits 03:46:32 APO.apo.jaeger 2060 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0011.fits 03:47:13 APO.apo.jaeger 2060 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0012.fits 03:47:54 APO.apo.jaeger 2060 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0013.fits 03:45:52 APO.apo.jaeger 2060 jaeger i text="[FVC]: RMS full fit 45.730 um." 03:46:36 APO.apo.jaeger 2060 jaeger i text="[FVC]: RMS full fit 15.210 um." 03:47:17 APO.apo.jaeger 2060 jaeger i text="[FVC]: RMS full fit 10.020 um." 03:47:58 APO.apo.jaeger 2060 jaeger i text="[FVC]: RMS full fit 12.840 um." 03:37:49 Config Loaded: 36183-3708 03:43:57 boss exposure 4.0, file=337954 03:44:38 boss exposure 4.0, file=337955 03:45:21 sp1.motor.move= 361, 362, 364 03:45:21 Hartmann collimate output sp1: offset: r= 659 (In Focus); b= -0.2 (In Focus) sp1: pred. move: spAverageMove= 361 sp1: pred. resid: r= 297, b= 0.9, txt= OK, spTemp = 0.5 03:45:41 boss exposure 4.0, file=337956 03:47:58 FVC Loop - RMS: 12.84 Delta RMS: -2.82 Image #: 0013 03:48:39 boss exposure 25.0, file=337957 03:51:36 boss exposure 900.0, file=337958 03:51:37 apogee.expose=apRaw-40730016, 43, A, Object 03:59:19 apogee.expose=apRaw-40730017, 43, B, Object 04:06:54 Config Loaded: 36183-3709 04:10:08 boss exposure 900.0, file=337959 04:10:09 apogee.expose=apRaw-40730018, 43, A, Object 04:17:50 apogee.expose=apRaw-40730019, 43, B, Object 04:25:47 Config Loaded: 36183-3710 04:26:32 APO.apo.boss 2085 boss e text="sp1daq readout failed: timed out (>)." A 'boss exposure recover' was issued a few times, but each time still ended in the state as 'READING 0.0'. A 'boss reloadControllers' was issued, then a 'boss exposure recover'. That seemed to work. The image was successfully read out to sdss5@sdss5-boss-icc. 04:38:29 boss exposure 0.0, file=337964 04:40:24 boss exposure 900.0, file=337965 04:40:25 apogee.expose=apRaw-40730020, 43, A, Object 04:48:06 apogee.expose=apRaw-40730021, 43, B, Object 04:59:43 Config Loaded: 36183-3711 05:00:36 boss exposure 900.0, file=337966 05:00:38 apogee.expose=apRaw-40730022, 43, A, Object 05:08:18 apogee.expose=apRaw-40730023, 43, B, Object 05:16:46 Config Loaded: 36183-3712 05:17:16 boss exposure 900.0, file=337967 05:17:18 apogee.expose=apRaw-40730024, 43, A, Object 05:24:58 apogee.expose=apRaw-40730025, 43, B, Object 05:35:31 Config Loaded: 36183-3713 05:36:08 boss exposure 900.0, file=337968 05:36:10 apogee.expose=apRaw-40730026, 43, A, Object 05:43:51 apogee.expose=apRaw-40730027, 43, B, Object 05:51:38 Config Loaded: 36183-3714 05:52:39 boss exposure 900.0, file=337969 05:52:40 apogee.expose=apRaw-40730028, 43, A, Object 06:00:21 apogee.expose=apRaw-40730029, 43, B, Object 06:12:50 Config Loaded: 36183-3716 06:13:44 boss exposure 900.0, file=337970 06:13:46 apogee.expose=apRaw-40730030, 43, A, Object 06:21:27 apogee.expose=apRaw-40730031, 43, B, Object 06:30:15 Config Loaded: 36183-3717 06:30:55 boss exposure 900.0, file=337971 06:30:56 apogee.expose=apRaw-40730032, 43, A, Object 06:38:37 apogee.expose=apRaw-40730033, 43, B, Object 06:47:28 apogee.expose=apRaw-40730034, 5, B, DomeFlat ====== Design 26127, Configuration 3718, Field 16163 06:53:40 CmdStarted: APO.apo 42 hal goto-field --guider-time 15 06:53:43 APO.apo.jaeger 2152 jaeger f error="The configuration was not executed. Try unwinding." 06:53:43 CmdDone: APO.apo.jaeger 2152 jaeger configuration reverse 06:53:43 APO.apo 41 hal e error="Command jaeger configuration reverse failed." jaeger unwind - success Try again... 06:56:10 CmdStarted: APO.apo 42 hal goto-field --guider-time 15 06:59:07 APO.apo.jaeger 2165 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0014.fits 06:59:51 APO.apo.jaeger 2165 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0015.fits 07:00:32 APO.apo.jaeger 2165 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0016.fits 07:01:13 APO.apo.jaeger 2165 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0017.fits 06:59:11 APO.apo.jaeger 2165 jaeger i text="[FVC]: RMS full fit 47.200 um." 06:59:11 APO.apo.jaeger 2165 jaeger i fvc_perc_95=83.33 06:59:11 APO.apo.jaeger 2165 jaeger i fvc_percent_reached=4.7 06:59:56 APO.apo.jaeger 2165 jaeger i text="[FVC]: RMS full fit 14.680 um." 06:59:56 APO.apo.jaeger 2165 jaeger i fvc_perc_95=27.76 06:59:56 APO.apo.jaeger 2165 jaeger i fvc_percent_reached=50.0 07:00:36 APO.apo.jaeger 2165 jaeger i text="[FVC]: RMS full fit 13.460 um." 07:00:36 APO.apo.jaeger 2165 jaeger i fvc_perc_95=26.86 07:00:36 APO.apo.jaeger 2165 jaeger i fvc_percent_reached=58.7 07:01:17 APO.apo.jaeger 2165 jaeger i text="[FVC]: RMS full fit 13.160 um." 07:01:17 APO.apo.jaeger 2165 jaeger i fvc_perc_95=24.88 07:01:17 APO.apo.jaeger 2165 jaeger i fvc_percent_reached=54.2 06:52:54 Config Loaded: 26127-3718 06:57:15 boss exposure 4.0, file=337972 06:57:57 boss exposure 4.0, file=337973 06:58:40 sp1.motor.move= -1092, -1090, -1086 06:58:41 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= 6.8 (Out of focus) sp1: pred. move: spAverageMove= -1092 sp1: pred. resid: r= 1092, b= 3.4, txt= OK, spTemp = 0.0 06:59:00 boss exposure 4.0, file=337974 07:01:17 FVC Loop - RMS: 13.16 Delta RMS: 0.30 Image #: 0017 07:01:58 boss exposure 25.0, file=337975 07:05:56 boss exposure 900.0, file=337976 07:05:57 apogee.expose=apRaw-40730035, 43, A, Object 07:13:38 apogee.expose=apRaw-40730036, 43, B, Object 07:12:47 APO.apo.cherno 2177 cherno f error="Command 'acquire -c -t 15.0' failed because of an uncaught error 'ExposerError: Weights sum to zero, can't be normalized'. See traceback in the log for more information." 07:12:47 CmdDone: APO.apo.cherno 2177 cherno acquire -c -t 15.0 Lots of clouds. The S/N for these exposures are most likely poor. Seeing 3.0-6.0". 06:52:54 Config Loaded: 26127-3718 06:57:15 boss exposure 4.0, file=337972 06:57:57 boss exposure 4.0, file=337973 06:58:40 sp1.motor.move= -1092, -1090, -1086 06:58:41 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= 6.8 (Out of focus) sp1: pred. move: spAverageMove= -1092 sp1: pred. resid: r= 1092, b= 3.4, txt= OK, spTemp = 0.0 06:59:00 boss exposure 4.0, file=337974 07:01:17 FVC Loop - RMS: 13.16 Delta RMS: 0.30 Image #: 0017 07:01:58 boss exposure 25.0, file=337975 07:05:56 boss exposure 900.0, file=337976 07:05:57 apogee.expose=apRaw-40730035, 43, A, Object 07:13:38 apogee.expose=apRaw-40730036, 43, B, Object 07:21:44 Config Loaded: 26128-3719 07:23:45 boss exposure 900.0, file=337977 07:23:47 apogee.expose=apRaw-40730037, 43, A, Object 07:31:27 apogee.expose=apRaw-40730038, 43, B, Object 07:39:13 Config Loaded: 26129-3720 07:40:15 boss exposure 900.0, file=337978 07:40:17 apogee.expose=apRaw-40730039, 43, A, Object 07:47:57 apogee.expose=apRaw-40730040, 43, B, Object configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59635/configuration_snapshot_59635_3721.pdf 07:57:34 OBSERVER.karen 194 jaeger : text="Configuration 3721 loaded." 07:57:34 CmdDone: OBSERVER.karen 194 jaeger configuration load 26130 Didn't officially say that the configuration was cloned, and jaeger did some new trajectory calculations. Same behavior as before. We cannot explain in. 07:57:31 Config Loaded: 26130-3721 07:58:21 apogee.expose=apRaw-40730041, 43, A, Object 07:58:22 boss exposure 900.0, file=337979 08:06:02 apogee.expose=apRaw-40730042, 43, B, Object 08:16:37 apogee.expose=apRaw-40730043, 5, B, DomeFlat ======= Design 27519, Configuration 3722, Field 16164 08:22:09 APO.apo.jaeger 2234 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0018.fits 08:22:54 APO.apo.jaeger 2234 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0019.fits 08:23:36 APO.apo.jaeger 2234 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0020.fits 08:24:16 APO.apo.jaeger 2234 jaeger d fvc_filename=/data/fcam/59635/fimg-fvc1n-0021.fits 08:22:14 APO.apo.jaeger 2234 jaeger i text="[FVC]: RMS full fit 46.780 um." 08:22:14 APO.apo.jaeger 2234 jaeger i fvc_perc_95=80.62 08:22:14 APO.apo.jaeger 2234 jaeger i fvc_percent_reached=5.7 08:22:58 APO.apo.jaeger 2234 jaeger i text="[FVC]: RMS full fit 15.960 um." 08:22:58 APO.apo.jaeger 2234 jaeger i fvc_perc_95=28.5 08:22:58 APO.apo.jaeger 2234 jaeger i fvc_percent_reached=47.2 08:23:40 APO.apo.jaeger 2234 jaeger i text="[FVC]: RMS full fit 13.280 um." 08:23:40 APO.apo.jaeger 2234 jaeger i fvc_perc_95=23.27 08:23:40 APO.apo.jaeger 2234 jaeger i fvc_percent_reached=60.2 08:24:20 APO.apo.jaeger 2234 jaeger i text="[FVC]: RMS full fit 14.920 um." 08:24:20 APO.apo.jaeger 2234 jaeger i fvc_perc_95=26.93 08:24:20 APO.apo.jaeger 2234 jaeger i fvc_percent_reached=56.4 08:17:12 Config Loaded: 27519-3722 08:20:19 boss exposure 4.0, file=337980 08:21:01 boss exposure 4.0, file=337981 08:21:43 sp1.motor.move= -276, -276, -279 08:21:43 Hartmann collimate output sp1: offset: r= 989 (In Focus); b= 4.8 (Out of focus) sp1: pred. move: spAverageMove= -276 sp1: pred. resid: r= 1265, b= 4.0, txt= OK, spTemp = 0.0 08:22:03 boss exposure 4.0, file=337982 08:24:20 FVC Loop - RMS: 14.92 Delta RMS: -1.64 Image #: 0021 08:25:02 boss exposure 25.0, file=337983 Lighter clouds now, but windy and seeing is still atrocious at 4.0". 08:29:01 boss exposure 900.0, file=337984 08:29:03 apogee.expose=apRaw-40730044, 43, A, Object 08:36:44 apogee.expose=apRaw-40730045, 43, B, Object Focus sweep. Seeing slightly better at 1.9" now. 08:49:29 Config Loaded: 27520-3723 08:52:19 boss exposure 900.0, file=337985 08:52:21 apogee.expose=apRaw-40730046, 43, A, Object 09:00:01 apogee.expose=apRaw-40730047, 43, B, Object 09:07:40 Config Loaded: 27521-3724 09:08:47 boss exposure 900.0, file=337986 09:08:48 apogee.expose=apRaw-40730048, 43, A, Object 09:16:29 apogee.expose=apRaw-40730049, 43, B, Object 09:26:54 Config Loaded: 27522-3725 09:27:33 boss exposure 900.0, file=337987 09:27:34 apogee.expose=apRaw-40730050, 43, A, Object 09:35:15 apogee.expose=apRaw-40730051, 43, B, Object 09:49:05 apogee.expose=apRaw-40730052, 5, B, DomeFlat ====== Design 34479, Configuration 3726, Field 16169 09:51:25 APO.apo.jaeger 2288 jaeger f error="The configuration was not executed. Try unwinding." 09:51:25 CmdDone: APO.apo.jaeger 2288 jaeger configuration reverse 09:51:25 APO.apo 38 hal e error="Command jaeger configuration reverse failed." 09:51:26 .mcp 0 mcp i command="move" 09:51:52 .tcc 0 tcc i AxisCmdState=Slewing,Slewing,Slewing; AxisErrCode=OK,OK,OK; AxisNoTrack=FFF 09:53:24 .mcp 0 mcp i trace="altitude reposition caused frame break" 09:53:24 .mcp 0 mcp ! command="move" 09:53:24 .mcp 0 mcp i trace="azimuth reposition caused frame break" 09:53:24 .mcp 0 mcp ! command="move" 09:53:24 .mcp 0 mcp i trace="rotator reposition caused frame break" 09:51:56 .tcc 0 tcc w BadAltStatus; AltStat=36.519848,0.000000,5152499516.45335,0x00001800; AltDTime=0.04 09:52:46 .tcc 0 tcc w BadRotStatus; RotStat=143.484198,0.000000,5152499566.68766,0x00000800; RotDTime=0.04 09:53:26 .tcc 0 tcc w BadAzStatus; AzStat=-65.403766,0.000000,5152499606.87345,0x00001800; AzDTime=0.01 09:55Z - Not proceeding with observations since the clouds are too thick. We will wait for an opening. 11:44Z - An opening never came. We closed and started cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0020 0.0040 Axial B: 0.0030 0.0030 Axial C: -0.1100 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: -2.67 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6700 -9.6710 Axial C: -9.6950 -9.6970 Trans D: 0.1060 0.1140 GALILS Commanded: 4897100 4898300 4917000 -50 -3450 Actual: 4859173 4859243 4859274 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.01 -0.02 0.02 0.01 42.88 ---- Telescope Status ---- 00:00Z Started ventilation 00:45Z encl25m : open 11:44Z encl25m : closed No interlocks bypassed Status at: 11:44Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: fps Counterweights at: 685 687 705 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 180 lbs, 21 psi ---- Software Used ---- Get Versions, 02/24/22, 23:05:20 -- STUI software: STUI : 1.7.17b1 alerts : 2.2.0 boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED apo : 3.0.1 fliswarm : 0.2.1-alpha.0 jaeger : 0.14.1-alpha.0 kaiju : 1.2.2-alpha.1 coordio : 1.2.2-alpha.1 fps_calibrati : APO.2021.12.25+960cb05 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : FAILED STUIAdditions : 3.1.12 -- done -- --------------Other Versions-------------- sdss-obstools : 3.7.10 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V34 Fiducials : v1_148 idlspec2d : v6_0_5 Kronos : 0.1.0 roboscheduler : 0.11.1 /dev/sdb1 2.0T 161G 1.9T 8% /home 10.25.1.230:/data 21T 18T 2.6T 88% /data ---- APOGEE Data Summary ---- APOGEE data list, mjd=59635, all raw_data: /data/apogee/utr_cdr/59635/apRaw-*.fits archive: /data/apogee/archive/59635/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59635/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59635 1 23:06 40730001 Dark 3 A 13.0 ---------- a-b-c - - 59635 2 23:21 40730002 Dark 60 A 13.0 ---------- a-b-c - - 59635 3 23:23 40730003 QuarFlat 10 A 13.0 ---------- a-b-c - 97 59635 4 23:25 40730004 Arc-Thar 12 B 13.5 ---------- a-b-c 1.55 - 59635 5 23:33 40730005 Arc-Une 40 B 13.5 ---------- a-b-c - - 59635 6 23:35 40730006 Arc-Thar 12 A 13.0 ---------- a-b-c 0.97 - 59635 7 23:42 40730007 Arc-Une 40 A 13.0 ---------- a-b-c - - 59635 8 23:53 40730008 Dark 60 A 13.0 ---------- a-b-c - - 59635 9 23:59 40730009 Arc---- 30 A 13.0 ---------- a-b-c - - FPI 59635 10 02:49 40730010 Object 43 A 13.0 16716-3706 a-b-c 0.94 - 59635 11 02:57 40730011 Object 43 B 13.5 16716-3706 a-b-c 1.43 - 59635 12 03:02 40730012 DomeFlat 5 B 13.5 16716-3706 a-b-c - 145 59635 13 03:26 40730013 Object 43 A 13.0 4791-3707 a-b-c 0.94 - 59635 14 03:33 40730014 Object 43 B 13.5 4791-3707 a-b-c 1.43 - 59635 15 03:36 40730015 DomeFlat 5 B 13.5 4791-3707 a-b-c - 144 59635 16 03:58 40730016 Object 43 A 13.0 36183-3708 a-b-c 0.93 - 59635 17 04:06 40730017 Object 43 B 13.5 36183-3708 a-b-c 1.42 - 59635 18 04:17 40730018 Object 43 A 13.0 36183-3709 a-b-c 0.94 - 59635 19 04:25 40730019 Object 43 B 13.5 36183-3709 a-b-c 1.43 - 59635 20 04:47 40730020 Object 43 A 13.0 36183-3710 a-b-c 0.94 - 59635 21 04:55 40730021 Object 43 B 13.5 36183-3710 a-b-c 1.43 - 59635 22 05:07 40730022 Object 43 A 13.0 36183-3711 a-b-c 0.94 - 59635 23 05:15 40730023 Object 43 B 13.5 36183-3711 a-b-c 1.43 - 59635 24 05:24 40730024 Object 43 A 13.0 36183-3712 a-b-c 0.93 - 59635 25 05:32 40730025 Object 43 B 13.5 36183-3712 a-b-c 1.43 - 59635 26 05:43 40730026 Object 43 A 13.0 36183-3713 a-b-c 0.94 - 59635 27 05:51 40730027 Object 43 B 13.5 36183-3713 a-b-c 1.43 - 59635 28 05:59 40730028 Object 43 A 13.0 36183-3714 a-b-c 0.94 - 59635 29 06:07 40730029 Object 43 B 13.5 36183-3714 a-b-c 1.43 - 59635 30 06:20 40730030 Object 43 A 13.0 36183-3716 a-b-c 0.94 - 59635 31 06:28 40730031 Object 43 B 13.5 36183-3716 a-b-c 1.43 - 59635 32 06:38 40730032 Object 43 A 13.0 36183-3717 a-b-c 0.94 - 59635 33 06:45 40730033 Object 43 B 13.5 36183-3717 a-b-c 1.44 - 59635 34 06:47 40730034 DomeFlat 5 B 13.5 36183-3717 a-b-c - 146 59635 35 07:13 40730035 Object 43 A 13.0 26127-3718 a-b-c 0.94 - 59635 36 07:20 40730036 Object 43 B 13.5 26127-3718 a-b-c 1.44 - 59635 37 07:31 40730037 Object 43 A 13.0 26128-3719 a-b-c 0.95 - 59635 38 07:38 40730038 Object 43 B 13.5 26128-3719 a-b-c 1.44 - 59635 39 07:47 40730039 Object 43 A 13.0 26129-3720 a-b-c 0.95 - 59635 40 07:55 40730040 Object 43 B 13.5 26129-3720 a-b-c 1.44 - 59635 41 08:05 40730041 Object 43 A 13.0 26130-3721 a-b-c 0.94 - 59635 42 08:13 40730042 Object 43 B 13.5 26130-3721 a-b-c 1.44 - 59635 43 08:17 40730043 DomeFlat 5 B 13.5 27519-3722 a-b-c - 145 59635 44 08:36 40730044 Object 43 A 13.0 27519-3722 a-b-c 0.95 - 59635 45 08:43 40730045 Object 43 B 13.5 27519-3722 a-b-c 1.44 - 59635 46 08:59 40730046 Object 43 A 13.0 27520-3723 a-b-c 0.95 - 59635 47 09:07 40730047 Object 43 B 13.5 27520-3723 a-b-c 1.44 - 59635 48 09:16 40730048 Object 43 A 13.0 27521-3724 a-b-c 0.95 - 59635 49 09:23 40730049 Object 43 B 13.5 27521-3724 a-b-c 1.44 - 59635 50 09:34 40730050 Object 43 A 13.0 27522-3725 a-b-c 0.95 - 59635 51 09:42 40730051 Object 43 B 13.5 27522-3725 a-b-c 1.44 - 59635 52 09:49 40730052 DomeFlat 5 B 13.5 27522-3725 a-b-c - 142 cannot read file : /data/apogee/utr_cdr/59635/apRaw-40730053.fits ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59635 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:17 b1-00337938 ----------- ---- bia Clo 0.0 -5453 -5310 -6695 2 23:17 r1-00337938 ----------- ---- bia Clo 0.0 -5453 -5310 -6695 3 23:19 b1-00337939 ----------- ---- dar Clo 20.0 -5453 -5310 -6695 4 23:19 r1-00337939 ----------- ---- dar Clo 20.0 -5453 -5310 -6695 5 23:21 b1-00337940 6597-17444 3701 arc Lef 4.1 -5453 -5310 -6695 HART 6 23:21 r1-00337940 6597-17444 3701 arc Lef 4.1 -5453 -5310 -6695 HART 7 23:22 b1-00337941 6597-17444 3701 arc Rig 4.1 -5453 -5310 -6695 HART 8 23:22 r1-00337941 6597-17444 3701 arc Rig 4.1 -5453 -5310 -6695 HART 9 23:24 b1-00337942 6597-17444 3701 arc Out 4.1 -6723 -6579 -7962 10 23:24 r1-00337942 6597-17444 3701 arc Out 4.1 -6723 -6579 -7962 11 23:26 b1-00337943 6597-17444 3701 fla Out 25.1 -6723 -6579 -7962 12 23:26 r1-00337943 6597-17444 3701 fla Out 25.1 -6723 -6579 -7962 13 02:28 r1-00337944 16716-19942 3706 arc Lef 4.1 -6723 -6579 -7962 HART 14 02:28 b1-00337944 16716-19942 3706 arc Lef 4.1 -6723 -6579 -7962 HART 15 02:28 r1-00337945 16716-19942 3706 arc Rig 4.1 -6723 -6579 -7962 HART 16 02:28 b1-00337945 16716-19942 3706 arc Rig 4.1 -6723 -6579 -7962 HART 17 02:29 b1-00337946 16716-19942 3706 arc Out 4.1 -4417 -4272 -5655 18 02:29 r1-00337946 16716-19942 3706 arc Out 4.1 -4417 -4272 -5655 19 02:33 r1-00337947 16716-19942 3706 fla Out 25.1 -4417 -4272 -5655 20 02:33 b1-00337947 16716-19942 3706 fla Out 25.1 -4417 -4272 -5655 21 02:42 r1-00337948 16716-19942 3706 sci Out 900.1 -4417 -4272 -5655 22 02:42 b1-00337948 16716-19942 3706 sci Out 900.1 -4417 -4272 -5655 23 03:10 b1-00337949 4791-17049 3707 arc Lef 4.1 -4417 -4272 -5655 HART 24 03:10 r1-00337949 4791-17049 3707 arc Lef 4.1 -4417 -4272 -5655 HART 25 03:11 b1-00337950 4791-17049 3707 arc Rig 4.1 -4417 -4272 -5655 HART 26 03:11 r1-00337950 4791-17049 3707 arc Rig 4.1 -4417 -4272 -5655 HART 27 03:12 r1-00337951 4791-17049 3707 arc Out 4.1 -5682 -5535 -6919 28 03:12 b1-00337951 4791-17049 3707 arc Out 4.1 -5682 -5535 -6919 29 03:15 b1-00337952 4791-17049 3707 fla Out 25.1 -5682 -5535 -6919 30 03:15 r1-00337952 4791-17049 3707 fla Out 25.1 -5682 -5535 -6919 31 03:18 b1-00337953 4791-17049 3707 sci Out 900.1 -5682 -5535 -6919 32 03:18 r1-00337953 4791-17049 3707 sci Out 900.1 -5682 -5535 -6919 33 03:43 b1-00337954 36183-21152 3708 arc Lef 4.1 -5682 -5535 -6919 HART 34 03:43 r1-00337954 36183-21152 3708 arc Lef 4.1 -5682 -5535 -6919 HART 35 03:44 b1-00337955 36183-21152 3708 arc Rig 4.1 -5682 -5535 -6919 HART 36 03:44 r1-00337955 36183-21152 3708 arc Rig 4.1 -5682 -5535 -6919 HART 37 03:45 b1-00337956 36183-21152 3708 arc Out 4.1 -5321 -5173 -6555 38 03:45 r1-00337956 36183-21152 3708 arc Out 4.1 -5321 -5173 -6555 39 03:48 b1-00337957 36183-21152 3708 fla Out 25.1 -5321 -5173 -6555 40 03:48 r1-00337957 36183-21152 3708 fla Out 25.1 -5321 -5173 -6555 41 03:51 b1-00337958 36183-21152 3708 sci Out 900.1 -5321 -5173 -6555 42 03:51 r1-00337958 36183-21152 3708 sci Out 900.1 -5321 -5173 -6555 43 03:51 b1-00337958 36183-21152 3708 sci Out 900.1 -5321 -5173 -6555 44 04:10 b1-00337963 36183-21152 3709 sci Out 900.1 -5321 -5173 -6555 45 04:10 r1-00337963 36183-21152 3709 sci Out 900.1 -5321 -5173 -6555 46 04:10 r1-00337963 36183-21152 3709 sci Out 900.1 -5321 -5173 -6555 47 04:10 b1-00337963 36183-21152 3709 sci Out 900.1 -5321 -5173 -6555 48 04:38 r1-00337964 ----------- ---- bia Clo 0.0 -5321 -5173 -6555 49 04:38 b1-00337964 ----------- ---- bia Clo 0.0 -5321 -5173 -6555 50 04:40 r1-00337965 36183-21152 3710 sci Out 900.1 -5321 -5173 -6555 51 04:40 b1-00337965 36183-21152 3710 sci Out 900.1 -5321 -5173 -6555 52 05:00 r1-00337966 36183-21152 3711 sci Out 900.1 -5321 -5173 -6555 53 05:00 b1-00337966 36183-21152 3711 sci Out 900.1 -5321 -5173 -6555 54 05:17 b1-00337967 36183-21152 3712 sci Out 900.1 -5321 -5173 -6555 55 05:17 r1-00337967 36183-21152 3712 sci Out 900.1 -5321 -5173 -6555 56 05:36 b1-00337968 36183-21152 3713 sci Out 900.1 -5321 -5173 -6555 57 05:36 r1-00337968 36183-21152 3713 sci Out 900.1 -5321 -5173 -6555 58 05:52 b1-00337969 36183-21152 3714 sci Out 900.1 -5321 -5173 -6555 59 05:52 r1-00337969 36183-21152 3714 sci Out 900.1 -5321 -5173 -6555 60 06:13 r1-00337970 36183-21152 3716 sci Out 900.1 -5321 -5173 -6555 61 06:13 b1-00337970 36183-21152 3716 sci Out 900.1 -5321 -5173 -6555 62 06:30 b1-00337971 36183-21152 3717 sci Out 900.1 -5321 -5173 -6555 63 06:30 r1-00337971 36183-21152 3717 sci Out 900.1 -5321 -5173 -6555 64 06:57 r1-00337972 26127-16163 3718 arc Lef 4.1 -5321 -5173 -6555 HART 65 06:57 b1-00337972 26127-16163 3718 arc Lef 4.1 -5321 -5173 -6555 HART 66 06:57 r1-00337973 26127-16163 3718 arc Rig 4.1 -5321 -5173 -6555 HART 67 06:57 b1-00337973 26127-16163 3718 arc Rig 4.1 -5321 -5173 -6555 HART 68 06:59 r1-00337974 26127-16163 3718 arc Out 4.1 -6413 -6263 -7641 69 06:59 b1-00337974 26127-16163 3718 arc Out 4.1 -6413 -6263 -7641 70 07:01 b1-00337975 26127-16163 3718 fla Out 25.1 -6413 -6263 -7641 71 07:01 r1-00337975 26127-16163 3718 fla Out 25.1 -6413 -6263 -7641 72 07:05 b1-00337976 26127-16163 3718 sci Out 900.1 -6413 -6263 -7641 73 07:05 r1-00337976 26127-16163 3718 sci Out 900.1 -6413 -6263 -7641 74 07:23 b1-00337977 26128-16163 3719 sci Out 900.1 -6413 -6263 -7641 75 07:23 r1-00337977 26128-16163 3719 sci Out 900.1 -6413 -6263 -7641 76 07:40 r1-00337978 26129-16163 3720 sci Out 900.1 -6413 -6263 -7641 77 07:40 b1-00337978 26129-16163 3720 sci Out 900.1 -6413 -6263 -7641 78 07:58 b1-00337979 26130-16163 3721 sci Out 900.1 -6413 -6263 -7641 79 07:58 r1-00337979 26130-16163 3721 sci Out 900.1 -6413 -6263 -7641 80 08:20 b1-00337980 27519-16164 3722 arc Lef 4.1 -6413 -6263 -7641 HART 81 08:20 r1-00337980 27519-16164 3722 arc Lef 4.1 -6413 -6263 -7641 HART 82 08:21 r1-00337981 27519-16164 3722 arc Rig 4.1 -6413 -6263 -7641 HART 83 08:21 b1-00337981 27519-16164 3722 arc Rig 4.1 -6413 -6263 -7641 HART 84 08:22 b1-00337982 27519-16164 3722 arc Out 4.1 -6689 -6539 -7920 85 08:22 r1-00337982 27519-16164 3722 arc Out 4.1 -6689 -6539 -7920 86 08:25 b1-00337983 27519-16164 3722 fla Out 25.1 -6689 -6539 -7920 87 08:25 r1-00337983 27519-16164 3722 fla Out 25.1 -6689 -6539 -7920 88 08:29 b1-00337984 27519-16164 3722 sci Out 900.1 -6689 -6539 -7920 89 08:29 r1-00337984 27519-16164 3722 sci Out 900.1 -6689 -6539 -7920 90 08:52 b1-00337985 27520-16164 3723 sci Out 900.1 -6689 -6539 -7920 91 08:52 r1-00337985 27520-16164 3723 sci Out 900.1 -6689 -6539 -7920 92 09:08 b1-00337986 27521-16164 3724 sci Out 900.1 -6689 -6539 -7920 93 09:08 r1-00337986 27521-16164 3724 sci Out 900.1 -6689 -6539 -7920 94 09:27 b1-00337987 27522-16164 3725 sci Out 900.1 -6689 -6539 -7920 95 09:27 r1-00337987 27522-16164 3725 sci Out 900.1 -6689 -6539 -7920 96 11:41 b1-00337988 ----------- ---- bia Clo 0.0 -6689 -6539 -7920 97 11:41 r1-00337988 ----------- ---- bia Clo 0.0 -6689 -6539 -7920 98 11:43 r1-00337989 ----------- ---- bia Clo 0.0 -6689 -6539 -7920 99 11:43 b1-00337989 ----------- ---- bia Clo 0.0 -6689 -6539 -7920 100 11:44 b1-00337990 ----------- ---- bia Clo 0.0 -6689 -6539 -7920 101 11:44 r1-00337990 ----------- ---- bia Clo 0.0 -6689 -6539 -7920 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 23:23:10 1.8 -1319 3.8 -1270 -0.2 02:29:20 -0.0 1979 -8.3 2307 -1.0 03:12:00 -0.2 -989 4.8 -1265 0.9 03:45:10 -0.1 659 -0.2 361 0.9 06:58:30 0.0 0 6.8 -1092 3.4 08:21:40 -0.0 989 4.8 -276 4.0