Subject: 2.5m obslog 20220215 (SJD 59626) From: kinemuchi@apo.nmsu.edu Date: Wed, 16 Feb 2022 13:13:09 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday February 15, 2022 (SJD 59626) ---- Observing Team ---- Dan Oravetz Karen Kinemuchi Jose Sanchez-Gallego John Donor ---- Observing Plan ---- https://wiki.sdss.org/pages/viewpage.action?spaceKey=COM&title=Commissioning+Nightly+Plan See numbered items in the log. ---- Observing Summary ---- Lost 3 hours due to weather (high wind, high dust, clouds) Opened at 5:30Z to clouds and periodic high wind gusts Observed: ========= Design 13029 Config id 3503; FVC images #1-2; no data; work on autofocus Design 721 Config id 3504; FVC images #3-4; no data Design 4678 Config id 3505-3506; FVC images #5-9; 1x900s + 1xAB (under config 3506) Design 10489 Config id 3507; FVC images #10-12; 1x900s + 1xAB Design 4583 Config id 3508; FVC images #13-16; 1x900s + 1xAB Design 4582 Config id 3509; FVC images #17-20; 1x900s + 1xAB Design 35989 Config id 3511-3523; FVC images #21-36; 12pt-dither sequence for APOGEE design. NOTES TO DAY STAFF AND DEVELOPERS: ================================== * Database is reporting low/zero S/N for APOGEE exposures. * We discovered that at the end of the hal domeflat sequence the apogee inst shutter closes. We have modified the APOGEE dither point sequence script on the wiki so that it will check at each iteration if the inst shutter is open. * See Problems in Details for error messages from the python scripts for gfa focus. * Dust counts ~10000 ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ Clear, but wind and dust are high and trending higher. All systems alive and responsive. This morning's APOGEE cals look fine. APOGEE evening cals and BOSS cals taken. Positioners are homed: 23:16:30 CmdStarted: APO.doravetz_2 25 jaeger unwind --status 23:16:31 APO.doravetz_2 25 jaeger i folded=T 23:16:31 APO.doravetz_2 25 jaeger : text="All positioners are folded." Ventilation delayed due to high wind and dust -- no sense in exposing things to rising dust if the wind will prevent opening. ============================================================================ NIGHT ============================================================================ 00:37Z Delaying opening due to high wind gust and rising dust counts. 04:33Z Wind and dust are decreasing but now it is cloudy. We remain closed. 04:59Z Cloudy but clearing. Ventilation begins. Doing a force fill on BOSS. 05:23:23 encl25m : open; 1. and 2. Jose is on site and might want to do some work on focus / autofocus John D. is also on site, so if Jose and John are up, observe designs from the Kronos queue as if we were in normal operations, and work together to get good estimates of efficiency and identify areas for potential improvement 05:30Z Setting up Kronos and loaded onto Hal. Loaded Design 13029 (Field 19185) Configuration ID 3503 05:32:50 Config Loaded: 13029-3503 Hartmann: 05:35:44 boss exposure 4.0, file=337564 05:36:25 boss exposure 4.0, file=337565 05:37:24 sp1.motor.move= 4256, 4256, 4255 05:37:24 Hartmann collimate output sp1: offset: r= 3628 (Out of focus); b= -15.3 (Out of focus) sp1: pred. move: spAverageMove= 4256 sp1: pred. resid: r= -628, b= -2.0, txt= OK, spTemp = 3.8 05:38:13 RMS full fit 42.840 um." 05:38:56 RMS full fit 88.600 um." 05:38:08 APO.doravetz.jaeger 465 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0001.fits 05:38:52 APO.doravetz.jaeger 465 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0002.fits 05:39:14 APO.doravetz 26 hal e error="FVC loop failed. RMS=88.6." 05:39:14 APO.doravetz 26 hal w text="Running cleanup tasks." 05:39:14 APO.doravetz 26 hal f 05:39:14 CmdDone: APO.doravetz 26 hal goto-field --guider-time 15 Accepted the RMS result and restarting the Hal command to get the rest of the BOSS cals. 05:38:56 FVC Loop - RMS: 88.60 Delta RMS: -45.76 Image #: 0002 BOSS cals: 05:42:48 boss exposure 4.0, file=337566 05:44:22 boss exposure 25.0, file=337567 05:44:51 CmdStarted: APO.doravetz.tcc 480 tcc track 106.642324577854, 73.0109419327394 icrs /rottype=object/rotang=50.9199/rotwrap=mid 05:45:15 CmdStarted: APO.doravetz_2 3 fliswarm reconnect 05:45:12 cherno.cherno 8297 fliswarm w text=gfa5,"no cameras connected." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node gfa6 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node gfa4 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node fvc is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node gfa5 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node gfa1 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node gfa3 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:16 APO.doravetz_2 3 fliswarm w text="Node gfa2 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 05:45:52 APO.doravetz.cherno 483 cherno f error="Acquisition failed: Timed out waiting for the exposure to finish." 05:45:52 CmdDone: APO.doravetz.cherno 483 cherno acquire -t 15.0 --full 05:47:35 APO.jose_boson 82 cherno w help="UsageError: Got unexpected extra argument (--stop)" 05:47:35 APO.jose_boson 82 cherno f error="Command 'acquire --stop' failed." 05:47:35 CmdDone: APO.jose_boson 82 cherno acquire --stop 05:49:44 CmdDone: APO.doravetz 29 hal goto-field -s acquire,guide --guider-time 15 05:51Z Stopping guiding and reloading the cherno. Clouds are back. 06:00:18 APO.Jose 3 cherno w text="Camera gfa3: not enough sources." 06:00:18 APO.Jose 3 cherno w text="Camera gfa4: not enough sources." 06:00:18 APO.Jose 3 cherno w text="Camera gfa2: not enough sources." 06:00:18 APO.Jose 3 cherno w text="Camera gfa5: not enough sources." 06:00:18 APO.Jose 3 cherno w text="Camera gfa6: not enough sources." 06:00:22 APO.Jose 3 cherno e acquisition_valid=F; did_correct=F 06:00:22 APO.Jose 3 cherno f error="Acquisition failed: Floating point nan values are not allowed in FITS headers." 06:00:22 CmdDone: APO.Jose 3 cherno acquire 06:08:04 CmdStarted: APO.Jose 5 cherno acquire --help 06:08:04 APO.Jose 5 cherno > 06:08:04 APO.Jose 5 cherno w help="Usage: cherno acquire [OPTIONS]" 06:08:04 APO.Jose 5 cherno w help 06:08:04 APO.Jose 5 cherno w help=" Runs the acquisition procedure." 06:08:04 APO.Jose 5 cherno w help 06:08:04 APO.Jose 5 cherno w help=Options: 06:08:04 APO.Jose 5 cherno w help=" -t, --exposure-time FLOAT Cameras exposure time." 06:08:04 APO.Jose 5 cherno w help=" -m, --cameras TEXT Comma-separated cameras to expose." 06:08:04 APO.Jose 5 cherno w help=" -c, --continuous Run acquisition in continuous mode." 06:08:04 APO.Jose 5 cherno w help=" --apply / --no-apply Whether to apply the correction." 06:08:04 APO.Jose 5 cherno w help=" -f, --full Applies the full correction once. Cannot be used" 06:08:04 APO.Jose 5 cherno w help=" with --continuous." 06:08:04 APO.Jose 5 cherno w help=" --help Show this message and exit." 06:08:04 APO.Jose 5 cherno : 06:08:04 CmdDone: APO.Jose 5 cherno acquire --help Doing a focus sweep. On obs2, Dylan's new script works but not on obs1. See Problems in Detail for error messages from Dylan's python scripts for gfa focus. Cannot find an optimal focus value. 06:25Z Restarted cherno actor and cherno acquire. Cloudy and getting guide stars is difficult at the moment. Searching for the keyword "fwhm_camera" in the log gives you the rough fwhm of the gfas. 06:28:14 APO.doravetz_2 16 cherno f error="Acquisition failed: Rotator c orrection too large. Not applying correction." 06:28:14 CmdDone: APO.doravetz_2 16 cherno acquire -c -t 30 06:29:07 CmdStarted: APO.doravetz_2 18 cherno acquire -c -t 30 06:29:14 CmdStarted: APO.doravetz 40 cherno set axes radec Look for keyword "focus_fit". 06:31:09 APO.doravetz_2 18 cherno i focus_fit=44,2.143,4.72e-06,0.0008333,6.287,-0.161,17.7 The second number is the FWHM fit (not measured) and the last number in this sequence is how many microns to move from the current secondary focus number. 06:35Z Fighting clouds. Focus ~ 710 2. John D. is also on site, so if Jose and John are up, observe designs from the Kronos queue as if we were in normal operations, and work together to get good estimates of efficiency and identify areas for potential improvement 06:45Z Try loading another field from Kronos. Loading Design 721 (Field 16340) Configuration ID 3504. 06:49:04 Config Loaded: 721-3504 06:50Z Executing Hal all stages on this design. Hartmann: 06:52:14 boss exposure 4.0, file=337568 06:52:56 boss exposure 4.0, file=337569 06:53:44 sp1.motor.move= -3062, -3062, -3062 06:53:44 Hartmann collimate output sp1: offset: r= -3298 (Out of focus); b= 8.9 (Out of focus) sp1: pred. move: spAverageMove= -3063 sp1: pred. resid: r= -235, b= -0.7, txt= OK, spTemp = 2.9 06:54:15 APO.doravetz.jaeger 507 jaeger i text="[FVC]: RMS full fit 46.440 um." 06:54:58 APO.doravetz.jaeger 507 jaeger i text="[FVC]: RMS full fit 92.660 um." 6:54:10 APO.doravetz.jaeger 507 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0003.fits 06:54:54 APO.doravetz.jaeger 507 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0004.fits 06:54:58 FVC Loop - RMS: 92.66 Delta RMS: -46.22 Image #: 0004 Skipping BOSS cals in order to catch the next cloud sucker hole. Clouds again. 07:03Z Trying to find a good field through Kronos and finding bugs. 07:31Z Fixing bugs. 07:38Z Clearing and trying another field Loading Design 4678 (Field 17017) Configuration ID 3505 07:42Z Running Hal on this design. Failed. 299 deadlocks and Jose is debugging. 07:41:07 Config Loaded: 4678-3505 jaeger actor was restarted. Doing a jaeger unwind and reloading this field. Design 4678 Configuration ID 3506 07:51Z Running Hal on this design 07:50:44 Config Loaded: 4678-3506 Hartmann: 07:52:10 boss exposure 4.0, file=337570 07:52:53 boss exposure 4.0, file=337571 07:53:37 sp1.motor.move= -1600, -1600, -1600 07:53:37 Hartmann collimate output sp1: offset: r= -1979 (Out of focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -1600 sp1: pred. resid: r= -379, b= -1.2, txt= OK, spTemp = 2.7 FVC Loop: 07:54:08 APO.doravetz.jaeger 130 jaeger i text="[FVC]: RMS full fit 44.020 um." 07:54:51 APO.doravetz.jaeger 130 jaeger i text="[FVC]: RMS full fit 15.560 um." 07:55:33 APO.doravetz.jaeger 130 jaeger i text="[FVC]: RMS full fit 12.650 um." 07:56:15 APO.doravetz.jaeger 130 jaeger i text="[FVC]: RMS full fit 10.080 um." 07:56:56 APO.doravetz.jaeger 130 jaeger i text="[FVC]: RMS full fit 14.190 um." 07:54:04 APO.doravetz.jaeger 130 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0005.fits 07:54:47 APO.doravetz.jaeger 130 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0006.fits 07:55:29 APO.doravetz.jaeger 130 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0007.fits 07:56:10 APO.doravetz.jaeger 130 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0008.fits 07:56:52 APO.doravetz.jaeger 130 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0009.fits 07:56:56 FVC Loop - RMS: 14.19 Delta RMS: -4.11 Image #: 0009 BOSS Cals 07:57:33 boss exposure 4.0, file=337572 07:59:06 boss exposure 25.0, file=337573 Doing a focus sweep Best focus is 330. gfa5 is not finding stars. Jose is working on this. We found that running the gfa focus sweep script in STUI tries to reset the focus value back to when the script started. This STUI script needs to be further refined if we are going to use it for later operations. 08:30:54 apogee.expose=apRaw-40640010, 43, A, Object 08:31:01 boss exposure 900.0, file=337574 08:38:35 apogee.expose=apRaw-40640011, 43, B, Object Loading new design 10489 (field 18356) Configuration ID 3507 08:55:04 Config Loaded: 10489-3507 Hartmann: 08:57:35 boss exposure 4.0, file=337575 08:58:17 boss exposure 4.0, file=337576 08:58:59 sp1.motor.move= -948, -948, -949 08:59:00 Hartmann collimate output sp1: offset: r= -1319 (Out of focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -948 sp1: pred. resid: r= -370, b= -1.2, txt= OK, spTemp = 2.2 FVC Loop: 08:59:30 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 44.480 um." 09:00:14 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 15.160 um." 09:00:57 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 14.470 um." 08:59:26 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0010.fits 09:00:09 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0011.fits 09:00:52 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0012.fits 09:00:57 FVC Loop - RMS: 14.47 Delta RMS: 0.69 Image #: 0012 BOSS cals: 09:01:34 boss exposure 4.0, file=337577 09:03:09 boss exposure 25.0, file=337578 Autofocus seems to be working now. Data: 09:08:22 boss exposure 900.0, file=337579 09:08:23 apogee.expose=apRaw-40640012, 43, A, Object 09:16:04 apogee.expose=apRaw-40640013, 43, B, Object 09:24Z Moving on to the next available field Loading Design 4583 Config id 3508 (Field 16963) 09:27:15 Config Loaded: 4583-3508 This time having the arc lamp warm up time accounted for to help reduce overhead time. 09:28Z Running hal set on this design Hartmann: 09:30:13 boss exposure 4.0, file=337580 09:30:57 boss exposure 4.0, file=337581 09:31:46 sp1.motor.move= -784, -783, -784 09:31:46 Hartmann collimate output sp1: offset: r= -989 (In Focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -783 sp1: pred. resid: r= -205, b= -0.6, txt= OK, spTemp = 1.9 FVC Loop: 09:32:16 APO.doravetz.jaeger 123 jaeger i text="[FVC]: RMS full fit 50.640 um." 09:33:00 APO.doravetz.jaeger 123 jaeger i text="[FVC]: RMS full fit 15.200 um." 09:33:41 APO.doravetz.jaeger 123 jaeger i text="[FVC]: RMS full fit 8.800 um." 09:34:22 APO.doravetz.jaeger 123 jaeger i text="[FVC]: RMS full fit 12.540 um." 09:32:12 APO.doravetz.jaeger 123 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0013.fits 09:32:56 APO.doravetz.jaeger 123 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0014.fits 09:33:37 APO.doravetz.jaeger 123 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0015.fits 09:34:18 APO.doravetz.jaeger 123 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0016.fits 09:34:22 FVC Loop - RMS: 12.54 Delta RMS: -3.74 Image #: 0016 BOSS Cals: 09:32:06 boss exposure 4.0, file=337582 09:35:05 boss exposure 25.0, file=337583 Autofocus seems to be ok, but seeing is poor. Seeing is about 2.5" DATA: 09:49:53 boss exposure 900.0, file=337584 09:49:54 apogee.expose=apRaw-40640014, 43, A, Object 09:57:34 apogee.expose=apRaw-40640015, 43, B, Object 10:06:45 apogee.expose=apRaw-40640016, 5, B, DomeFlat 10:08Z Loaded same field : 16963. Design 4582 Configuration ID 3509 via Kronos 10:09:24 Config Loaded: 4582-3509 10:09Z Running Hal on this field Hartmann: 10:12:30 boss exposure 4.0, file=337585 10:13:11 boss exposure 4.0, file=337586 10:14:01 sp1.motor.move= 29, 26, 20 10:14:01 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= 28 sp1: pred. resid: r= -357, b= -1.1, txt= OK, spTemp = 1.9 FVC Loop: 10:14:31 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 44.570 um." 10:15:15 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 15.510 um." 10:15:57 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 11.070 um." 10:16:38 APO.doravetz.jaeger 162 jaeger i text="[FVC]: RMS full fit 13.250 um." 10:14:27 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0017.fits 10:15:11 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0018.fits 10:15:52 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0019.fits 10:16:34 APO.doravetz.jaeger 162 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0020.fits 10:16:38 FVC Loop - RMS: 13.25 Delta RMS: -2.18 Image #: 0020 BOSS Cals: 10:14:21 boss exposure 4.0, file=337587 10:17:20 boss exposure 25.0, file=337588 Data: 10:21:11 boss exposure 900.0, file=337589 10:21:13 apogee.expose=apRaw-40640017, 43, A, Object 10:28:54 apogee.expose=apRaw-40640018, 43, B, Object 10:38:58 apogee.expose=apRaw-40640019, 5, B, DomeFlat ================================================================================ 4. observe APOGEE dither field (we are skipping #3 in the commissioning plan because those fields are not available at 10:00Z.) We have opened the APOGEE instrument shutter now. 10:48:46 APO.doravetz 34 jaeger f error="Failed generating paths: Attempt 6: 2 deadlocks remain but the number of retries has been exhausted" 10:48:46 CmdDone: APO.doravetz 34 jaeger configuration load Reloading to see if we can avoid the deadlocks. 10:49:45 CmdStarted: APO.doravetz 35 jaeger configuration load Loaded 5718 config id 3510 10:51:01 Config Loaded: 5718-3510 but this is the wrong design. Reloading the correct design Design 35989 Config id 3511 10:54:21 Config Loaded: 35989-3511 10:54:43 APO.doravetz.jaeger 197 jaeger f error="The configuration was not executed. Try unwinding." 10:54:43 CmdDone: APO.doravetz.jaeger 197 jaeger configuration reverse Did a jaeger unwind Reissued Hal command from slew stage and adding the BOSS cals. We want to see how the arcs are now timed with the FVC loop. Hartmann: 10:57:35 boss exposure 4.0, file=337590 10:58:16 boss exposure 4.0, file=337591 10:59:00 sp1.motor.move= -458, -456, -452 10:59:00 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= 0.8 (In Focus) sp1: pred. move: spAverageMove= -458 sp1: pred. resid: r= -201, b= -0.6, txt= OK, spTemp = 1.7 FVC Loop: 10:59:31 APO.doravetz.jaeger 210 jaeger i text="[FVC]: RMS full fit 50.690 um." 11:00:14 APO.doravetz.jaeger 210 jaeger i text="[FVC]: RMS full fit 17.380 um." 11:00:55 APO.doravetz.jaeger 210 jaeger i text="[FVC]: RMS full fit 14.440 um." 11:01:36 APO.doravetz.jaeger 210 jaeger i text="[FVC]: RMS full fit 15.180 um." 10:59:26 APO.doravetz.jaeger 210 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0021.fits 11:00:10 APO.doravetz.jaeger 210 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0022.fits 11:00:51 APO.doravetz.jaeger 210 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0023.fits 11:01:32 APO.doravetz.jaeger 210 jaeger d fvc_filename=/data/fcam/59626/fimg-fvc1n-0024.fits 11:01:36 FVC Loop - RMS: 15.18 Delta RMS: -0.74 Image #: 0024 BOSS cals: 10:59:20 boss exposure 4.0, file=337592 11:02:19 boss exposure 25.0, file=337593 At the base configuration: 11:07:01 apogee.expose=apRaw-40640020, 30, A, Object #1 11:14:07 Config Loaded: 35989-3512 11:14:46 RMS full fit 52.220 um." file: fimg-fvc1n-0025.fits 11:15:50 apogee.expose=apRaw-40640021, 30, A, Object Guider wasn't running, so we turned it on via command line. We received an error: 11:14:58 APO.doravetz.cherno 222 cherno f error="Acquisition failed: Weights sum to zero, can't be normalized" 11:14:58 CmdDone: APO.doravetz.cherno 222 cherno acquire -c -t 15.0 We manually turned on the guider with Boson, and guiding and autofocus appears to be ok. #2 11:21:22 Config Loaded: 35989-3513 11:22:01 RMS full fit 48.430 um. file: fimg-fvc1n-0026.fits 11:23:03 apogee.expose=apRaw-40640022, 30, A, Object #3 11:28:36 Config Loaded: 35989-3514 11:29:15 RMS full fit 46.560 um. file: fimg-fvc1n-0027.fits 11:30:18 apogee.expose=apRaw-40640023, 30, A, Object 11:29:16 APO.doravetz 42 cherno f error="Acquisition failed: Weights sum to zero, can't be normalized" 11:29:16 CmdDone: APO.doravetz 42 cherno acquire -c -t 15 We see that the guider does not cleanly turn itself back on from our old dither sequence script. Modifying the script so that cherno comes back on, along with autofocus, so that we don't have to manually do it each time. #4 11:37:24 Config Loaded: 35989-3515 11:38:04 RMS full fit 49.090 um." file: fimg-fvc1n-0028.fits 11:40:16 apogee.expose=apRaw-40640024, 30, A, Object Cherno did not restart with our modification of the script. Manually starting the data while we try to figure out to cleanly restart cherno in our dither sequence script. #5 11:46:50 Config Loaded: 35989-3516 11:47:28 RMS full fit 50.160 um." file: fimg-fvc1n-0029.fits 11:49:41 apogee.expose=apRaw-40640025, 30, A, Object We see that if we run the commands through Run Commands, the script still fails with the cherno acquire command. We get the error: 11:47:35 APO.doravetz_2 115 cherno f error="Acquisition failed: Weights sum to zero, can't be normalized" 11:47:35 CmdDone: APO.doravetz_2 115 cherno acquire -c -t 15 We will have to do the remaining dither points manually. We not sure how to avoid this particular error since it is new to us. #6 11:58:17 Config Loaded: 35989-3517 11:58:55 RMS full fit 50.530 um." file: fimg-fvc1n-0030.fits 11:59:58 apogee.expose=apRaw-40640026, 30, A, Object #7 12:07:24 Config Loaded: 35989-3518 12:08:05 RMS full fit 48.190 um file: fimg-fvc1n-0031.fits 12:08:57 apogee.expose=apRaw-40640027, 30, A, Object #8 12:15:41 Config Loaded: 35989-3519 12:16:23 RMS full fit 49.450 um." file: fimg-fvc1n-0032.fits 12:17:45 apogee.expose=apRaw-40640028, 30, A, Object #9 12:23:38 Config Loaded: 35989-3520 12:24:19 RMS full fit 48.920 um." file: fimg-fvc1n-0033.fits 12:25:10 apogee.expose=apRaw-40640029, 30, A, Object #10 12:32:32 Config Loaded: 35989-3521 12:33:13 RMS full fit 47.340 um file: fimg-fvc1n-0034.fits 12:34:05 apogee.expose=apRaw-40640030, 30, A, Object #11 12:40:02 Config Loaded: 35989-3522 12:40:43 RMS full fit 47.860 um." file: fimg-fvc1n-0035.fits 12:41:36 apogee.expose=apRaw-40640031, 30, A, Object #12 12:47:49 Config Loaded: 35989-3523 12:48:30 RMS full fit 47.690 um. file: fimg-fvc1n-0036.fits 12:49:22 apogee.expose=apRaw-40640032, 30, A, Object 12:55:26 apogee.expose=apRaw-40640033, 5, A, DomeFlat 13:04:24 encl25m : closed; 13:08Z Started morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0010 0.0030 Axial C: -0.1150 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 7.50 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6550 -9.6570 Axial B: -9.6670 -9.6710 Axial C: -9.6970 -9.6970 Trans D: 0.1120 0.1140 GALILS Commanded: 4888700 4885700 4908000 0 -3400 Actual: 4859224 4859136 4859211 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.99 -0.0 -0.02 -0.01 41.64 ---- Telescope Status ---- 05:00Z Ventilation 05:23Z Opened 13:04Z Closed Status at: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 685 686 704 703 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 130 lbs, 22.7 psi ---- Software Used ---- --- Afternoon disk space --- Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 168G 1.9T 9% /home 10.25.1.230:/data 21T 17T 3.3T 84% /data Get Versions, 02/15/22, 23:21:47 --- Software --- STUI : 1.7.16 alerts : 2.2.0 boss : 6.0.1 -boss.specMec : 3.0.4 guider : not connected ecamera : not connected apo : 3.0.1 fliswarm : 0.2.1-alpha.0 jaeger : 0.14.1-alpha.0 kaiju : 1.2.2-alpha.1 coordio : 1.2.2-alpha.1 fps_calibrati : APO.2021.12.25+960cb05 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.12 boson : 0.0.4-alpha.33 (0.0.4-alpha.33) ---- APOGEE Data Summary ---- 59625 morning cals ================== APOGEE data list, mjd=59625, all raw_data: /data/apogee/utr_cdr/59625/apRaw-*.fits archive: /data/apogee/archive/59625/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59625/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59625 66 13:22 40630066 Dark 60 A 13.0 ---------- a-b-c - - 59625 67 13:33 40630067 Dark 60 A 13.0 ---------- a-b-c - - 59625 68 13:44 40630068 Dark 60 A 13.0 ---------- a-b-c - - 59625 69 13:46 40630069 QuarFlat 10 A 13.0 ---------- a-b-c - 99 59625 70 13:47 40630070 QuarFlat 10 A 13.0 ---------- a-b-c - 99 59625 71 13:49 40630071 QuarFlat 10 A 13.0 ---------- a-b-c - 100 59625 72 13:51 40630072 Arc-Thar 12 B 13.5 ---------- a-b-c 0.50 - 59625 73 13:59 40630073 Arc-Une 40 B 13.5 ---------- a-b-c - - 59625 74 14:01 40630074 Arc-Thar 12 A 13.0 ---------- a-b-c -0.06 - 59625 75 14:08 40630075 Arc-Une 40 A 13.0 ---------- a-b-c - - 59625 76 14:14 40630076 Dark 30 A 13.0 ---------- a-b-c - - 59625 77 14:19 40630077 InteFlat 30 A 13.0 ---------- a-b-c - 279 59625 78 14:24 40630078 InteFlat 30 A 13.0 ---------- a-b-c - 280 59625 79 14:30 40630079 InteFlat 30 A 13.0 ---------- a-b-c - 279 59625 80 14:35 40630080 Dark 30 A 13.0 ---------- a-b-c - - 59625 81 14:41 40630081 Arc---- 30 A 13.0 ---------- a-b-c - - FPI ------------------------------------------------------------------------------ APOGEE data list, mjd=59626, all raw_data: /data/apogee/utr_cdr/59626/apRaw-*.fits archive: /data/apogee/archive/59626/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59626/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59626 1 23:13 40640001 Dark 3 A 13.0 ---------- a-b-c - - 59626 2 23:29 40640002 Dark 60 A 13.0 ---------- a-b-c - - 59626 3 23:31 40640003 QuarFlat 10 A 13.0 ---------- a-b-c - 99 59626 4 23:33 40640004 Arc-Thar 12 B 13.5 ---------- a-b-c 0.50 - 59626 5 23:41 40640005 Arc-Une 40 B 13.5 ---------- a-b-c - - 59626 6 23:43 40640006 Arc-Thar 12 A 13.0 ---------- a-b-c -0.11 - 59626 7 23:50 40640007 Arc-Une 40 A 13.0 ---------- a-b-c - - 59626 8 00:01 40640008 Dark 60 A 13.0 ---------- a-b-c - - 59626 9 00:07 40640009 Arc---- 30 A 13.0 ---------- a-b-c - - FPI 59626 10 08:38 40640010 Object 43 A 13.0 4678-3506 a-b-c -0.10 - 59626 11 08:45 40640011 Object 43 B 13.5 4678-3506 a-b-c 0.40 - 59626 12 09:15 40640012 Object 43 A 13.0 10489-3507 a-b-c -0.10 - 59626 13 09:23 40640013 Object 43 B 13.5 10489-3507 a-b-c 0.40 - 59626 14 09:57 40640014 Object 43 A 13.0 4583-3508 a-b-c -0.09 - 59626 15 10:04 40640015 Object 43 B 13.5 4583-3508 a-b-c 0.40 - 59626 16 10:07 40640016 DomeFlat 5 B 13.5 4583-3508 a-b-c - 143 59626 17 10:28 40640017 Object 43 A 13.0 4582-3509 a-b-c -0.11 - 59626 18 10:36 40640018 Object 43 B 13.5 4582-3509 a-b-c 0.39 - 59626 19 10:39 40640019 DomeFlat 5 B 13.5 4582-3509 a-b-c - 146 59626 20 11:11 40640020 Object 30 A 13.0 35989-3511 a-b-c -0.09 - 59626 21 11:20 40640021 Object 30 A 13.0 35989-3512 a-b-c -0.08 - 59626 22 11:27 40640022 Object 30 A 13.0 35989-3513 a-b-c -0.09 - 59626 23 11:35 40640023 Object 30 A 13.0 35989-3514 a-b-c -0.09 - 59626 24 11:45 40640024 Object 30 A 13.0 35989-3515 a-b-c -0.09 - 59626 25 11:54 40640025 Object 30 A 13.0 35989-3516 a-b-c -0.09 - 59626 26 12:04 40640026 Object 30 A 13.0 35989-3517 a-b-c -0.09 - 59626 27 12:13 40640027 Object 30 A 13.0 35989-3518 a-b-c -0.09 - 59626 28 12:22 40640028 Object 30 A 13.0 35989-3519 a-b-c -0.09 - 59626 29 12:30 40640029 Object 30 A 13.0 35989-3520 a-b-c -0.09 - 59626 30 12:39 40640030 Object 30 A 13.0 35989-3521 a-b-c -0.09 - 59626 31 12:46 40640031 Object 30 A 13.0 35989-3522 a-b-c -0.09 - 59626 32 12:54 40640032 Object 30 A 13.0 35989-3523 a-b-c -0.09 - 59626 33 12:55 40640033 DomeFlat 5 A 13.0 35989-3511 a-b-c - 145 cannot read file : /data/apogee/utr_cdr/59626/apRaw-40640034.fits ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59626 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:13 b1-00337557 ----------- ---- bia Clo 0.0 -3435 -3294 -4680 2 23:13 r1-00337557 ----------- ---- bia Clo 0.0 -3435 -3294 -4680 3 23:20 b1-00337558 35986-20955 3489 arc Lef 4.1 -3435 -3294 -4680 HART 4 23:20 r1-00337558 35986-20955 3489 arc Lef 4.1 -3435 -3294 -4680 HART 5 23:20 b1-00337559 35986-20955 3489 arc Rig 4.1 -3435 -3294 -4680 HART 6 23:20 r1-00337559 35986-20955 3489 arc Rig 4.1 -3435 -3294 -4680 HART 7 23:22 b1-00337560 35986-20955 3489 arc Lef 4.1 -2600 -2457 -3844 HART 8 23:22 r1-00337560 35986-20955 3489 arc Lef 4.1 -2600 -2457 -3844 HART 9 23:22 b1-00337561 35986-20955 3489 arc Rig 4.1 -2600 -2457 -3844 HART 10 23:22 r1-00337561 35986-20955 3489 arc Rig 4.1 -2600 -2457 -3844 HART 11 23:25 b1-00337562 35986-20955 3489 arc Out 4.1 -2577 -2432 -3817 12 23:25 r1-00337562 35986-20955 3489 arc Out 4.1 -2577 -2432 -3817 13 23:26 r1-00337563 35986-20955 3489 fla Out 30.1 -2577 -2432 -3817 14 23:26 b1-00337563 35986-20955 3489 fla Out 30.1 -2577 -2432 -3817 15 05:35 b1-00337564 13029-19185 3503 arc Lef 4.1 -2577 -2432 -3817 HART 16 05:35 r1-00337564 13029-19185 3503 arc Lef 4.1 -2577 -2432 -3817 HART 17 05:36 b1-00337565 13029-19185 3503 arc Rig 4.0 -2577 -2432 -3817 HART 18 05:36 r1-00337565 13029-19185 3503 arc Rig 4.0 -2577 -2432 -3817 HART 19 05:42 r1-00337566 13029-19185 3503 arc Out 4.1 1679 1824 438 20 05:42 b1-00337566 13029-19185 3503 arc Out 4.1 1679 1824 438 21 05:44 r1-00337567 13029-19185 3503 fla Out 25.1 1679 1824 438 22 05:44 b1-00337567 13029-19185 3503 fla Out 25.1 1679 1824 438 23 06:52 r1-00337568 721-16340 3504 arc Lef 4.1 1679 1824 438 HART 24 06:52 b1-00337568 721-16340 3504 arc Lef 4.1 1679 1824 438 HART 25 06:52 b1-00337569 721-16340 3504 arc Rig 4.0 1679 1824 438 HART 26 06:52 r1-00337569 721-16340 3504 arc Rig 4.0 1679 1824 438 HART 27 07:52 r1-00337570 4678-17017 3506 arc Lef 4.1 -1383 -1238 -2624 HART 28 07:52 b1-00337570 4678-17017 3506 arc Lef 4.1 -1383 -1238 -2624 HART 29 07:52 r1-00337571 4678-17017 3506 arc Rig 4.1 -1383 -1238 -2624 HART 30 07:52 b1-00337571 4678-17017 3506 arc Rig 4.1 -1383 -1238 -2624 HART 31 07:57 r1-00337572 4678-17017 3506 arc Out 4.1 -2983 -2838 -4224 32 07:57 b1-00337572 4678-17017 3506 arc Out 4.1 -2983 -2838 -4224 33 07:59 b1-00337573 4678-17017 3506 fla Out 25.1 -2983 -2838 -4224 34 07:59 r1-00337573 4678-17017 3506 fla Out 25.1 -2983 -2838 -4224 35 08:31 r1-00337574 4678-17017 3506 sci Out 900.1 -2983 -2838 -4224 36 08:31 b1-00337574 4678-17017 3506 sci Out 900.1 -2983 -2838 -4224 37 08:57 r1-00337575 10489-18356 3507 arc Lef 4.1 -2983 -2838 -4224 HART 38 08:57 b1-00337575 10489-18356 3507 arc Lef 4.1 -2983 -2838 -4224 HART 39 08:58 r1-00337576 10489-18356 3507 arc Rig 4.0 -2983 -2838 -4224 HART 40 08:58 b1-00337576 10489-18356 3507 arc Rig 4.0 -2983 -2838 -4224 HART 41 09:01 b1-00337577 10489-18356 3507 arc Out 4.1 -3931 -3786 -5173 42 09:01 r1-00337577 10489-18356 3507 arc Out 4.1 -3931 -3786 -5173 43 09:03 b1-00337578 10489-18356 3507 fla Out 25.1 -3931 -3786 -5173 44 09:03 r1-00337578 10489-18356 3507 fla Out 25.1 -3931 -3786 -5173 45 09:08 r1-00337579 10489-18356 3507 sci Out 900.1 -3931 -3786 -5173 46 09:08 b1-00337579 10489-18356 3507 sci Out 900.1 -3931 -3786 -5173 47 09:30 b1-00337580 4583-16963 3508 arc Lef 4.1 -3931 -3786 -5173 HART 48 09:30 r1-00337580 4583-16963 3508 arc Lef 4.1 -3931 -3786 -5173 HART 49 09:30 b1-00337581 4583-16963 3508 arc Rig 4.1 -3931 -3786 -5173 HART 50 09:30 r1-00337581 4583-16963 3508 arc Rig 4.1 -3931 -3786 -5173 HART 51 09:32 b1-00337582 4583-16963 3508 arc Out 4.1 -4715 -4569 -5957 52 09:32 r1-00337582 4583-16963 3508 arc Out 4.1 -4715 -4569 -5957 53 09:35 r1-00337583 4583-16963 3508 fla Out 25.1 -4715 -4569 -5957 54 09:35 b1-00337583 4583-16963 3508 fla Out 25.1 -4715 -4569 -5957 55 09:49 b1-00337584 4583-16963 3508 sci Out 900.1 -4715 -4569 -5957 56 09:49 r1-00337584 4583-16963 3508 sci Out 900.1 -4715 -4569 -5957 57 10:12 b1-00337585 4582-16963 3509 arc Lef 4.1 -4715 -4569 -5957 HART 58 10:12 r1-00337585 4582-16963 3509 arc Lef 4.1 -4715 -4569 -5957 HART 59 10:13 b1-00337586 4582-16963 3509 arc Rig 4.1 -4715 -4569 -5957 HART 60 10:13 r1-00337586 4582-16963 3509 arc Rig 4.1 -4715 -4569 -5957 HART 61 10:14 b1-00337587 4582-16963 3509 arc Out 4.1 -4686 -4543 -5937 62 10:14 r1-00337587 4582-16963 3509 arc Out 4.1 -4686 -4543 -5937 63 10:17 b1-00337588 4582-16963 3509 fla Out 25.1 -4686 -4543 -5937 64 10:17 r1-00337588 4582-16963 3509 fla Out 25.1 -4686 -4543 -5937 65 10:21 b1-00337589 4582-16963 3509 sci Out 900.1 -4686 -4543 -5937 66 10:21 r1-00337589 4582-16963 3509 sci Out 900.1 -4686 -4543 -5937 67 10:57 r1-00337590 35989-20958 3511 arc Lef 4.1 -4686 -4543 -5937 HART 68 10:57 b1-00337590 35989-20958 3511 arc Lef 4.1 -4686 -4543 -5937 HART 69 10:58 r1-00337591 35989-20958 3511 arc Rig 4.1 -4686 -4543 -5937 HART 70 10:58 b1-00337591 35989-20958 3511 arc Rig 4.1 -4686 -4543 -5937 HART 71 10:59 b1-00337592 35989-20958 3511 arc Out 4.1 -5144 -4999 -6389 72 10:59 r1-00337592 35989-20958 3511 arc Out 4.1 -5144 -4999 -6389 73 11:02 r1-00337593 35989-20958 3511 fla Out 25.1 -5144 -4999 -6389 74 11:02 b1-00337593 35989-20958 3511 fla Out 25.1 -5144 -4999 -6389 75 13:06 b1-00337594 ----------- ---- bia Clo 0.0 -5144 -4999 -6389 76 13:06 r1-00337594 ----------- ---- bia Clo 0.0 -5144 -4999 -6389 77 13:07 r1-00337595 ----------- ---- bia Clo 0.0 -5144 -4999 -6389 78 13:07 b1-00337595 ----------- ---- bia Clo 0.0 -5144 -4999 -6389 ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 23:22:02 boss exposure 4.0, file=337560 23:22:44 boss exposure 4.0, file=337561 23:23:26 sp1.motor.move= 23, 25, 27 23:23:26 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -2.2 (In Focus) sp1: pred. move: spAverageMove= 24 sp1: pred. resid: r= -683, b= -2.1, txt= OK, spTemp = 5.7 23:23:22 APO.doravetz_2 31 hartmann i r1MeanOffset=-0.10,"In Focus" 23:23:22 APO.doravetz_2 31 hartmann i b1MeanOffset=0.11,"In Focus" Time TSP1 R1 B1 Move1 B1Resid ============================================= 23:21:20 5.2 0 -5.2 835 -2.6 23:23:20 5.2 -659 -2.2 24 -2.1 05:37:00 3.4 3628 -15.3 4256 -2.0 06:53:30 2.5 -3298 8.9 -3063 -0.7 07:53:30 2.0 -1979 3.8 -1600 -1.2 08:58:50 1.7 -1319 1.8 -948 -1.2 09:31:30 1.5 -989 1.8 -783 -0.6 10:13:50 1.2 -329 -1.2 28 -1.1 10:58:50 1.1 -659 0.8 -458 -0.6 ---- Problems in Detail ---- Error message from running gfa_fwhm.py -c /opt/pyenv/versions/3.9.7/envs/observers39/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3440: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /opt/pyenv/versions/3.9.7/envs/observers39/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) 100%|███████████████████████████████████████████| 21/21 [00:20<00:00, 1.02it/s] Optimal focus not found /opt/pyenv/versions/3.9.7/envs/observers39/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3440: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /opt/pyenv/versions/3.9.7/envs/observers39/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) Error message from gfa_fwhm.py -p -w /opt/pyenv/versions/3.9.7/envs/observers39/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3440: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /opt/pyenv/versions/3.9.7/envs/observers39/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount)