Subject: 2.5m obslog 20220117 (SJD 59597)
From: doravetz@apo.nmsu.edu
Date: Tue, 18 Jan 2022 13:57:37 +0000 (UTC)
                       APACHE POINT OBSERVATORY SDSS 2.5M
                                 OBSERVING LOG
                          Monday January 17, 2022
                                  (SJD 59597)
----  Observing Team ----
Viktor Malanushenko
Dan Oravetz
---- Observing Plan ----
The same as last night:
If it's clear, please continue taking a mix of BOSS and APOGEE robot dither
sequences (12 points is fine) with jaeger fvc loop --no-apply at each dither. If possible, repeat one BOSS and one APOGEE design from the last two nights.
We will use --scale 0.999882 as before.
---- Observing Summary ----
Lost the first couple hours to clouds and some passing snow flurries. We opened at 04:15Z and the skies were clear with good seeing the rest of the night. Encountered several problems throughout the night, none of them necessarily new, just seemed to encounter them all tonight.
Design 35919 (APOGEE)
=====================
 - Two exposures were taken at the parent configuration and these produced good quickred results. The guiding wasn't correct during most of the dithered exposures, however, as a result of a large offset:
 - We discovered during the dither sequence that the guider wasn't applying corrections, but there were no red-text error messages associated. At some point  at the first dither loop the guider applied very large successive offsets resulting in a net -5 deg offset. We didn't see anything about a large fiducial error in rotator during the ~30 minutes in and around that time to suggest the rotator wasn't where it was supposed to be.
 - We learned that the correction_applied key was True (which is why we weren't seeing error messages), only the corrections were always 0.0. After discussion with Jose, the expectation going forward is that if correction_applied is all zeros, then did_correct should equal F. He'll also make the "rotator correction too large" an error so it stands out in the Log windows.
Design 35966 (BOSS)
===================
 - Completed a 12-point dither loop
Design 35965 (APOGEE)
=====================
 - We tried this one TWICE. The first time through there was no indication during the load/execute/gotoField/FVC loop/acquire/guide/etc. process that anything had gone wrong, but the first exposure and the first dither exposure both showed flux at ~0 counts. So we opted to re-try the design starting from the 'configuration load'. Second time through there were also no problems and all RMS values were in-spec, etc. Again, flux ~0 counts. ANYTHING WORTH CHECKING AT THE DESIGN LEVEL WITH THIS ONE?
Designs 35963 and 35961 (APOGEE)
================================
 - We never successfully executed a configuration for either of these two designs. In both cases we experienced VALUE_OUT_OF_RANGE errors when the positioners were in the folded state each time (i.e., no negative alpha values). For 35963 it was P0544 that threw the error and for 35961 it was P0808. We tried many combinations of reloading configurations to try for different trajectories and we also tried "nudging", but that didn't fix the problem either this time. In the interest of time, we skipped these and tried 35973 which DID execute properly instead.
Design 35973 (APOGEE)
=====================
 - Completed a 7-point dither sequence.
 - The guiding didn't work to begin with on this field. Once we changed the min_npix value to '20' (from the default of '50') the guiding corrections began and then eventually guiding converged well.
NOTES
=====
We had some trouble with the ALT brake at closing, but got it freed up and all is reporting fine again. Details at the end of the Log body.
---- Observing Log ----
Afternoon:
=========
23:00Z  FPS & GFA were turned on before I arrived on the telescope. I believe the day staff worked during the day.
--Test Actors, 23:02:06
23:05:42  APOGEE short dark
23:08:04  APOGEE evening cals started
23:20:02  boss exposure bias; boss exposure dark itime=20; files=336351,336352
23:24:40  hartmann collimate - failed. forgot turn on lamps.
23:25:09  hartmann collimate
23:26:35  Hartmann collimate output
   sp1: offset: r= 1649 (Out of focus);  b= -9.3 (Out of focus)
   sp1: pred. move: spAverageMove= 2302
   sp1: pred. resid: r= -653, b= -2.0, txt= OK, spTemp = 6.3
-- Collimate in spec.
----------
motorPosition
     sp1:   -2497,   -2458,   -2517;
----------
Median spectrographs temperatures:
     Tsp1= 6.4
     Outside air temperature= 4.2;
----------
23:32:25 boss arc, flat: files = 336357,336358
23:54Z start ventilation
23:59Z on the telescope all checked and ok,
Cloudy and radar coming to site. Nevertheless we will continue ventilation any time.
Night:
=====
02:20Z - Stopped ventilation due to approaching snow flurries.
03:22Z - Restarted ventilation. Snow has passed, clouds starting to clear.
04:15Z - Opened. Clear. Force-filled BOSS dewars.
****************************
Configuration 35919 (APOGEE)
****************************
Loaded and executed:
04:53:19 CmdDone: OBSERVER.viktorm 53 jaeger configuration load 35919 --scale 0.999882
04:53:19 OBSERVER.viktorm 53 jaeger : text="Configuration 2228 loaded."
04:55:40 CmdDone: OBSERVER.viktorm 55 jaeger configuration execute
04:55:40 OBSERVER.viktorm 55 jaeger i text="All positioners have reached their destinations."
GotoField:
05:00:31 CmdStarted: OBSERVER.viktorm_2 22 hal goto-field -s slew,fvc,acquire,guide --guider-time 15
The gotoField failed because the "FVC loop failed":
05:04:04 OBSERVER.viktorm_2 22 hal e error="FVC loop failed. RMS=12.87."
05:02:17 OBSERVER.viktorm_2.jaeger 329 jaeger i fvc_rms=49.88
05:03:04 OBSERVER.viktorm_2.jaeger 329 jaeger i fvc_rms=13.26
05:03:04 OBSERVER.viktorm_2.jaeger 329 jaeger i fvc_deltarms=36.62
05:03:47 OBSERVER.viktorm_2.jaeger 329 jaeger i fvc_rms=12.87
05:03:47 OBSERVER.viktorm_2.jaeger 329 jaeger i fvc_deltarms=0.39
Stopped and ran a second manual FVC loop:
05:05:43 CmdStarted: OBSERVER.viktorm 62 jaeger fvc loop
05:06:11 OBSERVER.viktorm 62 jaeger i fvc_rms=3.0
05:06:28 CmdDone: OBSERVER.viktorm 62 jaeger fvc loop
05:07:13 CmdStarted: OBSERVER.viktorm 68 jaeger fvc loop
05:07:40 OBSERVER.viktorm 68 jaeger i fvc_rms=3.03
05:07:57 CmdDone: OBSERVER.viktorm 68 jaeger fvc loop
05:10Z - Focus sweep. Best focus 500 um.
Guiding on and guide_rms stabilized:
05:24:26 OBSERVER.viktorm 101 cherno d guide_rms=21,0.068,0.066,0.095
Exposure at the parent configuration:
Exposure # 40350010 taken, but the shutter wasn't opened until the 6th read. We took another exposure after this to be certain.
Exposure # 40350011:
05:39:06 CmdDone: OBSERVER.viktorm 106 apogee expose nreads=30; object=Object
Started the dither loop:
----------
jaeger configuration dither 0.1
tcc axis stop
cherno set axes
jaeger fvc loop --no-apply
tcc axis init
tcc track
tui wait 3
cherno set axes radec rot
tui wait 20
apogee dither namedpos=A
apogee expose nreads=30 object=Object
tui wait 2
jaeger configuration reverse
tui wait 3
----------
We discovered during the dither loop there had evidently been a very large rotator offset (~ -5 DEGREES) implemented after we started tracking. It's not clear where this offset came from. The rotator fiducials that were read were not showing any large errors -- even though fiducial corrections aren't applied when tracking, the fiducials still get read and they were all acceptably small. Here are the successive cherno rot offsets being applied resulting in the ~ -5 deg offset:
05:42:06 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-1013.53,0.0
05:42:43 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-1541.33,0.0
05:43:19 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-2093.56,0.0
05:43:56 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-2671.27,0.0
05:44:35 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-3271.45,0.0
05:45:15 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-3877.02,0.0
05:45:56 OBSERVER.viktorm 101 cherno i acquisition_valid=T; did_correct=T; correction_applied=0.0,0.0,-4500.58,0.0
06:37:24 APO.Jose: It seems there was a point which it started applying rotation corrections but that only made things worse and worse.
We don't know what caused this behavior. Nothing looks unusual in the sequence of commands issued before/after. And later on when we manually removed the -5 deg offset, the guiding was just fine again.
05:40:20  Config Loaded: 35919-2229
05:40:31  encl25m : open;
05:42:18  apogee.expose=apRaw-40350012, 30, A, Object
05:47:41  Config Loaded: 35919-2228
05:47:52  Config Loaded: 35919-2230
05:49:51  apogee.expose=apRaw-40350013, 30, A, Object
05:55:13  Config Loaded: 35919-2228
05:55:24  Config Loaded: 35919-2231
05:57:14  apogee.expose=apRaw-40350014, 30, A, Object
06:02:38  Config Loaded: 35919-2228
06:02:50  Config Loaded: 35919-2232
06:04:40  apogee.expose=apRaw-40350015, 30, A, Object
06:10:01  Config Loaded: 35919-2228
06:10:12  Config Loaded: 35919-2233
06:12:05  apogee.expose=apRaw-40350016, 30, A, Object
06:17:28  Config Loaded: 35919-2228
06:17:39  Config Loaded: 35919-2234
06:19:30  apogee.expose=apRaw-40350017, 30, A, Object
06:24:51  Config Loaded: 35919-2228
06:25:03  Config Loaded: 35919-2235
06:26:56  apogee.expose=apRaw-40350018, 30, A, Object
06:32:18  Config Loaded: 35919-2228
06:32:30  Config Loaded: 35919-2236
06:34:21  apogee.expose=apRaw-40350019, 30, A, Object
06:39:43  Config Loaded: 35919-2228
06:39:55  Config Loaded: 35919-2237
06:41:44  apogee.expose=apRaw-40350020, 30, A, Object
06:47:07  Config Loaded: 35919-2228
06:47:19  Config Loaded: 35919-2238
06:49:08  apogee.expose=apRaw-40350021, 30, A, Object
06:54:31  Config Loaded: 35919-2228
06:54:43  Config Loaded: 35919-2239
06:56:32  apogee.expose=apRaw-40350022, 30, A, Object
07:01:55  Config Loaded: 35919-2228
07:02:07  Config Loaded: 35919-2240
07:03:58  apogee.expose=apRaw-40350023, 30, A, Object
07:09:21  Config Loaded: 35919-2228
07:11:00  apogee.expose=apRaw-40350024, 5, A, DomeFlat
**************************
Configuration 35966 (BOSS)
**************************
Loaded and executed:
07:16:34 CmdDone: OBSERVER.viktorm 241 jaeger configuration load 35966 --scale 0.999882
07:16:34 OBSERVER.viktorm 241 jaeger : text="Configuration 2241 loaded."
07:18:37 CmdDone: OBSERVER.viktorm 244 jaeger configuration execute
07:18:37 OBSERVER.viktorm 244 jaeger i text="All positioners have reached their destinations."
GotoField:
07:19:55 CmdStarted: OBSERVER.viktorm_2 24 hal goto-field -s slew,fvc,acquire,guide,boss_hartmann,boss_arcs,boss_flat,prepare --guider-time 15
The gotoField sequence failed at the FVC loop again:
07:26:42 OBSERVER.viktorm_2 24 hal e error="FVC loop failed. RMS=22.4."
Hartmanns:
07:22:18  boss exposure    4.0, file=336359
07:22:58  boss exposure    4.0, file=336360
07:23:44  sp1.motor.move= -2399, -2401, -2399
07:23:44  Hartmann collimate output
   sp1: offset: r= -1649 (Out of focus);  b= 9.9 (Out of focus)
   sp1: pred. move: spAverageMove= -2399
   sp1: pred. resid: r= 750, b= 2.4, txt= OK, spTemp = 1.5
Stopped and ran a second manual FVC loop:
07:27:46 CmdStarted: OBSERVER.viktorm 246 jaeger fvc loop
07:28:13 OBSERVER.viktorm 246 jaeger i fvc_rms=11.69
07:28:56 OBSERVER.viktorm 246 jaeger i fvc_rms=3.84
07:28:56 OBSERVER.viktorm 246 jaeger i fvc_deltarms=7.85
07:29:13 CmdDone: OBSERVER.viktorm 246 jaeger fvc loop
Back to field:
07:29:54 CmdStarted: OBSERVER.viktorm_2 25 hal goto-field -s acquire,guide,boss_arcs,boss_flat,slew --guider-time 15
Arc and flat:
07:32:26  boss exposure    4.0, file=336361
07:34:00  boss exposure   25.0, file=336362
Focus sweep: best focus 370 um.
BOSS exposure at parent configuration:
07:53:11  boss exposure  300.0, file=336364
07:59Z - Started dither loop:
jaeger configuration dither 0.2
boss exposure science itime=300
tui wait 2
jaeger configuration reverse
tui wait 3
07:59:35  Config Loaded: 35966-2242
08:03:13  boss exposure  300.0, file=336365
08:09:38  Config Loaded: 35966-2241
08:10:14  Config Loaded: 35966-2243
08:10:46  boss exposure  300.0, file=336366
08:17:07  Config Loaded: 35966-2241
08:17:19  Config Loaded: 35966-2244
08:17:51  boss exposure  300.0, file=336367
08:24:11  Config Loaded: 35966-2241
08:24:23  Config Loaded: 35966-2245
08:24:56  boss exposure  300.0, file=336368
08:31:16  Config Loaded: 35966-2241
08:31:28  Config Loaded: 35966-2246
08:32:00  boss exposure  300.0, file=336369
08:38:23  Config Loaded: 35966-2241
08:38:34  Config Loaded: 35966-2247
08:39:08  boss exposure  300.0, file=336370
08:45:28  Config Loaded: 35966-2241
08:45:40  Config Loaded: 35966-2248
08:46:12  boss exposure  300.0, file=336371
08:52:32  Config Loaded: 35966-2241
08:52:44  Config Loaded: 35966-2249
08:53:16  boss exposure  300.0, file=336372
08:59:37  Config Loaded: 35966-2241
08:59:49  Config Loaded: 35966-2250
09:00:21  boss exposure  300.0, file=336373
09:06:42  Config Loaded: 35966-2241
09:06:54  Config Loaded: 35966-2251
09:07:27  boss exposure  300.0, file=336374
09:13:49  Config Loaded: 35966-2241
09:14:01  Config Loaded: 35966-2252
09:14:33  boss exposure  300.0, file=336375
09:20:53  Config Loaded: 35966-2241
09:21:04  Config Loaded: 35966-2253
09:21:37  boss exposure  300.0, file=336376
09:27:56  Config Loaded: 35966-2241
****************************
Configuration 35965 (APOGEE)
****************************
Loaded and executed:
09:31:57 CmdStarted: OBSERVER.viktorm 308 jaeger configuration load 35965 --scale 0.999882
09:32:48 OBSERVER.viktorm 308 jaeger : text="Configuration 2254 loaded."
09:32:48 CmdDone: OBSERVER.viktorm 308 jaeger configuration load 35965 --scale 0.999882
09:34:11 OBSERVER.viktorm 309 jaeger i text="All positioners have reached their destinations."
GotoField:
09:38:39 CmdStarted: OBSERVER.viktorm_2 27 hal goto-field -s acquire,guide,fvc --guider-time 15
The gotoField failed because the "FVC loop failed":
09:40:54 OBSERVER.viktorm_2 27 hal e error="FVC loop failed. RMS=16.44."
Stopped and ran a second manual FVC loop:
09:42:17 CmdStarted: OBSERVER.viktorm_2.jaeger 392 jaeger fvc loop
09:42:44 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_rms=11.32
09:43:29 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_rms=6.83
09:43:29 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_deltarms=4.49
09:44:15 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_rms=5.65
09:44:15 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_deltarms=1.18
09:44:58 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_rms=5.75
09:44:58 OBSERVER.viktorm_2.jaeger 392 jaeger i fvc_deltarms=-0.1
09:45:16 CmdDone: OBSERVER.viktorm_2.jaeger 392 jaeger fvc loop
09:49Z - Focus sweeps. Best focus 300 um.
Guiding on and guide_rms stabilized:
10:04:26 OBSERVER.viktorm 334 cherno d guide_rms=532,0.057,0.01,0.058
Exposure at the parent configuration:
10:04:10  apogee.expose=apRaw-40350025, 30, A, Object
10:09Z - Started the dither loop:
----------
jaeger configuration dither 0.1
tcc axis stop
cherno set axes
jaeger fvc loop --no-apply
tcc axis init
tcc track
tui wait 3
cherno set axes radec rot
tui wait 20
apogee dither namedpos=A
apogee expose nreads=30 object=Object
tui wait 2
jaeger configuration reverse
tui wait 3
----------
09:32:46  Config Loaded: 35965-2254
09:40:38  FVC Loop - RMS:16.44 Delta RMS:  0.69
 Image #: 0026
09:44:58  FVC Loop - RMS: 5.75 Delta RMS: -0.10
 Image #: 0030
10:04:10  apogee.expose=apRaw-40350025, 30, A, Object
10:09:51  Config Loaded: 35965-2255
10:11:42  apogee.expose=apRaw-40350026, 30, A, Object
10:17:05  Config Loaded: 35965-2254
The flux was ~0 in the first exposure taken at the parent configuration and also at the first dither. We went back and checked the sequence of events and everything looked normal. We decided to restart from the beginning to see if we could reproduce this.
==========================================================================
10:30:08 CmdStarted: APO.doravetz_2 33 jaeger unwind --status
10:30:08 APO.doravetz_2 33 jaeger >
10:30:08 APO.doravetz_2 33 jaeger i folded=T
10:30:08 APO.doravetz_2 33 jaeger : text="All positioners are folded."
10:30:08 CmdDone: APO.doravetz_2 33 jaeger unwind --status
10:31:15 CmdDone: OBSERVER.viktorm 350 jaeger configuration load 35965 --scale 0.999882
10:31:15 OBSERVER.viktorm 350 jaeger : text="Configuration 2256 loaded."
10:32:29 CmdDone: OBSERVER.viktorm 351 jaeger configuration execute
10:32:29 OBSERVER.viktorm 351 jaeger i text="All positioners have reached their destinations."
10:33:13 CmdStarted: OBSERVER.viktorm_2 29 hal goto-field -s acquire,guide,fvc,prepare,slew --guider-time 15
10:35Z - GotoField failed to complete because of the RMS:
10:35:40 OBSERVER.viktorm_2 29 hal e error="FVC loop failed. RMS=14.18."
We're still sitting with axes halted, so started a manual FVC loop:
10:36:03 CmdStarted: OBSERVER.viktorm 352 jaeger fvc loop
10:36:30 OBSERVER.viktorm 352 jaeger i fvc_rms=5.26
10:37:13 OBSERVER.viktorm 352 jaeger i fvc_rms=5.56
10:37:13 OBSERVER.viktorm 352 jaeger i fvc_deltarms=-0.3
10:37:30 CmdDone: OBSERVER.viktorm 352 jaeger fvc loop
Guiding has converged and stabilized:
10:42:36 OBSERVER.viktorm_2.cherno 421 cherno d guide_rms=587,0.036,0.024,0.043
10:42Z - APOGEE shutter open. Starting exposure.
10:42:56  apogee.expose=apRaw-40350027, 30, A, Object
The exposure still shows ~0 flux. We decided to choose a new design to try.
10:54:09 CmdDone: OBSERVER.viktorm 356 jaeger configuration load 35963 --scale 0.999882
10:59:32 .jaeger 0 jaeger w text="Positioner 544 replied to SEND_TRAJECTORY_DATA UID=5031 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)"
10:59:32 OBSERVER.viktorm 365 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed"
10:59:48 APO.doravetz_2 36 jaeger i positioner_status=544,10.0127,170.0041,0x8db067b1,T,F,F,F,04.01.21,4,2,247
fps_snapshot_59597_0077.pdf
11:03:33 CmdDone: OBSERVER.viktorm 368 jaeger configuration load 35961 --scale 0.999882
11:03:33 OBSERVER.viktorm 368 jaeger : text="Configuration 2259 loaded."
11:03:59 .jaeger 0 jaeger w text="Positioner 808 replied to SEND_TRAJECTORY_DATA UID=5046 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)"
11:03:59 .jaeger 0 jaeger w text="Positioner 808 replied to SEND_TRAJECTORY_DATA UID=5046 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)"
11:03:59 OBSERVER.viktorm 369 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed"
11:03:59 CmdDone: OBSERVER.viktorm 369 jaeger configuration execute
Nothing looks wrong with P0808 - no negative values, it looks folded, etc. All the positioners are reporting a folded position.
We tried to "nudge" 808 using a 'goto 10.1 170.1' command. This command worked fine. Subsequent attempts at 'execute' produced the same VALUE_OUT_OF_RANGE error.
We reloaded the configuration again, but the execute still produced a VALUE_OUT_OF_RANGE error.
We tried about half a dozen more sequences of nudging 808 and reloading configurations, but each time we got the VALUE_OUT_OF_RANGE error.
In the interest of time, we decided to try yet another design.
****************************
Configuration 35973 (APOGEE)
****************************
Loaded and executed:
11:31:36 CmdStarted: OBSERVER.viktorm 393 jaeger configuration load 35973 --scale 0.999882
11:32:28 OBSERVER.viktorm 393 jaeger : text="Configuration 2263 loaded."
11:33:31 CmdDone: OBSERVER.viktorm 394 jaeger configuration execute
11:33:31 OBSERVER.viktorm 394 jaeger i text="All positioners have reached their destinations."
GotoField:
11:35:09 CmdStarted: OBSERVER.viktorm_2 33 hal goto-field -s acquire,guide,prepare,slew --guider-time 15
The gotoField failed because the "FVC loop failed":
11:37:21 CmdStarted: OBSERVER.viktorm_2.jaeger 441 jaeger fvc loop
11:37:49 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_rms=87.57
11:38:35 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_rms=47.42
11:38:35 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_deltarms=40.15
11:39:19 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_rms=39.38
11:39:19 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_deltarms=8.04
11:40:04 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_rms=39.14
11:40:04 OBSERVER.viktorm_2.jaeger 441 jaeger i fvc_deltarms=0.24
11:40:21 CmdDone: OBSERVER.viktorm_2.jaeger 441 jaeger fvc loop
Stopped and ran a second manual FVC loop:
11:40:49 CmdStarted: OBSERVER.viktorm 395 jaeger fvc loop
11:41:17 OBSERVER.viktorm 395 jaeger i fvc_rms=8.73
11:42:00 OBSERVER.viktorm 395 jaeger i fvc_rms=8.84
11:42:00 OBSERVER.viktorm 395 jaeger i fvc_deltarms=-0.11
11:42:17 CmdDone: OBSERVER.viktorm 395 jaeger fvc loop
The acquire failed. We were just seeing output from cherno like this:
11:54:50 OBSERVER.viktorm 404 cherno d fwhm_camera=gfa3,651,1.13,0.56,0.5,0.11,7
11:54:50 OBSERVER.viktorm 404 cherno d fwhm_camera=gfa2,651,1.08,0.54,0.49,0.09,6
11:54:50 OBSERVER.viktorm 404 cherno d fwhm_camera=gfa5,651,1.18,0.59,0.55,0.07,9
11:54:50 OBSERVER.viktorm 404 cherno d fwhm_camera=gfa6,651,1.17,0.58,0.52,0.12,5
11:54:50 OBSERVER.viktorm 404 cherno d fwhm_camera=gfa4,651,1.08,0.54,0.52,0.03,7
11:54:52 OBSERVER.viktorm 404 cherno i camera_solution=gfa3,651,F,-999.0,-999.0,-999.0,-999.0,-999.0
11:54:52 OBSERVER.viktorm 404 cherno i camera_solution=gfa2,651,F,-999.0,-999.0,-999.0,-999.0,-999.0
11:54:53 OBSERVER.viktorm 404 cherno i camera_solution=gfa6,651,F,-999.0,-999.0,-999.0,-999.0,-999.0
11:54:53 OBSERVER.viktorm 404 cherno i camera_solution=gfa4,651,F,-999.0,-999.0,-999.0,-999.0,-999.0
11:54:53 OBSERVER.viktorm 404 cherno i camera_solution=gfa5,651,F,-999.0,-999.0,-999.0,-999.0,-999.0
11:54:53 OBSERVER.viktorm 404 cherno e acquisition_valid=F; did_correct=F
We adjusted min_npix to 20 and that helped because we immediately started getting rotation corrections. Then we reissued the 'set axes radec rot' and all the guiding was working.
Guiding converged:
11:59:35 OBSERVER.viktorm 407 cherno d guide_rms=660,0.045,0.038,0.059
Started exposure at parent configuration:
11:58:48  apogee.expose=apRaw-40350028, 30, A, Object
12:05Z - Started dither loop:
12:04:24  Config Loaded: 35973-2264
12:06:14  apogee.expose=apRaw-40350029, 30, A, Object
12:11:37  Config Loaded: 35973-2263
12:11:49  Config Loaded: 35973-2265
12:13:39  apogee.expose=apRaw-40350030, 30, A, Object
12:19:02  Config Loaded: 35973-2263
12:19:13  Config Loaded: 35973-2266
12:21:04  apogee.expose=apRaw-40350031, 30, A, Object
12:26:26  Config Loaded: 35973-2263
12:26:38  Config Loaded: 35973-2267
12:28:28  apogee.expose=apRaw-40350032, 30, A, Object
12:33:50  Config Loaded: 35973-2263
12:34:01  Config Loaded: 35973-2268
12:35:53  apogee.expose=apRaw-40350033, 30, A, Object
12:41:16  Config Loaded: 35973-2263
12:41:27  Config Loaded: 35973-2269
12:43:20  apogee.expose=apRaw-40350034, 30, A, Object
12:48:43  Config Loaded: 35973-2263
12:48:55  Config Loaded: 35973-2270
12:50:47  apogee.expose=apRaw-40350035, 30, A, Object
12:56:11  Config Loaded: 35973-2263
Ended with dome flat:
12:57:22  apogee.expose=apRaw-40350036, 5, A, DomeFlat
13:15Z - Closed. Turned off FPS. Started morning cals.
After we pulled the enclosure on and initialized the axes in order to move the telescope to stow, the ALT brake got stuck. When the axes were initialized and the brake attempted to disengage, there was a "sputtering" sound in the pneumatics as if the brake were not fully disengaging.
At the same time the telescope drifted downwards about 0.75 deg from the 5-deg position (not sure why, we haven't seen many balance issues lately). The combination of the downward drift and the brake not being fully engaged or disengaged led to a state in which the brake was contacting the ALT drive surface but it had gotten torqued downward so that the internal switch that relays the ALT brake status was no longer in contact with the brake mechanism hardware.
The MCP reported the ALT brake was neither engaged nor disengaged even though physically it was contacting the ALT drive surface. This in-between brake state did not allow the ALT axis to initialize. It was possible to reach in and physically actuate the switch with a screwdriver and see the brake status change in the MCP to 'OFF'. That state allows an ALT.INIT to send and put things in a state that allows a MOVE attempt. Problem was even though we could send a MOVE command in that case, the brake was still sufficiently contacting the surface that ALT wouldn't move. Having verified that, a few well-placed taps with a rubber mallet freed the brake; the switch became contacted properly when the brake disengaged and we moved back to stow.
---- Mirror Numbers ----
PRIMARY:
--------
Scale: 1.000000
MIGS		TONIGHT	NOMINAL
Axial A:	 0.0060	 0.0040
Axial B:	 0.0040	 0.0030
Axial C:	-0.1180	-0.1150
Trans D:	-0.0340	-0.0338
Lateral E:	 0.3315	 0.3315
Lateral F:	 0.5350	 0.5350
GALILS
Commanded:  12500 -1600 5300 47050 -8800 -8400
Actual:     12500 -1600 5300 47050 -8800 -8400
SETMIR VALUES
PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3
PrimOrient:    1.07 -59.51 -9.21 -512.13 1906.31
SECONDARY:
----------
Focus:      0.0000
Air temp:     4.17
Altitude:   30.0
MIGS		TONIGHT	NOMINAL
Axial A:	-9.6560	-9.6570
Axial B:	-9.6680	-9.6710
Axial C:	-9.6970	-9.6970
Trans D:	 0.1160	 0.1140
GALILS
Commanded: 4889700 4886850 4905700 0 -3450
Actual:    4859230 4859299 4859243 0 -3450
SETMIR VALUES
SecDesOrient: 3857.0 0.0 0.0 0.0 42.22
SecOrient:    3857.03 0.01 0.01 0.62 42.25
---- Telescope Status ----
Status at 13:56Z:
Telescope stowed at:    121.0, 30.0, 0.0 mount
Instrument mounted:     FPS
Counterweights at:      685 687 705 704
LN2 autofill systems:   Connected and turned on
180L LN2 dewar scales:  SP1 85 lbs, 22 psi
---- Software Used ----
  --- STUI software ---
STUI          : 1.7.16
alerts        : 2.2.0
apo           : FAILED
boss          : 6.0.1
-boss.specMec : 3.0.4
guider        : FAILED
ecamera       : FAILED
fliswarm      : 0.2.1-alpha.0
jaeger        : 0.13.2-alpha.0
kaiju         : 1.2.2-alpha.0
coordio       : 1.2.1-alpha.2
fps_calibrati : APO.2021.12.25
cherno        : 0.1.1-alpha.0
hal           : 0.1.1-alpha.0
hartmann      : 2.0.2
hub           : 5.0.1a0
mcp           : 2.1
-mcp.plc      : 34
-mcp.azFiduci : v1_148
-mcp.altFiduc : v1_148
-mcp.rotFiduc : v1_148
tcc           : 1.4.0
apogee        : not available
apogeecal     : not available
apogeeql      : v1_7_0dev
STUIAdditions : 3.1.11
  --- Other Versions ---
sdss-obstools       : 3.7.7b0
TPM                 : 1.2-31-gd7eb
MCP                 : 2.1