Subject: 2.5m obslog 20220217 (SJD 59628)
From: viktorm@apo.nmsu.edu
Date: Fri, 18 Feb 2022 11:28:32 +0000 (UTC)
                       APACHE POINT OBSERVATORY SDSS 2.5M
                                 OBSERVING LOG
                          Thursday February 17, 2022
                                  (SJD 59628)
----  Observing Team ----
Dan Oravetz
Viktor Malanushenko
Jose Sanchez-Gallego
John Donor
Joe Burchett
---- Observing Plan ----
Continue to work with John and Jose to identify areas of improvement.
https://wiki.sdss.org/display/COM/Commissioning+Nightly+Plan#CommissioningNightlyPlan-CommissioningNightlyPlan
Thursday February 17
If closed, run the following FVC tests:
1.    Using the latest version of jaeger, load 10 designs from the queue and for each one reconfigure the FPS, move the rotator to the position it would have on the sky, and take a single FVC loop. After the loop, for each configuration, take three fvc loop --no-apply. The purpose of the latter is to see how stable the model is. In the log, include the range of configuration IDs for the 10 designs.
2.    Ask José to revert to a previous version of jaeger that uses the old FVC fitting code. Repeat 1 with 10 new designs from the queue.
---- Observing Summary ----
At the evening: radar + cloudy + high humidity. After 1:30Z: clear sky + high humidity. After 8:35Z: clear sky + reduced humidity + strong wind and snow blowing.
We have made FVC tests for the next config:
— Using the latest version of jaeger —
17093-3543, RMS: 11.70
 5294-3544, RMS: 13.94
15766-3545, RMS: 16.27
15766-3546, RMS:  9.81
15693-3547, RMS: 15.67
15599-3548, RMS: 11.22
15680-3549, RMS: 12.01
15268-3550, RMS: 10.61
15175-3552, RMS: 14.70
18550-3556, RMS: 49.78
— Revert to a previous version of jaeger :
jaeger i version=0.13.2-alpha.0
  721-3557, RMS:  8.10
 4696-3558, RMS:  4.32
13077-3559, RMS: 36.37
10510-3560, RMS: 24.38
10514-3561, RMS: 40.84
15175-3562, RMS:  8.87
  721-3563, RMS: 13.41
 4696-3564, RMS: 14.46
10510-3566, RMS:  5.20
10514-3568, RMS: 18.95
Remarks:
========
-- The histogram of positioner errors using the older version of jaeger was considerably tighter than with the newer version. Jose will talk to Conor about the changes to the centroiding code in the newest version.
-- We skip the next configurations due to the error 'Trajectory failed with error: Something went wrong sending the trajectory' :
 18535-3553; 18535-3554; 18535-3555; 13077-3565;
-- We have 'Jaeger collision:'  444, 184
-- We load configuration 10514-3567 instead of 10510-3566 due to the 'Jaeger collision:  184'
-- Jose cut a new Boson version that incorporates a "preload" functionality to save on overhead. It allows one to load the next design in the queue and compute the trajectories and save that info to memory (nothing written to the database) while finishing up the observing on the previous field.
-- We started modifying some of the Grafana plots to look at the new 'actors' bucket as part of the new service Jose started. This contains all of the new actors/keywords.
-- Additional miscellaneous enhancements made to Kronos (e.g., including an APOGEE exposure S/N plot)
See details in section 'Problems in Detail'
---- Observing Log ----
============================================================================
                               AFTERNOON CHECKOUT
============================================================================
Cloudy, high humidity, intermittent snow flurries upon arrival.
The 'fliswarm reconnect' didn't work, but a 'fliswarm reconnect -c gfa' was successful. See 'Problems in Detail' for associated STUI Log messages.
All systems alive and responsive.
This morning's APOGEE cals look fine. APOGEE evening cals and BOSS cals taken.
Positioners are homed:
23:34:53 CmdStarted: APO.doravetz 16 jaeger unwind --status
23:34:53 APO.doravetz 16 jaeger i folded=T
23:34:53 APO.doravetz 16 jaeger : text="All positioners are folded."
Ventilation delayed due to weather.
============================================================================
                                    NIGHT
============================================================================
Turned off amplifiers of Alt and Az, turn ON amplifiers of ROT. Then unpress 'Stop button' to move Rotator and use ‘hal goto field’ commands inside enclosure.
So for each configuration we did:
HAL: ' load from queue ', then -> Goto Field , then -> run script 'Run_Commands':
==========================
tcc axis stop
jaeger fvc loop
tui wait 2
jaeger fvc loop --no-apply
tui wait 2
jaeger fvc loop --no-apply
tui wait 2
jaeger fvc loop --no-apply
tui wait 2
jaeger configuration reverse
==========================
======= last version of jaeger ===================
jaeger i version=0.14.1-alpha.0
=======================================================
01# 02:30:36  Config Loaded: 17093-3543
02:36:52  FVC Loop - RMS: 11.70 Delta RMS:   0.80 Image #: 0005 ; fvc_perc_95=19.45 ; fvc_percent_reached=73.5
02# 02:40:14  Config Loaded: 5294-3544
02:49:53  FVC Loop - RMS: 13.94 Delta RMS:   0.99 Image #: 0008 ;  fvc_perc_95=25.81 ; fvc_percent_reached=52.8
03# 02:53:27  Config Loaded: 15766-3545
03:00:14  FVC Loop - RMS: 16.27 Delta RMS:  -0.52 Image #: 0012 ; fvc_perc_95=28.69 ;  fvc_percent_reached=53.3
04# 03:02:34  Config Loaded: 15766-3546
03:12:00  FVC Loop - RMS:  9.81 Delta RMS:   0.96 Image #: 0017 ;  fvc_perc_95=15.6 ; fvc_percent_reached=83.5
05# 03:14:52  Config Loaded: 15693-3547
03:21:48  FVC Loop - RMS: 15.67 Delta RMS:  -3.37 Image #: 0021 ;  fvc_perc_95=19.03 ; fvc_percent_reached=71.7
03:22:35 APO.doravetz 94 jaeger i text="[FVC]: RMS full fit 16.270 um." ; fvc_perc_95=21.32 ;  fvc_percent_reached=64.6
03:23:22 APO.doravetz 95 jaeger i text="[FVC]: RMS full fit 15.570 um." ; fvc_perc_95=18.23 ; fvc_percent_reached=74.9
03:24:11 APO.doravetz 96 jaeger i text="[FVC]: RMS full fit 16.760 um." ; fvc_perc_95=22.12 ; fvc_percent_reached=62.2
06# 03:28:23  Config Loaded: 15599-3548
03:34:04  FVC Loop - RMS: 11.22 Delta RMS:   0.97 Image #: 0029 ; fvc_perc_95=20.27 ; fvc_percent_reached=72.9
03:35:38 APO.doravetz 103 jaeger i text="[FVC]: RMS full fit 12.180 um." ; fvc_perc_95=21.4 ;  fvc_percent_reached=68.2
03:36:26 APO.doravetz 104 jaeger i text="[FVC]: RMS full fit 11.810 um." ; fvc_perc_95=19.84 ; fvc_percent_reached=69.5
03:36:52  Jaeger collision:  184
  03:37:47 CmdDone: APO.doravetz 106 jaeger unlock
  03:38:36 CmdDone: APO.doravetz 107 jaeger configuration reverse
07# 03:40:01  Config Loaded: 15680-3549
03:46:05  FVC Loop - RMS: 12.01 Delta RMS:   0.46 Image #: 0036 ; fvc_perc_95=19.64 ;  fvc_percent_reached=74.2
03:46:52 APO.doravetz 110 jaeger i text="[FVC]: RMS full fit 11.160 um." ; fvc_perc_95=18.08 ; fvc_percent_reached=76.8
03:47:39 APO.doravetz 111 jaeger i text="[FVC]: RMS full fit 11.980 um." ; fvc_perc_95=18.76 ; fvc_percent_reached=76.5
03:48:26 APO.doravetz 112 jaeger i text="[FVC]: RMS full fit 12.210 um." ; fvc_perc_95=20.29 ; fvc_percent_reached=67.4
08# 03:51:47  Config Loaded: 15268-3550
03:57:17  FVC Loop - RMS: 10.61 Delta RMS:   0.13 Image #: 0044 ; fvc_perc_95=17.83 ; fvc_percent_reached=75.4
- 03:58:03 APO.doravetz 116 jaeger i text="[FVC]: RMS full fit 10.590 um." ; fvc_perc_95=17.34 ; fvc_percent_reached=75.0
- 03:58:51 APO.doravetz 117 jaeger i text="[FVC]: RMS full fit 11.770 um." ; fvc_perc_95=21.78 ; fvc_percent_reached=73.5
- 03:59:38 APO.doravetz 118 jaeger i text="[FVC]: RMS full fit 15.490 um." ; fvc_perc_95=26.24 ; fvc_percent_reached=46.2
04:03:24  Config Loaded: 15318-3551 - skip it: ...Failed executing the correction trajectory ...
04:12:15 APO.doravetz: 04:08:00 APO.doravetz 121 jaeger f error=\"Failed processing image: Failed executing the correction trajectory: Something went wrong during the trajectory: Some positioners did not reach their destinations. Trajectory failed 7.24 seconds after start\"
04:08:00 CmdDone: APO.doravetz 121 jaeger fvc loop
04:13:30 CmdDone: APO.doravetz 123 jaeger fvc loop
04:14:36 CmdDone: APO.doravetz 125 jaeger fvc loop
04:13:26  Jaeger collision:  444
04:14:30  Jaeger collision:  444
09# 04:18:31  Config Loaded: 15175-3552
04:20:55 APO.doravetz: 04:20:04 APO.doravetz 129 jaeger f error=\"Failed processing image: The exposure succeeded but did not output the filename.\"
-- repeat it:
04:23:53  FVC Loop - RMS: 14.70 Delta RMS:  -2.19 Image #: 0055 ;  fvc_perc_95=24.46 ; fvc_percent_reached=50.5
04:24:40 APO.doravetz 133 jaeger i text="[FVC]: RMS full fit 15.260 um." ; fvc_perc_95=26.3 ; fvc_percent_reached=46.5
04:25:27 APO.doravetz 134 jaeger i text="[FVC]: RMS full fit 17.010 um." ; fvc_perc_95=27.52 ; fvc_percent_reached=38.5
04:28:31  Config Loaded: 18535-3553 — skip it:
04:28:58 .jaeger 0 jaeger w text=\"Positioner 63 replied to SEND_TRAJECTORY_DATA UID=5122 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)\"
04:28:59 APO.doravetz_2.jaeger 188 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59628/trajectory-59628-0055.json
04:28:59 APO.doravetz_2.jaeger 188 jaeger f error=\"Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed\"
04:28:59 CmdDone: APO.doravetz_2.jaeger 188 jaeger configuration execute
04:37:37  Config Loaded: 18535-3554 - skip it
04:38:52  Config Loaded: 18535-3555 - skip it
10# 04:41:06  Config Loaded: 18550-3556
04:44:01  FVC Loop - RMS: 49.78 Delta RMS:  -1.83 Image #: 0060 ; fvc_perc_95=30.35 ; fvc_percent_reached=36.3
04:44:48 APO.doravetz 152 jaeger i text="[FVC]: RMS full fit 49.610 um." ;  fvc_perc_95=29.66 ; fvc_percent_reached=35.9
04:45:35 APO.doravetz 153 jaeger i text="[FVC]: RMS full fit 48.910 um." ; fvc_perc_95=27.22 ; fvc_percent_reached=47.1
04:46:22 APO.doravetz 154 jaeger i text="[FVC]: RMS full fit 48.370 um." ; fvc_perc_95=29.14 ; fvc_percent_reached=38.9
======= revert to a previous version of jaeger ===================
jaeger i version=0.13.2-alpha.0
=======================================================
01# 04:50:24  Config Loaded: 721-3557
04:56:27  FVC Loop - RMS:  8.10 Delta RMS:   0.54 Image #: 0067
04:57:13 APO.doravetz 159 jaeger i text="[FVC]: RMS full fit 8.110 um."
04:57:58 APO.doravetz 160 jaeger i text="[FVC]: RMS full fit 8.200 um."
04:58:44 APO.doravetz 161 jaeger i text="[FVC]: RMS full fit 7.910 um."
02# 05:01:15  Config Loaded: 4696-3558
05:06:35  FVC Loop - RMS:  4.32 Delta RMS:   1.06 Image #: 0075
05:07:20  FVC Loop - RMS:  4.48 Delta RMS:   1.06 Image #: 0076
05:08:05  FVC Loop - RMS:  4.48 Delta RMS:   1.06 Image #: 0077
05:08:51  FVC Loop - RMS:  4.33 Delta RMS:   1.06 Image #: 0078
03# 05:14:21  Config Loaded: 13077-3559
05:17:24  FVC Loop - RMS: 36.37 Delta RMS: -24.93 Image #: 0081
05:18:10 APO.doravetz 177 jaeger i text="[FVC]: RMS full fit 35.980 um."
05:18:55 APO.doravetz 178 jaeger i text="[FVC]: RMS full fit 36.060 um."
05:19:41 APO.doravetz 179 jaeger i text="[FVC]: RMS full fit 35.970 um."
04# 05:23:32  Config Loaded: 10510-3560
05:28:05  FVC Loop - RMS: 24.38 Delta RMS:   0.13 Image #: 0088
05:28:51 APO.doravetz 183 jaeger i text="[FVC]: RMS full fit 24.010 um."
05:29:36 APO.doravetz 184 jaeger i text="[FVC]: RMS full fit 23.910 um."
05:30:22 APO.doravetz 185 jaeger i text="[FVC]: RMS full fit 24.120 um."
05# 05:36:25  Config Loaded: 10514-3561
05:40:38  FVC Loop - RMS: 40.84 Delta RMS:   0.66 Image #: 0094
05:41:23 APO.doravetz 190 jaeger i text="[FVC]: RMS full fit 40.890 um."
05:42:08 APO.doravetz 191 jaeger i text="[FVC]: RMS full fit 40.850 um."
05:42:54 APO.doravetz 192 jaeger i text="[FVC]: RMS full fit 40.970 um."
06# 05:46:57  Config Loaded: 15175-3562
05:52:18  FVC Loop - RMS:  8.87 Delta RMS:   0.65 Image #: 0102
05:53:04 APO.doravetz 196 jaeger i text="[FVC]: RMS full fit 8.530 um."
05:53:49 APO.doravetz 197 jaeger i text="[FVC]: RMS full fit 8.750 um."
05:54:34 APO.doravetz 198 jaeger i text="[FVC]: RMS full fit 8.410 um."
07# 05:58:58  Config Loaded: 721-3563
06:03:30  FVC Loop - RMS: 13.41 Delta RMS:   0.59 Image #: 0109
06:04:16 APO.doravetz 202 jaeger i text="[FVC]: RMS full fit 13.400 um."
06:05:02 APO.doravetz 203 jaeger i text="[FVC]: RMS full fit 13.430 um."
06:05:47 APO.doravetz 204 jaeger i text="[FVC]: RMS full fit 13.530 um."
08# 06:07:57  Config Loaded: 4696-3564
06:11:28  FVC Loop - RMS: 14.46 Delta RMS:  -0.20 Image #: 0116
06:12:13 APO.doravetz 208 jaeger i text="[FVC]: RMS full fit 14.480 um."
06:12:58 APO.doravetz 209 jaeger i text="[FVC]: RMS full fit 14.460 um."
06:13:44 APO.doravetz 210 jaeger i text="[FVC]: RMS full fit 14.260 um."
06:16:04  Config Loaded: 13077-3565
06:16:25 APO.doravetz_2.jaeger 145 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed"
— skip it
09# 06:19:13  Config Loaded: 10510-3566
06:28:54  FVC Loop - RMS:  5.20 Delta RMS:   0.45 Image #: 0123
06:29:40 APO.doravetz_2 4 jaeger i text="[FVC]: RMS full fit 5.300 um."
06:30:25 APO.doravetz_2 5 jaeger i text="[FVC]: RMS full fit 5.170 um."
06:31:10 APO.doravetz_2 6 jaeger i text="[FVC]: RMS full fit 5.160 um."
06:34:07  Config Loaded: 10514-3567
06:35:03  Jaeger collision:  184
- 06:35:57 CmdDone: APO.doravetz_2 9 jaeger unlock
- 06:36:01 CmdDone: APO.doravetz_2 10 jaeger configuration reverse
- 06:36:04 CmdDone: APO.doravetz_2 11 jaeger unlock
- 06:36:21 CmdDone: APO.doravetz_2 12 jaeger status 184
- 06:37:37 CmdDone: APO.doravetz_2 13 jaeger goto 184 203 85
- 06:38:50 CmdDone: APO.doravetz_2 14 jaeger explode 5
- 06:39:03 CmdDone: APO.doravetz_2 15 jaeger configuration reverse
- 06:40:27 CmdDone: APO.doravetz_2 16 jaeger unwind
Reload current configration using commend:
06:42:10 CmdDone: APO.doravetz_2 17 jaeger configuration load 10514 --no-clone
10# 06:42:07  Config Loaded: 10514-3568
06:46:20  FVC Loop - RMS: 18.95 Delta RMS:  -3.04 Image #: 0130
06:47:05 APO.doravetz_2 20 jaeger i text="[FVC]: RMS full fit 19.020 um."
06:47:50 APO.doravetz_2 21 jaeger i text="[FVC]: RMS full fit 18.880 um."
06:48:36 APO.doravetz_2 22 jaeger i text="[FVC]: RMS full fit 19.060 um."
---- Mirror Numbers ----
PRIMARY:
--------
Scale: 1.000000
MIGS		TONIGHT	NOMINAL
Axial A:	 0.0050	 0.0040
Axial B:	-0.0010	 0.0030
Axial C:	-0.1110	-0.1150
Trans D:	-0.0341	-0.0338
Lateral E:	 0.3315	 0.3315
Lateral F:	 0.5350	 0.5350
GALILS
Commanded:  12500 -1600 5300 47050 -8800 -8400
Actual:     12500 -1600 5300 47050 -8800 -8400
SETMIR VALUES
PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3
PrimOrient:    1.07 -59.51 -9.21 -512.13 1906.31
SECONDARY:
----------
Focus:      0.0000
Air temp:    -5.00
Altitude:   30.0
MIGS		TONIGHT	NOMINAL
Axial A:	-9.6580	-9.6570
Axial B:	-9.6720	-9.6710
Axial C:	-9.6960	-9.6970
Trans D:	 0.1050	 0.1140
GALILS
Commanded: 4896900 4897150 4915900 -50 -3450
Actual:    4859224 4859192 4859331 -50 -3450
SETMIR VALUES
SecDesOrient: 3857.0 0.0 0.0 0.0 42.22
SecOrient:    3857.02 -0.05 -0.01 -0.01 42.9
---- Telescope Status ----
Closed all night
Status at:  11:24:25Z
Telescope Position:  121.0,  30.0,   0.0 mount
Instrument mounted:  FPS
Counterweights at:  685.5, 686.8, 704.7, 704.1
LN2 autofill systems:  Connected and turned on
180L LN2 dewar scale:  SP1  147.0 lbs,   23.3 psi
---- Software Used ----
  --- Afternoon disk space ---
(observers39) [observer@sdss5-hub: ~]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sdb1                2.0T  168G  1.9T   9% /home
10.25.1.230:/data         21T   17T  3.2T  85% /data
  --- Software ---
STUI          : 1.7.17b1
boson         : 0.0.4-alpha.34
alerts        : 2.2.0
boss          : 6.0.1
-boss.specMec : 3.0.4
guider        : not connected
ecamera       : not connected
apo           : 3.0.1
fliswarm      : 0.2.1-alpha.0
jaeger        : 0.14.1-alpha.0
kaiju         : 1.2.2-alpha.1
coordio       : 1.2.2-alpha.1
fps_calibrati : APO.2021.12.25+960cb05
cherno        : 0.1.1-alpha.0
hal           : 0.1.1-alpha.0
hartmann      : 2.0.2
hub           : 5.0.1a0
mcp           : 2.1
-mcp.plc      : 34
-mcp.azFiduci : v1_148
-mcp.altFiduc : v1_148
-mcp.rotFiduc : v1_148
tcc           : 1.4.0
apogee        : not available
apogeecal     : not available
apogeeql      : FAILED
STUIAdditions : 3.1.12
sdss-obstools : 3.7.10
TPM           : 1.2-31-gd7eb
MCP           : 2.1
PLC           : V34
Fiducials     : v1_148
idlspec2d     : v6_0_5
Kronos        : 0.1.0
roboscheduler : 0.11.1
---- APOGEE Data Summary ----
59627 morning cals:
(observers39) [observer@sdss5-hub: ~]$ list_ap -m 59627
APOGEE data list,   mjd=59627,  all
raw_data:  /data/apogee/utr_cdr/59627/apRaw-*.fits
archive:  /data/apogee/archive/59627/apR-[a,b,c]-*.apz
quickred:  /data/apogee/quickred/59627/apq-*.fits
------------------------------------------------------------------------------
 mjd   i   UT   File/Exp  Imtype Nread A/B Dth  Design-Cfg  Arch Offset %Flux Comm
 ------------------------------------------------------------------------------
59627  10 10:12  40650010    Dark   60  A  13.0  ----------  a-b-c    -   -
59627  11 10:22  40650011    Dark   60  A  13.0  ----------  a-b-c    -   -
59627  12 10:33  40650012    Dark   60  A  13.0  ----------  a-b-c    -   -
59627  13 10:35  40650013  QuarFlat 10  A  13.0  ----------  a-b-c    -   98
59627  14 10:37  40650014  QuarFlat 10  A  13.0  ----------  a-b-c    -   98
59627  15 10:39  40650015  QuarFlat 10  A  13.0  ----------  a-b-c    -   99
59627  16 10:41  40650016  Arc-Thar 12  B  13.5  ----------  a-b-c  0.51  -
59627  17 10:48  40650017  Arc-Une  40  B  13.5  ----------  a-b-c    -   -
59627  18 10:50  40650018  Arc-Thar 12  A  13.0  ----------  a-b-c -0.07  -
59627  19 10:58  40650019  Arc-Une  40  A  13.0  ----------  a-b-c    -   -
59627  20 11:03  40650020    Dark   30  A  13.0  ----------  a-b-c    -   -
59627  21 11:08  40650021  InteFlat 30  A  13.0  ----------  a-b-c    -  279
59627  22 11:14  40650022  InteFlat 30  A  13.0  ----------  a-b-c    -  279
59627  23 11:19  40650023  InteFlat 30  A  13.0  ----------  a-b-c    -  279
59627  24 11:25  40650024    Dark   30  A  13.0  ----------  a-b-c    -   -
59627  25 11:30  40650025  Arc----  30  A  13.0  ----------  a-b-c    -   -   FPI
------------------------------------------------------------------------------
APOGEE data list,   mjd=59628,  all
raw_data:  /data/apogee/utr_cdr/59628/apRaw-*.fits
archive:  /data/apogee/archive/59628/apR-[a,b,c]-*.apz
quickred:  /data/apogee/quickred/59628/apq-*.fits
------------------------------------------------------------------------------
 mjd   i   UT   File/Exp  Imtype Nread A/B Dth  Design-Cfg  Arch Offset %Flux Comm
 ------------------------------------------------------------------------------
59628   1 23:27  40660001    Dark    3  A  13.0  ----------  a-b-c    -   -
59628   2 23:46  40660002    Dark   60  A  13.0  ----------  a-b-c    -   -
59628   3 23:48  40660003  QuarFlat 10  A  13.0  ----------  a-b-c    -   98
59628   4 23:51  40660004  Arc-Thar 12  B  13.5  ----------  a-b-c  0.49  -
59628   5 23:58  40660005  Arc-Une  40  B  13.5  ----------  a-b-c    -   -
59628   6 00:00  40660006  Arc-Thar 12  A  13.0  ----------  a-b-c -0.11  -
59628   7 00:07  40660007  Arc-Une  40  A  13.0  ----------  a-b-c    -   -
59628   8 00:18  40660008    Dark   60  A  13.0  ----------  a-b-c    -   -
59628   9 00:24  40660009  Arc----  30  A  13.0  ----------  a-b-c    -   -   FPI
59628  10 08:07  40660010   Object  43  A  13.0  4678-3572  a-b-c   err  -   Test
    cannot read file : /data/apogee/utr_cdr/59628/apRaw-40660011.fits
------------------------------------------------------------------------------
---- eBOSS Data Summary ----
Reading FITS from /data/spectro//59628
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  n  Time  Filename    design-field cfg  flav  shu   Exp     colA   colB   colC Hrt
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  1 23:28 b1-00337642 ----------- ---- bia Clo    0.0  -3655  -3510  -4899
  2 23:28 r1-00337642 ----------- ---- bia Clo    0.0  -3655  -3510  -4899
  3 23:43 b1-00337643 13077-19205 3542 arc Lef    4.1  -3655  -3510  -4899 HART
  4 23:43 r1-00337643 13077-19205 3542 arc Lef    4.1  -3655  -3510  -4899 HART
  5 23:44 b1-00337644 13077-19205 3542 arc Rig    4.1  -3655  -3510  -4899 HART
  6 23:44 r1-00337644 13077-19205 3542 arc Rig    4.1  -3655  -3510  -4899 HART
  7 23:54 b1-00337645 13077-19205 3542 arc Lef    4.1  -5255  -5110  -6500 HART
  8 23:54 r1-00337645 13077-19205 3542 arc Lef    4.1  -5255  -5110  -6500 HART
  9 23:54 b1-00337646 13077-19205 3542 arc Rig    4.1  -5255  -5110  -6500 HART
 10 23:54 r1-00337646 13077-19205 3542 arc Rig    4.1  -5255  -5110  -6500 HART
 11 23:58 b1-00337647 13077-19205 3542 arc Out    4.1  -5392  -5246  -6634
 12 23:58 r1-00337647 13077-19205 3542 arc Out    4.1  -5392  -5246  -6634
 13 00:00 b1-00337648 13077-19205 3542 fla Out   30.1  -5392  -5246  -6634
 14 00:00 r1-00337648 13077-19205 3542 fla Out   30.1  -5392  -5246  -6634
 15 11:17 b1-00337649 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 16 11:17 r1-00337649 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 17 11:18 r1-00337650 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 18 11:18 b1-00337650 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 19 11:20 r1-00337651 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 20 11:20 b1-00337651 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 21 11:21 b1-00337652 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
 22 11:21 r1-00337652 ----------- ---- bia Clo    0.0  -5392  -5246  -6634
---- BOSS Spectrographs Collimation ----
Afternoon focus check
---------------------
23:54:16  boss exposure    4.0, file=337645
23:54:57  boss exposure    4.0, file=337646
23:55:39  sp1.motor.move= -137, -136, -134
23:55:39  Hartmann collimate output
   sp1: offset: r= -659 (In Focus);  b= -1.2 (In Focus)
   sp1: pred. move: spAverageMove= -137
   sp1: pred. resid: r= -522, b= -1.6, txt= OK, spTemp = 2.1
23:55:35 OBSERVER.telescope 160 hartmann i r1MeanOffset=-0.10,"In Focus"
23:55:35 OBSERVER.telescope 160 hartmann i b1MeanOffset=0.06,"In Focus"
Time        TSP1   R1    B1   Move1   B1Resid
=============================================
23:45:10    1.3  -1979   3.8  -1600  -1.2
23:55:30    1.3   -659  -1.2   -137  -1.6
---- Weather Log ----
				Wind	1 um	DIMM		IRSC
Time	Temp C	Dewp C	MPH	Dir	Dust	fwhm	mean	sigma
19:00	-5.9	-6.3	12.9	232.0	dpDep
19:15	-5.9	-6.2	5.2	233.2	dpDep
19:30	-6.2	-6.8	11.0	269.1	dpDep
19:45	-5.9	-6.5	10.4	264.3	dpDep
20:00	-6.0	-6.6	9.2	258.6	dpDep
20:15	-5.8	-6.3	9.7	254.6	dpDep
20:30	-5.8	-6.6	10.0	236.1	dpDep
20:45	-5.1	-5.8	8.6	247.8	dpDep
21:00	-4.8	-6.1	8.9	199.9	dpDep
21:15	-4.8	-5.9	9.7	264.3	dpDep
21:30	-4.7	-5.8	11.5	236.6	dpDep
21:45	-4.9	-6.1	12.5	252.5	dpDep
22:00	-5.2	-6.9	15.0	252.2	dpDep
22:15	-5.3	-6.9	13.9	237.5	dpDep
22:30	-5.2	-6.1	8.1	268.3	dpDep
22:45	-5.4	-6.2	10.6	265.4	dpDep
23:00	-5.4	-6.2	9.3	267.4	dpDep
23:15	-5.4	-6.4	8.5	263.1	dpDep
23:30	-5.3	-6.3	11.4	244.1	dpDep
23:45	-5.3	-6.1	8.9	213.2	dpDep
00:00	-5.8	-6.6	7.4	263.1	dpDep
00:15	-5.3	-6.3	6.0	257.0	dpDep
00:30	-5.7	-6.5	4.8	258.7	dpDep
00:45	-5.8	-7.3	6.3	278.1	dpDep
01:00	-6.3	-7.4	5.1	258.8	dpDep
01:15	-6.6	-7.6	6.1	254.5	dpDep
01:30	-6.6	-7.6	4.8	258.0	dpDep
01:45	-6.7	-7.9	5.2	267.6	dpDep
02:00	-6.6	-7.6	8.6	300.8	dpDep
02:15	-6.7	-7.8	6.6	342.9	dpDep
02:30	-6.5	-7.6	2.9	339.3	dpDep
02:45	-6.6	-7.6	1.2	247.3	dpDep
03:00	-6.8	-7.8	4.2	346.1	dpDep
03:15	-6.8	-7.8	2.7	311.5	dpDep
03:30	-6.9	-7.9	8.7	325.3	dpDep
03:45	-7.0	-8.2	7.9	337.9	dpDep
04:00	-7.1	-8.2	11.8	330.1	dpDep
04:15	-7.1	-8.5	8.3	343.1	dpDep
04:30	-7.0	-8.5	5.0	269.6	dpDep
04:45	-7.1	-8.7	11.5	326.2	dpDep
05:00	-7.2	-9.2	11.3	343.8	dpDep
05:15	-7.2	-9.6	5.4	341.1	dpDep
05:30	-7.3	-9.8	15.5	331.1	dpDep
05:45	-7.4	-10.1	14.9	321.6	dpDep
06:00	-7.3	-10.2	10.4	326.0	dpDep
06:15	-7.2	-10.0	9.2	345.8	dpDep
06:30	-6.9	-10.1	10.1	308.8	dpDep
06:45	-6.9	-9.9	12.2	325.1	dpDep
07:00	-6.9	-10.0	9.3	319.6	dpDep
07:15	-6.9	-9.8	14.3	313.8	dpDep
07:30	-6.9	-9.8	2.2	343.5	dpDep
07:45	-7.0	-9.7	7.6	353.8	dpDep
08:00	-7.1	-9.7	9.3	4.7	dpDep
08:15	-7.2	-9.7	9.5	355.5	dpDep
08:30	-6.6	-10.9	16.0	313.8	dpDep
08:45	-6.1	-11.5	9.2	340.0	dpDep
09:00	-6.2	-12.5	12.9	336.9	260
09:15	-6.5	-11.9	11.2	1.0	275
09:30	-6.5	-12.1	11.2	358.4	303
09:45	-6.4	-12.1	20.3	354.1	279
10:00	-6.6	-11.9	14.2	4.1	336
10:15	-6.8	-11.9	11.7	357.4	297
10:30	-6.7	-11.8	10.2	29.5	284
10:45	-6.6	-11.9	16.3	348.0	319
11:00	-6.7	-12.1	6.1	342.9	271
11:15	-6.6	-12.6	21.0	358.3	295
---- Problems in Detail ----
A 'fliswarm reconnect' produces the following warning messages and no GFA connection is made:
23:32:06 CmdStarted: APO.doravetz 14 fliswarm reconnect
23:32:06 APO.doravetz 14 fliswarm >
23:32:06 APO.doravetz 14 fliswarm w text="Node gfa6 is not pinging back or the Docker daemon is not running. Try rebooting the computer."
23:32:06 APO.doravetz 14 fliswarm w text="Node gfa4 is not pinging back or the Docker daemon is not running. Try rebooting the computer."
23:32:06 APO.doravetz 14 fliswarm w text="Node fvc is not pinging back or the Docker daemon is not running. Try rebooting the computer."
23:32:06 APO.doravetz 14 fliswarm w text="Node gfa5 is not pinging back or the Docker daemon is not running. Try rebooting the computer."
23:32:06 APO.doravetz 14 fliswarm w text="Node gfa1 is not pinging back or the Docker daemon is not running. Try rebooting the computer."
23:32:06 APO.doravetz 14 fliswarm w text="Node gfa3 is not pinging back or the Docker daemon is not running. Try rebooting the computer."
23:32:06 APO.doravetz 14 fliswarm w text="Node gfa2 is not pinging back or the Docker daemon is not running. Try rebooting the computer."
However, using a 'fliswarm reconnect -c gfa' works fine.
A 'Failed executing the correction trajectory':
04:12:15 APO.doravetz: 04:08:00 APO.doravetz 121 jaeger f error=\"Failed processing image: Failed executing the correction trajectory: Something went wrong during the trajectory: Some positioners did not reach their destinations. Trajectory failed 7.24 seconds after start\"
04:08:00 CmdDone: APO.doravetz 121 jaeger fvc loop
04:13:30 CmdDone: APO.doravetz 123 jaeger fvc loop
04:14:36 CmdDone: APO.doravetz 125 jaeger fvc loop
04:28:58 .jaeger 0 jaeger w text=\"Positioner 63 replied to SEND_TRAJECTORY_DATA UID=5122 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)\"
04:28:59 APO.doravetz_2.jaeger 188 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59628/trajectory-59628-0055.json
04:28:59 APO.doravetz_2.jaeger 188 jaeger f error=\"Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed\"
06:16:25 APO.doravetz_2.jaeger 145 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed"
===============================================
03:36:52  Jaeger collision:  184
04:13:26  Jaeger collision:  444
04:14:30  Jaeger collision:  444
06:35:03  Jaeger collision:  184
===============================================
04:20:55 APO.doravetz: 04:20:04 APO.doravetz 129 jaeger f error=\"Failed processing image: The exposure succeeded but did not output the filename.\"