Subject: Re: [sloan-25mlog 6815] 2.5m obslog 20230913 (SJD 60201)
From: Conor Sayres
Date: Thu, 14 Sep 2023 12:19:36 -0400
Ah, ok that probably explains it.  I did see the robots moving from loop =
to loop, but I guess they were just inching their way to their goals =
from home.  Yes, that=E2=80=99s probably it!
This brings up a nuance that=E2=80=99s a little confusing.  jaeger =
configuration random automatically moves the robots (without needing a =
configuration execute).  When loading a configuration from a design, you =
need to explicitly execute it after it=E2=80=99s loaded. =20
> On Sep 14, 2023, at 12:06 PM, Audrey Oravetz  =
wrote:
>=20
>> Hi Audrey, All,
>>=20
>> Thanks for the tests.  Strange that FVC rms is so different for real
>> designs vs random designs.  The numbers spit out from the real =
designs are
>> totally insane.  Must be a bug somewhere still, and hopefully not a
>> problem with the designs themselves.  I=C3=A2=C2=80=C2=99m sure =
we=C3=A2=C2=80=C2=99ll figure that out
>> soon.  For the moment, lets hold off on configuring for real designs
>> incase we=C3=A2=C2=80=C2=99re doing something strange with the =
robots.
>=20
> After going back through my notes, I believe the reason for this is
> because the robots were never executed and never moved off home. I =
should
> have spotted this error. Can I try again tonight?
>=20
>>=20
>> I think the fvc loops for the random designs actually look pretty =
good,
>> perc90 seemed to achieve 7-8 microns after 3 loops.  Those with =
higher rms
>> must be explained by a few very large outliers (likely the disabled
>> robots).  Anyway it gives me some stuff to look at.
>>=20
>> Last comment, you might use =C3=A2=C2=80=C2=9Cjaeger configuration =
reverse=C3=A2=C2=80=C2=9D rather than
>> =C3=A2=C2=80=C2=9Cjaeger unwind=C3=A2=C2=80=C2=9D.  That saves some =
time and is more guaranteed to work
>> without deadlocking (and needing =C3=A2=C2=80=C2=9Cexplodes=C3=A2=C2=80=
=C2=9D etc.)
>=20
> There were a few instances where 'reverse' gave many OUT-OF-RANGE =
errors
> so 'unwind' was the goto method.
>=20
> Audrey
>=20
>>=20
>> Conor
>>=20
>>> On Sep 14, 2023, at 4:51 AM, asimmons@apo.nmsu.edu wrote:
>>>=20
>>>=20
>>>                      APACHE POINT OBSERVATORY SDSS 2.5M
>>>=20
>>>                                OBSERVING LOG
>>>                         Wednesday September 13, 2023
>>>=20
>>>                                 (SJD 60201)
>>>=20
>>> ----  Observing Team ----
>>>=20
>>> Audrey Oravetz
>>>=20
>>>=20
>>> ---- Observing Plan ----
>>>=20
>>> FVC testing
>>>=20
>>>=20
>>> ---- Observing Summary ----
>>>=20
>>> Tested FVC loops with jaeger modifications and the new command:
>>>=20
>>> =C3=A2=C2=80=C2=9Cjaeger fvc loop =C3=A2=C2=80=C2=94max-iterations 5 =
=C3=A2=C2=80=C2=94target-90-percentile 5
>>> =C3=A2=C2=80=C2=94no-write-summary"
>>>=20
>>> The data can be found in the log body, but in essence the KRONOS =
designs
>>> and random --danger designs both did not converge within 5 FVC =
cycles.
>>>=20
>>> With Jose's help, now designs can be loaded from KRONOS without =
"jaeger
>>> e error=3D"Failed retrieving design: operands could not be broadcast
>>> together with shapes ... ", but it was a temporary fix.
>>>=20
>>> Worked on additional FVC scans:
>>>=20
>>> Round 1: fimg-fvc1n-0033.fits - fimg-fvc1n-0143.fits(starting and =
ending
>>> with ROT=3D0)
>>> Round 2: fimg-fvc1n-0144.fits - fimg-fvc1n-0254.fits (starting with
>>> ROT=3D5)
>>>=20
>>>=20
>>> ---- Observing Log ----
>>>=20
>>> Night:
>>> =3D=3D=3D=3D=3D=3D
>>>=20
>>> Ran APOGEE evening cals.
>>> (observers39) [observer@sdss5-hub: ~]$ list_ap
>>> APOGEE data list,   mjd=3D60201,  all
>>> raw_data:  /data/apogee/utr_cdr/60201/apRaw-*.fits
>>> archive:  /data/apogee/archive/60201/apR-[a,b,c]-*.apz
>>> quickred:  /data/apogee/quickred/60201/apq-*.fits
>>> =
--------------------------------------------------------------------------=
----
>>> mjd   i   UT   File/Exp  Imtype Nread A/B Dth  Design-Cfg  Arch =
Offset
>>> %Flux/FPI Comm
>>> =
--------------------------------------------------------------------------=
----
>>> 60201   1 01:21  46390001    Dark   60  A  13.0  ----------  a-b-c   =
 -
>>> -
>>> 60201   2 01:23  46390002  QuarFlat 10  A  13.0  ----------  a-b-c   =
 -
>>> -
>>> 60201   3 01:26  46390003  Arc-Thar 12  B  13.5  ----------  a-b-c  =
1.03
>>> -
>>> 60201   4 01:33  46390004  Arc-Une  40  B  13.5  ----------  a-b-c   =
 -
>>> -
>>> 60201   5 01:35  46390005  Arc-Thar 12  A  13.0  ----------  a-b-c  =
0.45
>>> -
>>> 60201   6 01:42  46390006  Arc-Une  40  A  13.0  ----------  a-b-c   =
 -
>>> -
>>> 60201   7 01:53  46390007    Dark   60  A  13.0  ----------  a-b-c   =
 -
>>> -
>>> 60201   8 01:59  46390008  Arc----  30  A  13.0  ----------  a-b-c   =
 -
>>> 86  FPI
>>> =
--------------------------------------------------------------------------=
----
>>>=20
>>> Turned on the FPS. Force filled B1. Turned on the rotator amplifier.
>>> Unable to move in alt/az just like yesterday.
>>>=20
>>> Restarted jaeger with:
>>> sdss5@sdss5-hub > systemctl --user restart jaeger
>>>=20
>>> Disabled robots 535 and 478 because of repeated false alarms.
>>>=20
>>> Random configurations a few times to test:
>>> 03:10:33 CmdStarted: APO.aoravetz 51 jaeger configuration random
>>> 03:12:38 .jaeger 0 jaeger i fps_status=3D0x11
>>> 03:12:39 APO.aoravetz 51 jaeger i folded=3DF
>>> 03:12:39 APO.aoravetz 51 jaeger i text=3D"All positioners have =
reached
>>> their destinations."
>>> 03:12:39 APO.aoravetz 51 jaeger :
>>> 03:12:39 CmdDone: APO.aoravetz 51 jaeger configuration random
>>> etc.
>>>=20
>>> Random --danger configurations a few times to test:
>>> 03:23:16 CmdStarted: APO.aoravetz 57 jaeger configuration random
>>> --danger
>>> 03:23:16 APO.aoravetz 57 jaeger >
>>> 03:23:17 APO.aoravetz 57 jaeger i text=3D"Creating random =
configuration."
>>> 03:25:50 .jaeger 0 jaeger i fps_status=3D0x11
>>> 03:25:51 APO.aoravetz 57 jaeger i folded=3DF
>>> 03:25:51 APO.aoravetz 57 jaeger i text=3D"All positioners have =
reached
>>> their destinations."
>>> 03:25:51 APO.aoravetz 57 jaeger :
>>> 03:25:52 CmdDone: APO.aoravetz 57 jaeger configuration random =
--danger
>>> etc
>>>=20
>>> =3D=3D=3D=3D=3D
>>>=20
>>> Gathering FVC loops from existing KRONOS designs:
>>>=20
>>> 03:38:11 CmdStarted: APO.aoravetz_2 3 jaeger configuration preload
>>> 03:38:11 APO.aoravetz_2 3 jaeger >
>>> 03:38:11 APO.aoravetz_2 3 jaeger i text=3D"Preloading design =
361228."
>>> 03:39:31 APO.aoravetz_2 4 jaeger : text=3D"Configuration 10605 =
loaded."
>>> 03:39:31 CmdDone: APO.aoravetz_2 4 jaeger configuration load
>>> --from-preloaded
>>>=20
>>> 03:40:19 CmdStarted: APO.aoravetz 64 jaeger fvc loop =
--max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> 03:40:45 APO.aoravetz 64 jaeger i fvc_rms=3D17136.19
>>> 03:40:45 APO.aoravetz 64 jaeger i fvc_perc_90=3D24265.6
>>>=20
>>> 03:41:26 APO.aoravetz 64 jaeger i fvc_rms=3D17324.84
>>> 03:41:26 APO.aoravetz 64 jaeger i fvc_perc_90=3D24759.3
>>>=20
>>> 03:42:06 APO.aoravetz 64 jaeger i fvc_rms=3D17708.52
>>> 03:42:06 APO.aoravetz 64 jaeger i fvc_perc_90=3D26025.9
>>>=20
>>> 03:42:46 APO.aoravetz 64 jaeger i fvc_rms=3D18087.09
>>> 03:42:46 APO.aoravetz 64 jaeger i fvc_perc_90=3D27451.9
>>>=20
>>> 03:43:26 APO.aoravetz 64 jaeger i fvc_rms=3D18400.06
>>> 03:43:26 APO.aoravetz 64 jaeger i fvc_perc_90=3D29251.7
>>>=20
>>> ********************************************************
>>> 03:39:26  Field-Design-Configuration: 104204-361228-10605
>>> ********************************************************
>>> 03:43:29  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 18400.06
>>>          fvc_deltarms=3D -312.97
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0005.fits
>>>          fvc_perc_90=3D 29251.70
>>>          fvc_percent_reached=3D 4.90
>>>=20
>>> 03:43:29 APO.aoravetz 64 jaeger f error=3D"The FVC loop failed."
>>> 03:43:29 CmdDone: APO.aoravetz 64 jaeger fvc loop --max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> Trying again...
>>> 03:44:06 CmdStarted: APO.aoravetz 65 jaeger fvc loop =
--max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> 03:44:32 APO.aoravetz 65 jaeger i fvc_rms=3D18399.72
>>> 03:44:32 APO.aoravetz 65 jaeger i fvc_perc_90=3D29250.8
>>>=20
>>> 03:45:11 APO.aoravetz 65 jaeger i fvc_rms=3D18534.77
>>> 03:45:11 APO.aoravetz 65 jaeger i fvc_perc_90=3D31056.6
>>>=20
>>> 03:45:50 APO.aoravetz 65 jaeger i fvc_rms=3D18458.19
>>> 03:45:50 APO.aoravetz 65 jaeger i fvc_perc_90=3D32506.5
>>>=20
>>> 03:46:29 APO.aoravetz 65 jaeger i fvc_rms=3D18178.22
>>> 03:46:29 APO.aoravetz 65 jaeger i fvc_perc_90=3D32881.3
>>>=20
>>> 03:47:07 APO.aoravetz 65 jaeger i fvc_rms=3D17700.65
>>> 03:47:07 APO.aoravetz 65 jaeger i fvc_perc_90=3D32489.8
>>>=20
>>> 03:47:10 APO.aoravetz 65 jaeger f error=3D"The FVC loop failed."
>>> 03:47:10 CmdDone: APO.aoravetz 65 jaeger fvc loop --max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> 03:47:10  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 17700.65
>>>          fvc_deltarms=3D 477.57
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0010.fits
>>>          fvc_perc_90=3D 32489.80
>>>          fvc_percent_reached=3D 19.20
>>>=20
>>> 03:49:10 CmdStarted: APO.aoravetz 76 jaeger unwind
>>> 03:49:10 APO.aoravetz 76 jaeger >
>>> 03:49:10 APO.aoravetz 76 jaeger i folded=3DF
>>> 03:49:10 APO.aoravetz 76 jaeger f error=3D"Command 'unwind' failed =
because
>>> of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot
>>> generate paths'. See traceback in the log for more information."
>>> jaeger explode 10
>>> jaeger unwind
>>>=20
>>> =3D=3D=3D
>>>=20
>>> Another KRONOS design to test:
>>>=20
>>> 03:56:46 CmdStarted: APO.aoravetz_2 5 jaeger configuration preload
>>> 03:56:46 APO.aoravetz_2 5 jaeger >
>>> 03:56:46 APO.aoravetz_2 5 jaeger i text=3D"Preloading design =
364229."
>>> 03:56:46 CmdStarted: jaeger.jaeger 21 cherno get-scale --max-age =
3600
>>> 03:56:46 jaeger.jaeger 21 cherno >
>>> 03:56:46 jaeger.jaeger 21 cherno w text=3D"Not enough points to =
calculate
>>> median scale."
>>> 03:56:46 jaeger.jaeger 21 cherno : scale_median=3D-999.0
>>> 03:56:46 CmdDone: jaeger.jaeger 21 cherno get-scale --max-age 3600
>>> 03:56:46 APO.aoravetz_2 5 jaeger w text=3D"Invalid guider scale."
>>> 03:56:46 APO.aoravetz_2 5 jaeger i text=3D"Focal scale: 0.999759."
>>> =
configuration_snapshot=3D/data/logs/jaeger/configuration_snapshots/60201/c=
onfiguration_snapshot_60201_10606.pdf
>>> 03:58:06 APO.aoravetz_2 6 jaeger : text=3D"Configuration 10606 =
loaded."
>>> 03:58:06 CmdDone: APO.aoravetz_2 6 jaeger configuration load
>>> --from-preloaded
>>>=20
>>> 03:58:25 CmdStarted: APO.aoravetz 87 jaeger fvc loop =
--max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> 03:58:51 APO.aoravetz 87 jaeger i fvc_rms=3D17464.95
>>> 03:58:51 APO.aoravetz 87 jaeger i fvc_perc_90=3D24331.6
>>>=20
>>> 03:59:31 APO.aoravetz 87 jaeger i fvc_rms=3D17700.93
>>> 03:59:31 APO.aoravetz 87 jaeger i fvc_perc_90=3D25398.2
>>>=20
>>> 04:00:11 APO.aoravetz 87 jaeger i fvc_rms=3D18167.54
>>> 04:00:11 APO.aoravetz 87 jaeger i fvc_perc_90=3D26826.8
>>>=20
>>> 04:00:52 APO.aoravetz 87 jaeger i fvc_rms=3D18683.88
>>> 04:00:52 APO.aoravetz 87 jaeger i fvc_perc_90=3D28473.3
>>>=20
>>> 04:01:32 APO.aoravetz 87 jaeger i fvc_rms=3D19118.73
>>> 04:01:32 APO.aoravetz 87 jaeger i fvc_perc_90=3D30351.4
>>>=20
>>> ********************************************************
>>> 03:58:03  Field-Design-Configuration: 104657-364229-10606
>>> ********************************************************
>>> 04:01:35  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 19118.73
>>>          fvc_deltarms=3D -434.85
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0015.fits
>>>          fvc_perc_90=3D 30351.40
>>>          fvc_percent_reached=3D 2.60
>>>=20
>>> 04:01:35 APO.aoravetz 87 jaeger f error=3D"The FVC loop failed."
>>> 04:01:35 CmdDone: APO.aoravetz 87 jaeger fvc loop --max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> =3D=3D=3D=3D=3D
>>>=20
>>> Now trying with a random --danger configuration
>>>=20
>>> 04:02:48 CmdStarted: APO.aoravetz 89 jaeger configuration random
>>> --danger
>>>=20
>>> 04:05:32 APO.aoravetz 89 jaeger i folded=3DF
>>> 04:05:32 APO.aoravetz 89 jaeger i text=3D"All positioners have =
reached
>>> their destinations."
>>> 04:05:32 APO.aoravetz 89 jaeger :
>>> 04:05:33 CmdDone: APO.aoravetz 89 jaeger configuration random =
--danger
>>>=20
>>> 04:05:54 CmdStarted: APO.aoravetz 90 jaeger fvc loop =
--max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> 04:06:20 APO.aoravetz 90 jaeger i fvc_rms=3D50.58
>>> 04:06:20 APO.aoravetz 90 jaeger i fvc_perc_90=3D77.2
>>>=20
>>> 04:06:58 APO.aoravetz 90 jaeger i fvc_rms=3D8.98
>>> 04:06:58 APO.aoravetz 90 jaeger i fvc_perc_90=3D10.6
>>>=20
>>> 04:07:33 APO.aoravetz 90 jaeger i fvc_rms=3D26.62
>>> 04:07:33 APO.aoravetz 90 jaeger i fvc_perc_90=3D7.5
>>>=20
>>> 04:08:09 APO.aoravetz 90 jaeger i fvc_rms=3D21.49
>>> 04:08:09 APO.aoravetz 90 jaeger i fvc_perc_90=3D7.5
>>>=20
>>> 04:08:44 APO.aoravetz 90 jaeger i fvc_rms=3D24.37
>>> 04:08:44 APO.aoravetz 90 jaeger i fvc_perc_90=3D7.6
>>>=20
>>> 04:08:47  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 24.37
>>>          fvc_deltarms=3D -2.88
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0020.fits
>>>          fvc_perc_90=3D 7.60
>>>          fvc_percent_reached=3D 68.20
>>>=20
>>> 04:08:47 APO.aoravetz 90 jaeger f error=3D"The FVC loop failed."
>>> 04:08:47 CmdDone: APO.aoravetz 90 jaeger fvc loop --max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> 04:14:39 CmdStarted: APO.aoravetz 91 jaeger unwind
>>>=20
>>> 04:14:40 APO.aoravetz 91 jaeger f error=3D"Command 'unwind' failed =
because
>>> of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot
>>> generate paths'. See traceback in the log for more information."
>>> 04:14:40 CmdDone: APO.aoravetz 91 jaeger unwind
>>>=20
>>> jaeger explode 10
>>> jeager unwind
>>>=20
>>> =3D=3D=3D=3D=3D
>>>=20
>>> Another trial with random --danger:
>>>=20
>>> 04:17:47 CmdStarted: APO.aoravetz 94 jaeger configuration random
>>> --danger
>>> 04:17:47 APO.aoravetz 94 jaeger >
>>> 04:17:47 APO.aoravetz 94 jaeger i text=3D"Creating random =
configuration."
>>> 04:17:48 .jaeger 0 jaeger i
>>> =
snapshot=3D/data/logs/jaeger/snapshots/60201/fps_snapshot_60201_0034.pdf
>>>=20
>>> 04:20:44 APO.aoravetz 95 jaeger i fvc_rms=3D52.73
>>> 04:20:44 APO.aoravetz 95 jaeger i fvc_perc_90=3D74.6
>>>=20
>>> 04:21:22 APO.aoravetz 95 jaeger i fvc_rms=3D10.69
>>> 04:21:22 APO.aoravetz 95 jaeger i fvc_deltarms=3D42.04
>>>=20
>>> FAILED AFTER LOOP #2
>>>=20
>>> 04:21:30  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 10.69
>>>          fvc_deltarms=3D 42.04
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0022.fits
>>>          fvc_perc_90=3D 9.80
>>>          fvc_percent_reached=3D 57.70
>>>=20
>>> 04:21:27 .jaeger 0 jaeger w text=3D"Positioner 842 replied to
>>> START_TRAJECTORY UID=3D10735 with 'INVALID_TRAJECTORY'.
>>> (JaegerUserWarning)"
>>> 04:21:27 .jaeger 0 jaeger f text=3D"A collision was detected in =
positioner
>>> 507. Sending SEND_TRAJECTORY_ABORT and locking the FPS."
>>> 04:21:27 .jaeger 0 jaeger w text=3D"Locking the FPS. =
(JaegerUserWarning)"
>>> 04:21:28 APO.aoravetz 95 jaeger e error=3D"Failed processing image: =
Failed
>>> executing the correction trajectory: Something went wrong during the
>>> trajectory: START_TRAJECTORY failed"
>>>=20
>>> =3D=3D=3D=3D
>>>=20
>>> Trying again...
>>>=20
>>> 04:24:06 CmdStarted: APO.aoravetz 99 jaeger configuration random
>>> --danger
>>> 04:24:06 APO.aoravetz 99 jaeger >
>>> 04:24:07 APO.aoravetz 99 jaeger i text=3D"Creating random =
configuration."
>>> 04:24:09 .jaeger 0 jaeger i
>>> =
snapshot=3D/data/logs/jaeger/snapshots/60201/fps_snapshot_60201_0039.pdf
>>>=20
>>> 04:28:21 APO.aoravetz 100 jaeger i fvc_rms=3D57.6
>>> 04:28:21 APO.aoravetz 100 jaeger i fvc_perc_90=3D79.0
>>>=20
>>> 04:28:59 APO.aoravetz 100 jaeger i fvc_rms=3D53.93
>>> 04:28:59 APO.aoravetz 100 jaeger i fvc_perc_90=3D10.5
>>>=20
>>> 04:29:33 APO.aoravetz 100 jaeger i fvc_rms=3D20.4
>>> 04:29:33 APO.aoravetz 100 jaeger i fvc_perc_90=3D7.4
>>>=20
>>> 04:29:48 .jaeger 67 jaeger i text=3D"[FVC]: Processed HDU written to
>>> /data/fcam/60201/proc-fimg-fvc1n-0025.fits"
>>> 04:30:07 APO.aoravetz 100 jaeger i fvc_rms=3D32.51
>>> 04:30:07 APO.aoravetz 100 jaeger i fvc_perc_90=3D7.4
>>>=20
>>> 04:30:40 APO.aoravetz 100 jaeger i fvc_rms=3D31.63
>>> 04:30:40 APO.aoravetz 100 jaeger i fvc_perc_90=3D7.0
>>>=20
>>> 04:30:44  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 31.63
>>>          fvc_deltarms=3D 0.88
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0027.fits
>>>          fvc_perc_90=3D 7.00
>>>          fvc_percent_reached=3D 72.60
>>>=20
>>> 04:30:44 APO.aoravetz 100 jaeger f error=3D"The FVC loop failed."
>>> 04:30:44 CmdDone: APO.aoravetz 100 jaeger fvc loop --max-iterations =
5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> =3D=3D=3D=3D
>>>=20
>>> One more time...
>>>=20
>>> 04:33:26 CmdStarted: APO.aoravetz 102 jaeger configuration random
>>> --danger
>>> 04:33:26 APO.aoravetz 102 jaeger >
>>> 04:33:27 APO.aoravetz 102 jaeger i text=3D"Creating random =
configuration."
>>>=20
>>> 04:36:42 CmdStarted: APO.aoravetz 103 jaeger fvc loop =
--max-iterations 5
>>> --target-90-percentile 5 --no-write-summary
>>> 04:36:42 APO.aoravetz 103 jaeger >
>>>=20
>>> 04:37:01 jaeger.jaeger 53 fliswarm i
>>> filename=3Dfvc,fvc1n,/data/fcam/60201/fimg-fvc1n-0028.fits
>>>=20
>>> 04:37:09 APO.aoravetz 103 jaeger i fvc_rms=3D51.28
>>> 04:37:09 APO.aoravetz 103 jaeger i fvc_perc_90=3D73.5
>>>=20
>>> 04:37:45 APO.aoravetz 103 jaeger i fvc_deltarms=3D40.86
>>> 04:37:45 APO.aoravetz 103 jaeger i fvc_perc_90=3D10.1
>>>=20
>>> 04:38:18 APO.aoravetz 103 jaeger i fvc_rms=3D9.1
>>> 04:38:18 APO.aoravetz 103 jaeger i fvc_perc_90=3D7.4
>>>=20
>>> 04:38:52 APO.aoravetz 103 jaeger i fvc_rms=3D8.99
>>> 04:38:52 APO.aoravetz 103 jaeger i fvc_perc_90=3D7.4
>>>=20
>>> 04:39:25 APO.aoravetz 103 jaeger i fvc_rms=3D8.42
>>> 04:39:25 APO.aoravetz 103 jaeger i fvc_perc_90=3D7.0
>>>=20
>>> 04:39:28  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 8.42
>>>          fvc_deltarms=3D 0.57
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0032.fits
>>>          fvc_perc_90=3D 7.00
>>>          fvc_percent_reached=3D 70.60
>>>=20
>>> 04:39:28 APO.aoravetz 103 jaeger f error=3D"The FVC loop failed."
>>> 04:39:28 CmdDone: APO.aoravetz 103 jaeger fvc loop --max-iterations =
5
>>> --target-90-percentile 5 --no-write-summary
>>>=20
>>> =3D=3D=3D=3D=3D=3D=3D=3D
>>>=20
>>> FVC rotational scans
>>>=20
>>>       a. configure robots randomly =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=85=
=C2=93jaeger configuration random
>>> =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=A2=C2=80=C2=9Ddanger=C3=83=C2=A2=C3=A2=
=C2=82=C2=AC=C3=82=C2=9D
>>>       b. move the rotator in 10 degree increments between 0-360 (0, =
10,
>>> 20, =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=82=C2=A6
>>> 340, 350)
>>>       c. at each rotator position, take 3x =C3=83=C2=A2=C3=A2=C2=82=C2=
=AC=C3=85=C2=93jaeger fvc loop
>>> =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=A2=C2=80=C2=9Dno-apply
>>> =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=A2=C2=80=C2=9Dfbi-level =
4=C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=82=C2=9D
>>>       d. send robots home =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=85=C2=93ja=
eger configuration reverse"
>>>       e. repeat steps a-d 4 times.  Every other time offset the =
rotator
>>> positions
>>> by 5
>>> degrees (5, 15, 25, =C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=82=C2=A6, 345, =
355)
>>>       f. note the first and last fvc image for each full rotator =
scan,
>>> no need to
>>> keep
>>> track of every image, I can find the rotator angles from the =
headers.  I
>>> only need
>>> to know when you=C3=83=C2=A2=C3=A2=C2=82=C2=AC=C3=A2=C2=84=C2=A2ve =
begun a new scan with different robot
>>> positions.
>>>=20
>>> ****FIRST ROUND****
>>>=20
>>> 05:44:42 CmdStarted: APO.aoravetz2 884 jaeger configuration random
>>> --danger
>>> 05:44:42 APO.aoravetz2 884 jaeger >
>>> 05:44:42 APO.aoravetz2 884 jaeger i text=3D"Creating random
>>> configuration."
>>>=20
>>> 05:47:53  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 46.27
>>>          fvc_deltarms=3D 0.57
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0033.fits
>>>          fvc_perc_90=3D 67.20
>>>          fvc_percent_reached=3D 1.40
>>> ...
>>>=20
>>> 06:55:54  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 45.31
>>>          fvc_deltarms=3D 0.57
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0143.fits
>>>          fvc_perc_90=3D 62.90
>>>          fvc_percent_reached=3D 2.40
>>>=20
>>> fimg-fvc1n-0033.fits - fimg-fvc1n-0143.fits
>>>=20
>>> jaeger unwind
>>>=20
>>> *** SECOND ROUND****
>>>=20
>>> 07:00:50 CmdDone: APO.aoravetz2 73 jaeger configuration random =
--danger
>>> 07:00:50 .tcc 0 tcc i TCCPos=3Dnan,nan,nan;
>>> AxePos=3D120.998821,90.035696,0.003630;
>>> ObjInstAng=3Dnan,nan,5201391274.1618919;
>>> SpiderInstAng=3Dnan,nan,5201391274.1618919
>>> 07:00:50 .tcc 0 tcc i AxisCmdState=3DHalted,Halted,Halted;
>>> AxisErrCode=3DHaltRequested,HaltRequested,HaltRequested; =
AxisNoTrack=3DTTT
>>> 07:00:54 .jaeger 0 jaeger i
>>> =
snapshot=3D/data/logs/jaeger/snapshots/60201/fps_snapshot_60201_0054.pdf
>>>=20
>>> 07:01:42  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 50.06
>>>          fvc_deltarms=3D 0.57
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0144.fits
>>>          fvc_perc_90=3D 67.60
>>>          fvc_percent_reached=3D 1.20
>>>=20
>>> 08:09:05  FVC LOOP AND BOSS CALS
>>>          fvc_rms=3D 48.97
>>>          fvc_deltarms=3D 0.57
>>>          fvc_filename=3D/data/fcam/60201/fimg-fvc1n-0254.fits
>>>          fvc_perc_90=3D 64.80
>>>          fvc_percent_reached=3D 1.20
>>>=20
>>> fimg-fvc1n-0144.fits - fimg-fvc1n-0254.fits
>>>=20
>>> Turned the FPS off after homing the robots. Started APOGEE morning =
cals.
>>>=20
>>>=20
>>> ---- Mirror Numbers ----
>>>=20
>>>=20
>>> N/A
>>>=20
>>> ---- Telescope Status ----
>>>=20
>>> Status at:
>>> Telescope stowed at:    121.0, 30.0,   0.0  mount
>>> Instrument mounted:     FPS
>>> Counterweights at:          470
>>> LN2 autofill systems:   Connected and turned on
>>> 180L LN2 dewar scales:  SP1    220    lbs,  20   psi
>>>=20
>>>=20
>>> ---- Software Used ----
>>>=20
>>> Get Versions, 09/14/23, 07:44:01
>>> -- STUI software:
>>> STUI          : 1.7.18b1
>>> alerts        : 2.3.0
>>> boss          : 6.0.2-alpha.0
>>> -boss.specMec : 3.0.4
>>> apo           : 3.0.1
>>> fliswarm      : 0.4.0
>>> jaeger        : 1.4.1a0
>>> kaiju         : 1.3.1
>>> coordio       : 1.7.1
>>> fps_calibrati : 2023.07.27+7ea4ef6
>>> cherno        : 0.6.3
>>> astrometrynet : 0.93
>>> hal           : 0.6.0b5
>>> hartmann      : 2.0.2
>>> hub           : 5.0.1a0
>>> mcp           : 2.1
>>> -mcp.plc      : 35
>>> -mcp.azFiduci : v1_149
>>> -mcp.altFiduc : v1_149
>>> -mcp.rotFiduc : v1_149
>>> tcc           : 1.4.0
>>> apogee        : not availble
>>> apogeecal     : not availble
>>> apogeeql      : v1_7_0dev
>>> STUIAdditions : FAILED
>>> Boson         : 0.3.2
>>> -- done --
>>> --------------Other Versions--------------
>>> sdss-obstools       : 3.9.3
>>> TPM                 : 1.2-31-gd7eb
>>> MCP                 : 2.1-1-ge461
>>> PLC                 : V35
>>> Fiducials           : v1_149
>>> idlspec2d           : v6_1_0
>>> Kronos              : 1.6.2a
>>> roboscheduler       : 1.4.0
>>>=20
>>> /dev/sdb1                2.0T  290G  1.8T  15% /home
>>> /dev/sdd1                 22T  9.0T   12T  43% /data
>>>=20
>>> You have new mail in /var/spool/mail/observer
>>>=20
>>> ---- Time Tracking ----
>>>=20
>>> Wed 13 Sep 2023 60201 0.01 02:05 02:20 11:37 18:48 04:07 --- dark ( =
9.3
>>> h) plus 0.2 h for 12 deg design
>>>=20
>>> *=3D*=3D*=3D Observatory: APO
>>> *=3D*=3D*=3D MJD: 60201
>>>      Fill night time in Hours below
>>> *=3D*=3D*=3D Total bright time                     : 0
>>> *=3D*=3D*=3D Bright time, queued observations      : 0
>>> *=3D*=3D*=3D Bright time, commissioning/engineering: 0
>>> *=3D*=3D*=3D Bright time lost to weather           : 0
>>> *=3D*=3D*=3D Bright time lost to technical issues  : 0
>>>=20
>>> *=3D*=3D*=3D Total dark time                       : 9.3
>>> *=3D*=3D*=3D Dark time, queued observations        : 0
>>> *=3D*=3D*=3D Dark time, commissioning/engineering  : 9.3
>>> *=3D*=3D*=3D Dark time lost to weather             : 0
>>> *=3D*=3D*=3D Dark time lost to technical issues    : 0
>>>=20
>>>=20
>>>=20
>>>=20
>>> ____________________________________________________________
>>> Logs of Nightly APO Activities
>>> =
https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/priv=
ate/sloan-25mlog/index.html__;!!K-Hz7m0Vt54!hMp0YZLS_j8sJNO_9eWH5mn6Lv5BHT=
MQjNfJa8bxFYvNOyRC1qKKvBXnhTRkr3ajU_rygGt5HQbcJcn46Ro$
>>=20
>>=20
>=20
>=20