Subject: Re: [sloan-25mlog 7541] 2.5m obslog 20241201 (SJD 60646)
From: Kaike
Date: Mon, 2 Dec 2024 10:27:51 -0700
Hi Jose and Dan,
  Could it be possible that the recover procedure fails due to a failure =
of reading out from camera to netburner? When will the data in the =
netburner be cleared (or overridden), in the process of prepare for the =
next exposure or prepare for next readout from camera to the netburner?
Thanks,
-Kaike
> On Dec 2, 2024, at 6:22=E2=80=AFAM, Daniel Oravetz via sloan-25mlog =
 wrote:
>=20
>=20
>                       APACHE POINT OBSERVATORY SDSS 2.5M
>=20
>                                 OBSERVING LOG
>                          Sunday December 01, 2024
>=20
>                                  (SJD 60646)
>=20
> ----  Observing Team ----
>=20
> Dan Oravetz
>=20
>=20
> (3) On 103940-674791-17538, there was a Netburner error, but despite =
many attempts at the recovery procedure the exposure could not be =
recovered from the daq.
>=20
> The science exposure sequence on 103940-674792-17539 aborted in hal =
because the previous BOSS exposure's (376995) readout timed-out as a =
result of a "sp1daq readout failed: Invalid response from Netburner" =
message at 07:22:07Z. However, whenever I tried the usual recovery =
procedure (and I tried many attempts at this), the message from BOSS was =
that there was no exposure to act on; i.e., nothing in the daq to =
re-read or recover.
>=20
> When I was checking the status of the controllers, most of the time =
all three BOSS controllers replied, but a couple of times sp1daq did not =
reply and at least once it actually did show a 'DEAD' status (which =
isn't unusual when we have to recover exposures and is actually the norm =
when we have a Netburner error). After reloading controllers several =
times (which is sometimes necessary during the recovery procedure), and =
making sure that there was an 'OK' state for all three BOSS controllers, =
there still was "no exposure to act on" when I tried to recover the =
exposure.
>=20
> Ultimately, I tried all of the 'boss exposure [...]' status commands =
-- 'pause', 'readout', 'abort', recover', etc. -- to see if I could =
change the BOSS Exp Status and none of them had any effect. When I =
eventually tried a 'boss iccStatus' it did automatically force a readout =
attempt, which at first seemed hopeful, but was unsuccessful with each =
attempt (the readout would just count down to 0.0 and never move beyond =
the 'Reading 0.0' state).
>=20
> After all of these unsuccessful attempts at recovering the exposure, =
all three BOSS controllers were repeatedly replying with an 'OK' state, =
but before restarting bossICC as the next step I was resigned to try the =
'boss clearExposure' command (which clears any attempt at recovering the =
exposure, even though everything up to this point was unsuccessful =
anyway). That command did clear the BOSS Exp Status from the 'Reading =
0.0' state to an 'Idle' state, so I tried a bias and the bias read-out =
successfully. I did not see an urgent need to restart bossICC at that =
point because the BOSS controllers (and BOSS itself) seemed normally =
responsive, so I moved on to the next field.
>=20
> I didn't have any problems with BOSS the rest of the night.
>=20