Date: Tue, 29 Nov 2022 12:52:00 -0700
Subject: Re: [sloan-25mlog 6375] Re: 2.5m obslog 20221128 (SJD 59912)
From: "Audrey Oravetz"
Thank you for digging into this further today, Dan.
It proves very difficult at night with keeping observations running at the
highest efficiency possible and sleuthing these sorts of problems when
they arise. Also, this problem happened when my attention was diverted
towards setting up the next field so I did not catch it until ~2-3 minutes
after the error message was posted.
Audrey
>> 10:15:33 hal.hal 10763 boss e text="sp1daq readout failed: Invalid
>> response from Netburner. (> 0x7f30ccfc62b0>>)."
>> The netburner error occurred right before the next gotoField sequence.
>> There was a message 2 minutes after the netburner error that read
>> “Readout timeout expired, aborting exposure." I went through the
>> procedure to recover the science exosure BOSS expNum=349256, but it was
>> already too late. 15 minutes were lost because this exposure was never
>> recovered.
>
> This was an especially odd case because the BOSS exposure state never
> actually went into an ABORTED state to allow for the recovery procedure to
> work, even though the accompanying error message says it was "aborting
> exposure."
>
> The exposure was started:
>
> 2022-11-29 10:00:10,845Z
> Then preread and readout:
>
> 2022-11-29 10:15:10,847Z  2022-11-29 10:15:15,018Z
> But no ABORTED exposureState after that -- the next stage is a flush
> before the test bias after the controller connections were restored:
>
> 2022-11-29 10:21:22,835Z
> It's also interesting that there wasn't an IDLE stage that preceded that
> FLUSHING state.
>
> It doesn't look like the hal "expose" cleanup tasks would've cleared the
> exposure either because hal acknowledged the BOSS expose hadn't completed
> once the timeout message spawned, so hal seemed to have left things be:
>
> 2022-11-29 10:17:09,201Z  cancelling it."
>
> And in the goto-field macro that was run when readout was in progress
> (which is completely normal), there weren't any steps that involved BOSS
> at all. Even so, the goto-field "prepare" stage doesn't look like it would
> attempt to clear any pending BOSS exposure even if BOSS stages had been
> part of the goto-field command this time because there are a bunch of
> "awaits" for any pending BOSS readouts.
>
> Hopefully just a one-off event.
>
> ____________________________________________________________
> Logs of Nightly APO Activities
> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>