Date: Sun, 25 Feb 2024 10:56:15 -0700
Subject: Re: [sloan-25mlog 7055] 2.5m obslog 20240224 (SJD 60365)
From: John Downey
On 2/25/24 06:17, dmbiz@apo.nmsu.edu wrote:
>                         APACHE POINT OBSERVATORY SDSS 2.5M
>
>                                   OBSERVING LOG
>                            Saturday February 24, 2024
>
>                                    (SJD 60365)
>
>
> Notes
> -----
> - dust collected:  0.5K
>
> - 48 min were lost to the micro not talking BOSS error. Controller reloads didn't help.
>    Boxes and connectors on the telescope were inspected. Finally, the bossICC restart
>    helped restore connection.
Hi Dmitry - good job getting it sorted out.  My only observation at this
point - and this is 100% the clarity of hindsight - would be to try the
bossICC restart right after the reload controllers fails.  If that too
fails, then take the time to go look for connection issues.  In the
historical context, this is a very minor point, you did very well to get
this resolved in under an hour regardless.
My feeling overall is that examples of this failure mode have been much
less frequent and, like this one, of much shorter duration than they
used to be before the 2020 LN2 line reroute. Re-routing the LN2 lines
away from the saddlebag was chosen as a 90% solution because it was
clear that the problem came from condensation in the SCSI connectors
between the saddlebags and the micro chassis, condensation caused by the
locally cold conditions during a camera fill.  The LN2 lines were routed
behind and above those SCSI connectors, on the order of one cm from
them.  Moving them more than 15 cm away has largely solved the hours
long or sometimes days long problems we used to have.
However, those SCSI connectors remain old, dating to the camera upgrade
for BOSS from the original SDSS cameras.  And we are using them well
outside their designed environmental conditions.  In this case, I
believe the micro not talking failure mode was just a glitch from which
the software was unable to recover, but that should not rule out the
possibility of a corroded connection to the saddlebag causing the micro
to lose it's mind.  In that case, I do not expect a bossICC reboot to
produce such a satisfactory result, so it is a worst case condition of
which we must remain aware.
>
>
> ____________________________________________________________
> Logs of Nightly APO Activities
> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html