Subject: Re: [sloan-25mlog 4541] Re: 2.5m obslog 20190617 (SJD 58652)
From: Karen Kinemuchi
Date: Tue, 18 Jun 2019 14:15:35 -0600
> On Jun 18, 2019, at 9:49 AM, John Downey  wrote:
>>=20
>> (4) The BOSS DAQ controllers were intermittently unresponsive during =
checkout.
>> No debris, no bugs seen, but jiggling the ethernet cable seemed to =
restore
>> consistent communication thereafter.
>=20
> Did this start with just one DAQ or Micro and then spread to the =
other?
>=20
If I remember correctly, the first warning I saw was for Sp2, and after =
trying to reloadControllers, the boss ping command would then report =
that BOTH of them were dead.  It=E2=80=99s hard to catch the first =
instance of a non-responding DAQ as it is only one line of red, but you =
notice something is wrong when the BOSS instrument gui appears to have =
hung at the pre-read/flush stage.
After a boss ping command and you see the daqs are not communicating, =
the next thing that happens is a critical alert is thrown and the alerts =
window will tell you that the SP?Micro is not talking.  Last night it =
was both Micros not talking, so I had to search for the Micro box and =
inspect them. =20
You can see the green light next to the ethernet cables (peel back the =
black tape a little), and I usually inspect for debris or moths.  I =
don=E2=80=99t unplug the ethernet, but I do firmly seat them in the =
socket.  For whatever reason, that=E2=80=99s what makes this problem go =
away.
Since this happened during afternoon checkout and we didn=E2=80=99t do =
any co-obs with MaNGA throughout the night, there are no additional =
incidences to report.  When we observed cart 6 at the beginning of the =
night, the BOSS communication behaved normally.
Sorry I can=E2=80=99t provide more clues to this problem.
> That is more or less what we saw the last time at the end of April =
(58603), and what I recall seeing from startup after summer shutdown.
> It would be good to track whether it's always the same one that starts =
the problem.  On 58603 it was SP2's DAQ.
>=20
>>=20
>>=20
>> Rebooted the MIGS box while checking the spectrographs.  Redoing =
mirror numbers
>> to fix the out of spec M2 numbers.
>>=20
>=20
> M2 mig numbers probably off due to cool-warm-cool cycling since we had =
the enclosure off for handoff checks.  If the mig readings are updating =
correctly, home them and see if you get close to the nominals.  For M2 =
it should only take one home cycle, but if it didn't bring them back to =
spec I might try another before accepting new ones. =20
>=20
> I had mentioned accepting new ones yesterday evening via email before =
going back and looking at how they had been tracking nominal the last =
week or so, not realizing what a sudden and large change this =
represented.  After looking at some previous night logs it was clear =
that a home should fix it, but Karen had already accepted new nominals - =
so she had to do it again after homing.
>=20
> The one thing is that homing should be done when the telescope has =
been at a stable temperature for a few hours - which generally won't be =
a problem for night staff, but a good thing to remember for us working =
during the day. If we home during a warm day, we'll probably see greater =
inconsistencies between readings and nominals than if we home at similar =
temps to when nominals were last taken.
>=20
Yeah, sorry about that.  While I was having to deal with the ethernet =
cables and the micro boxes, the telescope was at zenith so I could reach =
the MIGS box.  I figured while I was there, I could reach the MIGS =
reboot button and then take care of the M2 numbers after I fixed the =
BOSS issue.  Since this all happened around when I started the lower =
enclosure ventilation (to gently bring the building to ambient), the =
temps would have been unstable even then to do a home of M2.  We chose =
to do the M2 home later, which probably had the telescope at a much more =
stable temp.  We reaccepted the nominals again, and I believe they are =
back to the previous values of last week (I=E2=80=99ll need to check the =
log).