Date: Tue, 25 May 2021 09:57:09 -0600
Subject: Re: 2.5m obslog 20210524 (SJD 59359)
From: "Daniel Oravetz"
> - in the evening we've got apogee.arrayPower serious alert and were able
>   to fix it, see log. No time lost for the problem.
>
> - connected to this problem, Dylan attempted to see how often the
>   arrayPower was checked in the past, through the Grafana/Influx.
>   Besides our today's few requests during the problem, the arrayPower
>   was not checked any time between January and April. As we can see,
>   to check the apogee.arrayPower, "apogee status" should be issued.
>   We opened COS-50 ticket on investigating this issue (to Jose at
>   the moment, for the further dispatching).
We had discussed this issue in January after a problem on January 11 and
COS-28 was opened:
https://jira.sdss.org/browse/COS-28
In that ticket and the comments therein it was determined that APOGEE
needs to output its status more often for the relevant keywords to be
updated regularly, which would allow alertsActor to monitor the Leach
controller. JohnW recommended that we "wait until later in the spring when
effort on changing the APOGEE ICC to include the BPR telemetry starts in
earnest. At that point, since the ICC will get edited, we can revisit the
issue."
And, FYI, here's the discussion about the issue from the 1/12/21 Ops
Telecon captured in the telecon notes:
https://wiki.sdss.org/pages/viewpage.action?pageId=71172396
"6)Apogee leach detector controller was reset (1/11).
The instrument still takes data, but reads out all zeros. John Wilson
talked to Matt Nelson. The APOGEE team doesn’t think this has ever
happened before. Elevated noise seen in telemetry on the detector
temperature indicates some kind of problem with electric feedback between
the Lakeshore sensors and leach controller. There are instantaneous dips
in the temperatures. Would be good to have an alert to monitor keywords
without messing with the icc controller, but after further discussion,
this may be more work to implement at the moment for a possibly rare
problem. The alerts actor doesn’t ask about new keywords, just monitors,
but APOGEE doesn’t output new keyword updates regularly. There are no
periodic status outputs from APOGEE-- the user has to ask for it. John
Wilson would like time to keep digging as to why it happened.Nathan saw
good calibrations in the morning, but they were bad in the evening, so it
must have happened during the day. APOGEE actor log should be ok, so Matt
will continue to dig. The logs shouldn’t have been restarted last night."