Subject: Re: [sloan-25mlog 4840] 2.5m obslog 20191226 (SJD 58844)
From: Kaike Pan
Date: Sat, 28 Dec 2019 08:57:14 -0700
Thanks Dan for the input.=20
I think that it may take some time to fix the issue in a perfect way, so =
let=E2=80=99s approach the
issue in several steps.
1. Looks like that mcp knows whether a slew is good (executable) or not. =
 mcp should be=20
able to send out a =E2=80=9Cfailure=E2=80=9D message (like =
out-of-closed-loop) to sop without moving any axis=20
when a bad slew is detected, so, sop will fail immediately,
2. Not sure how currently  tcc (mcp) and sop communicate on =
successfulness of a slow.=20
Looks like the communication only depends on the slew timer.  Can mcp or =
tcc check where=20
axes are actually at and report the slew successfulness to sop basing on =
where axes are?=20
3. Look into sop code to see whether =E2=80=9CX=E2=80=9D and =E2=80=9Cstop=
=E2=80=9D options of sop can be reworked out so
that they can work properly. What we need is that the command sequence =
is stopped immediately
whenever =E2=80=9CX=E2=80=9D or =E2=80=9CSTOP=E2=80=9D is detected? Do =
we need to check and make sure whether collimator
pistons are moved the same steps and clean up properly  if they have =
been moved?=20
4. Digging more on tcc and mcp to find the cause of bad slews and fix =
properly.=20
Because SDSS-V will benefit on these works, we should make efforts on =
them.
Thanks,
-Kaike
> On Dec 28, 2019, at 1:51 AM, Daniel Oravetz  =
wrote:
>=20
>>                       APACHE POINT OBSERVATORY SDSS 2.5M
>>=20
>>                                 OBSERVING LOG
>>                          Thursday December 26, 2019
>>=20
>>                                  (SJD 58844)
>> Notes:
>> =3D=3D=3D=3D=3D=3D
>> (3) We experienced two more episodes of bad slew behavior tonight. =
Both
>> occurred
>>    while using SOP to go to field during cart changes. This happened
>> around
>>    3:40Z while switching to Cart 3 and around 7:00Z while switching =
to
>> Cart 2.
>>    In both cases, when the SOP go to field command was issued ROT =
began
>> slewing
>>    but ALT and AZ did not, even though the reported values for ALT =
and AZ
>> did
>>    change as expected. In both cases restarting SOP and re-trying =
solved
>> the
>>    issue.
>=20
> This ties back into ticket #2919 (and #2203 is probably associated, =
too)
> from the sop side of things, but from the mcp side we had also =
discussed
> -- during a telecon about six weeks ago -- having all axes "fail," or
> abort, in instances of "bad slew behavior" so that 'sop gotoField' =
would
> quickly go to a "Failed" state. Was a consensus ever reached as far as =
how
> to move forward with handling this behavior?
>=20
> The current mcp handling of "bad slew behavior" is to stay put, which =
is
> certainly preferred compared to any runaway slews. The problem is that
> sopActor doesn't recognize that the telescope isn't where it should be
> when the predicted slew time completes -- sopActor doesn't seem to =
care
> that the telescope never actually arrives at the target destination =
and it
> just proceeds to the hartmanns, cals, etc. Since the sop 'Stop'
> functionality doesn't reliably work it is generally faster just to =
restart
> sop altogether.
>=20
> When we used to experience an axis going out of closed loop during =
"bad
> slew behavior," that out-of-closed-loop axis state triggered a =
"Failed"
> gotoField sequence. This was actually an easier state from which to
> recover within sop because one could just re-click the gotoField =
button
> after clearing the offending axis state -- there was no need to mess =
with
> the 'Stop' function for the gotoField sequence because it failed
> automatically.
>=20
> What should be the accepted way of handling these situations going
> forward? Is it indeed the case that the underlying cause is related to =
the
> TCC, e.g.,  as it was described following the event reported on =
58801?:
>=20
> "[the 58801 misbehavior was due to] a reposition command that the =
system
> does when the axis is more than 0.01 degrees off when a PVT is
> requested... believe the problem is caused by the TCC not knowing =
about
> the reposition command (to get the PVT starting point right), so the =
time
> it takes to reposition would cause the PVT to start after the =
commanded
> T..."
>=20
> In the long term, is it feasible to expect the TCC to be fixed to =
address
> this? Or is it more likely to rely on an indefinite workaround: (1)
> restart SOP from stageManager and re-gotoField; (2) exploit the =
intended
> functionality of sop 'Stop' to abort a gotoField and then just =
gotoField
> again; (3) change MCP to revert back to the out-of-closed-loop =
behavior so
> that 'sop gotoField' quickly gets into a "Failed" state; (4) in =
principle
> we could press the control-room E-stop after issuing an 'axis stop', =
which
> would take the axes out of closed loop and should therefore get the
> gotoField state to 'Failed.'
>=20
>=20
>=20
>=20
> ____________________________________________________________
> Logs of Nightly APO Activities
> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html