Subject: Re: [sloan-25mlog 2250] Re: 2.5m obslog 20150417 (SJD 57130)
From: Dan Long
Date: Mon, 20 Apr 2015 13:04:55 -0600
Yes, this has definitely been around for a while.  Here is an email from =
Craig regarding a ticket (mcp/7241) filed on the problem in 2007.
-Dan
On Mar 14, 2007, at 11:17 AM, cloomis@astro.princeton.edu wrote:
Synopsis: Slews/moves crossing the once-a-day time rollover are =
mishandled
Fix: Don't issue a slew between, say, 23:58Z and a bit after 00:00Z.=20
=3D=3D=3D=3D=3D=3D=3D
pvt.c, line 373, in calc_frames():
    dt =3D fframe->end_time - lframe->end_time;
 looks awful promising. That should almost certainly be
      dt =3D sdss_delta_time(fframe->end_time, lframe->end_time);
- craig
Release Notes:=20
State-Changed-From-To: open->analyzed
State-Changed-By: cloomis
State-Changed-When: Wed, 14 Mar 2007 14:17:40 -0400
State-Changed-Why:
Spotted a culprit.
Description-Changed-From-To:=20
 When the individual parts of a TCC->MCP slew command cross 00:00Z, =
which is when the MCP's "SDSS" clock is reset from 86400 s to 0s, the =
slew can misbehave, sometimes extremely badly. One example is slightly =
annotated below. Note that the time to watch is not the log timestamp =
but the last ("T") argument to the MOVE commands, and/or the arg to =
MS.OFF, and/or the last field in the responses from STATUS/DRIFT.
 There are other instances in the logs. Recently, 2007-02-28 23:58Z and =
2007-03-13 23:58Z.
 All I have checked is that the move_cmd arguments appear to be handled =
properly (i.e. passed through sdss_delta_time), and that sdss_delta_time =
does work correctly under the circumstances (at least as compiled on a =
different box). So calc_frames is the next place to look. The MS.OFF =
logic might be worth pursuing. I suspect it is a general MOVE-across-0 =
issue, only noticeably here because there is no immediately following =
MOVE, which you'd get during tracking.
 [ See the effective and not terribly onerous workaround..... ]
2007-01-28 23:58:27.318Z ROT INIT
2007-01-28 23:58:28.843Z  OK
2007-01-28 23:58:30.334Z ROT SET.TIME  1 28 2007 23 59 03.18
2007-01-28 23:58:31.203Z  OK
2007-01-28 23:58:31.718Z ROT STATUS
2007-01-28 23:58:31.765Z -0.002943 0.000000 86344.708933 0 0.000000 OK
2007-01-28 23:58:31.878Z ROT STATUS
2007-01-28 23:58:31.928Z -0.001504 0.000000 86344.869650 0 0.000000 OK
2007-01-28 23:58:32.278Z ROT STATUS
2007-01-28 23:58:32.328Z -0.000065 0.000000 86345.269384 0 0.000000 OK
2007-01-28 23:58:57.997Z ROT DRIFT
2007-01-28 23:58:58.095Z 0.000000 0.000000 86371.061106 OK
2007-01-28 23:58:58.316Z ROT MS.ON
2007-01-28 23:58:58.321Z  OK
							##### The =
transmission of the actual slew segments starts=20
							##### here, and =
ends with the MS.OFF
2007-01-28 23:58:58.780Z ROT MOVE 0.0000000 0.0000000 86373.70000
2007-01-28 23:58:58.787Z  OK
2007-01-28 23:58:59.220Z ROT MOVE 0.0651042 0.1562500 86374.95000
2007-01-28 23:58:59.225Z  OK
2007-01-28 23:58:59.701Z ROT MOVE 6.8151042 1.8437500 86381.70000
2007-01-28 23:58:59.708Z  OK
2007-01-28 23:59:00.060Z ROT MOVE 9.2500000 2.0000000 86382.95000
2007-01-28 23:59:00.065Z  OK
2007-01-28 23:59:00.466Z ROT MOVE 10.7500000 2.0000000 86383.70000
2007-01-28 23:59:00.467Z  OK
2007-01-28 23:59:00.865Z ROT MOVE 13.1848958 1.8437500 86384.95000
2007-01-28 23:59:00.868Z  OK
2007-01-28 23:59:01.303Z ROT MOVE 19.9348958 0.1562500 86391.70000
2007-01-28 23:59:01.308Z  OK
2007-01-28 23:59:01.701Z ROT MOVE 20.0000000 0.0000000 86392.95000
2007-01-28 23:59:01.708Z  OK
								#####  =
Clock rollover, after end of actual move...
2007-01-28 23:59:02.058Z ROT MOVE 20.0000000 0.0000000 33.45000
2007-01-28 23:59:02.061Z  OK
2007-01-28 23:59:02.338Z ROT MS.OFF 32.45000
2007-01-28 23:59:02.342Z  OK
2007-01-28 23:59:02.717Z ROT STATUS
2007-01-28 23:59:02.764Z 0.269964 0.359375 86375.708665 0 0.000000 OK
2007-01-28 23:59:04.638Z ROT STATUS
2007-01-28 23:59:04.686Z 1.416819 0.839062 86377.629324 0 0.000000 OK
2007-01-28 23:59:06.598Z ROT STATUS
2007-01-28 23:59:06.647Z 3.540635 1.329687 86379.589659 0 0.000000 OK
2007-01-28 23:59:08.638Z ROT STATUS
2007-01-28 23:59:08.685Z 6.765880 1.839062 86381.629271 0 0.000000 OK
2007-01-28 23:59:10.638Z ROT STATUS
2007-01-28 23:59:10.685Z 10.696955 2.000000 86383.629186 0 0.000000 OK
2007-01-28 23:59:12.638Z ROT STATUS
2007-01-28 23:59:12.685Z 14.453312 1.660937 86385.629178 0 0.000000 OK
2007-01-28 23:59:14.638Z ROT STATUS
2007-01-28 23:59:14.688Z 17.278003 1.160937 86387.629291 0 0.000000 OK
2007-01-28 23:59:16.637Z ROT STATUS
2007-01-28 23:59:16.688Z 19.103368 0.660937 86389.629269 0 0.000000 OK
2007-01-28 23:59:18.638Z ROT STATUS
2007-01-28 23:59:18.688Z 19.928686 0.160937 86391.629254 0 0.000000 OK
								##### =
Slew looks pretty good to here, after
								##### =
the _motion_ triplets have been processed.
								##### =
But the axis now zooms off:
2007-01-28 23:59:20.718Z ROT STATUS
2007-01-28 23:59:20.768Z 20.015424 0.066523 86393.709036 0 0.000000 OK
2007-01-28 23:59:22.678Z ROT STATUS
2007-01-28 23:59:22.727Z 20.709223 0.771796 86395.670324 0 0.000000 OK
2007-01-28 23:59:24.718Z ROT STATUS
2007-01-28 23:59:24.766Z 23.711137 2.319020 86397.709129 0 0.000000 OK
2007-01-28 23:59:26.718Z ROT STATUS
2007-01-28 23:59:26.765Z 30.533782 4.645268 86399.709288 0 0.000000 OK
2007-01-28 23:59:28.638Z ROT STATUS
2007-01-28 23:59:28.684Z 42.227475 7.639843 1.629174 0 0.000000 OK
								##### =
And maybe stops _fast_: 7.6 deg/s in no=20
								##### =
more than 2 seconds.
2007-01-28 23:59:30.638Z ROT STATUS
2007-01-28 23:59:30.688Z 46.901482 0.000000 3.629298 2048 0.000000 OK
->  When the individual parts of a TCC->MCP slew command cross 00:00Z, =
which is when the MCP's "SDSS" clock is reset from 86400 s to 0s, the =
slew can misbehave, sometimes extremely badly. One example is slightly =
annotated below. Note that the time to watch is not the log timestamp =
but the last ("T") argument to the MOVE commands, and/or the arg to =
MS.OFF, and/or the last field in the responses from STATUS/DRIFT.
 There are other instances in the logs. Recently, 2007-02-28 23:58Z and =
2007-03-13 23:58Z.
 All I have checked is that the move_cmd arguments appear to be handled =
properly (i.e. passed through sdss_delta_time), and that sdss_delta_time =
does work correctly under the circumstances (at least as compiled on a =
different box). So calc_frames is the next place to look. The MS.OFF =
logic might be worth pursuing. I suspect it is a general MOVE-across-0 =
issue, only noticeably here because there is no immediately following =
MOVE, which you'd get during tracking.
On Apr 18, 2015, at 8:06 PM, Kaike Pan  wrote:
> Definitely, this is a not a new problem. As far as know, it has =
happened at least five times during past five years.
> ALL of them have a common thing: UT rollover. However, it doesn=92t =
always happen when a slew is requested
> at that time. Not sure what combination trigger it.=20
>=20
> -kaike
>=20
>> On Apr 18, 2015, at 6:26 PM, Jon Brinkmann  =
wrote:
>>=20
>> There are thousands of MOVE commands sent to the MCP near the SJD =
rollover
>> over the years (see attached log of SDSS3/SDSS4 MOVEs).  These recent
>> incidents are the first I've heard of any problems.  =46rom the =
snippet of
>> code Fritz sent, it looks like the MCP properly handles the rollover.
>>=20
>> Jon
>>=20
>> On Sat, Apr 18, 2015 at 05:17:57PM -0600, Fritz Stauffer wrote:
>>> Here=92s a snippet of mcp code that I have on my laptop.  This code =
is old and may not be what is currently running.
>>>=20
>>> This code piece is from the code that processes the move command.  =
The move command is overloaded, and it can have multiple parameters.  =
The move command that is suspect has 3 parameters.
>>>=20
>>> The code rejects a move command if the move=92s time is less than =
the current time using a routine to decide when a rollover happens.  =
Here=92s the routine.  It looks like there is a 400 second window that =
should be avoided.
>>>=20
>>> -fs
>>>=20
>>> =
/*=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> **
>>> ** Determine the interval between two timestamps, given in seconds =
since
>>> ** midnight UTC ~ 5pm MST.
>>> **
>>> ** The routine assumes that within 400s of this roll-over time the =
smaller
>>> ** time really belongs to the following day, and acts accordingly
>>> **
>>> =
**=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> */
>>> double
>>> sdss_delta_time(double t2, double t1)
>>> {
>>> if(t2 >=3D 0 && t2  86000 && t1 >>    return((86400 - t1) + t2);
>>> } else if(t1 >=3D 0 && t1  86000 && t2 >>    return((t2 - 86400) - t1);
>>> } else {
>>>    return(t2 - t1);
>>> }
>>> }
>>>=20
>>>=20
>>>=20
>>> On Apr 18, 2015, at 4:49 PM, Fritz Stauffer  =
wrote:
>>>=20
>>>> I am pretty sure the MCP handles this just fine.  Deciding that the =
day changed is far easier than programming the hardware to run the axes. =
I am pretty sure FNAL got this right.
>>>>=20
>>>> I don=92t think it is in the MCP, and it is a red herring to =
consider reprogramming it.
>>>>=20
>>>> On Apr 18, 2015, at 4:10 PM, Russell Owen  wrote:
>>>>=20
>>>>>=20
>>>>> In any case, the axis controllers require =93seconds in the day=94 =
for historical reasons. If anyone ever wants to reprogram the MCP to =
take the full date as MJD seconds then it would be trivial to modify the =
TCC and would eliminate any danger of such things happening.
>>>>=20
>>>> ____________________________________________________________
>>>> Logs of Nightly APO Activities
>>>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>>>=20
>>> ____________________________________________________________
>>> Logs of Nightly APO Activities
>>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>> =
____________________________________________________________=
>> Logs of Nightly APO Activities
>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>=20
> ____________________________________________________________
> Logs of Nightly APO Activities
> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html