Date: Tue, 28 Nov 2017 19:29:43 +0000
From: Jon Brinkmann
Subject: Re: [sloan-25mlog 3778] Re: 2.5m obslog 20171127 (SJD 58085)
It's clearly an internal MCP problem.  From /data/logs/from.MCP/2017-11-27T18\:00\:04.log
	2017-11-28 09:56:22.165 ALT STATUS
	2017-11-28 09:56:22.208 66.329239 -0.000685 35782.149335 0 0.000000 OK
	2017-11-28 09:56:22.245
	2017-11-28 09:56:22.245 AZ STATUS
	2017-11-28 09:56:22.291 75.462399 -0.004266 35782.233962 0 0.000000 OK
	2017-11-28 09:56:22.327
	2017-11-28 09:56:22.327 ROT STATUS
	2017-11-28 09:56:22.385 243.274173 -0.002189 35782.316167 0 0.000000 OK
	2017-11-28 09:56:23.585
	2017-11-28 09:56:23.625 ALT MOVE 66.3597360 0.0033999 35792.01282
	2017-11-28 10:14:59.176 AZ INIT
	2017-11-28 10:15:00.415  OK
	2017-11-28 10:15:08.616 AZ INIT
	2017-11-28 10:15:10.271  OK
	2017-11-28 10:15:10.271
	2017-11-28 10:15:10.271 ROT INIT
	2017-11-28 10:15:11.636  OK
	2017-11-28 10:15:11.679 AZ MOVE 69.9719610 -0.0055789 36912.01282
	2017-11-28 10:15:11.818  OK
	2017-11-28 10:15:11.857 ROT MOVE 240.1422116 -0.0035275 36912.01282
while there's no interruption of commands coming over the network from
the TCC to the MCP.  From /data/logs/to.MCP/2017-11-27T18\:00\:06.log
	2017-11-28 09:59:53.590 \x000\x00aAZ INIT
	2017-11-28 09:59:53.590 \x000\x00aROT INIT
	2017-11-28 09:59:53.666 \x000\x00aALT MOVE 67.0547976 0.0033857 35997.01282
	2017-11-28 09:59:58.627 \x000\x00aAZ MOVE 74.5330741 -0.0044674 35997.01282
	2017-11-28 09:59:58.627 \x000\x00aALT INIT
	2017-11-28 10:00:08.547 \x000\x00aAZ INIT
	2017-11-28 10:00:08.629 \x000\x00aROT INIT
	2017-11-28 10:00:08.630 \x000\x00aALT MOVE 67.1055442 0.0033844 36012.01282
As Fritz noted, the MCP was responding to the our network monitoring queries
the entire time, but was NOT responding to the TCC commands.
On Tue, Nov 28, 2017 at 07:33:44AM -0700, Fritz Stauffer wrote:
>
>     We are unsure whether hub was affected by this hiccup.  We think there was
>     a
>     hiccup in the network around 10:00Z, but we are unable to track down that
>     down.
>
>
> I saw the MCP reboot about 10:14Z and return at 10:15Z, meaning the MCP was
> responding on the network.  I checked the network switch traffic and there was
> nothing unusual.
>
> The error messages you saw can also happen if the MCP is heavily loaded, out of
> resources, an edge case triggered a problem in the code and there are more
> reasons that I have seen over the years.
>
> The network was not the problem.
> ____________________________________________________________
> Logs of Nightly APO Activities
> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html