Date: Tue, 22 Jan 2002 08:43:30 -0800
From: Russell E Owen
Subject: RE: 2.5m obslog 20020120 (MJD 52295)
> > >We saw the TCC watcher error shown below multiple times tonight:
>> >timed out while writing to hub message mailbox:: (Jan 21 01:43:26 MST)
>>
>> This was mentioned in a previous log, too. Sorry I did not respond then.
>> I believe this means the TCC cannot communicate with one of sessions. Ie.
>>   ^^^^^^^
>> there is something like a telrun session going on with a permanent ctrl-S.
>> The best way to address this (although not during the night) is a telstop
>> to kill all listening processes. I suggest doing this before tonight's
>> observing.
>
>Although Scot's incantation works, if I remember correctly (Russell?), a
>simple 'show users' command in TCC will remove any orphan message hub users.
>This leaves everyone else connected ok.  Or is this a different problem?
Hi, I'm back. Pete is right. Here is more information about this problem.
"timed out while writing to message hub mailbox" means that the message hub got so busy that some process timed out while trying to output a message. (All messages are sent to the hub, which redistributes them to the users.) It suggests that the message hub is getting overloaded.
This error was last reported about a year ago, at which time I implemented automatic deletion of moribund users and the "show users" command which gives diagnostic information and also deletes moribund users.
If you see this error again, please do a SHOW USERS and report the output (or let me know, preferably with a timestamp, and I'll hunt it out of the log).
It is a fairly disturbing error. Generally it should take no time at all to output messages. I have implemented long queues for messages going into the hub and the hub should easily be able to keep up. I assume there is some sort of limit on how many users we can have and how many messages/second, but I'd be surprised if we are pushing those limits unless we have a ton of extra users. One possibility is to increase the priority of the message hub above that of the other TCC processes; I have been reluctant to do this, for fear it'll break other things. First I'd like to see the output of SHOW USERS to get some idea what sort of load the message hub is seeing.
By the way, one of the messages that may have been lost is "SlewEnd"; if true, that might explain why SOP got confused and thought the telescope was still slewing. Independent of fixing the problem in the TCC, it would probably be wise to improve SOP's code so it detects other indications of a slew being finished, as well (such as the "TTT" axis state), just to make it maximally robust.
Regards,
-- Russell
P.S. I just tried to ssh into the machine with the logs but the ssh server doesn't seem to be running. I'll work on solving that, then investigate the other problem.