From: kaike pan
Subject: Re: Delayed Slew On 54498
Date: Thu, 7 Feb 2008 10:13:54 -0700
On Feb 7, 2008, at 9:42 AM, Dan Long wrote:
> Hi Kaike,
>
>   I'm not familiar enough to give a definitive answer here, maybe
> Jon, Fritz, or Russell can.  I can guess that it is probably a
> communications issue between the sockets created by the sop session
> and the corresponding mailbox on the TCC.
>
>    With some regularity I see messages in telrun sessions
> complaining that user so-and-so's mailbox is full and output is
> suspended until that changes.  I know that this can be caused by
> hitting ctrl-s in a telrun session and having all the broadcasts
> pile up until the mailbox is full.  There is probably something
> analogous happening in the connection between the sop session and
> its tcc connection.  Why that is I couldn't say, and short of
> closing and reestablishing the connection I don't know a more
> subtle way to correct it.
>
>   One possibly related issue which may give a clue to the problem
> is that a few times over the last month there have been timeouts
> during a spectro exposure where the DA apparently didn't read the
> exposure, and it was lost.  If this is caused by a similar delay in
> communication between sop and the camera(s), it may an indication
> of a host2 communications problem.
   Thanks Dan,  Here is an another piece of information for
communication on sd1.  Last week, when we had sd1
ICC communication problem, we rebooted spectro crates. It took about
10 minutes for crates to come back. It
usually take 3 minutes or so.  This maybe somewhat related too.
  -kaike
>
> 				Cheers,
>
> 				  Dan
>
>
>>     I want to clarify one thing here. The rebooting did not cause
>> any science time loss.
>> The MCP rebooting process took less than 1 minute and was done in
>> the afternoon
>> check out.
>>
>>  As a rule, if there will be a science time loss and we are not
>> very sure one machine is
>> the cause of the problem, we will not reboot it.  However, if it
>> will not cause any science
>> time loss and one observer thinks that one machine may be the
>> cause, we may try to
>> reboot the machine in the afternoon because it's not harmful to
>> reboot a computer.
>>
>> On average, I will say we reboot one machine every two weeks
>> (except for DA crates which
>> hang on more often, and we usually know when they need to be
>> rebooted). Given number
>> of computers we have there, this is one reboot every 5 months or
>> so per computer. It is fair
>> to state that observers applied their adjustment well so far.
>> There is a good number of times
>> that rebooting a computer made problems disappear (fixed or
>> coincidences?) although we
>> did not fully understood the problem because of the complexion of
>> the whole system.
>>
>>   Thanks to Dan for tracking down the slew delay was due to the
>> delay between sop
>> session and the TCC.
>> Robert, Dan:  any further clue on this. What cause the delay
>> between sop and TCC?
>> what can be done to fix it if this happen again? Thanks!
>>
>>   -kaike
>>
>>
>>