From: stephanie snedden
Subject: Re: 2.5m obslog 20070928 (MJD 54372)
Date: Sat, 29 Sep 2007 12:59:14 -0600
Many thanks to Jon and the observers for dealing with this problem in
time for Supernova tonight!
On Sep 29, 2007, at 5:08 AM, Dmitry Bizyaev wrote:
>
>   id5 was not happy at all today:
>
>   When we started doghouse bias, watcher complined about id5 with the
> fellowing messages:
>
> 1) Status: Inactive, Run: 0, (and other run # 6991-6993), Frame: -1
> (and many
> others, 12, 20)
> dsc W chksumMismatch Checksum mismatch (expected -2075405749 vs.
> computed
> 323857328)  (dsc_W_chksumMismatch Checksum mismatch (expected
> -2075405749 vs. computed 323857328) : 2007-09-28
> 23:44:55Z )
> 2) id5's archiver and tape drives
> frameCatcher: Invalid frame idRR-006993-g5-0016.fit  ( : 2007-09-28
> 23:54:15Z)
>
> It looked like that files were corrupted somewhere during data
> transfer from
> the pool to host1 (file1). So ew tried
> 1) rebooted all photo and astro crates, that did not do anything good,
> 2) rebooted host2 without any luck again.
> 3) rebooted file1 and all crates.
>
> We got micro error messgae:
> Camera Serial Line
> server is not responding (Last packet: 2007-09-28 23:51:32Z)
>
> But when ping imager, cc camCheck, and cc imstat, it showed imager is
> communicating.
>
> After we rebooted sdssfiles1 and all crates, we did a bias run (6994).
> Every node was taking data, and id 5 stoped after the first frame.
> However, iccSoundOff said id 5 was available, and watcher did not
> complained about this. Jon sound that VCI has fallen behind (by
> doing dsclat id5). We tried again bias run (6995), id 5 did the same.
>
> During another attempt bias run (6996), we got this watcher message
> after Jon restarted readout for id5 (readoutBegin id5).
> dsc E monPriorityHigh The monitor task has identified a serious
> problem:
> Archiver has fallen behind  (50 frames are not archived), and
> message from
> MCP.
>
> telnet reading on port 41788: S errno ECONNRESET (2007-09-29 01:36:11Z
> sdssmcp) (telnet reading on port 41788: S_errno_ECONNRESET
> (2007-09-29 01:36:11Z sdssmcp): 2007-09-29 01:36:23Z ).
>
> After Jon VCI rebooted, we rebooted crates again. During nother try of
> bias run, id 5 stoped at frame 17 with the same error message as we
> had in very beginning.
>
> Kurt was called and left messages on his machines in office and
> home. Then
> Eric and Fritz were called. Eric
>
> After Jon Clean contact in CPU and VCI card, we tried another bias
> run. We
> still had problem of chksum and framecatcher on id5. But its VCI
> behaves
> correctly.
>
> Because the bias levels for CCD25 and CCD 45 were not right, so we
> did an
> iack on imger, then tok bias. This did not work out. Then we replaced
> VCI card for id 5. After the replacement, we took two bias runs
> (7000 and
> 7001), they are looked good. Thanks to Jon for helping.