Date: Tue, 29 Apr 2003 16:57:44 -0500
From: Don Holmgren
Subject: Re: 2.5m obslog 20030428 (MJD 52758) - archiver info
On Tue, 29 Apr 2003, Howard Brewington wrote:
>
>                        APACHE POINT OBSERVATORY SDSS 2.5M
>
>                                  OBSERVING LOG
>                              Monday April 28, 2003
>
>                                   (MJD 52758)
>
> ---=== OBSERVING TEAM ===---
>
> Swing:          Pete Newman
> Night:          Steph Snedden, Howard Brewington
> Support:        Craig Loomis, Jon Brinkmann
>
> ---=== OBSERVING PLAN ===---
>
>         Science!
>
> ---=== OBSERVING SUMMARY ===---
>
> Imaged all night; completed strips 31,N and 27,S.  Yeee-hah!!
>
> ---=== OBSERVING LOG ===---
>
> Swing shift notes
> -----------------
>
> We had "fun" getting the DA to write to tape this afternoon.  See problem
> section 1 for details.
> ++++++++++++++++++++++++++++++++
>
> ---=== PROBLEMS IN DETAIL ===---
>
> Problem 1 - Archiver errors
> ---------------------------
>
> During preflight checks, we encountered multiple problems with the DA.  These
> manifested themselves as dual tape drive "failures" although we (Pete, Craig,
> Jon B) are fairly sure the drives themselves are ok.  The first indication of
> the problem occured during the initNight -tapeLoad process, then subsequently
> in writing to the tapes during doghouse biases, several of which were
> discarded.
>
> The details are in the murmur log (in /skycat/oplogs/murmurlog/52757/).  The
> key seems to be an error code seen on two different crates:
>
> 2003-04-28 21:41:47Z sdssid4    dArch      dsc_E_trcErr dscArch_CheckPrevOp:
> rstatus = 0x8001421c, drive 1
>
> 2003-04-28 21:41:49Z sdssid4    dArch      dsc_E_trcErr dscArch_CheckPrevOp:
> rstatus = 0x8001421c, drive 2
>
> 2003-04-28 21:41:52Z sdssid2    dArch      dsc_E_trcErr dscArch_CheckPrevOp:
> rstatus = 0x8001421c, drive 1
>
> 2003-04-28 21:41:54Z sdssid2    dArch      dsc_E_trcErr dscArch_CheckPrevOp:
> rstatus = 0x8001421c, drive 2
> This error code does NOT correspond to anything in the DLT drive manuals.  If
> anyone reading this (Don H!) can decode 0x8001421c into English or tell us
> where to look, please do!
> We did determine that this error code was also associated with other similar
> failures in the last few months.  In all cases, it seems that that the
> appropriate magic to solve the problem was rebooting the  crates.  Tonight,
> the first reboot stopped id4's problem,  and it took a second reboot to
> apparently solve id2's problem.  We were careful to reboot one crate at a time
> and let it come up before booting the next one.
> The plans is for Craig and/or JB to contact Don Holmgren and/or Ciprico
> (makers of the RimFire SCSI controllers in the DA) to try to understand the
> error codes seen.
>
0x8001421c is the murmur code from the Rimfire software routines which
corresponds to the "Status Block List Full" error.  This error happens
when a SCSI command to one of the tape drives never completes (the drive
never reports back the status for the command to the controller, or
perhaps the RimFire controller drops the status).  Unfortunately,
because of a design flaw in the RimFire firmware, this always leads to a
full status block list error, and the only way to recover is to
re-initialize the RimFire controller.  The archiver autorecovery
routines do this for you, leaving the drive which was involved offline.
If both drives have errors, the autorecovery routines disable archiving
on that ICC.
It's not surprising that you'll find this particular error code in prior
error logs.  It is a very low level error from the RimFire controller
which catches errors not reported directly by the tape drives.  Nearly
all of these should be handled by the archiver via autorecovery.
I took a close look at the murmur logs and I agree that the drives are
OK.  On both 'id2 and 'id4 this error occurs at exactly the same frame
number (21) on all four drives during the pre-flight bias run.  That's
too much of a coincidence - somehow the sequence of tape movement
commands issued prior to starting this run is leaving an uncompleted
command in the RimFire controller status block lists.  As best I can
tell from the logs, the same sequence of commands is issued to all six
ICC boards (from "initNight -tapeLoad", I gather).  I don't know why
there are resulting RimFire problems on only 'id2 and 'id4.  I did note
that "initNight -tapeLoad" does considerable tape movement; for example,
it rewinds, unloads, and loads the same tapes three times.
How frequently are we seeing this error scenario (both drives go down
very soon after doing initNight)?  If rarely, I suggest not worrying too
much.  But, if this is happening a lot we'll have to investigate
further, and perhaps add some diagnostic code to initNight.
Don H.