Date: Sat, 4 Mar 2000 08:55:40 -0600 (CST) From: Don Holmgren Subject: Re: obs log 20000303 (mjd 51607) - Archiver notes On Sat, 4 Mar 2000, Jeffrey Hayes wrote: > [mail to: obs-report-25m@apo.nmsu.edu > include MJD in subject] > > Friday March 03 2000 > > Apache Point Observatory SDSS 2.5m > > Observing Log > > (MJD 51607) > > Death of Archiver: > ------------------ > > This happened at 01:50. This happened without an error being shown by Watcher. > We ended the drift at this point. > > The sequence of events was as follows at this point was as follows: > > im> iccExec [photoList] statusDump archState > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- -------------- > ----------- > archState pub r mur End of tape condition reached Current sta > te > 1 Status entries found > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- -------------- > ----------- > archState pub r mur End of tape condition reached Current sta > te > 1 Status entries found > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- -------------- > ----------- > archState pub r mur End of tape condition reached Current sta > te > 1 Status entries found > Error: {id1 {ICC id1 (crate 1, board 0) is not available > Bad ICC}} {id2 {ICC id2 (crate 1, board 1) is not availab > le > Bad ICC}} {id3 {ICC id3 (crate 1, board 2) is not availab > le > Bad ICC}} {id4 {}} {id5 {}} {id6 {}} "End of tape condition" means that either the field pause limit was reached, or the archiver sensed that a given tape drive was within 5 fields of the end of tape. The archiver is paused, the tapes rewound, and the tapes unloaded. "Bad ICC" messages that pop up on all 3 boards of a crate almost certainly mean that the PT-VME link has gone down. Once PT-VME is down, no communications with that crate are possible - so, archiver commands are not processed, gangs are not fetched, etc... On Monday Ron and I will put together a procedure to try to recover the PT-VME link in the middle of a run. > > im> iccExec [photoList] statusDump archDrvSts > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- ------------------------- > archDrvSts pub r mur (array) Current status of drives > No tape in drive > No tape in drive > Not initialized > 1 Status entries found > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- ------------------------- > archDrvSts pub r mur (array) Current status of drives > No tape in drive > No tape in drive > Not initialized > 1 Status entries found > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- ------------------------- > archDrvSts pub r mur (array) Current status of drives > No tape in drive > No tape in drive > Not initialized > 1 Status entries found > Error: {id1 {ICC id1 (crate 1, board 0) is not available > Bad ICC}} {id2 {ICC id2 (crate 1, board 1) is not available > Bad ICC}} {id3 {ICC id3 (crate 1, board 2) is not available > Bad ICC}} {id4 {}} {id5 {}} {id6 {}} The auto-rewind/unloads on id4-6 result in these tape drive states. id1-3 are incommunicado. > > > Then a archRewind was done, and then an archState queried. The results were: > > im> iccExec [photoList] statusDump archState > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- ------------------------- > archState pub r mur End of tape condition reached Current state > 1 Status entries found > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- ------------------------- > archState pub r mur End of tape condition reached Current state > 1 Status entries found > NAME ACCESS TYPE VALUE LOW HIGH DESCRIPTION > ------------------ ------------- ----- -------- -------- -------- ------------------------- > archState pub r mur End of tape condition reached Current state > 1 Status entries found > Error: {id1 {ICC id1 (crate 1, board 0) is not available > Bad ICC}} {id2 {ICC id2 (crate 1, board 1) is not available > Bad ICC}} {id3 {ICC id3 (crate 1, board 2) is not available > Bad ICC}} {id4 {}} {id5 {}} {id6 {}} > > We changed out the tapes (as it was about the right time to do so), and > and before mounting the tapes, > Kinney booted PHOTO crate 1; then she tried to follow the Holmgren document > for "starting a run-new tapes". The archMount command did *not* cause the nodes > in PHOTO crate 2 to switch to ready-to-archive. After issuing an archResume, > all the drives in PHOTO crate 2 unloaded their tapes. According to the murmur log, the tapes in id1-6 did mount. However, immediately on id4-6 a field pause occurred, resulting in 4-6 tapes being rewound and unloaded, and "End of tape condition" declared. An "archFldPause 99999" on all of the boards was needed before the archResume. > There were complaints about an endTape when an archStatus was > queried for the drives in PHOTO crate 2. We then followed the "handling > a non-responsive archiver" The values for archEnable were 1 for all > drives. An archInit brought all the drives in PHOTO crate 2 back to a > normal state. We went back the "starting a run-new tapes" proceedure. > Once the archMount command was issued, we got the same complaints about an end > of tape, with the drives in PHOTO crate 2. We did an archPause, manually > unloaded the tapes, rebooted PHOTO crate 2, and once again followed the > "starting a run-new tapes" proceedure. This time all went as expected, and > we ran endNight -frameCheck to requeue the unarchived frames. The tapes were probably unloaded at that time. Before endTape, do an archReadSense and check for "82"'s at the end of the number, which indicates no tape in drive. Don H.