Subject: obs log 51599 24Feb00 Date: Fri, 25 Feb 2000 07:59:01 -0700 (MST) From: sjnk@apo.nmsu.edu (Scot Kleinman) Thursday Feb 24 2000 Apache Point Observatory SDSS 2.5m Observing Log (MJD 51599) Observing Team: Jeffrey Hayes (APO) Ellyne Kinney (APO) Scot Kleinman (APO) Atsuko Nitta Kleinman (APO) Dan Long (APO) Ron Rechenmacher(FNAL) Goal: shakedown and exercise imager and DA system. Weather: cloudy, humid, dusty, and a little snow. Bottom Line: no on-sky time, but lots of biases and archiver shaking. There was much battling with host today into the early evening. Finally mth was set up as host and we got things up and running. New versions of the watcher and iop were built against the latest version of the DA. astroda: v14_16 iop: v3_8_6 sop: v3_8_6 watcher:v2_6_3 mcp: NOCVS|Feb 24 2000|19:34:40 We began taking bias data at about 20:00. Shortly after we started running ID2-1 stopped archiving. It dismounted itself and the cleaning light came on. The auto-recovery feature kicked in, and worked beautifully!! When we issued the archResume ID2 happily started archiving to ID2-2 alone. Host was accidentally rebooted while in the middle of a bias drift. The VME side happily continued to receive and archive imager data. When host came back up and the appropriate software restarted, the drift was ended successfully! Jon Brinkmann disconnected the vacuum pump from the spectrographs at about 19:30 so the telescope could be moved tonight. He has one left to pump on tomorrow before all are cooled. We continued taking bias data to exercise the new DA version, while waiting on the sky to clear. Dan noticed tonight that he could not adjust the counterweights to a position which should be allowed. He get the response "invalid counterweight position" when he input 710 to a ^ command. The counterweights were at the upper limits when he issued the command, and could have something to do with the error. At around 00:30, the tapes stopped archiving, presumably from hitting an archFldPause set by someone (iop presumably, but I am not sure). We swapped tapes and issued an archResume. This didn't work as the archFldPause was not reset, so it paused itself immediately upon resuming. Much flailing ensued (some archMounts, archLoads, archUnloads, etc.) and we ended up with the following successful sequence for replacing tapes during a run: 1) reset archFldPause to something large (we used 99999) ex. iccExec [photoList] archFldPause 99999 2) reset archUseDrv from 0 to 1 ex. iccExec [photoList] archUseDrv 1 iccExec [photoList] archUseDrv 2 3) reset archEnable from 0 to 1 ex. iccExec [photoList] archEnable 4) resume archiving ex. iccExec [photoList] archResume At this point, archiving resumed fine, including on both drives of ID2! (We also put the cleaning tape in all drives at before inserting the new tapes in IDS1-6.) At some point during all this, the sp server crashed. We ended up killing the im server and restarting everything. The Watcher's "server" button then turned red, but remained connected and repeated attempts to reconnect failed to change the color of the button, although it did appear to remain connected. We noticed the QDepth was not decreasing very quickly (we were continuing with our biases) once we got everything resumed again. For example: Time id6 QDepth 03:15 192 03:35 191 03:50 192 During this time, Ron noticed that ID1's blinking was different than the others. A check on archErrCount showed that it was 14,000 something on ID1-1. ID6 (as well as the rest) were more reasonable numbers, although ID5-2 was 2500 something while ID2-2, for example, was 106. We stopped the bias drift run 1194 at 03:50 because we got Watcher error messages that said: TEXTONLY *** WARNING &&& dscArch: Shutting down tape ID 1 ... 03:45:32 MST ) dsc E ivFrameCtx Invalid Frame context (dsc_E_ivFrameCtx Invalid Frame context : Feb 25 03:45:32 MST) Ron knew the second error was fatal and was very bad, so he suggested we reboot the crate (IDs1-3). We issued an archPause, unloaded the tapes from IDs1-3 and hit the little red button. We put a new tape in ID1-1 only to see if the system could handle it gracefully (and since ID1-1 is the only drive we suspected of errors anyhow). To resume archiving again, we issued the following commands iccExec {id1 id2 id3} archFldPause 99999 iccExec {id1 id2 id3} archLoad 1 iccExec {id1 id2 id3} archLoad 2 iccExec {id1 id2 id3} archMount 1 2000 iccExec {id1 id2 id3} archMount 2 2000 endNight -checkFrame answer y to the re-queue the frames question and watch tons of stuff flash by the screen. (Dan - here's an example of when to answer "y" to that question.) While endNight -checkFrame was proceeding, we started a new bias drift (run 1195) just for grins to see if we could do that while the pooled frames were being re-queued. We could. ID1-1 still having lots of errors. We could tell this both by looking at the archErrCount and archRewriteCnt status entries and by noticing the lights on id1-1 flickered about twice as often as those in id1-2 did, looking like it was rewriting nearly every frame while id1-2 was fine the first time, and just waited for id1-1 to finish its business. We decided to take id1-1 offline and suggest it is either swapped tomorrow, or checked again tomorrow with another tape. (Email to jb and cloomis was sent indicating this.) archPause id1 archNoUseDrv 1 archResume This worked fine, too! Later, we did an archNoUseDrv on id5-2 during a drift, followed by an archUsrDrv. This didn't do anything bad, which was good, but also, presumably the few frames that were being written to id5-1 while id5-2 was turned off to the other didn't get written to id5-2 when it was brought back up. At least nothing crashed, but had this been real data, id5-2 would be missing a few frames. The Queue seemed to be emptying now faster than it was before: Time ID6 QDepth ID1 Depth 6:06 132 231 6:26 130 143 6:36 129 135 6:42 129 134 6:52 101 101 7:01 80 74 We ended the drift around 6:40, so the archiver had some hope of catching up for an attempt at endNight. This is the result of endNight: im> endNight 51599 endNight gangCheck: untransferred gangs? checking runs 1193 1194 1195 none all gangs transferred endNight frameCheck: unarchived frames? checking runs 1193 1194 1195 none all frames archived endNight parseMurmur: finished parsing murmur log endNight framelogFix: Fixing the frame logs is only necessary for parked scans. Do you need to run framelogFix? [n]:n endNight shiptape: Determining which tapes are to be sent to Fermi Error: couldn't execute "readlabel": No such file or directory im> All tapes taken tonight will be dumped in the recycling bin. Dan and Scot for all.