Date: Wed, 26 Jan 2000 03:43:34 -0700 From: Dan Long Subject: 2.5m obslog 000125 (MJD 51589) TUESDAY JANUARY 25, 2000 APACHE POINT OBSERVATORY SDSS 2.5m OBSERVING LOG Observering team: Dan Long (APO) Ellyne Kinney (APO) Since the secondary did not go into the telescope today and the telescope remains immobile we did more iop/imager/da sop/spectro testing. We began the night by trying to run endNight for last night's data incorporating today's software fixes. id2 was not responding. Rebooting only id2 did not solve the problem. Rebooting the entire photo 1 crate did fix the problem. For details, see Ellyne's notes at the end of the log. Last night's tapes are on the shelf ready to be shipped, and Craig got the appropriate e-mail, with the correct tape names on it. Once endNight finished we tried Connie's prescription for swapping fibers between id4 and id5. This involved editing a copy of camera.tcl and physically swapping data fibers between id4 and id5. The bottom line is that the high bias values moved with the swap, indicating the source of the problem is at the imager end. We did get some interesting effects from ccds 24, and 35 due to some configuration error when we did the swap from different number of amplifiers reading out chips in dewars 4 and 5 ( 24 has 2, 25 has 1, 34 has 1, 35 has two ). If you're interested, take a look at /data/astro/long/bias_24.fits on host. Run 1073 is a bias run with the swap in. Run 1074 is another bias run without the swap. More work will have to be done to identify the problem at the imager regarding the high bias levels from dewar 5. In the mean time Mamoru Doi was going to start his calibrator work. However he ran into problems trying to get staring frames from the imager. The goStare 0 -init command would bomb out just after reporting leap seconds with beginData has failed: Operation not permitted sending SIGCONT to process 219 249 Error Connie has issued a detailed problem report (PR-720) on this. We then turned our attention to the spectrographs. They were cooled this evening. Unfortunately they were also warmed this evening. For some reason, the watcher is not reporting the ccd temperatures for ccds 0 through 4 ( is this a bug, or has it been masked? The entries aren't yellowed, they are blank ). By the time we realized this and did a tcheck by hand, ccds 3 and 1 had warmed to around -35 C. This was around 22:40. It was after 01:00 before they were cool again. We then proceeded to take biases, 20 in all frames 1778 - 1797. 3 darks were taken ( out of the 5 requested ), so item 11.6 of the shakedown list isn't quite completed. The frame numbers were 1798-1800. The mechanicals check, Item 11.5, was completed however. There were some problems encountered in setting up to fill the spectrographs. First, I was unable to get the rotator to move without issuing an AXIS INIT from the TCC. (PR-718) I turned on the rotator amplifier only. I released all stop buttons. I issued an amplifier reset from the Menu. I heard a clunk from the rotator the first time I reset the amplifier. The status became *-A-*. Subsequent attempts at a reset had no effect. The amplifier leds were all green. The directional rotator permits were true. Only the rotator PLC permit was false. I then issued an AXIS INIT from the TCC. This solved the reset problem in the rotator. It also led to the second thing which happened. When I issued the AXIS INIT from the TCC, the windscreen jolted in both Azimuth and Elevation, despite the fact that the drive amplifiers were powered off for those axes. It also briefly released the brakes. None of this should happen if the amplifiers are off.(PR-719) Ellyne also worked tonight on cutting a version of the MCP. It is called v2_3. Details on running/reverting will be forthcoming. I swapped dewars on the imager at a little after 03:00. There was quite a bit left in the old dewar, but I didn't want to risk it running out. However I left the old dewar for use on the spectrographs rather than putting it back as an empty and wasting 1/3 of a dewar( or worse having someone confuse it with full ). I also noticed when I switched dewars that nothing was connected to the vent. Is there a reason for this? Are we changing the way the 180l dewars are connected to the imager? There are fixtures and plumbing for the vent side connected to the dog house but the hoses are not connected to it. EndNight was run and completed successfully with the exception of the summary plots to be mailed to Princeton. It complained that the summary postscript file did not exist ( summary226559.ps ). Tonight's tapes are also on the shelf ready for shipment. WEATHER Skies tonight varied between mostly clear and partly cloudy. The temperature was around 40. Winds were from the southwest at 15 to 20 MPH, humidity was around 40%, and dust counts were around 300. Seeing tonight was between 1.1 and 1.6 arcseconds on DIMM. Dan Details of endNight attempts: We started by trying to archive the files from last night using 'endNight 51586 -doImager'. The first time we got dumped out of iop with the following error message: id2: issuing archive 1071 56.52 0 1 1 id2: issuing archive 1071 57.12 0 1 1 id2: issuing archive 1071 57.22 0 1 1 id2: issuing archive 1071 57.32 0 1 1 /p/astroda/v14_10/bin/dscRun[422]: 211465 Bus error(coredump) Greping the murmur log revealed the following failure: Jan 24 19:12:57 sdssid2 dArch dsc_E_archStsFail Failed to receive exabyte command completion status Jan 24 19:16:10 sdssid2 dArch dsc_E_archStsFail Failed to receive exabyte command completion status We restarted iop and tried again. Got the same error messages in murmur plus: Jan 24 19:16:48 sdssid2 dArch dsc_E_archStsFail Failed to receive exabyte command completion status Jan 24 19:16:48 sdssid2 dArch dsc_E_archStsFail Failed to receive exabyte command completion status Jan 24 19:23:14 sdssid2 dArch TEXTONLY *** FATAL *** dscArch: While double writing frames Jan 24 19:23:14 sdssid2 dArch TEXTONLY ---> Frame 1070 4.12 not archived Jan 24 19:23:17 sdssid2 dArch dsc_E_trcErr dscArch_CheckPrevOp: rstatus 0x8001421c, drive 1 Jan 24 19:23:17 sdssid2 dArch dsc_E_archShutDown1 Archiver shutting down first drive Jan 24 19:23:17 sdssid2 dArch dsc_E_archWrFail Tape write error while archiving frame 1070 6.22 Jan 24 19:23:19 sdssid2 dArch dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 2 Jan 24 19:23:19 sdssid2 dArch dsc_E_archShutDown2 Archiver shutting down second drive Jan 24 19:23:20 sdssid2 dArch dsc_E_archWrFail Tape write error while archiving frame 1070 6.22 Jan 24 19:23:22 sdssid2 dArch dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 1 Jan 24 19:23:24 sdssid2 dArch dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 2 And the following in iop: Waiting for command response from id2 (crate 1, ICC 1), archiver server (client 2) Waiting for command response from id2 (crate 1, ICC 1), archiver server (client 2) So we started to follow Don's troubleshooting guide. We asked for an 'archPause 1' but iop never came back from this command and the 'archDrvSts' did not change from Archiver writing to Paused. At this point we manually unloaded the tapes and rebooted crate2. We immediately started getting watcher errors at this point, including a critical error about poolFullFrame and poolFullGang. Does this mean that the disk for the archiver pool is full. Where do we go to find the descriptions of what these error messages mean? We checked the astroDa reference manual under section 6, but did not find a reference for either. We checked the pool* status Objects for id2 and confirmed that the pool was full. We also note that the poolSize was 99999999. Did we hit some limit. poolSize pub r int 99999999 0 0 Frame Pool size (blocks) [0 indicates n poolFree pub r int 0 0 0 Available Frame Pool space (blocks) We then tried and archReadSens for drive 1 and 2. We did not get a response from the archiver. Jim was called. He suggested another reboot of the system, which did not fix the problem. He then suggested that we try to talk to id2 directly via the dscRun command. We did not get a response from any of the archiver commands or the pool commands. Jim suggested we call Ron Rechenmacher. We paged him. While we were waiting for Ron to call back, Dan suggested that we try rebooting the whole crate. (We had only rebooted id2 on previous attempts.) That fixed the problem. When Jim called back to see how we were doing, he recommended that in general we should only rebooting whole crates, instead individual archivers. Once we commanded the archivers to remount and resume, we were able to finish archive last night's tape. They are on the shelf to be shipped and Craig got the appropriate e-mail, with the correct tape names on it.