Date: Sun, 06 Feb 2000 10:32:44 -0700 From: "s. a. snedden" Subject: 2.5-m Obs Log (MJD 51580) SATURDAY FEBRUARY 5, 2000 APACHE POINT OBSERVATORY SDSS 2.5m OBSERVING LOG (MJD 51580) Observing team: Jeffrey Hayes (APO)--first shift Scot Kleinman (APO)--second shift Atsuko Kleinman (APO)--second shift Stephanie Snedden (APO)--third shift Visitors: Connie Rockosi (UC) Andrew Connolly (UP) David Schlegel (PU) Adrian Pope (CMU) David Johnston (UC) Mark Subbarao (UC) Robert Lupton (PU) The plan: repeat of last night, do spectroscopy as on a "typical" run. During the afternoon, Hayes replaced the ID1-2 tape drive. We had an sdsshost crash earlier in the day with PTVME errors. Robert and Craig performed diagnostics and as of 15:00 sdsshost was okay. At 16:00 we wanted to try running "endNight" on last night's data, and at first all seemed to go well, but on checking ID1-2 again, we found that the drive thought the tape was write protected. We changed tapes to a new one, and after a few minutes, found that ID1-2 thought that tape was write protected as well (it clearly was not). We then took ID1-2 out to check the jumpers -- to see if they were put in correctly (we verified this by taking out ID1-1 for comparison), and found no problems with the jumper positions. We reassembled the rack and power-up again. No luck. ID1-2 still thinks any and all tapes put into it are write protected, and now ID2-1 had problems: there is a continuous flashing of the write protect, clean tape and tape in use lights and the operate handle light is continuously lit. We tried to unload the tape ID2-1 and found that it had never been loaded. At roughly 17:30 we wanted to refocus the spectroscopic guider. We waited for Connie to put the shutter back in (to address the problem mentioned last night of the shutter not acting properly). This is when the fun began. We repeatedly power cycled the guider and the guider Mac and still could get no repsonse from the guider. Connie and Jeffrey the opened the guider control box and noted that the error light on the Paralan was constantly on. We the swapped the fibers to the engineering cameras found the same problem: the error light on the engineering camera Paralan was constantly on. This was not the case at the guider Mac end and so we suspected a problem with a fiber. After a call to Craig, we got hold of his sniffer to check the fibers. This produced no problematic fibers. A final call to Craig gave us the idea of swapping the Paralan at the Mac end. Success! We had a talking guider by 22:00. French fixed the ff petals this am. See his daylog report. Hayes then went on to address the problem of the 2 non-functioning DLT drives. At 23:00 I finally got back to swapping ID1-2 and ID2-1, and put them in the rack. We reloaded the tapes into the new drives, and by 23;30, we had an archiver that, on the basis of the lights, seemed to be fine. We tried endNight again and got this meessage just after the "Are the tapes in the drives still 51579's? [y]:" prompt: Error: {id1 {Tape drive does not exist}}] We did an archInit... we got the same error. We rebooted the crate..The id2-1 drive came up with the cleaning light on. We ran the cleaning tape & it appeared to be okay. Then we tried initNight again. We got the same error as before. We noticed that the # of drives status was 0: archNTapes pub r int 0 0 6 Number of tape drives We also noted that the status was: archDrvSts pub r mur (array) Current status of drives Not initialized Not initialized Not initialized We re-booted the crate again. id2-1 asked to be cleaned again. After clearing the cleaning light, this time by unloading and loading the tape, initNight was tried again and failed at the same point with the same message. The Tape drive controller can't see any tape drives for id1... Is the controller dead? It does see the id2 drives. (PR da/784). id3 sent out a critical error and gave these messages: id3 reported on the VME node: dsc E noDev Device SCSI ID %u LUN %u does not exist dsc E loadFail Frame Pool Directory load from disk failed dsc E noDev Device SCSI ID LUN 0 does not exist on the archiver node: poolFullFrame (0 Frames until node dies) poolFullGang (0 Gangs until node dies) However there was still space in the pool. At this point we gave up running endNight. The tapes were left in the drives, and nothing was logged out. At the end of the night, Steph did try an endNight 51580 -doSpectro. ************************************** Spectroscopy: The evening started with cartridge 2, plate 256 on the telescope, however, no exposures were taken because of the guider shutter problems mentioned previously. Cartridge 4, plate 318 (M 67) was mounted at 22:30. A preCalib was run, followed by three 900 sec. exposures and a postCalib. The diamond pattern (nodding pattern) with no binning was then tested successfully. Cartridge 3, plate 179 (FASTT) was installed at 02:00. Three short exposures were taken. The diamond pattern was tested again and a tracking test was performed. All tests were successful. Clouds started coming in at this time. At 03:30 cartridge 5, plate 295 was installed. A preCalib, four 900 sec. exposures and a postCalib were taken. Guiding was then tested. All tests completed succesfully. This field was followed into morning twilight. 20 sec exposures were taken from 05:58 until 06:10. This last tracking test was at an airmass of about 1.4, and the stars stayed in the fibers for as long as they could be seen in morning twilight. It was decided to keep cartridge 5 on the telescope at closing. endNight 51580 -doSpectro was run, amd appeared to archive all the spectroscopy data. There was this error message, however: will check if tape was labelled with $SDSSTAPE_DIR/bin/labeltapes Read error: Not enough space 0+0 records in 0+0 records out Error reading the VOL1 label, ========tape error, or unlabled tape=========== Data did appear to be archived to volume 1, however. Volume 2 did not have this problem...the label appeared to be okay. I also got the "Cannot open /astrolog/51580/idTarTape-.par: No such file or directory" message. In /astrolog/51580/ there was a idTarTape-JL1816.par file, but none for JL1815. The rest of the copying completed normally. The written tapes are JL 1815 and JL 1816. They were placed on the shelf in the 2.5-m control room. Fresh tapes were put in the drives. ********************************** Problems: Archiver troubles: See the beginning paragraphs of the log. Much of the afternoon and early part of the night was spent working on id1. A bug was noted: When the flat field screen close command is issued, then followed closely by calibLamps ff on calibLamps ne off calibLamps hgcd off the lamps would go on, then shut back off. If you wait a while so that the petals finish opening, then turn on the lamps, they stay on. Alternatively, if you turn on the lamps first, then close the petals, the lamps stay on.(PR sop/785) Another (old) bug: Flat field lamp 1 is still not coming on. Occasionally it flickers on, but mostly it stays off. We got a warning on watcher when /data/spectro's free space dropped to 48%, and were told we had 2920 Mby of free space. Do we really want a a watcher error under these conditions? ************************************* Mirror positions: primary: Received=" 1, 1, 1, 1, 1, 1 axis homed" 0 2 I Received=" 000005900, -000003300, 000000050, 000004200, 000014600, 000 014150 commanded position" 0 2 I Received=" 000005900, -000003300, 000000050, 000004200, 000014600, 000 014150 actual position" secondary: focus 1350 for spectroscopy. 0 2 I Received=" 1, 1, 1, 1, 1 axis homed" 0 2 I Received=" 001411450, 001398550, 001388300, -000001900, -000006550 comma nded position" 0 2 I Received=" 001411448, 001398604, 001388198, -000001900, -000006550 actua l position" The MIG TPM link did not appear to be working. It gave these messages: [NULL_LOG,tMIG0,4,W] 1476l > 120l ... delay has been stepped [NULL_LOG,tMIG0,6,E] MIG is off or channel is in error state. [NULL_LOG,tMIG0,1,W] MIG Data[0] wasn't updated. [NULL_LOG,tMIG0,6,E] MIG is off or channel is in error state. [NULL_LOG,tMIG0,1,W] MIG Data[1] wasn't updated. [NULL_LOG,tMIG0,6,E] MIG is off or channel is in error state. [NULL_LOG,tMIG0,1,W] MIG Data[2] wasn't updated. [NULL_LOG,tMIG0,6,E] MIG is off or channel is in error state. [NULL_LOG,tMIG0,1,W] MIG Data[3] wasn't updated. [NULL_LOG,tMIG0,6,E] MIG is off or channel is in error state. [NULL_LOG,tMIG0,1,W] MIG Data[4] wasn't updated. [NULL_LOG,tMIG0,6,E] MIG is off or channel is in error state. [NULL_LOG,tMIG0,1,W] MIG Data[5] wasn't updated. [NULL_LOG,tMIG0,0,I] MIG Channel 2 = 0.000000 [NULL_LOG,tMIG0,0,I] MIG Channel 2 = 0.000000 [NULL_LOG,tMIG0,0,I] MIG Channel 3 = 0.000000 [NULL_LOG,tMIG0,0,I] MIG Channel 4 = 0.000000 [NULL_LOG,tMIG0,0,I] MIG Channel 5 = 0.000000 [NULL_LOG,tMIG0,0,I] MIG Channel 6 = 0.000000 LN2: Spectros 1 and 2 were filled at 01:30 and topped off at 07:30. The imager dewar was swapped out at 07:30. On closing, the Alt_PLC_Perm and Wind_Alt_PLC_Perm interlocks were found to be bypassed. They were left in that state. Weather: The night started out mostly clear. A band of clouds stayed near the eastern horizon most of the evening. At about 02:00 high clouds started moving in from the west. Temperatures were mild, ranging from 3 to 2 C. Winds were out of the west, shifting to the northwest at about midnight. As they shifted, they increased from 5 mph to 15-20 mph. Dust counts were low, 40-50 particles/foot^3. Relative humidity was about 30%.