Date: Thu, 14 Oct 1999 09:04:18 -0600 From: Dan Long Subject: 2.5m obslog 991013 ( MJD 51465 ) WEDNESDAY OCTOBER 13, 1999 APACHE POINT OBSERVATORY SDSS 2.5m OBSERVING LOG Observering team: Dan Long (APO) Scot Kleinman (APO) Dave Schlegel (PU) Scott Burles (UC) Tonight's plan was to get imaging data for stripes 82 S between 310 and 0 RA, 76 N between 0 and 35 RA, and 86 N between 35 and 60 RA, and if the seeing was very good get a focus sweep. After a rocky start ( overall we lost maybe half an hour due to the lengthy checkout, and about an hour and a half to archiver problems ) we ended up with most of what we aimed to get tonight. We got about half of the equatorial stripe 82 S, all of stripe 76 N in reasonable seeing of 1.2 to 1.5 arcsec, and all of stripe 86 in similar seeing. For some reason tonight, every scan was right on target according to skyGang ( observed and catalog stars on top of each other ). Also instead of a rotator offset of -.08 deg which all scans last night averaged, tonight's scans asked for a rotator offset of about -.018 deg. What's different from last night? Well the optics had to be homed, and the normal azimuth fiducial worked. The mig numbers would indicate that the mirrors are in the same place as last night. Scot noticed when he set the Az fiducial that the first time the positional change was quite large. He couldn't recall how large but it was at least tens of arcseconds and maybe even arcminutes. This could certainly account for poor pointing and even explain why the meridian scan last night had no Dec error. This should be checked to see how large the difference really is when setting on fiducial 32 instead of 33. If it really is that large we cannot use fiducial 32 to set Az until a better reference position is calculated. The afternoon checkout took much longer than normal due to communication problems to the secondary and having to home the optics after the power cycling the galils got during the day. We started around 17:30 and it was about 20:00 when we started our bias drift. RUN SUMMARY 1038: Test 1039: Bias run of about 23 frames 1040: 82 S RA 310 to 324 Science after frame 35 ( Sorry Jen ) Was to be stripe 82 S setup. Since we had to take time to replace the imager dewar, and had problems getting skyGang to run, we ended up drifting into the desired scan region. Rather than set up again, we continued this scan and called it a science scan. The last lskip was around frame 35. This scan ran until Frame 173 when we had to stop due to id1 frame pool filling. Nice globular around frame 165. scanStripe 82 115 -strip S rotator offset -.023 no boresight offset change needed according to skyGang 1041: Test bias to see if we were back on the air after dealing with id1 archiver problem. 1042: Setup for 76 N scanStripe 76 170 -strip N rotator offset -.018 no boresight offset change needed according to skyGang 1043: 76 S RA 0 to 37 deg 00:50 to 03:20 Science scan Like run 1040, the slew was right on according to skyGang, so no offset correction was needed. Seeing was 1.2 to 1.5 arcseconds during the run. 1044: Setup for 86 N scanStripe 86 -155 -strip N rotator offset -.017 no boresight offset change needed according to skyGang 1045: 86 N RA 35 to 68 deg 03:45 to 05:55 Science scan Good seeing throughout. Starting around 1.5 and improving to 1.1 by the end. 1046: Start of focus sweep, but camera rebooted right after first focus change. TROUBLES I found the secondary mirror galil wouldn't talk during the afternoon check. After the power outage today the primary was lost but I could telnet to it. However the secondary would not respond, even after a few times of power cycling the power supply to the controller/logging out the port. I got Craig and after some more poking around he took a laptop to the controller itself and found it was operational and talking. Once it was connected back to the terminal server, one could telnet to it. The best guess is that it got into some state after the power hit which even logging out the port didn't cure. I had to home optics after power outage today. Aren't the galils supposed to be converted to ups power at some point? Primary Oscillation around zenith. Nothing new, just noting that it happened. Trying to run skyGang on stripe 82 we got the following error astrom> flInit /data/astro/51465/1040 -koCat /p/act/v1_0/ko/koActCat-82.fit astrom> skyGang 1040 45 55 -xBore 22.7 -useHdr 2 -leading -diff Open /XWIN PGPLOT /xw: Failed to allocate pixmap. Error: tclChainElementGetByPos: chain h1099 is empty This was using version v2_14. Is this a bug or something I'm doing wrong? Forshadowing things to come, we had trouble with initNight -tapeLoad. Below is what iop spilled out after the first time I issued an initNight -tapeLoad. Wed Oct 13 20:01:04 MDT 1999: tapeWaitFor reports success Wed Oct 13 20:01:42 MDT 1999: IOP Not ready (blew up on pad): tapeMount: Request sense failure Failed to receive exabyte command completion status invoked from within "iccExec $tapeList archReadSense 1" invoked from within "set sense1 [iccExec $tapeList archReadSense 1]..." (procedure "tapeRewind" line 5) invoked from within "tapeRewind $tapeList " invoked from within "if {$doRewind} { tapeRewind $tapeList }" (procedure "tapeLoad" line 5) invoked from within "tapeLoad $tapeList $doRewind $doUnload" Repeating initNight -tapeLoad completed without error. Later the watcher gave complaints about id1 tape drive error, but qDepth was always zero, so I wasn't sure how to interpret it as it appeared to be keeping up. This was a ruse. Finally the pool filled and scan was stopped just before it would have crashed. Had to reboot photo 1 to get id1 tape drives to work. Neither drive needed cleaning, they were not rewound, and the claim was that the qDepth was zero. Pausing and resetting that particular drive didn't seem to bring it back to life, so the crate was rebooted. This necessitated manually archiving the frames left unarchived in the pool. The catasRecovery command was used... specifically catasRecovery 51465 { 1039 1040 } frames -noarchived Since the pool had filled the watcher was rightly complaining that the id1 pool was critically full. However, once frames were being archived again and the statistics for id1 on the watcher showed dozens of frames had been archived, the error persisted. The queue depth had maxed out at 300 and more were being added by catasRecovery as soon as they were archived. I assume 300 is a real limit on the queue depth. To be fair to the watcher, something was still up with id1, as I couldn't start a drift either. The complaint was ... id1 {Failed to send archiving message} ... Once catasRecovery stopped queuing unarchived frames, the queue depth began dropping below 300. At this point I could start a drift. However the watcher continued to display the poolFrameFull critical error. It continued beeping until the queue depth eventually reached zero, over an hour and a half after we resumed taking data. A footnote to the archiver dealings. I noticed that id3 is reporting many more archErrCounts than the other nodes ex ( 90 2817 compared to id1's 12 24 ). endNight won't run because of one unarchived frame which apparently isn't possible to archive. At the very end of the log is a transcript of what I get when I run endNight -doImager, and what I managed to do running it piecemeal. Finally, the imager micros rebooted themselves twice at the end of the night. Once at about 06:00 and again around 07:40. Both times I "iack-ed" them and everything seemed OK afterwards. MIRROR NUMBERS Primary: setmir: 0 -27.4 7.8 427 1495 galil: 6400, -3350, -450, 11050, 7350, 7050 linear gauges 1 = 1.9265 2 = 0.5595 3 = 0.5585 5 = 1.9060 6 = 0.4070 4 = 4.7530 7 = 4.8280 8 = 4.9950 Secondary: ( inst=imager focus=-1667 alt=63.9 deg T=11.1 C) setmir: -436 -110.5 45 0 155.2 galil: -734276, -581699, -333400, -10066, -7299 Available plates for observing are Cartridge Current Plate 2 189 21:00 4 205 01:00 5 209 02:30 1 235 05:00 TPM data was logged for the start of the night in "/mcptpm/tpm-514675". Notice the typo in the mjd... there's no going back typing in the tpm. Nitrogen notes: Both spectrographs were filled at 18:30 once the telescope was open, and again at 07:00 after we closed. The imager dewar was changed at about 21:00 after having been changed at Noon, and at 05:00. Three 180 L dewars in one day... Eeeek! EndNight for MJD 51464 completed once I stepped through it by hand. Someone will want to verify this before blowing gangs away. WEATHER Skies tonight were clear. The temperature was steady in the low 50s. Winds were light and variable mostly from the North and East, humidity was around 30%, and dust counts were under 1000. Seeing tonight was between 1.0 and 2.2 arcseconds on the imager, and between 1.1 and 2.6 arcseconds on DIMM. ENDNIGHT ENTRAILS im> endNight -doImager 51465 endNight gangCheck: untransferred gangs? checking runs 1038 1039 1040 1041 1042 1043 1044 1045 1046 none endNight frameCheck: unarchived frames? checking runs 1038 1039 1040 1041 1042 1043 1044 1045 1046 id1 1040 173 51 1 0 # node run frames ccd completed? permanent? Found unarchived frames in runs 1040 1040 1040 1040 1040 1040 issuing a: catasRecovery 51465 1038 1039 1040 1041 1042 1043 1044 1045 1046 frames Are the tapes in the drives still 51465's? [y]: writing unarchived fields to tape id1: issuing archive 1040 173.51 0 1 1 Rerun endNight to check recovery after archiving to tapes completed im> im> poolDir *1040* comes back empty As it appears to be gone from the pool, and is the last frame of a run, I decided to proceed with endNight by hand, issuing the following sequence which I believe to be the equivalent of endNight -doImager ( without the frame and gang check ). endNight -parseMurmur endNight -ioptape endNight -shiptape endNight -copy endNight -ccd endNight -scpPPD endNight -mailSum im> endNight 51465 -parseMurmur endNight parseMurmur: im> endNight 51465 -ioptape Of course this only archived /astrolog. It took a few tries to figure out how to get it to do the rest, so there are 3 copies of /astrolog on this tape. The following is what I sent to get it to do the imaging gang files. im> set fd [open /dev/tty w]; fcntl $fd NONBLOCK im> exec ioptape -mjdDir 51465 -mail 1 -doImager 1 2>@$fd ==================================== Thu Oct 14 07:33:24 MDT 1999 we will select a directory from /data/astro* and /data/photo* and use tape drives /dev/mt/tps0d3nrnsvc /dev/mt/tps1d6nrnsvc we will status the tape drive, this may take some time.... Controller: SCSI Device: Quantum: DLT4000 CC37Q7 Status: 0x8024260 Drive type: DLT Media : READY, writable, at EOD, block 854 rewinding /dev/mt/tps0d3nrnsvc Media : READY, writable, at BOT will check if tape was labelled with $SDSSTAPE_DIR/bin/labeltapes 1+0 records in 0+1 records out VOL1JL1387 SH624F0702386 3 Thu Oct 14 07:33:37 MDT 1999 skipping file mark 1 Thu Oct 14 07:33:40 MDT 1999 skipping file mark 2 Thu Oct 14 07:33:44 MDT 1999 skipping file mark 3 we will status the tape drive, this may take some time.... Controller: SCSI Device: Quantum: DLT4000 CC37Q7 Status: 0x8024260 Drive type: DLT Media : READY, writable, at EOD, block 854 rewinding /dev/mt/tps1d6nrnsvc Media : READY, writable, at BOT will check if tape was labelled with $SDSSTAPE_DIR/bin/labeltapes 1+0 records in 0+1 records out VOL1JL1388 SH628F1055528 3 Thu Oct 14 07:33:57 MDT 1999 skipping file mark 1 Thu Oct 14 07:34:00 MDT 1999 skipping file mark 2 Thu Oct 14 07:34:04 MDT 1999 skipping file mark 3 enter one or more run numbers to archive [default: all of 1038 1039 1040 1041 1042 1043 1044 1045 1046] Will archive mjdDir: 51465 runs 1038 1039 1040 1041 1042 1043 1044 1045 1046 Thu Oct 14 07:34:24 MDT 1999 Gross (maximum) estimate. Archiving 28942166 blocks at 3073 blocks/s. Finish at Thu Oct 14 10:11:23 1999 skipping to end of tape /dev/mt/tps0d3nrnsvc skipping to end of tape /dev/mt/tps1d6nrnsvc Thu Oct 14 07:34:33 MDT 1999 Starting /astrolog archiving. 358023 blocks to archive. Estimate finish at Thu Oct 14 07:36:29 1999 Thu Oct 14 07:36:07 MDT 1999 /astrolog finished. Thu Oct 14 07:36:07 MDT 1999 Starting /data/astro/51465/1038 archiving. 35446 blocks to archive. Estimate finish at Thu Oct 14 07:36:18 1999 Thu Oct 14 07:36:23 MDT 1999 /data/astro/51465/1038 finished. Thu Oct 14 07:36:23 MDT 1999 Starting /data/photo/51465/1038 archiving. 18752 blocks to archive. Estimate finish at Thu Oct 14 07:36:29 1999 Thu Oct 14 07:36:33 MDT 1999 /data/photo/51465/1038 finished. Thu Oct 14 07:36:33 MDT 1999 Starting /data/astro/51465/1039 archiving. 141577 blocks to archive. Estimate finish at Thu Oct 14 07:37:19 1999 Thu Oct 14 07:37:27 MDT 1999 /data/astro/51465/1039 finished. Thu Oct 14 07:37:27 MDT 1999 Starting /data/photo/51465/1039 archiving. 70039 blocks to archive. Estimate finish at Thu Oct 14 07:37:49 1999 Thu Oct 14 07:37:51 MDT 1999 /data/photo/51465/1039 finished. Thu Oct 14 07:37:52 MDT 1999 Starting /data/astro/51465/1040 archiving. 1419673 blocks to archive. Estimate finish at Thu Oct 14 07:45:33 1999 Thu Oct 14 07:46:27 MDT 1999 /data/astro/51465/1040 finished. Thu Oct 14 07:46:29 MDT 1999 Starting /data/photo/51465/1040 archiving. 750745 blocks to archive. Estimate finish at Thu Oct 14 07:50:33 1999 Thu Oct 14 07:50:53 MDT 1999 /data/photo/51465/1040 finished. Thu Oct 14 07:50:54 MDT 1999 Starting /data/astro/51465/1041 archiving. 26602 blocks to archive. Estimate finish at Thu Oct 14 07:51:02 1999 Thu Oct 14 07:51:04 MDT 1999 /data/astro/51465/1041 finished. Thu Oct 14 07:51:04 MDT 1999 Starting /data/photo/51465/1041 archiving. 9428 blocks to archive. Estimate finish at Thu Oct 14 07:51:07 1999 Thu Oct 14 07:51:10 MDT 1999 /data/photo/51465/1041 finished. Thu Oct 14 07:51:10 MDT 1999 Starting /data/astro/51465/1042 archiving. 344995 blocks to archive. Estimate finish at Thu Oct 14 07:53:02 1999 Thu Oct 14 07:53:17 MDT 1999 /data/astro/51465/1042 finished. Thu Oct 14 07:53:18 MDT 1999 Starting /data/photo/51465/1042 archiving. 181936 blocks to archive. Estimate finish at Thu Oct 14 07:54:17 1999 Thu Oct 14 07:54:20 MDT 1999 /data/photo/51465/1042 finished. Thu Oct 14 07:54:22 MDT 1999 Starting /data/astro/51465/1043 archiving. 2118267 blocks to archive. Estimate finish at Thu Oct 14 08:05:51 1999 Thu Oct 14 08:07:04 MDT 1999 /data/astro/51465/1043 finished. Thu Oct 14 08:07:06 MDT 1999 Starting /data/photo/51465/1043 archiving. 1114411 blocks to archive. Estimate finish at Thu Oct 14 08:13:09 1999 Thu Oct 14 08:13:19 MDT 1999 /data/photo/51465/1043 finished. Thu Oct 14 08:13:20 MDT 1999 Starting /data/astro/51465/1044 archiving. 199064 blocks to archive. Estimate finish at Thu Oct 14 08:14:24 1999 Thu Oct 14 08:14:33 MDT 1999 /data/astro/51465/1044 finished. Thu Oct 14 08:14:33 MDT 1999 Starting /data/photo/51465/1044 archiving. 102675 blocks to archive. Estimate finish at Thu Oct 14 08:15:06 1999 Thu Oct 14 08:15:14 MDT 1999 /data/photo/51465/1044 finished. Thu Oct 14 08:15:15 MDT 1999 Starting /data/astro/51465/1045 archiving. 1990025 blocks to archive. Estimate finish at Thu Oct 14 08:26:02 1999 Thu Oct 14 08:26:56 MDT 1999 /data/astro/51465/1045 finished. Thu Oct 14 08:26:58 MDT 1999 Starting /data/photo/51465/1045 archiving. 1049137 blocks to archive. Estimate finish at Thu Oct 14 08:32:39 1999 Thu Oct 14 08:32:50 MDT 1999 /data/photo/51465/1045 finished. Thu Oct 14 08:32:50 MDT 1999 Starting /data/astro/51465/1046 archiving. 203487 blocks to archive. Estimate finish at Thu Oct 14 08:33:56 1999 Thu Oct 14 08:34:08 MDT 1999 /data/astro/51465/1046 finished. Thu Oct 14 08:34:09 MDT 1999 Starting /data/photo/51465/1046 archiving. 102675 blocks to archive. Estimate finish at Thu Oct 14 08:34:42 1999 Thu Oct 14 08:34:49 MDT 1999 /data/photo/51465/1046 finished. 36415 blocks 35468 blocks 18784 blocks Thu Oct 14 07:36:33 MDT 1999 141644 blocks 70132 blocks Thu Oct 14 07:37:52 MDT 1999 1420288 blocks 751660 blocks Thu Oct 14 07:50:54 MDT 1999 26620 blocks 9448 blocks Thu Oct 14 07:51:10 MDT 1999 345148 blocks 182164 blocks Thu Oct 14 07:54:21 MDT 1999 2119172 blocks 1115764 blocks Thu Oct 14 08:13:20 MDT 1999 199156 blocks 102808 blocks Thu Oct 14 08:15:14 MDT 1999 1990876 blocks 1050412 blocks Thu Oct 14 08:32:50 MDT 1999 203580 blocks 102808 blocks Thu Oct 14 08:34:49 MDT 1999 im> im> endNight 51465 -shiptape endNight shiptape: What is label of tape in /dev/tps0d3 (left drive)? JL1387 Shipping: JL1387 im> endNight 51465 -copy endNight copy: done copying to /usrdevel/observer/ppd/51465 im> endNight 51465 -ccd endNight ccd: done copying to /usrdevel/observer/astrolog/51465 im> exec $env(IOP_DIR)/bin/ppd.sh 51465 1 0 sdReport-*.par is missing in 51465 It's not clear that scpPPD worked. im> set runList [findRunsDone 51465] 1038 1039 1040 1041 1042 1043 1044 1045 1046 im> set sciRuns [findSciRuns $runList] run list: 1038 1039 1040 1041 1042 1043 1044 1045 1046 which are the science runs? 1040 1043 1045 1040 1043 1045, correct? [y] 1040 1043 1045 im> foreach run $sciRuns { mailRunSummary $run -mjd 51465} Open /CPS Close /CPS Open /CPS Close /CPS Open /CPS Close /CPS Open /CPS Close /CPS Open /CPS Close /CPS Open /CPS Close /CPS im> Dan