Date: Mon, 11 Oct 1999 07:16:29 -0600 From: Scot Kleinman Subject: obs log 10oct99 (51462) Sunday October 10, 1999 Apache Point Observatory SDSS 2.5m Observing Log (MJD 51462) Observing Team: Jeffrey Hayes (APO) Scot Kleinman (APO) Steve Kent (FNAL) -- PIC David Schegel (PU) Eric Neilson (JHU) - [P|M]T Plan for tonight: Start with engcam to see if we can figure out what went wrong last night with the last spectro cartridge (2 hours and we couldn't find a star). Then switch to spectro (4 plates ready) until imager is cool and conditions are photometric (estimated time is around midnight for the camera to be ready.) Summary: 2.5 spectro plates done, imager should be cool now. No images from the engcam. One telescope runaway, mirror oscillations, MCP crash and other minor gotchas. The spectro dewars lasted all night. There was no resolution of last night's problems, although we sucessfully observed the same plate tonight. We were late in starting due to the imager being on the telescope through the initial stages of its refill. While we were waiting, we brought the primary out of it's -80000 safe position and it immediately started oscillating. We repeatedly put it both at -80000 and a couple times at the RLIM positions (-120000) and still no luck stopping the oscillations. A phone call to French informed us that the mirror has been sensitive both at large alt (~90 deg) and low alt (which we already knew about). So, we brought the mirror back to -80000 and waited until we could change instruments and move the telescope from 90deg (where it was) to something more acceptable to the primary. We later set the mirror the its nominal collimated position via the tcc_prim host on the TCC without trouble at an altitude of ~60deg. The then MIG mirror values were: [MIG,tMIG,0,I] MIG Channel 1 = 1.926500 [MIG,tMIG,0,I] MIG Channel 2 = 0.560500 [MIG,tMIG,0,I] MIG Channel 3 = 0.559000 [MIG,tMIG,0,I] MIG Channel 4 = 8.006000 [MIG,tMIG,0,I] MIG Channel 5 = 1.907500 [MIG,tMIG,0,I] MIG Channel 6 = 0.407500 [MIG,tMIG,0,I] MIG Channel 7 = 8.094000 [MIG,tMIG,0,I] MIG Channel 8 = 8.244000 with a focus set for -3400 (approx. engcam value) prim stat via the TCC revealed: 0 3 I Received=" 000006400, -000003350, -000000450, 000011050, 000007350, 000007050 current position" There are all looked like normal values. (We also realized that the mirror values we recorded last night were from when the mirror was at -80000, so they unfortunately, are meaningless to debug our end of night problems.) When getting ready to mount the engcam, we managed to break the two fiber optic cables on the side of the instrument with the manipulator as we were removing it from the cartridge storage rack. The cables are unfortunately aligned with the trunion and just below it, so coming in low with the manipulator sheared the suckers right off. We called Craig and he told us where to find replacement fibers. After mucking with the connectors, we got the replacement fibers installed and proceeded to mount the instrument. While we were mounting the engcam, Jeff's sdss-commish login session on monx2 totally shut down, logging him out of all his processes. This happened once last night as well. We do not know why. Unfortunately, we could not get an image from the engineering camera. An attempted gcam doread would come back with a ppk(?) error 116. We reverted to our one guide camera trick and rebooted the guider mac. Still no images. Jeff and David connected the data fibers back to the spectrograph and we changed the instrument in the TCC and were then able to take successful guide images with the spectrograph guider -verifying the guider mac was ok (and adding a second guider trick to our arsenal). Unfortunately, on putting the fibers back, we could still not get an image and we gave up, switching to the spectrograph (cartridge 4, plates 194). This was about 23:00. The engcam balance was about 280 on the CWs, and the spectrograph was about 360. At 23:30, we got imager secondary dewar pressure critical errors from the Watcher. We then setup iop and did a cc ln2stat to get more information. (The Watcher could benefit from providing the observers with more detailed error messages that they can actually understand. We have frequent discussions about what a given error message might actually mean and what we should be doing about it. Ex. What is a vt+ voltage and what do we do if one goes out of spec? What is wrong with an axis when we get a axis MCP error?) The cc ln2stat command returned: LN2_FILL ON FILLTIME 60_MINUTES NEXT_FILL 1_MINUTES NORM_RETRIG 1 FILL_MODE COLD WARM_FILLTIME 15_MINUTES WARM_FILLS 2 WARM_RETRIG 6 FILLFAULT NO EMPTY_TRIGGER NO # SETUP EMPTY AUTOFILL_DEWAR_1 1 0 AUTOFILL_DEWAR_2 1 0 AUTOFILL_DEWAR_3 1 0 AUTOFILL_DEWAR_4 1 0 AUTOFILL_DEWAR_5 1 0 AUTOFILL_DEWAR_6 1 0 AUTOFILL_DEWAR_7 1 0 AUTOFILL_DEWAR_8 1 0 2NDLN2_FILL OFF We then did a cc starfillseq and the errors did not go away. We went to the telescope (coincidentally, the telescope had aborted during a slew, so we checked that out while we were down there, simply using the Menu to stop, hold, and reset the amps) and found the large supply dewar empty. This was surprising since Jon Brinkman replaced the dewar only a few hours earlier. (As we recall, this was around 18:30, the same time we were refilling the spectrograph dewars.) We changed the dewar and did a new cc starfillseq and things seemed to get better (at least the Watcher stopped beeping at us!). At ~00:20, after focusing the spectrograph (no trouble finding stars), these are the following MIG numbers: (focus=1500 with inst=spectro) [MIG,tMIG,0,I] MIG Channel 1 = 1.927000 [MIG,tMIG,0,I] MIG Channel 2 = 0.560000 [MIG,tMIG,0,I] MIG Channel 3 = 0.558500 [MIG,tMIG,0,I] MIG Channel 4 = 3.070000 [MIG,tMIG,0,I] MIG Channel 5 = 1.907000 [MIG,tMIG,0,I] MIG Channel 6 = 0.407500 [MIG,tMIG,0,I] MIG Channel 7 = 3.153000 [MIG,tMIG,0,I] MIG Channel 8 = 3.300000 At 00:34, we started guiding on plate 194, cartridge 4 and by 00:39, we were taking science spectra (!!). Throughout the night, the Watcher was giving MCP axes errors as we specified in a GNATS report filled out last night. The text of the errors are as follows: altitude (0: Oct 11 00:35:45 MDT) azimuth (0: Oct 11 00:35:45 MDT) rotator (0: Oct 11 00:35:45 MDT) We also got similar TCC errors, sometimes coincident with the MCP errors, sometimes not. I think only once, though, did the telescope really have any problems moving during these errors. By 01:00, the temp plots of the imager CCDs (could we get axes labels/units on the plots?) showed that the CCDs were still a couple hours away from being cold, so we carried on with the spectroscopy. 01:30 - 3rd science exposure finished- start calibrations. Seeing has been around 2" with excursions higher (as reported by the DIMM; FWHM of guider images around 3-3.5". We finished the last guider image with an efficiency of 99%! 01:55 - Switch to cartridge 5, plate 209 - the fateful cartridge from last night. Will we see stars? (The imager CCDs are still cooling and the seeing is not great.) At 02:00, Jeff shut off the bug zapper in the bottom of the 3.5m as it seemed to be shorting out and he could smell ozone. 02:20 -Success! We see guide stars (after an easyFiber and twiddling focus to 1525)! We have no idea what happened last night. FWHMs are about 1.8". The DIMM is reading 1.7". By 02:40, the CCDs in dewar 4 (numbers 14,24, etc.) have cooled down to within spec (according to the Watcher), but the others are still out. Some have about 10 degrees of cooling left. We had to power cycle the spec. mechanicals once during this plate's 4 exposures. 03:55 ended 4th science exposure of plate 209. Start calibrations. During the switch to the next cartridge, the MCP died during an interactive Menu session and had to be rebooted. This occurred after an "lb" command then a "^30". It looks like it took the "^30", but died before it could move the counterweights. We then had to reset the fiducials, of course, but we could not get the telescope to find the valid 120deg Az fiducial even after many attempts. We eventually settled for the 105deg (non-valid - what exactly is the difference between a valid and non-valid fiducial?) fiducial and carried on. We have had mixed luck in recent past running off the 105 or 135 deg Az fiducial. We also re-started the TPM logs (51462-3: see below). 04:30 Switch to cartridge 1 plate 235. Seeing still not quite good enough for imager and not all chips are cooled completely and twilight is near, anyhow. We had trouble finding any stars and mucked around a while. Around 5:10, the telescope ran away in altitude. We were slewing to an FK5 star near the spectro field. The desired slew position was: -60.26, 37.92, 133.11. When the telescope got there, it said "Slew End", but kept moving in altitude to 90.10 (where we noticed it). It appeared to run away very quickly, getting to 90deg in only 3-4 TCC reporting cycles. We then slewed to the stow position in azimuth and Jeff pushed on the telescope while I manned the stop button. After he got it a few degrees, I pressed the stop button and the telescope was at 87deg alt. We could then move it normally with the MCP Menu. A happy recovery. We came back and slewed to a nearby fk5 star. We found it, but it had few counts. We hoped for the best and went for the field stars, but couldn't find them. David ran to the telescope and saw that the flat field flaps were closes (we did not tell them to do so- is this related to the runaway?). We opened them and happily found our field guide stars. At ~5:40, we got the field and proceeded to tweak guiding and do calibrations. Started science exposures at 5:50. We finished the second exposure at 06:30 and decided to close, after putting the primary at the safe -80,000 position. By 06:30, all but CCDs 64, 46, 56, 92 had their CCDtemps in spec (but see the Watcher note below) and these plus 55, 52, 91 had their heaterv values out of spec. Note the spec dewars held all night. We refilled at the end of the night, around 7 am. Throughout the night and last night as well, the Watcher periodically appeared to lose connection to cameras. On the main display, the Camera title changed to No Connection, then would spontaneously change back to camera. Also, at some point the Watcher decided all the Imager CCD temps were within spec, but a right click on ccdtemp in the voltages section revealed a few chips that were still flagged red and had temps just outside of the desired range. We recorded the TPM logs in tpm-51462.dat, tpm-51462-2.tpm (oops) and tpm-51462-3.dat. Since I (Scot) thought my account was setup for the fnal scp command, where Jeff's didn't appear to be, I ran endNight 51462 -doSpectro to see if we could successfully complete the entire endNight script! It did and there was much rejoicing. While closing the telescope, the MCP was scrolling numerous messages about waiting for the next frame despite the fact that we had done an axis stop and a telstop from our TCC sessions. We survived and rebooted the MCP after the telescope was stowed and the dewars filled. The dewars were filled at 7:00 am, cartridge 1 is on the telescope and the primary is at -80000. Scot and Jeffrey