Date: Fri, 03 Mar 2000 07:06:18 -0700 From: "s. a. snedden" Subject: 2.5-meter Obs Log (MJD 51606) Tuesday Mar 02 2000 Apache Point Observatory SDSS 2.5m Observing Log (MJD 51606) Observing Team: Scot Kleinman (APO) (3 p.m.-midnight) Atsuko Nitta Kleinman (APO) (3 p.m.-midnight) Stephanie Snedden (APO) (11 p.m.-8 a.m.) Dan Long (APO) (11 p.m.-8 a.m.) Bing Chen (JHU) PT Visitors: Steve Kent (FNAL) [official head honcho] Robert Lupton esq. (PU) Larry Carey (UW) French Leger (FNAL) Jim Gunn (PU) (phone) Connie Rockosi (UofC) (phone) Russell Owen (UW) (phone) Ron Rechenmacher (FNAL) (phone) THE PLAN: ========= Imaging if weather and seeing remain acceptable. FASTT field, then onto stripes 9 and 36. If seeing gets bad, then spectroscopy with science plates and plate 317 having top priority. SYNOPSIS FOR NIGHT: =================== Clear. Imager FWHM were about 1.5 arcsec. Software bugs kept us from getting science data for eight hours. We took imaging data on stripe 9 S starting at lambda=-15 during the second half of the night. We scanned FASTT field J at the end of the night. We had DA problems that will affect shipping of the data tapes. TELESCOPE STATUS: ================= Imager is mounted with counterweights at 720V. Telescope is stowed at 30 deg altitude. INTERLOCKS: =========== None bypassed. SOFTWARE USED: ============== IOP: v3_8_8 Watcher: v2_6_4 MCP: NOCVS|Mar 2 2000|15:05:47 -- later switched to v3_1_2 WEATHER SYNOPSIS (see below for hourly data): ============================================= Clear. Winds out of the west at 10 mph. DIMM seeing variable--1 arcsec at best with a bad spell going to 4 arcsec at 2:30, then settling to about 1.5 arcsec after 03:00. We had seeing of about 1.5 arcsec (FWHM). GENERAL COMMENTS (see Problems below for details): ================================================== Imager focus was was about 330 at the start of the night, but the out of focus donuts didn't look very good. This led us to eventually realize that there was a problem with the primary actuators (See "Horendous Donuts" in the Problem section). We then had problems with azimuth amplifier aborts again (see "Azimuth aborts" in the Problems section). These were ultimately worked around by running in unmonitored mode in the azimuth axis, on an older version of the MCP. At about 02:20 we started science imaging on stripe 9 S at a starting lambda of -15 deg. We encountered more archiving problems (See the problems section), but kept the drift going anyway, because we had lost so much time to the previous problem. We ended the drift at about 05:00 and then scanned across FASTT field J. We ended imaging with a bias drift. DATA ACQUISITION (see IOP/SOP output below): =========================================== IMAGING: -------- Run 1225, 64 frames: bias run. Run 1226, 82 frames: initial FASTT field setup (10,-81 N) Run 1227, 316 frames: trial 2 of same FASTT field with relaxed primary Run 1228, 199 frames: Engineering The above were not archived or saved (lost in poolInit). They were all out of focus, anyway, and the primary actuators were not set correctly. Run 1229, 76 frames: Labeled science, but really setup (ignore) Run 1230, 31 frames: Setup (ignore) Run 1231, 233 frames: Science scan of 9 S, starting lambda=-15 Run 1232, 21 frames: Setup (ignore) Run 1233, 72 frames: Science - FASTT field J Run 1234, 57 frames: Bias id-1 data is not on tape (See DA problems below). As a result, endNight was not run. The day shift will have to archive this data from the hard drive, and run endNight. This will delay shipping of tapes, of course. ***** N.B. Until this data is archived, a poolInit must not be done! **** SPECTROSCOPY: ------------- None MIRRORS: ======== Primary: setmir( cmd ): 0 -3.9 3.9 213.5 213.5 galil( act ): 4850 -2350 100 -9500 50 -100 linear gauges before* after 2-10-00(after latest collimation) 1 = Axial A = 1.9320 1.9315 1.9315 2 = Axial B = 0.5565 0.5565 0.5570 3 = Axial C = 0.5565 0.5565 0.5570 4 = Transverse = 0.4029 0.4085 0.4075 5 = East Lateral = 1.9835 1.9005 1.8980 6 = West Lateral = 0.4839 0.4000 0.3990 Secondary: ( inst=imager focus=-243 alt=44.8 deg T=2.0 C) setmir( cmd ): -454.31 5 -5 50 14.94 setmir( act ): -454.29 4.96 -5.01 49.99 14.56 galil ( act ): -560301 -573239 -583394 -3750 -8550 linear gauges 1 = Axial A = 0.390 2 = Axial B = 0.428 3 = Axial C = 0.417 4 = Transverse = -0.028 *This was the primary position before the mirror specific code was reloaded for the primary. LN2 SCHEDULE: ============= Spectrographs filled at 18:30 APO Time. Spectrographs re-filled at 06:00. Imager dewar swapped out at 06:15. A dewar about 1/3 full was left with the cart in the LN2 shed for use on the spectrographs. WEATHER DATA: ============= Wind Time Temp F Dewp F MPH Direction Dust Sky Precip. 16:01 36 8 13 327 (NNW) 239 clear none 17:01 38 10 18 296 (WNW) 197 clear none 18:01 35 10 14 303 (WNW) 241 clear none 19:01 35 10 13 280 (W) 202 clear none 20:01 36 10 17 280 (W) 353 clear none 21:01 35 11 10 287 (WNW) 417 clear none 22:01 35 12 12 290 (WNW) 453 clear none 23:01 35 12 7 265 (W) 494 clear none 00:01 36 11 7 257 (WSW) 464 clear none 01:01 35 7 7 229 (SW) 390 clear none 02:01 37 5 9 269 (W) 363 clear none 03:01 36 -1 11 243 (WSW) 255 clear none 04:01 33 5 9 194 (SSW) 396 clear none 05:01 33 6 8 189 (S) 413 clear none 06:01 33 2 11 221 (SW) 348 clear none DATA LOGS FROM IOP/SOP: ======================= Not available. PROBLEMS: ========= Optics: ------- We started the evening, sitting at our FASTT field, waiting for twilight. Once we started observing, we noticed our out of focus donuts were not uniformly illuminated. We noticed the secondary's 3 axial mig readings were not as close to each other as they usually are. We decided to home the secondary, and that's when things got interesting. Our first few home attempts did not work quite right - in one, for example, the Galil values for Axials A,B,&C were about -7.6e6, -8.8e6, -7.3e6 while the MIG values read about 5.5, 4.3, and 6.2, respectively. We then did a setmir to 0 0 0 0 0 (no piston, no tilt) and it reported more or less that it had succeeded, but the MIG readings (again for axials A,B,&C) begged to differ. They read -0.0020, -1.2769, and -0.6439. Connie and Jim Gunn helped us diagnose things to this level and by this time, Larry had appeared on the scene, so we re-homed and re setmir'd and this time got much more reasonable results (although we had repeated the homings earlier with similar unhappy results): it homed to approx. all -7250000 Galil units and 5.7 in Mig readings. The setmir 0 0 0 0 0 this time resulted in numbers about -0.06, -0.03, and -0.03. At this point, we realized the primary wasn't exactly where we expected it would be. Migs A-D looked OK, but E & F were out. Attempts to home the primary failed in axis E&F. Around this time, Russell called in and the crew started to diagnose further. Dan's notes follow: We had problems with the optics tonight. Both the secondary and primary were not in proper position and there were problems getting them back to nominal positions. The secondary was considerably out of position with the axial actuators a significantly different positions. Home and move commands were not completing properly. It took repeated home commands to get the secondary back to normal. The primary was found to be significantly off laterally. The migs showed it to be nearly a tenth of an inch from the nominal position. We soon found that the lateral links were never going off air during a move of the lateral actuators. The air pressure and solenoid were checked and found working. Toggling the bit 9 in the primary galil would indeed activate the solenoid ( SB9 and CB9 ). Russell Owen traced the fault to the mirror specific code for the primary galil. Somehow it had been left out or cleared during recent upgrades. This kept the galil from controlling the links during a lateral move. The result is that the links were always on air and the actuators could not change position reliably since they were pushing on the mirror through the links. Once the code was uploaded normal operation was restored and the actuators were homed and repositioned, and the primary relaxed. The images looked much better after this. A documentation note: Russell also told us to home the primary in E&F first, then put it back to the nominal position, then do D and put it back, then A,B,&C. This is not documented anywhere and no one present here today has ever done it this way in the past. If this is needed, we need to document it and spread the word. findRotator ----------- We tried to do a findRotator command in iop and first, found we had to specify a threshold which is not documented in the sense that it's needed, nor what a -threshold is. When we discovered through trial and error that -threshold is needed, we got the following error which Sir Lupton claims to have fixed and checked in: im> findRotator h5 16 -threshold 3000 nPeaks=146 at thresh= 3000 Continue? [y]: y Error: can't read "skyData": no such variable im> tb can't read "skyData": no such variable while executing "findTrans $peaks $lskip $skyData..." invoked from within "set data [findTrans $peaks $lskip $skyData..." (procedure "findRotator" line 43) invoked from within "findRotator h5 16 -threshold 3000" Once Robert got it working with the unknown -threshold parameter, we got a rotator error of -0.003 +- 0.007. A PR was issued. Flat Field Screens ------------------ At the beginning of the night, sometime after we moved the building off the telescope, while we were fiducializing (isn't English great?), the flat field screens opened by themselves. Twice in the night, during a drift, the petals closed by themselves and had to be reopened by hand. Hint: when your stars look like flowers, you are out of focus and the flat field screens are closed. When we started the science drift, the out of focus donuts had a bite taken out of them...ONE of the petals had spontaneously closed! This is a bad problem that must be fixed immediately. Azimuth aborts -------------- We had a strange problem with the azimuth brake coming on / azimuth amp aborting tonight ( see PR 876 ). It seemed that the abort was coming before the brake, however clearing the brake would temporarily clear the abort on the amplifier. The abort would return a short time later and the brake would come on. The amplifier status was seen both in the Menu and on the amplifiers themselves. The interlocks were in a curious state when this was happening. az_mtr_cw_perm az_mtr_ccw_perm wind_mtr_cw_perm wind_mtr_ccw_perm These were all red. However all the logic feeding them (anded together ) were all green. This didn't help track down the problem, but may provide some clue to what the problem isn't. We could not get this problem to go away, so we reverted to the "latest" version of the MCP ( v3_1_2 ). This seemed to solve the problem, but a short time later after a slew to the home position completed and I was beginning to set the fiducials, both altitude and azimuth stopped and the brakes came on. The windscreen remained active. After a few attempts at resetting the amps and clearing the brakes without success, I tried the following. Pushed in a stop button, waited a few seconds and released it. Reset the amplifiers( * ). - state did not show a change in mcp Took the monitor feature off the brakes( & ). Cleared the brakes. Did a stop then hold for each axis( s then h ). This worked and the amplifiers did not abort the rest of the night. I did put the monitoring feature back on for the altitude. It seems that there is some sort of unclearable state the amplifiers/mcp get in sometimes which may be some race condition which only stop/hold can stop. This is not acceptable since it means that there is no way to get the amps to clear with an AXIS INIT from the TCC. The above list of commands may well have worked using the current version of the mcp, but I did not want to take more time reloading it and resetting fiducials to find out. Note that this seems similar to the behavior noticed a few days ago ( PR 868 ). However I think the work tonight shows that this and possibly what was seen the other day does not have to do with an overheating amplifier, just some race condition in the interlock/mcp logic. DA -- We had several archiver problems during the night. The first with id6, the second with id4. Ron phoned in just as we noticed the id6 problem and helped to diagnose it. The professional's view was that id6-2 was dead in a bad way, so turn it off and keep going. id4-2 acted up and was also deactivated. After the primary mirror actuator problem was fixed, we decided to start fresh with the DA, so we paused & unloaded all the tapes and rebooted the crates, re-loaded and mounted new tapes. The old tapes were set aside for recycling. Since all previous data was bad, we went ahead and did a poolInit. Archiving on the science scan (run 1231) proceeded normally until 02:39:48, and then both id-1 and id-2 stopped archiving. The errors: archUseDrv is 0 (drive 1) archEnable is 0 archDrvSts (drive 1=disabled) We then did the following to id-1 to recover (it did not recover by itself): im> iccExec id1 archPause 1 {id1 {}} im> iccExec id1 archUnload {id1 {}} im> iccExec id1 archInit {id1 {}} im> iccExec id1 archLoad 1 2000 {id1 {}} im> iccExec id1 archReadSense 1 Waiting for command response from id1 (crate 1, ICC 0), archiver server (client 2) Waiting for command response from id1 (crate 1, ICC 0), archiver server (client 2) Error: {id1 {Request sense failure Failed to receive exabyte command completion status}} At this point, we decided to continue with id-1 not archiving, rather than stop the run and reboot the crates (too much time had already been lost tonight!). The data will have to be archived from hard disk during the day. *** Don't do a poolInit until you know the data has been archived! ***