Date: Sat, 12 Feb 2000 07:47:12 -0700 From: Dan Long Subject: obslog 000211 (MJD 51586) FRIDAY, FEBRUARY 11, 2000 APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG (MJD 51586) Observing team: Jeff Hayes (APO) -- first shift Scot Kleinman (APO) -- second shift Atsuko Kleiman (APO) -- second shift Dan Long (APO) -- third shift Bing Chen (JHU) -- PT Visitors: Steve Kent (FNAL) Plan for tonight: Spectroscopy for the first half (FASTT field and focal plane mapping), and if weather permits, imaging stripe 9 and FASTT fields during the second half. Should the weather be bad in the second half, we continue with spectroscopy. Note that any off equatorial scans should have lskips in them, science, engineering or whatever. In summary the first half of the night was lost due to DA/host problems, so no spectroscopy was done. The second half was much smoother. We got a three-and-a-half hour science scan of stripe 9 N in reasonable seeing and clear skies. We also scanned FASTT field K at the end of the night. Cartridge ,Plate 316 was mounted in the afternoon so the FASTT field could be obtained at high airmass. Also during the afternoon the guider was focused. This should resolve the issue of 3" fwhm in 1" seeing. Fiducials were set around 18:20, and the building opened around 19:00 after some trouble getting the telescope to move in altitude below 15 degrees. See PROBLEMS IN DETAIL section for more details. Also around 19:00 an imager bias drift was started. There were problems with ad1 not seeing udp packets. It was traced by Craig to a bad ethernet connection. At 21:30 the photo 1 crate died and upon rebooting took down sdsshost. All crates were rebooted when host came back up since VME communications were not possible from iop. Other fallout was that the guider Mac had to be rebooted, and that all of the telnet connections to the instruments had hung and needed to be logged out. Before this small crisis, we were on FASTT field 316 at 21:00 and had acquired 4 guide stars down the fibers. After recovering from the host crash the guider reacquired the target, but the spectrographs were not talking. By the time things were back up and running, the plate was no longer at high airmass and the moon was nearly down. We decided to switch to imaging. We had problems in getting primary instrument latch 1 to latch up the camera. And, the other two would latch only after holding the button down for a minute or more. The rotator was at -16 minutes. The lift was at 1600 lb force. We tried putting the protection bolts in both before and after latching and it didn't matter. On the third attempt after backing the imager off and trying again, primary latch 1 fired after about a minute of holding the button. ( PR 822 ). Imaging Runs: ------------- 1137: Junk bias. 1138: Real bias of about 30 frames. 1139: Setup for stripe 9 N 1140: Science run stripe 9 N -35. About 370 frames long. Clear skies. Seeing 1.0" - 1.6" until around frame 280. 1.5" to 2.1" 280 to 370. 1141: Setup for FASTT field K. 1142: FASTT field K in less than ideal seeing ( 1.5" to 1.8" ), but clear. Spectroscopy: ------------- None tonight. Rearranging to avoid the moon, we would like cartridge 4 plate 266 unplugged and replaced with plate 289. EndNight: Tapes Post Crash Pre Crash 1 2 1 2 ID1 JL1854 JL1862 JL1793 JL1813 ID2 JL1855 JL1863 JL1794 JL1814 ID3 JL1858 JL1864 JL1801 JL1848 ID4 JL1859 JL1865 JL1802 JL1849 ID5 JL1860 JL1866 JL1803 JL1850 ID6 JL1861 JL1867 JL1804 JL1851 Gang JL1827 JL1828 Another consequence of the sdsshost crash this evening is that the tapes in at the start of the night were swapped since the crash happened while they were writing. They only have bias frames on them and may be corrupted. Also endNight claims there is a third set to go out, and I have no clue where it got the label numbers for them. JL1836 through JL1841 are _NOT_ part of 51586. They actually belong to 51584. Endnight -doImager completed successfully. The column 1 tapes are on the shelf ready to be shipped. MIRROR NUMBERS Primary: setmir: 0 -3.9 3.9 213.5 213.5 galil: 4850 -2350 100 -9500 50 -100 linear gauges 1 = Axial A = 1.9315 2 = Axial B = 0.5570 3 = Axial C = 0.5570 4 = Transverse = 0.4075 5 = East Lateral = 1.8990 6 = West Lateral = 0.3990 Secondary: ( inst=imager focus=-1680 alt=53.7 deg T=2.0 C) setmir: -422 5 -5 50 -34.2 galil: -519734 -532452 -542707 -1550 -6350 linear gauges 1 = Axial A = 0.3660 2 = Axial B = 0.3920 3 = Axial C = 0.3920 4 = Transverse = 0.0120 STATUS Software: current development version of sop watcher v2_6_1 iop v3_8_3 mcp v3_1_1 No interlocks were/are bypassed tonight. Spectro dewars were filled around 19:10, and again after the telescope was put away at 06:30. The imager dewar was replaced at 06:30 as well. The telescope currently has the imager mounted, CW setting 720. WEATHER Skies tonight were clear all night. The temperature was in the mid 30s. Winds were westerly at 10 to 25 MPH, humidity was around 30%, and dust counts were around 400. Seeing tonight was between 1.0 and 3.0 arcseconds on DIMM. PROBLEMS IN DETAIL ALT AXIS HALT: We had problems with the TCC and the MCP at this point. Using the TCC we commanded the telescope to a position where we could open the building via: track 121,6 mount /rotang=0 /rottype=mount We started to slew, but at ALT=15.28 we received a "slew end" message from the TCC. An axis init and a reissue of the above command did not result in any telescope motion. We exited the TCC and went down to the telescope to see if we could control the motions with the MCP: no luck, as every time we tried to clear the brake, and reset the amps, we would have the brake come back on. We checked the 15 degree stop, and it seemed to be behaving fine. We then exited the monitor programme for the MCP on ALT, and tried to get any response out of the MCP (i.e. clearing the brake). Still no success. We then used the manual paddle to bring the telescope down to ALT=6 degrees so that the building could be rolled off. Once this was done, we moved the telescope up (again, via the manual) to ALT=30 degrees. (NB: during all this time the AZ=121, and ROT=0.) We then switched over to the MCP again. We re-enabled the ALT monitor and tried to reset the amps: no success as the axis would abort and the brake came back on. Same for the other axes as well. We then tried an "axis init" in the TCC. This time all 3 axes reported ready in the MCP (i.e. 3 *** in the appropriate fields). We exited the MCP Menu and were (in theory) on the sky by 19:30. AD1 PROBLEMS: Around 19:00, we attempted to start an imager bias, but found ad1 problems. The Watcher error was no udp packets; the murmur error was no responses from heartbeat requests. We then went to the computer room and saw ad1's ethernet port on the switch showed no activity. Craig fiddled with some cables, and the connection came back. We also diagnosed this by doing a im>dsclat ad1 within iop and watching ad1 attempt to boot. It failed almost immediately trying to load it's OS. We verifed this by looking in the /var/adm/SYSLOG file on 'host and seeing connections from ad2,ad3, and ad4, but not ad1. We rebooted the astro crates and then ad2 had trouble. We rebooted again and ad2 was fine, but ad1, ad3, and ad4 were not, etc. Eventually, after a few more reboots, everything came back up properly. We rebooted the photo crates as well, just to make sure all were happy. The loose connection for ad1 may have been real, but the various different responses during several reboots are definitely disconcerting. PHOTO 1 AND SDSSHOST CRASH: At 21:30 we had a problem in getting id1, id2 and id3 to respond. Observation showed us that the photo 1 crate was dead (i.e. all the FOXI lights were red). We then proceeded to reboot the crate. Almost immediately, sdsshost crashed. We lost the ability to talk to the imager, as iop disappeared. Steve Kent tried an "iccExec readoutStop" without success, but the "iccExec readoutAbort" did stop the goStare. sdsshost rebooted successfully after about 10 minutes. At this point, a call to Craig was put in to notify him of the problem. Hayes then tried the following commands for both the current iop (v3_8_2) and the most recent (v_3_8_3), both of which produced the following: sdsshost:hayes> setup iop sdsshost:hayes> iop Switching umask from 02 to 002 dscRun using system v14_11 Warning: murmur server not running Executing commands in /p/dervish/v7_9/etc/dervishStartup.tcl: Executing commands in /p/astrotools/v4_12/etc/astrotoolsStartup.tcl astroda OAC version v14_11 Using HVE link to ICC Error mapping shared data area (memory) mmap failed for /dev/mmem for offset 0xB8000000 (0x00100000 bytes) MURMUR MESSAGE 22:00:00 Fri Feb 11 Arg# 1 (string): " (memory)" Appname: "v14_11 2221 OAC" -- MURMUR CLIENT STATUS : "Can't CONNECT to MURMUR server" -- Occurred when trying to report message : Errno message : MURMUR MESSAGE 22:00:00 Fri Feb 11 Arg# 1 (string): " /dev/mmem" Arg# 2 (ulong): 3087007744 Arg# 3 (ulong): 1048576 Appname: "v14_11 2221 OAC" MURMUR MESSAGE 22:00:00 Fri Feb 11 Arg# 1 (ulong): 2221 ********************************************************************** ********** * * * Backplane communications with the ICC are not possible. * * * * o Check connections between the machines. ** o Reboot (reset) the ICC. * * o Restart this (OAC) program. * * * ********************************************************************** ********** MURMUR MESSAGE 22:00:00 Fri Feb 11 Arg# 1 (ulong): 2221 We then rebooted all the crates (photo 1, photo 2, astro and spectro) and did a "ptvmetest". This tests were successful. All the monitors responded with the pleasent azure colour we have now come to expect. Atsuko reloaded the tapes