Date: Mon, 14 Feb 2000 03:35:20 -0700 (MST) From: Ellyne Kinney Subject: 2.5m obslog 000213 (MJD 51588) SATURDAY FEBRUARY 13, 2000 (MJD 51588) APACHE POINT OBSERVATORY SDSS 2.5m OBSERVING LOG Observing team: Ellyne Kinney (APO) Stephanie Snedden (APO) Bing Chen (PT) (JHU) Craig Loomis - by phone (APO) Visitors: Steve Kent (FNAL) John Peoples (FNAL) Alan Uomoto (JHU) The plan: Spectroscopy for the duration of the night. We will check plate 317 which is allegedly drilled for an HA of +3H at an HA of -3H, just to see if it is indeed drilled for the HA we think. We may also do focal plane mapping with a FASTT field plate drilled for HA=0. We had the telescope ready to go and out of the enclosure by 18:00 (all check-outs and fiducials set). There was a problem moving the enclosure --haven't seen this for a while. At 17:30 the enclosure failed to move and the control panel gave an "OL" (overload) error. Fortunately, the manual was in its place inside the control panel door, and we followed the instructions, issuing a "CTRL WRT". The first time, this did not work, so we issued the command again and waited a few minutes. This time the enclosure opened with no problems. The temperature at this time was +2C. In the meantime, we started a 'goDrift -bias' for the pipeline folks. We loaded and mounted tapes, then started a 'goDrift -bias'. id1 did not start archiving data. We got the following error message: "sdssid1 dArch dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 1" "sdssid1 dArch dsc_E_trcErr dscArch_CheckPrevOp: rstatus = 0x8001421c, drive 2" So tonight, it was id1 acting up. We decided to try what worked for us last night. We archPaused the tapes, we unloaded them, rebooted both photo crates and the astrom crate, reloaded the tapes and did an 'initNight -tapeLoad'. As with last night, it worked. We started a 'goDrift -bias' and were able to archive to all nodes. Later, id1 stopped archiving again in the middle of a bias drift. When Steph started SOP, she first did a 'startMonitor', then realized she wanted a 'startMonitor -simple', so she quit saoImage from its window then issued the 'startMonitor -simple' command. No new saoImage, but no error message either. There were no saoImage processes running. Re-starting SOP fixed the problem, but this seems to be a bug to be fixed. (PR filed). When Steph went down to fill the spectrograph dewars, after SOP had been started she saw that the Hg arc lamps were on (all of them). No command to turn them on had been issued. A 'calibLamps hgcd off' turned them off, but we are at a loss to explain why they came on. We had followed the SOP cheat sheet through "test health of the spectrographs" with no glitches. We don't know when the lamps could have come on. (PR filed). While waiting for the first spectroscopy field, we did a powered drift on the equator (lambda=-106) and took biases, so Steve could check that the declination headers were reading correctly. Every one out of five (approx) was reading a non-zero declination. The correct positions were being broadcast, however. At about 22:00 we strted getting more error messages on id1: "dsc E trcErr dscArch CheckPrevOp rstatus = 0x8001421c, drive 1 (dsc_E_trcErr dsc_Arch_CheckPrevOp: rstatus=0x8001421c, drive1 : Feb 13 22:03:44 MST" "dsc E trcErr dscArch CheckPrevOp rstatus = 0x8001421c, drive 2 (dsc_E_trcErr dsc_Arch_CheckPrevOp: rstatus=0x8001421c, drive2 : Feb 13 22:03:44 MST" "dsc E archStsFail Failed to recieve exabyte command completion status (dsc_E_archStsFail Failed to recieve exabyte command completion status: Feb 13 22:03:44 MST" We fo1lowed Don Holmgrem troubleshooting guide for the tape drives starting under the heading "Handling a non-responsive archiver" We issued an 'archPause', 'archRewind', 'archReset', and an 'archInit'. The archiver did not go to a 'archState' of "Initializing". After a couple of minutes we got the following error mesage: "Feb 13 22:09:53 sdssid1 dLine dsc_E_archNotInited Archiver is not initialized" We then ended the drift and let the other archivers finish archiving, paused all the archivers, unloaded the tapes and rebooted photo1. Once this was done, we remounted and resumed archiving. Id1 was now working. At 21:15 we started on cartridge 5, plate 317. We centered and focussed on an fk5 star. The target stars were very faint even with a 60 sec exposure. The images used for calculating fiber rotation angles are 104-108. The 'starPattern' compute routine had difficulty centroiding, the counts were so low. When I tried a guiderFocus, I got an error: Error: list element in quotes followed by "Estimated" instead of space ftclGetUsage: Formal parameter list is invalid - must have 5 entries This appears to be a bug in the latest SOP version. Steve found the bug to be a pair of double quotes rather than single quotes in the tcl script. We then setup an older version (3_8_3) of SOP and took an image. The guider camera then appeared to stop reading out properly. See images 109-112. The images have an appearance of a readout or clocking problem. We powercycled the guider camera and rebooted the guider mac. We attempted a 'gcam doread' from both the TCC and the SOP interface. Then, any attempts to 'set inst' or talk to the camera from the TCC would cause the TCC session to lock up. A 'grabInstruments -drop' command let us at least talk to the TCC. We called Craig. We were unable to host to tcc_gcam. We logged out the port, shutdown the Guider Mac, power cycled guide camera electronics, restarted the Mac. When we set the instrument block in the TCC we got the following error on the Guider Mac: guide camera error: ppk error 86 We looked error 86 up in the Photometrics documentation and found that this stood for "SCSI protocol breakdown: no device or termination" Craig was also concerned that the TCC lat_node may have been corrupted. We shut down the guider mac, power cycled the a to s board, restarted the mac, did a 'telstop' in the TCC, hosted to tcc_gcam and got an OK prompt. We restarted 'telrun', and 'set inst=spectro'. A 'gcam doread' didn't write any image. So we tried the whole process again of shutting down the guider mac, powercycling the a-to-s board etc., This time the error message was: guide camera error: ppk error 141 Which is a "subsystem fault: CCS chip or memory". We tried the power cycle and telstop/telrun procedure again. We could not reproduce the subsytem fault error state. We got not error message the third time,. but we still can't communicate with the guider camera. The bad images have been saved in /usrdevel/ekinney/guider_badness. At 00:30 we gave up on the guider camera, and stowed the telescope. Photometrics will have to be contacted. We ran 'endNight' on the data collected tonight. We ran into trouble archiving the data on id1. As far as we can tell there are about 300 frame files in the id1 poolDir that are written to tape, but not marked as archived. frameCheck was run twice for these data and the archQDepth never changed from 300, and files that endNight reported as being archived were never marked as such in the poolDir. After the second attempt to get frameCheck to run we ran 'endNight 51588 -parseMurmur', 'endNight 51588 -framelogFix', 'endNight 51588 -shiptape', 'endNight 51588 -ioptape', and 'endNight 51588 -copy'. endNight finished at 3:30 am. ********************* Mirrors Primary: prim stat 0 2 I Received=" 1, 1, 1, 1, 1, 1 axis homed" 0 2 I Received=" 000004850, -000002350, 000000100, -000009500, 000000050, -000 000100 commanded position" 0 2 I Received=" 000004850, -000002350, 000000100, -000009500, 000000050, -000 000100 actual position" setmir values: PrimDesOrient=0.00, -3.90, 3.90, 213.50, 213.50; PrimConstRMS=0.00 Secondary: focus: SecFocus=1500.00 sec stat 0 2 I Received=" 1, 1, 1, 1, 1 axis homed" 0 2 I Received=" 001712850, 001700000, 001689700, -000006200, -000010800 comma nded position" 0 2 I Received=" 001712959, 001700216, 001689620, -000006200, -000010800 actua l position" setmir values: SecDesOrient=1350.00, 5.00, -5.00, 50.00, 69.97; SecConstRMS=0.00 ******************** Software: Cut v3_8_4 of IOP/SOP. Reverted to v3_8_3 of SOP during operations. LN2: The spectrograph dewars were filled at 19:00. They were topped off at 01:00. Tapes: JL1829 JL1868 JL1869 JL1870 JL1871 JL1872 JL1873 Interlocks: No interlocks were by-passed. Telescope: The telescope still has cartridge 5 mounted, plate 317. The counterweights are at 300. Weather: Skies were variably cloudy, with mostly cirrus. Winds were from 10-15mph out of the WNW. Dust particles spiked at 120 particles/ft^3, but for most of the night were lower than 100 particles/ft^3. Relative humidity climbed from 50%-75%. DIMM seeing dropped to 1 arcsec, with periods of subarcsec seeing at around 20:00, then bounced around between 1 and 2 arcsec.