Date: Thu, 9 Nov 2006 12:46:54 -0600 From: Dan Long Subject: Re: 2.5m obslog 20061108 (MJD 54048) Hi Kurt, I'm pretty sure the switch I power cycled only controls the photo 1 crate. There is another switch on the enclosure for the pool disks, and at least its cooling fans remained on during the time the photo 1 crate was powered down. Another item that I didn't really make clear in the log was that the first attempt at rebooting id1 was a ctrl-x via dsclat to id1, thus only rebooting that one node. The boot stopped in the same place that the subsequent boot of the whole crate via the reset button on the card. I didn't however try power cycling the pool disks first. I don't know if it is possible to power cycle an individual disk, or if one would have to do id1 and id2's disks together (id3's scsi cable goes to a different enclosure and may indeed be on a separate power switch). If other nodes are still active, is there a problem cycling the power on those pool disks? Cheers, Dan >Dan, >Unfortunately, it is not clear why id1 hung in the first place, but >I have a guess as to why its reboot failed initially. I suspect >that the hang occurred around the same time as a SCSI read or write >and this confused one or more elements in the link between the 5500 >and the SCSI disk. > >At the teststand, I often see the symptom that you describe unless I >am careful to only reboot each ICC when there is no SCSI transfer >happening (symptom = reboot stalls when trying to configure SCSI). >The way that I have found to recover from such a situation is to >power cycle the pool disk(s) and then reboot the ICC. In the log, >you mentioned that you power-cycled the photo1 crate. Did this >include the pool disks connected to the ICCs in that crate? If not, >then something else must be going on. >Kurt > >long@apo.nmsu.edu wrote: > >> APACHE POINT OBSERVATORY SDSS 2.5M >> >> OBSERVING LOG >> Wednesday November 8, 2006 >> >> (MJD 54048) >> >>---=== OBSERVING TEAM ===--- >> >> >> >>Swing: Shannon Watters >> >>Night: Dan Long >> >>Also Wik: Robert Lupton >> >> >>---=== OBSERVING SUMMARY ===--- >> >> The reason for the split into two runs was that id1 hung. >> Unlike last night, the node was still alive, and all was restored >> without card swapping, though it did take a power cycle of photo 1. >> There are some hints that there may be a SCSI communications >> problem at play. See log and problems section for details. >> >> >>---=== OBSERVING LOG ===--- >> >>Night: >>------ >> >>08:20Z id1 died frame 719, lamda -145. A couple reboot attempts >>only got as far as "Configuring SCSI for Pool Disks, ID 1". >>Output of >> "i" and boot sequence before hang in problem section. >> >> Shut off power to photo 1 crate and checked SCSI cable connections >> between ID1 and pool disk. All looked okay although the connector >> at the disk end isn't mounted to the enclosure. Powered up >>photo 1 and this time everything came up, including id1. >> >>---=== PROBLEMS IN DETAIL ===--- >> >> >> State of id1 after hang. >> >>sdssid1> i >> >> NAME ENTRY TID PRI STATUS PC SP >>ERRNO DELAY >>---------- ------------ -------- --- ---------- -------- -------- >>------- ----- >>tExcTask excTask 1dffcda0 0 PEND 220024 1dffcc80 >>0 0 >>tLogTask logTask >>1dffa200 0 PEND 220024 1dffa0f0 0 0 >>tShell shell 1d3e7520 1 READY 21931c 1d3e7100 >>8800dd 0 >>tWdbTask wdbTask 1dd36700 3 PEND 2139c0 1dd365d0 >>0 0 >>tTimerTask timerTask 1dd2e790 3 PEND 220024 1dd2e660 >>3d0002 0 >>tcpLnkSrv tcpLnkSrv 1d3ffde0 3 PEND 2139c0 1d3ffa70 >>3d0002 0 >>dIntern inDataTsk 1c5f2120 3 PEND 220024 1c5f1e10 >>0 0 >>ntpd ntpd 1db1d2b0 4 PEND+T 2139c0 1db1cf50 >>3d0002 7 >>tScsiTask scsiMgr 1dff7bb0 5 PEND 2139c0 1dff7aa0 >>0 0 >>dAssem assemTask 1c5ec390 6 PEND 220024 1c5e9a80 >>3d0002 0 >>ntpTZenv usrTZenv 1da39a10 8 PEND 2139c0 1da39940 >>0 0 >>dLine lineTask 1d2a4e80 8 DELAY 218c40 1d28f830 >>3d0002 1 >>vciWatch1 vciWatchDog 1d2736e0 10 DELAY 218c40 1d2733a0 >>0 2 >>dArch archTsk 1bc0a930 10 PEND+T 220024 1bc0a4a0 >>3d0004 955 >>dSCD scdTask 1d31cba0 12 PEND+T 220024 1d31c8d0 >>3d0004 859200 >>dCLI cliTask 1db8dcf0 16 PEND 220024 1db8db00 >>0 0 >>dTclShr cliTask 1d7c24f0 16 PEND 220024 1d7c2300 >>3d0002 0 >>tTelnetd telnetd 1dd39e40 18 PEND 2139c0 1dd39cd0 >>0 0 >>tNetTask netTask 1ddc87c0 50 PEND 2139c0 1ddc86d0 >>0 0 >>dTCClisten tccListen 1d3b1360 52 PEND+T 2139c0 1d3b0db0 >>3d0002 5978 >>tPortmapd portmapd 1dd3c9c0 54 PEND 2139c0 1dd3c760 >>3d0002 0 >>dCtrlSrv clntSvTsk 1d317b60 60 PEND 2139c0 1d317950 >>0 0 >>dArchSrv clntSvTsk 1d2cf9f0 62 PEND 2139c0 1d2cf7e0 >>0 0 >>dDataSrv clntSvTsk 1d2f5db0 64 PEND 220024 1d2f3250 >>0 0 >>tMurRouter mur_route_st 1dc2e430 70 PEND 220024 1dc2de50 >>20 0 >>dAnal frameTask 1bc058f0 74 PEND 220024 1bc05590 >>22 0 >>dIccMon monTask 1d3e24e0 80 DELAY 218c40 1d3df280 >>3d0002 2761 >>monBackup monTask 1d3c9c20 80 DELAY 218c40 1d3c69c0 >>0 2752 >>tTraceDump trcDmpTsk 1dc293f0 150 PEND 220024 1dc29220 >>0 0 >>dIdle idle 1dc26ac0 200 READY 1dd14108 1dc26a50 >>0 0 >>value = 0 = 0x0 >>sdssid1> >> >>sdssid1> >>This is as far as the boot got before jiggling SCSI cables and >>power cycling photo 1. >> >> >>Press any key to stop auto-boot... >>0 >>auto-booting... >> >> >>boot device : gei >>unit number : 0 processor number : 0 host name >>: sdsshost2 >>file name : /tftpboot/mv5500/vxWorks >>inet on ethernet (e) : 10.25.1.111:ffffff00 >>host inet (h) : 10.25.1.3 >>gateway inet (g) : 10.25.1.3 >>user (u) : vxworks >>ftp password (pw) : daveberg >>flags (f) : 0x8 target name (tn) : sdssid1 >>startup script (s) : /linuxp/astrobase/node/sdssid1/sdssid1.s >> >>Attached TCP/IP interface to gei0. >>Attaching network interface lo0... done. >>Loading... 1567104 >>Starting at 0x100000... >> >>Configuring SCSI for Pool Disks, ID 1 >> >> >>DSS DSS DSS DSS DSS Digital Sky Survey DSS DSS DSS DSS DSS DSS >>DSS *** The information posted to SDSS mailing lists is proprietary *** >>DSS This is message 3244 in the sdss-25mlog archive, URL >>DSS http://www.astro.princeton.edu:81/sdss-25mlog/msg.3244.html >>DSS The index is at http://www.astro.princeton.edu:81/sdss-25mlog/INDEX.html >>DSS To join/leave the list, go to >>DSS >>http://www.astro.princeton.edu:81/cgi-bin/mailinglists.pl/show_subscription?list=sdss-25mlog >>DSS To post a message, mail it to sdss-25mlog@astro.princeton.edu >>DSS >>DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS >>