Subject: Re: 2.5m obslog 000420 (MJD 51655) Date: Fri, 21 Apr 2000 14:53:24 -0500 From: Ronald Rechenmacher long@apo.nmsu.edu said: > goDrift and the DA? > ------------------- > Once we got the camera back to life and happy, we tried a new goDrift. > It never started because it couldn't communicate with id2. We ctrl-c'd > iop and it then got hung up on id3. A new ctrl-C gave us back the > prompt. So we rebooted crates id1-3 and remounted the tapes and did a > new goDrift. This time it complained about id5 and id6. Ditto for the > reboot and a new goDrift which then complained about (you guessed it) > ad2-4. We rebooted those and got a beginData error in iop with ad4 > reporting Error with P3. A call to Ron and we saw that _inDataTsk was > suspended (weird!) and dLine was suspended (the real problem). Ron > couldn't figure out why, but will be working on it. A fresh reboot of > the astro crated and goDrift worked. Go figure. I did and will do some more ("Go figure"-ing :) First, the "couldn't communicate with id2 ... ad2-4" has got me baffled so far. A filtered portion of the murmur log: Apr 20 19:00:57 sdssid1 dArchSrv TEXTONLY archCmd: archResume Apr 20 19:00:57 sdssid2 dArchSrv TEXTONLY archCmd: archResume Apr 20 19:00:57 sdssid3 dArchSrv TEXTONLY archCmd: archResume Apr 20 19:00:58 sdssid4 dArchSrv TEXTONLY archCmd: archResume Apr 20 19:00:58 sdssid5 dArchSrv TEXTONLY archCmd: archResume Apr 20 19:00:58 sdssid6 dArchSrv TEXTONLY archCmd: archResume Apr 20 19:01:54 sdsshost IDA 136201 TEXTONLY Murmur Heartbeat to ... Apr 20 19:01:54 sdssid1 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssid4 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssid5 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssid6 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssad1 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssad2 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssad3 dDataSrv dsc_S_success Normal successful... Apr 20 19:01:54 sdssad4 dDataSrv dsc_S_success Normal successful... Apr 20 19:02:17 sdsshost IDA 136201 TEXTONLY Waiting for...response from id2 The "Heartbeat" command is accomplished using and "iccExec -async" command. With "-async" option, the command is issued to all the specified ICCs then serially, the responses are checked. The log shows that both id2 and id3 did not respond. This is strange. The PT link appears to be working, but somehow both id2 and 3 had problem. The problem appears to be with the PT link communications because it does not appear that the other possible cause (a crash of the boards) happened (because there was no sign of a "dump" file for either). This particular high-level Heartbeat command started a command on all 10 ICC nodes and because the responses are checked serially (in this case id1 -> id6, ad1 -> ad4), only id1 completed (completely completed :). The (appropriate) Ctrl-C left all other ICC nodes in a "non-idle" state. There is an "expert" commands to clear the nodes. In this case, ICCs in all the crates needed to be cleared. We have seen these "waiting for command response" problems before. In this case, it appears as if id2/3 did not fully receive the command. The frequency of this problem is low and therefore it is a bit difficult. After I clean up some more of the less difficult things I will look more closely at the code to try to determine where the problem is. As for the next error with ad4 (I do not know where "Error with P3" comes from), I just finished doing some work on this. It appears that our VME crates have a fundamental single transfer problem. The MVME167s can get bus errors when 3 or 4 of them are doing multiple single transfers simultaneously. I tested all crates transferring to the VCIs and to the Vigra cards; errors occur with both. It actually smells of a classic bus grant catch and pass problem but it's hard to say at this point. This situation doesn't happen that often during data taking as most transfers are block transfers. Currently when the VME nodes boot up, a data pattern is written across VME using single transfers. Apparently there is a significant probability (especially in the astro crate) that a VME bus error will occur. I have tested using block transfers for this initialization and it appears to solve the problem. I will cut a new version of the DA today. This problem more than likely explains the occasional failure of a node to come up after a reboot. --Ron