Date: Tue, 21 May 2024 22:57:56 -0000 Subject: Re: [sloan-25mlog 7249] Re: 2.5m obslog 20240520 (SJD 60451) From: cscoresby@apo.nmsu.edu To clarify: I did not end up needing to reset the daq last night. By the time John showed up last night, BOSS was pinging the daq with an OK status. The weather was also poor---I was not taking exposures for about an hour due to clouds---so I can't say with certainty about when exactly communication was restored. My impression is that the commands Kaike and I sent trying to fix it *eventually* made its way through, though again, I can't say how much time it ended up needing. -Conner > I added Jose to the conversation for his comments. > > I think that restarting bossICC automatically reload controllers. We tried > to reloadControllers after the restarting the ICC last night due to > restarting didn’t establish daq communication. > > >> On May 21, 2024, at 12:57 PM, John wrote: >> >> When I was here around midnight, I had the impression that the DAQ had >> come back on it's own without a reset, but I did not ask specifically. >> Perhaps it had come back following an earlier reset. >> >> That it appears to have responded to the first controller reload but >> hung during readout should be a clue. This really strikes me as a >> process problem rather than anything else. >> >> My impression has always been that on startup or restart of the spectro >> the order of operations is to reload controllers and then do the iacks. >> I don't believe they are strictly required following a non-reset >> controller reload, but I always follow reloadControllers with both iacks >> to be on the safe side. I also will typically do a camcheck following >> the first iack - no point in doing the mechIack if the camcheck fails in >> some way. >> >> I believe that the DAQ is mostly ignorant of what happens past the >> netburner card, so I am a bit skeptical that a fiber connection problem >> on 192.168.1.x could cause it to hang up - BUT it is also suspicious >> that the exposure recover command has been repeatedly associated with a >> DAQ hang.... Apparently the DAQ sometimes doesn't like something it is >> told to do during exposure recovery. >> >> On 5/21/24 11:33 AM, Kaike via sloan-25mlog wrote: >> >>> Yes. I did watch outcome of commends. The first reloadControllers seem >>> worked, after reloadConreollers commands, boss ping command shows >>> sp1mech, sp1cam, and sp1daq were alive, boss exposure recover initiated >>> the readout process as expected, but hung the process, fellow up >>> reloadControllers did not establish daq communication, then bossICC was >>> restarted, iack, mechIack, and reloadController still could not get daq >>> communication back. Then Conner checked fiber and cable connection on >>> media converter, found nothing, tried boss iack and mechIack, and >>> reloadController again, that didn’t work. Just to be clear, Conner >>> did push the reset button before I logout. For further reference, So, >>> my question here is: was any action performed to get the daq >>> communication back or the communication restored itself without any >>> intervention? >>> >>> >>>> On May 21, 2024, at 11:02 AM, John wrote: >>>> >>>> Two things to remember about working with BOSS problems. >>>> >>>> First is that the hardware is very old. Issue commands with a very >>>> long pause between them - say 20 sec minimum. Ideally there is some >>>> feedback in the log window and you can watch to see that your command >>>> was completed before doing the next one. That is not always the case >>>> however, so pauses between commands are a good idea. >>>> >>>> Second is that there is no sure fire way to make this a simple >>>> sequence of steps that will always reliably provide the result you >>>> want. If the steps followed do not provide the desired result of a >>>> working DAQ, then you have to stop and think about what else you could >>>> try before going with the reset. In most cases, resets are not >>>> actually required - something else is out of sequence and a reset >>>> simply fixes the flawed manual sequence or a zombie process somewhere >>>> that is hanging everything up. When there is an unexplainable "fixed >>>> itself" result like last night, that tells me a reset was not really >>>> necessary. In some cases, a reset will not solve the problem either. >>>> When we had problems with condensation in data connectors pre-2020, >>>> resetting the DAQ and the command micro would never fix them. That >>>> was an actual physical problem rather than a communications problem, >>>> and resets did not help. >>>> >>>> While there is nothing actually wrong with going to reset early in the >>>> process, the trouble with forming that habit is that we never make any >>>> progress on learning what is actually causing the recurring problem so >>>> that we can fix it. It is also entirely reasonable to debate whether >>>> we wish to prevent problems like this or simply accept them and >>>> document and apply fixes that have worked in the past. Though I am >>>> not fond of the latter approach, given the age of the hardware and the >>>> minimal staffing we have to troubleshoot it, I can accept that if >>>> resets work reliably perhaps our time is best spent solving other >>>> problems. >>>> >>>> It is unfortunate for us that this hardware is so old, and there is >>>> little in the way of information on how it is supposed to work - what >>>> talks to what in which order. There are bits and pieces of info all >>>> over the place, as well as many generations of troubleshooting steps, >>>> but no overall explanation of the architectural design of the system >>>> and no outline of how that design was implemented. >>>> >>>> On 5/21/24 9:22 AM, Kaike via sloan-25mlog wrote: >>>>>> On May 21, 2024, at 5:39 AM, Conner via sloan-25mlog >>>>>> wrote: >>>>>> >>>>>> >>>>>> APACHE POINT OBSERVATORY SDSS 2.5M >>>>>> >>>>>> OBSERVING LOG >>>>>> Monday May 20, 2024 >>>>>> >>>>>> (SJD 60451) >>>>>> >>>>>> ---- Observing Team ---- >>>>>> >>>>>> Conner Scoresby >>>>>> Kaike Pan (support) >>>>>> John Downey (support) >>>>>> >>>>>> ---- Observing Plan ---- >>>>>> >>>>>> >>>>>> Mon 20 May 2024 60451 0.95 03:00 03:17 10:41 12:11 >>>>>> 19:36 --- bright ( 8.5 h, w/ 1.0 h for 8 deg design) >>>>>> >>>>>> 6 BOSS dithers >>>>>> >>>>>> ---- Observing Summary ---- >>>>>> >>>>>> Cloudy night with subpar seeing. About 2 hours at the end of the >>>>>> first half were lost to clouds, though we never closed. >>>>>> >>>>>> Notes: >>>>>> >>>>>> 1. The sp1daq lost connection at around 0:400Z A reloadControllers >>>>>> and a restart on bossICC did not immediately restore connection to >>>>>> sp1daq. The connection was eventually restored; the loss of the daq >>>>>> lined up well with the cloud cover, so the loss of time could have >>>>>> been worse. Best way is to do a boss reloadControllers, restart >>>>>> bossICC, reload controllers again, and eventually the issue should >>>>>> resolve itself. Missing pictures of the daq reset button were taken >>>>>> and added to the BOSS Troubleshooting wiki page. >>>>>> >>>>> Hello Conner, >>>>> Did you do everything later to restore the daq communication? Such >>>>> as push daq reset button.. >>>>> >>>>> If I recall correctly, >>>>> Together we did: boss reloadControllers, boss ping (had three ok), >>>>> boss exposure recover. Boss readout started but hung at the end of >>>>> readout, at this point, another check found daq is not connected, >>>>> then we restarted bossICC, reloadControllers, boss iack, boss >>>>> mechIack, daq still was not connected after this sequence. Then you >>>>> checked fiber and cables connected to media converter, found >>>>> nothing. I tried reloadControllers anyway, nothing happened. So, >>>>> either what you did re-established the connection or the issue >>>>> resolved itself later on. >>>>> >>>>> Thanks, >>>>> >>>>> -Kaike >>>>> >>>>> >>>>> >>>>> >>>>> ____________________________________________________________ >>>>> Logs of Nightly APO Activities >>>>> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html >>> ____________________________________________________________ >>> Logs of Nightly APO Activities >>> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html > > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html >