From: =?UTF-8?B?Sm9zw6kgU8OhbmNoZXotR2FsbGVnbw==?= Date: Thu, 5 May 2022 10:46:33 -0700 Subject: Re: [sloan-25mlog 6052] Re: 2.5m obslog 20220504 (SJD 59704) --000000000000aa509e05de47527f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Dan, I agree this probably requires more discussion. We added the closing the shutter in the cleanup stage recently because not doing it was causing persistence in APOGEE due to the BOSS calibrations. There are a few things we can do: - Close the shutters at the end of a successful hal expose command, but not if it failed. Close all the shutters in the prepare stage of the goto-field macro. This won't prevent APOGEE persistence if somebody take= s BOSS cals manually. - Got back to never closing the shutters at the end of a hal expose, but close them at the beginning of hal goto-field. - Make STUI/a future widget in Boson open shutters on manual exposures with some reasonable logic (e.g., biases close certain shutters, object opens all shutters). There's probably no ideal solution that doesn't require having to pay some attention to the shutters when doing things manually, though. But ultimately we really need to understand the problem with the controller, and I agree this is looking more and more as a networking issue= . Thanks, Jos=C3=A9 On May 5, 2022 at 10:24:45, Daniel Oravetz wrote: > - at 03:26:16 we found that the second APOGEE dither exposure did not > > start automatically. After noticing it we ran it manually, but on the 9th > > read noticed that the cold shutter got closed. We opened it manually. Not= e > > also that in the fits headers that exposure, 41420015, has dither 'B'. In > > reality, and sky line analysis shows it, it was taken in the A. See more > > information in the 'Log' section. > > > There was an APOGEE "Controller is busy" error which is why the exposure > didn't start (and perhaps the commanded dither move preceeding the > exposure never completed also a result?): > > 2022-05-05 03:26:16,162Z command: Controller is busy..." > > We've been seeing these error messages more often lately (could it > indicate a general network problem down at the telescope when coupled wit= h > the problem with the weather status communication issues we've been > experiencing recently, too?). > > Looks like we've had this error four times within the last week: > > 2022-04-29T07:12:00.log:2022-04-30 03:18:53,909Z errorMessage=3D"cannot execute command: Controller is busy..." > 2022-05-02T07:12:00.log:2022-05-03 02:36:01,338Z errorMessage=3D"cannot execute command: Controller is busy..." > 2022-05-03T07:12:00.log:2022-05-03 23:58:25,042Z errorMessage=3D"cannot execute command: Controller is busy..." > 2022-05-04T07:12:00.log:2022-05-05 03:26:16,162Z errorMessage=3D"cannot execute command: Controller is busy..." > > When HAL sees that failure it runs its cleanup tasks: > > 2022-05-05 03:26:16,635Z > Part of the cleanup tasks for HAL's 'expose' macro is to close the APOGEE > shutter: > > 2022-05-05 03:26:21,639Z > client.v3061 9215 shutter close > > When starting the next APOGEE exposure manually, that explains why the > shutter was closed at least. > > We might consider filing a ticket to re-evaluate the HAL 'expose' cleanup > tasks since we've had that 'shutter close' one bite us before when an > exposure was in progress, too. There's probably a good reason for it, but > might be worth asking about it. > > Dan > > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html > --000000000000aa509e05de47527f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Dan,I agree this probably requires more discussion.= We added the closing the shutter in the cleanup stage recently because not= doing it was causing persistence in APOGEE due to the BOSS calibrations. T= here are a few things we can do:Close the shutters at the end of a successful hal expose comm= and, but not if it failed. Close all the shutters in the prepare stage of t= he goto-field macro. This won't prevent APOGEE persistence if somebody = takes BOSS cals manually.Got back to never closing the shutters at= the end of a hal expose, but close them at the beginning of hal goto-field= .Make STUI/a future widget in Boson open shutters on manual exposu= res with some reasonable logic (e.g., biases close certain shutters, object= opens all shutters).There's = probably no ideal solution that doesn't require having to pay some atte= ntion to the shutters when doing things manually, though.But ultimately we really need to understand= the problem with the controller, and I agree this is looking more and more= as a networking issue.Thanks,= Jos=C3=A9 On May 5, 2022 at 10:24:45, D= aniel Oravetz wrote: =20 - at 03:26:16 we found that the second APOGE= E dither exposure did not start = automatically. After noticing it we ran it manually, but on the 9th read noticed that the cold shutter got = closed. We opened it manually. Note also that in the fits headers that exposure, 41420015, has dither '= B'. In reality, and sky line= analysis shows it, it was taken in the A. See more information in the 'Log' section.There was an APOGEE "Controller is busy" error which is wh= y the exposuredidn't start (and perhaps the commanded dither move p= receeding theexposure never completed also a result?):2022-05-0= 5 03:26:16,162Z < =C2=A00 9214 =C2=A0f errorMessage=3D"cannot execu= tecommand: Controller is busy..."We've been seeing the= se error messages more often lately (could itindicate a general network= problem down at the telescope when coupled withthe problem with the we= ather status communication issues we've beenexperiencing recently, = too?).Looks like we've had this error four times within the las= t week:2022-04-29T07:12:00.log:2022-04-30 03:18:53,909Z < =C2=A0= 0 8621 =C2=A0ferrorMessage=3D"cannot execute command: Controller i= s busy..."2022-05-02T07:12:00.log:2022-05-03 02:36:01,338Z < = =C2=A00 8933 =C2=A0ferrorMessage=3D"cannot execute command: Contro= ller is busy..."2022-05-03T07:12:00.log:2022-05-03 23:58:25,042Z &= lt; =C2=A00 9077 =C2=A0ferrorMessage=3D"cannot execute command: Co= ntroller is busy..."2022-05-04T07:12:00.log:2022-05-05 03:26:16,16= 2Z < =C2=A00 9214 =C2=A0ferrorMessage=3D"cannot execute command= : Controller is busy..."When HAL sees that failure it runs its= cleanup tasks:2022-05-05 03:26:16,635Z < 1 22 w text=3D"Ru= nning cleanup tasks."Part of the cleanup tasks for HAL's &= #39;expose' macro is to close the APOGEEshutter:2022-05-05 = 03:26:21,639Z > client.v3061 9215 shutter closeWhen starting the= next APOGEE exposure manually, that explains why theshutter was closed= at least.We might consider filing a ticket to re-evaluate the HAL = 'expose' cleanuptasks since we've had that 'shutter clo= se' one bite us before when anexposure was in progress, too. There&= #39;s probably a good reason for it, butmight be worth asking about it.= Dan________________________________________________________= ____Logs of Nightly APO Activities https://trac.sdss.org/= tracmailman/browser/private/sloan-25mlog/index.html --000000000000aa509e05de47527f--