Subject: Re: [sloan-25mlog 4980] 2.5m obslog 20200213 (SJD 58893) From: Kaike Pan Date: Fri, 14 Feb 2020 16:41:23 -0700 Thanks to Dan, John, and Elena for pushing on this. So, look like that we will not have this kind of alerts with the new = alertsActor. If that is the case, it will be great and should not be an issue for us.=20 Sounds that the old alertsActor checks the predicated tccPos rather than = actual axis positions. When the telescope is halted, there is no predicated axis positions, so = gets nan.=20 -Kaike > On Feb 14, 2020, at 9:40 AM, Donor, John wrote: >=20 > Hi Dan, >=20 > Yes unfortunately this error does make sense; I have no idea why the = old actor was written this way. >=20 > First I'll say it's not a problem in the new actor! So there's no = reason to spend too much time on this. >=20 > I'm not sure why the TCC and apogee actors don't have an "aliveAt" = keyword. But the keywords we watch (tccPos, ditherPosition, and rootURL = I think) are output very reliably, so it works out. If someone were to = do some work on these other actors and decide to add an aliveAt keyword, = I'm sure that would be fine (and I think I've heard from some that they = would prefer to have that), but I just don't think it's necessary. All = the problems we have right now are due to old code that doesn't work as = intended (and possibly never did). And it will be replaced very soon (as = soon as I figure out why I can't start it with stageManager...) >=20 > And I just saw Elena's email come through: Elena the new alerts config = file is = /home/sdss4/products/Linux64/twistedAlertsActor/git/python/twistedAlertsAc= tor/etc/alertsActions.yml (hopefully I typed that correctly); there is = also a /home/sdss4/.alertsActions.yml, but it is intended to be dynamic = and is modified somewhat for testing at the moment. The old alerts actor = is in /home/sdss4/products/Linux64/alertsActor/; there is a config file = for emails and ports, etc, but the actual alerts are all implemented in = python. The structure is fairly obvious when you look at it. You may = prefer to find the version on either the sdss or APO github for easy = reading. >=20 > Cheers, > John >=20 > =EF=BB=BFOn 2/14/20, 10:26 AM, "Daniel Oravetz" = wrote: >=20 > Hi John, all, >=20 >>=20 >> APACHE POINT OBSERVATORY SDSS 2.5M >>=20 >> OBSERVING LOG >> Thursday February 13, 2020 >>=20 >> (SJD 58893) >=20 >> Also, as usual every 10 seconds we have warnings in the STUI log >> window like >> 22:52:49 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: >> KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" >=20 > I looked at the TCC documentation and the alertsActor configuration = file > and these errors make sense as to why we're seeing them, but we = should > probably resolve to find a better way to check the TCC heartbeat, = i.e., a > different keyword, perhaps. >=20 > The alertsActor configuration file uses the tccPos keyword for the = tcc > heartbeat check. When the telescope is tracking on sky this keyword = gets > updated regularly with "normal" numerical values and everything is = happy; > e.g., for alt: >=20 > 02/14/2020 12:47:36.402101084 52.955 > 02/14/2020 12:47:38.571900929 52.9561 > 02/14/2020 12:47:38.588490996 52.9563 > 02/14/2020 12:47:41.312126448 52.9589 > 02/14/2020 12:47:46.311108473 52.9629 > 02/14/2020 12:47:51.364155376 52.967 > 02/14/2020 12:47:56.291199265 52.9709 > 02/14/2020 12:48:01.313211344 52.9749 > 02/14/2020 12:48:05.320100105 52.9783 > 02/14/2020 12:48:06.227122589 52.979 > 02/14/2020 12:48:11.464086870 52.983 >=20 > The problem is that "axes that are halted or not available are = listed as > 'nan'." E.g., again for alt: >=20 > 02/14/2020 12:50:04.925135692 45 > 02/14/2020 12:50:42.228159067 nan > 02/14/2020 12:50:52.407522831 nan > 02/14/2020 12:51:02.467108090 nan > 02/14/2020 12:51:06.925112711 90 > 02/14/2020 12:52:00.279314827 nan > 02/14/2020 12:52:10.354867369 nan > 02/14/2020 12:52:20.582490483 nan > 02/14/2020 12:52:30.713705410 nan > 02/14/2020 12:52:40.817600202 nan > 02/14/2020 12:52:50.994349836 nan > 02/14/2020 12:53:01.085319511 nan > 02/14/2020 12:53:11.159516869 nan > 02/14/2020 12:53:21.242423741 nan > 02/14/2020 12:53:31.350629448 nan > 02/14/2020 12:53:41.442489675 nan > 02/14/2020 12:53:51.589533416 nan > 02/14/2020 12:54:01.684601052 nan > 02/14/2020 12:54:11.770988706 nan > 02/14/2020 12:54:21.973573563 nan > 02/14/2020 12:54:32.083736657 nan > 02/14/2020 12:54:42.161305356 nan > 02/14/2020 12:54:52.268362222 nan > 02/14/2020 12:55:04.660431189 nan > 02/14/2020 12:55:08.260914439 nan > 02/14/2020 12:55:08.454232849 nan > 02/14/2020 12:55:13.322231503 30 >=20 > That means when the telescope is sitting in a parked/stow state = with a > stop button in, the alertsActor will report an alert and the STUI = Logs get > inundated with warning messages, even though the TCC is not = disconnected. > Note that "during a slew, tccPos gives the *end* position of the = slew, not > the current predicted position", that's why '45', '90', '30' show = up > exactly and alone when looking at the tccPos:alt example above = (slews to > gangChange, instChange, and stow, respectively); and the = interspersed > periods of 'nan' reflect when the telescope was parked at those = positions > with axes unavailable/stop button in. >=20 > The aliveAt keyword is used for heartbeat checks when that keyword = is > available, but only boss and mcp seem to have that keyword. Is = there a > particular reason why all actors could not have that keyword in = their > actorkeys to use as a dedicated keyword for the heartbeat checks? >=20 > Dan >=20 >=20 >=20 > ____________________________________________________________ > Logs of Nightly APO Activities > = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html