Subject: Re: [sloan-25mlog 4034] Re: 2.5m obslog 20180428 (SJD 58237) From: John Downey Date: Sun, 29 Apr 2018 09:04:35 -0700 --Apple-Mail=_24BBDB9C-E9DE-4736-8B58-681242ABA5D6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Apr 29, 2018, at 9:00 AM, Kaike Pan wrote: >=20 >=20 >=20 >> On Apr 29, 2018, at 9:51 AM, John Downey > wrote: >>=20 >>>=20 >>> On Apr 29, 2018, at 8:33 AM, Kaike Pan > wrote: >>>=20 >>>=20 >>>=20 >>>> On Apr 29, 2018, at 9:29 AM, John Downey > wrote: >>>>=20 >>>>>=20 >>>>> On Apr 29, 2018, at 8:24 AM, Kaike Pan > wrote: >>>>>=20 >>>>>=20 >>>>>=20 >>>>>> On Apr 29, 2018, at 9:15 AM, John Downey > wrote: >>>>>>=20 >>>>>>>>=20 >>>>>>>> Remarks: >>>>>>>> -------- >>>>>>>> - Lost time to APOGEE due to weather : 7.2 hours >>>>>>>> - Lost time due to software problems: 34 min (see below) >>>>>>>> - Pointing models looks good=20 >>>>>>>> - dust: 3300 counts >>>>>>>>=20 >>>>>>>> -- At 10:46Z after cartridge change and run gotoField we got = 'MCP unavailable'. >>>>>>>> We waited until the telescope stops and tried to 'soft' reboot = from sdss-host2. >>>>>>>> All processes on the 'host2' were slowed down.=20 >>>>>>>> By that we did 'hard' reboot MCP. Then after gotoField all the = axis were with >>>>>>>> big fiducials error and no guiding stars. We moved the = telescope at zenith and >>>>>>>> did 'accept' fiducials. Lost time due to MCP problems was 34 = min. >>>>>>>>=20 >>>>>>>=20 >>>>>>> Thanks Dan and Viktor. =20 >>>>>>> I assume that the =E2=80=9Csoft=E2=80=9D reboot refer to = rebooting MCP from MCP Menu that was started from sass-host2. >>>>>>> Anyway, if sass-host2 runs slow, please reboot MCP from its = crates.=20 >>>>>>>=20 >>>>>>> If fiducial errors are really big, I suggest that we slew the = telescope to alt=3D60 (or 75) deg, check it is indeed near 60 (or 75) = degrees, and >>>>>>> accept altitude fiducial there, then slew to instrument change = to accept fiducials on AZ and rotator. If we directly slew to alt=3D90 = and=20 >>>>>>> altitude is off too much, we may end up pass 90 deg when slew = there without accept alt fiducial first.=20 >>>>>>>=20 >>>>>>=20 >>>>>> The MCP interface should be running on hub, not host2. >>>>>>=20 >>>>>> Is this just a slip of typing? As I recall, the MCP interface = looks very odd when run on host2, so I'm going to guess this is a typo. >>>>>=20 >>>>> It is not a typo. We can reboot MCP from a MCP Menu started from = sdss-host2 but not from a MCP Menu started from sdss-hub. >>>>=20 >>>> That sounds to me like a very strange thing arrangement, "clunky" = is the word I would use. >>>>=20 >>>> I can't say as I'm surprised however, we have a lot of that in the = computers here. >>>>=20 >>>> My advice is to always physically reboot - unless the host2 = interface is running before the MCP becomes unavailable, I don't see any = way that starting one to reboot could possibly work. Just a waste of = time that could be spent walking down to the enclosure to push the = button. >>>=20 >>> Observers usually start MCP at the beginning of nights. If it was = not running before MCP becomes unavailable, one cannot start the = interface >>> at all. >>=20 >> So you are running an MCP interface on host2 in the control room, = while the one at the podium is running on hub? >=20 > Yes. Because the migration MCP to hub doesn=E2=80=99t have all = functionality work at 100% level yet. >=20 >>=20 >> Does remote reboot typically work when the MCP is hung? >=20 > If it works when MCP hangs but sdss-host2 works properly. Last night, = looks like both MCP and sdss-host2 had issues at the=20 > same time, not sure if they are related. Ok, that all makes sense once explained. Even after 3 years I get surprised by the "temporary" work arounds that = have become established procedures. As work arounds they make sense, as = procedures they strike me as very strange. >=20 >>=20 >>>=20 >>>>=20 >>>>>=20 >>>>>>=20 >>>>>> If rebooting at 90 degrees, you may register a 314,xxx count = error noted in ticket #2853. We reboot so infrequently that I don't = know if it will cause the error, but given that a couple other actions = cause the MEI to read a spurious fiducial at 90 degrees, it wouldn't = surprise me much if a reboot did the same. >>>>>>=20 >>>>>> In such a case, slew down to 88 degrees and accept the 88.8 = fiducial (that is the correct #6 location). >>>>>> After accepting that one, slew down to the next, around 75 = degrees, and verify that the error is low. >>>>>>=20 >>>>>>>=20 >>>>>>> ____________________________________________________________ >>>>>>> Logs of Nightly APO Activities > >>>>>>> = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html = --Apple-Mail=_24BBDB9C-E9DE-4736-8B58-681242ABA5D6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On Apr 29, 2018, at 9:00 AM, Kaike Pan = wrote:On Apr 29, 2018, at 9:51 AM, John Downey wrote:On Apr 29, 2018, at 8:33 AM, Kaike = Pan wrote:On Apr = 29, 2018, at 9:29 AM, John Downey wrote:On Apr 29, 2018, at 8:24 AM, Kaike = Pan wrote:On Apr = 29, 2018, at 9:15 AM, John Downey wrote:Remarks:--------- Lost time to APOGEE due to weather : = 7.2 hours- Lost time due to software problems: 34 min = (see below)- Pointing models looks good - dust: 3300 = counts-- At 10:46Z after cartridge change = and run gotoField we got 'MCP unavailable'.We waited = until the telescope stops and tried to 'soft' reboot from = sdss-host2.All processes on the 'host2' were slowed = down. By = that we did 'hard' reboot MCP. Then after gotoField all the axis were = withbig fiducials error and no guiding stars. We moved = the telescope at zenith anddid 'accept' fiducials. Lost = time due to MCP problems was 34 min.Thanks Dan and Viktor. I assume that the =E2=80=9Csoft=E2=80=9D reboot refer to = rebooting MCP from MCP Menu that was started from sass-host2.Anyway, if sass-host2 runs slow, please reboot MCP from its = crates. If fiducial errors are really big, I suggest = that we slew the telescope to alt=3D60 (or 75) deg, check it is indeed = near 60 (or 75) degrees, andaccept altitude fiducial = there, then slew to instrument change to accept fiducials on AZ and = rotator. If we directly slew to alt=3D90 and altitude is = off too much, we may end up pass 90 deg when slew there without accept = alt fiducial first. The MCP interface should be running on hub, = not host2.Is this just a slip of typing? As I = recall, the MCP interface looks very odd when run on host2, so I'm going = to guess this is a typo.It is not a typo. We can reboot MCP from a MCP Menu = started from sdss-host2 but not from a MCP Menu started from = sdss-hub.That sounds to me like a very strange thing = arrangement, "clunky" is the word I would use.I can't say as I'm surprised however, we have a = lot of that in the computers here.My advice is to always physically reboot - = unless the host2 interface is running before the MCP becomes = unavailable, I don't see any way that starting one to reboot could = possibly work. Just a waste of time that could be spent walking = down to the enclosure to push the button.Observers usually start = MCP at the beginning of nights. If it was not running before MCP becomes = unavailable, one cannot start the interfaceat = all.So you are running an MCP interface on host2 in the = control room, while the one at the podium is running on = hub?Yes. = Because the migration MCP to hub doesn=E2=80=99t have all functionality = work at 100% level yet.Does = remote reboot typically work when the MCP is = hung?If it works when MCP hangs but sdss-host2 works properly. = Last night, looks like both MCP and sdss-host2 had issues at = the same time, not sure if they are = related.Ok, that all makes sense once = explained.Even after 3 years I get = surprised by the "temporary" work arounds that have become established = procedures. As work arounds they make sense, as procedures they = strike me as very strange.If rebooting at 90 degrees, you may = register a 314,xxx count error noted in ticket #2853. We reboot so = infrequently that I don't know if it will cause the error, but given = that a couple other actions cause the MEI to read a spurious fiducial at = 90 degrees, it wouldn't surprise me much if a reboot did the = same.In such = a case, slew down to 88 degrees and accept the 88.8 fiducial (that is = the correct #6 location).After = accepting that one, slew down to the next, around 75 degrees, and verify = that the error is low.____________________________________________________________Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/= index.html= --Apple-Mail=_24BBDB9C-E9DE-4736-8B58-681242ABA5D6--