Subject: Re: [sdss3-25mlog 1780] 2.5m obslog 20140602 (MJD 56811)
From: Dan Long
Date: Tue, 3 Jun 2014 10:31:12 -0600
Hi all,
   Looking through the telemetry from last night, this is what I found.  =
First, the PLC was happy with the state of things.  All axes were live =
and permits granted when all axes stopped moving at 04:37Z, and it =
continued this way well after 04:37Z.  It appears that the =
communications to the axes was stopped at this time.  It cannot be =
further confirmed because the TCC and MCP logs also stopped at this =
time.  It certainly appears that part of the hub or some connections to =
it went away.=20
   John, is there maybe some hint in the hub log (which did continue =
past 04:37Z)?
-Dan
On Jun 3, 2014, at 5:16 AM, dmbiz@apo.nmsu.edu wrote:
>=20
>                       APACHE POINT OBSERVATORY SDSS 2.5M
>=20
>                                 OBSERVING LOG
>                          Monday June 02, 2014
>=20
>                                  (MJD 56811)
>=20
> ----  Observing Team ----
>=20
> Patrick Gaulme
> Dmitry Bizyaev
>=20
>=20
> ---- Observing Plan ----
>=20
> 2 Jun 2014  56811.0  0.34  03:28 10:35  13:11 20:19  BOSS  7.1 hours
> Start APOGEE at 15 h LST
>=20
>=20
>=20
> ---- Observing Summary ----
>=20
> BOSS plates exposed:=20
> 12 7419 : DONE
>=20
> APOGEE plates exposed:
> 6 7381 000+75_A 1 x ABBA , 1 missing, 4 faint fibers
> 7 6891 005+03_A 2 x ABBA, no missing, 3 faint
> 3 6895 008+02_A 2 x ABBA, no missing, 2 faint
> 5 5927 SGRC3_A  2 x ABBA, 3 missing, 1 faint
>=20
>=20
> LN2 large dewars show low pressure (in TPM), which makes LN2 fill very =
slow:
> evening fill last ~30 minutes.=20
>=20
>=20
> Some 1h 20m were lost due to a problem, most probably with the =
rotator's
> amplifier, see log for events tracking.=20
>=20
>=20
.
.
.
> After finishing the exposure, we started getting errors about dead =
tcc:
> 4:39:01 .alerts 0 alerts e alert=3Dtcc.heartbeat,critical,"The TCC's =
been dead for
> at least 122s",enabled,noack,""
> 04:39:02 .boss 3975 boss w SP2R1HeaterVRead=3D-0.527
> 04:39:02 .boss 3975 boss w camCheck=3DSP2R1HeaterVRead
> 04:39:11 .alerts 0 alerts e alert=3Dtcc.heartbeat,critical,"The TCC's =
been dead
> for at least 132s",enabled,noack,""
> 04:39:21 .alerts 0 alerts e alert=3Dtcc.heartbeat,critical,"The TCC's =
been dead
> for at least 142s",enabled,noack,""
>=20
> Tried to hub startNubs tcc25m, did not help.
>=20
> When I logged in at tcc25m, the process TELRUN behaved as it is =
starting for the
> first time. I tried telstop/telrun again. Then hubstartNubs for =
tcc25m, then
> reseter alerts actor. After reseting, the alertsActor stopped =
complaining.
> Nevertheless, ROT axis continued to show bad status (AO in MCP). =
Attempt to
> clear it with axis stop/axis init did not success.=20
>=20
> Checked slip detector. Checked amplifiers.=20
> The ROT amplifier showed more red LEDs than usual. Thus, the upper =
right corner
> light was red. Tried to power cycle it, with ~ 1 min wait time between =
off/on.
> After powering on, same red lights came on.=20
> 4:58:51 APO.Patrick.sop 378 tcc F Text=3D"Move rejected; this =
instrument has no
> rotator"
> 04:58:51 CmdDone: APO.Patrick.sop 378 tcc track 121.000000, 90.000000
> mount/rottype=3Dmount/rotangle=3D0.000000
> 04:58:51 APO.Patrick 125 sop w text=3D"Failed to start slew"
>=20
> I loaded correct instrument block only after this, but it did not =
allow axis
> init to clear the rotator.=20
>=20
> Call in French (he was on site). French saw the amplifiers LEDs. He =
moved
> rotator manually a little, then power cycled the amplifiers again. =
After this
> the LEDs got mostly green. We were able to move telescope via tcc.=20
> Next step, loadCArtridge command failed to work.=20
> Issued mcp info,
> then reload guiderActor at hub25m.=20
> By mistake, we reloaded sopActor first, with no changes after doint =
this.=20
> Reloading guider actor helped. Cartridge was successfully loaded.=20
>=20
> 05:58:11 .tcc 0 tcc F Modu=3D"prt_Read"; Text=3D"read timed out on =
port LTA11:"
> 05:58:11 .tcc 0 tcc F Modu=3D"prt_Init_Forth"; Text=3D"initialization =
command init
> failed"
>=20
> 05:58:39 APO.Patrick 134 sop f text=3D"Failed to close screens, warm =
up lamps, and
> slew to field"
> 05:58:39 CmdDone: APO.Patrick 134 sop gotoField =20
>=20
> For some reason, telescope failed to star moving after gotoField.
>=20
> We moved it through tcc track just a few degrees of its position. Then =
issued
> the "gotoField again. It started working after that and went to the =
field when
> commanded.
>=20
> Given the problem started at 04:39Z, with telescope received to the =
next field at
> 06:05Z, minus ~ 6 min for the normal cartridge change (with slew back =
and fore),
> we lost 1h 20m due to this problem.
>=20
> French's version of what happened is that the rotator got almost stuck =
at
> certain position, with high current, amplifier was overloaded and =
turned to a
> wrong status, and then the problem propagated to software domain.=20
>=20