Subject: Re: [sloan-25mlog 4623] Re: 2.5m obslog 20190907 (SJD 58734) From: John Downey Date: Sun, 8 Sep 2019 11:52:28 -0600 --Apple-Mail=_D507D2C6-5076-4EEA-9A96-B0C6611E5F5B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I never re-connected the MCP last night, that was probably the problem. > On Sep 8, 2019, at 11:48 AM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote: >=20 > guiderActor won=E2=80=99t let you load a cartridge without first = asking the MCP to read the encoding of the cartridge to get its number. = In principle it should not hang, but if the MCP was in a weird state or = not responding it may hang, waiting forever for the MCP to reply. >=20 > If the MCP is not going to be fixed tonight and you still want to be = able to load a cart, I made a change to guiderActor that would allow you = to force load a cart without commanding the MCP. Let me know if you want = me to install it. >=20 > Jos=C3=A9 >=20 >=20 > On September 8, 2019 at 04:15:11, kinemuchi@apo.nmsu.edu = (kinemuchi@apo.nmsu.edu = ) wrote: >=20 >>=20 >> APACHE POINT OBSERVATORY SDSS 2.5M=20 >>=20 >> OBSERVING LOG=20 >> Saturday September 07, 2019=20 >>=20 >> (SJD 58734)=20 >>=20 >> ---- Observing Team ----=20 >>=20 >> Audrey Oravetz=20 >> Karen Kinemuchi=20 >>=20 >> Dylan Gatlin (afternoon training)=20 >>=20 >>=20 >>=20 >> ---- Observing Plan ----=20 >>=20 >>=20 >> Sat 7 SEP 2019 58734.0 0.66 01:55 11:34 18:00 03:39 ---- =20 >> APOGEE-2 until 07:43 ( 5.8 h), then MaNGA ( 3.8 h) =20 >>=20 >> ---- Observing Summary ----=20 >>=20 >> Tonight is lost due to weather and ongoing MCP issues.=20 >>=20 >>=20 >>=20 >> Notes:=20 >> Unable to load a cartridge into STUI. This may be related to the MCP = work=20 >> earlier today.=20 >>=20 >> Projects: be aware we are running out of disk space.=20 >> Filesystem Size Used Avail Use% Mounted on=20 >> /dev/vdd 688G 635G 54G 93% /home=20 >> /dev/vdc1 11T 9.7T 1.2T 90% /data=20 >>=20 >>=20 >> ---- Observing Log ----=20 >>=20 >> Afternoon:=20 >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=20 >>=20 >> Dylan began training with afternoon checkout, which as anybody can = see below,=20 >> was not the best choice since everything is in disarray. =20 >>=20 >> The MCP is offline tonight. =20 >>=20 >> We gathered mirror numbers, but ran into a complication with = resetting the focus=20 >> and scale values to 0 and 1, respectively, because of the TCC track = command=20 >> failing with:=20 >>=20 >> 02:01:52 OBSERVER.dgatlin 118 tcc f text=3D"Slew failed: = Sub-command(s) failed:=20 >> DRIFT: not connected; DRIFT: not connected; DRIFT: not connected"=20 >> 02:01:52 CmdDone: OBSERVER.dgatlin 118 tcc track 121,30 mount=20 >>=20 >> The good news is JohnD was able to reconnect the TPM so we have a set = of mirror=20 >> numbers, but we do not feel comfortable resetting nominal mirror = positions not=20 >> knowing if the mirrors are in their correct, nominal, orientations. =20= >>=20 >> Disk space. We will need to address this soon. =20 >>=20 >> /dev/vda2 7.7G 5.9G 1.5G 81% /=20 >> tmpfs 3.4G 0 3.4G 0% /dev/shm=20 >> /dev/vda1 194M 128M 57M 70% /boot=20 >> /dev/vdb 19G 8.8G 9.1G 49% /exports/hub-local=20 >> /dev/vdd 688G 635G 54G 93% /home=20 >> /dev/vdc1 11T 9.7T 1.2T 90% /data=20 >>=20 >> We are obviously unable to gather MaNGA cals because the cameras are = still offline. =20 >>=20 >> What we didn't know is that we are unable to load the cartridge with = the=20 >> MCP/TCC down. We attempted to load the cart in preparation to take = cals with it,=20 >> but the process hung, and many attempts to restart the process with = stageManager=20 >> failed. =20 >>=20 >> We attempted to load cartridge two ways: using the "guider = loadCartridge" =20 >> command, which failed, and through the SOP loadCartridge button, = which also =20 >> failed. This hung the guider actor, and we had to restart the actor = through =20 >> stageManager.=20 >>=20 >> We had to kill the stubborn guider process with a kill -9 command, = which may not =20 >> be the most graceful way to do this. This command allowed us to = restart the =20 >> guider actor regardless.=20 >>=20 >> We also checked at the telescope with the guider. We did check the = breakers for =20 >> the guider and they looked fine. We reseated the ethernet and were = able to see =20 >> the green lights for the connection. When we went down to turn off = the =20 >> amplifiers later in the night, we noted that the guider showed no = activity (no =20 >> green lights).=20 >>=20 >> We suspect the power is not going to the guider. The software shows = that the =20 >> guider and gcameraICC actors are running. With the trouble with the = guider, we =20 >> were unable to take guider calibrations this evening.=20 >>=20 >> This was the conclusion of afternoon checkout, with nothing working. = Luckily we=20 >> have many clouds and rainstorms around, which would prevent us from = opening =20 >> anyway.=20 >>=20 >> Night:=20 >> =3D=3D=3D=3D=3D=3D=20 >> 03:37Z We remain closed. Lots of nearby radar and clouds.=20 >>=20 >> Pump monitoring:=20 >> @03:30Z=20 >> SP1: 1.5 e-6=20 >> SP2: 1.2 e-7=20 >>=20 >> @08:31Z=20 >> SP1: 1.5 e-6=20 >> SP2: 1.2 e-7=20 >> 05:50Z Rain sensor went off.=20 >>=20 >> 08:30Z Still raining on site.=20 >>=20 >>=20 >> 10:21Z Humidity dropping, but completely overcast. Radar moving out.=20= >>=20 >> 11:04Z Humid again. Giving up. Starting APOGEE morning cals.=20 >>=20 >> ---- Mirror Numbers ----=20 >>=20 >>=20 >> PRIMARY:=20 >> --------=20 >> Scale: 1.000000;=20 >>=20 >> MIGS TONIGHT NOMINAL=20 >> Axial A: -5.33 tpm_MIG1A --- Invalid channel name=20 >> Axial B: -5.05 tpm_MIG1B --- Invalid channel name=20 >> Axial C: -5.271 tpm_MIG1C --- Invalid channel name=20 >> Trans D: -0.578 tpm_MIG1DT --- Invalid channel name=20 >> Lateral E: 9.3472 9.3345=20 >> Lateral F: 14.4399 14.3637=20 >>=20 >> GALILS=20 >> Commanded: 11200 -1850 6850 29650 21250 21050=20 >> Actual: 11200 -1850 6850 29650 21250 21050=20 >>=20 >> SETMIR VALUES=20 >> PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12=20 >> PrimOrient: 0.0 -11.4 6.58 375.38 620.77=20 >>=20 >> SECONDARY:=20 >> ----------=20 >>=20 >> Focus: 0.0000=20 >> Air temp: 15.8=20 >> Altitude: 30.0=20 >>=20 >> MIGS TONIGHT NOMINAL=20 >> Axial A: -6.665 -6.668=20 >> Axial B: -0.039 tpm_MIG2B --- Invalid channel name=20 >> Axial C: -6.672 tpm_MIG2C --- Invalid channel name=20 >> Trans D: -0.074 -0.074=20 >>=20 >> GALILS=20 >> Commanded: 1058550 1101450 1079600 -7250 -6550=20 >> Actual: 1058718 1101244 1079676 -7250 -6550=20 >>=20 >> SETMIR VALUES=20 >> SecDesOrient: 857.0 2.7 12.75 -38.35 156.51=20 >> SecOrient: 857.0 2.61 12.74 -38.43 156.78=20 >>=20 >>=20 >>=20 >> ---- Telescope Status ----=20 >>=20 >>=20 >> Never opened.=20 >>=20 >> No interlocks bypassed=20 >>=20 >> tatus at: 11:11:09 =20 >> Telescope stowed at: 121.0, 30.0, 121.0 mount=20 >> Instrument mounted: unknown=20 >> Counterweights at: None, None, None, None =20 >> LN2 autofill systems: Connected and turned on=20 >> 180L LN2 dewar scales: SP1 ___ lbs, ___ psi=20 >> SP2 ___ lbs, ___ psi=20 >>=20 >>=20 >> ---- Software Used ----=20 >>=20 >> Get Versions, 09/08/19, 01:01:37=20 >> -- STUI software: =20 >> STUI : 1.7.8=20 >> alerts : trunk=20 >> apo : trunk=20 >> boss : v4_0=20 >> -boss.specMec : 3.0.4=20 >> gcamera : v2_2=20 >> guider : 3.8.1=20 >> hartmann : 1.6.0=20 >> hub : Tag: v4.2=20 >> mcp : FAILED=20 >> -mcp.plc : FAILED=20 >> -mcp.azFiduci : FAILED=20 >> -mcp.altFiduc : FAILED=20 >> -mcp.rotFiduc : FAILED=20 >> platedb : 2.8.0=20 >> sop : 3.12.3=20 >> tcc : 1.3.1=20 >> apogee : not availble=20 >> apogeecal : not availble=20 >> apogeeql : v1_6_1=20 >> APO-local : trunk=20 >> -- done --=20 >>=20 >> -- Other versions ----=20 >> idlspec2d : v5_7_12=20 >> plate-mapper3 : NOCVS:v6_0_12=20 >> petunia : 1.4.2=20 >> autoscheduler : 1.3.0=20 >> sdss_python_module: trunk=20 >> mangadrp : trunk exported=20 >>=20 >>=20 >> ____________________________________________________________=20 >> Logs of Nightly APO Activities >=20 >> = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html = =20 > ____________________________________________________________ > Logs of Nightly APO Activities > > = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html = --Apple-Mail=_D507D2C6-5076-4EEA-9A96-B0C6611E5F5B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 I never re-connected the MCP last night, that was probably = the problem.On = Sep 8, 2019, at 11:48 AM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote:guiderActor won=E2=80=99t let you load a cartridge without = first asking the MCP to read the encoding of the cartridge to get its = number. In principle it should not hang, but if the MCP was in a weird = state or not responding it may hang, waiting forever for the MCP to = reply.If the MCP is not going to = be fixed tonight and you still want to be able to load a cart, I made a = change to guiderActor that would allow you to force load a cart without = commanding the MCP. Let me know if you want me to install it.Jos=C3=A9On September = 8, 2019 at 04:15:11, kinemuchi@apo.nmsu.edu (kinemuchi@apo.nmsu.edu) wrote:APACHE POINT OBSERVATORY = SDSS 2.5M OBSERVING LOG Saturday September 07, 2019 (SJD 58734) ---- Observing = Team ---- Audrey Oravetz Karen Kinemuchi Dylan = Gatlin (afternoon training) ---- Observing Plan ---- Sat 7 SEP 2019 58734.0 0.66 01:55 11:34 18:00 = 03:39 ---- APOGEE-2 until 07:43 ( 5.8 h), then = MaNGA ( 3.8 h) ---- Observing = Summary ---- Tonight is lost due to = weather and ongoing MCP issues. Notes: Unable to load a = cartridge into STUI. This may be related to the MCP work earlier today. Projects: = be aware we are running out of disk space. Filesystem = Size Used Avail Use% Mounted on /dev/vdd 688G 635G = 54G 93% /home /dev/vdc1 11T 9.7T 1.2T 90% = /data ---- Observing = Log ---- Afternoon: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dylan began training with afternoon checkout, which as = anybody can see below, was not the best choice since = everything is in disarray. The = MCP is offline tonight. We = gathered mirror numbers, but ran into a complication with resetting the = focus and scale values to 0 and 1, respectively, = because of the TCC track command failing = with: 02:01:52 OBSERVER.dgatlin 118 = tcc f text=3D"Slew failed: Sub-command(s) failed: DRIFT: not connected; DRIFT: not connected; DRIFT: not = connected" 02:01:52 CmdDone: OBSERVER.dgatlin 118 = tcc track 121,30 mount The good news = is JohnD was able to reconnect the TPM so we have a set of = mirror numbers, but we do not feel comfortable = resetting nominal mirror positions not knowing if = the mirrors are in their correct, nominal, orientations. Disk space. We will need to address this = soon. /dev/vda2 7.7G 5.9G 1.5G = 81% / tmpfs 3.4G 0 3.4G 0% /dev/shm /dev/vda1 194M 128M 57M 70% /boot /dev/vdb = 19G 8.8G 9.1G 49% /exports/hub-local /dev/vdd 688G = 635G 54G 93% /home /dev/vdc1 11T 9.7T 1.2T 90% = /data We are obviously unable to = gather MaNGA cals because the cameras are still offline. What we didn't know is that we are unable to = load the cartridge with the MCP/TCC down. We = attempted to load the cart in preparation to take cals with it, but the process hung, and many attempts to restart the = process with stageManager failed. We attempted to load cartridge two ways: using = the "guider loadCartridge" command, which = failed, and through the SOP loadCartridge button, which = also failed. This hung the guider actor, and = we had to restart the actor through stageManager. We had to = kill the stubborn guider process with a kill -9 command, which may = not be the most graceful way to do this. This = command allowed us to restart the guider actor = regardless. We also checked at the = telescope with the guider. We did check the breakers for the guider and they looked fine. We reseated the ethernet and = were able to see the green lights for the = connection. When we went down to turn off the amplifiers later in the night, we noted that the guider = showed no activity (no green lights). We suspect the power is not going to the = guider. The software shows that the guider and = gcameraICC actors are running. With the trouble with the guider, = we were unable to take guider calibrations = this evening. This was the conclusion = of afternoon checkout, with nothing working. Luckily we have many clouds and rainstorms around, which would prevent = us from opening anyway. Night: =3D=3D=3D=3D=3D=3D 03:37Z We remain closed. Lots of nearby radar and = clouds. Pump monitoring: @03:30Z SP1: 1.5 e-6 SP2: 1.2 e-7 @08:31Z SP1: 1.5 e-6 SP2: 1.2 e-7 05:50Z Rain sensor went = off. 08:30Z Still raining on = site. 10:21Z Humidity = dropping, but completely overcast. Radar moving out. 11:04Z Humid again. Giving up. Starting APOGEE = morning cals. ---- Mirror Numbers = ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial = A: -5.33 tpm_MIG1A --- Invalid channel name Axial B: -5.05 tpm_MIG1B --- Invalid channel = name Axial C: -5.271 tpm_MIG1C --- Invalid = channel name Trans D: -0.578 tpm_MIG1DT --- = Invalid channel name Lateral E: 9.3472 = 9.3345 Lateral F: 14.4399 14.3637 GALILS Commanded: 11200 = -1850 6850 29650 21250 21050 Actual: 11200 -1850 = 6850 29650 21250 21050 SETMIR = VALUES PrimDesOrient: 0.0 -11.37 6.87 375.47 = 621.12 PrimOrient: 0.0 -11.4 6.58 375.38 = 620.77 SECONDARY: ---------- Focus: = 0.0000 Air temp: 15.8 Altitude: = 30.0 MIGS TONIGHT NOMINAL Axial A: -6.665 -6.668 Axial B: = -0.039 tpm_MIG2B --- Invalid channel name Axial C: = -6.672 tpm_MIG2C --- Invalid channel name Trans D: = -0.074 -0.074 GALILS Commanded: 1058550 1101450 1079600 -7250 -6550 Actual: 1058718 1101244 1079676 -7250 -6550 SETMIR VALUES SecDesOrient: = 857.0 2.7 12.75 -38.35 156.51 SecOrient: 857.0 2.61 = 12.74 -38.43 156.78 ---- Telescope Status ---- Never opened. No interlocks bypassed tatus= at: 11:11:09 Telescope stowed at: 121.0, = 30.0, 121.0 mount Instrument mounted: = unknown Counterweights at: None, None, None, = None LN2 autofill systems: Connected and = turned on 180L LN2 dewar scales: SP1 ___ lbs, ___ = psi SP2 ___ lbs, ___ psi ---- Software Used ---- Get Versions, 09/08/19, 01:01:37 -- STUI = software: STUI : 1.7.8 alerts : trunk apo : trunk boss : v4_0 -boss.specMec : = 3.0.4 gcamera : v2_2 guider : = 3.8.1 hartmann : 1.6.0 hub : = Tag: v4.2 mcp : FAILED -mcp.plc = : FAILED -mcp.azFiduci : FAILED -mcp.altFiduc : FAILED -mcp.rotFiduc : = FAILED platedb : 2.8.0 sop : = 3.12.3 tcc : 1.3.1 apogee : not = availble apogeecal : not availble apogeeql : v1_6_1 APO-local : = trunk -- done -- --= Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_12 petunia : = 1.4.2 autoscheduler : 1.3.0 sdss_python_module: trunk mangadrp : = trunk exported ____________________________________________________________&nb= sp;Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/= index.html ____________________________________________________________Logs of Nightly APO = Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/= index.html= --Apple-Mail=_D507D2C6-5076-4EEA-9A96-B0C6611E5F5B--