Subject: Re: [sloan-25mlog 5089] Re: 2.5m obslog 20200706 (SJD 59037) From: Russet McMillan Date: Tue, 7 Jul 2020 14:18:28 -0600 I THINK there have been years in the not-too-distant past (since the = TCCs were upgraded in 2013) where the shutdowns were separate, but I = couldn't swear to it. Most years 2.5m has a longer shutdown than 3.5, = so it is more common for 3.5m to be up while 2.5m is down than vice = versa. But I don't think this is unprecedented. If the problem is with the machine zaphod25m, why would disassembling = the other telescope cause that machine to go down? Maybe it was just a = mistake? > On Jul 7, 2020, at 2:14 PM, = wrote: >=20 > Have we not seen this because the shut downs always coincided before? = There was another bug I saw that no one must of caught for years in the = weather page last night. >=20 > Shane >=20 >=20 > -----Original Message----- > From: Russet McMillan =20 > Sent: Tuesday, July 7, 2020 2:01 PM > To: Conor Sayres > Cc: Russet McMillan ; kinemuchi@apo.nmsu.edu; = obs-report-25m@apo.nmsu.edu 2.5m ; Tracey = Fernandez ; shane thomas > Subject: Re: [sloan-25mlog 5089] Re: 2.5m obslog 20200706 (SJD 59037) >=20 > Tell me tcc25m is not using 3.5m temperatures to decide where to = focus? The 3.5m temps are broken because the telescope was disassembled = yesterday, but why would that affect 2.5m? >=20 >> On Jul 7, 2020, at 1:55 PM, Conor Sayres wrote: >>=20 >> Hi Guys, >>=20 >> Here=E2=80=99s what I=E2=80=99ve discovered about the issue last = night. >>=20 >> The root of this issue is most likely that the machine: = zaphod25m-p.apo.nmsu.edu with IP address 10.25.1.250 seems offline (or = at least un-pingable from the hub). >>=20 >> The apoActor queries zaphod to discover mirror/truss temperatures, = and then reports those to the TCC automatically on a timer. The TCC = uses these mirror temperatures in conjunction with a collimation model = to decide where exactly to move the mirrors. The apoActor was feeding = the TCC NaN values, which broke things. >>=20 >> For now, I=E2=80=99ve hacked in a check to the TCC where if any value = is an NaN, it will default to 10.0 C. I=E2=80=99ve tested this just = now, but having some observers eyes early on tonight would be good. >>=20 >> The TCC is currently running as the test version. To restart the = TCC: >>=20 >> $ lsst >> $ setup tcc -t test >> $ tcc start | stop | restart >>=20 >> If you end up restarting the mirror controllers, be sure to grab the=20= >> tagged versions (don=E2=80=99t use the -t test flag) $ lsst $ setup = tcc $ sec=20 >> start | stop | restart $ prim start | stop | restart >>=20 >> This hack should probably remain in the future, so after a few days = we can discuss a more long term fix option and tag a new TCC. >>=20 >> Conor >>=20 >>> On Jul 7, 2020, at 5:19 AM, kinemuchi@apo.nmsu.edu wrote: >>>=20 >>>=20 >>> APACHE POINT OBSERVATORY SDSS 2.5M >>>=20 >>> OBSERVING LOG >>> Monday July 06, 2020 >>>=20 >>> (SJD 59037) >>>=20 >>> ---- Observing Team ---- >>>=20 >>> Audrey Oravetz >>> Karen Kinemuchi >>>=20 >>> ---- Observing Plan ---- >>>=20 >>> Mon 6 JUL 2020 59037.0 APOGEE-2 (8.4 h) >>>=20 >>> ---- Observing Summary ---- >>>=20 >>> Entire night lost to nearby radar, clouds, and problem with = secondary=20 >>> mirror for focusing. >>>=20 >>> APOGEE lost 8.4 hours. >>> NOTES TO DAY STAFF AND DEVELOPERS: >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>=20 >>> * We had a problem with getting the secondary to respond while = trying to focus. >>> We contacted JohnD. We will contact Conor for help. See night log=20= >>> for details. >>>=20 >>> * Dust counts: 2200 >>>=20 >>>=20 >>> ---- Observing Log ---- >>>=20 >>> Afternoon: >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>=20 >>> Actors ok, disk space ok, guider dark ok, mirror numbers ok, APOGEE=20= >>> morning and evening cals ok. >>>=20 >>> MaNGA collimation check with cart 1: >>>=20 >>> motorPosition >>> sp1: 25535, 24756, 23148; =20 >>> sp2: 26971, 26760, 26539; =20 >>> ---------- >>> Median spectrographs temperatures: >>> Tsp1=3D 10.5, Tsp2=3D 14.9; =20 >>> Outside air temperature=3D 18.4; >>>=20 >>> 00:07:47 boss exposure 4.0, file=3D315266, C >>> 00:08:57 boss exposure 4.0, file=3D315267, C >>> 00:10:25 sp1.motor.move=3D -141, -142, -140 >>> 00:10:25 Hartmann collimate output on cart #1 >>> sp1: offset: r=3D -989 (In Focus); b=3D -2.2 (In Focus)=20 >>> sp2: offset: r=3D -315 (In Focus); b=3D -1.0 (In Focus)=20 >>> sp1: pred. move: spAverageMove=3D -141 >>> sp2: pred. move: spAverageMove=3D 0 >>> sp1: pred. resid: r=3D -848, b=3D -2.7, txt=3D OK, spTemp =3D 10.5 >>> sp2: pred. resid: r=3D -315, b=3D -1.0, txt=3D OK, spTemp =3D 14.9 >>> 00:21:31 boss exposure 25.0, file=3D315268, C >>> 00:24:20 boss exposure 4.0, file=3D315269, C >>>=20 >>> The blue cameras may be in spec tonight, depending on the = temperatures. >>>=20 >>> *Telescope check - All ok.=20 >>>=20 >>> *Did not started ventilating due to nearby storms. >>>=20 >>> Night: >>> =3D=3D=3D=3D=3D=3D >>>=20 >>> 03:05Z We remain closed due to nearby storms. >>> 09:30Z Radar clearing, but still cloudy. Ventilation begins. >>>=20 >>> Clearing >>>=20 >>> 10:33:10 encl25m : open; >>>=20 >>> 10:28:29 loadCart: Cart 9, Plate 10093; >>> 10:28:29 survey: APOGEE-2&MaNGA; lead: APOGEE lead >>>=20 >>> Secondary is not responding while we are trying to focus up the = guider. >>>=20 >>> 10:43:16 OBSERVER.aoravetz 42 tcc f text=3D"Sub-command(s) failed: = move nan,6.3439,11.2267,-39.8733,-19.2797: Too many function calls" >>> 10:43:16 CmdDone: OBSERVER.aoravetz 42 tcc set focus=3D-100.0/incr >>>=20 >>> 10:46:10 .tcc 0 tcc w Text=3D"Collimation Failure: Too many function = calls" >>>=20 >>> We are trying to reconnect the secondary for focus. Following=20 >>> procedure from Observing tips on 9/4/2019. Procedure did not work. >>>=20 >>> tcc mirr init sec and tcc mirror connect commands did not work. >>>=20 >>> Tried disabling and enabling collimate, did not work. >>>=20 >>> Homing the secondary. Telescope at 70 degrees. =20 >>>=20 >>> We are having trouble with the tcc proc enable collimate >>>=20 >>> tcc mirror status >>> 11:10:26 OBSERVER.aoravetz 61 tcc i=20 >>> mirrorConnState=3DConnected,Connected,NotAvailable >>>=20 >>> Systems are reporting mirror collimate is disabled. >>>=20 >>> Called JohnD. =46rom our homing the secondary, we were able to = confirm=20 >>> that the galil were active and moving. We are still getting the=20 >>> warning of Too many function calls and a "nan" for the piston value. = =20 >>> Maybe a software issue for Conor? >>>=20 >>> Running out of time, so we are closing. >>>=20 >>> 11:49:10 encl25m : closed; >>> 11:58Z Starting morning cals. >>>=20 >>>=20 >>>=20 >>> ---- BOSS Spectrographs Collimation Summary ---- >>>=20 >>> Afternoon focus check: >>>=20 >>> Time Cart R1 B1 Move1 B1Resid TSP1 R2 B2 Move2 = B2Resid TSP2 >>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >>> 00:09 1-12603A -989 -2.2 -141 -2.7 10.5 -315 -1.0 0 = -1.0 14.9 >>>=20 >>> 00:10:21 OBSERVER.aoravetz.sop 483 hartmann i r1MeanOffset=3D-0.15,"In= Focus" >>> 00:10:21 OBSERVER.aoravetz.sop 483 hartmann i b1MeanOffset=3D0.11,"In = Focus" >>> 00:10:21 OBSERVER.aoravetz.sop 483 hartmann i r2MeanOffset=3D-0.05,"In= Focus" >>> 00:10:21 OBSERVER.aoravetz.sop 483 hartmann i b2MeanOffset=3D0.05,"In = Focus" >>>=20 >>> ---- Mirror Numbers ---- >>>=20 >>>=20 >>> PRIMARY: >>> -------- >>> Scale: 1.000000; >>>=20 >>> MIGS TONIGHT NOMINAL >>> Axial A: -5.327 -5.328 >>> Axial B: -5.049 -5.052 >>> Axial C: -5.268 -5.268 >>> Trans D: -0.575 -0.579 >>> Lateral E: 9.3472 9.3472 >>> Lateral F: 14.5034 14.5161 >>>=20 >>> GALILS >>> Commanded: 11200 -1850 6850 29650 21250 21050 >>> Actual: 11200 -1850 6850 29650 21250 21050 >>>=20 >>> SETMIR VALUES >>> PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 >>> PrimOrient: 0.0 -11.4 6.58 375.38 620.77 >>>=20 >>> SECONDARY: >>> ---------- >>>=20 >>> Focus: 0.0000 >>> Air temp: 18.5 >>> Altitude: 30.5 >>>=20 >>> MIGS TONIGHT NOMINAL >>> Axial A: -6.665 -6.663 >>> Axial B: -0.026 -0.025 >>> Axial C: -6.67 -6.669 >>> Trans D: -0.627 -0.627 >>>=20 >>> GALILS >>> Commanded: 1089300 1127400 1113650 -7250 -6550 >>> Actual: 1059058 1100847 1079348 -7250 -6550 >>>=20 >>> SETMIR VALUES >>> SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 >>> SecOrient: 856.94 2.68 12.52 -38.57 156.73 >>>=20 >>>=20 >>>=20 >>> ---- Telescope Status ---- >>>=20 >>> 09:30Z Started ventilation >>> 10:30Z encl25m : open >>> 11:49Z encl25m : closed >>>=20 >>> No interlocks bypassed >>>=20 >>> Status at: 12:02:21=20 >>> Telescope stowed at: 121.0, 30.0, 0.0 mount >>> Instrument mounted: cartridge =3D 6 >>> Counterweights at: 336, 318, 317, 303=20 >>> LN2 autofill systems: Connected and turned on >>> 180L LN2 dewar scales: SP1 180.0 lbs, 22.3 psi=20 >>> SP2 178.8 lbs, 22.2 psi=20 >>>=20 >>> ---- Software Used ---- >>>=20 >>> --- Afternoon disk space --- >>> /dev/vdd 688G 552G 136G 81% /home >>> /dev/vdc1 11T 9.2T 1.7T 86% /data >>>=20 >>> --- STUI software --- >>> STUI : 1.7.8 >>> alerts : 2.0.1 >>> apo : trunk >>> boss : v4_1_2 >>> -boss.specMec : 3.0.4 >>> gcamera : v2_2 >>> guider : 3.8.1 >>> hartmann : 1.7.1 >>> hub : v4.2.4 >>> mcp : 1.11 >>> -mcp.plc : 34 >>> -mcp.azFiduci : v1_142 >>> -mcp.altFiduc : v1_142 >>> -mcp.rotFiduc : v1_142 >>> platedb : 2.8.0 >>> sop : 3.12.7 >>> tcc : 1.4.0 >>> apogee : not availble >>> apogeecal : not availble >>> apogeeql : v1_6_1 >>> APO-local : not availble >>>=20 >>> --- Other versions --- >>> idlspec2d : v5_7_12 >>> plate-mapper3 : NOCVS:v6_0_12 >>> petunia : 1.5.2 >>> autoscheduler : 1.5.2 >>> sdss_python_module: trunk >>> mangadrp : trunk exported >>>=20 >>> ---- APOGEE Data Summary ---- >>>=20 >>> [observer@sdss4-hub ~]$ list_ap -m 59036 -morning >>> APOGEE data list, mjd=3D59036, morning >>> raw_data: /data/apogee/utr_cdr/59036/apRaw-*.fits >>> archive: /data/apogee/archive/59036/apR-[a,b,c]-*.apz >>> quickred: /data/apogee/quickred/59036/ap2D-a-*.fits.fz >>> = --------------------------------------------------------------------------= ------=20 >>> mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch = Offset %Flux Comm=20 >>> = --------------------------------------------------------------------------= ------ >>> 59036 1 10:40 34740009 Dark 60 A 13.0 1-12603 a-b-c = - - =20 >>> 59036 2 10:50 34740010 Dark 60 A 13.0 1-12603 a-b-c = - - =20 >>> 59036 3 11:01 34740011 Dark 60 A 13.0 1-12603 a-b-c = - - =20 >>> 59036 4 11:12 34740012 QuarFlat 10 A 13.0 1-12603 a-b-c = - 45=20 >>> 59036 5 11:14 34740013 QuarFlat 10 A 13.0 1-12603 a-b-c = - 45=20 >>> 59036 6 11:16 34740014 QuarFlat 10 A 13.0 1-12603 a-b-c = - 45=20 >>> 59036 7 11:17 34740015 Arc-Thar 12 A 13.0 1-12603 a-b-c = 3.90 - =20 >>> 59036 8 11:20 34740016 Arc-Une 40 A 13.0 1-12603 a-b-c = - - =20 >>> 59036 9 11:27 34740017 Arc-Thar 12 B 13.5 1-12603 a-b-c = 4.39 - =20 >>> 59036 10 11:29 34740018 Arc-Une 40 B 13.5 1-12603 a-b-c = - - =20 >>> 59036 11 11:37 34740019 Dark 30 A 13.0 1-12603 a-b-c = - - =20 >>> 59036 12 11:42 34740020 InteFlat 30 A 13.0 1-12603 a-b-c = - 0=20 >>> 59036 13 11:47 34740021 InteFlat 30 A 13.0 1-12603 a-b-c = - 0=20 >>> 59036 14 11:53 34740022 InteFlat 30 A 13.0 1-12603 a-b-c = - 0=20 >>> 59036 15 11:58 34740023 Dark 30 A 13.0 1-12603 a-b-c = - - =20 >>> = --------------------------------------------------------------------- >>> ----------- >>>=20 >>> ---- Problems in Detail ---- >>>=20 >>> Example of warnings we received while trying to focus: >>>=20 >>> 10:43:10 .tcc 0 tcc w Text=3D"Collimation Failure: Too many function = calls" >>> 10:43:10 .tcc 0 tcc w Text=3D"Collimation Failure: Too many function = calls" >>>=20 >>> in red: >>> 10:43:16 OBSERVER.aoravetz 42 tcc f text=3D"Sub-command(s) failed: = move nan,6.3439,11.2267,-39.8733,-19.2797: Too many function calls" >>> 10:43:16 CmdDone: OBSERVER.aoravetz 42 tcc set focus=3D-100.0/incr >>> 10:45:13 OBSERVER.aoravetz 44 tcc f text=3D"Sub-command(s) failed: = move nan,6.3248,11.2312,-39.8688,-18.3565: Too many function calls" >>> 10:45:13 CmdDone: OBSERVER.aoravetz 44 tcc set focus=3D-10.0/incr >>> 10:49:46 OBSERVER.aoravetz 50 tcc f text=3D"Sub-command(s) failed: = move nan,6.2785,11.2424,-39.8576,-16.1236: Too many function calls" >>> 10:49:46 CmdDone: OBSERVER.aoravetz 50 tcc set focus=3D0.0 >>> 10:50:36 OBSERVER.aoravetz 51 tcc f text=3D"Sub-command(s) failed: = move nan,6.2698,11.2445,-39.8555,-15.7052: Too many function calls" >>> 10:50:36 CmdDone: OBSERVER.aoravetz 51 tcc set focus=3D-10.0/incr >>>=20 >>> We now homed the secondary without any trouble. We watched the MiG=20= >>> numbers on TPM as well as on Mirror Status gui. >>>=20 >>> 11:20:05 OBSERVER.aoravetz 71 tcc f text=3D"Sub-command(s) failed: = move nan,7.7824,10.9473,-40.1527,-88.6688: Too many function calls" >>> 11:20:05 CmdDone: OBSERVER.aoravetz 71 tcc set focus=3D-10.0/incr >>>=20 >>>=20 >>> ____________________________________________________________ >>> Logs of Nightly APO Activities =20 >>> = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index. >>> html >>=20 >> ____________________________________________________________ >> Logs of Nightly APO Activities =20 >> = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.h >> tml >=20 >=20