From: Tracy Naugle Subject: Re: [sloan-25mlog 5682] Re: 2.5m obslog 20211010 (SJD 59498) Date: Mon, 11 Oct 2021 08:26:46 -0600 We are looking at a UPS that might have something to with it.. James is look= ing into it and can give you more information or I can once I know for sure.= Thanks Tracy Naugle Sent from my iPhone > On Oct 11, 2021, at 8:23 AM, Conor Sayres wrote: >=20 > =EF=BB=BFI just looked into the mirror stuff. >=20 > Seems like the galil was replying when I logged in, although the connectio= n felt very slow. I wonder if there is a network issue. >=20 > Restarted mirror controllers and tcc, and the mirrors showed up (although t= hey are reporting not homed). >=20 > Now it seems the tcc cannot connect to the telescope axes? It seems like t= he MCP is failing the =E2=80=9CINIT=E2=80=9D command that the TCC is sending= . So there=E2=80=99s still some weirdness going on. >=20 >> On Oct 11, 2021, at 6:41 AM, dmbiz@apo.nmsu.edu wrote: >>=20 >>=20 >> APACHE POINT OBSERVATORY SDSS 2.5M >>=20 >> OBSERVING LOG >> Sunday October 10, 2021 >>=20 >> (SJD 59498) >>=20 >> ---- Observing Team ---- >>=20 >> Dmitry Bizyaev >> Audrey Oravetz (Remotely) >>=20 >> Kaike Pan (remote troubleshooting) >> Jose Sanchez-Gallego (remote troubleshooting) >> Conor Sayres (remote troubleshooting) >> James Parker (remote troubleshooting) >>=20 >>=20 >> ---- Observing Plan ---- >>=20 >> Focus vs Altitude test >>=20 >>=20 >> ---- Observing Summary ---- >>=20 >> The entire night was lost to problems of a range of severity.=20 >>=20 >>=20 >> - We were unable to turn on FF lamps on the telescope=20 >> [JEP will fix it in the morning] >> - After successful evening calibrations we started getting=20 >> dark APOGEE frames, including the FPI one. Fixed, necessary >> calibrations taken. >> - Unable to connect to mirror controllers. We didn't fix it=20 >> tonight. Need Shane to check connection in the morning.=20 >> - PMSS vacuum gauge is out of spec. TraceyN thinks that=20 >> it is still OK, and he'll check it in the morning.=20 >>=20 >> - A problem from last nights: counterweight CW #2 reads 217 in MCP, >> but actually shows 438 on the box in the lower enclosure.=20 >>=20 >> - dust accumulated ~ 2K due to ventilation without opening in the evening= . =20 >>=20 >> ---- Observing Log ---- >>=20 >> Afternoon >> --------- >>=20 >> All systems were found responsive except for ecamera.=20 >> After it was turned on, neither reconnect nor startNubs >> were able to make it available. Restarted on sdss5-hub >> via systemctl, then started working.=20 >>=20 >> Upon checkout, 3 more problems discovered.=20 >>=20 >> 1. PMSS Vacuum gauge in the lower enclosure says 17, the spec range is 5-= 15. >>=20 >> 2. TPM gui cannot turn on FF lamp. The GUI shows that lamp is turning on,= >> but actually it is not. >>=20 >> 3. Both old and new LUI cannot grab mirror numbers, and bring up python e= rror: >>=20 >> Traceback (most recent call last): >> File "/home/tpm/sdss/tpm_linux/py/m4l.py", line 383, in=20 >> m4l() >> File "/home/tpm/sdss/tpm_linux/py/m4l.py", line 119, in __init__ >> self.m4lprint (nominalMigs, tpm, tcc) >> File "/home/tpm/sdss/tpm_linux/py/m4l.py", line 144, in m4lprint >> reply +=3D 'PrimDesOrient: ' + tcc['PrimDesOrient'] + '\n' >> KeyError: 'PrimDesOrient' >>=20 >> The actual m4l script just hangs silently when run.=20 >> I checked and cannot see the PrimDesOrient keyword broadcasted >> by the tcc. Although, 2500 port on the tcc machine is accepting >> and shows something reasonable. Not reporting PrimDesOrient >> may explain the m4l hanging. Investigating more.=20 >>=20 >> Also the Mirror Status in STUI shows "Sec state" as "iter 1 of 3" >> as in progress, infinitely.=20 >>=20 >> 4. At the end, after successful evening calibrations an attempt to get >> an FPI spectrum brought up a dark.=20 >> 23:50Z ventilation started.=20 >> Given the out of range PMSS parameter, I hesitate to move the telescope >> until talking to our engineers.=20 >>=20 >>=20 >> Night >> ----- >>=20 >>=20 >> The problems were worked on parallel to each other, so a threaded view=20= >> of them follows. =20 >>=20 >> 1. The PMSS Vacuum gauge is out of spec. Pictures were sent to Tracey.=20= >> He thinks that it is safe to move the telescope in these conditions.=20 >> He'll take a look at this issue tomorrow.=20 >>=20 >> 2. FF lamps. Turning them on and off in STUI and TPM showed that they are= on, but the actual inspection on the telescope revealed that the FF lamps a= re not turning on, any of them. At the same time, Ne and HgCd lamps work as e= xpected.=20 >>=20 >> We blamed MCP first, and restarted the tlamp process, than all mcp. It di= d not help. >> Contacted James Parker. He logged in remotely, but was unable to see a ca= use of the problem. It sounds like it is a hardware problem. He'll work on i= t tomorrow.=20 >>=20 >> The problem prevents us from taking flat field images with the engineerin= g camera.=20 >>=20 >> 4. We checked the calbox and did not see anything unusual in the status o= f power supplies or iBootBar. After Jose restarted apogeecal actor, and we r= einitialized apogee, without success, the APOGEE shutters were suspected. We= opened/closed each of the shutters three times, and then were able to get s= uccessful spectra with signal.=20 >> 3. The superficial manifestation of the problem as inability to get numbe= rs >> or status keywords from mirrors were traced down to problems with tcc. R= estarting tcc did not help, s well as clearing ports. With Joe and Conor in S= TUI we tried to track down the problem, and finally Conor was unable to talk= to the mirror controllers. The stopping/starting tcc/mirror services on sds= s5-tcc machine was able to restore connection to axes, but not to the mirror= controllers. Same message is repeating every 10 seconds: >> 07:38:55 .tcc 0 tcc w mirrorConnState=3DConnecting,Connecting,NotAvailabl= e >> and the "connecting" never ends.=20 >>=20 >> Conor suspects that there is no connection because of either terminal ser= ver, or the galil box is out of power, or there is no connection between the= m.=20 >> Visual inspection of the terminal server Xyplex did not show any signs of= >> problems: wires are connected tightly, some LEDs are on, the Xyplex is po= wered on. We also reseted the galil box, with no signs of improvement.=20 >>=20 >> Also, on the secondary we see >> 0 2 i yourUserID=3D2; numUsers=3D2 >> 0 2 i UserInfo=3D1, 127.0.0.1 >> 0 2 i UserInfo=3D2, 10.25.1.139 >> 0 2 i version=3D"1.1.3" >> 0 2 i text=3D'mirror name: mir25mSec' >> status >> 0 2 i yourUserID=3D2; numUsers=3D2 >> 0 2 i UserInfo=3D1, 127.0.0.1 >> 0 2 i UserInfo=3D2, 10.25.1.139 >>=20 >> 0 2 w galilConnState =3D 'Failed', "[Failure instance: Traceback (failure= with no frames): : C= onnection was refused by other side: 111: Connection refused. >> ]" >> 0 2 f text=3D"Device Not Connected" >>=20 >> and the same on the primary: >> 0 2 i UserInfo=3D1, 127.0.0.1 >> 0 2 i UserInfo=3D2, 10.25.1.139 >> 0 2 i version=3D"1.1.3" >> 0 2 i text=3D'mirror name: mir25mPrim' >> status >> 0 2 i yourUserID=3D2; numUsers=3D2 >> 0 2 i UserInfo=3D1, 127.0.0.1 >> 0 2 i UserInfo=3D2, 10.25.1.139 >> 0 2 w galilConnState =3D 'Failed', "[Failure instance: Traceback (failure= with no frames): : C= onnection was refused by other side: 111: Connection refused. >> ]" >> 0 2 f text=3D"Device Not Connected" >>=20 >> At the same time, the address 10.25.1.139 is alive: >> (observers39) [observer@sdss5-hub: ~]$ ping 10.25.1.139 >> PING 10.25.1.139 (10.25.1.139) 56(84) bytes of data. >> 64 bytes from 10.25.1.139: icmp_seq=3D1 ttl=3D64 time=3D0.028 ms >> 64 bytes from 10.25.1.139: icmp_seq=3D2 ttl=3D64 time=3D0.039 ms >> 64 bytes from 10.25.1.139: icmp_seq=3D3 ttl=3D64 time=3D0.031 ms >>=20 >>=20 >> After a few hour of attempts it was decided to continue tomorrow,=20 >> with Shane online. A message was sent to Shane.=20 >>=20 >> 05:30Z ventilation stopped. Doors closed.=20 >>=20 >> 10:36Z called the night >>=20 >> APOGEE calibrations started, >>=20 >> ecamera is turned off >>=20 >> STUI memory usage on the podium laptop is 8Gb;=20 >> I hope it will hold during the day >>=20 >>=20 >> ---- Mirror Numbers ---- >>=20 >> Traceback (most recent call last): >> File "/home/tpm/sdss/tpm_linux/py/m4l.py", line 383, in >> m4l() >> File "/home/tpm/sdss/tpm_linux/py/m4l.py", line 119, in __init__ >> self.m4lprint (nominalMigs, tpm, tcc) >> File "/home/tpm/sdss/tpm_linux/py/m4l.py", line 144, in m4lprint >> reply +=3D 'PrimDesOrient: ' + tcc['PrimDesOrient'] + '\n' >> KeyError: 'PrimDesOrient' >>=20 >> After attempts to fix (see log), we've got the following mirror >> numbers. They look unreal, until fixed.=20 >>=20 >> -------- >> PRIMARY: >> -------- >> Scale: 4 >>=20 >> MIGS TONIGHT NOMINAL >> Axial A: 0.0050 0.0080 >> Axial B: 0.0310 0.0330 >> Axial C: -0.1140 -0.1140 >> Trans D: -0.0375 -0.0395 >> Lateral E: 0.3310 0.3310 >> Lateral F: 0.5345 0.5335 >>=20 >> GALILS >> Commanded: 12500 -1600 5350 47050 -8800 -8400 >> Actual: 12500 -1600 5350 47050 -8800 -8400 >>=20 >> SETMIR VALUES >> PrimDesOrient: 0 >> PrimOrient: 0 >>=20 >> SECONDARY: >> ---------- >>=20 >> Focus: 48 >> Air temp: 6.44 >> Altitude: 30.000035067407072 >>=20 >> MIGS TONIGHT NOMINAL >> Axial A: -9.3030 -9.4610 >> Axial B: -9.3300 -9.4760 >> Axial C: -9.3570 -9.5050 >> Trans D: 0.4190 0.0880 >>=20 >> GALILS >> Commanded: 4446250 4454350 4474200 13500 10100 >> Actual: 4422916 4420371 4421612 13500 10100 >>=20 >> SETMIR VALUES >> SecDesOrient: 0 >> SecOrient: 0 >>=20 >> ---- Telescope Status ---- >>=20 >> 23:50Z Started ventilation >> 05:30Z Stopped ventilation >>=20 >> Never open >>=20 >> No interlocks bypassed >>=20 >> Status at: 10:36Z >> Telescope stowed at: 121.0, 30.0, 0.0 mount >> Instrument mounted: cartridge =3D engcam =20 >> Counterweights at: 438 >> LN2 autofill systems: Connected and turned on >> 180L LN2 dewar scales: SP1 146 lbs, 22 psi >>=20 >>=20 >>=20 >>=20 >> ---- Software Used ---- >>=20 >>=20 >> TUI : 1.7.13 >> alerts : 2.1.6 >> apo : 3.0.0 >> boss : 6.0.0 >> -boss.specMec : 3.0.4 >> guider : 4.0.0 >> ecamera : FAILED >> fliswarm : 0.1.1-alpha.0 >> hartmann : 2.0.0 >> hub : 5.0.0a0 >> mcp : 2.0-7-g8799 >> -mcp.plc : 34 >> -mcp.azFiduci : v1_146 >> -mcp.altFiduc : v1_146 >> -mcp.rotFiduc : v1_146 >> sop : 0.1.0-alpha.0 >> tcc : 1.4.0 >> apogee : not availble >> apogeecal : not availble >> apogeeql : v1_7_0dev >> STUIAdditions : origin/sdss5 >>=20 >>=20 >> /dev/sdb1 2.0T 95G 2.0T 5% /home >> 10.25.1.230:/data 21T 15T 5.9T 71% /data >>=20 >>=20 >> ---- APOGEE Data Summary ---- >>=20 >> Last morning: >>=20 >> 59497 11 10:12 39350011 Dark 60 A 13.0 ------- a-b-c - - = =20 >> 59497 12 10:22 39350012 Dark 60 A 13.0 ------- a-b-c - - = =20 >> 59497 13 10:33 39350013 Dark 60 A 13.0 ------- a-b-c - - = =20 >> 59497 14 10:35 39350014 QuarFlat 10 A 13.0 ------- a-b-c - 99= =20 >> 59497 15 10:37 39350015 QuarFlat 10 A 13.0 ------- a-b-c - 99= =20 >> 59497 16 10:38 39350016 QuarFlat 10 A 13.0 ------- a-b-c - 100= =20 >> 59497 17 10:41 39350017 Arc---- 12 A 13.0 ------- a-b-c - - = =20 >> 59497 18 10:48 39350018 Arc---- 40 A 13.0 ------- a-b-c - - = =20 >> 59497 19 10:50 39350019 Arc---- 12 A 13.0 ------- a-b-c - - = =20 >> 59497 20 10:57 39350020 Arc---- 40 A 13.0 ------- a-b-c - - = =20 >> 59497 21 11:03 39350021 Dark 30 A 13.0 ------- a-b-c - - = =20 >> 59497 22 11:08 39350022 InteFlat 30 A 13.0 ------- a-b-c - 279= =20 >> 59497 23 11:13 39350023 InteFlat 30 A 13.0 ------- a-b-c - 279= =20 >> 59497 24 11:19 39350024 InteFlat 30 A 13.0 ------- a-b-c - 280= =20 >> 59497 25 11:24 39350025 Dark 30 A 13.0 ------- a-b-c - - = =20 >>=20 >> This evening: >> -------------------------------------------------------------------------= -----=20 >> mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux= Comm=20 >> -------------------------------------------------------------------------= ----- >> 59498 1 23:32 39360001 Dark 3 A 13.0 ------- a-b-c - - = =20 >> 59498 2 23:58 39360002 Dark 60 A 13.0 ------- a-b-c - - = =20 >> 59498 3 00:00 39360003 QuarFlat 10 A 13.0 ------- a-b-c - 98= =20 >> 59498 4 00:03 39360004 Arc---- 12 A 13.0 ------- a-b-c - - = =20 >> 59498 5 00:10 39360005 Arc---- 40 A 13.0 ------- a-b-c - - = =20 >> 59498 6 00:12 39360006 Arc---- 12 A 13.0 ------- a-b-c - - = =20 >> 59498 7 00:19 39360007 Arc---- 40 A 13.0 ------- a-b-c - - = =20 >> 59498 8 00:30 39360008 Dark 60 A 13.0 ------- a-b-c - -=20= >> All darks below, see log - we had problems=20 >> 59498 9 01:44 39360009 Arc---- 30 A 13.0 ------- a-b-c - - = FPI >> 59498 10 02:01 39360010 Arc---- 5 A 13.0 ------- a-b-c - - = test >> 59498 11 03:05 39360011 Arc---- 3 A 13.0 ------- a-b-c - - = test >> 59498 12 04:13 39360012 Arc---- 5 A 13.0 ------- a-b-c - - = test >> 59498 13 04:16 39360013 Arc---- 10 A 13.0 ------- a-b-c - - = test >> 59498 14 04:45 39360014 Arc---- 3 A 13.0 ------- a-b-c - - = test >> 59498 15 04:47 39360015 Arc---- 5 A 13.0 ------- a-b-c - - = test >> All darks above because of problems.=20 >> The spectra below have signal. >> 59498 16 04:52 39360016 Arc---- 5 A 13.0 ------- a-b-c - - = test >> 59498 17 04:57 39360017 Arc---- 20 A 13.0 ------- a-b-c - - = FPI >> -------------------------------------------------------------------------= -----=20 >>=20 >>=20 >>=20 >> ---- eBOSS Data Summary ---- >>=20 >> Reading FITS from /data/spectro//59498 >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -= - -=20 >> n Time Filename crt plt flav shu Exp colA colB colC Hrt >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -= - -=20 >> 1 23:31 b1-00333957 -1 -1 bia Clo 0.0 0 0 0 =20 >> 2 23:31 r1-00333957 -1 -1 bia Clo 0.0 0 0 0 =20 >> 3 23:39 b1-00333958 -1 -1 dar Clo 20.0 0 0 0 =20 >> 4 23:39 r1-00333958 -1 -1 dar Clo 20.0 0 0 0 =20 >>=20 >>=20 >> ____________________________________________________________ >> Logs of Nightly APO Activities >> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html= >=20 > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html