Subject: 2.5m obslog 20210820 (SJD 59447) From: doravetz@apo.nmsu.edu Date: Sat, 21 Aug 2021 10:44:14 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday August 20, 2021 (SJD 59447) ---- Observing Team ---- Dmitry Bizyaev Dan Oravetz Ryan Wagner (support) ---- Observing Plan ---- Check voltages and fluxes on Th-Ar and UNe lamps. Pointing Model. ---- Observing Summary ---- Proper APOGEE ThAr and UNe STUI control restored after swapping AC cords. Home/relax mirrors completed. After home/relax completed we were unable to connect to engCam; we kept getting a "no route to host" error. We spent most of the night swapping fiber optic network cables behind the rotator within a PMSS panel with Ryan's direction and also tried two different fiber-to-copper converter switches. Eventually we found that the problem seems to have been a lack of fiber connection downstairs in the network cabinet at a converter switch. (See Log for details). After spending the night diagnosing the network issues we did not have time left for pointing model work. ---- Observing Log ---- Afternoon --------- Checked what we can check. Corrected fiducials on AZ and ALT axes. Checked APOGEE calibration lamps and found that they are mis-wired. When I turn on Th-Ar from STUI, the power goes to UNe block (marked, upper). Same, if UNe is turned on in STUI, Th-Ar block is active (marked, the bottom one). Notified John Wilson. Evening APOGEE calibrations postponed to not damage APOGEE chips with long bright Th-Ar exposures. 00:25Z ventilation started. John Wilson sent an instruction on how to swap power cords on the iBootBar. The power cords are swapped. Calibrations taken (with 10 reads) and show correct behavior of the lamps 38850006 (ThAr) and 38850007 (UNe). Standard evening calibration started. The floor in the upper enclosure broomed - some metal shavings (swarf) noticed. Night ----- Homing M1 E, F "position error", "999999999, 999999999, 999999999, 999999999, 000000050, 000000000" D "position error", "999999999, 999999999, 999999999, 000000000, 999999999, 999999999" A, B, C "position error", "-000000050, -000000050, 000000000, 999999999, 999999999, 999999999" Homing M2 D "position error", "999999999, 999999999, 999999999, 000000000, 999999999" E "position error", "999999999, 999999999, 999999999, 999999999, -000000050" A, B, C "position error", "-000000108, -000000114, -000000058, 999999999, 999999999 (after 3 more home commands) To this point the home-relax procedures are done Errors: We couldn not connect to the engCam: ERROR: curl_easy_perform error(7 in buffer 1, size=0): couldn't connect to host. URL :http://10.25.1.71:80/SESSION?Open: ERROR: curl_easy_perform error(7 in buffer 1, size=0): couldn't connect to host. URL :http://10.25.1.71:80/SESSION?Close: 2021-08-21 02:50:34.785Z cmds 20 CommandLink.py:116 > 0 0 w text="BAD THING: could not connect to camera: Alta camera connection is down" 2021-08-21 02:50:34.786Z cmds 20 CommandLink.py:116 > 0 0 w text="BAD THING: failed to connect to camera! Try 'gcamera reconnect', I suppose. Alta camera connection is down" ERROR: curl_easy_perform error(7 in buffer 1, size=0): couldn't connect to host. URL :http://10.25.1.71:80/SESSION?Close: APOGEE.DLL - CApnCamera::~CApnCamera()2021-08-21 02:50:36.802Z cmds 20 CommandLink.py:116 > 2 4 i version="v2_2" 2021-08-21 02:50:36.802Z cmds 20 CommandLink.py:116 > 2 4 i stack=1 2021-08-21 02:50:36.803Z cmds 20 CommandLink.py:116 > 2 4 w cameraConnected=False 2021-08-21 02:50:36.803Z cmds 20 CommandLink.py:116 > 2 4 i simulating=Off,None,1 2021-08-21 02:50:36.803Z cmds 20 CommandLink.py:116 > 2 4 : 2021-08-21 02:50:38.958Z cmds 20 Actor.py:469 new cmd: Command(source=01725950 cmdr=OBSERVER.dmbiz cid=2 mid=5 rawCmd=setTemp temp=-35) 2021-08-21 02:50:38.972Z cmds 20 Actor.py:426 2 5 i text="setting camera cooler setpoint to -35.0 degC" 2021-08-21 02:50:38.972Z cmds 20 CommandLink.py:116 > 2 5 f text=Camera not connected: cannot set temperature. 2021-08-21 02:50:38.972Z cmds 20 CommandLink.py:116 > 2 5 w text="this command has already been finished!!!! (OBSERVER.dmbiz 5): setTemp temp=-35" We contacted Ryan and with his help we spent most of the night swapping fiber optic network cable connections and tried two different gigabit converter switches. We visually verified the continuity of the #7/#8 fibers that run from the fiber joining panel located within the back of the PMSS - when shining a light through the ends that connect to the north side of the patch panel, light is seen at the ends that connect to the west converter switch that's in the rotator wrap. Still, the camera is not connecting and we get a "no route to host" error. Next step will be to check in the lower enclosure for something causing the connection problem. The problem seems to have been a lack of connection downstairs in the network cabinet. There's a gigabit converter switch that sits below the Xyplex terminal server. The fiber optic patch cables feeding that converter switch (#7/#8) were NOT connected to the fiber optic cables that now go to the engineering camera (#3/#4 on the south side of the fiber joining plate within the PMSS). Here is the CURRENT STATE of things with gcamera and ecamera working: engCam --> RJ45 ethernet --> (West) CSG14H converter switch --> fiber optic patch cables #7/#8 --> fiber joining plate inside PMSS --> fiber optic patch cables #3/#4 --> lower enclosure network rack --> fiber optic patch cable #4 goes to black ST connector that feeds the lower-enclosure CSG14H converter switch & fiber optic patch cable #3 goes to red ST connector on that switch --> RJ45 ethernet to network switch Port #10 ORIGINALLY, it seems things were wired like this: fibers #7/#8 inside the rotator wrap mated with #7/#8 at the fiber joining plate and then in the lower enclosure #7/#8 from the south side of the joining plate fed the lower-enclosure CSG14H switch -- the numbering was consistent from point to point: OLD state: engCam --> RJ45 ethernet --> (West) CSG14H converter switch --> fiber optic patch cables #7/#8 --> fiber joining plate inside PMSS --> fiber optic patch cables #7/#8 --> lower enclosure network rack --> fiber optic patch cable #8 went to black ST connector that feeds the lower-enclosure CSG14H converter switch & fiber optic patch cable #7 went to red ST connector on that switch --> RJ45 ethernet to network switch Port #10 When connections got moved during the FVC work as a result of broken fibers, the fiber connection number-matching got disrupted at the fiber joining plate when spare connections were being utilized. After Ryan directed us to try some different fiber connections at the joining plate we ultimately ended with #7/#8 in the rotator wrap mated with #3/#4 at the fiber joining plate, but in the lower enclosure the converter switch was still looking for #7/#8 instead of #3/#4, so it wasn't until we swapped-in #3/#4 that the engCam/gcamera talked. For the record, the #3/#4 fibers WERE plugged into the #12/#11 (respectively) spots on the upper fiber patch panel located at the top of the lower-enclosure network cabinet. We unplugged #3/#4 from those locations and moved them to the converter switch. Also, the #7/#8 patch cables that WERE plugged into the lower-enclosure converter switch are now not connected to anything. We took some pictures and notes, so we can put things back (or direct someone) if we need to revert anything, but we left things in the working "CURRENT" state described above. One more note: the #3/#4 fiber pair on the ROTATOR-WRAP SIDE of the fiber joining panel doesn't appear to go to anything -- they are coiled up in the wrap and capped. So, hopefully, having commandeered the #3/#4 ports at the fiber joining panel for use by the engCam/gcamera converter switch doesn't cause a problem. 10:42Z - Called it a night. Not enough time to start pointing model work. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0080 0.0080 Axial B: 0.0330 0.0330 Axial C: -0.1180 -0.1140 Trans D: -0.0393 -0.0395 Lateral E: 0.3310 0.3310 Lateral F: 0.5325 0.5335 GALILS Commanded: 12500 -1600 5350 47050 -8800 -8400 Actual: 12500 -1600 5350 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.3 -8.75 -512.73 1906.51 PrimOrient: 0.01 -59.26 -8.77 -512.68 1906.6 SECONDARY: ---------- Focus: 0.0000 Air temp: 17.06 Altitude: 29.999996178438877 MIGS TONIGHT NOMINAL Axial A: -9.4560 -9.4610 Axial B: -9.4690 -9.4760 Axial C: -9.5010 -9.5050 Trans D: 0.0880 0.0880 GALILS Commanded: 4630650 4625350 4647850 0 -3400 Actual: 4607174 4607218 4607149 0 -3400 SETMIR VALUES SecDesOrient: 3657.0 0.0 0.0 0.0 42.22 SecOrient: 3656.95 0.02 0.0 0.0 41.45 ---- Telescope Status ---- Status at close: Telescope stowed at: 121.0, 30.0, -100 mount Instrument mounted: engCam Counterweights at: 398 402 400 400 ---- Software Used ---- STUI : 1.7.13 alerts : trunk apo : trunk boss : unknown -boss.specMec : Offline gcamera : v2_2 guider : unknown hartmann : 1.8.0 hub : Tag: v4.2 mcp : 2.0-7-g8799 -mcp.plc : 34 -mcp.azFiduci : v1_146 -mcp.altFiduc : v1_146 -mcp.rotFiduc : v1_146 platedb : 2.8.0 sop : 4.0.8 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : unknown STUIAdditions : 3.1.7 Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 654G 34G 96% /home 10.25.1.230:/data 21T 14T 7.0T 66% /data Setting up: idlspec2d Flavor: Linux64 Version: v5_7_12 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: trunk Setting up: |specflat Flavor: Linux64 Version: v1_20 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |sdss_python_module Flavor: Linux64 Version: v1_6_7 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.9.7 Setting up: |psycopg2 Flavor: Linux64 Version: 2.5.3 Setting up: |idlutils Flavor: Linux64 Version: v5_5_16 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: |photolog Flavor: Linux64 Version: trunk Setting up: actorcore Flavor: Linux64 Version: v4_1_6 Setting up: |opscore Flavor: Linux64 Version: v2_5 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v30_2 Setting up: |sdsstools Flavor: Linux64 Version: v4_33 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_13 Setting up: |RO Flavor: Linux64 Version: 3.6.9 --- Other versions --- idlspec2d : v5_7_12 sdss_python_module: trunk