Subject: 2.5m obslog 20210414 (SJD 59319) From: kpan@apo.nmsu.edu Date: Thu, 15 Apr 2021 11:04:42 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday April 14, 2021 (SJD 59319) ---- Observing Team ---- Viktor Malaushenko Kaike Pan Conor Sayres (tcc) John Donor (Petunia) ---- Observing Plan ---- Wed 14 Apr 2021 59319 0.09 02:43 11:22 09:14 17:54 --- KK BHM ( 8.6 h) ---- Observing Summary ---- The entire night was lost to clouds. BHM lost 8.6 hours. Note: ==== 1. Half dozen actors, three web pages were left in non-working status this evening, it took us two calls, 2.5 hours to get things back. See the log for more details. ---- Observing Log ---- Afternoon: ========== When we came in at 5:00 PM, a half dozen actors were found not running. Restarting actors is not a bid deal, but not starting actors usually does not guarantee that they will work properly, especially when they are not stopped properly. For example, statuses of tcc and mcp show that they are both fine, but they are not talking each other, BOSS and SOS are ok, but SOS does reduce new data. status checks indicate both tcc and MCP were ok, but STUI shows axes were disconnected. "tcc axis connect" could not get them connected. We restarted tcc, no change. Then we decided to check MCP, the running version is /home/mcp /sdss/tagged/2.0, all processes were running, nothing was found out of the spec. We decided to restart it anyway. During the "mcp start", we saw some unusual messages: connect: connection refused waiting for tmtcc_az to stop ... waiting for tmtcc_az to stop tmtcc_az process did not stop, skipping tmtcc_alt stop process requested waiting for tmtcc_alt to stop ... waiting for tmtcc_alt to stop tmtcc_alt process did not stop, skipping tmtcc_rot stop process requested waiting for tmtcc_rot to stop ... waiting for tmtcc_rot to stop tmtcc_rot process did not stop, skipping This restarting didn't resolve the issue, and the MCO console keeps rolling the message: connect connection refused. We decided to add "mcp reset" before "mcp start", After MCP restarting, we still cloud not connect axes, and all three axes have position of zero (we recorded all axis positions before restart MCP). Then we called Conor who suggested us to stop tcc, then restart MCP to see if we could move the telescope with MCP. We did so. After the MCP restarting, AZ and rotator have their positions back correctly (121 and 0.0). Altitude axis position was set to be 43.65 when it was at 30:00:58.4. We saw the differences, so we went ahead to restart tcc, then"tcc axis connect" worked. At this point, thing seems to work except that we still need to reset altitude position. However, we still was the weird message of "connect: connection refused" on the MCP console. Then Conor suggested us to stop MCP, then load specific version of 2.0 with "module load mcp/2.0" before restart it. We did so, but still see the weird message. Because things seems work, we went ahead reset altitude position, issue fake slews with e-Stop engaged. Things worked. Now the only thing that worries us is the weird message on the MCP console. We thought that it might be left over for the very first MCP restarting, So, we logout from MCP, and logged back in to see if we still see these messages. No, no weird messages any more. I decided to do test slews and mounted the first cartridge of the night at about 6:50 PM. Our next issue is Petunia was not running. After restarted it, we still see "service is not available". I called John Donor who started nginx with "sudo" which observers don't have password for. Then Petunia started to work. We also need to restart Apogee webapp and SOS to get them work properly.We got things back at 7:30 PM. [observer@sdss4-hub ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/vda2 7.7G 6.5G 911M 88% / tmpfs 3.4G 0 3.4G 0% /dev/shm /dev/vda1 194M 133M 52M 73% /boot /dev/vdb 19G 8.8G 9.1G 49% /exports/hub-local /dev/vdd 688G 604G 84G 88% /home /dev/vdc1 11T 6.8T 4.0T 63% /data 10.25.1.230:/data 21T 11T 10T 51% /sdss5_data 23:35Z apogee short dark -ok 23:38Z apogee evening cals started 23:32Z boss biases -ok 23:41Z guider dark -ok 23:42Z - sop collimateBoss 23:45:31 Hartmann collimate output on cart #9 sp1: offset: r= 329 (In Focus); b= -7.3 (Out of focus) sp1: pred. move: spAverageMove= 1321 sp1: pred. resid: r= -992, b= -3.1, txt= OK, spTemp = 7.1 ---------- motorPosition sp1: 5309, 5280, 5373; ---------- Median spectrographs temperatures: Tsp1= 7.1 Outside air temperature= 12.7; ---------- 01:44:27 sop doBossCalibs nflat=1 narc=1, files=329455,329456 BOSS collimation was checked with cart 9. No adjustment was made. Night: ===== 01:30Z: Thick clouds, not open. 08:00Z: Stopped ventilation. 11:00Z: started morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: -4.9270 -4.9280 Axial B: -4.6670 -4.6660 Axial C: -5.7680 -5.7770 Trans D: -4.3250 -4.3890 Lateral E: 0.3745 0.3750 Lateral F: 0.5710 0.5675 GALILS Commanded: 5850 -4500 14850 28200 25200 24900 Actual: 5850 -4500 14850 28200 25200 24900 SETMIR VALUES PrimDesOrient: 0.0 69.89 100.0 375.3 622.11 PrimOrient: 0.04 69.89 100.0 375.3 622.11 SECONDARY: ---------- Focus: 0.0000 Air temp: 10.78 Altitude: 43.508118169747355 MIGS TONIGHT NOMINAL Axial A: -9.3910 -9.4420 Axial B: -9.3550 -9.4070 Axial C: -9.4300 -9.4960 Trans D: 0.1080 0.0420 GALILS Commanded: 4550450 4529900 4593100 850 -4000 Actual: 4540453 4519496 4562563 850 -4000 SETMIR VALUES SecDesOrient: 3603.13 -12.45 -3.95 15.02 47.37 SecOrient: 3604.25 -12.49 -4.05 14.98 46.59 ---- Telescope Status ---- 01:00Z Started ventilation N/A encl25m : open N/A encl25m : closed No interlocks bypassed Status at: 11:10Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 7 Counterweights at: 445 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 98 lbs,22.8 psi ---- Software Used ---- Get Versions, 04/14/21, 23:44:30 -- STUI software: STUI : 1.7.12 alerts : 2.0.1 apo : v1_11 boss : 5.0.1 -boss.specMec : 3.0.4 gcamera : v2_2 guider : 3.9.4 hartmann : 1.8.0 hub : v4.2.4 mcp : 2.0 -mcp.plc : 34 -mcp.azFiduci : v1_144 -mcp.altFiduc : v1_144 -mcp.rotFiduc : v1_144 platedb : 2.8.0 sop : 4.0.8 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.5 -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v7_0_9 petunia : 2.2.3 autoscheduler : 1.5.4 sdss_python_module: trunk mangadrp : trunk exported