Subject: 2.5m obslog 20200209 (SJD 58889) From: kpan@apo.nmsu.edu Date: Mon, 10 Feb 2020 12:46:39 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday February 09, 2020 (SJD 58889) ---- Observing Team ---- Viktor Malanushenko Kaike Pan Elena Malanushenko - afternoon and STATUS script ---- Observing Plan ---- Sun 9 FEB 2020 58889.0 1.00 01:39 12:56 03:54 15:14 eng - Night APOGEE-2 (11.3 h) ---- Observing Summary ---- The entire night is lost to clouds and snow. APOGEE lost 11.3 hours. We restarted tron and took test exposures. Reconnected APOGEE and telemetry. It is snowing. We removed snow on the path to the enclosure, placed some absorbent pads on the floor in the upper enclosures. Notes: ======= 1. We experienced another wind_AZ_ready issue during afternoon check.This one took 30 minutes to get the bit turning green. We tried power cycle the wind baffle power switch and cycled all e-stop buttons, none of them worked. The last attempt, power cycling the wind baffle switch turn the wind_AZ-ready green. We still don't understand why the last attempt worked. (see log for more info). ---- Observing Log ---- Afternoon ------------- All actors tested, all looks good [observer@sdss4-hub ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 537G 151G 79% /home > 2G /dev/vdc1 11T 9.0T 1.8T 84% /data > 60G 22:48:39 Apogee short dark and evening cals have been started Spectro bias and dark: 22:48:43 boss exposure 0.0, file=308378, C 22:50:04 boss exposure 20.0, file=308379, C DOS is alive Guider dark is ok Mirror numbers are in spec. Spectro collimation: 22:55:43 Hartmann collimate output on cart #2 sp1: offset: r= -2638 (Out of focus); b= 4.8 (Out of focus) sp2: offset: r= 1893 (Out of focus); b= -3.0 (In Focus) sp1: pred. move: spAverageMove= -2090 sp2: pred. move: spAverageMove= 1419 sp1: pred. resid: r= -548, b= -1.7, txt= OK, spTemp = 4.3 sp2: pred. resid: r= 474, b= 1.5, txt= OK, spTemp = 5.3 collimation is in spec motorPosition sp1: 17928, 17112, 15485; sp2: 18539, 18301, 18153; Median spectrographs temperatures: Tsp1= 4.3, Tsp2= 5.2; Outside air temperature= 5.1; Spectro calibrations 23:00:04 boss exposure 25.0, file=308382, C 23:02:51 boss exposure 4.0, file=308383, C Cals are in spec Petunia is alive. Cart 2 is the first cartridge for the night, and it is currently mounted. We experienced another wind_AZ_ready problem. The bit would not cleared. We tried cycling all e-stop buttons, included ones in lower, upper, the control room, on rails, along with power cycling the wind-baffle switch several times. None of these combinations worked. We actually decided to call JohnD. But just before we left the enclosure, Viktor decided to power cycle the wind baffle switch one more time, and magic happened. Before power cycling the wind baffle switch, an "tcc axis init" was executed, all plc bits were cleared except for the wind-AZ-ready. After the power cycling, the wind-AZ-ready became green immediately (without "axis init"). During the trouble shooting process, we checked mcp processes and confirmed all are running. and checked wind screen PLC control box for LED lights, and wind baffle motor box several times, we noticed that the wind baffle motor box shows "AL _ _ _" when the wind_AZ_ready is red,the same code as wind_AZ_ready is green (things work normally). The screen PLC control box LED ligh configuration doesn't change neither. Night: ===== We restarted tron. Then reconnected telemetry and apogee. We find that the STUI script doesn't work after the tron restarting, It complains: Failed: unsupported operand type(s) for -: 'float' and 'str'. Elena investigated on the issue, and found that bossModel.SP1R0CCDTempNom, bossModel.SP1B2CCDTempNom, bossModel.SP2R0CCDTempNom and bossModel.SP2B2CCDTempNom did not have values, and the script set them \"n/a\", string. After a while the script work properly again. We suspect that these nominals values may be updated with "boss camcheck" (we tried boss status and specstat, they did help). Anyway, in the worse case, we may just wait a while to have the nominal values being updated by whatever system checks before the script will work again. ---- BOSS Spectrographs Collimation Summary ---- Afternoon Time Inst r1 b1 move1 b1pred Tsp1 r2 b2 move2 b2pred Tsp2 -------------- ---------------------------- ---------------------------- 22:54 2-12263A -2638 4.8 -2090 -1.7 4.3 1893 -3.0 1419 1.5 5.3 all in spec Night ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: -5.33 -5.332 Axial B: -5.037 -5.031 Axial C: -5.271 -5.27 Trans D: -0.582 -0.579 Lateral E: 9.3599 9.3472 Lateral F: 14.5161 14.4907 GALILS Commanded: 11200 -1850 6850 29650 21250 21050 Actual: 11200 -1850 6850 29650 21250 21050 SETMIR VALUES PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 PrimOrient: 0.0 -11.4 6.58 375.38 620.77 SECONDARY: ---------- Focus: 0.0000 Air temp: 6.1 Altitude: 30.1 MIGS TONIGHT NOMINAL Axial A: -6.661 -6.661 Axial B: -0.028 -0.023 Axial C: -6.659 -6.661 Trans D: -0.066 -0.07 GALILS Commanded: 1093650 1132800 1133000 -7250 -6550 Actual: 1058693 1101244 1079600 -7250 -6550 SETMIR VALUES SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 SecOrient: 856.94 2.64 12.73 -38.44 156.76 ---- Telescope Status ---- Started ventilation encl25m : open, never open. encl25m : closed No interlocks bypassed Status at: 12:50Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 2 Counterweights at: LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 177 lbs, 21 psi SP2 178 lbs, 21 psi ---- Software Used ---- Get Versions, 02/09/20, 22:53:15 -- STUI software: STUI : 1.7.8 alerts : 0.0.1 apo : trunk boss : v4_1_2 -boss.specMec : 3.0.4 gcamera : v2_2 guider : 3.8.1 hartmann : 1.7.1 hub : v4.2.3 mcp : 1.10-6-g6450 -mcp.plc : 34 -mcp.azFiduci : v1_141 -mcp.altFiduc : v1_141 -mcp.rotFiduc : v1_141 platedb : 2.8.0 sop : 3.12.7 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_6_1 APO-local : not availble -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_12 petunia : 1.5.0 autoscheduler : 1.4.1 sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- Last night morning calibrations: [observer@sdss4-hub ~]$ list_ap -m 58888 -morning APOGEE data list, mjd=58888, morning raw_data: /data/apogee/utr_cdr/58888/apRaw-*.fits archive: /data/apogee/archive/58888/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/58888/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 58888 1 13:15 33260089 Dark 60 B 13.5 5-12323 a-b-c - - 58888 2 13:26 33260090 Dark 60 B 13.5 5-12323 a-b-c - - 58888 3 13:36 33260091 Dark 60 B 13.5 5-12323 a-b-c - - 58888 4 13:47 33260092 QuarFlat 10 B 13.5 5-12323 a-b-c - 30 58888 5 13:49 33260093 QuarFlat 10 B 13.5 5-12323 a-b-c - 31 58888 6 13:51 33260094 QuarFlat 10 B 13.5 5-12323 a-b-c - 31 58888 7 13:53 33260095 Arc-Thar 12 A 13.0 5-12323 a-b-c -6.37 - 58888 8 13:55 33260096 Arc-Une 40 A 13.0 5-12323 a-b-c - - 58888 9 14:02 33260097 Arc-Thar 12 B 13.5 5-12323 a-b-c -5.87 - 58888 10 14:04 33260098 Arc-Une 40 B 13.5 5-12323 a-b-c - - 58888 11 14:12 33260099 Dark 30 A 13.0 5-12323 a-b-c - - 58888 12 14:17 33260100 InteFlat 30 A 13.0 5-12323 a-b-c - 0 58888 13 14:22 33260101 InteFlat 30 A 13.0 5-12323 a-b-c - 0 58888 14 14:28 33260102 InteFlat 30 A 13.0 5-12323 a-b-c - 0 58888 15 14:33 33260103 Dark 30 A 13.0 5-12323 a-b-c - - --------------------------------------------------------------------------------