Subject: 2.5m obslog 20180329 (SJD 58207) From: elenam@apo.nmsu.edu Date: Fri, 30 Mar 2018 09:25:28 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday March 29, 2018 (SJD 58207) ---- Observing Team ---- Dan Oravetz Elena Malanushenko John Downey (support) Conor Sayres (support) Jon Brinkmann (support) Dan Long (support) ! ---- Observing Plan ---- Thu 29 MAR 2018 58207.0 Flex Eng/APOGEE-2 (10.4 h) ---- Observing Summary ---- We experienced abnormal behavior with the wind baffle motion during afternoon checkout and after 8 hours of trying to diagnose the problem it still is not clear where exactly the problem lies. The magnetostrictive sensor and the spare were both tested on the bench and the spare was ultimately swapped-in after it seemed to perform as expected on the bench (the "old" sensor that had been in place on the telescope did not test correctly on the bench). Despite the swap of the sensor, the windbaffle motion is still not behaving correctly and at this point it seems the problem is in the signal that comes down into the lower enclosure. There will certainly be more details to follow tomorrow, but as for tonight the telescope is inoperable. Because of hardware problem, we did not open. It was Eng time, Eng lost 10.4 h. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive. Based on last night's focus residuals, no adjustments were required of the blue rings for tonight. After testing the fiducial crossings were all good and then mounting cartridge 8 at the instChange position, I initialized the axes in preparation of slewing back to stow. However, after the 'init' the telescope began to move downward uncommanded despite the axis not being clear to move (the MCP reported '*AB*', which means Axis Status = A(aborted) and Amp Status = B (brake)). An e-stop was engaged at ~86 degrees and then subsequent attempts at initialization failed to complete. After another try, the init cleared the axis, but then altitude began moving upward, again uncommanded: the stop button gets pulled, then we issue the init (either from tcc or mcp), and then altitude starts moving immediately upward. Here's a time stamp from which to reference things: 23:32:23 OBSERVER.telescope: 22:36:50 CmdStarted: OBSERVER.telescope 351 tcc track 121,90 mount /rota=0 /rotty=mount 22:36:51 OBSERVER.telescope 351 tcc i AxisCmdState=Drifting,Drifting,Drifting; AxisErrCode=OK,OK,OK; AxisNoTrack=FFF 22:36:51 OBSERVER.telescope 351 tcc i ObjName=\"\" 22:36:51 OBSERVER.telescope 351 tcc i ObjSys=Mount,0 22:36:51 OBSERVER.telescope 351 tcc i ObjPos=121.000000,0.000000,5029079819.3378572,90.000000,0.000000,5029079819.3378572 22:36:51 OBSERVER.telescope 351 tcc i RotType=Mount 22:36:51 OBSERVER.telescope 351 tcc i RotPos=0.000000,0.000000,5029079819.3378572 22:36:51 OBSERVER.telescope 351 tcc i ObjArcOff=0.000000,0.000000,5029079819.3378572,0.000000,0.000000,5029079819.3378572 22:36:51 OBSERVER.telescope 351 tcc i Boresight=0.000000,0.000000,5029079819.3378572,0.000000,0.000000,5029079819.3378572 22:36:51 OBSERVER.telescope 351 tcc i GuideOff=0.000000,0.000000,5029079819.3378572,0.000000,0.000000,5029079819.3378572,0.000000,0.000000,5029079819.3378572 22:36:51 OBSERVER.telescope 351 tcc i CalibOff=0.000000,0.000000,5029079819.3378572,0.000000,0.000000,5029079819.3378572,0.000000,0.000000,5029079819.3378572 22:36:51 OBSERVER.telescope 351 tcc i ObjZPMPos=0.000000,0.000000,5029079829.7729015,90.000000,0.000000,5029079829.7729015 22:36:51 OBSERVER.telescope 351 tcc i ObjNetPos=0.000000,0.000000,5029079829.7729015,90.000000,0.000000,5029079829.7729015 22:36:54 OBSERVER.telescope 351 tcc i AxisCmdState=Slewing,Slewing,Slewing 22:36:54 OBSERVER.telescope 351 tcc > SlewBeg=5029079831.77; SlewDuration=17.46; SlewNumIter=2; MoveItems=NYNNYYYYY 22:37:08 CmdStarted: OBSERVER.telescope 352 tcc axis init 22:37:11 OBSERVER.telescope 351 tcc f text=\"Slew failed: 'TCC25mActor' object has no attribute 'log'\" 22:37:11 CmdDone: OBSERVER.telescope 351 tcc track 121,90 mount /rota=0 /rotty=mount JohnD came on site to investigate and we called Conor as well. Conor could not find anything wrong with the TCC to explain the behavior and we rebooted it twice nonetheless just in case. We also called JonB and he couldn't find a smoking gun after looking into the MCP. The MCP did crash on us at one point later on, however, and we rebooted it. We used the manual pendant to slew the telescope to 121,30,00 so we could force a fill of the BOSS 10L dewars. We consulted with Dan Long about the magnetostrictive sensor and located his procedure to operate with one amplifier for altitude. We tried to bypass each of the two Glentek altitude amplifiers and with one of the two off then issue the 'init', but in both cases the altitude still was moving upward, so the problem did not appear to be isolated to one of the amplifiers. We swapped in the spare magnetostrictive sensor, and at least this time the telescope did not immediately move upward. Instead, the wind baffle slowly moved downward until it contacted the bumper with telescope. So this was at least different behavior than we had been seeing. Both the spare and the old magnetostrictive sensor were then tested on the bench and the spare seemed to perform as expected: it produced an expected signal (voltage increased from about 0 to 12 VDC as the magnet moved along the length of the available stroke). The "old" sensor was reading a fairly constant ~5.5VDC despite the magnet's position along the stroke. The spare magnetostrictive sensor was then put back in place on the telescope and we zeroed the telescope's position within the windbaffle and adjusted the sensor's magnet to the zero position. A small subsequent altitude move of ~2 degrees seemed to complete, but the windbaffle continued to slowly move downward. When we tried further 'init's, the windbaffle again continued to slowly move downward until the bumper was hit. JohnD checked the windbaffle speed signal in the lower enclosure windbaffle control cabinet during some additional attempted movements the measured signal was not as it was expected to be. This is where investigation will resume tomorrow. ============================================================================ NIGHT ============================================================================ No observations tonight. Morning calibrations have been done. ---- BOSS Spectrographs Collimation Summary ---- --- Afternoon focus check --- 22:09:17 boss exposure arc, expNum=265510, expTime=4.0 22:10:28 boss exposure arc, expNum=265511, expTime=4.0 22:11:51 sp1.motor.move= 22, 24, 23 22:11:51 Hartmann analysis for cartridge 16: sp1 offset: r = -659 steps (In Focus); b = -2.2 deg (In Focus) sp2 offset: r = 631 steps (In Focus); b = 2.0 deg (In Focus) sp1 pred. move: spAverageMove = 24 steps sp2 pred. move: spAverageMove = 0 steps sp1 pred. residual: r = -683 steps, b = -2.1 deg, spTemp = 8.4 C sp2 pred. residual: r = 630 steps, b = 2.0 deg, spTemp = 7.1 C 22:11:48 OBSERVER.doravetz.sop 343 hartmann i r1MeanOffset=-0.10,"In Focus" 22:11:48 OBSERVER.doravetz.sop 343 hartmann i r2MeanOffset=0.10,"In Focus" 22:11:48 OBSERVER.doravetz.sop 343 hartmann i b1MeanOffset=0.11,"In Focus" 22:11:48 OBSERVER.doravetz.sop 343 hartmann i b2MeanOffset=-0.10,"In Focus" ---- Telescope Offsets and Scale ---- None ---- Focus Log ---- None ---- Telescope Status ---- No ventilation No____ encl25m : open No____ encl25m : closed Status at: 09:02:19 Telescope stowed at: 121.0, 29.8, 0.0 mount Instrument mounted: cartridge = 8 Counterweights at: 431, 432, 432, 434 LN2 autofill systems: Connected and turned on -180L LN2 dewar scales: SP1 237.1 lbs, 22.4 psi SP2 270.0 lbs, 22.1 psi ---- Software Used ---- --- Afternoon disk space --- /dev/vdd 688G 585G 104G 86% /home /dev/vdc1 11T 7.2T 3.7T 67% /data --- STUI software --- STUI : 1.7.3 alerts : trunk apo : trunk boss : v4_0 -boss.specMec : 3.0.4 gcamera : v2_2 guider : v3_7 hartmann : v1_5 hub : Tag: v4.2 mcp : v7_0_4 -mcp.plc : 34 -mcp.azFiduci : v1_130 -mcp.altFiduc : v1_130 -mcp.rotFiduc : v1_130 platedb : v2_7 sop : 3.10.1 tcc : 1.2.2 apogee : not availble apogeecal : not availble apogeeql : v1_6_1 APO-local : not availble --- Other versions --- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_11 petunia : 1.0.0 autoscheduler : trunk/ sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- [observer@sdss4-hub ~]$ list_ap -m 58206 -morning APOGEE data list, mjd=58206, morning raw_data: /data/apogee/utr_cdr/58206/apRaw-*.fits archive: /data/apogee/archive/58206/apR-[a,b,c]-*.apz quickred: /data/apogee/archive/58206/apR-[a,b,c]-*.apz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 58206 1 12:40 26440092 Dark 60 B 13.5 16-7338 a-b-c - - 58206 2 12:51 26440093 Dark 60 B 13.5 16-7338 a-b-c - - 58206 3 13:01 26440094 Dark 60 B 13.5 16-7338 a-b-c - - 58206 4 13:12 26440095 QuarFlat 10 B 13.5 16-7338 a-b-c - 45 58206 5 13:14 26440096 QuarFlat 10 B 13.5 16-7338 a-b-c - 45 58206 6 13:16 26440097 QuarFlat 10 B 13.5 16-7338 a-b-c - 45 58206 7 13:18 26440098 Arc-Thar 12 A 13.0 16-7338 a-b-c 0.08 - 58206 8 13:20 26440099 Arc-Une 40 A 13.0 16-7338 a-b-c - - 58206 9 13:27 26440100 Arc-Thar 12 B 13.5 16-7338 a-b-c 0.57 - 58206 10 13:29 26440101 Arc-Une 40 B 13.5 16-7338 a-b-c - - 58206 11 13:37 26440102 Dark 30 A 13.0 16-7338 a-b-c - - 58206 12 13:42 26440103 InteFlat 30 A 13.0 16-7338 a-b-c - 0 58206 13 13:47 26440104 InteFlat 30 A 13.0 16-7338 a-b-c - 0 58206 14 13:53 26440105 InteFlat 30 A 13.0 16-7338 a-b-c - 0 58206 15 13:58 26440106 Dark 30 A 13.0 16-7338 a-b-c - - -------------------------------------------------------------------------------- APOGEE data list, mjd=58207, all raw_data: /data/apogee/utr_cdr/58207/apRaw-*.fits archive: /data/apogee/archive/58207/apR-[a,b,c]-*.apz quickred: /data/apogee/archive/58207/apR-[a,b,c]-*.apz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 58207 1 21:42 26450001 Dark 3 A 13.0 -1- -1 a-b-c - - 58207 2 21:45 26450002 Dark 3 A 13.0 -1- -1 a-b-c - - 58207 3 21:48 26450003 Dark 60 A 13.0 16-7338 a-b-c - - 58207 4 21:59 26450004 QuarFlat 10 A 13.0 16-7338 a-b-c - 49 58207 5 22:01 26450005 Arc-Thar 12 A 13.0 16-7338 a-b-c 0.04 - 58207 6 22:03 26450006 Arc-Une 40 A 13.0 16-7338 a-b-c - - 58207 7 22:10 26450007 Arc-Thar 12 B 13.5 16-7338 a-b-c 0.53 - 58207 8 22:13 26450008 Arc-Une 40 B 13.5 16-7338 a-b-c - - 58207 9 22:20 26450009 Dark 60 A 13.0 -1- -1 a-b-c - - 58207 10 07:39 26450010 Dark 60 A 13.0 8-10304 a-b-c - - 58207 11 07:50 26450011 Dark 60 A 13.0 8-10304 a-b-c - - 58207 12 08:00 26450012 Dark 60 A 13.0 8-10304 a-b-c - - 58207 13 08:11 26450013 QuarFlat 10 A 13.0 8-10304 a-b-c - 49 58207 14 08:13 26450014 QuarFlat 10 A 13.0 8-10304 a-b-c - 49 58207 15 08:15 26450015 QuarFlat 10 A 13.0 8-10304 a-b-c - 49 58207 16 08:17 26450016 Arc-Thar 12 A 13.0 8-10304 a-b-c 0.07 - 58207 17 08:19 26450017 Arc-Une 40 A 13.0 8-10304 a-b-c - - 58207 18 08:26 26450018 Arc-Thar 12 B 13.5 8-10304 a-b-c 0.57 - 58207 19 08:28 26450019 Arc-Une 40 B 13.5 8-10304 a-b-c - - 58207 20 08:36 26450020 Dark 30 A 13.0 8-10304 a-b-c - - 58207 21 08:41 26450021 InteFlat 30 A 13.0 8-10304 a-b-c - 0 58207 22 08:46 26450022 InteFlat 30 A 13.0 8-10304 a-b-c - 0 58207 23 08:52 26450023 InteFlat 30 A 13.0 8-10304 a-b-c - 0 58207 24 09:00 26450024 Dark 30 A 13.0 ------- x-x-x - - -------------------------------------------------------------------------------- ---- eBOSS Data Summary ---- Reading FITS from /data/spectro/58207/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n time filename cart plt flav hrt exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 21:49 b1-00265505 16 7338 bia Clo 0.0 31029 30599 29679 21:49 b2-00265505 16 7338 bia Clo 0.0 21022 22013 21632 21:49 r1-00265505 16 7338 bia Clo 0.0 31029 30599 29679 21:49 r2-00265505 16 7338 bia Clo 0.0 21022 22013 21632 2 21:53 b1-00265506 16 7338 arc Lef 4.1 31029 30599 29679 HART 21:53 b2-00265506 16 7338 arc Lef 4.2 21022 22013 21632 HART 21:53 r1-00265506 16 7338 arc Lef 4.1 31029 30599 29679 HART 21:53 r2-00265506 16 7338 arc Lef 4.2 21022 22013 21632 HART 3 21:54 b1-00265507 16 7338 arc Rig 4.1 31029 30599 29679 HART 21:54 b2-00265507 16 7338 arc Rig 4.1 21022 22013 21632 HART 21:54 r1-00265507 16 7338 arc Rig 4.1 31029 30599 29679 HART 21:54 r2-00265507 16 7338 arc Rig 4.1 21022 22013 21632 HART 4 22:05 b1-00265508 16 7338 arc Lef 4.1 28939 28510 27589 HART 22:05 b2-00265508 16 7338 arc Lef 4.2 25120 26112 25731 HART 22:05 r1-00265508 16 7338 arc Lef 4.1 28939 28510 27589 HART 22:05 r2-00265508 16 7338 arc Lef 4.2 25120 26112 25731 HART 5 22:06 b1-00265509 16 7338 arc Rig 4.1 28939 28510 27589 HART 22:06 b2-00265509 16 7338 arc Rig 4.2 25120 26112 25731 HART 22:06 r1-00265509 16 7338 arc Rig 4.1 28939 28510 27589 HART 22:06 r2-00265509 16 7338 arc Rig 4.2 25120 26112 25731 HART 6 22:09 b1-00265510 16 7338 arc Lef 4.1 29123 28692 27773 HART 22:09 b2-00265510 16 7338 arc Lef 4.2 25278 26271 25890 HART 22:09 r1-00265510 16 7338 arc Lef 4.1 29123 28692 27773 HART 22:09 r2-00265510 16 7338 arc Lef 4.2 25278 26271 25890 HART 7 22:10 b1-00265511 16 7338 arc Rig 4.1 29123 28692 27773 HART 22:10 b2-00265511 16 7338 arc Rig 4.2 25278 26271 25890 HART 22:10 r1-00265511 16 7338 arc Rig 4.1 29123 28692 27773 HART 22:10 r2-00265511 16 7338 arc Rig 4.2 25278 26271 25890 HART - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- MaNGA Data Summary ---- Reading FITS from /data/spectro/58207/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 18:00 4.0 -13.3 12.0 0.7 1589 18:15 4.2 -13.3 13.3 0.5 1558 18:30 4.8 -13.1 2.3 330.3 1558 18:45 5.5 -15.5 0.8 344.5 1524 19:00 5.4 -13.7 11.0 301.1 1404 19:15 5.0 -14.8 11.1 355.0 1238 19:30 5.6 -13.9 10.2 317.7 1258 19:45 6.1 -13.7 10.3 326.3 1219 20:00 5.6 -13.8 6.1 260.8 1113 20:15 6.7 -11.9 15.3 327.6 1167 20:30 9.0 -12.2 13.3 304.8 1106 20:45 7.8 -13.4 13.0 291.7 1186 21:00 7.1 -13.8 10.7 269.9 1171 21:15 8.0 -13.1 2.9 326.3 1028 21:30 6.7 -13.4 10.4 310.7 1197 21:45 5.9 -13.8 15.6 315.6 1053 22:00 6.9 -13.3 1.7 287.4 1079 22:15 8.5 -12.6 13.8 299.8 1061 22:30 7.7 -12.6 17.2 264.5 850 22:45 7.0 -13.7 8.8 260.6 883 23:00 6.3 -13.8 6.6 297.6 908 23:15 5.8 -13.8 4.3 267.3 853 23:30 5.9 -13.1 9.7 290.3 935 23:45 5.8 -13.1 1.9 242.0 985 00:00 5.7 -13.1 5.2 262.6 897 00:15 7.6 -12.0 2.2 251.2 1006 00:30 6.4 -13.3 9.2 272.6 908 00:45 5.7 -12.9 6.6 262.3 878 01:00 6.3 -12.8 5.3 287.5 887 01:15 5.8 -13.1 4.8 280.0 898 01:30 5.3 -13.3 6.1 313.3 876 01:45 5.4 -13.4 7.7 280.7 797 02:00 5.4 -13.2 8.0 304.5 781 02:15 5.6 -13.6 6.5 258.0 743 02:30 5.6 -13.3 8.6 252.3 806 02:45 5.5 -13.1 12.4 231.8 792 03:00 5.7 -13.0 10.7 233.8 793 03:15 5.4 -13.1 9.0 246.0 777 03:30 5.4 -12.9 11.5 230.9 825 03:45 5.5 -12.9 8.8 248.3 858 04:00 5.5 -12.7 11.5 240.6 821 04:15 5.4 -12.8 11.3 243.3 770 04:30 5.9 -12.6 11.2 266.0 735 04:45 6.0 -12.7 11.5 263.1 580 05:00 6.1 -12.9 11.7 264.0 605 05:15 5.9 -13.0 13.6 261.5 579 05:30 5.7 -13.2 12.1 263.5 590 328 2.78 05:45 5.8 -13.2 11.4 260.0 532 328 2.65 06:00 5.8 -13.7 11.2 263.1 542 328 6.08 06:15 5.7 -13.3 11.5 236.4 520 328 2.75 06:30 5.6 -14.0 10.2 244.3 515 328 2.90 06:45 5.7 -13.6 8.7 224.3 548 328 2.67 07:00 5.6 -13.4 10.4 228.8 583 328 2.88 07:15 5.3 -13.2 7.9 231.7 567 328 2.87 07:30 5.1 -13.6 8.4 210.5 517 328 3.17 07:45 5.2 -13.8 8.4 218.6 519 328 2.28 08:00 5.1 -13.7 7.1 189.5 491 328 3.02 08:15 5.0 -13.9 7.1 185.1 499 328 2.82 08:30 4.9 -13.9 8.7 184.1 505 328 2.70 08:45 5.0 -13.7 10.5 186.8 516 328 2.98 09:00 5.0 -13.7 7.2 183.3 520 328 3.12