Subject: 2.5m obslog 20170427 (SJD 57871) From: pgaulme@apo.nmsu.edu Date: Fri, 28 Apr 2017 11:27:27 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday April 27, 2017 (SJD 57871) ---- Observing Team ---- Kaike Pan Patrick Gaulme Online help fron Jon Holtzman and Nathan DeLee for apogee issues ---- Observing Summary ---- All time lost for dust, winds, and clouds. Time loss: eBOSS 4.5 h, MaNGA 3.7 h Notes: - We had a serious issue with APOGEE during the whole night. During checkouts, we could not run APOGEE correctly as the Dither and Shutter status were unknown. See log and problem in detail section. APOGEE was not available for the whole night. - We swapped dewar 6 because the pressure was too low. - We tried to bypass SOP to have MANGA only and it did not work, as it has happened previously with isapogee. We updated ticket 2759. - We took a one test MANGA exposure with command lines given that APOGEE was unavailable and that the bypass was not working. - We got one out-of-closed-loop event ---- Observing Plan ---- Thu 27 APR 2017 57871.0 0.03 02:57 11:05 10:18 18:29 ---- eBOSS until 07:25 ( 4.5 h), then MaNGA ( 3.7 h) ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ 23:30Z Started afternoon checkouts %... Major issue with APOGEE We were not able to do the checkouts of APOGEE as both the dither and shutter status were not read in anyway. We called Jon H and Nathan to get help. The problem appears to be new and not understood. Jon asked help to Matt Nelson who was not available on the moment, For the night time observations APOGEE is used as piggyback on MANGA and won't be available tonight. %... Miscellaneous - Test slews (121,30, 121,90, 161,90, rota = 0, 40, 0) - Did 'platedb showActive' All BOSS/APOGEE carts are ready - Then, stui->script->startUp->plugging - Tested the actors - Check calib lamps - guider dark stack=9 time=15 - guider flat %... eBOSS/MANGA - sop doBossCalibs nbias=1 ndark=1 darktime=20 - Hartmann: sop collimateBoss 00:15:33 Hartmann collimate output on cart #6 sp1: offset: r= -329 (In Focus); b= -4.2 (Out of focus) sp2: offset: r= 2209 (Out of focus); b= -3.9 (Out of focus) sp1: pred. move: spAverageMove= 510 sp2: pred. move: spAverageMove= 1734 sp1: pred. resid: r= -839, b= -2.6, txt= OK, spTemp = 10.2 sp2: pred. resid: r= 474, b= 1.5, txt= OK, spTemp = 9.5 %... Took 1 arc 1 flat from SOP. 00:17:08 boss exposure 25.0, file=242968, C 00:21:14 boss exposure 4.0, file=242891 %... Status ---------- motorPosition sp1: 23500, 23388, 19840; sp2: 20877, 21547, 20807; ---------- Median spectrographs temperatures: Tsp1= 10.2, Tsp2= 9.4; Outside air temperature= 13.0; %... SOS test > guider loadCartridge cartridge=10 force > sop doBossCalibs nbias=1 ndark=1 darktime=20 %... Declaring BOSS test exposures into tests. $ ssh eboss@sdss4-eboss $ setup idlspec2d $ idl $ apofix, 241549, /test %... Declaring MaNGA test exposures into tests. $ ssh -X manga@sdss-manga $ dosMarkExposure -e 242968 -m 57871 -s "test" We did not star ventilation because dust levels were way too high (red) ============================================================================ NIGHT ============================================================================ Sky cleared up at about 3am local time (09:00Z) and dust + winds dropped down. The time to open, the sky was cloudy again. 09:15:43 encl25m : open; Given that APOGEE was not available, we tried to bypass SOP by using isMANGAdither, which was not working. We updated the ticket 2759. For test, took one exposure by using these command lines guider decenter on guider mangaDither ditherPosition=N boss exposure time=900 The data should not be of any good quality given the cloud coverage. 11:15:43 encl25m : closed ---- APOGEE Data Summary ---- APOGEE data list, mjd=57871, all raw_data: /data/apogee/utr_cdr/57871/apRaw-*.fits archive: /data/apogee/archive/57871/apR-[a,b,c]-*.apz quickred: /data/apogee/archive/57871/apR-[a,b,c]-*.apz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 57871 1 23:34 23090001 Dark 3 ? 0.0 6-9048 a-b-c - - 57871 2 23:52 23090002 Dark 3 ? 0.0 6-9048 a-b-c - - -------------------------------------------------------------------------------- ---- eBOSS Data Summary ---- Reading FITS from /data/spectro/57871/ 1 00:23 b1-00242970 10 9620 bia Clo 0.0 23500 23388 19840 00:23 b2-00242970 10 9620 bia Clo 0.0 20877 21547 20807 00:23 r1-00242970 10 9620 bia Clo 0.0 23500 23388 19840 00:23 r2-00242970 10 9620 bia Clo 0.0 20877 21547 20807 2 00:25 b1-00242971 10 9620 dar Clo 20.0 23500 23388 19840 00:25 b2-00242971 10 9620 dar Clo 20.0 20877 21547 20807 00:25 r1-00242971 10 9620 dar Clo 20.0 23500 23388 19840 00:25 r2-00242971 10 9620 dar Clo 20.0 20877 21547 20807 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- MaNGA Data Summary ---- Reading FITS from /data/spectro/57871/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n time filename crt plt flav Shu Dth exp colA colB colC Hrt b1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 00:07 b1-00242964 6 9048 bia Clo C 0.0 22990 22878 19328 00:07 b2-00242964 6 9048 bia Clo C 0.0 19144 19813 19073 00:07 r1-00242964 6 9048 bia Clo C 0.0 22990 22878 19328 00:07 r2-00242964 6 9048 bia Clo C 0.0 19144 19813 19073 2 00:08 b1-00242965 6 9048 dar Clo C 20.0 22990 22878 19328 00:08 b2-00242965 6 9048 dar Clo C 20.0 19144 19813 19073 00:08 r1-00242965 6 9048 dar Clo C 20.0 22990 22878 19328 00:08 r2-00242965 6 9048 dar Clo C 20.0 19144 19813 19073 3 00:12 b1-00242966 6 9048 arc Lef C 4.1 22990 22878 19328 HART 00:12 b2-00242966 6 9048 arc Lef C 4.2 19144 19813 19073 HART 00:12 r1-00242966 6 9048 arc Lef C 4.1 22990 22878 19328 HART 00:12 r2-00242966 6 9048 arc Lef C 4.2 19144 19813 19073 HART 4 00:13 b1-00242967 6 9048 arc Rig C 4.1 22990 22878 19328 HART 00:13 b2-00242967 6 9048 arc Rig C 4.2 19144 19813 19073 HART 00:13 r1-00242967 6 9048 arc Rig C 4.1 22990 22878 19328 HART 00:13 r2-00242967 6 9048 arc Rig C 4.2 19144 19813 19073 HART 5 00:17 b1-00242968 6 9048 fla Out C 25.1 23500 23388 19840 00:17 b2-00242968 6 9048 fla Out C 25.1 20877 21547 20807 00:17 r1-00242968 6 9048 fla Out C 25.1 23500 23388 19840 00:17 r2-00242968 6 9048 fla Out C 25.1 20877 21547 20807 6 00:21 b1-00242969 6 9048 arc Out C 4.1 23500 23388 19840 00:21 b2-00242969 6 9048 arc Out C 4.2 20877 21547 20807 00:21 r1-00242969 6 9048 arc Out C 4.1 23500 23388 19840 00:21 r2-00242969 6 9048 arc Out C 4.2 20877 21547 20807 7 09:48 b1-00242972 6 9048 arc Lef C 4.1 23500 23388 19840 HART 09:48 b2-00242972 6 9048 arc Lef C 4.2 20877 21536 20807 HART 09:48 r1-00242972 6 9048 arc Lef C 4.1 23500 23388 19840 HART 09:48 r2-00242972 6 9048 arc Lef C 4.2 20877 21536 20807 HART 8 09:49 b1-00242973 6 9048 arc Rig C 4.1 23500 23388 19840 HART 09:49 b2-00242973 6 9048 arc Rig C 4.2 20877 21536 20807 HART 09:49 r1-00242973 6 9048 arc Rig C 4.1 23500 23388 19840 HART 09:49 r2-00242973 6 9048 arc Rig C 4.2 20877 21536 20807 HART 9 09:51 b1-00242974 6 9048 arc Out C 4.1 27921 27810 24259 09:51 b2-00242974 6 9048 arc Out C 4.2 23399 24058 23328 09:51 r1-00242974 6 9048 arc Out C 4.1 27921 27810 24259 09:51 r2-00242974 6 9048 arc Out C 4.2 23399 24058 23328 10 09:52 b1-00242975 6 9048 fla Out C 25.1 27921 27810 24259 09:52 b2-00242975 6 9048 fla Out C 25.2 23399 24058 23328 09:52 r1-00242975 6 9048 fla Out C 25.1 27921 27810 24259 09:52 r2-00242975 6 9048 fla Out C 25.2 23399 24058 23328 11 10:52 b1-00242976 6 9048 sci Out N 900.1 27921 27810 24259 10:52 b2-00242976 6 9048 sci Out N 900.2 23399 24058 23328 10:52 r1-00242976 6 9048 sci Out N 900.1 27921 27810 24259 10:52 r2-00242976 6 9048 sci Out N 900.2 23399 24058 23328 12 11:18 b1-00242977 6 9048 bia Clo N 0.0 27921 27810 24259 11:18 b2-00242977 6 9048 bia Clo N 0.0 23399 24058 23328 11:18 r1-00242977 6 9048 bia Clo N 0.0 27921 27810 24259 11:18 r2-00242977 6 9048 bia Clo N 0.0 23399 24058 23328 13 11:19 b1-00242978 6 9048 bia Clo N 0.0 27921 27810 24259 11:19 b2-00242978 6 9048 bia Clo N 0.0 23399 24058 23328 11:19 r1-00242978 6 9048 bia Clo N 0.0 27921 27810 24259 11:19 r2-00242978 6 9048 bia Clo N 0.0 23399 24058 23328 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- Telescope Status ---- 08:50 Started ventilation 09:15 encl25m : open 11:15 encl25m : closed No interlocks bypassed Status at: 11:22Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 6 Counterweights at: LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 240 lbs, 19 psi SP2 104 lbs, 19 psi ---- Software Used ---- Get Versions, 04/28/17, 00:12:53 -- STUI software: STUI : 1.7.1b6 alerts : trunk+svn66561 apo : trunk+svn158476M boss : v4_0 -boss.specMec : 3.0.4 gcamera : v2_1 guider : v3_6_1 hartmann : v1_4_6 hub : Tag: sdss4_v3_0 mcp : NOSVN -mcp.plc : 34 -mcp.azFiduci : v1_124 -mcp.altFiduc : v1_124 -mcp.rotFiduc : v1_124 platedb : v2_5_1 sop : v3_8_2 tcc : 1.2.2 apogee : not availble apogeecal : not availble apogeeql : v1_6_1 APO-local : trunk -- done -- -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_9 petunia : trunk autoscheduler : v7_1_1 sdss_python_module: v2_5 mangadrp : trunk exported ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: 0.003 0.001 Axial B: 0.76 0.76 Axial C: 0.524 0.524 Trans D: 0.99 0.989 Lateral E: 9.1313 9.1313 Lateral F: 14.3383 14.3256 GALILS Commanded: 10250 -900 6850 16950 4850 5000 Actual: 10250 -900 6850 16950 4850 5000 SETMIR VALUES PrimDesOrient: 0.0 3.12 -1.95 -106.75 -170.8 PrimOrient: 0.0 3.04 -1.75 -107.3 -171.7 SECONDARY: ---------- Focus: 0.0000 Air temp: 14.6 Altitude: 30.3 MIGS TONIGHT NOMINAL Axial A: -0.00115 -0.00115 Axial B: -6.653 -6.653 Axial C: -1.308 -1.309 Trans D: -0.183 -0.179 GALILS Commanded: 1622700 1577050 1640050 -8250 -15650 Actual: 1599617 1568455 1584430 -8250 -15650 SETMIR VALUES SecDesOrient: 1257.0 -2.03 -9.35 40.0 278.8 SecOrient: 1256.99 -2.02 -9.36 39.83 278.89 ---- Problems in Detail ---- Here is Jon Holtzman's message to Matt Nelson to explain the issue with APOGEE: From: Jon Holtzman Sent: Thursday, April 27, 2017 6:31 PM To: Wilson, John C. (jcw6z); Matt Nelson Cc: Nathan De Lee; Patrick Gaulme Subject: APOGEE communication issues: HELP NEEDED Matt, There seems to be a communcations problem with the APOGEE-N instrument. Dither and shutter status are showing as off. Earlier today, ICS Log started showing: 2017 04 27 117 13 03 11 149871545 VacPressure:1.11E-6 LN2Lev: 85.846 2017 04 27 117 13 03 11 158960441 APSurvey Time: 2308.844272465285 APSurveyDay: 2308 2017 04 27 117 13 08 11 159976454 VacPressure:1.12E-6 LN2Lev: 85.771 2017 04 27 117 13 08 11 169060234 APSurvey Time: 2308.8477448032427 APSurveyDay: 2308 2017 04 27 117 13 13 11 167674916 VacPressure:1.11E-6 LN2Lev: 85.746 2017 04 27 117 13 13 11 176793907 APSurvey Time: 2308.851217118063 APSurveyDay: 2308 2017 04 27 117 13 15 10 513283720 soTimout on cmd Rsp to: MG _CN0 Trying again... 2017 04 27 117 13 15 11 514265178 Ex on cmd retry: Read timed out 2017 04 27 117 13 15 11 523327695 NFE parsing Double: 2017 04 27 117 13 15 12 515282754 soTimout on cmd Rsp to: RPF Trying again... 2017 04 27 117 13 15 13 516488992 Ex on cmd retry: Read timed out 2017 04 27 117 13 15 13 525436028 NFE parsing Int: CMD: RPF 2017 04 27 117 13 15 14 517744264 soTimout on cmd Rsp to: MG _LFF Trying again... 2017 04 27 117 13 15 15 520263621 Ex on cmd retry: Read timed out 2017 04 27 117 13 15 15 530398998 NFE parsing Double: earlier today, and later: 2017 04 27 117 23 34 42 860254420 Raw file: /data/2309/apRaw-23090001-003.fits 2017 04 27 117 23 34 42 869411926 Populating HeaderVector with 33 2017 04 27 117 23 34 43 025559435 Summary file: /summary/2309/apRaw-23090001.fits 2017 04 27 117 23 34 43 035553199 Monitor stopped 2017 04 27 117 23 34 43 046489136 --------------------------------------------- 2017 04 27 117 23 35 37 145800161 Galil CommNOTOK RSP: 2017 04 27 117 23 36 37 146058841 Galil CommNOTOK RSP: 2017 04 27 117 23 37 37 146361977 Galil CommNOTOK RSP: 2017 04 27 117 23 38 36 254176576 VacPressure:1.12E-6 LN2Lev: 93.827 2017 04 27 117 23 38 36 263402395 APSurvey Time: 2309.2855352430634 APSurveyDay: 2309 2017 04 27 117 23 38 37 146259236 Galil CommNOTOK RSP: 2017 04 27 117 23 39 37 146901748 Galil CommNOTOK RSP: 2017 04 27 117 23 40 37 147165147 Galil CommNOTOK RSP: 2017 04 27 117 23 41 37 147520053 Galil CommNOTOK RSP: 2017 04 27 117 23 42 37 147812220 Galil CommNOTOK RSP: 2017 04 27 117 23 43 19 892011674 CamCMD: OBSERVER.Patrick 77 shutter K 2017 04 27 117 23 43 36 258377902 VacPressure:1.12E-6 LN2Lev: 93.746 2017 04 27 117 23 43 36 267569612 APSurvey Time: 2309.289007511572 APSurveyDay: 2309 2017 04 27 117 23 43 37 147720090 Galil CommNOTOK RSP: Galil seems to be responding to pings at 192.168.2.100 Instrument is scheduled for second half of the night tonight …. Holtz.