Subject: 2.5m obslog 20140510 (MJD 56788) From: kinemuchi@apo.nmsu.edu Date: Sun, 11 May 2014 11:48:56 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday May 10, 2014 (MJD 56788) ---- Observing Team ---- Moses Marchante Karen Kinemuchi Joe Huehnerhoff at the beginning of the night for flat field screen problem. ---- Observing Plan ---- 10 May 2014 56788.0 0.76 03:08 10:51 11:20 19:04 APOGEE 8.3 hours ---- Observing Summary ---- APOGEE OBSERVING: Cart 8 (Plate 7370: 270+75_A) 2xABBA [missing fiber #35,184; faint fibers #88,158,268] Cart 9 (Plate 7369: 270+75_A) 2xABBA [missing fiber #36; faint fibers #284] Cart 6 (Plate 7361: 000+75_A) 2xABBA+sky flats [missing fiber #181; faint fibers #15,34,43,133,297] Cart 3 (Plate 7440: K16_075+11_A) 2xABBA [missing fiber #258; faint fiber #250] Cart 7 (Plate 6927: 060+00_A) 2xABBA [no missing fibers; faint fibers #197,208,258,268,292] Cart 4 (Plate 6074: 060+00_A) 1xABBA+1xAB [missing fibers #278; faint fibers #252,284] File number 12260045 (a dome flat) did not archive. Noticed during afternoon check out that boss morning cals had failed with boss.SP2MicroNotTalking boss.SP1MicroNotTalking error. reloaded controllers and ran camcheck; all systems returned to normal We discovered the flat field panel motors were burning out again at the beginning of the night. We operated with two of the flat field panels open all night. Please see the Observing Log and Problems in Detail sections for more detail. Please also see Joe's daylog email regarding this problem for his assessment. We restarted the computer controlling sdss-display. All windows froze during afternoon checkout. The restart cleared up the problem. The weather pages are now updating properly. ---- Observing Log ---- Afternoon: ---------- 23:45Z Started the afternoon checkout. Checked email and changeNotes Checked 180-L dewars weight Cleaned APOGEE gang connector with air. Did 3-read dark Did APOGEE evening cals Checked the images Checked status of rain sensor Recorded the software versions Recorded and checked mirror numbers Did 'platedb showActive' All BOSS/APOGEE carts are ready Tested the actors Check calib lamps Guider/gcamera tests: a) gcamera status (simulate=off; temp=-40) b) guider dark stack=9 time=15 c) guider status d) guider flat Spectro tests and focus a) guider loadCartridge b) boss exposure bias itime=0 c) boss exposure dark itime=20 d) sop hartmann No Boss Tonight e) sop doCalibs f) checked xMid and waveMid values g) declared calibs as test Checked that carts were in bay correctly; All BOSS/Apogee carts are ready 01:08Z Opened doors/louvers; turned on fans Mounted first cart of the night cart 8 hub25m> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 95G 20G 71G 22% / /dev/sda4 793G 366G 387G 49% /home /dev/sda1 99M 36M 58M 39% /boot tmpfs 5.9G 0 5.9G 0% /dev/shm sdssfiles1-p://data1/data/MARVELS 1.8T 516G 1.3T 29% /MARVELS /dev/drbd1 13T 11T 1.7T 87% /data NIGHT: ============= 02:10:44 encl25m : open; 02:18Z Flat field petals failed to open. We noticed an electrical burning smell during opening near the telescope baffle. 02:53Z Joe is on site and he inspected the fuses to the motors for the flat field petals. He replaced one fuse and checked the motors near the 12:00 to the 5:30 screens. He tested the current on the motor near the 3:00 position, and it had high AC current. He concluded that the switch was the problem, and had been most likely running for a while. We regained control of the flat field petals via MCP, but have left 2 of the petals open for operations tonight (the 1:30 and 3:30 panels). Joe is searching for a replacement switch, but would like to inspect the motors carefully for any damage. We are finishing the force fill of spectrographs and will continue on with APOGEE observing, but with two flat field screens open (the 1:30 and 3:00 panels). We were delayed at the start, so we lost about 30 minutes of science time (based on when 12 degree twilight was and when we took our first science exposure). 00:55:59 loadCart: ct=8, pl=7370, sd=A; 03:16:44 sop.gotoField = running; 03:18:16 sop.gotoField = failed; 03:23Z Applied ffs Bypass in the SOP gui to be able to complete the goToField command. Completed goToField. GSOGTF; clear skies with moon, seeing is 1.5"; very windy. 56788 9 03:12 12260009 Dark 10 A 13.0 8-7370 a-b-c - - 56788 10 03:14 12260010 Dark 10 A 13.0 8-7370 a-b-c - - 56788 11 03:27 12260011 Object 47 A 13.0 8-7370 a-b-c - - 56788 12 03:36 12260012 Object 47 B 13.5 8-7370 a-b-c - - 56788 13 03:44 12260013 Object 47 B 13.5 8-7370 a-b-c - - 56788 14 03:52 12260014 Object 47 A 13.0 8-7370 a-b-c - - 56788 15 04:01 12260015 Object 47 A 13.0 8-7370 a-b-c - - 56788 16 04:09 12260016 Object 47 B 13.5 8-7370 a-b-c - - 56788 17 04:18 12260017 Object 47 B 13.5 8-7370 a-b-c - - 56788 18 04:26 12260018 Object 47 A 13.0 8-7370 a-b-c - - 56788 19 04:34 12260019 DomeFlat 5 A 13.0 8-7370 a-b-c - 63 56788 20 04:38 12260020 Dark 10 A 13.0 8-7370 a-b-c - - 56788 21 04:40 12260021 Dark 10 A 13.0 8-7370 a-b-c - - aptest 56788 12260019 flat= /data/apogee/utr_cdr/56788/apRaw-12260019.fits master= /home/observer/bin/apRaw-03720068.fits nfibers= 300 average ratio= 0.558198315381 missing fibers (flat/master -------------------------------------------------------------------------------- 03:27 8-7370A 1.000300 -2962 -946 373 28.1 69.4 13.1 27 253 1.6 apog 03:35 8-7370A 1.000330 -3258 -1267 301 22.6 70.2 13.2 30 256 1.4 apog 03:44 8-7370A 1.000312 -3058 -1136 283 16.8 70.8 13.4 31 246 1.6 apog 03:52 8-7370A 1.000302 -2978 -1053 284 10.6 71.2 13.0 28 240 1.5 apog 04:00 8-7370A 1.000299 -2952 -1034 283 4.4 71.4 13.1 22 238 1.5 apog 04:09 8-7370A 1.000300 -2959 -1032 288 -2.0 71.5 13.1 18 260 1.5 apog 04:17 8-7370A 1.000296 -2917 -997 289 -8.3 71.3 13.2 30 246 1.5 apog 04:26 8-7370A 1.000299 -2952 -1027 288 -14.6 70.9 13.2 25 247 1.8 apog 04:47 9-7369A 1.000302 -2981 -1037 299 -29.1 69.3 13.1 22 247 1.8 apog 04:56 9-7369A 1.000329 -3252 -1268 290 -34.2 68.3 13.1 24 253 1.8 apog 05:04 9-7369A 1.000322 -3175 -1192 300 -38.8 67.3 13.1 21 240 1.6 apog 05:12 9-7369A 1.000322 -3198 -1189 304 -43.1 66.1 13.0 20 257 1.9 apog 05:21 9-7369A 1.000325 -3214 -1199 319 -46.9 64.9 12.7 31 243 1.8 apog 05:29 9-7369A 1.000332 -3271 -1238 327 -50.5 63.5 12.8 20 253 1.9 apog 05:37 9-7369A 1.000330 -3258 -1220 331 -53.7 62.2 12.6 30 241 1.7 apog 05:46 9-7369A 1.000326 -3220 -1174 342 -56.7 60.7 12.4 22 222 1.7 apog 06:06 6-7361A 1.000374 -3692 -1570 361 -29.8 73.7 12.2 22 250 1.5 apog 06:14 6-7361A 1.000386 -3816 -1670 361 -35.9 72.7 11.7 27 240 1.7 apog 06:23 6-7361A 1.000388 -3835 -1672 374 -41.3 71.6 11.8 20 264 1.7 apog 06:31 6-7361A 1.000392 -3867 -1683 387 -46.1 70.4 11.7 25 249 1.6 apog 06:40 6-7361A 1.000397 -3915 -1718 390 -50.5 69.1 11.5 25 236 1.4 apog 06:48 6-7361A 1.000392 -3867 -1659 408 -54.3 67.7 11.4 29 233 1.5 apog 06:56 6-7361A 1.000396 -3905 -1672 424 -57.7 66.2 11.1 22 248 1.5 apog 07:05 6-7361A 1.000408 -4023 -1756 433 -60.9 64.7 11.1 24 243 1.5 apog 07:38 3-7440A 1.000467 -4615 -2164 459 118.8 46.1 10.5 20 263 2.0 apog 07:46 3-7440A 1.000480 -4736 -2270 461 118.5 47.6 10.7 25 255 2.2 apog 07:55 3-7440A 1.000469 -4631 -2184 466 118.3 49.2 10.3 20 251 1.8 apog 08:03 3-7440A 1.000462 -4561 -2128 469 118.2 50.7 9.9 20 245 1.8 apog 08:11 3-7440A 1.000463 -4564 -2134 471 118.1 52.3 10.1 20 261 2.1 apog 08:20 3-7440A 1.000446 -4408 -2008 471 118.1 53.8 10.1 21 253 1.8 apog 08:28 3-7440A 1.000436 -4306 -1933 468 118.1 55.4 9.9 16 260 1.8 apog 08:37 3-7440A 1.000433 -4268 -1912 470 118.2 57.0 9.9 18 259 1.7 apog 08:58 7-6927A 1.000344 -3395 -1199 447 85.2 55.4 9.4 17 244 1.7 apog 09:06 7-6927A 1.000318 -3137 -999 440 83.8 57.2 9.4 17 256 1.8 apog 09:15 7-6927A 1.000307 -3035 -911 445 82.3 58.9 9.4 19 233 1.6 apog 09:23 7-6927A 1.000299 -2949 -848 442 80.7 60.7 9.4 12 252 1.5 apog 09:32 7-6927A 1.000300 -2959 -858 444 79.0 62.4 9.2 21 241 1.5 apog 09:40 7-6927A 1.000296 -2921 -818 456 77.2 64.1 9.1 19 265 1.6 apog 09:48 7-6927A 1.000289 -2857 -759 467 75.1 65.8 9.2 18 266 1.5 apog 09:57 7-6927A 1.000293 -2895 -791 469 72.8 67.5 9.0 15 264 1.8 apog 10:19 4-6074A 1.000132 -1302 538 488 65.1 72.0 8.7 18 250 1.6 apog 10:28 4-6074A 1.000115 -1134 688 501 61.4 73.6 8.5 19 250 1.3 apog 10:36 4-6074A 1.000108 -1067 746 506 57.0 75.1 8.6 17 270 1.7 apog 10:45 4-6074A 1.000112 -1108 708 504 51.8 76.5 8.5 22 252 1.4 apog 10:53 4-6074A 1.000101 -997 791 495 45.5 77.9 8.5 25 239 1.3 apog 11:01 4-6074A 1.000100 -990 792 493 37.8 79.1 8.4 20 244 1.5 apog ---- Weather Log ---- --- Weather --- Time Inst Temp DP Dif Humid Wind Dir Dust,1um IRSC IRSCm -------------------------------------------------------------------------------- 03:27 8-7370A 13.1 -10.2 23.3 18 27 253 1790 17.2 1574 1.6 apog 03:35 8-7370A 13.2 -10.2 23.4 18 30 256 1846 17.2 1574 1.4 apog 03:44 8-7370A 13.4 -10.2 23.6 18 31 246 1560 7.1 801 1.6 apog 03:52 8-7370A 13.0 -10.2 23.2 18 28 240 1471 7.0 802 1.5 apog 04:00 8-7370A 13.1 -10.2 23.3 18 22 238 1642 7.6 802 1.5 apog 04:09 8-7370A 13.1 -10.2 23.3 18 18 260 1601 7.9 801 1.5 apog 04:17 8-7370A 13.2 -10.1 23.3 18 30 246 1831 5.8 801 1.5 apog 04:26 8-7370A 13.2 -10.1 23.3 18 25 247 1800 7.4 802 1.8 apog 04:47 9-7369A 13.1 -9.9 23.0 19 22 247 1686 7.3 802 1.8 apog 04:56 9-7369A 13.1 -9.8 22.9 19 24 253 1709 6.0 802 1.8 apog 05:04 9-7369A 13.1 -9.9 23.0 19 21 240 1639 7.5 802 1.6 apog 05:12 9-7369A 13.0 -9.8 22.8 19 20 257 1738 7.5 801 1.9 apog 05:21 9-7369A 12.7 -9.8 22.5 19 31 243 1658 7.2 802 1.8 apog 05:29 9-7369A 12.8 -9.8 22.6 19 20 253 1741 7.9 802 1.9 apog 05:37 9-7369A 12.6 -9.8 22.4 20 30 241 1924 7.4 802 1.7 apog 05:46 9-7369A 12.4 -9.8 22.2 20 22 222 2160 6.0 802 1.7 apog 06:06 6-7361A 12.2 -9.0 21.2 21 22 250 1988 10.7 802 1.5 apog 06:14 6-7361A 11.7 -9.2 20.9 22 27 240 1863 8.4 801 1.7 apog 06:23 6-7361A 11.8 -8.4 20.2 23 20 264 2514 8.3 801 1.7 apog 06:31 6-7361A 11.7 -7.9 19.6 24 25 249 2055 8.0 802 1.6 apog 06:40 6-7361A 11.5 -7.9 19.4 24 25 236 2110 7.8 801 1.4 apog 06:48 6-7361A 11.4 -7.4 18.8 26 29 233 2302 8.1 801 1.5 apog 06:56 6-7361A 11.1 -7.8 18.9 25 22 248 2269 7.8 801 1.5 apog 07:05 6-7361A 11.1 -7.8 18.9 25 24 243 2336 7.7 801 1.5 apog 07:38 3-7440A 10.5 -7.9 18.4 26 20 263 2263 7.3 802 2.0 apog 07:46 3-7440A 10.7 -7.9 18.6 26 25 255 2411 8.6 801 2.2 apog 07:55 3-7440A 10.3 -8.2 18.5 26 20 251 2423 6.9 802 1.8 apog 08:03 3-7440A 9.9 -8.2 18.1 27 20 245 2345 6.3 802 1.8 apog 08:11 3-7440A 10.1 -8.1 18.2 27 20 261 2220 7.7 802 2.1 apog 08:20 3-7440A 10.1 -8.1 18.2 27 21 253 2315 7.9 802 1.8 apog 08:28 3-7440A 9.9 -8.1 18.0 27 16 260 2247 5.9 802 1.8 apog 08:37 3-7440A 9.9 -8.1 18.0 27 18 259 2231 7.6 802 1.7 apog 08:58 7-6927A 9.4 -8.2 17.6 28 17 244 2252 8.5 802 1.7 apog 09:06 7-6927A 9.4 -8.2 17.6 28 17 256 2192 6.7 802 1.8 apog 09:15 7-6927A 9.4 -8.1 17.5 28 19 233 2172 7.9 802 1.6 apog 09:23 7-6927A 9.4 -8.4 17.8 27 12 252 2200 7.4 802 1.5 apog 09:32 7-6927A 9.2 -8.4 17.6 27 21 241 2285 7.7 802 1.5 apog 09:40 7-6927A 9.1 -8.5 17.6 28 19 265 2167 7.6 802 1.6 apog 09:48 7-6927A 9.2 -8.4 17.6 27 18 266 2307 7.0 802 1.5 apog 09:57 7-6927A 9.0 -8.7 17.7 27 15 264 2187 6.2 802 1.8 apog 10:19 4-6074A 8.7 -8.6 17.3 28 18 250 2239 7.6 802 1.6 apog 10:28 4-6074A 8.5 -8.6 17.1 28 19 250 2347 7.9 802 1.3 apog 10:36 4-6074A 8.6 -8.9 17.5 28 17 270 2273 7.6 803 1.7 apog 10:45 4-6074A 8.5 -9.1 17.6 27 22 252 2268 8.7 802 1.4 apog 10:53 4-6074A 8.5 -9.0 17.5 27 25 239 2324 7.9 802 1.3 apog 11:01 4-6074A 8.4 -9.0 17.4 28 20 244 2329 7.1 803 1.5 apog ---- Telescope Status ---- 01:08Z Started ventilation 02:10Z encl25m : open 11:25Z encl25m : closed No interlocks bypassed Status at: 11:40:26 Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 14 Counterweights at: 425, 425, 424, 424 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 111.0 lbs, 20.1 psi SP2 122.8 lbs, 22.7 psi --------------- ---- Software Used ---- Get Versions, 05/11/14, 00:07:51 ----STUI software: STUI: 1.3.1b1 2013-10-22 alerts: v1_3_0 apo: v1_10 boss: v2_5_16 boss.daq: 1.3.2 boss.specMech: 3.0.4 gcamera: v1_3_7 guider: v1_6_3 hub: Tag: sdss3_v1_2_0 mcp: v6_6_18 mcp.plc: 29 mcp.azFiducialVersion: v1_111 mcp.altFiducialVersion: v1_111 mcp.rotFiducialVersion: v1_111 platedb: v1_1_0 sop: v1_3_2 sos: v1_2_0 sos.idlspec2d: v5_6_9 tcc: 2.14.0 apogee: -not availble- apogeecal: -not availble- apogeeql: v1_5_3 -----Other software sdssProcedure(sdss3): v3_06 APO-local scripts : trunk -- done -- -- Other versions ---- idlspec2d (sos3) --- v5_6_9 plate-mapper3 ------ NOCVS:v5_0_14a ---- Mirror Numbers ---- 11:47Z Sorry - we were dealing with critical alerts at the time and forgot to put in mirror numbers during afternoon checkout. ---- Problems in Detail ---- 02:15Z As we moved the enclosure off of the telescope, we noticed an electrical burning smell near the telescope. When we moved the telescope to stow position for the force fill, we tried to open the flat field screens. This failed, and considering the proximity of the electrical smell, we concluded that there was a problem with the flat field screen motors. 02:53Z Joe is on site and he inspected the fuses to the motors for the flat field petals. He replaced one fuse and checked the motors near the 12:00 to the 5:30 screens. He tested the current on the motor near the 3:00 position, and it had high AC current. He concluded that the switch was the problem, and had been most likely running for a while. We regained control of the flat field petals via MCP, but have left 2 of the petals open for operations tonight (the 1:30 and 3:30 panels). Joe is searching for a replacement switch, but would like to inspect the motors carefully for any damage. Please see Joe's day log report for 20140510 to read his assessment. We lost about 30 minutes of science time (mostly waiting for the restarted force fill to finish and doing all the usual prep at the beginning of the night).