Subject: 2.5m obslog 20160627 (SJD 57567) From: kinemuchi@apo.nmsu.edu Date: Tue, 28 Jun 2016 11:27:37 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday June 27, 2016 (SJD 57567) ---- Observing Team ---- Dmitry Bizyaev Karen Kinemuchi ---- Observing Summary ---- Lost entire night to high humidity. MaNGA lost 3.7h ; APOGEE lost 3.9h Notes to day staff and developers: ---------------------------------- * We had several critical alerts due to the CCD Temp of SP2B2 being too low. We found in the subsequent biases some weird electronic noise. See log and/or infrastructure thread for details. Now THE B2 BIASES ARE BAD! Counts are ~5400 in upper left, ~8000 in upper right. Lower left and right are at ~2000 counts. * Both LN2 tanks are low * There is a bird in the enclosure ---- Observing Plan ---- Mon 27 JUN 2016 57567.0 0.50 03:38 11:17 15:02 22:42 ---- MaNGA until 07:21 ( 3.7 h), then APOGEE-2 ( 3.9 h) We actually plan to expose RM plate during MaNGA time. ---- Observing Log ---- Afternoon: ---------- Hardware, software, actors, plates, guider checked. No problems. Collimation with cart 14: 00:53:35 Hartmann collimate output on cart #14 sp1: offset: r= -1979 (Out of focus); b= 0.8 (In Focus) sp2: offset: r= 0 (In Focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -1118 sp2: pred. move: spAverageMove= -283 sp1: pred. resid: r= -861, b= -2.7, txt= OK, spTemp = 15.8 sp2: pred. resid: r= 283, b= 0.9, txt= OK, spTemp = 15.2 Guider cart= 14, plate= 7340, side= A (loaded); ---------- motorPosition sp1: 24982, 25297, 25085; sp2: 4643, 4101, 4352; ---------- Median spectrographs temperatures: Tsp1= 15.8, Tsp2= 15.2; Outside air temperature= 16.8; ---------- Boss CCD Temp : sp1r0= -125.0 (-126.6); sp1b2= -99.6 (-100.0); sp2r0= -134.0 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 11.0; sp2LN2: 8.0; ---------- gcameraTemp= -40.0 (-40.0) ---------- Apogee: vacuum = 1.09e-06 ( 85 ) ---------- Blue ring was not moved. 01:09Z ventilation started. STUI on the podium was restarted. Night: ----- 02:16:09 encl25m : open; Humidity rising. 02:50:09 encl25m : closed due to humidity. Ventilation ends. 05:18Z We received a few critical alerts with the SP2 camera temp. We noticed that the SP2 temperature is bouncing between -94 to -153. We have taken some biases and noticed that there is some weird electronic noise in B2. See images: file #223657,223658,223659 for B2. 05:24:26 .alerts 0 alerts e alert=boss.SP2B2CCDTempRead,critical,"Reported by camCheck",enabled,noack,"" We are issuing boss reloadControllers and taking another bias (file #223660). Electronic noise is still there. reloadControllers on sp2 only. File #223661 shows obvious electronic noise in one quadrant. 05:36Z Status shows the temps are in acceptable range now. Taking another bias (file #223662). Electronic noise appears in at least 3 quadrants. Took another bias (file #223663) 05:11:13 boss exposure 0.0, file=223657 05:15:27 boss exposure 0.0, file=223658 05:18:20 boss exposure 0.0, file=223659 05:21:49 boss exposure 0.0, file=223660 05:28:41 boss exposure 0.0, file=223661 05:36:21 boss exposure 0.0, file=223662 05:45:10 boss exposure 0.0, file=223663 Output on SoS: sdR-r1-00223650.fit.gz: WARNING: r1 Amp 0 high read noise = 3.65 sdR-r1-00223651.fit.gz: WARNING: r1 Amp 0 high read noise = 3.66 sdR-r1-00223654.fit.gz: WARNING: r1 Amp 0 high read noise = 3.67 sdR-r1-00223655.fit.gz: WARNING: r1 Amp 0 high read noise = 3.67 sdR-r1-00223657.fit.gz: WARNING: r1 Amp 0 high read noise = 3.59 sdR-b2-00223657.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-b2-00223658.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-r1-00223658.fit.gz: WARNING: r1 Amp 0 high read noise = 3.60 sdR-r1-00223659.fit.gz: WARNING: r1 Amp 0 high read noise = 3.60 sdR-b2-00223659.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-b2-00223659.fit.gz: WARNING: b2 Amp 3 crazy low read noise = 0.74 e-; Are CCD voltages correct? sdR-b2-00223660.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-b2-00223660.fit.gz: WARNING: b2 Amp 3 crazy low read noise = 0.90 e-; Are CCD voltages correct? sdR-r1-00223660.fit.gz: WARNING: r1 Amp 0 high read noise = 3.60 sdR-r1-00223661.fit.gz: WARNING: r1 Amp 0 high read noise = 3.61 sdR-b2-00223661.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-b2-00223661.fit.gz: WARNING: b2 Amp 2 high read noise = 26.50 sdR-b2-00223662.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-r1-00223662.fit.gz: WARNING: r1 Amp 0 high read noise = 3.61 sdR-r1-00223663.fit.gz: WARNING: r1 Amp 0 high read noise = 3.61 sdR-b2-00223663.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-b2-00223663.fit.gz: WARNING: b2 Amp 3 crazy low read noise = 0.81 e-; Are CCD voltages correct? We went down to the telescope and checked the camera electronics box on SP2. The values we saw as we cycled through each CCD led were between 97 and 99 on the display. CCD -5 and +30 had the 97 values. We also saw the LN2 line to SP2 was leaking a little. We removed the frost on the connector and reattached the LN2 line to SP2. No more leaks. Took another bias, more electronic noise. 06:13:17 boss exposure 0.0, file=223664 Called Connie and left a message. 07:42Z SP2 B2 temps have been stable for over an hour. Boss CCD Temp : sp1r0= -125.0 (-126.6); sp1b2= -99.6 (-100.0); sp2r0= -134.0 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 10.0; sp2LN2: 10.0; Took another bias: 08:42:08 boss exposure 0.0, file=223665 The electronic noise section of B2 bias is at 8000 counts. The noise appears as a clear step in the image. SoS shows: sdR-b2-00223665.fit.gz: WARNING: bias exploding in b2 crazy quadrant! sdR-b2-00223665.fit.gz: WARNING: b2 Amp 2 high read noise = 8.07 sdR-b2-00223665.fit.gz: WARNING: b2 Amp 3 high read noise = 5.11 sdR-r1-00223665.fit.gz: WARNING: r1 Amp 0 high read noise = 3.60 We are seeing in the biases that upper left quadrant has ~5400 counts upper right quadrant has ~8000 counts lower left quadrant ~2000 counts lower right quadrant ~2000 counts 10:54Z And now our night's watch ends. 11:10Z Started morning cals for eBOSS and APOGEE. ---- Telescope Status ---- 01:09Z Started ventilation 02:16Z encl25m : open 02:50Z encl25m : closed No interlocks bypassed Status at: 11:10:11 Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 14 Counterweights at: 433, 434, 432, 433 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 52.2 lbs, 14.4 psi SP2 85.9 lbs, 19.8 psi --------------- ---- Software Used ---- Get Versions, 06/28/16, 00:51:03 -- STUI software: STUI : 1.7.1b5 alerts : trunk+svn66561 apo : trunk+svn158476M boss : v4_0 -boss.specMec : 3.0.4 gcamera : v2_1 guider : v3_4 hartmann : v1_4_3 hub : Tag: UNTAGGED_REVISION 70716:70717 mcp : v6_6_18 -mcp.plc : 34 -mcp.azFiduci : v1_119 -mcp.altFiduc : v1_119 -mcp.rotFiduc : v1_119 platedb : v2_4 sop : v3_8_1 tcc : 1.2.2 apogee : not availble apogeecal : not availble apogeeql : v1_6_1 APO-local : trunk -- done -- Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 448G 241G 66% /home /dev/vdc1 11T 8.3T 2.5T 78% /data Setting up: idlspec2d Flavor: Linux64 Version: v5_7_12 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: trunk Setting up: |specflat Flavor: Linux64 Version: v1_20 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |sdss_python_module Flavor: Linux64 Version: v1_6_7 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.9.7 Setting up: |psycopg2 Flavor: Linux64 Version: 2.5.3 Setting up: |idlutils Flavor: Linux64 Version: v5_5_16 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: |photolog Flavor: Linux64 Version: trunk Setting up: actorcore Flavor: Linux64 Version: v4_0 Setting up: |opscore Flavor: Linux64 Version: v2_4 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_13 Setting up: |RO Flavor: Linux64 Version: 3.6.9 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: -0.121 -0.125 Axial B: 0.115 0.112 Axial C: -0.023 -0.028 Trans D: -1.353 -1.357 Lateral E: 10.9728 10.9728 Lateral F: 12.5984 12.6111 GALILS Commanded: 11500 -1200 5850 19600 24150 23900 Actual: 11500 -1200 5850 19600 24150 23900 SETMIR VALUES PrimDesOrient: 0.0 -24.21 6.52 356.1 1323.0 PrimOrient: 0.0 -24.31 6.14 356.65 1322.6 SECONDARY: ---------- Focus: 0.0000 Air temp: 17.2 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.024 -0.023 Axial B: -0.717 -0.724 Axial C: -0.733 -0.732 Trans D: 0 -0.009 GALILS Commanded: 1613400 1567750 1570100 -1700 -8150 Actual: 1613425 1567781 1570093 -1700 -8150 SETMIR VALUES SecDesOrient: 1257.0 4.35 -15.05 25.0 111.05 SecOrient: 1257.03 4.34 -15.08 25.25 111.22