Date: Sun, 13 Jan 2013 14:55:10 -0700 (MST) Subject: Re: [sdss3-25mlog 1027] Re: 2.5m obslog 20130112 (MJD 56305) From: "Daniel Oravetz" The fill from the 180L dewars was completed by pushing the fill button while still in auto mode. We did not toggle to manual mode. A couple months back we had a situation in which one of the 10L dewars had been _too_ full of LN2 such that there wasn't a sufficient gas head to establish a pressure differential and complete a fill to the primary dewar/CCDs when a fill was triggered. In this case the pressure had read low (can't recall how close to 0, but it was low enough to trigger the SecondaryDewarPressure Critical Alert) because there was too much LN2. French remedied this problem by using the pressure from the 180L dewar to expel LN2 from the secondary/10L dewar and the pressure increased accordingly. I think this had been a full sensor problem that had allowed too much LN2 to transfer before stopping the fill. Having experienced that situation, when we arrived at the telescope after the Critical Alert had been issued and saw that the sp1 180L dewar head was very snowy, and the autofill lines and the connection between the 180L dewar and the 10L dewar also appeared snowy, we thought perhaps a fill had just completed from the 180L dewar and because it was so cold (-15C), there hadn't been sufficient boil-off to establish the gas head, so the pressure was reading low. We tried a 'startfillseq' (fill from 10L dewar to camera) and could hear the sound of the switch (solenoid?) that precedes/triggers the fill, but only sp2 was actually transferring LN2 during the fill sequence (at first, this led us to think maybe another bad full sensor and the 10L dewar was too full). During this sequence in which sp2 was properly filling from its 10L dewar, but sp1 was not transferring LN2 from its 10L dewar, the 10L dewar pressures were 11 (sp2) and 4 (sp1). When we checked the dewar weight plots we saw that it had been ~16 hours since the last fill from the 180L dewars completed (we no longer thought the 10L dewar could possibly be too full). We decided to force the fill from the 180L dewars and watch, and both 10L dewars were accepting LN2 as expected and at the conclusion of this force-fill-in-auto-mode, the sp1 180L dewar weight dropped about 20lbs, which I think is indicative of a typical (i.e., 10L dewars ~empty) auto-triggered fill. We did notice that there was noticeable snow/frozen condensation on metal surfaces behind the telescope; even the telescope laptop felt like it had a thin layer of ice on it. Not sure just how much the cold temperatures or the combination of cold temperatures and high humidity producing the frozen condensation were to blame for last night's problem. Dan > I, too. It is, of course, true that the 10L pressure is mainteined and > restored after a fill by boiloff, which is largely generated by radiation > from the walls of the dewars, so one would expect this to happen more > slowly when it is very cold. But in the bout of REALLY low temperatures > (-30? -40?) a couple or three years ago I think we had no trouble of > this sort. > > --j > > > > On Sun, 13 Jan 2013, Dan Long wrote: > >> Hi Audrey and Dan, >> >> I'm curious about the low dewar pressure event last night. What did >> you do to make the 180 L fill the 10 L dewars? Did you just push the >> fill button with the system in auto mode, or did you have to force the >> fill by switching to manual? >> >> Since the lines were hooked up, the empty sensor in SP1's 10L should >> have triggered a fill. It could have been prevented by a poor ground >> on the connectors, but then pushing the fill button should have >> suffered the same problem. If the empty sensor in SP1's 10L failed, >> then it would be consistent with a fill working by pushing the fill >> button. >> >> It is also a bit puzzling that the dewar pressure was low but not >> zero, indicating SP1's 10L may not have been empty, but just low on >> pressure. >> >> Cheers, >> >> Dan >> >> >> On Jan 13, 2013, at 5:06 AM, asimmons@apo.nmsu.edu wrote: >> >>> >>> APACHE POINT OBSERVATORY SDSS 2.5M >>> >>> OBSERVING LOG >>> Saturday January 12, 2013 >>> >>> (MJD 56305) >>> >>> ---- Observing Team ---- >>> >>> Daniel Oravetz >>> Audrey Oravetz >>> >>> Kaike Pan (afternoon) >>> >>> >>> ---- Observing Plan ---- >>> >>> 12 Jan 2013 56305.0 0.12 01:30 12:54 01:57 13:24 BOSS 11.4 hours >>> >>> >>> ---- Observing Summary ---- >>> >>> We never opened, mainly due to low temperatures. The lowest the >>> temperature got >>> was -15C. It mostly stayed around -14C. We had a few snow flurries at >>> the >>> beginning of the night as well, but nothing accumulated on the ground. >>> >>> Note: >>> ===== >>> >>> The west door failed to activate when we tried to close it. We were >>> forced to >>> use the chain to manually close it, but luckily it only needed to move >>> a few >>> feet. Perhaps this problem could be related to the very cold >>> temperatures. >>> >>> At 09:05Z, we received a critical alert for sp1 dewar pressure. We >>> initiated a >>> fill and solved the problem. See log for more details. >>> >>> >>> ---- Observing Log ---- >>> >>> Afternoon: >>> --------- >>> BOSS collimation was checked with cart 10. Both SP1 and SP2 blue focus >>> rings were adjusted by -2 degrees. >>> >>> COLLIMATE: Camera b1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) >>> COLLIMATE: Predict blue camera ring movement of -4.0 degrees for >>> spectro-1 >>> COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) >>> COLLIMATE: Predict blue camera ring movement of 4.4 degrees for >>> spectro-2 >>> COLLIMATE: Predict (red) piston movement of 256 steps for spectro-1 >>> COLLIMATE: Issue command: "boss moveColl spec=sp1 a=0 b=0 c=0" >>> COLLIMATE: Predict (red) piston movement of 555 steps for spectro-2 >>> COLLIMATE: Issue command: "boss moveColl spec=sp2 a=255 b=255 c=255" >>> >>> motorPosition >>> sp1: -9120, -9127, -9420; >>> sp2: -6059, -7246, -7361; >>> >>> Median spectrographs temperatures: >>> Tsp1= -2.7, Tsp2= -3.4; >>> Outside air temperature= -8.6 >>> >>> All systems and actors were checked. Migs controllers was not >>> communicating, was >>> rebooted. >>> >>> 23:10Z mounted cart 12, started ventilation. >>> >>> Night: >>> ====== >>> >>> We had high wind gusts, and the temperature was -10C at sunset. We did >>> not open >>> right away. >>> >>> 03:17Z - Stopped ventilating. Humidity quickly climbed to 98%. >>> Temperature at >>> -13.5C. Snow flurries. The west door did not operate normally. We had >>> to use the >>> chain to manually close the door (about 1-2 feet). This is probably >>> related to >>> the very low temperature. >>> >>> 07:27Z - Ran APOGEE scheduler and started morning cals. Started BOSS >>> cals. >>> >>> 09:02Z - Critical alert - sp1dewarpressure. A primary fill was issued >>> with no >>> help. Sp1 primary dewar pressure was a 4psi, whereas Sp2 dewar pressure >>> was at >>> 11psi. A secondary fill was forced, and then another primary. This >>> solved the >>> problem. We noticed that the last time the spectrographs filled was 16 >>> hours >>> ago, so it should have automatically begun another fill but didn't. >>> 09:01:25 .alerts 0 alerts e >>> alert=boss.SP1SecondaryDewarPress,critical,"Reported >>> by camCheck",enabled,noack,"" >>> >>> 11:59Z - The sky cleared and humidity improved, but the temperature >>> remained at >>> -15C. There is no hope of opening tonight. >>> >>> >>> ---- Telescope Status ---- >>> >>> 23:30Z Started ventilation >>> 03:17Z - Stopped ventilating. >>> >>> No interlocks bypassed >>> >>> Status at: 11:46Z >>> Telescope stowed at: 121.0, 30.0, 0.0 mount >>> Instrument mounted: cartridge = 12 >>> Counterweights at: 280 >>> LN2 autofill systems: Connected and turned on >>> 180L LN2 dewar scales: SP1 212 lbs, 19 psi >>> SP2 215 lbs, 20 psi >>> >>> >>> >>> >>> ---- Software Used ---- >>> >>> STUI: 1.2 2012-12-19; >>> alerts: v1_2_1; >>> apo: ** FAILED **; >>> boss: v2_5_13; >>> boss.daq: 1.3.2; >>> boss.specMech: 3.0.1; >>> gcamera: v1_2_10; >>> guider: v1_4_1; >>> hub: Tag: sdss3_v1_2_0; >>> mcp: v6_6_16; >>> mcp.plc: 28; >>> mcp.azFiducialVersion: v1_107; >>> mcp.altFiducialVersion: v1_107; >>> mcp.rotFiducialVersion: v1_107; >>> platedb: v1_0_5; >>> sop: v1_2_16; >>> sos: v1_2_0; >>> sos.idlspec2d: v5_6_2; >>> tcc: 2.14.0; >>> apogee: ** not availble ** >>> apogeecal: ** not availble ** >>> apogeeql: v1_4_1 >>> >>> idlspec2d (sos3) --- v5_6_2 >>> plate-mapper3 ------ NOCVS:v5_0_16 >>> >>> >>> >>> ---- Mirror Numbers ---- >>> >>> >>> PRIMARY: >>> -------- >>> Scale: 1.000000 >>> >>> MIGS TONIGHT NOMINAL >>> Axial A: 5.361 5.357 >>> Axial B: 5.212 5.221 >>> Axial C: 5.519 5.516 >>> Trans D: 10.353 10.374 >>> Lateral E: 10.6553 10.6299 >>> Lateral F: 12.2809 12.2428 >>> >>> GALILS >>> Commanded: 11300 -650 5500 21900 13500 13050 >>> Actual: 11300 -650 5500 21900 13500 13050 >>> >>> SETMIR VALUES >>> PrimDesOrient: 0.00 0.00 0.00 612.35 1500.90 >>> PrimOrient: 0.00 0.00 0.00 612.07 1500.38 >>> >>> SECONDARY: >>> ---------- >>> >>> Focus: 0.00 >>> Air temp: -9.5 >>> Altitude: 30.0 >>> >>> MIGS TONIGHT NOMINAL >>> Axial A: 7.039 7.041 >>> Axial B: 7.047 7.059 >>> Axial C: 7.027 7.041 >>> Trans D: 0.335 0.33 >>> >>> GALILS >>> Commanded: 1576550 1594600 1580000 -3550 -6650 >>> Actual: 1578036 1593690 1579775 -3550 -6650 >>> >>> SETMIR VALUES >>> SecDesOrient: 1257.00 3.00 5.00 0.00 115.81 >>> SecOrient: 1257.08 3.04 4.26 -0.66 116.11 >>> >>> >>> >>> _____________________________________________________ >>> Message 1025 of sdss3-25mlog - Logs of nightly APO activities. >>> https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1025.html >>> sdss3-25mlog@sdss3.org >>> https://sdss3.org/mailman/listinfo/sdss3-25mlog >> >> _____________________________________________________ >> Message 1026 of sdss3-25mlog - Logs of nightly APO activities. >> https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1026.html >> sdss3-25mlog@sdss3.org >> https://sdss3.org/mailman/listinfo/sdss3-25mlog >> > _____________________________________________________ > Message 1027 of sdss3-25mlog - Logs of nightly APO activities. > https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1027.html > sdss3-25mlog@sdss3.org > https://sdss3.org/mailman/listinfo/sdss3-25mlog >