Subject: Re: [sdss3-25mlog 1027] Re: 2.5m obslog 20130112 (MJD 56305) From: Dan Long Date: Mon, 14 Jan 2013 08:00:41 -0700 Hi Dan, Thanks for the added info. The frosty connection between the 180 and = 10 L dewars is an indication that it had been trying to fill but being = stopped short, probably due to a bad ground connection. The manually = triggered fill must have either gotten lucky with the ground (or perhaps = you reseated the connection first?). Either way, we will check the = condition of the ground for the connections. -Dan On Jan 13, 2013, at 2:55 PM, Daniel Oravetz wrote: > The fill from the 180L dewars was completed by pushing the fill button > while still in auto mode. We did not toggle to manual mode. >=20 > A couple months back we had a situation in which one of the 10L dewars = had > been _too_ full of LN2 such that there wasn't a sufficient gas head to > establish a pressure differential and complete a fill to the primary > dewar/CCDs when a fill was triggered. In this case the pressure had = read > low (can't recall how close to 0, but it was low enough to trigger the > SecondaryDewarPressure Critical Alert) because there was too much LN2. > French remedied this problem by using the pressure from the 180L dewar = to > expel LN2 from the secondary/10L dewar and the pressure increased > accordingly. I think this had been a full sensor problem that had = allowed > too much LN2 to transfer before stopping the fill. >=20 > Having experienced that situation, when we arrived at the telescope = after > the Critical Alert had been issued and saw that the sp1 180L dewar = head > was very snowy, and the autofill lines and the connection between the = 180L > dewar and the 10L dewar also appeared snowy, we thought perhaps a fill = had > just completed from the 180L dewar and because it was so cold (-15C), > there hadn't been sufficient boil-off to establish the gas head, so = the > pressure was reading low. We tried a 'startfillseq' (fill from 10L = dewar > to camera) and could hear the sound of the switch (solenoid?) that > precedes/triggers the fill, but only sp2 was actually transferring LN2 > during the fill sequence (at first, this led us to think maybe another = bad > full sensor and the 10L dewar was too full). During this sequence in = which > sp2 was properly filling from its 10L dewar, but sp1 was not = transferring > LN2 from its 10L dewar, the 10L dewar pressures were 11 (sp2) and 4 = (sp1). > When we checked the dewar weight plots we saw that it had been ~16 = hours > since the last fill from the 180L dewars completed (we no longer = thought > the 10L dewar could possibly be too full). We decided to force the = fill > from the 180L dewars and watch, and both 10L dewars were accepting LN2 = as > expected and at the conclusion of this force-fill-in-auto-mode, the = sp1 > 180L dewar weight dropped about 20lbs, which I think is indicative of = a > typical (i.e., 10L dewars ~empty) auto-triggered fill. >=20 > We did notice that there was noticeable snow/frozen condensation on = metal > surfaces behind the telescope; even the telescope laptop felt like it = had > a thin layer of ice on it. Not sure just how much the cold = temperatures or > the combination of cold temperatures and high humidity producing the > frozen condensation were to blame for last night's problem. >=20 > Dan >=20 >=20 >> I, too. It is, of course, true that the 10L pressure is mainteined = and >> restored after a fill by boiloff, which is largely generated by = radiation >> from the walls of the dewars, so one would expect this to happen more >> slowly when it is very cold. But in the bout of REALLY low = temperatures >> (-30? -40?) a couple or three years ago I think we had no trouble of >> this sort. >>=20 >> --j >>=20 >>=20 >>=20 >> On Sun, 13 Jan 2013, Dan Long wrote: >>=20 >>> Hi Audrey and Dan, >>>=20 >>> I'm curious about the low dewar pressure event last night. What = did >>> you do to make the 180 L fill the 10 L dewars? Did you just push = the >>> fill button with the system in auto mode, or did you have to force = the >>> fill by switching to manual? >>>=20 >>> Since the lines were hooked up, the empty sensor in SP1's 10L = should >>> have triggered a fill. It could have been prevented by a poor = ground >>> on the connectors, but then pushing the fill button should have >>> suffered the same problem. If the empty sensor in SP1's 10L failed, >>> then it would be consistent with a fill working by pushing the fill >>> button. >>>=20 >>> It is also a bit puzzling that the dewar pressure was low but not >>> zero, indicating SP1's 10L may not have been empty, but just low on >>> pressure. >>>=20 >>> Cheers, >>>=20 >>> Dan >>>=20 >>>=20 >>> On Jan 13, 2013, at 5:06 AM, asimmons@apo.nmsu.edu wrote: >>>=20 >>>>=20 >>>> APACHE POINT OBSERVATORY SDSS 2.5M >>>>=20 >>>> OBSERVING LOG >>>> Saturday January 12, 2013 >>>>=20 >>>> (MJD 56305) >>>>=20 >>>> ---- Observing Team ---- >>>>=20 >>>> Daniel Oravetz >>>> Audrey Oravetz >>>>=20 >>>> Kaike Pan (afternoon) >>>>=20 >>>>=20 >>>> ---- Observing Plan ---- >>>>=20 >>>> 12 Jan 2013 56305.0 0.12 01:30 12:54 01:57 13:24 BOSS 11.4 = hours >>>>=20 >>>>=20 >>>> ---- Observing Summary ---- >>>>=20 >>>> We never opened, mainly due to low temperatures. The lowest the >>>> temperature got >>>> was -15C. It mostly stayed around -14C. We had a few snow flurries = at >>>> the >>>> beginning of the night as well, but nothing accumulated on the = ground. >>>>=20 >>>> Note: >>>> =3D=3D=3D=3D=3D >>>>=20 >>>> The west door failed to activate when we tried to close it. We were >>>> forced to >>>> use the chain to manually close it, but luckily it only needed to = move >>>> a few >>>> feet. Perhaps this problem could be related to the very cold >>>> temperatures. >>>>=20 >>>> At 09:05Z, we received a critical alert for sp1 dewar pressure. We >>>> initiated a >>>> fill and solved the problem. See log for more details. >>>>=20 >>>>=20 >>>> ---- Observing Log ---- >>>>=20 >>>> Afternoon: >>>> --------- >>>> BOSS collimation was checked with cart 10. Both SP1 and SP2 blue = focus >>>> rings were adjusted by -2 degrees. >>>>=20 >>>> COLLIMATE: Camera b1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) >>>> COLLIMATE: Predict blue camera ring movement of -4.0 degrees for >>>> spectro-1 >>>> COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) >>>> COLLIMATE: Predict blue camera ring movement of 4.4 degrees for >>>> spectro-2 >>>> COLLIMATE: Predict (red) piston movement of 256 steps for = spectro-1 >>>> COLLIMATE: Issue command: "boss moveColl spec=3Dsp1 a=3D0 b=3D0 = c=3D0" >>>> COLLIMATE: Predict (red) piston movement of 555 steps for = spectro-2 >>>> COLLIMATE: Issue command: "boss moveColl spec=3Dsp2 a=3D255 b=3D255 = c=3D255" >>>>=20 >>>> motorPosition >>>> sp1: -9120, -9127, -9420; >>>> sp2: -6059, -7246, -7361; >>>>=20 >>>> Median spectrographs temperatures: >>>> Tsp1=3D -2.7, Tsp2=3D -3.4; >>>> Outside air temperature=3D -8.6 >>>>=20 >>>> All systems and actors were checked. Migs controllers was not >>>> communicating, was >>>> rebooted. >>>>=20 >>>> 23:10Z mounted cart 12, started ventilation. >>>>=20 >>>> Night: >>>> =3D=3D=3D=3D=3D=3D >>>>=20 >>>> We had high wind gusts, and the temperature was -10C at sunset. We = did >>>> not open >>>> right away. >>>>=20 >>>> 03:17Z - Stopped ventilating. Humidity quickly climbed to 98%. >>>> Temperature at >>>> -13.5C. Snow flurries. The west door did not operate normally. We = had >>>> to use the >>>> chain to manually close the door (about 1-2 feet). This is probably >>>> related to >>>> the very low temperature. >>>>=20 >>>> 07:27Z - Ran APOGEE scheduler and started morning cals. Started = BOSS >>>> cals. >>>>=20 >>>> 09:02Z - Critical alert - sp1dewarpressure. A primary fill was = issued >>>> with no >>>> help. Sp1 primary dewar pressure was a 4psi, whereas Sp2 dewar = pressure >>>> was at >>>> 11psi. A secondary fill was forced, and then another primary. This >>>> solved the >>>> problem. We noticed that the last time the spectrographs filled was = 16 >>>> hours >>>> ago, so it should have automatically begun another fill but didn't. >>>> 09:01:25 .alerts 0 alerts e >>>> alert=3Dboss.SP1SecondaryDewarPress,critical,"Reported >>>> by camCheck",enabled,noack,"" >>>>=20 >>>> 11:59Z - The sky cleared and humidity improved, but the temperature >>>> remained at >>>> -15C. There is no hope of opening tonight. >>>>=20 >>>>=20 >>>> ---- Telescope Status ---- >>>>=20 >>>> 23:30Z Started ventilation >>>> 03:17Z - Stopped ventilating. >>>>=20 >>>> No interlocks bypassed >>>>=20 >>>> Status at: 11:46Z >>>> Telescope stowed at: 121.0, 30.0, 0.0 mount >>>> Instrument mounted: cartridge =3D 12 >>>> Counterweights at: 280 >>>> LN2 autofill systems: Connected and turned on >>>> 180L LN2 dewar scales: SP1 212 lbs, 19 psi >>>> SP2 215 lbs, 20 psi >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> ---- Software Used ---- >>>>=20 >>>> STUI: 1.2 2012-12-19; >>>> alerts: v1_2_1; >>>> apo: ** FAILED **; >>>> boss: v2_5_13; >>>> boss.daq: 1.3.2; >>>> boss.specMech: 3.0.1; >>>> gcamera: v1_2_10; >>>> guider: v1_4_1; >>>> hub: Tag: sdss3_v1_2_0; >>>> mcp: v6_6_16; >>>> mcp.plc: 28; >>>> mcp.azFiducialVersion: v1_107; >>>> mcp.altFiducialVersion: v1_107; >>>> mcp.rotFiducialVersion: v1_107; >>>> platedb: v1_0_5; >>>> sop: v1_2_16; >>>> sos: v1_2_0; >>>> sos.idlspec2d: v5_6_2; >>>> tcc: 2.14.0; >>>> apogee: ** not availble ** >>>> apogeecal: ** not availble ** >>>> apogeeql: v1_4_1 >>>>=20 >>>> idlspec2d (sos3) --- v5_6_2 >>>> plate-mapper3 ------ NOCVS:v5_0_16 >>>>=20 >>>>=20 >>>>=20 >>>> ---- Mirror Numbers ---- >>>>=20 >>>>=20 >>>> PRIMARY: >>>> -------- >>>> Scale: 1.000000 >>>>=20 >>>> MIGS TONIGHT NOMINAL >>>> Axial A: 5.361 5.357 >>>> Axial B: 5.212 5.221 >>>> Axial C: 5.519 5.516 >>>> Trans D: 10.353 10.374 >>>> Lateral E: 10.6553 10.6299 >>>> Lateral F: 12.2809 12.2428 >>>>=20 >>>> GALILS >>>> Commanded: 11300 -650 5500 21900 13500 13050 >>>> Actual: 11300 -650 5500 21900 13500 13050 >>>>=20 >>>> SETMIR VALUES >>>> PrimDesOrient: 0.00 0.00 0.00 612.35 1500.90 >>>> PrimOrient: 0.00 0.00 0.00 612.07 1500.38 >>>>=20 >>>> SECONDARY: >>>> ---------- >>>>=20 >>>> Focus: 0.00 >>>> Air temp: -9.5 >>>> Altitude: 30.0 >>>>=20 >>>> MIGS TONIGHT NOMINAL >>>> Axial A: 7.039 7.041 >>>> Axial B: 7.047 7.059 >>>> Axial C: 7.027 7.041 >>>> Trans D: 0.335 0.33 >>>>=20 >>>> GALILS >>>> Commanded: 1576550 1594600 1580000 -3550 -6650 >>>> Actual: 1578036 1593690 1579775 -3550 -6650 >>>>=20 >>>> SETMIR VALUES >>>> SecDesOrient: 1257.00 3.00 5.00 0.00 115.81 >>>> SecOrient: 1257.08 3.04 4.26 -0.66 116.11 >>>>=20 >>>>=20 >>>>=20 >>>> _____________________________________________________ >>>> Message 1025 of sdss3-25mlog - Logs of nightly APO activities. >>>> = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1025.h= tml >>>> sdss3-25mlog@sdss3.org >>>> https://sdss3.org/mailman/listinfo/sdss3-25mlog >>>=20 >>> _____________________________________________________ >>> Message 1026 of sdss3-25mlog - Logs of nightly APO activities. >>> = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1026.h= tml >>> sdss3-25mlog@sdss3.org >>> https://sdss3.org/mailman/listinfo/sdss3-25mlog >>>=20 >> _____________________________________________________ >> Message 1027 of sdss3-25mlog - Logs of nightly APO activities. >> = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1027.h= tml >> sdss3-25mlog@sdss3.org >> https://sdss3.org/mailman/listinfo/sdss3-25mlog >>=20 >=20 >=20