Date: Fri, 11 Sep 2009 08:22:09 -0700 From: Constance Rockosi Subject: Re: 2.5m obslog 20090910 (MJD 55085) I looked at the darks based in Dmitry's comments: > The 10 minutes darks with b1,b2 chips look smooth, with very few and rather > natural looking cosmic ray hits. In a contrast, r1,r2 look MUCH dirtier, and > this is not a constant pattern. Suspicious. The R2 data in exposure 101186 looks like a BOSS red dark, and the noise in all four quadrants is about right in DN, 1.3-1.7 or so. Those long cosmic rays are what we get with these thick red CCDs. But the R1 noise values are much too low, > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Thursday September 10, 2009 > > (MJD 55085) > > ---=== OBSERVING TEAM ===--- > > Observers: > Howard Brewington > Dmitry Bizyaev > > > > ---=== OBSERVING PLAN ===--- > > Guider and software tests! > > > > ---=== OBSERVING SUMMARY ===--- > > > > Night was humid and cloudy, sometimes rainy and foggy. > > We fulfilled request > from Kyle and did sequences of dithered flats. Software (boss, sop) crashed > many times (see log), and finally we were unable to complete all dithered > flats. In addition to finished 1-s dithered flats with camera sp1 yesterday, > we finished > sp2 - 1s, 2s, 4s and > sp1 - 2s, 4s > 7 darks x10 min also were taken. > > Since we are dealing with scripts, as sop ditheredFlat, it would be nice to > see the number of images in the sequence that have been finished. > > > The 10 minutes darks with b1,b2 chips look smooth, with very few and rather > natural looking cosmic ray hits. In a contrast, r1,r2 look MUCH dirtier, and > this is not a constant pattern. Suspicious. > > > > > > ---=== OBSERVING LOG ===--- > > Afternoon: > ---------- > 23:00Z Started the afternoon checkout. > > 23;15Z Restarted the servers. > > 23:18Z Checked the mirror numbers. > > 23:23Z Filesystem Size Used Avail Use% Mounted on > /dev/sda3 142G 123G 12G 92% / > /dev/sda1 99M 30M 64M 33% /boot > /dev/sdb1 917G 828G 43G 96% /data > sdssfiles1p:/data1/data > 1.5T 770G 623G 56% /files > hub25mp:/home 92G 48G 45G 52% /home_hub25m > > 23:40Z Verified that the STUI actors were happy; also checked the status > of the hub25m and perms. > Reseted photo crate 2: id6 did not respond. > > Spectrographs sp1, so2 are in-spec for the current temperature, 11.5C. > > > Night: > ------ > > 01:38Z night looks now wet and cloudy. We continue to complete Kayle's test: > dithered flats for sp1 and sp2, with 1,2,4,10 sec exposures. > flat sequence started. See the summary of calibrations below. > > 04:41Z BOSS Sp1 micro not talking error. > 04:42:14 APO.hjbrew 30 boss f text="command failed: ICCError('command BLUEROFF > is active, cannot send new command 2112 LINES',) at > /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/Forth.py:126" > 04:42:36 .alerts 0 alerts e alert=boss.SP1MicroNotTalking,critical,"Reported > by camCheck",enabled,noack,"" > > We issued "boss iack", "boss clearExposure", and then "hub startNubs boss", > and started the sequence over. > > > 05:39:08 APO.hjbrew.sop 247 boss f text="command failed: > CommError(, > "TDS(TCPPort(name=None:2100 to t-g-sdss-2.apo.nmsu.edu:2100) ok=True > initialized=False) send '' '', received ''0''") at > /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/Forth.py:320" > > sequence of dithered flats stopped, lamps gone off. > ... retaking. > > > 05:54:25 boss ping shows "sp1cam DEAD". We did "status" and few more pings. > sp1 gets alive after this. Howard whistled "dixy", probably ot helped. > - iach, clearExposure, couple of pings again, and then the sp1 camera came back. > > 06:03:25 .boss 3175 boss f text="command failed: ValueError("invalid > literal > for int() with base 10: 'NaN'",) at > /home/sdss3/products/Linux64/boss/v1_1/python/boss/Commands/MechCmd.py:71" > > 06:06:06Z At this point we tried 3 times to get dithered flat with expTime=10. > All attempts failed. We decide to switch to "sp1, 2 sec". > > FAiled again, > Disregarding that "boss" actor is alive, we stopdead it and start with > stageManager. Then startNubs, and iack. > > We are sure that we used "boss clearExposure" before, but now it fails: > 06:15:17 APO.hjbrew 86 boss f text="command failed: > AttributeError("'Exposure' object has no attribute 'cmd'",) at > boss_main.py:53" > > Interesting (and incorrect): when Howard issues a command (sop ditheredFlat, > more precisely), I don't see it in my log window (Howard does see it in his), > I just see results. > > Nevertheless, we try to continue. > > 06:27:54 APO.dmbiz 32 sop w text="1 tasks failed to respond; heard from: uv > ffs ne tcc wht ff hgcd boss gcamera" > > We cannot start. "sop status" report failure of comunication with lamps: > > 06:30:44 33 sop status06:30:44 .hub 0 cmds d NewCmd=180716; > CmdTime=1252650679.05; Cmdr="APO.dmbiz"; CmdrMID=33; CmdActor="sop"; > CmdText="status"06:30:44 APO.dmbiz 33 sop i text="boss thread"06:30:44 > APO.dmbiz 33 sop i text="uv thread"06:30:44 APO.dmbiz 33 sop i text="HgCd > thread"06:30:44 APO.dmbiz 33 sop i text="Ne thread"06:30:44 APO.dmbiz 33 sop > i text="gcamera thread"06:30:45 APO.dmbiz 33 sop i text="FF thread"06:30:45 > APO.dmbiz 33 sop i text="tcc thread"06:30:45 APO.dmbiz 33 sop i text="wht > thread"06:30:45 APO.dmbiz 33 sop i text="ffs thread"06:30:46 APO.dmbiz 33 > sop w text="1 tasks failed to respond; heard from: uv ffs ne tcc wht ff hgcd > boss gcamera" > > > sop restart issued. Howard whistled "dixy" again. > Started dithered Flats again.06:35Z ... lamps are on and exposure started! > > 06:35:36 APO.hjbrew.sop 260 boss f text="command failed: NameError("global > name 'astrotime' is not defined",) at > /export/home/boss/devel/boss-trunk/python/boss/Commands/ExposureCmd.py:288" > > 06:37Z called Craig. > > Craig recommended to change "setup -r ." in our procedure for boss-icc > troubleshooting to "setup boss" from home directory. > > We repeated all steps (except "boss iack") to resurrect the boss interface. > As a result, setup finds files in correct place (i.e. runs the latest version) > and works correctly. > > 06:49Z next attempt to start the dithered flats sequence. > > 07:38Z sky is still cloudy, but humidity dropped below 80%, we start > ventilation. > > 08:25Z ventilation stopped because humidity runs up high again. > > it crashed again, at the sequence of dithered flats 10 sec exposures: > > 08:56:18 APO.hjbrew.sop 382 boss f text="command failed: CommError( ontrollers.TCPPort.TCPPort object at 0x1947c410>, 'poll timeout (60.0 secs) > reading character.') at > /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/TCPPort.py:125" > > It failed again: > > 09:15:58 APO.hjbrew.sop 398 boss f text="command failed: > CommError(, > "TDS(TCPPort(name=None:2100 to t-g-sdss-2.apo.nmsu.edu:2100) ok=True > initialized=False) send ''R'', received ''U''") at > /home/sdss3/products/Linux64/boss/v1_1/python/boss/Controllers/Forth.py:320" > > 09:40Z we got desperate of finishing the 10 sec dithered flats, it gets failed > after a few exposures every time we try them. > We mount plate 666 (black foam, found in some unusual place, on the floor of > janitor's room in plug lab) and are doing darks requested by Lyle (10x10 min). > > After 5 darks the process failed again: > > 10:58:10 .boss 253 boss f text="command failed: ValueError("invalid literal > for int() with base 10: 'NaN'",) at > /home/sdss3/products/Linux64/boss/v1_1/python/boss/Commands/MechCmd.py:71" > > Restarted boss. Restarted sop. Started 2 more darks. > > The summary of the exposures: > > --- > > Dithered Flats > > start frame finish frame sp expTime > ---------------------------------- > 101046 101067 2 1 second > 101068 101089 2 2 seconds > 101090 101111 2 4 seconds > 101124 101145 1 2 seconds > 101146 101167 1 4 seconds > > ============================ > > Darks > > frame # expTime > ---------------------------------- > 101180 600 seconds > 101181 600 seconds > 101182 600 seconds > 101183 600 seconds > 101184 600 seconds > 101186 600 seconds > 101187 600 seconds > > --- > > > 11:50Z endNight finished. > > ---=== IMAGING RUN SUMMARY ===--- > > Run Time Stripe Lambda Last Flavor Comments > Start End Begin End Frame > ------------------------------------------------------------------------- > > 8027 00:08Z 00:26Z 100 O 61.40 65.89 44 ignore > ---=== IMAGING RUN DETAILS ===--- > > ---=== SKIPPY RESULTS ===--- > > Run Frame nFrames stars muErr muRms nuErr nuRms rot az el > --------------------------------------------------------------------------- > > ---=== LTMATCH RESULTS ===--- > > Run Field nFields alt az nGood rowMean rowSig colMean colSig rot > ------------------------------------------------------------------------ > ---=== SPECTROSCOPY DATA SUMMARY ===--- > > Summary Checked (y/n): > QA Procedures Done (y/n): > > UT Exp Time flavor comment (S/N)^2 totals > ========================================== b1 r1 b2 r2 > > ---=== TELESCOPE OFFSETS AND SCALE I ===--- > > Time Instrument Az Alt Rot Scale > pos offset pos offset pos offset > ------------------------------------------------------------------------------ > ---=== TELESCOPE OFFSETS AND SCALE II ===--- > > > > ---=== FOCUS LOG ===--- > > setmir piston Temp Wind > Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm > ------------------------------------------------------------------------------ > ---=== WEATHER LOG ===--- > > Wind > Time Temp F Dewp F MPH Direction Dust DIMM Sky > > 23:16Z 53 49 6 29 (NNE) 525 - > 23:49Z 53 49 8 62 (ENE) 542 - > 00:21Z 51 49 11 74 (ENE) 591 - clouds > 00:33Z 51 49 10 41 (NE) 597 - '' > 01:06Z 51 48 13 38 (NE) 715 - '' > 01:38Z 51 48 13 76 (ENE) 644 - '' > 02:10Z 51 45 13 67 (ENE) 520 - '' > 02:42Z 51 44 12 76 (ENE) 459 - '' > 03:14Z 50 45 9 81 (E) 488 - '' > 03:46Z 52 43 2 335 (NNW) 453 - '' > 04:19Z 53 41 0 326 (NW) 453 - '' > 04:51Z 52 42 7 107 (ESE) 506 - '' > 05:23Z 51 42 7 90 (E) 482 - '' > 05:55Z 51 43 9 86 (E) 522 - '' > 06:27Z 51 42 9 94 (E) 538 - '' > 06:59Z 51 41 10 87 (E) 439 - '' > 07:32Z 51 40 13 86 (E) 363 - '' > 08:04Z 49 42 14 84 (E) 490 - '' > 08:36Z 49 44 14 79 (E) 685 - '' > 09:08Z 49 46 11 68 (ENE) 811 - '' > 09:40Z 51 43 8 55 (NE) 681 - '' > 10:12Z 51 43 6 88 (E) 762 - '' > 10:45Z 52 40 1 239 (WSW) 697 - '' > 11:17Z 51 40 1 252 (WSW) 502 - '' > 11:49Z 50 42 1 95 (E) 651 - '' > ---=== TELESCOPE STATUS ===--- > > 07:38Z Doors and louvers opened, fans on. > 08:25Z ventilation stopped, doors closed. > > Handpaddle at the cartridge lift is switched to the manual side. > > Status at: 11:35Z > Telescope stowed at: 30,121,00 mount > Instrument mounted: Cartridge 11 > Counterweights at: 250 units > LN2 autofill systems: Connected and turned on. > 180L LN2 dewar scales: Spectro 213 lbs, 16.5 psi > Imager 187 lbs, 23.4 psi > > > > > ---=== SOFTWARE USED ===--- > > IOP/SOP: v4_40_0 > Watcher: v2_36_0 > MCP: NOSVN > TPM: tpm_v3_5_0 > AstroDa: v15_13_1 > TCC: TCC 2.11.0 2009-06-24 > sdssProcedures: v2_44 > SoS: v5_1_8 > plate-mapper: v4_3_1 > > ---=== MIRROR NUMBERS ===--- > > PRIMARY: > -------- > Scale: 1.000000 > MIGS TONIGHT NOMINAL > Axial A -0.6930 -0.6940 > Axial B -0.0300 -0.0410 > Axial C -0.3710 -0.3710 > Trans D 11.6130 11.6170 > Lateral E 10.3759 10.3505 > Lateral F 11.9380 11.9253 > GALILS > Commanded: 9900. 400. 5850. 1200. 2950. 2750. > Actual: 9900. 400. 5850. 1200. 2950. 2750. > SETMIR VALUES > PriDesOrient: 0.00 -3.30 5.10 278.08 178.06 > PriOrient: 0.00 -3.04 5.26 277.70 178.07 > SECONDARY: > ---------- > Focus: 0.00 > Air Temp.: 12.5 > Alt.: 30.002867 > MIGS TONIGHT NOMINAL > Axial A 18.6200 18.6150 > Axial B 15.6260 15.6120 > Axial C 13.8420 13.8170 > Trans D -11.7880 -11.7780 > GALILS > Commanded: 1597816. 1566232. 1588191. -7600. -12400. > Actual: 1597700. 1566200. 1588100. -7600. -12400. > SETMIR VALUES > SecDesOrient: 1257.00 -4.00 -9.00 10.00 234.42 > SecOrient: 1257.06 -4.01 -9.01 10.18 234.70 > ---=== PROBLEMS IN DETAIL ===--- > > Collimation with cart. 11. > Telescope's T = 11.5 C. > > IDL> collimate, 101044 > % Compiled module: COLLIMATE. > > Reading /data/spectro/55085/sdR-b1-00101044.fit > SDSSPROC: Read noise = 0.585158 0.435157 0.582165 0.557161 electrons > Reading /data/spectro/55085/sdR-b1-00101045.fit > SDSSPROC: Read noise = 0.587488 0.447892 0.582358 0.557670 electrons > COLLIMATE: Min offset = -1.80 pix > COLLIMATE: Max offset = 1.80 pix > COLLIMATE: RMS across CCD = 0.87 pix > COLLIMATE: Mean offset = -0.18 pix > COLLIMATE: Camera b1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) > COLLIMATE: Predict blue camera ring movement of 6.4 degrees for spectro-1 > COLLIMATE: (if red is already in focus) > Log file = Collimate-55085-b1-00101044.log > PostScript file = Collimate-55085-b1-00101044.ps > > Reading /data/spectro/55085/sdR-b2-00101044.fit > SDSSPROC: Read noise = 1.73533 1.87991 1.92955 1.96183 electrons > Reading /data/spectro/55085/sdR-b2-00101045.fit > SDSSPROC: Read noise = 1.78719 1.91790 1.92945 1.96153 electrons > COLLIMATE: Min offset = -1.00 pix > COLLIMATE: Max offset = 1.20 pix > COLLIMATE: Mean offset = -0.23 pix > COLLIMATE: Camera b2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) > COLLIMATE: Predict blue camera ring movement of 8.3 degrees for spectro-2 > COLLIMATE: (if red is already in focus) > Log file = Collimate-55085-b2-00101044.log > PostScript file = Collimate-55085-b2-00101044.ps > > Reading /data/spectro/55085/sdR-r1-00101044.fit > SDSSPROC: Read noise = 1.65777 1.74204 1.61479 2.90231 electrons > Reading /data/spectro/55085/sdR-r1-00101045.fit > SDSSPROC: Read noise = 1.65816 1.74076 1.61525 2.92042 electrons > COLLIMATE: Min offset = -1.80 pix > COLLIMATE: Max offset = 1.80 pix > COLLIMATE: RMS across CCD = 0.82 pix > COLLIMATE: Mean offset = -0.03 pix > COLLIMATE: Camera r1 appears to be IN-FOCUS (|mean| COLLIMATE: Predict (red) piston movement of 314 steps for spectro-1 > COLLIMATE: Issue SOP command: "sp1; mechPiston 314" > Log file = Collimate-55085-r1-00101044.log > PostScript file = Collimate-55085-r1-00101044.ps > > Reading /data/spectro/55085/sdR-r2-00101044.fit > SDSSPROC: Read noise = 3.23377 3.32897 3.29331 3.70840 electrons > Reading /data/spectro/55085/sdR-r2-00101045.fit > SDSSPROC: Read noise = 2.43857 2.62400 2.39312 2.59019 electrons > COLLIMATE: Min offset = -1.20 pix > COLLIMATE: Max offset = 0.40 pix > COLLIMATE: RMS across CCD = 0.35 pix > COLLIMATE: Mean offset = -0.06 pix > COLLIMATE: Camera r2 appears to be IN-FOCUS (|mean| COLLIMATE: Predict (red) piston movement of 543 steps for spectro-2 > COLLIMATE: Issue SOP command: "sp2; mechPiston 543" > Log file = Collimate-55085-r2-00101044.log > PostScript file = Collimate-55085-r2-00101044.ps > > Looks in good collimation. > > > DSS DSS DSS DSS DSS Digital Sky Survey DSS DSS DSS DSS DSS DSS > DSS *** The information posted to SDSS mailing lists is proprietary *** > DSS This is message 4718 in the sdss-25mlog archive, URL > DSS http://www.astro.princeton.edu:81/sdss-25mlog/msg.4718.html > DSS The index is at http://www.astro.princeton.edu:81/sdss-25mlog/INDEX.html > DSS To join/leave the list, go to > DSS http://www.astro.princeton.edu:81/cgi-bin/mailinglists.pl/show_subscription?list=sdss-25mlog > DSS To post a message, mail it to sdss-25mlog@astro.princeton.edu > DSS > DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS > > !DSPAM:10028,4aaa3a23294573225631196! > -- Associate Professor, Assoc. Astronomer UC Santa Cruz, UCO/Lick Observatory Office: 831 459 5246 crockosi@ucolick.org http://www.ucolick.org/~crockosi