Subject: 2.5m obslog 20220127 (SJD 59607) From: kgrabowski@apo.nmsu.edu Date: Fri, 28 Jan 2022 11:59:48 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday January 27, 2022 (SJD 59607) ---- Observing Team ---- Dylan Gatlin Katie Grabowski STUI: Jose Sanchez Gallego, Conor Sayres Zoom: Dmitry Bizyaev, Karen Kinemuchi, Kaike Pan Arlin Cortes, Pablo Vera, Consuelo Gonzalez Avila, Paulo Assuncao Lago ---- Observing Plan ---- Continue doing 18-exposure BOSS 0.1 dither sequences with fvc --no-apply as yesterday. Science design 16716 (YSO-centric field) - observability best early in the night, i.e., before 23:00 local time. Please update to Boson alpha.28 although there may be an update during the night to continue debugging the guider problem. Depending on conditions and whether the intra/extra-focal filters were installed tonight we may do some testing for automatic focusing. ---- Observing Summary ---- 00:36:51 APO.dgatlin 48 tcc f Exception=RuntimeError; text="No earth orientation data for this TAI date" Conor updated the earthpred.dat file. We restarted tcc GFA 2 has a 4mm r filter and GFA 5 now has a 6mm r filter. All the others remain the same 5mm thickness. Rough focus offsets [0., -85.3, -60.93, 0, 67.99, 11.85] assuming that GFA 4 is zero. Flat field lamp '0111' in the header of the boss images doesn't seem to be turning on, and SOS rejects the flat images and doesn't reduce the science data. Completed science exposure and a few boss dithers, but closed for high humidity and wind. Light dusting of snow kept us closed for the second half. Jose tested some FPS alerts, more will need to be done with the new chiller. Snow on the grass, not the sidewalks. ---- Observing Log ---- Afternoon: APOGEE cals started BOSS cals taken 22:53:02 boss exposure 4.0, file=336729 22:53:43 boss exposure 4.0, file=336730 22:54:29 sp1.motor.move= 2633, 2633, 2632 22:54:29 Hartmann collimate output sp1: offset: r= 2308 (Out of focus); b= -9.3 (Out of focus) sp1: pred. move: spAverageMove= 2632 sp1: pred. resid: r= -324, b= -1.0, txt= OK, spTemp = 3.9 Focus in spec, mirror numbers not adjusted Mirror numbers in spec The out-of-plane filters were installed today, the 4mm was installed on sextant 2 and the 6mm was installed on sextant 5. 23:40Z Started ventilation Clear for now, possible clouds incoming later in the night. Ready to open at sunset. 00:23:27 CmdStarted: APO.dgatlin 44 tcc track 00:23:28 APO.dgatlin 44 tcc f Exception=RuntimeError; text="No earth orientation data for this TAI date" 00:23:28 CmdDone: APO.dgatlin 44 tcc track 00:23:33 CmdStarted: APO.dgatlin 45 tcc track 00:23:33 APO.dgatlin 45 tcc f Exception=RuntimeError; text="No earth orientation data for this TAI date" 00:23:33 CmdDone: APO.dgatlin 45 tcc track Called Conor. Conor logged in and got us back on track. Updated earthpred.dat MIGS TONIGHT NOMINAL Axial A: -9.8850 -9.6570 Axial B: -9.9020 -9.6710 Axial C: -9.9480 -9.6970 Trans D: 0.0000 0.1140 All the values seemed ok yesterday evening. 00:46:50 .tcc 0 tcc W Text="Galil is busy executing: A=4891627; B=4890715; C=4912386; D=-19; E=-3419; XQ #MOVE, showing cached status" We reset the MIGS on the PMSS MUX box. 01:02:49 encl25m : open; 01:11:14 Config Loaded: 16716-2922 01:21:29 Jaeger collision: 184 01:23:45 Jaeger collision: 184 01:23:49 APO.dgatlin_boson.jaeger 125 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 30.93 seconds after start" 01:23:49 CmdDone: APO.dgatlin_boson.jaeger 125 jaeger configuration execute 01:23:49 APO.dgatlin_boson 25 hal e error="Command jaeger configuration execute failed." Reloaded with no collisions ============================================================================== SCIENCE DESIGN 16716:YSO-centric ============================================================================== 01:26:57 Config Loaded: 16716-2923 01:28:08 boss exposure 4.0, file=336733 01:28:49 boss exposure 4.0, file=336734 1:29:37 sp1.motor.move= 3118, 3117, 3117 01:29:37 Hartmann collimate output sp1: offset: r= 2638 (Out of focus); b= -11.3 (Out of focus) sp1: pred. move: spAverageMove= 3118 sp1: pred. resid: r= -480, b= -1.5, txt= OK, spTemp = 2.8 Initial teration: fvc_rms=71.1 fvc_rms=15.17 fvc_deltarms=55.93 01:31:22 APO.dgatlin_boson.jaeger 136 jaeger i fvc_rms=9.68 01:31:22 APO.dgatlin_boson.jaeger 136 jaeger i fvc_deltarms=5.49 fvc_rms=6.45 fvc_deltarms=3.23 01:32:41 FVC Loop - RMS: 6.40 Delta RMS: 0.05 Image #: 0005 01:33:18 boss exposure 4.0, file=336735 01:34:53 boss exposure 25.0, file=336736 Hal reported an error that "cherno -acquire full timed out" but 20 seconds later the command completed normally. 01:37:22 APO.dgatlin_boson.cherno 147 cherno : 01:37:22 CmdDone: APO.dgatlin_boson.cherno 147 cherno acquire -t 15.0 --full 01:37:59 CmdStarted: APO.dgatlin 70 cherno acquire -ct 15 Best current focus= 550 Seeing ~2" 01:55:59 apogee.expose=apRaw-40450010, 46, A, Object 01:56:06 boss exposure 900.0, file=336737 02:04:12 apogee.expose=apRaw-40450011, 46, B, Object 02:12:22 apogee.expose=apRaw-40450012, 46, B, Object 02:12:29 boss exposure 900.0, file=336738 02:20:39 apogee.expose=apRaw-40450013, 46, A, Object 02:28:49 apogee.expose=apRaw-40450014, 46, A, Object 02:28:53 boss exposure 900.0, file=336739 02:37:01 apogee.expose=apRaw-40450015, 46, B, Object 02:45:10 apogee.expose=apRaw-40450016, 43, B, Object 02:45:16 boss exposure 900.0, file=336740 02:52:56 apogee.expose=apRaw-40450017, 43, A, Object sdR-b1-00336736.fit.gz: WARNING: Reject flat: Flat-field lamps not turned on! FF '0 1 1 1' one lamp is turned off 03:04:27 boss exposure 25.0, file=336741 03:06:53 boss exposure 4.0, file=336742 03:08:33 apogee.expose=apRaw-40450018, 5, A, DomeFlat 03:13:01 Config Loaded: 35926-2924 No trouble unwinding or reconfiguring. 03:19:30 boss exposure 4.0, file=336743 03:20:10 boss exposure 4.0, file=336744 03:20:56 sp1.motor.move= -296, -297, -296 03:20:56 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -0.2 (In Focus) sp1: pred. move: spAverageMove= -297 sp1: pred. resid: r= -361, b= -1.1, txt= OK, spTemp = -0.4 RMS full fit 53.460 um fvc_rms=7.94 fvc_deltarms=45.52 03:22:41 FVC Loop - RMS:10.08 Delta RMS: -2.14 Image #: 0008 03:23:17 boss exposure 4.0, file=336745 03:24:53 boss exposure 25.0, file=336746 03:30:53 boss exposure 25.0, file=336747 The SOS pipeline rejected both flats. Last flat lamp didn't turn on until image started reading out (at least according to the system status window in STUI). Manually took the second flat. The flat field lamps were commanded on, but the cartoon continued to show the lamp turning on and off un-commanded. Best focus ~100 Seeing ~3.5" guide_rms=203,0.116,0.05,0.126 Haven't seen it improve less than 0.111 03:52:03 boss exposure 300.0, file=336748 03:58:38 Config Loaded: 35926-2925 fvc_rms=54.15 04:00:23 boss exposure 300.0, file=336749 04:15:06 Config Loaded: 35926-2926 fvc_rms=52.5 04:16:53 boss exposure 300.0, file=336750 04:29:50 encl25m : closed; High wind gusts and dewpoint dropping from 3.4 to 2.6. Wind has been out of the ENE and seeing has been worsening throughout the night. 6:15Z- rain sensor went off due to blowing snow, possibly new precip. Jose restarted jaeger, tested FPS alerts, and power-cycled a few things. 08:08:43 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-01-28 08:08:36; last seen 126 sec ago",disabled,noack,-1 Critical alert triggered for FPS heartbeat. jaeger is now checking and outputting alert keywords, but those are not yet connected to alertsActor. Light dusting of snow on the grass. Didn't stick to the sidewalk. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0050 0.0040 Axial B: 0.0020 0.0030 Axial C: -0.1140 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 2.50 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.8850 -9.6570 Axial B: -9.9020 -9.6710 Axial C: -9.9480 -9.6970 Trans D: 0.0000 0.1140 GALILS Commanded: 4894350 4893250 4913650 0 -3400 Actual: 4858764 4858695 4859047 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.7 -0.11 -0.0 -0.0 41.72 PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0030 0.0040 Axial B: 0.0000 0.0030 Axial C: -0.1130 -0.1150 Trans D: -0.0340 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5345 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: -2.78 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6600 -9.6570 Axial B: -9.6730 -9.6710 Axial C: -9.6970 -9.6970 Trans D: 0.1050 0.1140 GALILS Commanded: 4890900 4890750 4910100 -50 -3450 Actual: 4859010 4859488 4858997 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.92 0.09 0.05 0.03 41.58 ---- Telescope Status ---- 23:40Z Started ventilation 01:02Z encl25m : open 04:29Z encl25m : closed No interlocks bypassed Status at:11:52Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 685 687 705 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 141 lbs, 21 psi ---- Software Used ---- -- STUI software: STUI : 1.7.16 alerts : 2.2.0 apo : 3.0.1 boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED fliswarm : 0.2.1-alpha.0 jaeger : 0.13.2-alpha.0 kaiju : 1.2.2-alpha.1 coordio : 1.2.1-alpha.2 fps_calibrati : APO.2021.12.25+960cb05 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.11 -- done -- --------------Other Versions-------------- sdss-obstools : 3.7.8 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V34 Fiducials : v1_148 idlspec2d : v6_0_4 Kronos : 0.1.0 roboscheduler : FAILED /dev/sdb1 2.0T 158G 1.9T 8% /home 10.25.1.230:/data 21T 15T 5.4T 74% /data ---- APOGEE Data Summary ---- MJD UTC Design-Config Exposure Type Dith Reads Arch -------------------------------------------------------------------------------- 59606 11:46:36 35958-2912 40440042 Dark A 60 a-b-c 59606 11:57:14 35958-2912 40440043 Dark A 60 a-b-c 59606 12:07:53 35958-2912 40440044 Dark A 60 a-b-c 59606 12:18:44 35958-2912 40440045 Quartzflat A 10 a-b-c 59606 12:20:32 35958-2912 40440046 Quartzflat A 10 a-b-c 59606 12:22:20 35958-2912 40440047 Quartzflat A 10 a-b-c 59606 12:24:15 35958-2912 40440048 ThAr Arc B 12 a-b-c 59606 12:26:28 35958-2912 40440049 UNe Arc B 40 a-b-c 59606 12:33:45 35958-2912 40440050 ThAr Arc A 12 a-b-c 59606 12:35:58 35958-2912 40440051 UNe Arc A 40 a-b-c 59606 12:43:14 35958-2912 40440052 Dark A 30 a-b-c 59606 12:48:34 35958-2912 40440053 Internalflat A 30 a-b-c 59606 12:53:54 35958-2912 40440054 Internalflat A 30 a-b-c 59606 12:59:14 35958-2912 40440055 Internalflat A 30 a-b-c 59606 13:04:45 35958-2912 40440056 Dark A 30 a-b-c 59606 13:10:18 35958-2912 40440057 FPI Lamp A 30 a-b-c ThAr Offsets: ['-0.53', '0.53', '-0.52'] UNe Offsets: ['-0.51', '0.51', '-0.50'] Object Offsets: Max: 0.57, Min: 0.00, Mean: 0.41 APOGEE data list, mjd=59607, all raw_data: /data/apogee/utr_cdr/59607/apRaw-*.fits archive: /data/apogee/archive/59607/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59607/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59607 1 22:33 40450001 Dark 3 A 13.0 ---------- a-b-c - - 59607 2 22:47 40450002 Dark 60 A 13.0 ---------- a-b-c - - 59607 3 22:49 40450003 QuarFlat 10 A 13.0 ---------- a-b-c - 99 59607 4 22:51 40450004 Arc-Thar 12 B 13.5 ---------- a-b-c 0.49 - 59607 5 22:58 40450005 Arc-Une 40 B 13.5 ---------- a-b-c - - 59607 6 23:01 40450006 Arc-Thar 12 A 13.0 ---------- a-b-c -0.12 - 59607 7 23:08 40450007 Arc-Une 40 A 13.0 ---------- a-b-c - - 59607 8 23:19 40450008 Dark 60 A 13.0 ---------- a-b-c - - 59607 9 23:24 40450009 Arc---- 30 A 13.0 ---------- a-b-c - - FPI 59607 10 02:03 40450010 Object 46 A 13.0 16716-2923 a-b-c -0.12 - 59607 11 02:11 40450011 Object 46 B 13.5 16716-2923 a-b-c 0.39 - 59607 12 02:20 40450012 Object 46 B 13.5 16716-2923 a-b-c 0.39 - 59607 13 02:28 40450013 Object 46 A 13.0 16716-2923 a-b-c -0.12 - 59607 14 02:36 40450014 Object 46 A 13.0 16716-2923 a-b-c -0.11 - 59607 15 02:44 40450015 Object 46 B 13.5 16716-2923 a-b-c 0.39 - 59607 16 02:52 40450016 Object 43 B 13.5 16716-2923 a-b-c 0.39 - 59607 17 03:00 40450017 Object 43 A 13.0 16716-2923 a-b-c -0.11 - 59607 18 03:09 40450018 DomeFlat 5 A 13.0 16716-2923 a-b-c - 125 cannot read file : /data/apogee/utr_cdr/59607/apRaw-40450019.fits ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59607 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:44 r1-00336727 ----------- ---- bia Clo 0.0 -8540 -8490 -8560 2 22:44 b1-00336727 ----------- ---- bia Clo 0.0 -8540 -8490 -8560 3 22:45 r1-00336728 ----------- ---- dar Clo 20.0 -8540 -8490 -8560 4 22:45 b1-00336728 ----------- ---- dar Clo 20.0 -8540 -8490 -8560 5 22:53 b1-00336729 35958-20927 2912 arc Lef 4.1 -8540 -8490 -8560 HART 6 22:53 r1-00336729 35958-20927 2912 arc Lef 4.1 -8540 -8490 -8560 HART 7 22:53 r1-00336730 35958-20927 2912 arc Rig 4.1 -8540 -8490 -8560 HART 8 22:53 b1-00336730 35958-20927 2912 arc Rig 4.1 -8540 -8490 -8560 HART 9 22:56 r1-00336731 35958-20927 2912 arc Out 4.1 -5907 -5857 -5928 10 22:56 b1-00336731 35958-20927 2912 arc Out 4.1 -5907 -5857 -5928 11 22:59 r1-00336732 35958-20927 2912 fla Out 25.1 -5907 -5857 -5928 12 22:59 b1-00336732 35958-20927 2912 fla Out 25.1 -5907 -5857 -5928 13 01:28 r1-00336733 16716-19942 2923 arc Lef 4.1 -5907 -5857 -5928 HART 14 01:28 b1-00336733 16716-19942 2923 arc Lef 4.1 -5907 -5857 -5928 HART 15 01:28 r1-00336734 16716-19942 2923 arc Rig 4.1 -5907 -5857 -5928 HART 16 01:28 b1-00336734 16716-19942 2923 arc Rig 4.1 -5907 -5857 -5928 HART 17 01:33 r1-00336735 16716-19942 2923 arc Out 4.1 -2789 -2740 -2811 18 01:33 b1-00336735 16716-19942 2923 arc Out 4.1 -2789 -2740 -2811 19 01:34 r1-00336736 16716-19942 2923 fla Out 25.1 -2789 -2740 -2811 20 01:34 r1-00336736 16716-19942 2923 fla Out 25.1 -2789 -2740 -2811 21 01:34 b1-00336736 16716-19942 2923 fla Out 25.1 -2789 -2740 -2811 22 01:56 r1-00336737 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 23 01:56 r1-00336737 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 24 01:56 b1-00336737 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 25 02:12 r1-00336738 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 26 02:12 b1-00336738 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 27 02:28 r1-00336739 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 28 02:28 b1-00336739 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 29 02:45 b1-00336740 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 30 02:45 r1-00336740 16716-19942 2923 sci Out 900.1 -2789 -2740 -2811 31 03:04 b1-00336741 16716-19942 2923 fla Out 25.1 -2789 -2740 -2811 32 03:04 r1-00336741 16716-19942 2923 fla Out 25.1 -2789 -2740 -2811 33 03:06 b1-00336742 16716-19942 2923 arc Out 4.1 -2789 -2740 -2811 34 03:06 r1-00336742 16716-19942 2923 arc Out 4.1 -2789 -2740 -2811 35 03:19 r1-00336743 35926-20895 2924 arc Lef 4.1 -2789 -2740 -2811 HART 36 03:19 b1-00336743 35926-20895 2924 arc Lef 4.1 -2789 -2740 -2811 HART 37 03:20 r1-00336744 35926-20895 2924 arc Rig 4.1 -2789 -2740 -2811 HART 38 03:20 b1-00336744 35926-20895 2924 arc Rig 4.1 -2789 -2740 -2811 HART 39 03:23 b1-00336745 35926-20895 2924 arc Out 4.1 -3085 -3037 -3107 40 03:23 r1-00336745 35926-20895 2924 arc Out 4.1 -3085 -3037 -3107 41 03:24 r1-00336746 35926-20895 2924 fla Out 25.1 -3085 -3037 -3107 42 03:24 b1-00336746 35926-20895 2924 fla Out 25.1 -3085 -3037 -3107 43 03:30 r1-00336747 35926-20895 2924 fla Out 25.1 -3085 -3037 -3107 44 03:30 b1-00336747 35926-20895 2924 fla Out 25.1 -3085 -3037 -3107 45 03:30 r1-00336747 35926-20895 2924 fla Out 25.1 -3085 -3037 -3107 46 03:52 b1-00336748 35926-20895 2924 sci Out 300.1 -3085 -3037 -3107 47 03:52 r1-00336748 35926-20895 2924 sci Out 300.1 -3085 -3037 -3107 48 04:00 b1-00336749 35926-20895 2925 sci Out 300.1 -3085 -3037 -3107 49 04:00 r1-00336749 35926-20895 2925 sci Out 300.1 -3085 -3037 -3107 50 04:16 b1-00336750 35926-20895 2926 sci Out 300.1 -3085 -3037 -3107 51 04:16 r1-00336750 35926-20895 2926 sci Out 300.1 -3085 -3037 -3107