Subject: 2.5m obslog 20220612 (SJD 59743) From: dgatlin@apo.nmsu.edu Date: Mon, 13 Jun 2022 10:21:35 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday June 12, 2022 (SJD 59743) ---- Observing Team ---- Dan Oravetz Dylan Gatlin ---- Observing Plan ---- If clear, observe from the queue plus do one BOSS dither on design 35994 , 36002, or 36018, per instructions below with a different configuration rotator angle, note change below to /rota=100 instead of 190. Do the dither field after the telescope has settled. Remember to do a cherno offset command after completing the dither before returning to queue observing! ---- Observing Summary ---- Stayed closed for radar first, but later smoke. We completed the remaining BOSS flat fields that were scheduled yesterday but not observed - 111287, 111289, 111291, 111293. Observed: --------- Time field-dsgn-conf Design type BOSS/APOGEE exp complete? Comment =========================================================================== - During yesterday's morning cals, we got the following alert, which aborted the cals. We fixed it in the afternoon with a dither init Serious apogee.ditherIndexer at 2022-06-12 11:59:46 UT found Off - We had a netburner readout error during the flats, which we recovered as exposure 343671 ---- Engineering Issues ---- M2 Trans D MIG (Y) appears to be in program mode or the battery is dead (displays red-colored 0.0000 in the MCPTPM GUI) and needs to be reset. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ 18:10Z - Came in to check on things and found this serious alert: Serious apogee.ditherIndexer at 2022-06-12 11:59:46 UT found Off The morning cals did not make it beyond the quartz flats as a result. The APOGEE instrument window in STUI indicated an 'Off' status for Dither and Shutter and 'Limits 1 1 1 1 1 1' for Collimator. An 'apogee dither init' and an 'apogee coll init' were required to restore the status of Dither, Shutter, and Collimator to 'OK', but the Collimator Piston, Pitch, and Yaw were all zeroes. The recovery procedure indicates that "if the collimator positions come back up as all 0's, a backup save file has to be restored, and an 'init' command reissued." I copied the ~observer/coll_backup/collpos.per file to ~observer/ics_logs and reissued 'apogee dither init' as instructed, but the collimator orientation is still all zeroes. I checked the logs and the apogee.collOrient keyword (which supplies piston, pitch, and yaw) has been all zeroes since around the end of January (please see below). The mount position also changed to zero at that time. Emailed JohnW and JonH seeking clarification as to whether we need to update the recovery procedure as it pertains to all zeroes for collOrient. Afternoon disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 177G 1.9T 9% /home 10.25.1.230:/data 21T 14T 7.0T 66% /data Powered things on. Positioners are home/folded: 19:25:43 CmdStarted: APO.doravetz 79 jaeger unwind --status 19:25:44 APO.doravetz 79 jaeger i folded=T 19:25:44 APO.doravetz 79 jaeger : text="All positioners are folded." 19:25:44 CmdDone: APO.doravetz 79 jaeger unwind --status Loaded a manual folded configuration: 19:26:09 CmdStarted: APO.doravetz 80 jaeger configuration load --from-positions 19:26:14 APO.doravetz 80 jaeger d text="Focal plane scale: 0.9998." 19:26:14 APO.doravetz 80 jaeger d configuration_loaded=5887,-999,-999,-999.0, -999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files /0058XX/confSummary-5887.par,F 19:26:18 APO.doravetz 80 jaeger i configuration_snapshot=/data/logs/jaeger /configuration_snapshots/59742/configuration_snapshot_59742_5887.pdf 19:26:18 APO.doravetz 80 jaeger d design_preloaded=-999 19:26:18 APO.doravetz 80 jaeger : text="Configuration 5887 loaded." 19:26:18 CmdDone: APO.doravetz 80 jaeger configuration load --from-positions Took a '--no-apply' FVC image: 19:27:00 APO.doravetz 81 jaeger i text="FVC iteration 1" 19:27:44 APO.doravetz 81 jaeger d fvc_filename=/data/fcam/59743/fimg-fvc1n-0001.fits 19:27:51 APO.doravetz 81 jaeger i fvc_centroid_method=zbplus 19:27:52 APO.doravetz 81 jaeger i fvc_rms=45.870000000000005 19:27:52 APO.doravetz 81 jaeger i fvc_perc_90=69.3 19:27:52 APO.doravetz 81 jaeger i fvc_percent_reached=0.8 Powered off the FPS for now. - Started ventilation. ============================================================================ NIGHT ============================================================================ We decided to test the apogee dome flat behavior at different rotator angles. We rotated in increments of 30 from 0 to 360 and we never saw MTP 6 to be faint. We also tried holding the gang connector to see if we could get it into a bad spot. We didn't notice anything in those trials either. We decided to continue the BOSS dome flat tests since we were unable to open. # Folded 03:23:17 boss exposure 25.0, file=343555 03:25:05 boss exposure 25.0, file=343556 03:26:54 boss exposure 25.0, file=343557 03:28:42 boss exposure 25.0, file=343558 03:30:33 boss exposure 25.0, file=343559 03:32:21 boss exposure 25.0, file=343560 03:34:10 boss exposure 25.0, file=343561 03:35:58 boss exposure 25.0, file=343562 03:37:47 boss exposure 25.0, file=343563 03:39:35 boss exposure 25.0, file=343564 # Configured 03:52:32 Config Loaded: 21500-111287-5888 03:56:12 boss exposure 25.0, file=343565 03:58:00 boss exposure 25.0, file=343566 03:59:49 boss exposure 25.0, file=343567 04:02:00 Config Loaded: 21500-111287-5889 04:03:03 boss exposure 25.0, file=343568 04:04:51 boss exposure 25.0, file=343569 04:06:40 boss exposure 25.0, file=343570 04:08:52 Config Loaded: 21500-111287-5890 04:09:54 boss exposure 25.0, file=343571 04:11:43 boss exposure 25.0, file=343572 04:13:32 boss exposure 25.0, file=343573 04:15:44 Config Loaded: 21500-111287-5891 04:16:48 boss exposure 25.0, file=343574 04:18:36 boss exposure 25.0, file=343575 04:20:25 boss exposure 25.0, file=343576 04:22:36 Config Loaded: 21500-111287-5892 04:23:39 boss exposure 25.0, file=343577 04:25:27 boss exposure 25.0, file=343578 04:27:17 boss exposure 25.0, file=343579 04:29:31 Config Loaded: 21500-111287-5893 04:30:35 boss exposure 25.0, file=343580 04:32:23 boss exposure 25.0, file=343581 04:34:12 boss exposure 25.0, file=343582 04:36:24 Config Loaded: 21500-111287-5894 04:37:27 boss exposure 25.0, file=343583 04:39:17 boss exposure 25.0, file=343584 04:41:05 boss exposure 25.0, file=343585 04:43:17 Config Loaded: 21500-111287-5895 04:44:20 boss exposure 25.0, file=343586 04:46:08 boss exposure 25.0, file=343587 04:47:56 boss exposure 25.0, file=343588 04:50:08 Config Loaded: 21500-111287-5896 04:51:12 boss exposure 25.0, file=343589 04:53:00 boss exposure 25.0, file=343590 04:54:49 boss exposure 25.0, file=343591 04:57:02 Config Loaded: 21500-111287-5897 04:58:04 boss exposure 25.0, file=343592 04:59:53 boss exposure 25.0, file=343593 05:01:41 boss exposure 25.0, file=343594 05:10:56 Config Loaded: 21502-111289-5898 05:13:47 boss exposure 25.0, file=343595 05:15:36 boss exposure 25.0, file=343596 05:17:24 boss exposure 25.0, file=343597 05:19:36 Config Loaded: 21502-111289-5899 05:20:38 boss exposure 25.0, file=343598 05:22:27 boss exposure 25.0, file=343599 05:24:18 boss exposure 25.0, file=343600 05:26:30 Config Loaded: 21502-111289-5900 05:27:32 boss exposure 25.0, file=343601 05:29:20 boss exposure 25.0, file=343602 05:31:09 boss exposure 25.0, file=343603 05:33:21 Config Loaded: 21502-111289-5901 05:34:23 boss exposure 25.0, file=343604 05:36:12 boss exposure 25.0, file=343605 05:38:00 boss exposure 25.0, file=343606 05:40:12 Config Loaded: 21502-111289-5902 05:41:18 boss exposure 25.0, file=343607 05:43:06 boss exposure 25.0, file=343608 05:44:54 boss exposure 25.0, file=343609 05:47:06 Config Loaded: 21502-111289-5903 05:48:08 boss exposure 25.0, file=343610 05:49:57 boss exposure 25.0, file=343611 05:51:45 boss exposure 25.0, file=343612 05:53:57 Config Loaded: 21502-111289-5904 05:55:00 boss exposure 25.0, file=343613 05:56:48 boss exposure 25.0, file=343614 05:58:37 boss exposure 25.0, file=343615 06:00:48 Config Loaded: 21502-111289-5905 06:01:50 boss exposure 25.0, file=343616 06:03:39 boss exposure 25.0, file=343617 06:05:27 boss exposure 25.0, file=343618 06:07:39 Config Loaded: 21502-111289-5906 06:08:41 boss exposure 25.0, file=343619 06:10:32 boss exposure 25.0, file=343620 06:12:24 boss exposure 25.0, file=343621 06:14:36 Config Loaded: 21502-111289-5907 06:15:38 boss exposure 25.0, file=343622 06:17:27 boss exposure 25.0, file=343623 06:19:17 boss exposure 25.0, file=343624 06:30:20 Config Loaded: 21504-111291-5908 06:32:32 boss exposure 25.0, file=343625 06:34:21 boss exposure 25.0, file=343626 06:36:09 boss exposure 25.0, file=343627 06:38:23 Config Loaded: 21504-111291-5909 06:39:27 boss exposure 25.0, file=343628 06:41:18 boss exposure 25.0, file=343629 06:43:06 boss exposure 25.0, file=343630 06:45:20 Config Loaded: 21504-111291-5910 06:46:24 boss exposure 25.0, file=343631 06:48:12 boss exposure 25.0, file=343632 06:50:00 boss exposure 25.0, file=343633 06:52:14 Config Loaded: 21504-111291-5911 06:53:18 boss exposure 25.0, file=343634 06:55:06 boss exposure 25.0, file=343635 06:56:55 boss exposure 25.0, file=343636 06:59:08 Config Loaded: 21504-111291-5912 07:00:13 boss exposure 25.0, file=343637 07:02:01 boss exposure 25.0, file=343638 07:03:49 boss exposure 25.0, file=343639 07:06:03 Config Loaded: 21504-111291-5913 07:07:07 boss exposure 25.0, file=343640 07:08:55 boss exposure 25.0, file=343641 07:10:44 boss exposure 25.0, file=343642 07:12:58 Config Loaded: 21504-111291-5914 07:14:07 boss exposure 25.0, file=343643 07:15:55 boss exposure 25.0, file=343644 07:17:43 boss exposure 25.0, file=343645 07:19:58 Config Loaded: 21504-111291-5915 07:21:02 boss exposure 25.0, file=343646 07:22:50 boss exposure 25.0, file=343647 07:24:39 boss exposure 25.0, file=343648 07:26:52 Config Loaded: 21504-111291-5916 07:27:57 boss exposure 25.0, file=343649 07:29:45 boss exposure 25.0, file=343650 07:31:33 boss exposure 25.0, file=343651 07:33:50 Config Loaded: 21504-111291-5917 07:34:54 boss exposure 25.0, file=343652 07:36:42 boss exposure 25.0, file=343653 07:38:33 boss exposure 25.0, file=343654 07:42:37 Config Loaded: 21506-111293-5918 07:44:10 boss exposure 25.0, file=343655 07:45:58 boss exposure 25.0, file=343656 07:47:47 boss exposure 25.0, file=343657 07:50:02 Config Loaded: 21506-111293-5919 07:51:07 boss exposure 25.0, file=343658 07:52:56 boss exposure 25.0, file=343659 07:54:44 boss exposure 25.0, file=343660 07:57:00 Config Loaded: 21506-111293-5920 07:58:05 boss exposure 25.0, file=343661 07:59:54 boss exposure 25.0, file=343662 08:01:42 boss exposure 25.0, file=343663 08:03:58 Config Loaded: 21506-111293-5921 08:05:03 boss exposure 25.0, file=343664 08:06:52 boss exposure 25.0, file=343665 08:08:40 boss exposure 25.0, file=343666 08:10:56 Config Loaded: 21506-111293-5922 08:12:01 boss exposure 25.0, file=343667 08:13:50 boss exposure 25.0, file=343668 08:15:38 boss exposure 25.0, file=343669 # We ran into a netburner issue on 343669, we were unable to readout. We ran a reloadControllers twice, and an exposure recover after each. Weirdly, it worked the second time, but not the first. 08:30:18 boss exposure 25.0, file=343672 08:32:33 Config Loaded: 21506-111293-5923 08:33:39 boss exposure 25.0, file=343673 08:35:27 boss exposure 25.0, file=343674 08:37:18 boss exposure 25.0, file=343675 08:39:33 Config Loaded: 21506-111293-5924 08:40:38 boss exposure 25.0, file=343676 08:42:27 boss exposure 25.0, file=343677 08:44:18 boss exposure 25.0, file=343678 08:46:33 Config Loaded: 21506-111293-5925 08:47:39 boss exposure 25.0, file=343679 08:49:27 boss exposure 25.0, file=343680 08:51:18 boss exposure 25.0, file=343681 08:53:33 Config Loaded: 21506-111293-5926 08:54:37 boss exposure 25.0, file=343682 08:56:26 boss exposure 25.0, file=343683 08:58:18 boss exposure 25.0, file=343684 09:00:33 Config Loaded: 21506-111293-5927 09:01:39 boss exposure 25.0, file=343685 09:03:27 boss exposure 25.0, file=343686 09:05:18 boss exposure 25.0, file=343687 # Then the final exposures back at folded 09:09:46 boss exposure 25.0, file=343688 09:11:34 boss exposure 25.0, file=343689 09:13:23 boss exposure 25.0, file=343690 09:15:12 boss exposure 25.0, file=343691 09:17:00 boss exposure 25.0, file=343692 09:18:48 boss exposure 25.0, file=343693 09:20:37 boss exposure 25.0, file=343694 09:22:26 boss exposure 25.0, file=343695 09:24:18 boss exposure 25.0, file=343696 09:26:12 boss exposure 25.0, file=343697 10:08Z Called it and started morning cals ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0170 0.0140 Axial B: 0.0100 0.0050 Axial C: -0.0150 -0.0200 Trans D: 0.0080 0.0010 Lateral E: 0.3315 0.3315 Lateral F: 0.5345 0.5345 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 25.28 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6520 -9.6550 Axial B: -0.0450 -0.0470 Axial C: -9.7050 -9.7010 Trans D: 0.0000 0.1030 (likely this is in program mode and needs reset) GALILS Commanded: 4869050 4863400 4885600 -50 -3450 Actual: 4859154 4859173 4859205 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.97 -0.01 0.0 0.0 42.88 ---- Telescope Status ---- Status at: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 682 684 702 702 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 lbs, psi ---- Software Used ---- STUI : 1.7.17b1 alerts : 2.2.1 boss : 6.0.1 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.3.0 jaeger : 0.16.0a0 kaiju : 1.2.4-alpha.0 coordio : 1.4.0-alpha.0 fps_calibrati : APO.2022.04.21+8fae566 cherno : 0.1.1-alpha.0 hal : 0.2.1a0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.13 boson : 0.0.4-alpha.41 (0.0.4-alpha.41) sdss-obstools : 3.8.2 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V34 Fiducials : v1_148 idlspec2d : v6_0_7 Kronos : 1.3.1a roboscheduler : 1.2.2 ---- APOGEE Data Summary ---- 59742 morning cals did not complete beyond quartz flats apparently due to a communication problem with the dither/collimator mechanisms and the network. See Log section for a description of the recovery. MJD UTC Field-Design-Config Exposure Type Dith Reads Arch -------------------------------------------------------------------------------- 59742 18:31:18 100461- 37779-5884 41810001 Dark A 3 a-b-c 59742 18:32:51 100461- 37779-5884 41810002 Dark B 3 a-b-c 59742 19:11:59 100461- 37779-5884 41810003 Dark A 3 a-b-c 59742 19:14:25 100461- 37779-5884 41810004 Dark A 60 a-b-c 59742 19:25:18 100461- 37779-5884 41810005 Quartzflat A 10 a-b-c 59742 19:27:12 -999- -999-5887 41810006 ThAr Arc B 12 a-b-c 59742 19:29:25 -999- -999-5887 41810007 UNe Arc B 40 a-b-c 59742 19:36:42 -999- -999-5887 41810008 ThAr Arc A 12 a-b-c 59742 19:38:55 -999- -999-5887 41810009 UNe Arc A 40 a-b-c 59742 19:46:21 -999- -999-5887 41810010 Dark A 60 a-b-c 59742 19:57:13 -999- -999-5887 41810011 FPI Lamp A 30 a-b-c 59743 02:34:39 -999- -999-5887 41810012 Domeflat A 5 a-b-c 59743 02:41:07 -999- -999-5887 41810013 Domeflat A 5 a-b-c 59743 02:43:49 -999- -999-5887 41810014 Domeflat A 5 a-b-c 59743 02:45:31 -999- -999-5887 41810015 Domeflat A 5 a-b-c 59743 02:47:13 -999- -999-5887 41810016 Domeflat A 5 a-b-c 59743 02:48:55 -999- -999-5887 41810017 Domeflat A 5 a-b-c 59743 02:50:38 -999- -999-5887 41810018 Domeflat A 5 a-b-c 59743 02:52:20 -999- -999-5887 41810019 Domeflat A 5 a-b-c 59743 02:54:03 -999- -999-5887 41810020 Domeflat A 5 a-b-c 59743 02:55:45 -999- -999-5887 41810021 Domeflat A 5 a-b-c 59743 02:57:28 -999- -999-5887 41810022 Domeflat A 5 a-b-c 59743 02:59:10 -999- -999-5887 41810023 Domeflat A 5 a-b-c 59743 03:11:40 -999- -999-5887 41810024 Domeflat A 5 a-b-c 59743 03:13:33 -999- -999-5887 41810025 Domeflat A 5 a-b-c 59743 03:16:16 -999- -999-5887 41810026 Domeflat A 5 a-b-c ThAr Offsets: ['-0.50'] UNe Offsets: ['-0.52'] ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 19:17:01 boss exposure 4.0, file=343551 19:17:43 boss exposure 4.0, file=343552 19:18:25 sp1.motor.move= 17, 20, 18 19:18:25 Hartmann collimate output sp1: offset: r= -989 (In Focus); b= -3.2 (In Focus) sp1: pred. move: spAverageMove= 19 sp1: pred. resid: r= -1008, b= -3.2, txt= OK, spTemp = 12.3 19:18:21 APO.doravetz 56 hartmann i r1MeanOffset=-0.15,"In Focus" 19:18:21 APO.doravetz 56 hartmann i b1MeanOffset=0.16,"In Focus" Time Field-Design Temp R off B off Move Resid ================================================================================ 19:16:09 nan-nan 12.3 -1319 -2.2 -306 -3.2 19:18:21 nan-nan 12.3 -989 -3.2 19 -3.2 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 59743 Fill night time in Hours below *=*=*= Total bright time : 7:36 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 7:36 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 0 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 0 *=*=*= Dark time lost to technical issues : 0