From: "Donor, John" Subject: Re: [sloan-25mlog 7467] Re: 2.5m obslog 20241017 (SJD 60601) Date: Mon, 20 Jan 2025 18:55:40 +0000 --_000_IA2PR16MB63945DC533F5A6D1145B3C81FEE72IA2PR16MB6394namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable No idea where this email from October came from? Stuck in some mail server = backlog I guess. ________________________________ From: sloan-25mlog on behalf of Donor, John= Sent: Friday, October 18, 2024 10:04 AM To: Daniel Oravetz Cc: obs-report-25m@apo.nmsu.edu Subject: [sloan-25mlog 7467] Re: 2.5m obslog 20241017 (SJD 60601) [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless you = recognize the sender and know the content is safe. Hi Dan, I don't think this is permission issues actually. I think the server is jus= t rejecting connections. This will happen if it doesn't finish booting all = the way, or for many other reasons. I'm waiting to hear from Shane to see w= hat's up exactly. And it seems gateway specific? I can login to hub, etc, fine with VPN. Thanks, John On Oct 18, 2024 10:01 AM, Daniel Oravetz wrote: [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless you = recognize the sender and know the content is safe. Hi John, Shane, all, We've done some "cold start" tests in the past and I don't see any special permissions problems documented in restart instructions. Are these permission issues that hampered getting subsystems restarted a result of the failure mode that caused the crash, or are these permission problems related to a new configuration setup ("new" as in since whenever the last "cold start" tests occurred) that we should add to procedural documentation? Dan > Status update: things are no longer critical IMO, but I do still urgently > need Shane's help. > > I got in via VPN, but our gateway is still refusing connections. > > Tron is up and connected to APOGEE and BOSS. No critical alerts. > > I cannot connect to sdss5-fps, so I can't talk to jaeger. Hopefully the > FPS is in a safe state, I think Audrey could do that manually? > > apogeeql won't start because the DB is in a weird state. There is a cold > start issue with postgres on that machine that I haven't been able to > track down :/ (weird race condition IIRC) anyway, I can't fix postgres > without sudo and sudo is through ldap and ldap is down? Except that it le= t > me VPN in... > > I remember Shane said there was an issue with our ldap VM booting when we > did our last cold start test, but I thought he fixed it. Maybe something > similar? It is odd that I can login but not sudo when it's supposedly all > through ldap. Anyway, need Shane for this. > > I'm supposed to be on campus today so I'm going to get ready and head tha= t > way. I'll be available for more troubleshooting when folks are awake > probably. > > -John > ________________________________ > From: sloan-25mlog on behalf of Donor, > John > Sent: Friday, October 18, 2024 5:19 AM > To: asimmons@apo.nmsu.edu > Cc: obs-report-25m@apo.nmsu.edu > Subject: [sloan-25mlog 7464] Re: 2.5m obslog 20241017 (SJD 60601) > > > [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless yo= u > recognize the sender and know the content is safe. > > I am also getting "permission" issues, but based on the immediate refusal > it's more like ssh is refusing to let me connect. Assuming Shane restored > a current image, my first instinct is that gateway isn't booted all the > way? > > Anyway there's not much I can do if I can't login to gateway. I'll try to > get my VPN up; I forgot about that when my laptop was reimaged a few week= s > ago. But if Audrey couldn't login from site that won't help. > > On Oct 18, 2024 4:59 AM, asimmons--- via sloan-25mlog > wrote: > [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless yo= u > recognize the sender and know the content is safe. > > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Thursday October 17, 2024 > > (SJD 60601) > > ---- Observing Team ---- > > Audrey Oravetz > > Shane Thomas (support) > Kaike Pan (support) > > > ---- Observing Plan ---- > > > Thu 17 Oct 2024 60601 0.99 01:20 01:34 12:02 20:19 06:48 > --- bright (10.9 h, w/ 0.4 h for 12 deg design) > > ---- Observing Summary ---- > > The night was again lost to rain/storms/humidity. > > The hub crashed around midnight unexpectedly. With Shane's help, we were > able to get 10.25.190 back online with a manual power cycle. However, I > ran into various permission issues trying to actually log onto the > machine. > > At this moment, various systems are not being monitored: alerts, apo, > apogeeql, boss, jaeger, tcc, kronos, grafana. I need help reconnecting > these systems in the morning ASAP. > > > > ---- Observing Log ---- > > Afternoon: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 22:49:29 boss exposure 0.0, file=3D374700 > 22:51:02 boss exposure 20.0, file=3D374701 > 22:53:36 boss exposure 4.0, file=3D374702 > 22:54:16 boss exposure 4.0, file=3D374703 > 22:54:57 sp1.motor.move=3D 27, 27, 27 > 22:54:57 Hartmann collimate output > sp1: offset: r=3D -329 (In Focus); b=3D -1.2 (In Focus) > sp1: pred. move: spAverageMove=3D 28 > sp1: pred. resid: r=3D -357, b=3D -1.1, txt=3D OK, spTemp =3D 7.3 > 22:55:30 boss exposure 4.0, file=3D374704 > 22:58:10 boss exposure 25.0, file=3D374705 > > 22:58:18 Config Loaded: -999-16568 > > 22:59:00 jaeger.jaeger 1489 fliswarm i > filename_bundle=3D/data/fcam/60601/fimg-fvc1n-0001.fits > 22:59:00 jaeger.jaeger 1489 fliswarm : > 22:59:00 CmdDone: jaeger.jaeger 1489 fliswarm talk -c fvc expose 5.0 > > 23:00:19 APO.aoravetz2 44 fliswarm i > filename_bundle=3D/data/fcam/60601/fimg-fvc1n-0002.fits > 23:00:19 APO.aoravetz2 44 fliswarm : > 23:00:19 CmdDone: APO.aoravetz2 44 fliswarm talk -n fvc expose 50 > > fliswarm talk -c gfa expose --bias --count 10 > fliswarm talk -c gfa expose --dark --count 5 5 > > jaeger configuration random --danger > jaeger configuration reverse > > (observers39) [observer@sdss5-hub: ~]$ list_ap > APOGEE data list, mjd=3D60601, all > raw_data: /data/apogee/utr_cdr/60601/apRaw-*.fits > archive: /data/apogee/archive/60601/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/60601/apq-*.fits > -------------------------------------------------------------------------= ----- > mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ------------------------------------------------------------------------= ------ > 60601 1 22:59 50390001 Dark 60 B 13.5 ---------- a-b-c - > - > 60601 2 23:01 50390002 QuarFlat 10 B 13.5 ---------- a-b-c - > - > 60601 3 23:03 50390003 Arc-Thar 12 B 13.5 ---------- a-b-c 1.95 > - > 60601 4 23:11 50390004 Arc-Une 40 B 13.5 ---------- a-b-c - > - > 60601 5 23:13 50390005 Arc-Thar 12 A 13.0 ---------- a-b-c 1.40 > - > 60601 6 23:20 50390006 Arc-Une 40 A 13.0 ---------- a-b-c - > - > 60601 7 23:31 50390007 Dark 60 A 13.0 ---------- a-b-c - > - > 60601 8 23:37 50390008 Arc---- 30 A 13.0 ---------- a-b-c - > 73 FPI A > 60601 9 23:42 50390009 Arc---- 30 B 13.5 ---------- a-b-c - > 73 FPI B > -------------------------------------------------------------------------= ----- > > Did not ventilate due to clouds/radar. > > Night: > =3D=3D=3D=3D=3D=3D > 06:39Z - > > client_loop: ssh_packet_write_poll: Connection to 10.25.1.139 port 22: > Host is down > > (observers39) aoravetz@sdss5-obs2:~ > % ssh -X -Y -A observer@sdss5-hub > ssh: connect to host sdss5-hub port 22: Host is down > > --Test Actors, 06:42:52 > alerts ping - **FAILED** > apo ping - **FAILED** > apogee ping - ok > apogeecal ping - ok > apogeeql ping - **FAILED** > apogeefpi ping - ok > boss ping - ok > fliswarm ping - ok > jaeger ping - **FAILED** > cherno ping - ok > hal ping - ok > hub ping - ok > keys getFor=3Dhub version - ok > mcp ping - ok > msg (testing actors) - ok > perms status - ok > hartmann ping - ok > tcc show time - **FAILED** > > GRAFANA, KRONOS: > > Service Unavailable > > The server is temporarily unable to service your request due to > maintenance downtime or capacity problems. Please try again later. > > Contacted Jose, Kaike, and Shane. > > Shane said that the hub did crash, but the backup system failed to take > over. Shane instructed me to physically unplug the KVM 10 server in the > machine room (10.25.1.90) and plug it back in a few minutes later. > > Shane said that he was able to get the machine back up and running, but I > was never able to verify and log into hub due to some permissions issue. = I > tried on both my laptop and obs2, different wifi networks, various > passwords... > > audrey@ARCs-MacBook-Pro-5 ~ % ssh -X -Y -A observer@sdss5-hub.apo.nmsu.ed= u > observer@sdss5-hub.apo.nmsu.edu's password: > Permission denied, please try again. > > (observers39) aoravetz@sdss5-obs2:~ > % ssh observer@sdss5-hub > observer@sdss5-hub's password: > Permission denied, please try again. > > audrey@ARCs-MacBook-Pro-5 ~ % ssh -X -Y -A observer@sdss5-hub > kex_exchange_identification: Connection closed by remote host > Connection closed by UNKNOWN port 65535 > > audrey@ARCs-MacBook-Pro-5 ~ % ssh -X -Y -A -L > 8080:sdss5-hub.apo.nmsu.edu:80 observer@sdss5-gateway.apo.nmsu.edu > observer@sdss5-gateway.apo.nmsu.edu: Permission denied > (publickey,gssapi-keyex,gssapi-with-mic). > > (observers39) aoravetz@sdss5-obs2:~ > % ssh observer@sdss5-gateway.apo.nmsu.edu > ssh: connect to host sdss5-gateway.apo.nmsu.edu port 22: Operation timed > out > > Unable to run morningcals. Unable to get systems back in and running. > > > ---- Mirror Numbers ---- > > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS TONIGHT NOMINAL > Axial A: 0.0010 0.0040 > Axial B: -0.0015 -0.0014 > Axial C: -0.0060 -0.0010 > Trans D: -0.0180 -0.0090 > Lateral E: 0.3380 0.3380 > Lateral F: 0.5450 0.5435 > > GALILS > Commanded: 9800 -300 6750 19150 -3550 -3250 > Actual: 9800 -300 6750 19150 -3550 -3250 > > SETMIR VALUES > PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 > PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 > > SECONDARY: > ---------- > > Focus: 0.0000 > Air temp: 0.00 > Altitude: 30.0 > > MIGS TONIGHT NOMINAL > Axial A: -0.0870 -0.0830 > Axial B: 0.2050 0.2110 > Axial C: -0.0460 -0.0450 > Trans D: -0.0140 -0.0220 > > GALILS > Commanded: 5900000 5835200 5964100 -1500 -7250 > Actual: 5883681 5811643 5906263 -1500 -7250 > > SETMIR VALUES > SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 > SecOrient: 4657.01 -24.51 -17.52 17.68 125.27 > > > > ---- Telescope Status ---- > > No interlocks bypassed > > Status at: > Telescope stowed at: 121.0, 30.0, 0.0 mount > Instrument mounted: FPS > Counterweights at: 650s > LN2 autofill systems: Connected and turned on > 180L LN2 dewar scales: SP1 135 lbs, 21 psi > > > > > ---- Software Used ---- > > -- STUI software: > STUI : 1.7.19 > alerts : 2.3.0 > boss : 6.0.2 > -boss.specMec : 3.0.4 > apo : 4.1.0 > fliswarm : 0.5.1a0 > jaeger : 1.7.8a0 > kaiju : 1.4.0 > coordio : 1.11.3a0 > fps_calibrati : 2024.04.01+fc6df86 > cherno : 1.0.1a0 > astrometrynet : 0.93 > hal : 1.3.5 > hartmann : 2.0.2 > hub : 5.0.1a0 > mcp : 2.1-1-ge461 > -mcp.plc : 35 > -mcp.azFiduci : v1_149 > -mcp.altFiduc : v1_149 > -mcp.rotFiduc : v1_149 > tcc : 1.4.0 > apogee : not availble > apogeecal : not availble > apogeeql : v1_7_0dev > STUIAdditions : FAILED > Boson : 0.4.9 > -- done -- > > --------------Other Versions-------------- > sdss-obstools : 3.9.3 > TPM : 1.2-31-gd7eb > MCP : 2.1-1-ge461 > PLC : V35 > Fiducials : v1_149 > idlspec2d : v6_2_0 > Kronos : 2.0.0 > roboscheduler : 1.4.0 > > /dev/sdb1 2.0T 448G 1.6T 22% /home > /dev/sdd1 22T 17T 4.0T 81% /data > > > ---- APOGEE Data Summary ---- > > (observers39) [observer@sdss5-hub: ~]$ list_ap -m 60600 --morning > APOGEE data list, mjd=3D60600, all > raw_data: /data/apogee/utr_cdr/60600/apRaw-*.fits > archive: /data/apogee/archive/60600/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/60600/apq-*.fits > -------------------------------------------------------------------------= ----- > mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ------------------------------------------------------------------------= ------ > 60600 10 11:11 50380010 Dark 60 B 13.5 ---------- a-b-c - > - > 60600 11 11:22 50380011 Dark 60 B 13.5 ---------- a-b-c - > - > 60600 12 11:32 50380012 Dark 60 B 13.5 ---------- a-b-c - > - > 60600 13 11:34 50380013 QuarFlat 10 B 13.5 ---------- a-b-c - > - > 60600 14 11:36 50380014 QuarFlat 10 B 13.5 ---------- a-b-c - > - > 60600 15 11:38 50380015 QuarFlat 10 B 13.5 ---------- a-b-c - > - > 60600 16 11:40 50380016 Arc-Thar 12 B 13.5 ---------- a-b-c 1.98 > - > 60600 17 11:47 50380017 Arc-Une 40 B 13.5 ---------- a-b-c - > - > 60600 18 11:49 50380018 Arc-Thar 12 A 13.0 ---------- a-b-c 1.40 > - > 60600 19 11:57 50380019 Arc-Une 40 A 13.0 ---------- a-b-c - > - > 60600 20 12:02 50380020 Dark 30 A 13.0 ---------- a-b-c - > - > 60600 21 12:07 50380021 InteFlat 30 A 13.0 ---------- a-b-c - > - > 60600 22 12:13 50380022 InteFlat 30 A 13.0 ---------- a-b-c - > - > 60600 23 12:18 50380023 InteFlat 30 A 13.0 ---------- a-b-c - > - > 60600 24 12:24 50380024 Dark 30 A 13.0 ---------- a-b-c - > - > 60600 25 12:29 50380025 Arc---- 30 A 13.0 ---------- a-b-c - > 74 FPI A > 60600 26 12:35 50380026 Arc---- 30 B 13.5 ---------- a-b-c - > 73 FPI B > -------------------------------------------------------------------------= ----- > APOGEE data list, mjd=3D60601, all > raw_data: /data/apogee/utr_cdr/60601/apRaw-*.fits > archive: /data/apogee/archive/60601/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/60601/apq-*.fits > -------------------------------------------------------------------------= ----- > mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ------------------------------------------------------------------------= ------ > 60601 1 22:59 50390001 Dark 60 B 13.5 ---------- a-b-c - > - > 60601 2 23:01 50390002 QuarFlat 10 B 13.5 ---------- a-b-c - > - > 60601 3 23:03 50390003 Arc-Thar 12 B 13.5 ---------- a-b-c 1.95 > - > 60601 4 23:11 50390004 Arc-Une 40 B 13.5 ---------- a-b-c - > - > 60601 5 23:13 50390005 Arc-Thar 12 A 13.0 ---------- a-b-c 1.40 > - > 60601 6 23:20 50390006 Arc-Une 40 A 13.0 ---------- a-b-c - > - > 60601 7 23:31 50390007 Dark 60 A 13.0 ---------- a-b-c - > - > 60601 8 23:37 50390008 Arc---- 30 A 13.0 ---------- a-b-c - > 73 FPI A > 60601 9 23:42 50390009 Arc---- 30 B 13.5 ---------- a-b-c - > 73 FPI B > -------------------------------------------------------------------------= ----- > > > > ---- BOSS Data Summary ---- > > Reading FITS from /data/spectro//60601 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > n Time Filename design-field cfg flav shu Exp colA colB > colC Hrt > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > 1 22:49 b1-00374700 ----------- ---- bia Clo 0.0 2638 2723 270= 7 > 2 22:49 r1-00374700 ----------- ---- bia Clo 0.0 2638 2723 270= 7 > 3 22:51 b1-00374701 ----------- ---- dar Clo 20.0 2638 2723 270= 7 > 4 22:51 r1-00374701 ----------- ---- dar Clo 20.0 2638 2723 270= 7 > 5 22:53 b1-00374702 -999--999 16567 arc Lef 4.1 2638 2723 2707 > HART > 6 22:53 r1-00374702 -999--999 16567 arc Lef 4.1 2638 2723 2707 > HART > 7 22:54 r1-00374703 -999--999 16567 arc Rig 4.1 2638 2723 2707 > HART > 8 22:54 b1-00374703 -999--999 16567 arc Rig 4.1 2638 2723 2707 > HART > 9 22:55 r1-00374704 -999--999 16567 arc Out 4.1 2665 2750 2734 > 10 22:55 b1-00374704 -999--999 16567 arc Out 4.1 2665 2750 2734 > 11 22:58 b1-00374705 -999--999 16567 fla Out 25.1 2665 2750 2734 > 12 22:58 r1-00374705 -999--999 16567 fla Out 25.1 2665 2750 2734 > > > ---- Time Tracking ---- > > *=3D*=3D*=3D Observatory: APO > *=3D*=3D*=3D MJD: 60601 > Fill night time in Hours below > *=3D*=3D*=3D Total bright time : 10.9 > *=3D*=3D*=3D Bright time, queued observations : 0 > *=3D*=3D*=3D Bright time, commissioning/engineering: 0 > *=3D*=3D*=3D Bright time lost to weather : 10.9 > *=3D*=3D*=3D Bright time lost to technical issues : 0 > > *=3D*=3D*=3D Total dark time : 0 > *=3D*=3D*=3D Dark time, queued observations : 0 > *=3D*=3D*=3D Dark time, commissioning/engineering : 0 > *=3D*=3D*=3D Dark time lost to weather : 0 > *=3D*=3D*=3D Dark time lost to technical issues : 0 > > > Thu 17 Oct 2024 60601 0.99 01:20 01:34 12:02 20:19 06:48 > --- bright (10.9 h, w/ 0.4 h for 12 deg design) > > ____________________________________________________________ > Logs of Nightly APO Activities > https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/pri= vate/sloan-25mlog/index.html__;!!K6Z8K8YTIA!BJ5pzP9t8Bpyqe2aiFBM5-qkJECEMPJ= gOh1CBubP3M6D2av0CQjfIv1BFz8rb1w7wOnMQozHSXSAzCxpehj6$ > > ____________________________________________________________ > Logs of Nightly APO Activities > https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/pri= vate/sloan-25mlog/index.html__;!!K6Z8K8YTIA!DpprPny2JsZmnWmz3vxf0FVhK2053Ba= LiNbv3jJNjHNRlne1A6wmtXSnV1wQTHfnzR7G09k7zoVeW_6nOSf1$ > --_000_IA2PR16MB63945DC533F5A6D1145B3C81FEE72IA2PR16MB6394namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable P {margin-top:0;margin-bo= ttom:0;} No idea where this email from October came from? Stuck in some mail server = backlog I guess. From: sloan-25mlog on behalf of Donor, John Cc: obs-report-25m@apo.nmsu.edu Subject: [sloan-25mlog 7467] Re: 2.5m obslog 20241017 (SJD 60601) [EXTERNAL EMAIL WARNING= ] DO NOT CLICK LINKS or open attachments unless you recognize= the sender and know the content is safe. Hi Dan, I don't think this is permission issues actually. I think= the server is just rejecting connections. This will happen if it doesn't f= inish booting all the way, or for many other reasons. I'm waiting to hear f= rom Shane to see what's up exactly. And it seems gateway specific? I can login to hub, etc, f= ine with VPN. Thanks, John On Oct 18, 2024 10:01 AM, Daniel Oravetz <= ;doravetz@apo.nmsu.edu> wrote: [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or o= pen attachments unless you recognize the sender and know the content is saf= e. Hi John, Shane, all, We've done some "cold start" tests in the past and I don't see an= y special permissions problems documented in restart instructions. Are these permission issues that hampered getting subsystems restarted a result of the failure mode that caused the crash, or are these permission problems related to a new configuration setup ("new" as in since whenever = the last "cold start" tests occurred) that we should add to procedural documentation? Dan > Status update: things are no longer critical IMO, but I do still urgen= tly > need Shane's help. > > I got in via VPN, but our gateway is still refusing connections. > > Tron is up and connected to APOGEE and BOSS. No critical alerts. > > I cannot connect to sdss5-fps, so I can't talk to jaeger. Hopefully th= e > FPS is in a safe state, I think Audrey could do that manually? > > apogeeql won't start because the DB is in a weird state. There is a co= ld > start issue with postgres on that machine that I haven't been able to > track down :/ (weird race condition IIRC) anyway, I can't fix postgres= > without sudo and sudo is through ldap and ldap is down? Except that it= let > me VPN in... > > I remember Shane said there was an issue with our ldap VM booting when= we > did our last cold start test, but I thought he fixed it. Maybe somethi= ng > similar? It is odd that I can login but not sudo when it's supposedly = all > through ldap. Anyway, need Shane for this. > > I'm supposed to be on campus today so I'm going to get ready and head = that > way. I'll be available for more troubleshooting when folks are awake > probably. > > -John > ________________________________ > From: sloan-25mlog on behalf of = Donor, > John > Sent: Friday, October 18, 2024 5:19 AM > To: asimmons@apo.nmsu.edu > Cc: obs-report-25m@apo.nmsu.edu > Subject: [sloan-25mlog 7464] Re: 2.5m obslog 20241017 (SJD 60601) > > > [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless= you > recognize the sender and know the content is safe. > > I am also getting "permission" issues, but based on the imme= diate refusal > it's more like ssh is refusing to let me connect. Assuming Shane resto= red > a current image, my first instinct is that gateway isn't booted all th= e > way? > > Anyway there's not much I can do if I can't login to gateway. I'll try= to > get my VPN up; I forgot about that when my laptop was reimaged a few w= eeks > ago. But if Audrey couldn't login from site that won't help. > > On Oct 18, 2024 4:59 AM, asimmons--- via sloan-25mlog > wrote: > [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless= you > recognize the sender and know the content is safe. > >  = ; APACHE = POINT OBSERVATORY SDSS 2.5M > >  = ; &n= bsp; OBSERVING LOG >  = ; &n= bsp; Thursday October 17, 2024 > >  = ; &n= bsp; (SJD 60601) > > ---- Observing Team ---- > > Audrey Oravetz > > Shane Thomas (support) > Kaike Pan (support) > > > ---- Observing Plan ---- > > > Thu 17 Oct 2024 60601 0.99 0= 1:20 01:34 12:02 20:19 06:48 > --- bright (10.9 h, w/ 0.4 h for 12 deg design= ) > > ---- Observing Summary ---- > > The night was again lost to rain/storms/humidity. > > The hub crashed around midnight unexpectedly. With Shane's help, we we= re > able to get 10.25.190 back online with a manual power cycle. However, = I > ran into various permission issues trying to actually log onto the > machine. > > At this moment, various systems are not being monitored: alerts, apo, > apogeeql, boss, jaeger, tcc, kronos, grafana. I need help reconnecting= > these systems in the morning ASAP. > > > > ---- Observing Log ---- > > Afternoon: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 22:49:29 boss exposure 0.0, file=3D374700 > 22:51:02 boss exposure 20.0, file=3D374701 > 22:53:36 boss exposure 4.0, file=3D374702 > 22:54:16 boss exposure 4.0, file=3D374703 > 22:54:57 sp1.motor.move=3D 27, 27, 27 > 22:54:57 Hartmann collimate output > sp1: offset: r=3D -329 (In Focus); b=3D -1.2 (= In Focus) > sp1: pred. move: spAverageMove=3D 28 > sp1: pred. resid: r=3D -357, b=3D -1.1, txt=3D OK, s= pTemp =3D 7.3 > 22:55:30 boss exposure 4.0, file=3D374704 > 22:58:10 boss exposure 25.0, file=3D374705 > > 22:58:18 Config Loaded: -999-16568 > > 22:59:00 jaeger.jaeger 1489 fliswarm i > filename_bundle=3D/data/fcam/60601/fimg-fvc1n-0001.fits > 22:59:00 jaeger.jaeger 1489 fliswarm : > 22:59:00 CmdDone: jaeger.jaeger 1489 fliswarm talk -c fvc expose 5.0 > > 23:00:19 APO.aoravetz2 44 fliswarm i > filename_bundle=3D/data/fcam/60601/fimg-fvc1n-0002.fits > 23:00:19 APO.aoravetz2 44 fliswarm : > 23:00:19 CmdDone: APO.aoravetz2 44 fliswarm talk -n fvc expose 50 > > fliswarm talk -c gfa expose --bias --count 10 > fliswarm talk -c gfa expose --dark --count 5 5 > > jaeger configuration random --danger > jaeger configuration reverse > > (observers39) [observer@sdss5-hub: ~]$ list_ap > APOGEE data list, mjd=3D60601, all > raw_data: /data/apogee/utr_cdr/60601/apRaw-*.fits > archive: /data/apogee/archive/60601/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/60601/apq-*.fits > ----------------------------------------------------------------------= -------- > mjd i UT File/Exp Imty= pe Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ----------------------------------------------------------------= -------------- > 60601 1 22:59 50390001 Dark &= nbsp; 60 B 13.5 ---------- a-b-c = - > - > 60601 2 23:01 50390002 QuarFlat 10 B&nbs= p; 13.5 ---------- a-b-c - > - > 60601 3 23:03 50390003 Arc-Thar 12 B&nbs= p; 13.5 ---------- a-b-c 1.95 > - > 60601 4 23:11 50390004 Arc-Une 40 = B 13.5 ---------- a-b-c - > - > 60601 5 23:13 50390005 Arc-Thar 12 A&nbs= p; 13.0 ---------- a-b-c 1.40 > - > 60601 6 23:20 50390006 Arc-Une 40 = A 13.0 ---------- a-b-c - > - > 60601 7 23:31 50390007 Dark &= nbsp; 60 A 13.0 ---------- a-b-c = - > - > 60601 8 23:37 50390008 Arc---- 30 = A 13.0 ---------- a-b-c - > 73 FPI A > 60601 9 23:42 50390009 Arc---- 30 = B 13.5 ---------- a-b-c - > 73 FPI B > ----------------------------------------------------------------------= -------- > > Did not ventilate due to clouds/radar. > > Night: > =3D=3D=3D=3D=3D=3D > 06:39Z - > > client_loop: ssh_packet_write_poll: Connection to 10.25.1.139 port 22:= > Host is down > > (observers39) aoravetz@sdss5-obs2:~ > % ssh -X -Y -A observer@sdss5-hub > ssh: connect to host sdss5-hub port 22: Host is down > > --Test Actors, 06:42:52 > alerts ping - **FAILED** > apo ping - **FAILED** > apogee ping - ok > apogeecal ping - ok > apogeeql ping - **FAILED** > apogeefpi ping - ok > boss ping - ok > fliswarm ping - ok > jaeger ping - **FAILED** > cherno ping - ok > hal ping - ok > hub ping - ok > keys getFor=3Dhub version - ok > mcp ping - ok > msg (testing actors) - ok > perms status - ok > hartmann ping - ok > tcc show time - **FAILED** > > GRAFANA, KRONOS: > > Service Unavailable > > The server is temporarily unable to service your request due to > maintenance downtime or capacity problems. Please try again later. > > Contacted Jose, Kaike, and Shane. > > Shane said that the hub did crash, but the backup system failed to tak= e > over. Shane instructed me to physically unplug the KVM 10 server in th= e > machine room (10.25.1.90) and plug it back in a few minutes later. > > Shane said that he was able to get the machine back up and running, bu= t I > was never able to verify and log into hub due to some permissions issu= e. I > tried on both my laptop and obs2, different wifi networks, various > passwords... > > audrey@ARCs-MacBook-Pro-5 ~ % ssh -X -Y -A observer@sdss5-hub.apo.nmsu= .edu > observer@sdss5-hub.apo.nmsu.edu's password: > Permission denied, please try again. > > (observers39) aoravetz@sdss5-obs2:~ > % ssh observer@sdss5-hub > observer@sdss5-hub's password: > Permission denied, please try again. > > audrey@ARCs-MacBook-Pro-5 ~ % ssh -X -Y -A observer@sdss5-hub > kex_exchange_identification: Connection closed by remote host > Connection closed by UNKNOWN port 65535 > > audrey@ARCs-MacBook-Pro-5 ~ % ssh -X -Y -A -L > 8080:sdss5-hub.apo.nmsu.edu:80 observer@sdss5-gateway.apo.nmsu.edu > observer@sdss5-gateway.apo.nmsu.edu: Permission denied > (publickey,gssapi-keyex,gssapi-with-mic). > > (observers39) aoravetz@sdss5-obs2:~ > % ssh observer@sdss5-gateway.apo.nmsu.edu > ssh: connect to host sdss5-gateway.apo.nmsu.edu port 22: Operation tim= ed > out > > Unable to run morningcals. Unable to get systems back in and running. > > > ---- Mirror Numbers ---- > > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS = TONIGHT NOMINAL > Axial A: 0.0010 = 0.0040 > Axial B: -0.0015 -0.0014 > Axial C: -0.0060 -0.0010 > Trans D: -0.0180 -0.0090 > Lateral E: 0.3380 0.3380 > Lateral F: 0.5450 0.5435 > > GALILS > Commanded: 9800 -300 6750 19150 -3550 -3250 > Actual: 9800 -300 6750 19150 -3550 -3250 > > SETMIR VALUES > PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 > PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 > > SECONDARY: > ---------- > > Focus: 0.0000 > Air temp: 0.00 > Altitude: 30.0 > > MIGS = TONIGHT NOMINAL > Axial A: -0.0870 -0.0830 > Axial B: 0.2050 = 0.2110 > Axial C: -0.0460 -0.0450 > Trans D: -0.0140 -0.0220 > > GALILS > Commanded: 5900000 5835200 5964100 -1500 -7250 > Actual: 5883681 5811643 5906263 -1500 -7250 > > SETMIR VALUES > SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 > SecOrient: 4657.01 -24.51 -17.52 17.68 125.27 > > > > ---- Telescope Status ---- > > No interlocks bypassed > > Status at: > Telescope stowed at: 121.0, 30.0, 0.0&nb= sp; mount > Instrument mounted: FPS > Counterweights at: &nbs= p; 650s > LN2 autofill systems: Connected and turned on > 180L LN2 dewar scales: SP1 135 &nbs= p; lbs, 21 psi > > > > > ---- Software Used ---- > > -- STUI software: > STUI : 1.7.19 > alerts : 2.3.0 > boss : 6.0.2 > -boss.specMec : 3.0.4 > apo : 4.1.= 0 > fliswarm : 0.5.1a0 > jaeger : 1.7.8a0 > kaiju : 1.4.0 > coordio : 1.11.3a0 > fps_calibrati : 2024.04.01+fc6df86 > cherno : 1.0.1a0 > astrometrynet : 0.93 > hal : 1.3.= 5 > hartmann : 2.0.2 > hub : 5.0.= 1a0 > mcp : 2.1-= 1-ge461 > -mcp.plc : 35 > -mcp.azFiduci : v1_149 > -mcp.altFiduc : v1_149 > -mcp.rotFiduc : v1_149 > tcc : 1.4.= 0 > apogee : not availble > apogeecal : not availble > apogeeql : v1_7_0dev > STUIAdditions : FAILED > Boson : 0.4.9 > -- done -- > > --------------Other Versions-------------- > sdss-obstools : 3.9.3 > TPM &= nbsp; : 1.2-31-gd7eb > MCP &= nbsp; : 2.1-1-ge461 > PLC &= nbsp; : V35 > Fiducials = : v1_149 > idlspec2d = : v6_2_0 > Kronos &nbs= p; : 2.0.0 > roboscheduler : 1.4.0 > > /dev/sdb1 &= nbsp; 2.0T 448G 1.6T 22% /home > /dev/sdd1 &= nbsp; 22T 17T 4.0T 81= % /data > > > ---- APOGEE Data Summary ---- > > (observers39) [observer@sdss5-hub: ~]$ list_ap -m 60600 --morning > APOGEE data list, mjd=3D60600, all > raw_data: /data/apogee/utr_cdr/60600/apRaw-*.fits > archive: /data/apogee/archive/60600/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/60600/apq-*.fits > ----------------------------------------------------------------------= -------- > mjd i UT File/Exp Imty= pe Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ----------------------------------------------------------------= -------------- > 60600 10 11:11 50380010 Dark = 60 B 13.5 ---------- a-b-c - > - > 60600 11 11:22 50380011 Dark = 60 B 13.5 ---------- a-b-c - > - > 60600 12 11:32 50380012 Dark = 60 B 13.5 ---------- a-b-c - > - > 60600 13 11:34 50380013 QuarFlat 10 B 13= .5 ---------- a-b-c - > - > 60600 14 11:36 50380014 QuarFlat 10 B 13= .5 ---------- a-b-c - > - > 60600 15 11:38 50380015 QuarFlat 10 B 13= .5 ---------- a-b-c - > - > 60600 16 11:40 50380016 Arc-Thar 12 B 13= .5 ---------- a-b-c 1.98 > - > 60600 17 11:47 50380017 Arc-Une 40 B&nbs= p; 13.5 ---------- a-b-c - > - > 60600 18 11:49 50380018 Arc-Thar 12 A 13= .0 ---------- a-b-c 1.40 > - > 60600 19 11:57 50380019 Arc-Une 40 A&nbs= p; 13.0 ---------- a-b-c - > - > 60600 20 12:02 50380020 Dark = 30 A 13.0 ---------- a-b-c - > - > 60600 21 12:07 50380021 InteFlat 30 A 13= .0 ---------- a-b-c - > - > 60600 22 12:13 50380022 InteFlat 30 A 13= .0 ---------- a-b-c - > - > 60600 23 12:18 50380023 InteFlat 30 A 13= .0 ---------- a-b-c - > - > 60600 24 12:24 50380024 Dark = 30 A 13.0 ---------- a-b-c - > - > 60600 25 12:29 50380025 Arc---- 30 A&nbs= p; 13.0 ---------- a-b-c - > 74 FPI A > 60600 26 12:35 50380026 Arc---- 30 B&nbs= p; 13.5 ---------- a-b-c - > 73 FPI B > ----------------------------------------------------------------------= -------- > APOGEE data list, mjd=3D60601, all > raw_data: /data/apogee/utr_cdr/60601/apRaw-*.fits > archive: /data/apogee/archive/60601/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/60601/apq-*.fits > ----------------------------------------------------------------------= -------- > mjd i UT File/Exp Imty= pe Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ----------------------------------------------------------------= -------------- > 60601 1 22:59 50390001 Dark &= nbsp; 60 B 13.5 ---------- a-b-c = - > - > 60601 2 23:01 50390002 QuarFlat 10 B&nbs= p; 13.5 ---------- a-b-c - > - > 60601 3 23:03 50390003 Arc-Thar 12 B&nbs= p; 13.5 ---------- a-b-c 1.95 > - > 60601 4 23:11 50390004 Arc-Une 40 = B 13.5 ---------- a-b-c - > - > 60601 5 23:13 50390005 Arc-Thar 12 A&nbs= p; 13.0 ---------- a-b-c 1.40 > - > 60601 6 23:20 50390006 Arc-Une 40 = A 13.0 ---------- a-b-c - > - > 60601 7 23:31 50390007 Dark &= nbsp; 60 A 13.0 ---------- a-b-c = - > - > 60601 8 23:37 50390008 Arc---- 30 = A 13.0 ---------- a-b-c - > 73 FPI A > 60601 9 23:42 50390009 Arc---- 30 = B 13.5 ---------- a-b-c - > 73 FPI B > ----------------------------------------------------------------------= -------- > > > > ---- BOSS Data Summary ---- > > Reading FITS from /data/spectro//60601 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - = - - > - - > n Time Filename design-field= cfg flav shu Exp colA = ; colB > colC Hrt > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - = - - > - - > 1 22:49 b1-00374700 ----------- ---- bia Clo &n= bsp; 0.0 2638 2723 2707 > 2 22:49 r1-00374700 ----------- ---- bia Clo &n= bsp; 0.0 2638 2723 2707 > 3 22:51 b1-00374701 ----------- ---- dar Clo 2= 0.0 2638 2723 2707 > 4 22:51 r1-00374701 ----------- ---- dar Clo 2= 0.0 2638 2723 2707 > 5 22:53 b1-00374702 -999--999 16567 arc Lef &nb= sp; 4.1 2638 2723 2707 > HART > 6 22:53 r1-00374702 -999--999 16567 arc Lef &nb= sp; 4.1 2638 2723 2707 > HART > 7 22:54 r1-00374703 -999--999 16567 arc Rig &nb= sp; 4.1 2638 2723 2707 > HART > 8 22:54 b1-00374703 -999--999 16567 arc Rig &nb= sp; 4.1 2638 2723 2707 > HART > 9 22:55 r1-00374704 -999--999 16567 arc Out &nb= sp; 4.1 2665 2750 2734 > 10 22:55 b1-00374704 -999--999 16567 arc Out 4= .1 2665 2750 2734 > 11 22:58 b1-00374705 -999--999 16567 fla Out 25.1&nb= sp; 2665 2750 2734 > 12 22:58 r1-00374705 -999--999 16567 fla Out 25.1&nb= sp; 2665 2750 2734 > > > ---- Time Tracking ---- > > *=3D*=3D*=3D Observatory: APO > *=3D*=3D*=3D MJD: 60601 > Fill night time in Hours bel= ow > *=3D*=3D*=3D Total bright time &nbs= p; &= nbsp; : 10.9 > *=3D*=3D*=3D Bright time, queued observations &= nbsp; : 0 > *=3D*=3D*=3D Bright time, commissioning/engineering: 0 > *=3D*=3D*=3D Bright time lost to weather = : 10.9 > *=3D*=3D*=3D Bright time lost to technical issues : 0 > > *=3D*=3D*=3D Total dark time = &nb= sp; : 0 > *=3D*=3D*=3D Dark time, queued observations &nb= sp; : 0 > *=3D*=3D*=3D Dark time, commissioning/engineering : 0 > *=3D*=3D*=3D Dark time lost to weather &n= bsp; : 0 > *=3D*=3D*=3D Dark time lost to technical issues : 0 > > > Thu 17 Oct 2024 60601 0.99 0= 1:20 01:34 12:02 20:19 06:48 > --- bright (10.9 h, w/ 0.4 h for 12 deg design= ) > > ____________________________________________________________ > Logs of Nightly APO Activities > https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/priva= te/sloan-25mlog/index.html__;!!K6Z8K8YTIA!BJ5pzP9t8Bpyqe2aiFBM5-qkJECEMPJgO= h1CBubP3M6D2av0CQjfIv1BFz8rb1w7wOnMQozHSXSAzCxpehj6$ > > ____________________________________________________________ > Logs of Nightly APO Activities > https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/priva= te/sloan-25mlog/index.html__;!!K6Z8K8YTIA!DpprPny2JsZmnWmz3vxf0FVhK2053BaLi= Nbv3jJNjHNRlne1A6wmtXSnV1wQTHfnzR7G09k7zoVeW_6nOSf1$ > --_000_IA2PR16MB63945DC533F5A6D1145B3C81FEE72IA2PR16MB6394namp_--