From: "Donor, John" Subject: Re: [sloan-25mlog 6813] 2.5m obslog 20230912 (SJD 60200) Date: Wed, 13 Sep 2023 15:44:25 +0000 --_000_PH7PR16MB5216B57CF9741EA2F24815B2FEF0APH7PR16MB5216namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Audrey, all, Sorry about all the alerts chaos the last few days; all the problems are re= lated to shutdown type things that don't happen normally. But I think I've = managed to find reasonable ways to handle the chaos? I'm not 100% sure why you were seeing the timeout error last night, but I s= aw how annoying it was when I logged in this morning... Hopefully I've solv= ed it. It's behaving for now at least. We are still seeing "intermittent" behavior from camCheck? It seems that ca= mCheck messages are showing 1 alert, 2 alerts, 1 alert, 2 alerts, alternati= ng. The actor is handling it ok for now, hopefully it will continue to. But= I'm not sure it's worth looking into the cause of this camCheck behavior f= urther (probably not, but thought I'd mention it). Thanks, John ________________________________ From: sloan-25mlog on behalf of asimmons@ap= o.nmsu.edu Sent: Wednesday, September 13, 2023 4:02 AM To: obs-report-25m@apo.nmsu.edu Subject: [sloan-25mlog 6813] 2.5m obslog 20230912 (SJD 60200) [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless you = recognize the sender and know the content is safe. APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday September 12, 2023 (SJD 60200) ---- Observing Team ---- Audrey Oravetz ---- Observing Plan ---- FVC scans ---- Observing Summary ---- The night was devoted to FVC rotation images for Conor. The weather was not= conducive for regular operations despite still being in shutdown. There were 2 false collisions (out of 10 trials) using 'jaeger configuratio= n random', all with robot 535. There were 3 false collisions (out of 10 trials) using 'jaeger configuratio= n random --danger', with robot 535 and 478 (both are notorious for giving f= alse collisions). I was not able to load or preload current observable designs extracted from= KRONOS and execute FVC loops because of error like this: jaeger e error=3D"Failed retrieving design: operands could not be broadcast= together with shapes (226,10) (226,) " Round 1: fimg-fvc1n-0004.fits - fimg-fvc1n-0111.fits (starting and ending w= ith ROT=3D0) Round 2: fimg-fvc1n-0148.fits - fimg-fvc1n-0255.fits (starting with ROT=3D5= ) Round 3: fimg-fvc1n-0256.fits - fimg-fvc1n-0366.fits (starting and ending w= ith ROT=3D0) Round 4: fimg-fvc1n-0367.fits - fimg-fvc1n-0474.fits (starting with ROT=3D5= ) ---- Observing Log ---- Night: =3D=3D=3D=3D=3D=3D Filled B1 and removed LN2 line. Test slewed only in rotation. Turned on the= FPS. STUI is complaining with every B1 camCheck, that there is a timeout with th= e status: >> > >> > 01:59:12 APO.aoravetz 1070 stui F Timeout; Actor=3D'alerts';>> Cmd=3D"= status" It stopped around 02:23Z. B1 temperature is holding steady at -100, with go= od vacuum. APOGEE evening cals (observers39) [observer@sdss5-hub: ~]$ list_ap APOGEE data list, mjd=3D60200, all raw_data: /data/apogee/utr_cdr/60200/apRaw-*.fits archive: /data/apogee/archive/60200/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60200/apq-*.fits ---------------------------------------------------------------------------= --- mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Fl= ux/FPI Comm --------------------------------------------------------------------------= ---- 60200 1 02:30 46380001 Dark 60 A 13.0 ---------- a-b-c - - 60200 2 02:32 46380002 QuarFlat 10 A 13.0 ---------- a-b-c - - 60200 3 02:34 46380003 Arc-Thar 12 B 13.5 ---------- a-b-c 1.01 - 60200 4 02:42 46380004 Arc-Une 40 B 13.5 ---------- a-b-c - - 60200 5 02:44 46380005 Arc-Thar 12 A 13.0 ---------- a-b-c 0.45 - 60200 6 02:51 46380006 Arc-Une 40 A 13.0 ---------- a-b-c - - 60200 7 03:02 46380007 Dark 60 A 13.0 ---------- a-b-c - - 60200 8 03:08 46380008 Arc---- 30 A 13.0 ---------- a-b-c - 8= 8 FPI Started FVC scans for Conor: Could not restart mirror controllers to implement changes since the telesco= pe is immobilized in altitude and azimuth. =3D=3D=3D=3D=3D ** 10ish =93jaeger configuration random=94 and =93jaeger configuration reve= rses=94, note any false collisions, make sure stuff seems to be working Ex. 02:15:10 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0002.pdf 02:15:11 APO.aoravetz2 1156 stui F Timeout; Actor=3D'alerts'; Cmd=3D"status= " 02:15:11 APO.aoravetz2 1157 stui F Timeout; Actor=3D'alerts'; Cmd=3D"status= " 02:15:13 CmdStarted: APO.aoravetz2 5 jaeger configuration random 02:15:13 APO.aoravetz2 5 jaeger > 02:15:13 APO.aoravetz2 5 jaeger i text=3D"Creating random configuration." Snapshots 1-23 02:20:17 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D10.009464249014854; locked_beta=3D170.00253796577454 02:20:18 APO.aoravetz2 7 jaeger f error=3D"Trajectory failed with error: So= mething went wrong during the trajectory: The FPS got locked during the tra= jectory. Trajectory failed 30.52 seconds after start" 02:20:18 CmdDone: APO.aoravetz2 7 jaeger configuration random jaeger unlock jaeger explode 10 jaeger unwind 02:37:45 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D9.998941272497177; locked_beta=3D170.00009179115295 02:37:45 .jaeger 0 jaeger i fps_status=3D0x14 02:37:46 APO.aoravetz2 22 jaeger f error=3D"Trajectory failed with error: S= omething went wrong during the trajectory: The FPS got locked during the tr= ajectory. Trajectory failed 29.23 seconds after start" 02:37:46 CmdDone: APO.aoravetz2 22 jaeger configuration random =3D=3D=3D=3D=3D *** Do 10ish =93jaeger configuration random =97danger=94 and =93jaeger conf= iguration reverse=94, note any false collisions, make sure stuff seems to be working. Snapshots 24-52 02:54:07 CmdStarted: APO.aoravetz2 33 jaeger configuration random --danger 02:54:07 APO.aoravetz2 33 jaeger > 03:10:19 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D9.989084154367447; locked_beta=3D170.0010359287262 03:10:19 .jaeger 0 jaeger i fps_status=3D0x14 03:10:19 .tcc 0 tcc i AltTrackAdvTime=3D20,3.49,3.50,3.50,0.00 03:10:19 .mcp 0 mcp i trace=3D"altitude reposition caused frame break" 03:10:19 .mcp 0 mcp ! command=3D"move" 03:10:20 .tcc 0 tcc i TCCPos=3D121.000000,90.000000,60.000000; AxePos=3D120= .999210,90.035673,60.001091; ObjInstAng=3Dnan,nan,5201291428.3314857; Spide= rInstAng=3Dnan,nan,5201291428.3314857 03:10:20 .tcc 0 tcc i AxisCmdState=3DTracking,Tracking,Tracking; AxisErrCod= e=3DOK,OK,OK; AxisNoTrack=3DFFF 03:10:20 APO.aoravetz2 37 jaeger f error=3D"Trajectory failed with error: S= omething went wrong during the trajectory: The FPS got locked during the tr= ajectory. Trajectory failed 38.41 seconds after start" 03:10:20 CmdDone: APO.aoravetz2 37 jaeger configuration random --danger 03:10:24 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0029.pdf 03:10:24 .jaeger 0 jaeger w text=3D"Snapshot for locked FPS: /data/logs/jae= ger/snapshots/60200/fps_snapshot_60200_0029.pdf (JaegerUserWarning)" 03:20:10 .jaeger 0 jaeger f text=3D"A collision was detected in positioner = 478. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 03:20:10 .jaeger 0 jaeger w text=3D"Locking the FPS. (JaegerUserWarning)" 03:20:12 .jaeger 0 jaeger e locked=3DT; locked_by=3D478; locked_axes=3Dbeta= ; locked_alpha=3D10.004368722438812; locked_beta=3D169.999920129776 03:20:12 .jaeger 0 jaeger i fps_status=3D0x14 03:20:12 APO.aoravetz2 43 jaeger f error=3D"Trajectory failed with error: S= omething went wrong during the trajectory: The FPS got locked during the tr= ajectory. Trajectory failed 39.01 seconds after start" 03:20:12 CmdDone: APO.aoravetz2 43 jaeger configuration random --danger 03:20:14 .tcc 0 tcc i AzTrackAdvTime=3D20,3.49,3.50,3.50,0.00 03:20:14 .mcp 0 mcp i trace=3D"altitude reposition caused frame break" 03:20:14 .mcp 0 mcp ! command=3D"move" 03:20:14 .ecamera 877 ecamera i version=3D"2.3.0" 03:20:15 .tcc 0 tcc i TCCPos=3D121.000000,90.000000,60.000000; AxePos=3D120= .999203,90.035669,60.001091; ObjInstAng=3Dnan,nan,5201292023.3314857; Spide= rInstAng=3Dnan,nan,5201292023.3314857 03:20:15 .tcc 0 tcc i AxisCmdState=3DTracking,Tracking,Tracking; AxisErrCod= e=3DOK,OK,OK; AxisNoTrack=3DFFF 03:20:16 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0034.pdf 03:20:16 .jaeger 0 jaeger w text=3D"Snapshot for locked FPS: /data/logs/jae= ger/snapshots/60200/fps_snapshot_60200_0034.pdf (JaegerUserWarning)" 03:20:16 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0034.pdf 03:20:51 .jaeger 0 jaeger f text=3D"A collision was detected in positioner = 535. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 03:20:52 .jaeger 0 jaeger w text=3D"Locking the FPS. (JaegerUserWarning)" 03:20:53 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D224.1500186920166; locked_beta=3D123.47470879554749 03:20:53 .jaeger 0 jaeger i fps_status=3D0x14 03:20:53 APO.aoravetz2 45 jaeger f error=3D"Trajectory failed with error: S= omething went wrong during the trajectory: The FPS got locked during the tr= ajectory. Trajectory failed 3.89 seconds after start" 03:20:53 CmdDone: APO.aoravetz2 45 jaeger explode 10 03:45:35 .jaeger 0 jaeger f text=3D"A collision was detected in positioner = 535. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 03:45:35 .jaeger 0 jaeger w text=3D"Locking the FPS. (JaegerUserWarning)" 03:45:36 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D9.99064989387989; locked_beta=3D170.00436186790466 03:45:37 .jaeger 0 jaeger i fps_status=3D0x14 03:45:37 APO.aoravetz2 65 jaeger f error=3D"Trajectory failed with error: S= omething went wrong during the trajectory: The FPS got locked during the tr= ajectory. Trajectory failed 21.16 seconds after start" 03:45:37 CmdDone: APO.aoravetz2 65 jaeger configuration random --danger =3D=3D=3D=3D=3D=3D *** Configure for a few designs that you might observe if you had a working telescope, and do a normal fvc loop, make sure stuff seems to be working. 03:52:07 CmdStarted: APO.SeniorCondor 3 jaeger configuration preload 03:52:07 APO.SeniorCondor 3 jaeger > 03:52:07 APO.SeniorCondor 3 jaeger i text=3D"Preloading design 361228." 03:52:07 CmdStarted: jaeger.jaeger 1295 cherno get-scale --max-age 3600 03:52:07 jaeger.jaeger 1295 cherno > 03:52:08 jaeger.jaeger 1295 cherno w text=3D"Not enough points to calculate= median scale." 03:52:08 jaeger.jaeger 1295 cherno : scale_median=3D-999.0 03:52:08 CmdDone: jaeger.jaeger 1295 cherno get-scale --max-age 3600 03:52:08 APO.SeniorCondor 3 jaeger w text=3D"Invalid guider scale." 03:52:08 APO.SeniorCondor 3 jaeger i text=3D"Focal scale: 0.999816." 03:52:17 APO.SeniorCondor 3 jaeger e error=3D"Failed retrieving design: ope= rands could not be broadcast together with shapes (226,10) (226,) " 03:52:17 APO.SeniorCondor 3 jaeger f 03:52:17 CmdDone: APO.SeniorCondor 3 jaeger configuration preload 03:54:40 CmdStarted: APO.aoravetz2 72 jaeger configuration load 361228 03:54:40 APO.aoravetz2 72 jaeger > 03:54:40 APO.aoravetz2 72 jaeger i text=3D"Loading design 361228." 03:54:40 CmdStarted: jaeger.jaeger 1296 cherno get-scale --max-age 3600 03:54:40 jaeger.jaeger 1296 cherno > 03:54:40 jaeger.jaeger 1296 cherno w text=3D"Not enough points to calculate= median scale." 03:54:40 jaeger.jaeger 1296 cherno : scale_median=3D-999.0 03:54:40 CmdDone: jaeger.jaeger 1296 cherno get-scale --max-age 3600 03:54:40 APO.aoravetz2 72 jaeger w text=3D"Invalid guider scale." 03:54:40 APO.aoravetz2 72 jaeger i text=3D"Focal scale: 0.999817." 03:54:40 APO.aoravetz2 72 jaeger e error=3D"Failed retrieving design: opera= nds could not be broadcast together with shapes (226,10) (226,) " 03:54:40 APO.aoravetz2 72 jaeger f 03:54:40 CmdDone: APO.aoravetz2 72 jaeger configuration load 361228 03:55:13 CmdStarted: APO.SeniorCondor 4 jaeger configuration preload 03:55:13 APO.SeniorCondor 4 jaeger > 03:55:13 APO.SeniorCondor 4 jaeger i text=3D"Preloading design 364229." 03:55:13 CmdStarted: jaeger.jaeger 1297 cherno get-scale --max-age 3600 03:55:13 jaeger.jaeger 1297 cherno > 03:55:13 jaeger.jaeger 1297 cherno w text=3D"Not enough points to calculate= median scale." 03:55:13 jaeger.jaeger 1297 cherno : scale_median=3D-999.0 03:55:13 CmdDone: jaeger.jaeger 1297 cherno get-scale --max-age 3600 03:55:13 APO.SeniorCondor 4 jaeger w text=3D"Invalid guider scale." 03:55:13 APO.SeniorCondor 4 jaeger i text=3D"Focal scale: 0.999817." 03:55:19 APO.SeniorCondor 4 jaeger e error=3D"Failed retrieving design: ope= rands could not be broadcast together with shapes (111,10) (111,) " 03:55:19 APO.SeniorCondor 4 jaeger f 03:55:19 CmdDone: APO.SeniorCondor 4 jaeger configuration preload =3D=3D=3D=3D=3D *** Do 4ish manual FVC rotator scans: a. configure robots randomly =93jaeger configuration random =97dang= er=94 b. move the rotator in 10 degree increments between 0-360 (0, 10, 2= 0, =85 340, 350) c. at each rotator position, take 3x =93jaeger fvc loop =97no-apply= =97fbi-level 4=94 d. send robots home =93jaeger configuration reverse" e. repeat steps a-d 4 times. Every other time offset the rotator p= ositions by 5 degrees (5, 15, 25, =85, 345, 355) f. note the first and last fvc image for each full rotator scan, no= need to keep track of every image, I can find the rotator angles from the headers. I on= ly need to know when you=92ve begun a new scan with different robot positions. ****FIRST ROUND**** jaeger configuration random =97danger tcc track 121,90 mount /rota=3D0, 10, 20... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 ... ROT=3D0 04:00:21 FVC LOOP AND BOSS CALS fvc_rms=3D 17705.39 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0001.fits fvc_perc_90=3D 24636.10 fvc_percent_reached=3D 0.00 04:01:17 FVC LOOP AND BOSS CALS fvc_rms=3D 318.85 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0002.fits fvc_perc_90=3D 467.10 fvc_percent_reached=3D 0.00 04:01:59 FVC LOOP AND BOSS CALS fvc_rms=3D 40.84 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0003.fits fvc_perc_90=3D 63.40 fvc_percent_reached=3D 2.60 ... ROT=3D350 05:05:52 FVC LOOP AND BOSS CALS fvc_rms=3D 40.61 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0106.fits fvc_perc_90=3D 62.30 fvc_percent_reached=3D 2.00 05:06:21 FVC LOOP AND BOSS CALS fvc_rms=3D 40.51 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0107.fits fvc_perc_90=3D 62.90 fvc_percent_reached=3D 2.00 05:06:49 FVC LOOP AND BOSS CALS fvc_rms=3D 40.47 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0108.fits fvc_perc_90=3D 62.30 fvc_percent_reached=3D 1.40 ROT=3D0 05:10:26 FVC LOOP AND BOSS CALS fvc_rms=3D 40.72 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0109.fits fvc_perc_90=3D 61.50 fvc_percent_reached=3D 2.00 05:10:55 FVC LOOP AND BOSS CALS fvc_rms=3D 40.61 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0110.fits fvc_perc_90=3D 61.80 fvc_percent_reached=3D 2.60 05:11:23 FVC LOOP AND BOSS CALS fvc_rms=3D 40.58 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0111.fits fvc_perc_90=3D 61.20 fvc_percent_reached=3D 2.00 jaeger configuration reverse ****SECOND ROUND**** fvc images 112-147 were of the same random arrangement as the first round. = Reverse was not issued until a bit later. jaeger configuration random =97danger tcc track 121,90 mount /rota=3D5, 15, 25... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 , etc. etc. ROT=3D5 05:40:40 FVC LOOP AND BOSS CALS fvc_rms=3D 45.29 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0148.fits fvc_perc_90=3D 64.80 fvc_percent_reached=3D 1.80 05:41:09 FVC LOOP AND BOSS CALS fvc_rms=3D 45.4 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0149.fits fvc_perc_90=3D 64.10 fvc_percent_reached=3D 1.80 05:41:37 FVC LOOP AND BOSS CALS fvc_rms=3D 45.13 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0150.fits fvc_perc_90=3D 63.90 fvc_percent_reached=3D 1.80 ROT=3D355 06:41:12 FVC LOOP AND BOSS CALS fvc_rms=3D 44.96 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0253.fits fvc_perc_90=3D 64.90 fvc_percent_reached=3D 3.60 06:41:41 FVC LOOP AND BOSS CALS fvc_rms=3D 44.38 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0254.fits fvc_perc_90=3D 65.30 fvc_percent_reached=3D 4.20 06:42:13 FVC LOOP AND BOSS CALS fvc_rms=3D 44.77 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0255.fits fvc_perc_90=3D 64.80 fvc_percent_reached=3D 4.00 jaeger configuration reverse ****THIRD ROUND**** jaeger configuration random =97danger tcc track 121,90 mount /rota=3D0, 10, 20... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 , etc. etc. ROT=3D0 06:46:38 FVC LOOP AND BOSS CALS fvc_rms=3D 44.69 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0256.fits fvc_perc_90=3D 63.50 fvc_percent_reached=3D 2.00 06:47:07 FVC LOOP AND BOSS CALS fvc_rms=3D 44.86 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0257.fits fvc_perc_90=3D 63.90 fvc_percent_reached=3D 1.60 06:47:36 FVC LOOP AND BOSS CALS fvc_rms=3D 44.71 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0258.fits fvc_perc_90=3D 64.20 fvc_percent_reached=3D 2.20 ROT=3D350 07:47:18 FVC LOOP AND BOSS CALS fvc_rms=3D 45.34 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0361.fits fvc_perc_90=3D 63.60 fvc_percent_reached=3D 1.40 07:47:47 FVC LOOP AND BOSS CALS fvc_rms=3D 45.34 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0362.fits fvc_perc_90=3D 63.30 fvc_percent_reached=3D 1.00 07:48:16 FVC LOOP AND BOSS CALS fvc_rms=3D 45.19 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0363.fits fvc_perc_90=3D 63.60 fvc_percent_reached=3D 1.80 ROT=3D0 07:51:52 FVC LOOP AND BOSS CALS fvc_rms=3D 45.64 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0364.fits fvc_perc_90=3D 61.90 fvc_percent_reached=3D 2.40 07:52:21 FVC LOOP AND BOSS CALS fvc_rms=3D 45.4 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0365.fits fvc_perc_90=3D 62.00 fvc_percent_reached=3D 1.80 07:52:49 FVC LOOP AND BOSS CALS fvc_rms=3D 45.55 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0366.fits fvc_perc_90=3D 63.00 fvc_percent_reached=3D 2.20 jaeger configuration reverse ****FOURTH ROUND**** jaeger configuration random =97danger tcc track 121,90 mount /rota=3D5, 15, 25... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 , etc. etc. ROT=3D5 07:58:25 FVC LOOP AND BOSS CALS fvc_rms=3D 44.09 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0367.fits fvc_perc_90=3D 61.90 fvc_percent_reached=3D 2.00 07:58:53 FVC LOOP AND BOSS CALS fvc_rms=3D 43.86 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0368.fits fvc_perc_90=3D 62.30 fvc_percent_reached=3D 2.00 07:59:22 FVC LOOP AND BOSS CALS fvc_rms=3D 43.56 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0369.fits fvc_perc_90=3D 61.50 fvc_percent_reached=3D 2.00 ROT=3D355 08:59:35 FVC LOOP AND BOSS CALS fvc_rms=3D 43.88 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0472.fits fvc_perc_90=3D 61.60 fvc_percent_reached=3D 1.60 09:00:04 FVC LOOP AND BOSS CALS fvc_rms=3D 43.82 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0473.fits fvc_perc_90=3D 62.20 fvc_percent_reached=3D 2.00 09:00:34 FVC LOOP AND BOSS CALS fvc_rms=3D 44.29 fvc_deltarms=3D None fvc_filename=3D/data/fcam/60200/fimg-fvc1n-0474.fits fvc_perc_90=3D 63.10 fvc_percent_reached=3D 1.80 jaeger configuration reverse =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Hooked up the LN2 line, turned off the rotator amplifer, and turned off the= FPS. ---- Mirror Numbers ---- N/A ---- Telescope Status ---- Status at: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 470s LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 250 lbs, 20 psi ---- Software Used ---- Get Versions, 09/13/23, 04:59:06 -- STUI software: STUI : 1.7.18b1 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.4.0 jaeger : 1.4.1a0 kaiju : 1.3.1 coordio : 1.7.1 fps_calibrati : 2023.07.27 cherno : 0.6.3 astrometrynet : 0.93 hal : 0.6.0b5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.3.2 -- done -- --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_149 idlspec2d : v6_1_0 Kronos : 1.6.2a roboscheduler : 1.4.0 /dev/sdb1 2.0T 290G 1.8T 15% /home /dev/sdd1 22T 8.9T 12T 43% /data ---- Time Tracking ---- Wed 13 Sep 2023 60201 0.01 02:05 02:20 11:37 18:48 04:07 --- dark ( 9.3 h) = plus 0.2 h for 12 deg design *=3D*=3D*=3D Observatory: APO *=3D*=3D*=3D MJD: 60200 Fill night time in Hours below *=3D*=3D*=3D Total bright time : 0 *=3D*=3D*=3D Bright time, queued observations : 0 *=3D*=3D*=3D Bright time, commissioning/engineering: 0 *=3D*=3D*=3D Bright time lost to weather : 0 *=3D*=3D*=3D Bright time lost to technical issues : 0 *=3D*=3D*=3D Total dark time : 9.3 *=3D*=3D*=3D Dark time, queued observations : 0 *=3D*=3D*=3D Dark time, commissioning/engineering : 0 *=3D*=3D*=3D Dark time lost to weather : 9.3 *=3D*=3D*=3D Dark time lost to technical issues : 0 --_000_PH7PR16MB5216B57CF9741EA2F24815B2FEF0APH7PR16MB5216namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable P {margin-top:0;margin-bo= ttom:0;} Hi Audrey, all, Sorry about all the alerts chaos the last few days; all the problems are re= lated to shutdown type things that don't happen normally. But I think I've = managed to find reasonable ways to handle the chaos? I'm not 100% sure why you were seeing the timeout error last night, but I s= aw how annoying it was when I logged in this morning... Hopefully I've solv= ed it. It's behaving for now at least. We are still seeing "intermittent" behavior from camCheck? It see= ms that camCheck messages are showing 1 alert, 2 alerts, 1 alert, 2 alerts,= alternating. The actor is handling it ok for now, hopefully it will contin= ue to. But I'm not sure it's worth looking into the cause of this camCheck behavior further (probably not, but though= t I'd mention it). Thanks, John From: sloan-25mlog on behalf of asimmons@apo.nmsu.edu Sent: Wednesday, September 13, 2023 4:02 AM To: obs-report-25m@apo.nmsu.edu Subject: [sloan-25mlog 6813] 2.5m obslog 20230912 (SJD 60200) [EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or ope= n attachments unless you recognize the sender and know the content is safe.= &nb= sp; APACHE POINT OBSE= RVATORY SDSS 2.5M &nb= sp; = OBSERVING LOG &nb= sp; = Tuesday September 12, 2023 &nb= sp; = (SJD 60200) ---- Observing Team ---- Audrey Oravetz ---- Observing Plan ---- FVC scans ---- Observing Summary ---- The night was devoted to FVC rotation images for Conor. The weather was not= conducive for regular operations despite still being in shutdown. There were 2 false collisions (out of 10 trials) using 'jaeger configuratio= n random', all with robot 535. There were 3 false collisions (out of 10 trials) using 'jaeger configuratio= n random --danger', with robot 535 and 478 (both are notorious for giving f= alse collisions). I was not able to load or preload current observable designs extracted from= KRONOS and execute FVC loops because of error like this: jaeger e error=3D"Failed retrieving design: operands could not be broa= dcast together with shapes (226,10) (226,) " Round 1: fimg-fvc1n-0004.fits - fimg-fvc1n-0111.fits (starting and ending w= ith ROT=3D0) Round 2: fimg-fvc1n-0148.fits - fimg-fvc1n-0255.fits (starting with ROT=3D5= ) Round 3: fimg-fvc1n-0256.fits - fimg-fvc1n-0366.fits (starting and ending w= ith ROT=3D0) Round 4: fimg-fvc1n-0367.fits - fimg-fvc1n-0474.fits (starting with ROT=3D5= ) ---- Observing Log ---- Night: =3D=3D=3D=3D=3D=3D Filled B1 and removed LN2 line. Test slewed only in rotation. Turned on the= FPS. STUI is complaining with every B1 camCheck, that there is a timeout with th= e status: >> > >> > 01:59:12 APO.aoravetz 1070 stui F Timeout; Actor=3D'alerts';&= gt;> Cmd=3D"status" It stopped around 02:23Z. B1 temperature is holding steady at -100, with go= od vacuum. APOGEE evening cals (observers39) [observer@sdss5-hub: ~]$ list_ap APOGEE data list, mjd=3D60200, all raw_data: /data/apogee/utr_cdr/60200/apRaw-*.fits archive: /data/apogee/archive/60200/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60200/apq-*.fits ---------------------------------------------------------------------------= --- mjd i UT File/Exp Imtype Nr= ead A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ---------------------------------------------------------------------= --------- 60200 1 02:30 46380001 Dark = 60 A 13.0 ---------- a-b-c -&nbs= p; - 60200 2 02:32 46380002 QuarFlat 10 A 13= .0 ---------- a-b-c - - 60200 3 02:34 46380003 Arc-Thar 12 B 13= .5 ---------- a-b-c 1.01 - 60200 4 02:42 46380004 Arc-Une 40 B&nbs= p; 13.5 ---------- a-b-c - - 60200 5 02:44 46380005 Arc-Thar 12 A 13= .0 ---------- a-b-c 0.45 - 60200 6 02:51 46380006 Arc-Une 40 A&nbs= p; 13.0 ---------- a-b-c - - 60200 7 03:02 46380007 Dark = 60 A 13.0 ---------- a-b-c -&nbs= p; - 60200 8 03:08 46380008 Arc---- 30 A&nbs= p; 13.0 ---------- a-b-c - 88&nbs= p; FPI Started FVC scans for Conor: Could not restart mirror controllers to implement changes since the telesco= pe is immobilized in altitude and azimuth. =3D=3D=3D=3D=3D ** 10ish =93jaeger configuration random=94 and =93jaeger configuration reve= rses=94, note any false collisions, make sure stuff seems to be working Ex. 02:15:10 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0002.pdf 02:15:11 APO.aoravetz2 1156 stui F Timeout; Actor=3D'alerts'; Cmd=3D"s= tatus" 02:15:11 APO.aoravetz2 1157 stui F Timeout; Actor=3D'alerts'; Cmd=3D"s= tatus" 02:15:13 CmdStarted: APO.aoravetz2 5 jaeger configuration random 02:15:13 APO.aoravetz2 5 jaeger > 02:15:13 APO.aoravetz2 5 jaeger i text=3D"Creating random configuratio= n." Snapshots 1-23 02:20:17 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D10.009464249014854; locked_beta=3D170.00253796577454 02:20:18 APO.aoravetz2 7 jaeger f error=3D"Trajectory failed with erro= r: Something went wrong during the trajectory: The FPS got locked during th= e trajectory. Trajectory failed 30.52 seconds after start" 02:20:18 CmdDone: APO.aoravetz2 7 jaeger configuration random jaeger unlock jaeger explode 10 jaeger unwind 02:37:45 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D9.998941272497177; locked_beta=3D170.00009179115295 02:37:45 .jaeger 0 jaeger i fps_status=3D0x14 02:37:46 APO.aoravetz2 22 jaeger f error=3D"Trajectory failed with err= or: Something went wrong during the trajectory: The FPS got locked during t= he trajectory. Trajectory failed 29.23 seconds after start" 02:37:46 CmdDone: APO.aoravetz2 22 jaeger configuration random =3D=3D=3D=3D=3D *** Do 10ish =93jaeger configuration random =97danger=94 and =93jaeger conf= iguration reverse=94, note any false collisions, make sure stuff seems to be working.= Snapshots 24-52 02:54:07 CmdStarted: APO.aoravetz2 33 jaeger configuration random --danger 02:54:07 APO.aoravetz2 33 jaeger > 03:10:19 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D9.989084154367447; locked_beta=3D170.0010359287262 03:10:19 .jaeger 0 jaeger i fps_status=3D0x14 03:10:19 .tcc 0 tcc i AltTrackAdvTime=3D20,3.49,3.50,3.50,0.00 03:10:19 .mcp 0 mcp i trace=3D"altitude reposition caused frame break&= quot; 03:10:19 .mcp 0 mcp ! command=3D"move" 03:10:20 .tcc 0 tcc i TCCPos=3D121.000000,90.000000,60.000000; AxePos=3D120= .999210,90.035673,60.001091; ObjInstAng=3Dnan,nan,5201291428.3314857; Spide= rInstAng=3Dnan,nan,5201291428.3314857 03:10:20 .tcc 0 tcc i AxisCmdState=3DTracking,Tracking,Tracking; AxisErrCod= e=3DOK,OK,OK; AxisNoTrack=3DFFF 03:10:20 APO.aoravetz2 37 jaeger f error=3D"Trajectory failed with err= or: Something went wrong during the trajectory: The FPS got locked during t= he trajectory. Trajectory failed 38.41 seconds after start" 03:10:20 CmdDone: APO.aoravetz2 37 jaeger configuration random --danger 03:10:24 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0029.pdf 03:10:24 .jaeger 0 jaeger w text=3D"Snapshot for locked FPS: /data/log= s/jaeger/snapshots/60200/fps_snapshot_60200_0029.pdf (JaegerUserWarning)&qu= ot; 03:20:10 .jaeger 0 jaeger f text=3D"A collision was detected in positi= oner 478. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 03:20:10 .jaeger 0 jaeger w text=3D"Locking the FPS. (JaegerUserWarnin= g)" 03:20:12 .jaeger 0 jaeger e locked=3DT; locked_by=3D478; locked_axes=3Dbeta= ; locked_alpha=3D10.004368722438812; locked_beta=3D169.999920129776 03:20:12 .jaeger 0 jaeger i fps_status=3D0x14 03:20:12 APO.aoravetz2 43 jaeger f error=3D"Trajectory failed with err= or: Something went wrong during the trajectory: The FPS got locked during t= he trajectory. Trajectory failed 39.01 seconds after start" 03:20:12 CmdDone: APO.aoravetz2 43 jaeger configuration random --danger 03:20:14 .tcc 0 tcc i AzTrackAdvTime=3D20,3.49,3.50,3.50,0.00 03:20:14 .mcp 0 mcp i trace=3D"altitude reposition caused frame break&= quot; 03:20:14 .mcp 0 mcp ! command=3D"move" 03:20:14 .ecamera 877 ecamera i version=3D"2.3.0" 03:20:15 .tcc 0 tcc i TCCPos=3D121.000000,90.000000,60.000000; AxePos=3D120= .999203,90.035669,60.001091; ObjInstAng=3Dnan,nan,5201292023.3314857; Spide= rInstAng=3Dnan,nan,5201292023.3314857 03:20:15 .tcc 0 tcc i AxisCmdState=3DTracking,Tracking,Tracking; AxisErrCod= e=3DOK,OK,OK; AxisNoTrack=3DFFF 03:20:16 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0034.pdf 03:20:16 .jaeger 0 jaeger w text=3D"Snapshot for locked FPS: /data/log= s/jaeger/snapshots/60200/fps_snapshot_60200_0034.pdf (JaegerUserWarning)&qu= ot; 03:20:16 .jaeger 0 jaeger i snapshot=3D/data/logs/jaeger/snapshots/60200/fp= s_snapshot_60200_0034.pdf 03:20:51 .jaeger 0 jaeger f text=3D"A collision was detected in positi= oner 535. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 03:20:52 .jaeger 0 jaeger w text=3D"Locking the FPS. (JaegerUserWarnin= g)" 03:20:53 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D224.1500186920166; locked_beta=3D123.47470879554749 03:20:53 .jaeger 0 jaeger i fps_status=3D0x14 03:20:53 APO.aoravetz2 45 jaeger f error=3D"Trajectory failed with err= or: Something went wrong during the trajectory: The FPS got locked during t= he trajectory. Trajectory failed 3.89 seconds after start" 03:20:53 CmdDone: APO.aoravetz2 45 jaeger explode 10 03:45:35 .jaeger 0 jaeger f text=3D"A collision was detected in positi= oner 535. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 03:45:35 .jaeger 0 jaeger w text=3D"Locking the FPS. (JaegerUserWarnin= g)" 03:45:36 .jaeger 0 jaeger e locked=3DT; locked_by=3D535; locked_axes=3Dbeta= ; locked_alpha=3D9.99064989387989; locked_beta=3D170.00436186790466 03:45:37 .jaeger 0 jaeger i fps_status=3D0x14 03:45:37 APO.aoravetz2 65 jaeger f error=3D"Trajectory failed with err= or: Something went wrong during the trajectory: The FPS got locked during t= he trajectory. Trajectory failed 21.16 seconds after start" 03:45:37 CmdDone: APO.aoravetz2 65 jaeger configuration random --danger =3D=3D=3D=3D=3D=3D *** Configure for a few designs that you might observe if you had a working= telescope, and do a normal fvc loop, make sure stuff seems to be working. 03:52:07 CmdStarted: APO.SeniorCondor 3 jaeger configuration preload 03:52:07 APO.SeniorCondor 3 jaeger > 03:52:07 APO.SeniorCondor 3 jaeger i text=3D"Preloading design 361228.= " 03:52:07 CmdStarted: jaeger.jaeger 1295 cherno get-scale --max-age 3600 03:52:07 jaeger.jaeger 1295 cherno > 03:52:08 jaeger.jaeger 1295 cherno w text=3D"Not enough points to calc= ulate median scale." 03:52:08 jaeger.jaeger 1295 cherno : scale_median=3D-999.0 03:52:08 CmdDone: jaeger.jaeger 1295 cherno get-scale --max-age 3600 03:52:08 APO.SeniorCondor 3 jaeger w text=3D"Invalid guider scale.&quo= t; 03:52:08 APO.SeniorCondor 3 jaeger i text=3D"Focal scale: 0.999816.&qu= ot; 03:52:17 APO.SeniorCondor 3 jaeger e error=3D"Failed retrieving design= : operands could not be broadcast together with shapes (226,10) (226,) &quo= t; 03:52:17 APO.SeniorCondor 3 jaeger f 03:52:17 CmdDone: APO.SeniorCondor 3 jaeger configuration preload 03:54:40 CmdStarted: APO.aoravetz2 72 jaeger configuration load 361228 03:54:40 APO.aoravetz2 72 jaeger > 03:54:40 APO.aoravetz2 72 jaeger i text=3D"Loading design 361228."= ; 03:54:40 CmdStarted: jaeger.jaeger 1296 cherno get-scale --max-age 3600 03:54:40 jaeger.jaeger 1296 cherno > 03:54:40 jaeger.jaeger 1296 cherno w text=3D"Not enough points to calc= ulate median scale." 03:54:40 jaeger.jaeger 1296 cherno : scale_median=3D-999.0 03:54:40 CmdDone: jaeger.jaeger 1296 cherno get-scale --max-age 3600 03:54:40 APO.aoravetz2 72 jaeger w text=3D"Invalid guider scale."= 03:54:40 APO.aoravetz2 72 jaeger i text=3D"Focal scale: 0.999817."= ; 03:54:40 APO.aoravetz2 72 jaeger e error=3D"Failed retrieving design: = operands could not be broadcast together with shapes (226,10) (226,) "= 03:54:40 APO.aoravetz2 72 jaeger f 03:54:40 CmdDone: APO.aoravetz2 72 jaeger configuration load 361228 03:55:13 CmdStarted: APO.SeniorCondor 4 jaeger configuration preload 03:55:13 APO.SeniorCondor 4 jaeger > 03:55:13 APO.SeniorCondor 4 jaeger i text=3D"Preloading design 364229.= " 03:55:13 CmdStarted: jaeger.jaeger 1297 cherno get-scale --max-age 3600 03:55:13 jaeger.jaeger 1297 cherno > 03:55:13 jaeger.jaeger 1297 cherno w text=3D"Not enough points to calc= ulate median scale." 03:55:13 jaeger.jaeger 1297 cherno : scale_median=3D-999.0 03:55:13 CmdDone: jaeger.jaeger 1297 cherno get-scale --max-age 3600 03:55:13 APO.SeniorCondor 4 jaeger w text=3D"Invalid guider scale.&quo= t; 03:55:13 APO.SeniorCondor 4 jaeger i text=3D"Focal scale: 0.999817.&qu= ot; 03:55:19 APO.SeniorCondor 4 jaeger e error=3D"Failed retrieving design= : operands could not be broadcast together with shapes (111,10) (111,) &quo= t; 03:55:19 APO.SeniorCondor 4 jaeger f 03:55:19 CmdDone: APO.SeniorCondor 4 jaeger configuration preload =3D=3D=3D=3D=3D *** Do 4ish manual FVC rotator scans: a. configure robots randomly =93= jaeger configuration random =97danger=94 b. move the rotator in 10 degree= increments between 0-360 (0, 10, 20, =85 340, 350) c. at each rotator position, tak= e 3x =93jaeger fvc loop =97no-apply =97fbi-level 4=94 d. send robots home =93jaeger co= nfiguration reverse" e. repeat steps a-d 4 times.&nbs= p; Every other time offset the rotator positions by 5 degrees (5, 15, 25, =85, 345, 355) f. note the first and last fvc i= mage for each full rotator scan, no need to keep track of every image, I can find the rotator angles from the headers. = I only need to know when you=92ve begun a new scan with different robot positions. ****FIRST ROUND**** jaeger configuration random =97danger tcc track 121,90 mount /rota=3D0, 10, 20... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 ... ROT=3D0 04:00:21 FVC LOOP AND BOSS CALS fvc_rms=3D 177= 05.39 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0001.fits fvc_perc_90=3D= 24636.10 fvc_percent_re= ached=3D 0.00 04:01:17 FVC LOOP AND BOSS CALS fvc_rms=3D 318= .85 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0002.fits fvc_perc_90=3D= 467.10 fvc_percent_re= ached=3D 0.00 04:01:59 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 84 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0003.fits fvc_perc_90=3D= 63.40 fvc_percent_re= ached=3D 2.60 ... ROT=3D350 05:05:52 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 61 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0106.fits fvc_perc_90=3D= 62.30 fvc_percent_re= ached=3D 2.00 05:06:21 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 51 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0107.fits fvc_perc_90=3D= 62.90 fvc_percent_re= ached=3D 2.00 05:06:49 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 47 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0108.fits fvc_perc_90=3D= 62.30 fvc_percent_re= ached=3D 1.40 ROT=3D0 05:10:26 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 72 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0109.fits fvc_perc_90=3D= 61.50 fvc_percent_re= ached=3D 2.00 05:10:55 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 61 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0110.fits fvc_perc_90=3D= 61.80 fvc_percent_re= ached=3D 2.60 05:11:23 FVC LOOP AND BOSS CALS fvc_rms=3D 40.= 58 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0111.fits fvc_perc_90=3D= 61.20 fvc_percent_re= ached=3D 2.00 jaeger configuration reverse ****SECOND ROUND**** fvc images 112-147 were of the same random arrangement as the first round. = Reverse was not issued until a bit later. jaeger configuration random =97danger tcc track 121,90 mount /rota=3D5, 15, 25... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 , etc. etc. ROT=3D5 05:40:40 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 29 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0148.fits fvc_perc_90=3D= 64.80 fvc_percent_re= ached=3D 1.80 05:41:09 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 4 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0149.fits fvc_perc_90=3D= 64.10 fvc_percent_re= ached=3D 1.80 05:41:37 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 13 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0150.fits fvc_perc_90=3D= 63.90 fvc_percent_re= ached=3D 1.80 ROT=3D355 06:41:12 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 96 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0253.fits fvc_perc_90=3D= 64.90 fvc_percent_re= ached=3D 3.60 06:41:41 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 38 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0254.fits fvc_perc_90=3D= 65.30 fvc_percent_re= ached=3D 4.20 06:42:13 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 77 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0255.fits fvc_perc_90=3D= 64.80 fvc_percent_re= ached=3D 4.00 jaeger configuration reverse ****THIRD ROUND**** jaeger configuration random =97danger tcc track 121,90 mount /rota=3D0, 10, 20... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 , etc. etc. ROT=3D0 06:46:38 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 69 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0256.fits fvc_perc_90=3D= 63.50 fvc_percent_re= ached=3D 2.00 06:47:07 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 86 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0257.fits fvc_perc_90=3D= 63.90 fvc_percent_re= ached=3D 1.60 06:47:36 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 71 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0258.fits fvc_perc_90=3D= 64.20 fvc_percent_re= ached=3D 2.20 ROT=3D350 07:47:18 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 34 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0361.fits fvc_perc_90=3D= 63.60 fvc_percent_re= ached=3D 1.40 07:47:47 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 34 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0362.fits fvc_perc_90=3D= 63.30 fvc_percent_re= ached=3D 1.00 07:48:16 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 19 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0363.fits fvc_perc_90=3D= 63.60 fvc_percent_re= ached=3D 1.80 ROT=3D0 07:51:52 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 64 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0364.fits fvc_perc_90=3D= 61.90 fvc_percent_re= ached=3D 2.40 07:52:21 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 4 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0365.fits fvc_perc_90=3D= 62.00 fvc_percent_re= ached=3D 1.80 07:52:49 FVC LOOP AND BOSS CALS fvc_rms=3D 45.= 55 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0366.fits fvc_perc_90=3D= 63.00 fvc_percent_re= ached=3D 2.20 jaeger configuration reverse ****FOURTH ROUND**** jaeger configuration random =97danger tcc track 121,90 mount /rota=3D5, 15, 25... /rotty=3Dmount tcc axis stop jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 jaeger fvc loop --no-apply --fbi-level 4 , etc. etc. ROT=3D5 07:58:25 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 09 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0367.fits fvc_perc_90=3D= 61.90 fvc_percent_re= ached=3D 2.00 07:58:53 FVC LOOP AND BOSS CALS fvc_rms=3D 43.= 86 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0368.fits fvc_perc_90=3D= 62.30 fvc_percent_re= ached=3D 2.00 07:59:22 FVC LOOP AND BOSS CALS fvc_rms=3D 43.= 56 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0369.fits fvc_perc_90=3D= 61.50 fvc_percent_re= ached=3D 2.00 ROT=3D355 08:59:35 FVC LOOP AND BOSS CALS fvc_rms=3D 43.= 88 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0472.fits fvc_perc_90=3D= 61.60 fvc_percent_re= ached=3D 1.60 09:00:04 FVC LOOP AND BOSS CALS fvc_rms=3D 43.= 82 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0473.fits fvc_perc_90=3D= 62.20 fvc_percent_re= ached=3D 2.00 09:00:34 FVC LOOP AND BOSS CALS fvc_rms=3D 44.= 29 fvc_deltarms= =3D None fvc_filename= =3D/data/fcam/60200/fimg-fvc1n-0474.fits fvc_perc_90=3D= 63.10 fvc_percent_re= ached=3D 1.80 jaeger configuration reverse =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= Hooked up the LN2 line, turned off the rotator amplifer, and turned off the= FPS. ---- Mirror Numbers ---- N/A ---- Telescope Status ---- Status at: Telescope stowed at: 121.0, 30.0, 0.0 m= ount Instrument mounted: FPS Counterweights at: 47= 0s LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 250 lb= s, 20 psi ---- Software Used ---- Get Versions, 09/13/23, 04:59:06 -- STUI software: STUI : 1.7.18b1 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.4.0 jaeger : 1.4.1a0 kaiju : 1.3.1 coordio : 1.7.1 fps_calibrati : 2023.07.27 cherno : 0.6.3 astrometrynet : 0.93 hal : 0.6.0b5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.3.2 -- done -- --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM = : 1.2-31-gd7eb MCP = : 2.1-1-ge461 PLC = : V35 Fiducials : v1_= 149 idlspec2d : v6_= 1_0 Kronos &nb= sp; : 1.6.2a roboscheduler : 1.4.0 /dev/sdb1 = 2.0T 290G 1.8T 15% /home /dev/sdd1 = 22T 8.9T 12T 43% /da= ta ---- Time Tracking ---- Wed 13 Sep 2023 60201 0.01 02:05 02:20 11:37 18:48 04:07 --- dark ( 9.3 h) = plus 0.2 h for 12 deg design *=3D*=3D*=3D Observatory: APO *=3D*=3D*=3D MJD: 60200 Fill night time in Hours below *=3D*=3D*=3D Total bright time &nb= sp; = : 0 *=3D*=3D*=3D Bright time, queued observations = : 0 *=3D*=3D*=3D Bright time, commissioning/engineering: 0 *=3D*=3D*=3D Bright time lost to weather  = ; : 0 *=3D*=3D*=3D Bright time lost to technical issues : 0 *=3D*=3D*=3D Total dark time  = ; &n= bsp; : 9.3 *=3D*=3D*=3D Dark time, queued observations &n= bsp; : 0 *=3D*=3D*=3D Dark time, commissioning/engineering : 0 *=3D*=3D*=3D Dark time lost to weather &= nbsp; : 9.3 *=3D*=3D*=3D Dark time lost to technical issues : 0 --_000_PH7PR16MB5216B57CF9741EA2F24815B2FEF0APH7PR16MB5216namp_--