id3c.plans.flyscan_3idc#

Fly scan for 3-ID-C: area detector vs. motor.

Fly scan an EPICS motor and collect Eiger2 (or any AreaDetector) images.

Implementation note: this plan is software-correlated (no hardware gate signal). Frame-to-position pairing happens downstream from the monitor_during_decorator streams, joined by IOC timestamp. Waits inside the plan use ophyd Status objects (MoveStatus, SubscriptionStatus, AndStatus) driven by CA monitor callbacks rather than busy-poll loops. See flyscan-3idc-status- strategy.md (alongside this file) for the design history.

Usage from a command-line session:

from bits2606.startup import *           # provides RE, oregistry, etc.
from flyscan_3idc import flyscan, configure_adsimdet

# The plan: drive a fly scan and record a Bluesky run.
uid, = RE(
    flyscan(
        det_name: str = "eiger2",
        flymotor_name: str = "sample_stage.omega",
        p_start=5, p_end=10,
        exposures_per_egu=10,  # approximate
        t_period=0.05,
    )
)

# The standalone diagnostic: exercise the AD acquisition
# protocol (no plan, no RunEngine) for triage.
result = configure_adsimdet(adsimdet, capture_duration=3.0)

General outline#

Preparation 1. Validate input parameters 2. Collect metadata 3. Snapshot mutable state (stage_sigs, kind,

overridden signal values) for later restore
1. concurrently (motor moves while parameters are set) 1. send motor to initial position (``bps.abs_set(…,
  
  group=”taxi”)``)
  1. set most parameters for scan
  2. wait for motor to reach taxi position (bps.wait(group="taxi"))
2. set motor velocity for scan
Kickoff 1. stage devices 2. open run 3. subscribe to bespoke monitor streams (one stream per signal):
1. HDF writer’s frame count (det.hdf1.array_counter, carries EPICS timestamp for downstream sync with flymotor)
2. camera’s frame count (det.cam.array_counter, for cam-vs-HDF latency comparison)
3. motor position (flymotor.user_readback)
1. send motor to final position (bps.abs_set(..., group="scan"))
2. start detector acquiring (without a hardware gate signal, coordinating acquire-start to a specific motor position is extremely difficult; instead, acquire continuously for the entire span p_initial <= flymotor.position <= p_final and let downstream analysis select frames in [p_start, p_end] by timestamp/position). This is an intentional oversample.
3. build status objects for the monitor stage: - cam_stopped_status: cam.acquire == 0 - drain_status: HDF queue empty and idle - ``hdf_drain_status = AndStatus(cam_stopped_status,
  
  drain_status)`` — scan is done when both
  - watchdog_status: num_captured > 0 with timeout=no_frames_timeout
Monitor (monitor_loop) 1. CA monitor callback on det.hdf1.num_captured pushes

(timestamp, value) onto a bounded queue (the producer)
1. plan-side consumer wakes every _consumer_tick seconds, drains the queue, and emits one primary event per newly-captured frame (create / read(det) / read(flymotor) / save). bps.read returns cached monitor values — no extra CA traffic
2. stop detector acquiring when motor crosses p_end
3. raise RuntimeError if the watchdog times out without any frame arriving (RE will then STOP all in-motion movables)
4. exit when hdf_drain_status.done (cam has stopped AND every in-flight frame has been flushed)
5. after exit, bps.wait(group="scan") absorbs any motor settling past p_final
Conclusion (_cleanup) 1. stop motor (if still moving — checked via motor_is_moving) 2. stop cam acquire 3. stop hdf1 capture 4. wait for cam idle AND HDF queue drained

(wait_for_acquire_drained — uses AndStatus of SubscriptionStatus per signal)
1. verify the HDF5 file landed (full_file_name)
2. restore overridden signal values from CacheParameters, restore mutated stage_sigs dicts, restore mutated kind values
3. close run (handled by run_decorator)
4. unstage devices (handled by stage_decorator)

Functions#

`configure_adsimdet`(det, *[, ad_file_path, ...])	Configure & exercise an AD HDF5 detector without a plan.
`flyscan`([det_name, flymotor_name, p_start, p_end, ...])	Fly scan: move motor through range continuously acquiring detector frames.

Module Contents#

id3c.plans.flyscan_3idc.configure_adsimdet(det, *, ad_file_path='/tmp/flyscan/', ad_file_name='flyscan', ad_file_template='%s%s_%6.6d.h5', ad_file_number=1, acquire_time=0.02, acquire_period=0.1, capture_duration=2.0, num_capture=None, capture_arm_timeout=5.0, drain_timeout=10.0, do_capture=True, do_acquire=True)[source]#

Configure & exercise an AD HDF5 detector without a plan.

Diagnostic helper. No RunEngine, no plan, no stage_decorator — just straight ophyd put() calls in the order the IOC needs them.

Simulates the flyscan acquisition protocol:

Configure file destination & cam timings.

Cam in Continuous image_mode.

num_capture = UNLIMITED_FRAMES (capture until told to stop).

Arm capture (hdf1.capture.put(1)).

Wait for Capture_RBV == 'Capturing' — this avoids a race in which the cam starts producing frames before the HDF plugin is ready to receive them. Without this wait, the leading frames of a scan are silently dropped (not counted in dropped_arrays because the plugin isn’t even listening yet).

Start cam acquire.

Sleep capture_duration seconds (simulates the motor trajectory window in a real flyscan).

Stop capture (hdf1.capture.put(0)).

Drain: wait until num_queued_arrays == 0 so all in-flight frames flush to disk before the file is closed.

Stop cam acquire.

Snapshot relevant PVs and return.

Returns a dict of the post-operation PV snapshot.

Usage:

from flyscan_3idc import configure_adsimdet
result = configure_adsimdet(adsimdet, capture_duration=3.0)
for k, v in result.items():
    print(f"  {k}: {v}")

Parameters:

capture_duration (float) – Seconds to leave both capture and acquire active. Total file write count is approximately capture_duration / acquire_period.
capture_arm_timeout (float) – Maximum seconds to wait for Capture_RBV to transition to 'Capturing' after arming. Raises RuntimeError on timeout.
drain_timeout (float) – Maximum seconds to wait for num_queued_arrays to reach 0 after stopping capture. Logs a warning on timeout but does not raise.
do_capture (bool) – Skip arming capture or starting acquire, respectively. Useful for narrowing down which step misbehaves.
do_acquire (bool) – Skip arming capture or starting acquire, respectively. Useful for narrowing down which step misbehaves.

id3c.plans.flyscan_3idc.flyscan(det_name: str = 'adsimdet', flymotor_name: str = 'm1', p_start: float = 0, p_end: float = 5, exposures_per_egu: float = 2.0, t_period: float = 0.1, t_acquire: float = None, taxi_allowance: float = 0.5, compression: str = 'zlib', ad_file_name: str = 'flyscan', ad_file_path: str = '/tmp/flyscan', _consumer_tick: float = _CONSUMER_TICK_DEFAULT, _force_hdf_nonblocking: bool = False, md: dict = None)[source]#

Fly scan: move motor through range continuously acquiring detector frames.

The motor traverses p_initial → p_final, maintaining constant velocity between p_start → p_end to deliver num_frames frames within [p_start, p_end]. p_initial and p_final are computed from p_start, p_end, the motor’s .ACCL, and taxi_allowance; num_frames is computed from (p_end - p_start) * exposures_per_egu. Detector frames are acquired continuously during the traverse; downstream processing trims the data to [p_start, p_end] by motor position. An HDF5 file containing every captured frame is written next to the run (the path is in the run metadata under ad_file_path / ad_file_name).

Position geometry#

User-supplied: p_start and p_end (in-scan range). Derived: p_initial (parked, pre-scan) and p_final (coast, post-scan):

p_initial  <  p_start  <  p_end  <  p_final
|             |          |          |
|             |--scan----|          |
|--taxi-in---|           |---coast--|

p_start: the position at which the first useful frame should be captured. Downstream processing trims frames captured before this point.
p_end: the position at which the last useful frame should be captured. The plan stops the cam when the motor passes this point.
p_initial (derived): where the motor is parked before the scan, far enough below p_start that the motor reaches its scan velocity before it enters the acquisition region. Computed as p_start - d_taxi - taxi_allowance where d_taxi = 0.5 * scan_velocity * motor.ACCL.
p_final (derived): where the motor coasts to after the scan ends — far enough above p_end that the cam can finish processing its last frames before the motor stops. Computed symmetric to p_initial.

taxi_allowance (default 0.5, in motor EGU) is added to both ends as a slack margin on top of the acceleration-based distance. Increase it if the cam’s first/last frame is observed to fall outside [p_start, p_end]; decrease it if the scan takes too long to taxi.

Position units are whatever the motor reports (user_readback); typically engineering units (mm, degrees, etc.) — the motor’s .EGU field is recorded in run metadata.

Frame timing#

exposures_per_egu: target frame density. Combined with the scan range, gives num_frames = round(1 + (p_end - p_start) * exposures_per_egu) (fence-post counting: one frame at each endpoint plus exposures_per_egu frames per unit between).
t_period: seconds between successive frame exposures.
t_acquire: per-frame exposure time, in seconds. Defaults to t_period (continuous exposure). Must satisfy 0 < t_acquire <= t_period.

The scan velocity is computed as (p_end - p_start) / (num_frames * t_period). Pre-scan validation rejects velocities outside the motor’s .VBAS / .VMAX limits with a clear ValueError.

Detector & file#

det_name: ophyd device registry key for the area detector (default "adsimdet"). Must be an AreaDetector with an HDF5 plugin attached.
flymotor_name: ophyd device registry key for the motor (default "m1").
compression: HDF5 chunk compression name (default "zlib"). Validated against the HDF plugin’s compression.enum_strs at scan start; raises ValueError with the allowed list if the value isn’t supported by the IOC’s HDF plugin build.
ad_file_name: stem for the saved HDF5 file (default "flyscan"); the IOC appends an auto-incrementing number and the .h5 extension.
ad_file_path: directory on the IOC’s filesystem where the HDF5 file is written (default "/tmp/flyscan"). Must exist on the IOC’s filesystem. If the IOC runs in a container, this is the container’s view of the path, not the host’s. The plan checks this before staging and raises RuntimeError with a clear message if the path doesn’t exist.

What gets recorded#

Each call to RE(flyscan(...)) produces one bluesky run containing:

A primary event stream with one event per HDF frame accepted by the writer. Each event records the cam and HDF array counters and the motor’s reported position at the moment the consumer drained that frame from its queue. Treat this as a progress indicator and at-the-bench snapshot; use the monitor streams below for high-precision pairing.
Three monitor streams (adsimdet_cam_array_counter_monitor, adsimdet_hdf1_array_counter_monitor, m1_monitor) carrying IOC-timestamped values for downstream synchronization of frame counters with motor position.
A baseline stream (whatever apsbits configures).
Metadata under start: user-supplied scan parameters (p_start, p_end, exposures_per_egu, t_period, t_acquire, taxi_allowance, compression), derived geometry (p_initial, p_final, num_frames, scan_velocity, d_taxi, motor_accl, motor_egu), motor velocity limits, file destination, watchdog timeout, consumer_tick, plus anything you pass in md.
An HDF5 file with the actual image data at ad_file_path/ad_file_name_NNNNNN.h5.

Common usage#

From a bits2606 IPython session:

from bits2606.startup import *           # provides RE, oregistry
from flyscan_3idc import flyscan

# 50 frames over a 5-EGU range at 20 Hz:
uid, = RE(flyscan(p_start=0, p_end=5, exposures_per_egu=10,
                  t_period=0.05))

Override more defaults for a specific run:

uid, = RE(flyscan(
    flymotor_name="m1",
    p_start=0, p_end=10,
    exposures_per_egu=10, t_period=0.05, t_acquire=0.01,
    taxi_allowance=1.0,
    compression="lz4",
    ad_file_path="/tmp/myexperiment/",
    ad_file_name="sample42",
    md={"sample": "Ag behenate", "operator": "your-name"},
))

Common pitfalls#

“file_path does not exist” RuntimeError at scan start. The directory in ad_file_path doesn’t exist on the IOC’s filesystem. If the IOC is containerized, create the directory inside the container or use a path that’s visible there.
“scan_velocity exceeds motor max velocity” ValueError. The requested combination of position range and frame rate would require the motor to move faster than its .VMAX. Either reduce exposures_per_egu, increase t_period, or shorten p_end - p_start.
“compression=… not in HDF plugin’s allowed set” ValueError. The IOC’s HDF plugin doesn’t support the requested compression algorithm. Inspect det.hdf1.compression.enum_strs to see what is supported by this IOC build.
Watchdog: “no frames captured” RuntimeError mid-scan. The cam isn’t delivering frames to the HDF plugin. Likely the HDF plugin’s EnableCallbacks is Disable, the cam’s ArrayCallbacks is Disable, or the HDF plugin’s NDArrayPort doesn’t point at the cam. The RunEngine will have stopped the motor; investigate the IOC and try again.
The scan completes but the data dictionary’s ``num_captured`` is 0. The IOC resets NumCaptured_RBV to 0 after the HDF5 file is closed. Look at full_file_name (in _cleanup’s log line) and the actual file on disk to confirm what was saved.
“HDF plugin dropped N frame(s) during this run” FlyscanDataLossWarning at scan end. The HDF plugin couldn’t keep up with the cam at the requested rate, and N frames the cam produced are missing from the on-disk HDF5 file. The warning is emitted both to the log (WARNING level) and via Python’s warnings machinery (subclass of UserWarning). The plan uses blocking_callbacks="Yes" on the HDF plugin to throttle the cam to HDF’s write rate, so this should be rare — when it does occur, it usually means the cam emitted a burst before back-pressure propagated, or the HDF queue size is too small. Treat N > 0 as a data-integrity concern: increase t_period or reduce exposures_per_egu. Promote the warning to an exception with warnings.filterwarnings("error", category=flyscan_3idc.FlyscanDataLossWarning) to fail-fast in strict environments.

param det_name:: ophyd registry name of the area detector to fly.
type det_name:: str, default "adsimdet"
param flymotor_name:: ophyd registry name of the motor to fly.
type flymotor_name:: str, default "m1"
param p_start:: First in-scan position (motor units).
type p_start:: float, default 0
param p_end:: Last in-scan position (motor units).
type p_end:: float, default 5
param exposures_per_egu:: Frame density: frames per motor engineering unit. Total frame count is round(1 + (p_end - p_start) * exposures_per_egu). Must be positive.
type exposures_per_egu:: float, default 2.0
param t_period:: Time between successive frame exposures (seconds).
type t_period:: float, default 0.1
param t_acquire:: Per-frame exposure time (seconds). None (default) means “use t_period” (continuous exposure). Must satisfy 0 < t_acquire <= t_period.
type t_acquire:: float or None, default None
param taxi_allowance:: Extra distance (in motor EGU) added past the acceleration-based taxi region at each end of the scan. Increase if the first/last useful frame falls outside [p_start, p_end]; must be non-negative.
type taxi_allowance:: float, default 0.5
param compression:: HDF5 chunk compression name. Must match one of det.hdf1.compression.enum_strs if the IOC is reachable.
type compression:: str, default "zlib"
param ad_file_name:: HDF5 filename stem (IOC appends a number and .h5).
type ad_file_name:: str, default "flyscan"
param ad_file_path:: Directory on the IOC’s filesystem to write the HDF5 file.
type ad_file_path:: str, default "/tmp/flyscan"
param _consumer_tick:: Internal: wake-up tick for the per-frame event consumer. Increase if your run-engine subscriptions can’t keep up; decrease only for very high frame rates. Rarely needs to be changed.
type _consumer_tick:: float, default _CONSUMER_TICK_DEFAULT (20 ms)
param md:: Additional metadata to record under the run’s start document. Merged on top of the plan’s computed metadata.
type md:: dict, optional
rtype:: None (yields bluesky messages — pass to RE() to execute).
raises KeyError:: det_name or flymotor_name does not resolve to the expected ophyd device type in the registry.
raises ValueError:: Position ordering is wrong (p_end <= p_start), exposures_per_egu is non-positive, taxi_allowance is negative, t_acquire > t_period, computed num_frames is too small, computed scan_velocity is outside the motor’s limits, or compression is not in the HDF plugin’s enumeration.
raises RuntimeError:: IOC preflight failed (an expected PV did not connect), or the HDF plugin’s file path does not exist on the IOC’s filesystem, or the no-frames watchdog tripped during the scan.