id3c.plans.flyscan_3idc ======================= .. py:module:: id3c.plans.flyscan_3idc .. autoapi-nested-parse:: Fly scan for 3-ID-C: area detector *vs*. motor. Fly scan an EPICS motor and collect Eiger2 (or any AreaDetector) images. Implementation note: this plan is software-correlated (no hardware gate signal). Frame-to-position pairing happens downstream from the ``monitor_during_decorator`` streams, joined by IOC timestamp. Waits inside the plan use ophyd ``Status`` objects (``MoveStatus``, ``SubscriptionStatus``, ``AndStatus``) driven by CA monitor callbacks rather than busy-poll loops. See ``flyscan-3idc-status- strategy.md`` (alongside this file) for the design history. Usage from a command-line session:: from bits2606.startup import * # provides RE, oregistry, etc. from flyscan_3idc import flyscan, configure_adsimdet # The plan: drive a fly scan and record a Bluesky run. uid, = RE( flyscan( det_name: str = "eiger2", flymotor_name: str = "sample_stage.omega", p_start=5, p_end=10, exposures_per_egu=10, # approximate t_period=0.05, ) ) # The standalone diagnostic: exercise the AD acquisition # protocol (no plan, no RunEngine) for triage. result = configure_adsimdet(adsimdet, capture_duration=3.0) General outline --------------- 1. Preparation 1. Validate input parameters 2. Collect metadata 3. Snapshot mutable state (``stage_sigs``, ``kind``, overridden signal values) for later restore 4. concurrently (motor moves while parameters are set) 1. send motor to *initial* position (``bps.abs_set(..., group="taxi")``) 2. set *most* parameters for scan 3. wait for motor to reach taxi position (``bps.wait(group="taxi")``) 5. set motor velocity for scan 2. Kickoff 1. stage devices 2. open run 3. subscribe to bespoke monitor streams (one stream per signal): 1. HDF writer's frame count (``det.hdf1.array_counter``, carries EPICS timestamp for downstream sync with flymotor) 2. camera's frame count (``det.cam.array_counter``, for cam-vs-HDF latency comparison) 3. motor position (``flymotor.user_readback``) 4. send motor to *final* position (``bps.abs_set(..., group="scan")``) 5. start detector acquiring (without a hardware gate signal, coordinating acquire-start to a specific motor position is extremely difficult; instead, acquire **continuously** for the entire span ``p_initial <= flymotor.position <= p_final`` and let downstream analysis select frames in ``[p_start, p_end]`` by timestamp/position). This is an intentional oversample. 6. build status objects for the monitor stage: - ``cam_stopped_status``: ``cam.acquire == 0`` - ``drain_status``: HDF queue empty and idle - ``hdf_drain_status = AndStatus(cam_stopped_status, drain_status)`` — scan is done when both - ``watchdog_status``: ``num_captured > 0`` with ``timeout=no_frames_timeout`` 3. Monitor (``monitor_loop``) 1. CA monitor callback on ``det.hdf1.num_captured`` pushes ``(timestamp, value)`` onto a bounded queue (the *producer*) 2. plan-side *consumer* wakes every ``_consumer_tick`` seconds, drains the queue, and emits one ``primary`` event per newly-captured frame (``create / read(det) / read(flymotor) / save``). ``bps.read`` returns cached monitor values — no extra CA traffic 3. stop detector acquiring when motor crosses *p_end* 4. raise ``RuntimeError`` if the watchdog times out without any frame arriving (RE will then STOP all in-motion movables) 5. exit when ``hdf_drain_status.done`` (cam has stopped AND every in-flight frame has been flushed) 6. after exit, ``bps.wait(group="scan")`` absorbs any motor settling past *p_final* 4. Conclusion (``_cleanup``) 1. stop motor (if still moving — checked via ``motor_is_moving``) 2. stop cam acquire 3. stop hdf1 capture 4. wait for cam idle AND HDF queue drained (``wait_for_acquire_drained`` — uses ``AndStatus`` of ``SubscriptionStatus`` per signal) 5. verify the HDF5 file landed (``full_file_name``) 6. restore overridden signal values from ``CacheParameters``, restore mutated ``stage_sigs`` dicts, restore mutated ``kind`` values 7. close run (handled by ``run_decorator``) 8. unstage devices (handled by ``stage_decorator``) Functions --------- .. autoapisummary:: id3c.plans.flyscan_3idc.configure_adsimdet id3c.plans.flyscan_3idc.flyscan Module Contents --------------- .. py:function:: configure_adsimdet(det, *, ad_file_path='/tmp/flyscan/', ad_file_name='flyscan', ad_file_template='%s%s_%6.6d.h5', ad_file_number=1, acquire_time=0.02, acquire_period=0.1, capture_duration=2.0, num_capture=None, capture_arm_timeout=5.0, drain_timeout=10.0, do_capture=True, do_acquire=True) Configure & exercise an AD HDF5 detector without a plan. Diagnostic helper. No RunEngine, no plan, no stage_decorator — just straight ophyd ``put()`` calls in the order the IOC needs them. Simulates the flyscan acquisition protocol: 1. Configure file destination & cam timings. 2. Cam in ``Continuous`` image_mode. 3. ``num_capture = UNLIMITED_FRAMES`` (capture until told to stop). 4. Arm capture (``hdf1.capture.put(1)``). 5. **Wait** for ``Capture_RBV == 'Capturing'`` — this avoids a race in which the cam starts producing frames before the HDF plugin is ready to receive them. Without this wait, the leading frames of a scan are silently dropped (not counted in ``dropped_arrays`` because the plugin isn't even listening yet). 6. Start cam acquire. 7. Sleep ``capture_duration`` seconds (simulates the motor trajectory window in a real flyscan). 8. Stop capture (``hdf1.capture.put(0)``). 9. Drain: wait until ``num_queued_arrays == 0`` so all in-flight frames flush to disk before the file is closed. 10. Stop cam acquire. 11. Snapshot relevant PVs and return. Returns a dict of the post-operation PV snapshot. Usage:: from flyscan_3idc import configure_adsimdet result = configure_adsimdet(adsimdet, capture_duration=3.0) for k, v in result.items(): print(f" {k}: {v}") :param capture_duration: Seconds to leave both capture and acquire active. Total file write count is approximately ``capture_duration / acquire_period``. :type capture_duration: float :param capture_arm_timeout: Maximum seconds to wait for ``Capture_RBV`` to transition to ``'Capturing'`` after arming. Raises ``RuntimeError`` on timeout. :type capture_arm_timeout: float :param drain_timeout: Maximum seconds to wait for ``num_queued_arrays`` to reach 0 after stopping capture. Logs a warning on timeout but does not raise. :type drain_timeout: float :param do_capture: Skip arming capture or starting acquire, respectively. Useful for narrowing down which step misbehaves. :type do_capture: bool :param do_acquire: Skip arming capture or starting acquire, respectively. Useful for narrowing down which step misbehaves. :type do_acquire: bool .. py:function:: flyscan(det_name: str = 'adsimdet', flymotor_name: str = 'm1', p_start: float = 0, p_end: float = 5, exposures_per_egu: float = 2.0, t_period: float = 0.1, t_acquire: float = None, taxi_allowance: float = 0.5, compression: str = 'zlib', ad_file_name: str = 'flyscan', ad_file_path: str = '/tmp/flyscan', _consumer_tick: float = _CONSUMER_TICK_DEFAULT, _force_hdf_nonblocking: bool = False, md: dict = None) Fly scan: move motor through range continuously acquiring detector frames. The motor traverses ``p_initial → p_final``, maintaining constant velocity between ``p_start → p_end`` to deliver ``num_frames`` frames within ``[p_start, p_end]``. ``p_initial`` and ``p_final`` are computed from ``p_start``, ``p_end``, the motor's ``.ACCL``, and ``taxi_allowance``; ``num_frames`` is computed from ``(p_end - p_start) * exposures_per_egu``. Detector frames are acquired continuously during the traverse; downstream processing trims the data to ``[p_start, p_end]`` by motor position. An HDF5 file containing every captured frame is written next to the run (the path is in the run metadata under ``ad_file_path`` / ``ad_file_name``). Position geometry ----------------- User-supplied: ``p_start`` and ``p_end`` (in-scan range). Derived: ``p_initial`` (parked, pre-scan) and ``p_final`` (coast, post-scan):: p_initial < p_start < p_end < p_final | | | | | |--scan----| | |--taxi-in---| |---coast--| - ``p_start``: the position at which the first useful frame should be captured. Downstream processing trims frames captured before this point. - ``p_end``: the position at which the last useful frame should be captured. The plan stops the cam when the motor passes this point. - ``p_initial`` (derived): where the motor is parked before the scan, far enough below ``p_start`` that the motor reaches its scan velocity *before* it enters the acquisition region. Computed as ``p_start - d_taxi - taxi_allowance`` where ``d_taxi = 0.5 * scan_velocity * motor.ACCL``. - ``p_final`` (derived): where the motor coasts to after the scan ends — far enough above ``p_end`` that the cam can finish processing its last frames before the motor stops. Computed symmetric to ``p_initial``. ``taxi_allowance`` (default ``0.5``, in motor EGU) is added to both ends as a slack margin on top of the acceleration-based distance. Increase it if the cam's first/last frame is observed to fall outside ``[p_start, p_end]``; decrease it if the scan takes too long to taxi. Position units are whatever the motor reports (``user_readback``); typically engineering units (mm, degrees, etc.) — the motor's ``.EGU`` field is recorded in run metadata. Frame timing ------------ - ``exposures_per_egu``: target frame density. Combined with the scan range, gives ``num_frames = round(1 + (p_end - p_start) * exposures_per_egu)`` (fence-post counting: one frame at each endpoint plus ``exposures_per_egu`` frames per unit between). - ``t_period``: seconds between successive frame exposures. - ``t_acquire``: per-frame exposure time, in seconds. Defaults to ``t_period`` (continuous exposure). Must satisfy ``0 < t_acquire <= t_period``. The scan velocity is computed as ``(p_end - p_start) / (num_frames * t_period)``. Pre-scan validation rejects velocities outside the motor's ``.VBAS`` / ``.VMAX`` limits with a clear ``ValueError``. Detector & file --------------- - ``det_name``: ophyd device registry key for the area detector (default ``"adsimdet"``). Must be an AreaDetector with an HDF5 plugin attached. - ``flymotor_name``: ophyd device registry key for the motor (default ``"m1"``). - ``compression``: HDF5 chunk compression name (default ``"zlib"``). Validated against the HDF plugin's ``compression.enum_strs`` at scan start; raises ``ValueError`` with the allowed list if the value isn't supported by the IOC's HDF plugin build. - ``ad_file_name``: stem for the saved HDF5 file (default ``"flyscan"``); the IOC appends an auto-incrementing number and the ``.h5`` extension. - ``ad_file_path``: directory on the IOC's filesystem where the HDF5 file is written (default ``"/tmp/flyscan"``). **Must exist on the IOC's filesystem.** If the IOC runs in a container, this is the container's view of the path, not the host's. The plan checks this before staging and raises ``RuntimeError`` with a clear message if the path doesn't exist. What gets recorded ------------------ Each call to ``RE(flyscan(...))`` produces one bluesky run containing: - A ``primary`` event stream with one event per HDF frame accepted by the writer. Each event records the cam and HDF array counters and the motor's reported position at the moment the consumer drained that frame from its queue. Treat this as a progress indicator and at-the-bench snapshot; use the monitor streams below for high-precision pairing. - Three monitor streams (``adsimdet_cam_array_counter_monitor``, ``adsimdet_hdf1_array_counter_monitor``, ``m1_monitor``) carrying IOC-timestamped values for downstream synchronization of frame counters with motor position. - A ``baseline`` stream (whatever ``apsbits`` configures). - Metadata under ``start``: user-supplied scan parameters (``p_start``, ``p_end``, ``exposures_per_egu``, ``t_period``, ``t_acquire``, ``taxi_allowance``, ``compression``), derived geometry (``p_initial``, ``p_final``, ``num_frames``, ``scan_velocity``, ``d_taxi``, ``motor_accl``, ``motor_egu``), motor velocity limits, file destination, watchdog timeout, ``consumer_tick``, plus anything you pass in ``md``. - An HDF5 file with the actual image data at ``ad_file_path/ad_file_name_NNNNNN.h5``. Common usage ------------ From a bits2606 IPython session:: from bits2606.startup import * # provides RE, oregistry from flyscan_3idc import flyscan # 50 frames over a 5-EGU range at 20 Hz: uid, = RE(flyscan(p_start=0, p_end=5, exposures_per_egu=10, t_period=0.05)) Override more defaults for a specific run:: uid, = RE(flyscan( flymotor_name="m1", p_start=0, p_end=10, exposures_per_egu=10, t_period=0.05, t_acquire=0.01, taxi_allowance=1.0, compression="lz4", ad_file_path="/tmp/myexperiment/", ad_file_name="sample42", md={"sample": "Ag behenate", "operator": "your-name"}, )) Common pitfalls --------------- - **"file_path does not exist" RuntimeError at scan start.** The directory in ``ad_file_path`` doesn't exist on the IOC's filesystem. If the IOC is containerized, create the directory inside the container or use a path that's visible there. - **"scan_velocity exceeds motor max velocity" ValueError.** The requested combination of position range and frame rate would require the motor to move faster than its ``.VMAX``. Either reduce ``exposures_per_egu``, increase ``t_period``, or shorten ``p_end - p_start``. - **"compression=... not in HDF plugin's allowed set" ValueError.** The IOC's HDF plugin doesn't support the requested compression algorithm. Inspect ``det.hdf1.compression.enum_strs`` to see what *is* supported by this IOC build. - **Watchdog: "no frames captured" RuntimeError mid-scan.** The cam isn't delivering frames to the HDF plugin. Likely the HDF plugin's ``EnableCallbacks`` is ``Disable``, the cam's ``ArrayCallbacks`` is ``Disable``, or the HDF plugin's ``NDArrayPort`` doesn't point at the cam. The RunEngine will have stopped the motor; investigate the IOC and try again. - **The scan completes but the data dictionary's ``num_captured`` is 0.** The IOC resets ``NumCaptured_RBV`` to 0 after the HDF5 file is closed. Look at ``full_file_name`` (in ``_cleanup``'s log line) and the actual file on disk to confirm what was saved. - **"HDF plugin dropped N frame(s) during this run" FlyscanDataLossWarning at scan end.** The HDF plugin couldn't keep up with the cam at the requested rate, and ``N`` frames the cam produced are missing from the on-disk HDF5 file. The warning is emitted both to the log (WARNING level) and via Python's ``warnings`` machinery (subclass of ``UserWarning``). The plan uses ``blocking_callbacks="Yes"`` on the HDF plugin to throttle the cam to HDF's write rate, so this should be rare — when it does occur, it usually means the cam emitted a burst before back-pressure propagated, or the HDF queue size is too small. Treat ``N > 0`` as a data-integrity concern: increase ``t_period`` or reduce ``exposures_per_egu``. Promote the warning to an exception with ``warnings.filterwarnings("error", category=flyscan_3idc.FlyscanDataLossWarning)`` to fail-fast in strict environments. :param det_name: ophyd registry name of the area detector to fly. :type det_name: str, default ``"adsimdet"`` :param flymotor_name: ophyd registry name of the motor to fly. :type flymotor_name: str, default ``"m1"`` :param p_start: First in-scan position (motor units). :type p_start: float, default ``0`` :param p_end: Last in-scan position (motor units). :type p_end: float, default ``5`` :param exposures_per_egu: Frame density: frames per motor engineering unit. Total frame count is ``round(1 + (p_end - p_start) * exposures_per_egu)``. Must be positive. :type exposures_per_egu: float, default ``2.0`` :param t_period: Time between successive frame exposures (seconds). :type t_period: float, default ``0.1`` :param t_acquire: Per-frame exposure time (seconds). ``None`` (default) means "use ``t_period``" (continuous exposure). Must satisfy ``0 < t_acquire <= t_period``. :type t_acquire: float or None, default ``None`` :param taxi_allowance: Extra distance (in motor EGU) added past the acceleration-based taxi region at each end of the scan. Increase if the first/last useful frame falls outside ``[p_start, p_end]``; must be non-negative. :type taxi_allowance: float, default ``0.5`` :param compression: HDF5 chunk compression name. Must match one of ``det.hdf1.compression.enum_strs`` if the IOC is reachable. :type compression: str, default ``"zlib"`` :param ad_file_name: HDF5 filename stem (IOC appends a number and ``.h5``). :type ad_file_name: str, default ``"flyscan"`` :param ad_file_path: Directory on the IOC's filesystem to write the HDF5 file. :type ad_file_path: str, default ``"/tmp/flyscan"`` :param _consumer_tick: Internal: wake-up tick for the per-frame event consumer. Increase if your run-engine subscriptions can't keep up; decrease only for very high frame rates. Rarely needs to be changed. :type _consumer_tick: float, default ``_CONSUMER_TICK_DEFAULT`` (20 ms) :param md: Additional metadata to record under the run's ``start`` document. Merged on top of the plan's computed metadata. :type md: dict, optional :rtype: None (yields bluesky messages — pass to ``RE()`` to execute). :raises KeyError: ``det_name`` or ``flymotor_name`` does not resolve to the expected ophyd device type in the registry. :raises ValueError: Position ordering is wrong (``p_end <= p_start``), ``exposures_per_egu`` is non-positive, ``taxi_allowance`` is negative, ``t_acquire > t_period``, computed ``num_frames`` is too small, computed ``scan_velocity`` is outside the motor's limits, or ``compression`` is not in the HDF plugin's enumeration. :raises RuntimeError: IOC preflight failed (an expected PV did not connect), or the HDF plugin's file path does not exist on the IOC's filesystem, or the no-frames watchdog tripped during the scan. .. seealso:: :py:obj:`configure_adsimdet` Standalone diagnostic that exercises the same AD acquisition protocol without a plan or RunEngine. Useful for triaging an IOC that's misbehaving. :py:obj:`compute_flyscan_geometry` Pure-function helper that derives ``p_initial``, ``p_final``, and ``num_frames`` from the user-supplied kwargs; unit- testable without an IOC.