synApps sscan as 1D Flyer#

In this notebook, the EPICS sscan record is demonstrated as a bluesky Flyer. The EPICS sscan record will conduct a scan as configured from bluesky. After the scan, the data will be returned to bluesky for routine inclusion in the run’s documents.

NOTE: This notebook is under construction!

Overview#

The Bluesky framework has support for data acquisition using an external data collector. The support is for an ophyd fly-able device, which must be customized for the details of that external data collector. The design prototype collector is a fly scan controller (hence the name used by bluesky), operating outside the bluesky process, which performs its readings of detector(s) during continuous (on-the-fly) motion of one or more positioners. Triggering of the detectors must also be part of the external controller. The description is that this type of asynchronous acquisition requires fine-grained timing beyond the capabilities of bluesky.

Flying means: “Let the hardware take control, cache data externally, and then transfer all the data to the RunEngine at the end.” This is essential when the data acquisition rates are faster than the RunEngine or Python can go.

Note: As a consequence of the external nature of this Flyer device, the usual activity of the bluesky RunEngine pertaining to interruptions (pause, resume, stop, or abort) may not affect the progress of the external data collector. The RunEngine may only interrupt the external data collector as that collector allows in its interface with bluesky.

Any external data acquisition controller or data logger is suitable for use as a bluesky Flyer. It is not necessary for the external controller to conduct its acquisition at high rates.

The EPICS sscan record is an example of an external data acquisition controller. It supports various scan types (step, list, continuous) with flexible configuration for up to four positioners and dozens of detectors, as well as scans of up to four dimensions. The sscan record is used by many APS beamlines for routine data collection.

References#

The documentation for Flyers is distributed across the bluesky and ophyd packages:

Use the sscan record as an ophyd Flyer for data acquisition with the bluesky fly plan. Consider the case of 1D step scans using sscan record.

Suggest the noisy v m1 scan, done as 1-D step scan with sscan record, where noisy is the swait record calculating a Lorentzian peak based on m1 (EPICS motor record) position.

We’ll build a custom Python class, from apstools.synApps.SscanRecord (a subclass of ophyd.Device), that supports the EPICS sscan record and add the ophyd.FlyerInterface to that class. A class method will configure the Device staging to setup the sscan record for a 1-D step scan using the m1 positioner and the noisy detector. noisy is implemented as an EPICS swait record configured to recalculate a Lorentzian profile when the m1 readback value changes. The center, width, and height are set randomly as this bluesky instrument package is loaded. The center of the simulated peak will be defined somewhere between +/- 1.

Load the instrument package#

[1]:

%matplotlib widget

import os, pathlib, sys
sys.path.append(os.path.abspath(os.path.join(pathlib.Path.home(), "bluesky")))
from instrument.collection import *

/home/prjemian/bluesky/instrument/_iconfig.py
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/prjemian/Documents/projects/BCDA-APS/bluesky_training/docs/source/howto/.logs/ipython_console.log
Mode           : rotate
Output logging : True
Raw input log  : False
Timestamping   : True
State          : active

I Thu-17:53:40 - ############################################################ startup
I Thu-17:53:40 - logging started
I Thu-17:53:40 - logging level = 10
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/session_logs.py
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/collection.py
I Thu-17:53:40 - CONDA_PREFIX = /home/prjemian/.conda/envs/bluesky_2023_2

Exception reporting mode: Minimal

I Thu-17:53:40 - xmode exception level: 'Minimal'
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/mpl/notebook.py
I Thu-17:53:40 - #### Bluesky Framework ####
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/framework/check_python.py
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/framework/check_bluesky.py
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/framework/initialize.py
I Thu-17:53:40 - RunEngine metadata saved in directory: /home/prjemian/Bluesky_RunEngine_md
I Thu-17:53:40 - using databroker catalog 'training'
I Thu-17:53:40 - using ophyd control layer: pyepics
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/framework/metadata.py
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/epics_signal_config.py
I Thu-17:53:40 - Using RunEngine metadata for scan_id
I Thu-17:53:40 - #### Devices ####
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/devices/area_detector.py
I Thu-17:53:40 - /home/prjemian/bluesky/instrument/devices/calculation_records.py
I Thu-17:53:43 - /home/prjemian/bluesky/instrument/devices/fourc_diffractometer.py
I Thu-17:53:43 - /home/prjemian/bluesky/instrument/devices/ioc_stats.py
I Thu-17:53:43 - /home/prjemian/bluesky/instrument/devices/kohzu_monochromator.py
I Thu-17:53:43 - /home/prjemian/bluesky/instrument/devices/motors.py
I Thu-17:53:43 - /home/prjemian/bluesky/instrument/devices/noisy_detector.py
I Thu-17:53:43 - /home/prjemian/bluesky/instrument/devices/scaler.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/devices/shutter_simulator.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/devices/simulated_fourc.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/devices/simulated_kappa.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/devices/slits.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/devices/sixc_diffractometer.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/devices/temperature_signal.py
I Thu-17:53:44 - #### Callbacks ####
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/callbacks/spec_data_file_writer.py
I Thu-17:53:44 - #### Plans ####
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/plans/lup_plan.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/plans/peak_finder_example.py
I Thu-17:53:44 - /home/prjemian/bluesky/instrument/utils/image_analysis.py
I Thu-17:53:45 - #### Utilities ####
I Thu-17:53:45 - writing to SPEC file: /home/prjemian/Documents/projects/BCDA-APS/bluesky_training/docs/source/howto/20230413-175344.dat
I Thu-17:53:45 -    >>>>   Using default SPEC file name   <<<<
I Thu-17:53:45 -    file will be created when bluesky ends its next scan
I Thu-17:53:45 -    to change SPEC file, use command:   newSpecFile('title')
I Thu-17:53:45 - #### Startup is complete. ####

Add this notebook’s name to the RunEngine metadata and define the IOC’s prefix.

[2]:

RE.md["notebook"] = "sscan_1d_flyer"
ioc = "gp:"

Bluesky step scan#

We are told that the noisy signal will show a peak when m1 is moved over the range [-1 .. +1]. Use bluesky to show that peak.

[3]:

# show a pre-assembled bluesky.plans.scan() acquisition
RE(bp.scan([noisy], m1, -1.1, 1.1, 21, md=dict(purpose="demo bluesky scan plan")))



Transient Scan ID: 956     Time: 2023-04-13 17:53:45
Persistent Unique Scan ID: 'cfa4afcd-bcf0-4c35-ae0d-7f4daa927238'
New stream: 'label_start_motor'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |         m1 |      noisy |
+-----------+------------+------------+------------+
|         1 | 17:53:46.8 |    -1.1000 | 7442.55055 |
|         2 | 17:53:47.2 |    -0.9900 | 11249.11706 |
|         3 | 17:53:47.6 |    -0.8800 | 18986.60948 |
|         4 | 17:53:48.0 |    -0.7700 | 38984.13377 |
|         5 | 17:53:48.4 |    -0.6600 | 81758.13663 |
|         6 | 17:53:48.8 |    -0.5500 | 90661.67681 |
|         7 | 17:53:49.2 |    -0.4400 | 44616.92674 |
|         8 | 17:53:49.6 |    -0.3300 | 21037.26285 |
|         9 | 17:53:50.0 |    -0.2200 | 12314.44644 |
|        10 | 17:53:50.4 |    -0.1100 | 7477.76738 |
|        11 | 17:53:50.8 |     0.0000 | 5304.05319 |
|        12 | 17:53:51.2 |     0.1100 | 3719.11921 |
|        13 | 17:53:51.6 |     0.2200 | 2812.51654 |
|        14 | 17:53:52.0 |     0.3300 | 2297.41489 |
|        15 | 17:53:52.4 |     0.4400 | 1785.46562 |
|        16 | 17:53:52.8 |     0.5500 | 1508.59485 |
|        17 | 17:53:53.2 |     0.6600 | 1209.38156 |
|        18 | 17:53:53.6 |     0.7700 | 1024.83242 |
|        19 | 17:53:54.0 |     0.8800 |  919.60953 |
|        20 | 17:53:54.4 |     0.9900 |  765.85225 |
|        21 | 17:53:54.8 |     1.1000 |  655.33053 |
+-----------+------------+------------+------------+
generator scan ['cfa4afcd'] (scan num: 956)

[3]:

('cfa4afcd-bcf0-4c35-ae0d-7f4daa927238',)
../_images/howto__synapps_sscan_1d_flyer_6_2.png 
[4]:

import pprint
pprint.pprint(bec.peaks)

{
'com':
    {'noisy': -0.5404925440667286}
,
'cen':
    {'noisy': -0.597662063312674}
,
'max':
    {'noisy': (-0.55,
               90661.67681062203)}
,
'min':
    {'noisy': (1.1,
               655.3305297746574)}
,
'fwhm':
    {'noisy': 0.31034751314623277}
,
}

sscan as Bluesky Flyer#

Load some structures to be used.

[5]:

from ophyd import DeviceStatus, Signal
from ophyd.flyers import FlyerInterface
from apstools.synApps import SscanRecord

if save data is to be used

# TODO: if save data is to be used

# # configure saveData for data collection into MDA files:
# save_data.stage_sigs["file_system"] = "/tmp"
# save_data.stage_sigs["subdirectory"] = "saveData"
# save_data.stage_sigs["base_name"] = "sscan1_"
# save_data.stage_sigs["next_scan_number"] = 1
# save_data.stage_sigs["comment1"] = "testing"
# save_data.stage_sigs["comment2"] = "configured and run from ophyd"

The setup_staging_1D_step() method (below) configures the device’s stage_sigs dictionary to configure the sscan record for the 1-D scan. The bluesky RunEngine will only be responsible for configuring the sscan record and telling it to start. Once it has stopped, this device will collect the data from the sscan record’s array fields and yield it back to the RunEngine per the Bluesky event model.

  • The sscan record will record the time stamp for each point (relative to the scan’s starting time) if the text time or TIME is used as the readback of one of the positioners. At the expense of providing valuable timestamp information for the Bluesky event model interface, this reduces the number of possible positioners for use by the sscan record to three.

  • To record the positioner setpoint value for each point, the motor’s setpoint PV (.VAL field) is added as an additional detector.

  • This scan is preconfigured for the m1 positioner and the noisy detector.

[6]:

import time

class SscanFlyer_1D_StepSimple(FlyerInterface, SscanRecord):

    def __init__(self, *args, **kwargs):
        self._acquiring = False

        super().__init__(*args, **kwargs)

    def stage(self):
        super().stage()
        self.select_channels()

    def unstage(self):
        super().unstage()
        self.select_channels()

    def setup_staging_1D_step(self, start=-1.1, finish=1.1, num=21, ddelay=0.01, pdelay=0):
        """Configure sscan record for 1D step scan: noisy v. m1"""
        self.xref = dict(
            positioners=[m1, ],
            raw_detectors=[noisy, ],
            detectors=[noisy, m1.user_setpoint]  # include motor setpoints array
        )
        self.stage_sigs["number_points"] = num
        self.stage_sigs["pasm"] = "PRIOR POS"
        self.stage_sigs["positioner_delay"] = pdelay
        for i, p in enumerate(self.xref["positioners"]):
            self.stage_sigs[f"positioners.p{i+1}.setpoint_pv"] = p.user_setpoint.pvname
            self.stage_sigs[f"positioners.p{i+1}.readback_pv"] = p.user_readback.pvname
            self.stage_sigs[f"positioners.p{i+1}.start"] = start
            self.stage_sigs[f"positioners.p{i+1}.end"] = finish
        self.stage_sigs["detector_delay"] = ddelay
        for i, d in enumerate(self.xref["detectors"]):
            self.stage_sigs[f"detectors.d{i+1:02d}.input_pv"] = d.pvname

        # Get timestamp of each point in the scan.
        # This is a sscan record feature that returns the time since the scan started.
        # The time returned is relative to the first point of the scan.
        self.stage_sigs[f"positioners.p4.readback_pv"] = "time"  # or TIME (all upper case)

    def read_configuration(self):
        return {}

    def describe_configuration(self):
        return {}

    def kickoff(self):
        """Start the sscan record."""
        # self.setup_staging_1D_step()
        self.stage()
        time.sleep(0.1)

        # set(), do not `yield`, in kickoff()
        self.execute_scan.set(1)  # start the sscan record
        self._acquiring = True

        status = DeviceStatus(self)
        status.set_finished()  # means that kickoff was successful
        return status

    def complete(self):
        """Wait for sscan to complete."""
        logger.info("complete() starting")
        if not self._acquiring:
            raise RuntimeError("Not acquiring")

        st = DeviceStatus(self)
        cb_started = False

        def execute_scan_cb(value, timestamp, **kwargs):
            """Watch ``sscan.EXSC`` for completion."""
            value = int(value)
            if cb_started and value == 0:
                logger.info("complete() ending")
                self.unstage()
                self._acquiring = False
                self.execute_scan.unsubscribe(execute_scan_cb)
                if not st.done:
                    logger.info("Setting %s execute status to `done`.", self.name)
                    st.set_finished()

        self.execute_scan.subscribe(execute_scan_cb)
        # self.execute_scan.set(1)
        cb_started = True
        return st

    def describe_collect(self):
        """
        Provide schema & meta-data from collect().

        https://blueskyproject.io/ophyd/generated/ophyd.flyers.FlyerInterface.describe_collect.html
        """
        dd = {}
        dd.update(m1.describe())
        dd.update(noisy.describe())
        return {self.name: dd}

    def collect(self):
        """
        Retrieve all collected data (after complete()).

        Retrieve data from the flyer as proto-events.
        https://blueskyproject.io/ophyd/generated/ophyd.flyers.FlyerInterface.collect.html
        """
        if self._acquiring:
            raise RuntimeError("Acquisition still in progress. Call complete() first.")

        def get_data_from_sscan(obj, n):
            """Read a sscan array and return as Python list."""
            data = obj.read()[obj.name]
            data["value"] = list(data["value"][:n])
            return data

        def mkdoc(seq_num, values):
            """Bundle the dictionary of values into a raw event document."""
            timestamp = values.pop("__ts__")
            yield dict(
                seq_num=seq_num,
                time=timestamp,
                data={k: v for k, v in values.items()},
                timestamps={k: timestamp for k in values},
            )

        def read_sscan_data(scan):
            """Get the sscan arrays and yield as discrete events."""
            _cp = scan.current_point.read()[scan.current_point.name]
            n = _cp["value"]
            ts_last_point = _cp["timestamp"]

            # get the per-step time stamps from positioner 4
            ts_arr = self.positioners.p4.array.get(use_monitor=False)[:n]
            ts_arr = ts_last_point + ts_arr - ts_arr.max()

            results = dict(__ts__=list(ts_arr))  # __ts__ holds the timestamps, per point

            # This gets the full array for each item in one document
            for category, signals in scan.xref.items():
                for i, signal in enumerate(signals):
                    if category == "positioners":
                        item = f"p{i+1}"
                    elif category == "detectors":
                        item = f"d{i+1:02d}"
                    else:
                        continue
                    data = get_data_from_sscan(
                        getattr(scan, f"{category}.{item}.array"), n
                    )
                    results[signal.name] = data["value"]

            # yield all results one complete step at a time
            for i in range(n):
                yield from mkdoc(i+1, {k: results[k][i] for k in results})

        yield from read_sscan_data(self)
        self.unstage()

[7]:

flyer = SscanFlyer_1D_StepSimple(f"{ioc}scan1", name="flyer")
flyer.wait_for_connection()  # sscan records have _many_ channels and fields
flyer.reset()  # clear out any previous configuration

[8]:

flyer.setup_staging_1D_step(num=71)

RE(bp.fly([flyer], md=dict(purpose="demo bluesky fly plan with sscan record and 1-D step scan")))



Transient Scan ID: 957     Time: 2023-04-13 17:53:56
Persistent Unique Scan ID: '1802af12-916d-447d-82e7-f9d79e43e5fa'
New stream: 'label_start_motor'

I Thu-17:53:57 - complete() starting
I Thu-17:54:13 - complete() ending
I Thu-17:54:13 - Setting flyer execute status to `done`.

New stream: 'flyer'

/home/prjemian/.conda/envs/bluesky_2023_2/lib/python3.10/site-packages/event_model/__init__.py:224: UserWarning: The document type 'bulk_events' has been deprecated in favor of 'event_page', whose structure is a transpose of 'bulk_events'.
  warnings.warn(

[8]:

('1802af12-916d-447d-82e7-f9d79e43e5fa',)

[9]:

import matplotlib.pyplot as plt

run = cat[-1]
dataset = run.flyer.read()
x = dataset["m1_user_setpoint"]
y = dataset["noisy"]

plt.plot(x.values, y.values)
plt.title(f"scan_id={run.metadata['start']['scan_id']}")
plt.xlabel(x.name)
plt.ylabel(y.name)

[9]:

Text(0, 0.5, 'noisy')
../_images/howto__synapps_sscan_1d_flyer_15_1.png 
[10]:

dataset

[10]:

<xarray.Dataset>
Dimensions:           (time: 71)
Coordinates:
  * time              (time) float64 1.681e+09 1.681e+09 ... 1.681e+09 1.681e+09
Data variables:
    m1                (time) float64 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    m1_user_setpoint  (time) float64 -1.1 -1.069 -1.037 ... 1.037 1.069 1.1
    noisy             (time) float64 7.078e+03 8.256e+03 ... 682.6 679.9
[11]:

# compute the centroid and sigma from first and second moments of y(x):

print(f"centroid: {((y*x).sum()/y.sum()).data}")
print(f"sigma: {np.sqrt((y*x*x).sum()/y.sum()).data}")

centroid: -0.5366484118894111
sigma: 0.6149932116936468

References#

The documentation for Flyers is distributed across the bluesky and ophyd packages: