Output scan(s) to a SPEC data file.#

One of the common concerns is how to access data from bluesky’s database. The standard way is to replay the document stream from each of the scans through a bluesky callback that writes the data to the desired file format. Here, we write data to the SPEC file format.

Setup#

First, we must setup a bluesky session. We create a temporary catalog to store the data from this notebook’s data collection.

%matplotlib inline
import matplotlib.pyplot as plt
plt.ion()

from bluesky import RunEngine, plans as bp
from bluesky.callbacks.best_effort import BestEffortCallback
import databroker

cat = databroker.temp()
RE = RunEngine({})
RE.subscribe(cat.v1.insert)
RE.subscribe(BestEffortCallback())

1

SPEC data files usually contain a list of all the motor positions at the start of each scan. Starting with apstools release 1.6.11, it became possible to have the list of all ophyd-labeled motor devices logged to a separate stream at the start of every scan. The next configuration will create a label_start_motor stream with the positions. The SpecWriterCallback will find this stream and record the motor names and values in the #O and #P control lines, respectively.

from apstools.plans import label_stream_wrapper

def motor_start_preprocessor(plan):
    return label_stream_wrapper(plan, "motor", when="start")

RE.preprocessors.append(motor_start_preprocessor)

We need to create some scan data. We’ll use Gaussian peak profile calculated in a userCalc (swait record) and a motor from our EPICS IOC simulator gp:. Initialize the peak computation with some randomized parameters. The peak will be at some motor position between -1 .. +1. Print the actual values for our reference later on.

from apstools.devices import UserCalcN, setup_gaussian_swait
import numpy as np
from ophyd import EpicsMotor, EpicsSignal

IOC = "gp:"
motor = EpicsMotor(f"{IOC}m1", name="motor", labels=["motor"])
# define some extra motors to demonstrate #O & #P lines
omega = EpicsMotor(f"{IOC}m2", name="omega", labels=["motor"])
chi = EpicsMotor(f"{IOC}m3", name="chi", labels=["motor"])
phi = EpicsMotor(f"{IOC}m4", name="phi", labels=["motor"])
ttheta = EpicsMotor(f"{IOC}m5", name="ttheta", labels=["motor"])
atth = EpicsMotor(f"{IOC}m6", name="atth", labels=["motor"])
mx = EpicsMotor(f"{IOC}m7", name="mx", labels=["motor"])
my = EpicsMotor(f"{IOC}m8", name="my", labels=["motor"])
mz = EpicsMotor(f"{IOC}m9", name="mz", labels=["motor"])
swait = UserCalcN(f"{IOC}userCalc1", name="swait")
det = EpicsSignal(swait.calculated_value.pvname, name="det")

motor.wait_for_connection()
swait.wait_for_connection()
det.wait_for_connection()

setup_gaussian_swait(
    swait, motor.user_readback, 
    center=-0.5 + np.random.uniform(), 
    width=0.01 + 0.2*np.random.uniform(), 
    noise=0.05*(.95 + .1*np.random.uniform()),
    scale=100_000*(.95 + .1*np.random.uniform()), 
    )

print(f"calc = {swait.calculation.get()}")
print(f"A: x = {swait.channels.A.input_pv.get()}")
print(f"B: center = {swait.channels.B.input_value.get()}")
print(f"C: width = {swait.channels.C.input_value.get()}")
print(f"D: scale = {swait.channels.D.input_value.get()}")
print(f"E: noise= {swait.channels.E.input_value.get()}")

calc = D*(0.95+E*RNDM)/exp(((A-b)/c)^2)
A: x = gp:m1.RBV
B: center = -0.10578292257195432
C: width = 0.08364992146414521
D: scale = 100371.09148167742
E: noise= 0.05133473921276364

Finally, run the scan with the RunEngine:

RE(bp.scan([det], motor, -1.2, 1.2, 41))

Transient Scan ID: 1     Time: 2025-11-25 10:47:31
Persistent Unique Scan ID: 'de507b9d-c75b-465b-94f7-a7797b983f44'
New stream: 'label_start_motor'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |      motor |        det |
+-----------+------------+------------+------------+
|         1 | 10:47:36.5 |    -1.2000 |    0.00000 |
|         2 | 10:47:36.8 |    -1.1400 |    0.00000 |
|         3 | 10:47:37.1 |    -1.0800 |    0.00000 |
|         4 | 10:47:37.4 |    -1.0200 |    0.00000 |
|         5 | 10:47:37.7 |    -0.9600 |    0.00000 |
|         6 | 10:47:38.0 |    -0.9000 |    0.00000 |
|         7 | 10:47:38.3 |    -0.8400 |    0.00000 |
|         8 | 10:47:38.6 |    -0.7800 |    0.00000 |
|         9 | 10:47:38.9 |    -0.7200 |    0.00000 |
|        10 | 10:47:39.2 |    -0.6600 |    0.00000 |
|        11 | 10:47:39.5 |    -0.6000 |    0.00000 |
|        12 | 10:47:39.8 |    -0.5400 |    0.00000 |
|        13 | 10:47:40.1 |    -0.4800 |    0.00020 |
|        14 | 10:47:40.4 |    -0.4200 |    0.07199 |
|        15 | 10:47:40.7 |    -0.3600 |    9.70544 |
|        16 | 10:47:41.0 |    -0.3000 |  444.92528 |
|        17 | 10:47:41.3 |    -0.2400 | 7559.36977 |
|        18 | 10:47:41.6 |    -0.1800 | 44718.83240 |
|        19 | 10:47:41.9 |    -0.1200 | 93527.29018 |
|        20 | 10:47:42.2 |    -0.0600 | 72688.52312 |
|        21 | 10:47:42.5 |     0.0000 | 19567.78836 |
|        22 | 10:47:42.8 |     0.0600 | 1927.74181 |
|        23 | 10:47:43.1 |     0.1200 |   67.65720 |
|        24 | 10:47:43.4 |     0.1800 |    0.82258 |
|        25 | 10:47:43.7 |     0.2400 |    0.00377 |
|        26 | 10:47:44.0 |     0.3000 |    0.00001 |
|        27 | 10:47:44.3 |     0.3600 |    0.00000 |
|        28 | 10:47:44.6 |     0.4200 |    0.00000 |
|        29 | 10:47:44.9 |     0.4800 |    0.00000 |
|        30 | 10:47:45.2 |     0.5400 |    0.00000 |
|        31 | 10:47:45.5 |     0.6000 |    0.00000 |
|        32 | 10:47:45.8 |     0.6600 |    0.00000 |
|        33 | 10:47:46.1 |     0.7200 |    0.00000 |
|        34 | 10:47:46.4 |     0.7800 |    0.00000 |
|        35 | 10:47:46.7 |     0.8400 |    0.00000 |
|        36 | 10:47:47.0 |     0.9000 |    0.00000 |
|        37 | 10:47:47.3 |     0.9600 |    0.00000 |
|        38 | 10:47:47.6 |     1.0200 |    0.00000 |
|        39 | 10:47:47.9 |     1.0800 |    0.00000 |
|        40 | 10:47:48.2 |     1.1400 |    0.00000 |
|        41 | 10:47:48.5 |     1.2000 |    0.00000 |
+-----------+------------+------------+------------+
generator scan ['de507b9d'] (scan num: 1)

('de507b9d-c75b-465b-94f7-a7797b983f44',)
../_images/f9c601a731fd67806ab5dd733a7bcf5b9339345f802c95e9aa795b6505d8d89a.png ../_images/08696658921297d8c14cdc0b745d59784f959c93ee4f2ca8ad749d553080b2e6.png

get the most recent scan, by steps#

The databroker instance, cat, provides access to its scans by several means. One way is to consider cat as a list and retrieve the last item from the list. This will return a run, the common reference to be used.

For this first example, we’ll work through the steps one by one. The cat.v2 interface is the easiest to use (at this writing).

run = cat.v2[-1]
run

BlueskyRun
  uid='de507b9d-c75b-465b-94f7-a7797b983f44'
  exit_status='success'
  2025-11-25 10:47:31.736 -- 2025-11-25 10:47:48.607
  Streams:
    * label_start_motor
    * primary

As shown, the run has one data stream, name primary. The databroker provides a simple table view of this run:

run.primary.read()

/home/prjemian/.conda/envs/apstools/lib/python3.11/site-packages/databroker/intake_xarray_core/base.py:23: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 1kB
Dimensions:              (time: 41)
Coordinates:
  * time                 (time) float64 328B 1.764e+09 1.764e+09 ... 1.764e+09
Data variables:
    det                  (time) float64 328B 4.697e-70 4.115e-62 ... 1.489e-101
    motor                (time) float64 328B -1.2 -1.14 -1.08 ... 1.08 1.14 1.2
    motor_user_setpoint  (time) float64 328B -1.2 -1.14 -1.08 ... 1.08 1.14 1.2

Support function: specfile_example()#

We need to call the apstools.callbacks.SpecWriterCallback2() callback with the run’s documents.

Here, specfile_example() is a support function that can be used with one or more runs to create a SPEC data file (one Bluesky run will become one SPEC scan in the same file).

from apstools.callbacks import SpecWriterCallback2
from databroker._drivers.mongo_normalized import BlueskyMongoCatalog
from databroker._drivers.msgpack import BlueskyMsgpackCatalog
import warnings

DEMO_SPEC_FILE = "test_specdata.txt"

def specfile_example(runs, filename=DEMO_SPEC_FILE):
    """write one or more headers (scans) to a SPEC data file"""
    if isinstance(runs, databroker.core.BlueskyRun):
        runs = [runs]
    if not isinstance(runs, (list, BlueskyMsgpackCatalog, BlueskyMongoCatalog)):
        raise TypeError("Must give run object or list of run objects.")
    if len(runs) == 0:
        raise ValueError("Must provide one or more runs.")

    specwriter = SpecWriterCallback2()
    specwriter.spec_filename = filename
    for uid in runs:
        if isinstance(uid, str):
            run = runs[uid]
        else:
            run = uid
        if not isinstance(run, databroker.core.BlueskyRun):
            warnings.warn(f"Skipping {run=}, it is not a BlueskyRun object.")
            continue
        # to get the raw document stream, need the v1 interface
        h = run.catalog_object.v1[run.name]  # header
        for key, doc in h.db.get_documents(h):
            specwriter.receiver(key, doc)
        # lines = specwriter.prepare_scan_contents()
    print(f"Look at SPEC data file: {specwriter.spec_filename}")

Write the run as SPEC data#

Let’s write it as a SPEC data file (namely: spec1.dat):

import pathlib

spec1_dat = pathlib.Path("spec1.dat")
if spec1_dat.exists():  # re-write the file
    spec1_dat.unlink()  # delete the existing file

specfile_example(run, filename=spec1_dat)

Look at SPEC data file: spec1.dat

Let’s view file spec1.dat from disk storage (using the pycat IPython magic function):

%pycat spec1.dat

#F spec1.dat
#E 1764089251.7365375
#D Tue Nov 25 10:47:31 2025
#C Bluesky  user = prjemian  host = arf.jemian.org
#O0 atth  chi  motor  mx  my  mz  omega  phi
#O1 ttheta
#o0 atth  chi  motor  mx  my  mz  omega  phi
#o1 ttheta

#S 1  scan(detectors=['det'], num=41, args='['motor', -1.2, 1.2]', per_step='None')
#D Tue Nov 25 10:47:31 2025
#C Tue Nov 25 10:47:36 2025.  uid = de507b9d-c75b-465b-94f7-a7797b983f44
#P0 0 0 1.2 0 0 0 0 0
#P1 0
#MD versions = {'ophyd': '1.11.0', 'bluesky': '1.14.6'}
#MD plan_type = generator
#MD plan_name = scan
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_points = 41
#MD num_intervals = 40
#MD plan_args = {'detectors': ["EpicsSignal(read_pv='gp:userCalc1.VAL', name='det', timestamp=1764088504.230701, tolerance=1e-05, auto_monitor=False, string=False, write_pv='gp:userCalc1.VAL', limits=False, put_complete=False)"], 'num': 41, 'args': ["EpicsMotor(prefix='gp:m1', name='motor', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu'])", -1.2, 1.2], 'per_step': 'None'}
#MD hints = {'dimensions': [[['motor'], 'primary']]}
#MD plan_pattern = inner_product
#MD plan_pattern_module = bluesky.plan_patterns
#MD plan_pattern_args = {'num': 41, 'args': ["EpicsMotor(prefix='gp:m1', name='motor', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu'])", -1.2, 1.2]}
#N 5
#L motor  Epoch  Epoch_float  motor_user_setpoint  det
-1.2 5 4.798788785934 -1.2 4.697096076314e-70
-1.14 5 5.081346988678 -1.14 4.114682306176e-62
-1.08 5 5.382177829742 -1.08 1.191150081374e-54
-1.02 6 5.682698965073 -1.02 1.287762326763e-47
-0.96 6 5.984358787537 -0.96 4.982474832648e-41
-0.9 6 6.283968687057 -0.9 6.816012652073e-35
-0.84 7 6.58753156662 -0.84 3.40895401415e-29
-0.78 7 6.885951042175 -0.78 6.102627367248e-24
-0.72 7 7.186074256897 -0.72 3.855028249112e-19
-0.66 7 7.477291345596 -0.66 8.37741144642e-15
-0.6 8 7.788086652756 -0.6 6.602897660574e-11
-0.54 8 8.088327646255 -0.54 1.981647945317e-07
-0.48 8 8.388518571854 -0.48 0.0002010513966765
-0.42 9 8.689679145813 -0.42 0.07198503319617
-0.36 9 8.99031829834 -0.36 9.705444449639
-0.3 9 9.290639400482 -0.3 444.9252801024
-0.24 10 9.591261386871 -0.24 7559.369770398
-0.18 10 9.892528533936 -0.18 44718.83239994
-0.12 10 10.19314527512 -0.12 93527.29017926
-0.06 10 10.48501563072 -0.06 72688.52312141
0 11 10.79420518875 0 19567.78836187
0.06 11 11.09540462494 0.06 1927.74181425
0.12 11 11.39624977112 0.12 67.65719559994
0.18 12 11.69581484795 0.18 0.8225804876164
0.24 12 11.99742436409 0.24 0.003772408428037
0.3 12 12.29894351959 0.3 5.963670063734e-06
0.36 13 12.58793568611 0.36 3.346353217619e-09
0.42 13 12.88823223114 0.42 6.644520253063e-13
0.48 13 13.18871331215 0.48 4.840757860829e-17
0.54 13 13.49933886528 0.54 1.272406018296e-21
0.6 14 13.78994488716 0.6 1.210375244034e-26
0.66 14 14.09026098251 0.66 4.007957513344e-32
0.72 14 14.39210033417 0.72 4.60698256011e-38
0.78 15 14.69092798233 0.78 1.996277220905e-44
0.84 15 14.99262237549 0.84 2.970223944581e-51
0.9 15 15.30345225334 0.9 1.625809092382e-58
0.96 16 15.60361862183 0.96 3.065464820924e-66
1.02 16 15.89481163025 1.02 2.158397475032e-74
1.08 16 16.2053592205 1.08 5.307692599663e-83
1.14 17 16.50043964386 1.14 4.71594745842e-92
1.2 17 16.80608057976 1.2 1.489067902142e-101
#C Tue Nov 25 10:47:48 2025.  num_events_label_start_motor = 1
#C Tue Nov 25 10:47:48 2025.  num_events_primary = 41
#C Tue Nov 25 10:47:48 2025.  exit_status = success

We see that the output of the specfile_example() command includes the content of the SPEC file. For the remaining examples, we’ll skip this additional step to view the SPEC file contents from disk.

a specific scan#

The cat object allows us to access scans by UUID (or any shorter version that remains unique in the database). We show an example but have commented it out since those runs do not exist in our temporary databroker catalog.

# specfile_example(cat["37c188c0"], filename="spec3.dat")

specify a list of scans by UID#

Suppose we have a list of scans where we know the UID of each one, we can build a list of headers and write a SPEC data file with that list. Here, we have such a list of tuning scans. We show an example but have commented it out since those runs do not exist in our temporary databroker catalog.

# runs = [cat[uid] for uid in "957d83c c354fe37-e39f 42c".split()]
# specfile_example(runs, filename="spec_tunes.dat")

Find specific plans within a range of dates#

The cat object allows for filtering arguments based on any keywords in the start document and also by time. Here, we filter between certain dates and also by plan name. The dates are specified in ISO8601 format and can include precision beyond a millisecond. Here, we use the v2 interface to do the searches. We show examples how to pick between a set of dates.

Also, we write to the default data file: test_specdata.txt.

from databroker.queries import TimeRange
import datetime

test_specdata_txt = pathlib.Path("test_specdata.txt")
if test_specdata_txt.exists():  # will re-write the file
    test_specdata_txt.unlink()

query = {}
query.update(TimeRange(since="2019-02-19 17:00"))
query.update(TimeRange(until="2032-02-19 17:11:30"))
query.update(dict(plan_name="scan"))

runs = cat.v2.search(query)
for uid in runs:
    run = runs[uid]
    start_time = run.metadata["start"]["time"]
    isodate = datetime.datetime.fromtimestamp(start_time).isoformat(sep=" ")
    print(uid[:8], isodate)
specfile_example(runs, test_specdata_txt)

de507b9d 2025-11-25 10:47:31.736537
Look at SPEC data file: test_specdata.txt

%pycat test_specdata.txt

#F test_specdata.txt
#E 1764089251.7365375
#D Tue Nov 25 10:47:31 2025
#C Bluesky  user = prjemian  host = arf.jemian.org
#O0 atth  chi  motor  mx  my  mz  omega  phi
#O1 ttheta
#o0 atth  chi  motor  mx  my  mz  omega  phi
#o1 ttheta

#S 1  scan(detectors=['det'], num=41, args='['motor', -1.2, 1.2]', per_step='None')
#D Tue Nov 25 10:47:31 2025
#C Tue Nov 25 10:47:36 2025.  uid = de507b9d-c75b-465b-94f7-a7797b983f44
#P0 0 0 1.2 0 0 0 0 0
#P1 0
#MD versions = {'ophyd': '1.11.0', 'bluesky': '1.14.6'}
#MD plan_type = generator
#MD plan_name = scan
#MD detectors = ['det']
#MD motors = ['motor']
#MD num_points = 41
#MD num_intervals = 40
#MD plan_args = {'detectors': ["EpicsSignal(read_pv='gp:userCalc1.VAL', name='det', timestamp=1764088504.230701, tolerance=1e-05, auto_monitor=False, string=False, write_pv='gp:userCalc1.VAL', limits=False, put_complete=False)"], 'num': 41, 'args': ["EpicsMotor(prefix='gp:m1', name='motor', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu'])", -1.2, 1.2], 'per_step': 'None'}
#MD hints = {'dimensions': [[['motor'], 'primary']]}
#MD plan_pattern = inner_product
#MD plan_pattern_module = bluesky.plan_patterns
#MD plan_pattern_args = {'num': 41, 'args': ["EpicsMotor(prefix='gp:m1', name='motor', settle_time=0.0, timeout=None, read_attrs=['user_readback', 'user_setpoint'], configuration_attrs=['user_offset', 'user_offset_dir', 'velocity', 'acceleration', 'motor_egu'])", -1.2, 1.2]}
#N 5
#L motor  Epoch  Epoch_float  motor_user_setpoint  det
-1.2 5 4.798788785934 -1.2 4.697096076314e-70
-1.14 5 5.081346988678 -1.14 4.114682306176e-62
-1.08 5 5.382177829742 -1.08 1.191150081374e-54
-1.02 6 5.682698965073 -1.02 1.287762326763e-47
-0.96 6 5.984358787537 -0.96 4.982474832648e-41
-0.9 6 6.283968687057 -0.9 6.816012652073e-35
-0.84 7 6.58753156662 -0.84 3.40895401415e-29
-0.78 7 6.885951042175 -0.78 6.102627367248e-24
-0.72 7 7.186074256897 -0.72 3.855028249112e-19
-0.66 7 7.477291345596 -0.66 8.37741144642e-15
-0.6 8 7.788086652756 -0.6 6.602897660574e-11
-0.54 8 8.088327646255 -0.54 1.981647945317e-07
-0.48 8 8.388518571854 -0.48 0.0002010513966765
-0.42 9 8.689679145813 -0.42 0.07198503319617
-0.36 9 8.99031829834 -0.36 9.705444449639
-0.3 9 9.290639400482 -0.3 444.9252801024
-0.24 10 9.591261386871 -0.24 7559.369770398
-0.18 10 9.892528533936 -0.18 44718.83239994
-0.12 10 10.19314527512 -0.12 93527.29017926
-0.06 10 10.48501563072 -0.06 72688.52312141
0 11 10.79420518875 0 19567.78836187
0.06 11 11.09540462494 0.06 1927.74181425
0.12 11 11.39624977112 0.12 67.65719559994
0.18 12 11.69581484795 0.18 0.8225804876164
0.24 12 11.99742436409 0.24 0.003772408428037
0.3 12 12.29894351959 0.3 5.963670063734e-06
0.36 13 12.58793568611 0.36 3.346353217619e-09
0.42 13 12.88823223114 0.42 6.644520253063e-13
0.48 13 13.18871331215 0.48 4.840757860829e-17
0.54 13 13.49933886528 0.54 1.272406018296e-21
0.6 14 13.78994488716 0.6 1.210375244034e-26
0.66 14 14.09026098251 0.66 4.007957513344e-32
0.72 14 14.39210033417 0.72 4.60698256011e-38
0.78 15 14.69092798233 0.78 1.996277220905e-44
0.84 15 14.99262237549 0.84 2.970223944581e-51
0.9 15 15.30345225334 0.9 1.625809092382e-58
0.96 16 15.60361862183 0.96 3.065464820924e-66
1.02 16 15.89481163025 1.02 2.158397475032e-74
1.08 16 16.2053592205 1.08 5.307692599663e-83
1.14 17 16.50043964386 1.14 4.71594745842e-92
1.2 17 16.80608057976 1.2 1.489067902142e-101
#C Tue Nov 25 10:47:48 2025.  num_events_label_start_motor = 1
#C Tue Nov 25 10:47:48 2025.  num_events_primary = 41
#C Tue Nov 25 10:47:48 2025.  exit_status = success