Setup for for this beam line’s APS Data Management Python API client.
FIRST
The dm_setup(setup_file) function must be called first,
before any other calls to the dm package. The setup_file
argument is the bash script that activates the APS Data Management
conda environment for the workstation. That file contains definitions
of environment variables needed by the functions below.
Add APS Data Management environment variable definitions to this process.
This function reads the bash script, searching for lines that start with
“export “. Such lines define bash shell environment variables in the bash
script. This function adds those environment variables to the current
environment.
Find the “default” database (has the most recent run).
Note that here, database and catalog mean the same.
This routine looks at all the database instances defined in the
current session (console or notebook). If there is only one or no
database instances defined as objects in the current session, the
choice is simple. When there is more than one database instance in
the current session, then the one with the most recent run timestamp
is selected. In the case (as happens when starting with a new
database) that the current database has no runs and another
database instance is defined in the session and that additional
database has runs in it (such as the previous database), then the
database with the newest run timestamp (and not the newer empty
database) will be chosen.
RETURNS
object or None:
Bluesky database, an instance of databroker.catalog
Get values from a previous scan stream in a databroker catalog.
Optionally, select only those data with names including key_fragment.
Tip
If the output is truncated, use
pd.set_option('display.max_rows',300)
to increase the number of rows displayed.
PARAMETERS
scan_id
int or str :
Scan (run) identifier.
Positive integer value is scan_id from run’s metadata.
Negative integer value is since most recent run in databroker.
String is run’s uid unique identifier (can abbreviate to
the first characters needed to assure it is unique).
key_fragment
str :
Part or all of key name to be found in selected stream.
For instance, if you specify key_fragment="lakeshore",
it will return all the keys that include lakeshore.
db
object :
Bluesky database, an instance of databroker.catalog.
Default: will search existing session for instance.
stream
str :
Name of the bluesky data stream to obtain the data.
Default: ‘baseline’
query
dict :
mongo query dictionary, used to filter the results
Default: {}
"read" shows only the signals returned by obj.read()
default: None
cname
bool : Show the _control_ (Python, dotted) name in column name.
default: False
dname
bool : Show the _data_ (databroker, with underlines) name in
column dataname.
default: True
show_pv
bool : Show the EPICS process variable (PV) name in
column PV.
default: False
Note
Special case when show_pv=True:
If cname is not provided, it will be set True.
If dname is not provided, it will be set False.
use_datetime bool :
Show the EPICS timestamp (time of last update) in
column timestamp.
default: True
show_ancient bool :
Show uninitialized EPICS process variables.
In EPICS, an uninitialized PV has a timestamp of 1990-01-01 UTC.
This option enables or suppresses ancient values identified
by timestamp from 1989. These are values only defined in
the original .db file.
default: True
max_column_width int or None :
Truncate long columns to no more than this length. If not default,
then table will be formatted using pyRestTable.
default: None (will use 50)
table_style object :
Either apstools.utils.TableStyle.pandas (default) or
using values from apstools.utils.TableStyle.
Note
pandas.DataFrame wll truncate long text
to at most 50 characters.
use default OS mail utility (so no credentials needed)
EXAMPLE
Send email(s) when feedback_limits_approached
(a hypothetical boolean) is True:
# setupfromapstools.utilsimportEmailNotificationsSENDER_EMAIL="instrument_user@email.host.tld"email_notices=EmailNotifications(SENDER_EMAIL)email_notices.add_addresses(# This list receives email when send() is called."joe.user@goodmail.com","instrument_team@email.host.tld",# others?)# ... lateriffeedback_limits_approached:# send emails to listsubject="Feedback problem"message="Feedback is very close to its limits."email_notices.send(subject,message)
Return result is a dictionary with the statistical results for a peak
analysis, grouped in pairs (row, column) as it makes sense given
frame[rows][columns].
The \(x\) values are the index number along the respective axis.
List all plans. (Actually, lists all generator functions).
NOTE: Can only detect generator functions.
Bluesky plans are generator functions that generate
bluesky.Msg objects. There is a PR to define
a decorator that identifies a generator function
as a bluesky plan.
PARAMETERS
base object or None :
Object that contains plan methods (if None, use global namespace.)
(default: None)
trunc int :
Truncate long docstrings to no more than trunc characters.
(default: 50)
table_style object :
Either TableStyle.pyRestTable (default) or TableStyle.pandas,
using values from apstools.utils.TableStyle.
Note
pandas.DataFrame wll truncate long text to at most 50 characters.
Convenience function to get the run’s data. Default is the primary stream.
PARAMETERS
scan_id
int or str :
Scan (run) identifier.
Positive integer value is scan_id from run’s metadata.
Negative integer value is since most recent run in databroker.
String is run’s uid unique identifier (can abbreviate to
the first characters needed to assure it is unique).
db
object :
Bluesky database, an instance of databroker.catalog.
Default: will search existing session for instance.
stream
str :
Name of the bluesky data stream to obtain the data.
Default: ‘primary’
query
dict :
mongo query dictionary, used to filter the results
Default: {}
Convenience function to get value of key in run stream.
Defaults are last value of key in primary stream.
PARAMETERS
scan_id
int or str :
Scan (run) identifier.
Positive integer value is scan_id from run’s metadata.
Negative integer value is since most recent run in databroker.
String is run’s uid unique identifier (can abbreviate to
the first characters needed to assure it is unique).
key
str :
Name of the key (data column) in the table of the stream’s data.
Must match identically.
db
object :
Bluesky database, an instance of databroker.catalog.
Default: will search existing session for instance.
stream
str :
Name of the bluesky data stream to obtain the data.
Default: ‘primary’
query
dict :
mongo query dictionary, used to filter the results
Default: {}
int or str :
List index of value to be returned from column of table.
Can be 0 for first value, -1 for last value, "mean"
for average value, or "all" for the full list of values.
Default: -1
use_v1
bool :
Chooses databroker API version between ‘v1’ or ‘v2’.
Default: True (meaning use the v1 API)
Convenience function to list all keys (column names) in the scan’s stream (default: primary).
PARAMETERS
scan_id
int or str :
Scan (run) identifier.
Positive integer value is scan_id from run’s metadata.
Negative integer value is since most recent run in databroker.
String is run’s uid unique identifier (can abbreviate to
the first characters needed to assure it is unique).
key_fragment
str :
Part or all of key name to be found in selected stream.
For instance, if you specify key_fragment="lakeshore",
it will return all the keys that include lakeshore.
db
object :
Bluesky database, an instance of databroker.catalog.
Default: will search existing session for instance.
stream
str :
Name of the bluesky data stream to obtain the data.
Default: ‘primary’
query
dict :
mongo query dictionary, used to filter the results
Default: {}
This function provides a thin interface to the highly-reconfigurable
ListRuns() class in this package.
PARAMETERS
cat
object :
Instance of databroker v1 or v2 catalog.
keys
str or [str] or None:
Include these additional keys from the start document.
(default: None means "scan_idtimeplan_namedetectors")
missing
str:
Test to report when a value is not available.
(default: "")
hints_override bool:
For a key that appears in both the metadata and the hints,
override the metadata value if the same key is found in the hints.
(default: False)
ids
[int] or [str]:
List of uid or scan_id value(s).
Can mix different kinds in the same list.
Also can specify offsets (e.g., -1).
According to the rules for databroker catalogs,
a string is a uid (partial representations allowed),
an int is scan_id if positive or an offset if negative.
(default: None)
num
int :
Make the table include the num most recent runs.
(default: 20)
printing bool or str :
Deprecated.
reverse
bool :
If True, sort in descending order by sortby.
(default: True)
since
str :
include runs that started on or after this ISO8601 time
(default: "1995-01-01")
sortby
str :
Sort columns by this key, found by exact match in either
the start or stop document.
(default: "time")
tablefmt str :
Deprecated. Use table_style instead.
table_style object :
Either TableStyle.pyRestTable (default) or TableStyle.pandas,
using values from apstools.utils.TableStyle.
Note
pandas.DataFrame wll truncate long text to at most 50 characters.
timefmt
str :
The time key (also includes keys "start.time" and "stop.time")
will be formatted by the self.timefmt value.
See https://strftime.org/ for examples. The special timefmt="raw"
is used to report time as the raw value (floating point time as used in
python’s time.time()).
(default: "%Y-%m-%d%H:%M:%S",)
until
str :
include runs that started before this ISO8601 time
(default: 2100-12-31)
**query
dict :
Any additional keyword arguments will be passed to
the databroker to refine the search for matching runs
using the mongoquery package.
Part of the name to store the log file. Full name is
f"<log_path>/{file_name_base}.log" in present working directory.
log_pathstr
Part of the name to store the log file. Full name is
f"<log_path>/{file_name_base}.log" in present working directory.
default: (the present working directory)/LOG_DIR_BASE
levelint
Threshold for reporting messages with this logger. Logging messages
which are less severe than level will be ignored.
default: 10 (logging.DEBUG or DEBUG)
see: https://docs.python.org/3/library/logging.html#levels
maxBytes(optional) int
Log file rollover begins whenever the current log file is nearly
maxBytes in length. A new file is written when the current line will
push the current file beyond this limit.
default: 0
backupCount(optional) int
When backupCount is non-zero, the system will keep up to backupCount
numbered log files (with added extensions .1, ‘.2`, …). The current
log file always has no numbered extension. The previous log file is the
one with the lowest extension number.
default: 0
Note
When either maxBytes or backupCount are zero,
log file rollover never occurs, so you generally want to set
backupCount to at least 1, and have a non-zero maxBytes.
Get the names of all function parameters supplied by the caller.
This is used to differentiate user-supplied parameters from as-defined
parameters with the same value.
HOW TO USE THIS DECORATOR:
Decorate a function or method with this decorator and add an additional
_call_args=None kwarg to the function. The function can test _call_args
if a specific kwarg was supplied by the caller.
EXAMPLE:
@call_signature_decoratordeffunc1(a,b=1,c=True,_call_args=None):if'c'in_call_args:# Caller supplied this kwarg?pass
Note
With call_signature_decorator, it is not possible to get the names
of the positional arguments. Since positional parameters are not specified by
name, such capability is not expected to become a requirement.
Convert text so it can be used as a dictionary key.
Given some input text string, return a clean version
remove troublesome characters, perhaps other cleanup as well.
This is best done with regular expression pattern matching.
run or [run] :
The Bluesky callback to handle the stream of documents from a run. If
None, then use the bec (BestEffortCallback) from the IPython
shell.
(default:None)
sort
bool :
Sort the headers chronologically if True.
(default:True)
Given some input text string, return a clean version.
Remove troublesome characters, perhaps other cleanup as well.
This is best done with regular expression pattern matching.
The “sanitized” name fits this regular expression:
Dictionary with keys accessible as attributes (read-only).
All keys are accessible for read and write by dictionary-style references
(‘mmap[“key”]’ or ‘mmap.get(key)’).
Since dictionary keys are accessible as attributes, the names of existing
attributes (such as ‘clear’, ‘items’, keys’, values’, …) are reserved and
cannot be used as dictionary keys.
EXAMPLE:
>>> mmap=MMap(a=1,b=2)>>> mmapMMap(a=1, b=2)>>> mmap["a]1>>> mmap["a"]=2>>> mmap.a2>>> mmap.a=3AttributeError: Unknown: attribute='a'>>> mmapMMap(a=2, b=2)>>> mmap["clear"]=5KeyError: "key='clear' is a reserved name."
You can retrieve the value of a key by any of these dictionary methods:
mmap["a"]mmap.get("a")# Or supply a default if the key does not exist.# Here, the default is 1.2345mmap.get("a",1.2345)
and by any attribute method:
mmap.agetattr(mmap,"a")# Or supply a default if the key does not exist.getattr(mmap,"a",1.2345)
When code supports a parameter for which a user can provide
a local override, the code should import the overrides
object (from the override_params module),
and then register the parameter name, such as this example:
In the user’s configuration file that will override
the value of 45e-6 (such as can be loaded via
%run-iuser.py), import the overrides`
object (from the override_params module):
fromoverride_paramsimportoverrides
and then override the attribute(s) as desired:
overrides.set("minimum_step",1.0e-5)
With this override in place, the minstep value
(from pick())
will be 1e-5.
Get a pandas DataFrame object with all the overrides:
Note: This is not a bluesky plan. Call it as a normal Python function.
PARAMETERS
runs[run] or run:
List or runs or single run. A run is either a
bluesky.core.BlueskyRun object or a reference (uid, scan_id,
relative to most recent) to a BlueskyRun in the catalog.
xnamestr:
Name of the signal to plot on the x axis.
ynamestr:
Name of the signal to plot on the y axis.
appendbool:
(optional) If True, append to existing plot window.
Default: append=False
catobject:
(optional) Catalog to be used for finding a run by reference.
Default: return value from apstools.utils.getCatalog()
statsbool:
(optional) If True, compute and plot centroid and FWHM
(computed from sigma).
Default: stats=True
streamstr:
(optional) Name of the data stream in which to find “xname”
and “yname”.
Default: stream="primary"
titlestr:
(optional) Title to show on this plot.
Default: Metadata “title” keyword of first run (if found)
or scan_id and starting date/time of first run.
RETURNS
Returns a dict of statistics for each run indexed by scan_id,
if stats=True, else None. A computed fwhm key is added
to the statistics.
Find the plot(s) by name and replot with at most the last n lines.
Note: this is not a bluesky plan. Call it as normal Python function.
It is recommended to call trim_plot_by_name()before the
scan(s) that generate plots. Plots are generated from a RunEngine
callback, executed after the scan completes.
PARAMETERS
n
int :
number of plots to keep
plots
str, [str], or None :
name(s) of plot windows to trim
(default: all plot windows)
EXAMPLES:
trim_plot_by_name()# default of n=3, apply to all plotstrim_plot_by_name(5)# change from default of n=3trim_plot_by_name(5,"noisy_det vs motor")# just this plottrim_plot_by_name(5,["noisy_det vs motor","det noisy_det vs motor"]])
EXAMPLE:
# use simulators from ophydfromblueskyimportplansasbpfromblueskyimportplan_stubsasbpsfromophyd.simimport*snooze=0.25defscan_set():trim_plot_by_name()yield frombp.scan([noisy_det],motor,-1,1,5)yield frombp.scan([noisy_det,det],motor,-2,1,motor2,3,1,6)yield frombps.sleep(snooze)# repeat the_scans 15 timesuids=RE(bps.repeat(scan_set,15))
(new in release 1.3.5)
apstools.utils.plot.trim_plot_lines(bec, n, x, y)[source]#
Find the plot with axes x and y and replot with at most the last n lines.
Note: trim_plot_lines() is not a bluesky plan. Call it as
normal Python function.
EXAMPLE:
trim_plot_lines(bec,1,m1,noisy)
PARAMETERS
bec
object :
instance of BestEffortCallback
n
int :
number of plots to keep
x
object :
instance of ophyd.Signal (or subclass),
independent (x) axis
y
object :
instance of ophyd.Signal (or subclass),
dependent (y) axis
base class: read-only support for Excel files, treat them like databases
Use this class when creating new, specific spreadsheet support.
EXAMPLE
Show how to read an Excel file where one of the columns
contains a unique key. This allows for random access to
each row of data by use of the key.
classExhibitorsDB(ExcelDatabaseFileBase):''' content for exhibitors from the Excel file '''EXCEL_FILE=pathlib.Path("resources")/"exhibitors.xlsx"LABELS_ROW=2defhandle_single_entry(self,entry):'''any special handling for a row from the Excel file'''passdefhandleExcelRowEntry(self,entry):'''identify unique key (row of the Excel file)'''key=entry["Name"]self.db[key]=entry
Generic (read-only) handling of Excel spreadsheet-as-database
Note
This is the class to use when reading Excel spreadsheets.
In the spreadsheet, the first sheet should contain the table to be
used. By default (see keyword parameter labels_row), the table
should start in cell A4. The column labels are given in row 4. A
blank column should appear to the right of the table (see keyword
parameter ignore_extra). The column labels will describe the
action and its parameters. Additional columns may be added for
metadata or other purposes.
The rows below the column labels should contain actions and
parameters for those actions, one action per row.
To make a comment, place a # in the action column. A comment
should be ignored by the bluesky plan that reads this table. The
table will end with a row of empty cells.
While it’s a good idea to put the action column first, that is
not necessary. It is not even necessary to name the column
action. You can re-arrange the order of the columns and change
their names as long as the column names match what text strings
your Python code expects to find.
A future upgrade [1] will allow the table boundaries to be named by
Excel when using Excel’s FormatasTable[2] feature. For now,
leave a blank row and column at the bottom and right edges of the
table.
PARAMETERS
filename
str :
name (absolute or relative) of Excel spreadsheet file
labels_row
int :
Row (zero-based numbering) of Excel file with column labels,
default: 3 (Excel row 4)
ignore_extra
bool :
When True, ignore any cells outside of the table, default:
True.
Note that when True, a row of cells within the table will
be recognized as the end of the table, even if there are
actions in following rows. To force an empty row, use
a comment symbol # (actually, any non-empty content will work).
When False, cells with other information (in Sheet 1) will
be made available, sometimes with unpredictable results.
fromapstools.utilsimportExcelDatabaseFileGeneric,cleanupTextdefmyExcelPlan(xl_file,md={}):excel_file=pathlib.Path(xl_file).absolute()xl=ExcelDatabaseFileGeneric(excel_file)fori,rowinxl.db.values():# prepare the metadata_md={cleanupText(k):vfork,vinrow.items()}_md["xl_file"]=xl_file_md["excel_row_number"]=i+1_md.update(md)# overlay with user-supplied metadata# determine what action to takeaction=row["action"].lower()ifaction=="open":yield frombps.mv(shutter,"open")elifaction=="close":yield frombps.mv(shutter,"close")elifaction=="image":# your code to take an image, given **row as parametersyield frommy_image(**row,md=_md)elifaction=="scan":# your code to make a scan, given **row as parametersyield frommy_scan(**row,md=_md)else:print(f"no handling for row {i+1}: action={action}")# execute this plan through the RunEngineRE(myExcelPlan("spreadsheet.xlsx",md=dict(purpose="apstools demo"))
Conversion factor between full-width at half-maximum peak value and the computed
standard deviation, \(\sigma_c\), of the center of mass (centroid),
\(\mu\), of the peak.
Compute statistical measures of a 1-D array (or weighted array), using numpy.
Results are returned in MMap dictionary.
When suppplied, \(\vec{y}\) will be used as the weights of \(\vec{x}\).
Uses numpy[3] as an alternative to PySumReg. [4]
PARAMETERS
xlist | numpy.ndarray
\(\vec{x}\): 1-D array of numbers.
ylist | numpy.ndarray
\(\vec{y}\): (optional) 1-D array of numbers.
\(\vec{x}\) and \(\vec{y}\) must be of the same length.
Used as weights for \(\vec{x}\) in fitting analyses.
MMap Dictionary Keys
These keys are always defined.
key
type
description
max_x
float
Maximum value of \(\vec{x}\).
mean_x
float
\(\bar{x}\): Mean value of \(\vec{x}\).
median_x
float
Median value of \(\vec{x}\).
min_x
float
Minmum value of \(\vec{x}\).
n
int
\(n\): Length of \(\vec{x}\).
range_x
float
Difference between maximum and minimum values of \(\vec{x}\).
stddev_x
float
\(\sigma_x\) : Standard deviation of \(\vec{x}\).
These keys are added when \(\vec{y}\) is provided.
Requires \(\vec{x}\) & \(\vec{y}\) to be the same length.
key
type
description
centroid
float
\(\mu\) : Average of \(\vec{x}\), weighted by \(\vec{y}\). [5]
fwhm
float
Apparent width of the peak at half the maximum value of \(\vec{y}\). [6]
max_y
float
Maximum value of \(\vec{y}\).
mean_y
float
\(\bar{y}\): Mean value of \(\vec{y}\).
median_y
float
Median value of \(\vec{y}\).
min_y
float
Minmum value of \(\vec{y}\).
range_y
float
Difference between maximum and minimum values of \(\vec{y}\).
stddev_y
float
\(\sigma_y\) : Standard deviation of \(\vec{y}\).
These keys are added when \(\vec{y}\) is not constant.
key
type
description
sigma
float
\(\sigma_c\) : Standard error of \(\mu\), a statistical measure of the peak width. See variance. [7]