Linux Command & Wait for Finish#
Demonstrate how to launch a (Linux bash) shell command from Python and wait for it to finish.
This involves setting a command and receiving two different values (stdout and stderr). An ophyd.Signal is for setting and reading one value. The ophyd.Device can provide the multiple values needed.
This involves setting a command and receiving two different values (stdout and stderr). An ophyd.Signal is for setting and reading only one value. First we show how a Signal-based implementation would behave. Then, we show how the ophyd.Device can provide the multiple values needed.
To simulate a Linux command to be run, a bash shell script (doodle.sh) was created that runs a 5 second countdown printing to stdout (the terminal console).
1. Example shell command#
The example shell command is a bash script that executes a 5 second countdown. The script is shown first:
[1]:
!cat ./doodle.sh
#!/bin/bash
echo $(date): Doodle demonstration starting
echo $(date): sleep 5 seconds
for i in 5 4 3 2 1; do
echo $(date): countdown ${i}
sleep 1
done
echo $(date): Doodle demonstration complete
Now, run it to show how it works.
[2]:
!bash ./doodle.sh
Fri 23 Jul 2021 09:38:40 PM CDT: Doodle demonstration starting
Fri 23 Jul 2021 09:38:40 PM CDT: sleep 5 seconds
Fri 23 Jul 2021 09:38:40 PM CDT: countdown 5
Fri 23 Jul 2021 09:38:41 PM CDT: countdown 4
Fri 23 Jul 2021 09:38:42 PM CDT: countdown 3
Fri 23 Jul 2021 09:38:43 PM CDT: countdown 2
Fri 23 Jul 2021 09:38:44 PM CDT: countdown 1
Fri 23 Jul 2021 09:38:45 PM CDT: Doodle demonstration complete
2. Run from Python subprocess#
[3]:
import subprocess
import time
[4]:
command = "bash ./doodle.sh"
# Start the command
t0 = time.time()
process = subprocess.Popen(
command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
[5]:
# wait for the command to finish and collect the outputs.
stdout, stderr = process.communicate()
print(f"{time.time() - t0 = }")
time.time() - t0 = 5.049006938934326
[6]:
print(f"{stdout = }")
print(f"{stderr = }")
stdout = b'Fri 23 Jul 2021 09:38:45 PM CDT: Doodle demonstration starting\nFri 23 Jul 2021 09:38:45 PM CDT: sleep 5 seconds\nFri 23 Jul 2021 09:38:45 PM CDT: countdown 5\nFri 23 Jul 2021 09:38:46 PM CDT: countdown 4\nFri 23 Jul 2021 09:38:47 PM CDT: countdown 3\nFri 23 Jul 2021 09:38:48 PM CDT: countdown 2\nFri 23 Jul 2021 09:38:49 PM CDT: countdown 1\nFri 23 Jul 2021 09:38:50 PM CDT: Doodle demonstration complete\n'
stderr = b''
[7]:
# byte strings, must decode to see as string
print("stdout\n", stdout.decode('utf8'))
print("stderr\n", stderr.decode('utf8'))
stdout
Fri 23 Jul 2021 09:38:45 PM CDT: Doodle demonstration starting
Fri 23 Jul 2021 09:38:45 PM CDT: sleep 5 seconds
Fri 23 Jul 2021 09:38:45 PM CDT: countdown 5
Fri 23 Jul 2021 09:38:46 PM CDT: countdown 4
Fri 23 Jul 2021 09:38:47 PM CDT: countdown 3
Fri 23 Jul 2021 09:38:48 PM CDT: countdown 2
Fri 23 Jul 2021 09:38:49 PM CDT: countdown 1
Fri 23 Jul 2021 09:38:50 PM CDT: Doodle demonstration complete
stderr
3. As ophyd.Signal#
Since this is a demonstration, we show here why the Signal implementation just does not provide the right behavior.
An ophyd.Signal will be used to accept an input, launch the shell command in a subprocess from the Signal.set() method, and wait for the response using an ophyd.Status object.
Since a redefinition of the set() method is needed, it is necessary to create a subclass of ophyd.Signal.
[8]:
import ophyd
import threading
class ProcessSignal(ophyd.Signal):
process = None
_readback = None
stderr = None
def set(self, command, *, timeout=None, settle_time=None):
st = ophyd.status.Status(self)
def wait_process():
self._readback, self.stderr = self.process.communicate(timeout=timeout)
st._finished()
self._status = st
self.process = subprocess.Popen(
command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
# TODO: settle_time
threading.Thread(target=wait_process, daemon=True).start()
return st
Create the processor object and run (.set()) it. This will return immediately, before the shell script finishes. The return result is a Status object that bluesky will use to wait for the .set() operation to finish.
[9]:
t0 = time.time()
obj = ProcessSignal(name="obj")
st = obj.set("bash ./doodle.sh")
print(f"{st = }")
print(f"{time.time()-t0 = }")
st = Status(obj=ProcessSignal(name='obj', value=0.0, timestamp=1627094331.38266), done=False, success=False)
time.time()-t0 = 0.012959480285644531
The timeout was not configured. The shell script runs for 5 seconds so we use the status object to wait for it to complete.
[10]:
print(f"{obj._status = }")
print(f"{time.time()-t0 = }")
st.wait()
print(f"{obj._status = }")
print(f"{time.time()-t0 = }")
obj._status = Status(obj=ProcessSignal(name='obj', value=0.0, timestamp=1627094331.38266), done=False, success=False)
time.time()-t0 = 0.09882283210754395
obj._status = Status(obj=ProcessSignal(name='obj', value=b'Fri 23 Jul 2021 09:38:51 PM CDT: Doodle demonstration starting\nFri 23 Jul 2021 09:38:51 PM CDT: sleep 5 seconds\nFri 23 Jul 2021 09:38:51 PM CDT: countdown 5\nFri 23 Jul 2021 09:38:52 PM CDT: countdown 4\nFri 23 Jul 2021 09:38:53 PM CDT: countdown 3\nFri 23 Jul 2021 09:38:54 PM CDT: countdown 2\nFri 23 Jul 2021 09:38:55 PM CDT: countdown 1\nFri 23 Jul 2021 09:38:56 PM CDT: Doodle demonstration complete\n', timestamp=1627094331.38266), done=True, success=True)
time.time()-t0 = 5.05002236366272
Show what is returned from the read() method.
[11]:
obj.read()
[11]:
{'obj': {'value': b'Fri 23 Jul 2021 09:38:51 PM CDT: Doodle demonstration starting\nFri 23 Jul 2021 09:38:51 PM CDT: sleep 5 seconds\nFri 23 Jul 2021 09:38:51 PM CDT: countdown 5\nFri 23 Jul 2021 09:38:52 PM CDT: countdown 4\nFri 23 Jul 2021 09:38:53 PM CDT: countdown 3\nFri 23 Jul 2021 09:38:54 PM CDT: countdown 2\nFri 23 Jul 2021 09:38:55 PM CDT: countdown 1\nFri 23 Jul 2021 09:38:56 PM CDT: Doodle demonstration complete\n',
'timestamp': 1627094331.38266}}
The problem is seen after we try the .put() method
[12]:
t0 = time.time()
st = obj.put("bash ./doodle.sh")
print(f"{st = }")
print(f"{time.time()-t0 = }")
st = None
time.time()-t0 = 0.006703853607177734
As before, wait for it to finish and the value is still the input command. Note the put() method does not return its status object so we have to use a sleep timer.
[13]:
print(f"{obj.read() = }")
print(f"{time.time()-t0 = }")
time.sleep(5)
print(f"{obj.read() = }")
print(f"{time.time()-t0 = }")
obj.read() = {'obj': {'value': 'bash ./doodle.sh', 'timestamp': 1627094336.6018999}}
time.time()-t0 = 0.13547873497009277
obj.read() = {'obj': {'value': 'bash ./doodle.sh', 'timestamp': 1627094336.6018999}}
time.time()-t0 = 5.1512017250061035
The ophyd.Signal.put() method requests the Signal to go to the value and then waits for it to get there (that’s when it uses up its status object). The output of the shell script will never become the value of the command string. If we were to set obj._readback to be the output from the shell script, then the put() method would never return (it hangs because the readback value does not equal the input value).
Signal is not the right interface.
4. As ophyd.Device#
[14]:
import ophyd
import subprocess
import threading
import time
class ProcessDevice(ophyd.Device):
command = ophyd.Component(ophyd.Signal, value=None)
stdout = ophyd.Component(ophyd.Signal, value=None)
stderr = ophyd.Component(ophyd.Signal, value=None)
process = None
def trigger(self):
"""Start acquisition."""
if self.command.get() is None:
raise ValueError(f"Must set {self.name}.command. Cannot be `None`.")
st = ophyd.status.DeviceStatus(self)
def watch_process():
out, err = self.process.communicate()
# these are byte strings, decode them to get str
self.stdout.put(out.decode("utf8"))
self.stderr.put(err.decode("utf8"))
self.process = None
st._finished()
self._status = st
self.stderr.put(None)
self.stdout.put(None)
self.process = subprocess.Popen(
self.command.get(),
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
threading.Thread(target=watch_process, daemon=True).start()
return st
[15]:
obj = ProcessDevice(name="obj")
obj.stage_sigs["command"] = "bash ./doodle.sh"
[16]:
t0 = time.time()
obj.stage()
st = obj.trigger()
print(f"{time.time() - t0 = }s")
print(f"{st = }")
obj.read()
time.time() - t0 = 0.011952638626098633s
st = DeviceStatus(device=obj, done=False, success=False)
[16]:
OrderedDict([('obj_command',
{'value': 'bash ./doodle.sh', 'timestamp': 1627094342.1131032}),
('obj_stdout', {'value': None, 'timestamp': 1627094342.1135113}),
('obj_stderr', {'value': None, 'timestamp': 1627094342.113497})])
[17]:
st.wait()
print(f"{time.time() - t0 = }s")
print(f"{st = }")
obj.read()
time.time() - t0 = 5.063066720962524s
st = DeviceStatus(device=obj, done=True, success=True)
[17]:
OrderedDict([('obj_command',
{'value': 'bash ./doodle.sh', 'timestamp': 1627094342.1131032}),
('obj_stdout',
{'value': 'Fri 23 Jul 2021 09:39:02 PM CDT: Doodle demonstration starting\nFri 23 Jul 2021 09:39:02 PM CDT: sleep 5 seconds\nFri 23 Jul 2021 09:39:02 PM CDT: countdown 5\nFri 23 Jul 2021 09:39:03 PM CDT: countdown 4\nFri 23 Jul 2021 09:39:04 PM CDT: countdown 3\nFri 23 Jul 2021 09:39:05 PM CDT: countdown 2\nFri 23 Jul 2021 09:39:06 PM CDT: countdown 1\nFri 23 Jul 2021 09:39:07 PM CDT: Doodle demonstration complete\n',
'timestamp': 1627094347.1758268}),
('obj_stderr', {'value': '', 'timestamp': 1627094347.1758533})])
[18]:
obj.unstage()
[18]:
[ProcessDevice(prefix='', name='obj', read_attrs=['command', 'stdout', 'stderr'], configuration_attrs=[])]
[19]:
print(obj.stdout.get())
Fri 23 Jul 2021 09:39:02 PM CDT: Doodle demonstration starting
Fri 23 Jul 2021 09:39:02 PM CDT: sleep 5 seconds
Fri 23 Jul 2021 09:39:02 PM CDT: countdown 5
Fri 23 Jul 2021 09:39:03 PM CDT: countdown 4
Fri 23 Jul 2021 09:39:04 PM CDT: countdown 3
Fri 23 Jul 2021 09:39:05 PM CDT: countdown 2
Fri 23 Jul 2021 09:39:06 PM CDT: countdown 1
Fri 23 Jul 2021 09:39:07 PM CDT: Doodle demonstration complete
5. Run with bluesky#
This is a simplest implementation of the bluesky RunEngine with no custom callbacks, no table output, and no saving data anywhere. Capture the document stream from RE using a simple callback (document_printer()) that prints the content of each document.
In this demo, we do not show how to implement a timeout and or interrupt execution of the shell script.
[20]:
import bluesky
import bluesky.plans as bp
import pprint
def document_printer(key, doc):
print()
print(f"***{key}***")
pprint.pprint(doc)
RE = bluesky.RunEngine({})
RE(bp.count([obj]), document_printer)
***start***
{'detectors': ['obj'],
'hints': {'dimensions': [(('time',), 'primary')]},
'num_intervals': 0,
'num_points': 1,
'plan_args': {'detectors': ["ProcessDevice(prefix='', name='obj', "
"read_attrs=['command', 'stdout', 'stderr'], "
'configuration_attrs=[])'],
'num': 1},
'plan_name': 'count',
'plan_type': 'generator',
'scan_id': 1,
'time': 1627094347.901483,
'uid': 'b7fca79d-87de-446b-aec9-a14369899306',
'versions': {'bluesky': '1.7.0', 'ophyd': '1.6.1'}}
***descriptor***
{'configuration': {'obj': {'data': {},
'data_keys': OrderedDict(),
'timestamps': {}}},
'data_keys': {'obj_command': {'dtype': 'string',
'object_name': 'obj',
'shape': [],
'source': 'SIM:obj_command'},
'obj_stderr': {'dtype': 'string',
'object_name': 'obj',
'shape': [],
'source': 'SIM:obj_stderr'},
'obj_stdout': {'dtype': 'string',
'object_name': 'obj',
'shape': [],
'source': 'SIM:obj_stdout'}},
'hints': {'obj': {'fields': []}},
'name': 'primary',
'object_keys': {'obj': ['obj_command', 'obj_stdout', 'obj_stderr']},
'run_start': 'b7fca79d-87de-446b-aec9-a14369899306',
'time': 1627094353.0929906,
'uid': 'a2bfc0cb-1447-4be6-92e9-0b3b157e3c30'}
***event***
{'data': {'obj_command': 'bash ./doodle.sh',
'obj_stderr': '',
'obj_stdout': 'Fri 23 Jul 2021 09:39:08 PM CDT: Doodle demonstration '
'starting\n'
'Fri 23 Jul 2021 09:39:08 PM CDT: sleep 5 seconds\n'
'Fri 23 Jul 2021 09:39:08 PM CDT: countdown 5\n'
'Fri 23 Jul 2021 09:39:09 PM CDT: countdown 4\n'
'Fri 23 Jul 2021 09:39:10 PM CDT: countdown 3\n'
'Fri 23 Jul 2021 09:39:11 PM CDT: countdown 2\n'
'Fri 23 Jul 2021 09:39:12 PM CDT: countdown 1\n'
'Fri 23 Jul 2021 09:39:13 PM CDT: Doodle demonstration '
'complete\n'},
'descriptor': 'a2bfc0cb-1447-4be6-92e9-0b3b157e3c30',
'filled': {},
'seq_num': 1,
'time': 1627094353.3092813,
'timestamps': {'obj_command': 1627094347.901314,
'obj_stderr': 1627094353.0920315,
'obj_stdout': 1627094353.0919352},
'uid': '2d5d29de-8c80-46c1-8085-c1eb1c609b07'}
***stop***
{'exit_status': 'success',
'num_events': {'primary': 1},
'reason': '',
'run_start': 'b7fca79d-87de-446b-aec9-a14369899306',
'time': 1627094353.429601,
'uid': '9aa6d3c6-8bfc-4ded-8b9b-28a2f2455ffb'}
[20]:
('b7fca79d-87de-446b-aec9-a14369899306',)
Looks like we got the output from the bash shell script.