Linux Command & Wait for Finish

Demonstrate how to launch a (Linux bash) shell command from Python and wait for it to finish. This involves setting a command and receiving two different values (stdout and stderr). A custom subclass of ophyd.SignalRO executes the shell command and processes the results. We add a parse_response(stdout, stderr) method so any subclass can easily process the string output result of the Linux command.

To simulate a Linux command to be run, a bash shell script (doodle.sh) was created that runs a countdown (default: 5 seconds) printing to stdout (the terminal console).

Later, we replace the doodle.sh with other common shell commands.

Example shell command

The example shell command is a bash script that executes a 5 second countdown. The script is shown first:

%pycat ./doodle.sh

#!/bin/bash

echo $(date): Doodle demonstration starting

# optional argument is number of seconds to sleep, default is 5
counter=${1:-5}

until [ $counter -eq 0 ]; do
    echo $(date): countdown ${counter}
    sleep 1
    ((counter--))
done
echo $(date): Doodle demonstration complete

Now, run it to show how it works.

!bash ./doodle.sh

Fri Nov 8 04:33:26 PM CST 2024: Doodle demonstration starting
Fri Nov 8 04:33:26 PM CST 2024: countdown 5
Fri Nov 8 04:33:27 PM CST 2024: countdown 4
Fri Nov 8 04:33:28 PM CST 2024: countdown 3
Fri Nov 8 04:33:29 PM CST 2024: countdown 2
Fri Nov 8 04:33:30 PM CST 2024: countdown 1
Fri Nov 8 04:33:31 PM CST 2024: Doodle demonstration complete

Run from Python subprocess

There are several possibilities to run a shell command from Python. For various reasons, we choose subprocess.Popen() which allows us to start the command in one step, then wait for the process to complete in another step.

For more details, see the documentation.

import subprocess
import time

First step: Start the Linux command.

command = "bash ./doodle.sh"

# Start the command
t0 = time.time()
process = subprocess.Popen(
    command,
    shell=True,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

Second step: Wait for the command to finish.

# wait for the command to finish and collect the outputs.
stdout, stderr = process.communicate()
duration = time.time() - t0
print(f"{duration=:.4f}s")

duration=5.0260s

Show the results.

print(f"{stdout = }")
print(f"{stderr = }")

stdout = b'Fri Nov 8 04:33:31 PM CST 2024: Doodle demonstration starting\nFri Nov 8 04:33:31 PM CST 2024: countdown 5\nFri Nov 8 04:33:32 PM CST 2024: countdown 4\nFri Nov 8 04:33:33 PM CST 2024: countdown 3\nFri Nov 8 04:33:34 PM CST 2024: countdown 2\nFri Nov 8 04:33:35 PM CST 2024: countdown 1\nFri Nov 8 04:33:36 PM CST 2024: Doodle demonstration complete\n'
stderr = b''

Convert from byte strings to plain utf8 text.

# byte strings, must decode to see as string
print(f"stdout\n{stdout.decode('utf8')}")
print(f"stderr\n{stderr.decode('utf8')}")

stdout
Fri Nov 8 04:33:31 PM CST 2024: Doodle demonstration starting
Fri Nov 8 04:33:31 PM CST 2024: countdown 5
Fri Nov 8 04:33:32 PM CST 2024: countdown 4
Fri Nov 8 04:33:33 PM CST 2024: countdown 3
Fri Nov 8 04:33:34 PM CST 2024: countdown 2
Fri Nov 8 04:33:35 PM CST 2024: countdown 1
Fri Nov 8 04:33:36 PM CST 2024: Doodle demonstration complete

stderr

Custom ophyd.SignalRO subclass

Let’s show how to use a read-only Signal (SignalRO) to execute a pre-configured Linux shell command.

We’ll execute the shell command in the Signal’s .trigger() method using subprocess.communicate() in a thread. The .trigger() method returns a Status object. Once the Linux command finishes, any text returned by the command will be stored in the Signal’s ._readback attribute (to be returned by the .get() method). Any error output will be stored in the .stderr attribute.

We redefine the .trigger() method in a custom subclass of ophyd.SignalRO.

import ophyd
import threading


class ProcessSignal(ophyd.SignalRO):
    """Signal that returns output from a shell command."""

    command = "bash doodle.sh"
    status = None

    def parse_response(self, stdout, stderr):
        self._readback = stdout.decode("utf8")
        self.stderr = stderr.decode("utf8")

    def trigger(self):
        self.status = ophyd.status.Status()

        def action():
            """Calls command and waits for it to complete."""
            process = subprocess.Popen(
                self.command,
                shell=True,
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )

            # wait for the command to finish and collect the outputs.
            self.parse_response(*process.communicate())
            self.status._finished(success=True)

        threading.Thread(target=action, daemon=True).start()
        return self.status  # returns right away

Create the signal object. Print its initial value.

t0 = time.time()
signal = ProcessSignal(name="signal")
print(f"{signal.get()=}")

signal.get()=0.0

The default value of a new SignalRO object is 0.0. That will change once signal has completed its first Linux command.

Trigger the signal (run its .trigger() method). This returns immediately, before the shell command finishes. The return result is a Status object that bluesky will use to wait for the .trigger() operation to finish.

Until the Linux command finishes, the value returned by signal.get() is still unchanged.

status = signal.trigger()
print(f"{signal.get()=}")
print(f"{status = }")
print(f"{time.time()-t0 = }")

signal.get()=0.0
status = Status(obj=None, done=False, success=False)
time.time()-t0 = 0.011680364608764648

We use the status object to wait for the Linux command to complete. The shell script runs for 5 seconds, the status object is done in that time plus a smidgen.

print(f"{status = }")
print(f"{time.time()-t0 = }")
status.wait()

print(f"{signal.get()=}")
print(f"{status = }")
print(f"{time.time()-t0 = }")

status = Status(obj=None, done=False, success=False)
time.time()-t0 = 0.018770456314086914
signal.get()='Fri Nov 8 04:33:36 PM CST 2024: Doodle demonstration starting\nFri Nov 8 04:33:36 PM CST 2024: countdown 5\nFri Nov 8 04:33:37 PM CST 2024: countdown 4\nFri Nov 8 04:33:38 PM CST 2024: countdown 3\nFri Nov 8 04:33:39 PM CST 2024: countdown 2\nFri Nov 8 04:33:40 PM CST 2024: countdown 1\nFri Nov 8 04:33:41 PM CST 2024: Doodle demonstration complete\n'
status = Status(obj=None, done=True, success=True)
time.time()-t0 = 5.043781518936157

Linux system uptime Signal

Make a signal that provides a numerical value, so we can plot it.

The elapsed time since the Linux workstation was last started is updated in virtual file /proc/uptime. The file has two string values: uptime idletime. We want the first one.

We can modify the ProcessSignal class and change the command and the parse_response() method.

For more details, see the documentation.

class UptimeSignal(ProcessSignal):
    command = "cat /proc/uptime"

    def parse_response(self, stdout, stderr):
        self._readback = float(stdout.decode("utf8").split()[0])
        self.stderr = stderr.decode("utf8")


uptime = UptimeSignal(name="uptime")

Record a time series with the Bluesky Run Engine

When we record a time series of the system uptime, we expect a straight line plot. Try it. First, setup the minimum required bluesky objects.

from bluesky import RunEngine, plans as bp
from bluesky.callbacks.best_effort import BestEffortCallback

RE = RunEngine()
bec = BestEffortCallback()
RE.subscribe(bec)

0

Record the time series. Here, uptime is a detector. At each step, the count plan will trigger it, wait for the trigger to complete, then read the signal with its .read() method.

RE(bp.count([uptime], num=5, delay=1))

Transient Scan ID: 1     Time: 2024-11-08 16:33:42
Persistent Unique Scan ID: '2e05f6e6-bf6c-4fa0-815e-9da8f2bac8c5'
New stream: 'primary'
+-----------+------------+------------+
|   seq_num |       time |     uptime |
+-----------+------------+------------+
|         1 | 16:33:42.4 | 630362.860 |
|         2 | 16:33:43.3 | 630363.870 |
|         3 | 16:33:44.3 | 630364.870 |
|         4 | 16:33:45.4 | 630365.880 |
|         5 | 16:33:46.3 | 630366.870 |
+-----------+------------+------------+
generator count ['2e05f6e6'] (scan num: 1)

('2e05f6e6-bf6c-4fa0-815e-9da8f2bac8c5',)

Summary

SignalRO is the right class to use for a shell command that returns a single value.

One might think the ophyd.Signal class could be used where the command would be put and the returned value would be as above. The problem is the design of the Signal.put() method.

The ophyd.Signal.put() method requests the Signal to go to the value and then waits for it to get there (that’s when it uses up its status object). The output of the shell command will never become the value of the command string. If we were to set obj._readback to be the output from the shell command, then the put() method would never return (it hangs because the readback value does not equal the input value).

SignalRO, not Signal, is the right interface.