Python molurus (the Indian Python)

Working with subprocess

2012-02-05T08:46:00.001-08:00

The subprocess module provides some very useful functionality for working with external programs from Python applications, but is often complained about as being harder to use than it needs to be. See, for example, Kenneth Reitz’s Envoy project, which aims to provide an ease-of-use wrapper over subprocess. There’s also Andrew Moffat’s pbs project, which aims to let you do things like

from pbs import ifconfig
print ifconfig("eth0")

Which it does by replacing sys.modules['pbs'] with a subclass of the module type which overrides __getattr__ to look for programs in the path. Which is nice, and I can see that it would be useful in some contexts, but I don’t find that wc(ls("/etc", "-1"), "-l") is more readable than call(“ls /etc –1 | wc –l”) in the general case.

I’ve been experimenting with my own wrapper for subprocess, called sarge. The main things I need are:

I want to use command pipelines, but using subprocess out of the box often leads to deadlocks because pipe buffers get filled up.
I want to use bash-style pipe syntax on Windows as well as Posix, but Windows shells don’t support some of the syntax I want to use, like &&, ||, |& and so on.
I want to process output from commands in a flexible way, and communicate() is not always flexible enough for my needs - for example, if I need to process output a line at a time.
I want to avoid shell injection problems by having the ability to quote command arguments safely, and I want to minimise the use of shell=True, which I generally have to use when using pipelined commands.
I don’t want to set arbitrary limits on passing data between processes, such as Envoy’s 10MB limit.
subprocess allows you to let stderr be the same as stdout, but not the other way around - and I sometimes need to do that.

I’ve been working on supporting these use cases, so sarge offers the following features:

A simple run function which allows a rich subset of Bash-style shell command syntax, but parsed and run by sarge so that you can run cross-platform on Posix and Windows without cygwin:

>>> p = run('false && echo foo')
>>> p.commands
[Command('false')]
>>> p.returncodes
[1]
>>> p.returncode
1
>>> p = run('false || echo foo')
foo
>>> p.commands
[Command('false'), Command('echo foo')]
>>> p.returncodes
[1, 0]
>>> p.returncode
0

The ability to format shell commands with placeholders, such that variables are quoted to prevent shell injection attacks:

>>> from sarge import shell_format
>>> shell_format('ls {0}', '*.py')
"ls '*.py'"
>>> shell_format('cat {0}', 'a file name with spaces')
"cat 'a file name with spaces'"

The ability to capture output streams without requiring you to program your own threads. You just use a Capture object and then you can read from it as and when you want:

>>> from sarge import Capture, run
>>> with Capture() as out:
...     run('echo foobarbaz', stdout=out)
...
<sarge.Pipeline object at 0x175ed10>
>>> out.read(3)
'foo'
>>> out.read(3)
'bar'
>>> out.read(3)
'baz'
>>> out.read(3)
'\n'
>>> out.read(3)
''

A Capture object can capture the output from multiple commands:

>>> from sarge import run, Capture
>>> p = run('echo foo; echo bar; echo baz', stdout=Capture())
>>> p.stdout.readline()
'foo\n'
>>> p.stdout.readline()
'bar\n'
>>> p.stdout.readline()
'baz\n'
>>> p.stdout.readline()
''

Delays in commands are honoured in asynchronous calls:

>>> from sarge import run, Capture
>>> cmd = 'echo foo & (sleep 2; echo bar) & (sleep 1; echo baz)'
>>> p = run(cmd, stdout=Capture(), async=True) # returns immediately
>>> p.close() # wait for completion
>>> p.stdout.readline()
'foo\n'
>>> p.stdout.readline()
'baz\n'
>>> p.stdout.readline()
'bar\n'
>>>

Here, the sleep commands ensure that the asynchronous echo calls occur in the order foo (no delay), baz (after a delay of one second) and bar (after a delay of two seconds); the capturing works as expected.

Sarge hasn’t been released yet, but it’s not far off being ready. It’s meant for Python >= 2.6.5 and is tested on 2.6, 2.7, 3.1, 3.2 and 3.3 on Linux, Mac OS X, Windows XP and Windows 7 (not all versions are tested on all platforms, but the overall test coverage is comfortably over 90%).

I have released the sarge documentation on Read The Docs; I’m hoping people will read this and give some feedback about the API and feature set being proposed, so that I can fill in any gaps where possible and perhaps make it more useful to other people. Please add your comments here, or via the issue tracker on the BitBucket project for the docs.

A documentation viewer for Sphinx

2012-01-24T13:32:00.001-08:00

Eric Holscher, one of the creators of Read The Docs, recently posted about the importance of a documentation culture in Open Source development, and about things that could be done to encourage this. He makes some good points, and Read The Docs is a very nice looking showcase for documentation. Writing good documentation is difficult enough at the best of times, and one practical problem that I face when working on Sphinx documentation is that I often feel I have to break away from composing it to building it, to see how it looks - because the look of it on the page will determine how I want to refine it.

What I’ve tended to do is work iteratively by making some changes to the ReST sources, invoking make html and refreshing the browser to show how the built documentation looks. This is OK, but does break the flow more than a little (for me, anyway, but I can’t believe I’m the only one).

I had the idea that it would be nice to streamline the process somewhat, so that all I would need to do is to save the changed ReST source – the building and browser refresh would be automatically done, and if I had the editor and browser windows open next to each other in tiled fashion, I could achieve a sort of WYSIWYG effect with the changes appearing in the browser a second or two after I saved any changes.

I decided to experiment with this idea, and needed a browser which I could easily control (to get it to refresh on-demand). I decided to use Roberto Alsina’s 128-line browser, which is based on QtWebKit and PyQt. Roberto posted his browser code almost a year ago, and I knew I’d find a use for it one day :-)

I also needed to track changes to .rst files in the documentation tree, and since I do a fair amount of my Open Source development on Linux, I decided to use inotify functionality. Although there is a Python binding for this, I decided to use the command-line interface and the subprocess module in the standard library, because I wasn’t very familiar with inotify and the command-line interface is easier to experiment with.

The basic mechanism of the solution is that the browser watches for changes in source files in the documentation tree, invokes Sphinx to build the documentation, and then refreshes its contents. This is done in a separate thread:

class Watcher(QtCore.QThread):
    def run(self):
        self._stop = False
        watch_command = 'inotifywait -rq -e close_write --exclude \'"*.html"\' .'.split()
        make_command = 'make html'.split()
        while not self._stop:
            # Perhaps should put notifier access in a mutex - not bothering yet
            self.notifier = subprocess.Popen(watch_command)
            self.notifier.wait()
            if self._stop:
                break
            subprocess.call(make_command)
            # Refresh the UI ...
            self.parent().changed.emit()

    def stop(self):
        self._stop = True
        # Perhaps should put notifier access in a mutex - not bothering for now
        if self.notifier.poll() is None:    # not yet terminated ...
            self.notifier.terminate()

The thread invokes inotifywait, and waits for it to exit. This happens when a file is written in the documentation tree which has an extension other than .html, and this typically happens when a source file is edited and saved. The inotifywait command is usually available through a Linux package – on Ubuntu, for example, you can install it using sudo apt-get install inotify-tools. In the specific invocation used, the –r flag tells the program to recursively watch a particular directory, –q indicates that output should not be too verbose (it’s used for trouble-shooting only), –e close_write indicates that we’re only interested in files being closed after being opened for writing, and --exclude '"*.html"' indicates that we don’t care about writes to .html files.

The Watcher instance’s parent is the main (browser) window. In this we create a new custom event, changed, to be emitted when we want the window to know that the HTML has changed. This is done through the following snippets of code:

changed = QtCore.pyqtSignal()

which is declared in the main window class, and

self.watcher = Watcher(self)
self.changed.connect(self.wb.reload)
self.watcher.start()

which are added to the window’s constructor. Here, self.wb is the QWebView component of the browser which actually holds the browser content.

One last refinement is to save the browser window coordinates on exit and restore them on starting, so that if you have moved the window to a particular location, it will reappear there every time until you move it. First, we create a module-level QSettings instance:

settings = QtCore.QSettings("Vinay Sajip", "DocWatch")

and provide a couple of main window methods to load and save the settings:

    def load_settings(self):
        settings.beginGroup('mainwindow')
        pos = settings.value('pos')
        size = settings.value('size')
        if isinstance(pos, QtCore.QPoint):
            self.move(pos)
        if isinstance(size, QtCore.QSize):
            self.resize(size)
        settings.endGroup()

    def save_settings(self):
        settings.beginGroup('mainwindow')
        settings.setValue('pos', self.pos())
        settings.setValue('size', self.size())
        settings.endGroup()

When the main window is closed, we need to stop the watcher and save the settings. (We also need to call load_settings in the main window constructor.)

    def closeEvent(self, event):
        self.save_settings()
        self.watcher.stop()

The last thing is to construct the code which is invoked when the module is invoked as a script. Note that this very simplistic use is consistent with Sphinx’s quick-start script defaults.

if __name__ == "__main__":
    if not os.path.isdir('_build'):
        # very simplistic sanity check. Works for me, as I generally use
        # sphinx-quickstart defaults
        print('You must run this application from a Sphinx directory containing _build')
        rc = 1
    else:
        app=QtGui.QApplication(sys.argv)
        path = os.path.join('_build', 'html', 'index.html')
        url = 'file:///' + pathname2url(os.path.abspath(path))
        url = QtCore.QUrl(url)
        wb=MainWindow(url)
        wb.show()
        rc = app.exec_()
    sys.exit(rc)

The code (MIT licensed) is available from here. As it’s a single file standalone script, I haven’t considered putting it on PyPI – it’s probably easier to download it to a $HOME/bin or similar location, then you can invoke it in the docs directory of your project, run your editor, position the browser and editor windows suitably, and you’re ready to go! Here’s a screen-shot using doc-watch and gedit:

Please feel free to try it. Comments and suggestions are welcome.

Update: Another advantage of using the subprocess / command line approach to notification is that it’s easy to slot in a solution for a platform which doesn’t support inotify. Alternatives are available for both Windows and Mac OS X. For example, on Windows, if you have IronPython installed, the following script could be used to provide the equivalent functionality to inotifywait (for this specific application):

import clr
import os

from System.IO import FileSystemWatcher, NotifyFilters

stop = False

def on_change(source, e):
    global stop
    if not e.Name.endswith('.html'):
        stop = True
    print('%s: %s, stop = %s' % (e.FullPath, e.ChangeType, stop))

watcher = FileSystemWatcher(os.getcwd())
watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.FileName
watcher.EnableRaisingEvents = True
watcher.IncludeSubdirectories = True
watcher.Changed += on_change
watcher.Created += on_change

while not stop:
    pass

Whereas for Mac OS X, if you install the MacFSEvents package, the following script could be used to provide the equivalent functionality to inotifywait (again, for this specific application):

#!/usr/bin/env python

import os

from fsevents import Observer, Stream

stop = False

def on_change(e):
    global stop
    path = e.name
    if os.path.isfile(path):
        if not path.endswith('.html'):
            stop = True
    print('%s: %s, stop = %s' % (e.name, e.mask, stop))

observer = Observer()
observer.start()
stream = Stream(on_change, os.getcwd(), file_events=True)
observer.schedule(stream)
try:
    while not stop:
        pass
finally:
    observer.unschedule(stream)
    observer.stop()
    observer.join()