Friday, August 13. 2010
Passing pipes to subprocesses in Python in Windows
Passing pipes around in Windows is a bit more complicated than it is in Unix-like operating systems. This post is a bit of information about how file descriptor inheritance works in Windows and a quick example of how to do it.
The first difficulty to overcome is that file descriptors are not inherited by subprocesses in Windows as they are in Linux. However, OS file handles can be inheritable, and it is possible to retrieve the OS file handle associated with a C file descriptor using the _get_osfhandle function. It is also possible to convert the OS file handle back to a C file descriptor in the child process using _open_osfhandle. However, the OS file handle is not inheritable (see Python Bug 4708 - although on a side note, I don't think they should be default-inheritable since there is no preexec_fn in which to close them and prevent deadlock situations where a child holds the write end of a pipe open, but that is another matter), so to get an inheritable OS file handle it must be duplicated using DuplicateHandle.
The operating system-specific functions described above can be accessed through the _subprocess and msvcrt modules. Their use can be seen in the source for the subprocess module, if desired. However, note that the interface in _subprocess is not stable and should not be depended on, but since there is no alternative (that I am aware of - short of writing another module in C) it is the best solution. Now, an example:
parent.py:
import os
import subprocess
import sys
if sys.platform == "win32":
import msvcrt
import _subprocess
else:
import fcntl
# Create pipe for communication
pipeout, pipein = os.pipe()
# Prepare to pass to child process
if sys.platform == "win32":
curproc = _subprocess.GetCurrentProcess()
pipeouth = msvcrt.get_osfhandle(pipeout)
pipeoutih = _subprocess.DuplicateHandle(curproc, pipeouth, curproc, 0, 1,
_subprocess.DUPLICATE_SAME_ACCESS)
pipearg = str(int(pipeoutih))
else:
pipearg = str(pipeout)
# Must close pipe input if child will block waiting for end
# Can also be closed in a preexec_fn passed to subprocess.Popen
fcntl.fcntl(pipein, fcntl.F_SETFD, fcntl.FD_CLOEXEC)
# Start child with argument indicating which FD/FH to read from
subproc = subprocess.Popen(['python', 'child.py', pipearg], close_fds=False)
# Close read end of pipe in parent
os.close(pipeout)
if sys.platform == "win32":
pipeoutih.Close()
# Write to child (could be done with os.write, without os.fdopen)
pipefh = os.fdopen(pipein, 'w')
pipefh.write("Hello from parent.")
pipefh.close()
# Wait for the child to finish
subproc.wait()
child.py:
import os
import sys
if sys.platform == "win32":
import msvcrt
# Get file descriptor from argument
pipearg = int(sys.argv[1])
if sys.platform == "win32":
pipeoutfd = msvcrt.open_osfhandle(pipearg)
else:
pipeoutfd = pipearg
# Read from pipe
# Note: Could be done with os.read/os.close directly, instead of os.fdopen
pipeout = os.fdopen(pipeoutfd, 'r')
print pipeout.read()
pipeout.close()
Note: For this example to work, python must be on on the executable search path (i.e. in the %PATH% environment variable). If it is not, change the subprocess invocation to include the full path to python.exe. Note also that there is some complication with OS file handle leakage. When a _subprocess_handle object is garbage collected the handle that it holds is closed, unless it has been attached. Therefore, if the subprocess will outlive the scope in which the handle variable is defined, it must be detached (by calling its Detach method) to prevent closing during garbage collection. But this solution has its own problems as there is no way (that I am aware of) to close the handle from the child process, which means a file handle will be leaked.
Above caveats aside, this method does work for passing pipes between parent and child processes on Windows. I hope you find it useful.
