I’ve been pretty annoyed today that zsh isn’t automagically TAB-completing the names of Python unittest modules for me on the commandline. But then again, I haven’t spent much time really learning zsh.  Well, that’s over! I’m gonna have my cake and eat it, too, dammit!

I need to get these:

• https://github.com/clvv/fasd#readme
• https://github.com/robbyrussell/oh-my-zsh/tree/master/plugins

This guy has convinced me that I’m barely scratching the surface of what zsh can do…

I recently set up a Subversion pre-commit hook at work to check for the presence of an issue reference in the commit message. It was tricky enough to figure out that I thought I’d post the end result here. Many thanks to this post for flattening the learning curve for me. It’s an old article, but I was able to reuse a good chunk of the code it presents. And the ability to call it with --revision is really helpful for testing.

The tricky bit was dealing with our server’s slightly odd project layout. We have several repos that contain what amount to ‘sub-repos’, i.e., multiple directories that contain the conventional branches, tags and trunk sub-folders. We only wanted to enable the commit check on a subset of these ‘sub-repos’, so some futzing was required to do this filtering. I also made the script as generic as possible so that new types of checks can be added in the future. (The commented-out check_py_files_contain_no_tabs() call is an artifact of my proving to myself that I could, in fact, run a ‘battery’ of checks, if desired.)

First, here’s the base pre-commit script that calls the main Python script:

#!/bin/bash
REPOS="$1"
TXN="$2"
SVNLOOK=/usr/local/bin/svnlook
/usr/local/bin/python /repo/my-project/hooks/pre-commit.py "$REPOS" "$TXN" > /dev/null || exit 1
exit 0

This calls the following:

#!/usr/bin/env python

#  Folders in this set will be subject to commit sanity-checks.
#  Update as needed...
COMMIT_CHECK_FOLDERS = frozenset([
    'SubProject-A/branches',
    'SubProject-A/tags',
    'SubProject-A/trunk',
    'SubProject-B/branches',
    'SubProject-B/tags',
    'SubProject-B/trunk',   
])

def capture(cmd):
    """Capture a command's standard output."""
    import subprocess
    return subprocess.Popen(
        cmd.split(), stdout=subprocess.PIPE
    ).communicate()[0]

def check_log_msg_contains_issue(look_cmd):
    """Check whether the commit message references an issue

    Returns 0 if the commit message contains an issue reference
    (e.g., '[NL-1234]'), otherwise returns 1

    """
    import re
    msg = capture(look_cmd.format("log"))
    pat = r'\[[A-Z]{2,8}-\d{1,6}\]'
    if re.search(pat, msg) is None:
        sys.stderr.write(
            '----------------------------------------'
            '----------------------------------------\n'
            'Please include an issue number in your commit message. (If no '
            'issue is relevant,\njust include "[NL-0000]" somewhere in the '
            'message...)\n'
        )

        return 1
    else:
        return 0

def files_updated(look_cmd):
    """ List the files touched by the current transaction.

    The `svnlook changed` can be pretty esoteric, looking something like:

    A  + moved_dir
    M  + moved_dir/README
    D    stuff/fish.c
    A    stuff/loot/bloo.h
    C    stuff/loot/lump.c
     C   stuff/loot/glub.c
    R    xyz.c
    U    trunk/file1.cpp
    A    trunk/file2.cpp

    The file status info is guaranteed to be only 4 characters wide, though,
    so we can get the file list just by snipping off the first four
    characters...

    """
    def filename(line):
        return line[4:]

    return [
        filename(f)
        for f in capture(look_cmd.format("changed")).split("\n")
        if f  #  0:
        sys.stderr.write(
            '----------------------------------------'
            '----------------------------------------\n'
            "Please remove TABs from these files before committing:"
            "\n  {0}\n".format(
                "\n  ".join(py_files_with_tabs)
            )
        )
    return len(py_files_with_tabs)

def main():
    retval = 0
    usage = """
    %prog <SVN_REPO_PATH> <TRANSACTION_ID>

    Run pre-commit options on a repository transaction.
    """

    from optparse import OptionParser
    from os.path import dirname
    parser = OptionParser(usage=usage)
    parser.add_option(
        "-r", "--revision",
        help="Test mode [TRANSACTION_ID actually refers to a revision]",
        action="store_true",
        default=False
    )
    try:
        (opts, args) = parser.parse_args()
        if len(args) != 2:
            parser.print_help()
            sys.exit(1)

        #  The `look_cmd` var below is a string-formatting template containing
        #  a '{0}' placeholder for whichever svnlook sub-command needs to be
        #  invoked.  It eventually winds up expanding to things like:
        #
        #    svnlook cat <repos_path> --transaction  
        #
        #  or:
        #
        #    svn log <repos_path> --revision 
        #
        # etc, etc...
        #
        repos, txnum_or_revnum = args
        if opts.revision:
            look_opt = "--revision"
        else:
            look_opt = "--transaction"

        look_cmd = "/usr/local/bin/svnlook {0}" + " {0} {1} {2}".format(
            repos, look_opt, txnum_or_revnum
        )

        #  Run our commit checks if *any* of the files touched by this commit
        #  reside in (sub-)folders listed in COMMIT_CHECK_FOLDERS
        if commit_touches_checked_folders(look_cmd):
            retval += check_log_msg_contains_issue(look_cmd)
            #retval += check_py_files_contain_no_tabs(look_cmd)

    except Exception as e:
        import traceback
        sys.stderr.write("Unhandled exception in pre-commit script: ")
        traceback.print_exc()
        retval += 1

    return retval

if __name__ == "__main__":
  import sys
  sys.exit(main())

All in all, I’m pretty happy with the result. Adding the catchall except clause that prints a traceback turned out to be a nice debugging aid. (And I needed one, too!)

I recently came across this  blog post by Bill Hoffman of Kitware about parallel builds with CMake. Of particular interest were these utilities:

• Ninja
• jom

CMake will work with both of these. Bill also mentions that the command:

    cmake -–build . –-config Debug

can be used to build an IDE-based project in preference to, say, msbuild. This ought to be a big help dealing with the various combinations of Visual Studio command line build tools. (I only verified that this works with Visual C++ 2010 Express, but it looks like it should work fine with the other versions.)

Finally, this build tool (mentioned in the Ninja docs) looks quite interesting:

• https://github.com/apenwarr/redo

Remember this: Eclipse gets heartburn if you set LIBRARY_OUTPUT_PATH to ${CMAKE_BINARY_DIR}, so disable this temporarily if you want to go spelunking around a CMake-based project in Eclipse. Failure to do will cause the IDE to spit out a confusing error message along the lines of:

Error processing changed links in project description file.   Cannot create a link to
[blah-blah] because it overlaps the location of the project that contains the linked resource.

That one gets me every time, since I use Eclipse so infrequently. Now it won’t any more. (Note: I think this caveat applies to EXECUTABLE_OUTPUT_PATH as well…)

Holy @#?!, what a PITA. Thankfully, someone (a certain ‘schmichael’) had already gone through the pain and suffering of figuring this all this out. All I had to do was clone his fork and build/install it to $(HOME). Then I ran:

% cd ~/projects/virtual_envs
% python2.6 virtualenv-1.9.1/virtualenv.py --distribute mysql_sand
% source mysql_sand/bin/activate
(mysql_sand)% pip install mysql-python

Then, I was finally able to run Python scripts that use the MySQL bindings. Geesh…

This is a really nice (and thorough) cross-compilation walk-through:

• http://www.kitware.com/blog/home/post/426

I might need to build a(nother) ARM toolchain soon, and this will save a lot of finger-mumbling…

At some point, these might prove useful:

• http://stackoverflow.com/questions/11484700/python-example-for-reading-multiple-protobuf-messages-from-a-stream
• http://code.google.com/p/spinn3r-client/wiki/Protostream
• https://groups.google.com/forum/?fromgroups=#!topic/protobuf/xgmUqXVsK-o

It boils down to: how can I implement CodedInputStream in Python. (Why I might want to is another story, too long to tell right now…)

It turns out to be kind of a nuisance to sum a column of numbers in the shell. I needed to calculate the total ‘raw’ file size of a file tree, which meant I needed to not count the 4096 bytes (on Linux) for the directory inodes. I could easily get the file sizes with:

% find -type f | xargs du -b | cut -f1

But how to sum the output of this? This SO post helped me figure out that the paste utility is my friend in this case:

% find . -type f | xargs du -b | cut -f1 | paste -sd+ | bc

Nice. How did I make it this many years without knowing about paste? `:-| (There’s probably some option to du that will do the same thing, but I couldn’t manage to figure it out—and now I don’t need to…)

We’re considering using D-Bus as the primary IPC mechanism for a new product at work, and we need to get and idea of how complete its Windows support is. So I don’t forget, here’s how I managed to build it on a Windows 7 box at home using Visual Studio 10 Express:

1. Download and install expat_win32.
2. Clone the D-Bus git repo:

   c:\d>cd projects
   c:\d\projects>git clone git://anongit.freedesktop.org/dbus/dbus    

3. In a Windows command prompt, make a build folder somewhere and run these commands (or something similar):

   c:\d\projects>mkdir build\dbus && cd build\dbus
   c:\d\projects\build\dbus>"C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.Cmd" /x86 /Release
   c:\d\projects\build\dbus>cmake -DEXPAT_INCLUDE_DIR="C:/Program Files (x86)/Expat 2.1.0/Source/lib" ^
     -DEXPAT_LIBRARY="C:/Program Files (x86)/Expat 2.1.0/Bin/libexpat.lib" ^
     ..\..\dbus\cmake

   The SetEnv.Cmd script may live elsewhere, depending on your flavor of Windows and Visual Studio. Season to taste.

4. Type nmake to build everything. (For me, it built without errors.)

Unfortunately, the tests didn’t all pass. When I ran the command suggested in the README.win file, I got got crash pretty quickly:

c:\d\projects\build\dbus>bin\dbus-test.exe c:\d\projects\dbus\test\data
Test data in c:\d\projects\dbus\test\data
dbus-test: running string tests
dbus-test: checking for memleaks
dbus-test: running sysdeps tests
dbus-test: checking for memleaks
dbus-test: running data-slot tests
dbus-test: checking for memleaks
dbus-test: running misc tests
dbus-test: checking for memleaks
dbus-test: running address tests
dbus-test: checking for memleaks
dbus-test: running server tests
dbus-test: checking for memleaks
dbus-test: running object-tree tests
dbus-test: checking for memleaks
dbus-test: running signature tests
dbus-test: checking for memleaks
dbus-test: running marshalling tests
dbus-test: checking for memleaks
dbus-test: running marshal-recursive tests
>>> >>> Each value by itself 120 iterations
 0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 120 this test (120 cumula
tive)
>>> >>> Each value by itself with arrays as blocks 120 iterations
 0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 120 this test (240 cumula
tive)
>>> >>> All values in one big toplevel 1 iteration

A Window popped up saying “dbus-test.exe has stopped working,” after that.  Not sure what that’s about. It’s possible that I’m just missing some setup steps, or that this crash is a known-but-harmless bug. Or the current HEAD on master is busted. It could be a lot of things. I’ll post updates as I figure it out. I’m just really encouraged that Windows support has been folded into the main development branch!

I keep needing to refer to this example, so I may as well put it here for safe-keeping. It’s not perfect, but it kinda/sort gives a feel for how the dbus-python bindings work:

import sys
import dbus

if __name__ == '__main__':
    from os.path import basename
    if len(sys.argv) != 2:
        print "Usage: {0} ".format(basename(sys.argv[0]))
        sys.exit(1)

    devname = sys.argv[1]
    bus = dbus.SystemBus()
    obj = bus.get_object("org.freedesktop.UDisks", "/org/freedesktop/UDisks")
    iface = dbus.Interface(obj, 'org.freedesktop.UDisks')
    devices = iface.get_dbus_method('EnumerateDevices')()

    objpath = "/org/freedesktop/UDisks/devices/" + devname
    if dbus.ObjectPath(objpath) not in devices:
        print "Device '{0}' does not appear to exist...".format(devname)
        sys.exit(1)

    obj = bus.get_object("org.freedesktop.UDisks", objpath)
    iface = dbus.Interface(obj, 'org.freedesktop.UDisks.Device')
    files = iface.get_dbus_method('FilesystemListOpenFiles')()
    for fdata in files:
        if fdata[2]: print fdata[2].decode()