Continuous integration in python using watchdog

watchdog is a cross-platform python module for watching directories for changes.

I use watchdog to run continuous integration on my projects: every time a file changes (e.g. is saved, deleted, or created) in the directory I am watching, a script will automatically run unit tests, compile libraries, build docs, and run other tests, as necessary.

Below is a sample script using watchdog that monitors the current directory for changes to .py files and .rst files. When a .py file changes, it runs unit tests; when a .rst file changes, it builds the documentation. Here, I assume that the documentation is built using Sphinx, and is located in the `docs` subdirectory. `python -m unittest discover -b` is used to run the unit tests.

Of course, you could perform any actions you wanted, using any triggers you choose.

The first thing that you need to do is define an event handler based on watchdog.FileSystemEventHandler:

import os
from watchdog.events import FileSystemEventHandler

def getext(filename):
    "Get the file extension."

    return os.path.splitext(filename)[-1].lower()

class ChangeHandler(FileSystemEventHandler):
    """
    React to changes in Python and Rest files by
    running unit tests (Python) or building docs (.rst)
    "
""

    def on_any_event(self, event):
        "If any file or folder is changed"

        if event.is_directory:
            return
        if getext(event.src_path) == '.py':
            run_tests()
        elif getext(event.src_path) == '.rst':
            build_docs()
 

Here, I'm catching all file events, and reacting to changes to .py and .rst (reStructured text) files.

The next thing is to create a watchdog.Observer, and register your event handler.

import os
import time

from watchdog.observers import Observer

BASEDIR = os.path.abspath(os.path.dirname(__file__))

def main():
    """
    Called when run as main.
    Look for changes to code and doc files.
    "
""

    while 1:
   
        event_handler = ChangeHandler()
        observer = Observer()
        observer.schedule(event_handler, BASEDIR, recursive=True)
        observer.start()
        try:
            while True:
                time.sleep(1)
        except KeyboardInterrupt:
            observer.stop()
        observer.join()

if __name__ == '__main__':
    main()
 

Finally, define your actions. Here are mine for running unit tests and building my docs.

import os
import sys
import subprocess
import datetime
import time

BASEDIR = os.path.abspath(os.path.dirname(__file__))

def get_now():
    "Get the current date and time as a string"
    return datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S")

def build_docs():
    """
    Run the Sphinx build (`make html`) to make sure we have the
    latest version of the docs
    "
""

    print >> sys.stderr, "Building docs at %s" % get_now()
    os.chdir(os.path.join(BASEDIR, "docs"))
    subprocess.call(r'make.bat html')

def run_tests():
    "Run unit tests with unittest."

    print >> sys.stderr, "Running unit tests at %s" % get_now()
    os.chdir(BASEDIR)
    subprocess.call(r'python -m unittest discover -b')
 

Here is the full code listing:

"""
Monitors our code & docs for changes
"
""

import os
import sys
import subprocess
import datetime
import time

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

BASEDIR = os.path.abspath(os.path.dirname(__file__))

def get_now():
    "Get the current date and time as a string"
    return datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S")

def build_docs():
    """
    Run the Sphinx build (`make html`) to make sure we have the
    latest version of the docs
    "
""

    print >> sys.stderr, "Building docs at %s" % get_now()
    os.chdir(os.path.join(BASEDIR, "docs"))
    subprocess.call(r'make.bat html')

def run_tests():
    "Run unit tests with unittest."

    print >> sys.stderr, "Running unit tests at %s" % get_now()
    os.chdir(BASEDIR)
    subprocess.call(r'python -m unittest discover -b')

def getext(filename):
    "Get the file extension."

    return os.path.splitext(filename)[-1].lower()

class ChangeHandler(FileSystemEventHandler):
    """
    React to changes in Python and Rest files by
    running unit tests (Python) or building docs (.rst)
    "
""

    def on_any_event(self, event):
        "If any file or folder is changed"

        if event.is_directory:
            return
        if getext(event.src_path) == '.py':
            run_tests()
        elif getext(event.src_path) == '.rst':
            build_docs()

def main():
    """
    Called when run as main.
    Look for changes to code and doc files.
    "
""

    while 1:
   
        event_handler = ChangeHandler()
        observer = Observer()
        observer.schedule(event_handler, BASEDIR, recursive=True)
        observer.start()
        try:
            while True:
                time.sleep(1)
        except KeyboardInterrupt:
            observer.stop()
        observer.join()

if __name__ == '__main__':
    main()
 

You can download the code here.

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>