Microsoft’s Team Data Science Process Documentation has a lot of info/opinions on how to do data science with Azure. That also includes quite a general setup for testing Python code using custom Docker containers and a Kubernetes orchestrator.

However, for Python there is also a more “as a Service”-like solution that I will cover here.

Making a Python package

Python is not a core language for me, so I have relied on the official docs for making modules and packages as well as the popular post Packaging a python library.

I have made a small Python package that runs on both Python 2.7 and 3.6. The package is called vsts and is in a folder called vsts. The package is very simple, containing only one module for doing simple arithmetic:

vsts
├── run_tests.ps1
├── setup.py
├── tests
│   └── test_arithmetic.py
└── vsts
    ├── arithmetic.py
    └── __init__.py

The arithmetic module is also very simple, containing only two functions:

def add(x, y):
    """
    Return the sum of two numbers
    """
    return(x + y)

def subtract(x, y):
    """
    Return the difference of two numbers
    """
    return(x - y)

The __init__.py file imports these two functions and make them available directly in the vsts package:

from .arithmetic import add, subtract

Finally, there are two simple tests in test_arithmetic.py:

import vsts

def test_add():
    assert vsts.add(1, 1) == 2

def test_substract():
    assert vsts.subtract(1, 1) == 0

The tests are for the pytest framework. I use pytest because it can write the test results in formats that VSTS can import. Code coverage is computed with the package pytest-cov.

The setup.py script contains information for making the actual package (https://docs.python.org/3/distutils/setupscript.html):

#!/usr/bin/env python
# -*- encoding: utf-8 -*-

from setuptools import setup

setup(name='vsts',
      version='0.1.0',
      description='Demonstrating automated testing on VSTS',
      author='Robert Dahl Jacobsen',
      author_email='foo@bar.baz',
      license='MIT',
      packages=['vsts'],
      tests_require=['pytest', 'pytest-cov'],
      zip_safe=False)

The file run_tests.ps1 is covered in the VSTS section

Test locally

To run the tests locally, navigate to the root directory of the package and execute the following (in PowerShell on Windows or any shell on *nix):

python -m pytest tests --cov=vsts

$ python -m pytest tests --cov=vsts
======================== test session starts ========================
platform linux2 -- Python 2.7.12, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /home/robert/Documents/python/vsts, inifile:
plugins: cov-2.5.1
collected 2 items

tests/test_arithmetic.py ..                                   [100%]

---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name                 Stmts   Miss  Cover
----------------------------------------
vsts/__init__.py         1      0   100%
vsts/arithmetic.py       4      0   100%
----------------------------------------
TOTAL                    5      0   100%


===================== 2 passed in 0.03 seconds ======================

All tests run and all non-commented lines lines in __init__.py and arithmetic.py are executed during the tests.

Test in VSTS

In VSTS the project repository is called vsts.py. Testing in VSTS is found under the Build & Release tab. Create a new Build Definition under Builds. The bare minimum for a compiled language is a build, but we can also add more steps. Here are the steps I use:

Continuous integration is enabled under the Triggers tabs.

In Phase 1 I use a “Hosted VS2017” under “Agent queue”. This executes the code on a computer that I don’t have to maintain and thus eliminates the need for creating and running the Docker containers mentioned in the beginning. Note that only selected languages are available on the hosted solutions.

The PowerShell Script step is a custom build. PowerShell scripts can be executed in two ways: As an inline script where you write the script directly in the build definition. This is an easy solution for small scripts, but the script itself is not version controlled. The other option is to include the script in the repository, which is what I use here in run_tests.ps1:

python -m pip install .
python -m pip install -U pytest pytest-cov

python -m pytest tests --junit-xml=test-results.xml --cov=vsts --cov-report=xml:coverage.xml

By default the path is that of the script, but this can be changed in the “Advanced” section or programmatically through the build variable Build.SourcesDirectory.

First we test if the package actually installs correctly. The packages listed in tests_require in setup.py are not installed with the first pip command and are therefore installed manually.

The last line that run the tests reads as follows: Run the test scripts in the tests folder and write the results in the JUnit XML format in the file test-results.xml; compute code coverage for the folder vsts and write the results as coverage.xml.

When choosing “xml” for the cov-report it is in the Cobertura format. See all options with pytest --help.

Sometimes warnings (such as notification about an outdated pip) are written to the PowerShell error stream:

So even though your test scripts runs, no results are collected in the subsequent tasks. To circumvent this the box “Fail on Standard Error” must be unchecked in the Advanced section of PowerShell Script.

After running the PowerShell script for testing, we import test results and import code coverage into VSTS.

For the test results I use the default values:

• Test result format: JUnit.
• Test result files: **\TEST-*.xml

For the code coverage I use these values:

• Code coverage tool: Cobertura.
• Summary file: $(Build.SourcesDirectory)\coverage.xml

After running a test we get an output that looks like the following: