Skip to content

Speed up CI tests

Javier Duarte requested to merge parallel-tests into master

Created by: thesps

As we've been adding more tests, the CI testing takes longer to run. The timing varies a lot, but to begin with it would take ~15-20 minutes, while recently with more tests it's taking ~20-30 minutes. This PR speeds things up a bit by submitting each test_*.py file as a separate CI job, which can be assigned to different workers. Previously we just ran all pytests in one job, so each test would run sequentially. As we continue to develop more tests this should help avoid the CI really dragging.

Here the pipeline is generated dynamically: the main .gitlab-ci.yml job runs the generate_ci_yaml.py script, which creates a .yml file defining all the jobs from the available test files. That's uploaded as an artifact and used to trigger the next stage. So to add a new test file you can still just add the test_something.py file to the pytests directory and it will be picked up by the CI.

Here's a view of it running, with some jobs finished and some still going:

Screenshot 2021-09-30 at 17 03 43

The timing seems to now be in the 10-15 minute range, pretty much limited to the time it takes to run the slowest test file - usually the MNIST CNN one.

Merge request reports

Loading