Flaky specs

Automated Tests which do not pass consistently: they fail sometimes (flaky)

Spotted a flaky spec?

Flaky specs can be spotted in different ways. Often, these are seen after opening a pull request or after a merging one: in both these events the build is ran (as determined by our GitHub Actions configuration) and, at times, we can see a failing spec which passes upon a second or (several!) build re-runs.

If this happens, we need to register it so it can be fixed and track re-occurrences - we take the following steps, to do so:

• make sure the failing example is really flaky. Re-run the build a couple of times or follow the process below for diagnosing a flaky spec locally

• make a quick search on open and closed issues, to make sure the failure you're observing is really new; if that's the case, then move to the next step. If there already is an existing:

  • open issue - it's good to comment on the issue, pasting the build run in which you've observerd the most recent failure

  • closed issue - it may be sufficient to re-open a previously exiting issue, if the context is useful.

• Ok, so it's a new flaky spec 🙈 We need to open a tech-debt issue on our main repo:

  • the title of the issue should contain [Flaky] as well as the relative path to the test file containing the flaky example, for example:

  [Flaky] ./spec/system/admin/enterprise_fees_spec.rb

  • Make sure the issue you've opened indicates:

    • the line number for the failing example

    • the link to the build run, in which the flaky spec occurred

• Give a quick notice on the #testing channel, that a new flaky spec was spotted.

Diagnosing flaky tests

We have a nice script for this, which makes it easy for testers to run a spec or example repeatedly, using all available CPU threads and tracking a generated log file ✨

Run the script

The script can be run on your local repo folder, by following the syntax:

./script/rspec-slow-repeat <number of times to run the spec> <spec file or example>

for example:

script/rspec-slow-repeat 100 ./spec/system/admin/configuration/taxonomies_spec.rb:45

Here we run the context on line 45 of the taxonomies_spec.rb, on hundred times.

It will display passing runs with Pass., failing runs with !!! Fail !!! and a pass percentage when it finishes, something like 97 of 100 passed (97%). However, it will not display which examples are failing and why they are failing.

To find out, we'll need to look at the gererated log file.

Track the output with the log

The script above will generate a log file, in which the RSpec output is recorded. The log file is located at ./tmp/rspec.log .

Catching failing specs like a pro 💪

Open two terminal windows:

• On the left window run the script following the indicated syntax

• On the right window track the log file, by running tail -f ./tmp/rspec.log

Now you can see the specs passing/failing 🍿

Happy debugging!