> For the complete documentation index, see [llms.txt](https://ofn-user-guide.gitbook.io/ofn-testing-handbook/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ofn-user-guide.gitbook.io/ofn-testing-handbook/testers-toolkit/flaky-specs.md).

# Flaky specs

Automated Tests which do not pass consistently: they fail sometimes (flaky)

## Spotted a flaky spec?

Flaky specs can be spotted in different ways. Often, these are seen after opening a pull request or after a merging one: in both these events the build is ran (as determined by our GitHub Actions configuration) and, at times, we can see a failing spec which passes upon a second or (several!) build re-runs.

If this happens, we need to register it so it can be fixed and track re-occurrences - we take the following steps, to do so:

* make sure the failing example is really flaky. Re-run the build a couple of times or follow the process below for diagnosing a flaky spec locally
* make a quick search on open and closed issues, to make sure the failure you're observing is really new; if that's the case, then move to the next step. If there already is an existing:
  * open issue - it's good to comment on the issue, pasting the build run in which you've observerd the most recent failure
  * closed issue - it may be sufficient to re-open a previously exiting issue, if the context is useful.
* Ok, so it's a new flaky spec :see\_no\_evil: We need to open a tech-debt issue on our main repo:

  * the title of the issue should contain `[Flaky]` as well as the relative path to the test file containing the flaky example, for example: 

  [\[Flaky\] ./spec/system/admin/enterprise\_fees\_spec.rb](https://github.com/openfoodfoundation/openfoodnetwork/issues/11330)

  * Make sure the issue you've opened indicates:
    * the line number for the failing example
    * the link to the build run, in which the flaky spec occurred
* Give a quick notice on the #testing channel, that a new flaky spec was spotted.

## Diagnosing flaky tests

We have a [nice script](https://github.com/openfoodfoundation/openfoodnetwork/blob/master/script/rspec-slow-repeat) for this, which makes it easy for testers to run a spec or example repeatedly, using all available CPU threads and tracking a generated log file :sparkles:

### Run the script

The script can be run on your local repo folder, by following the syntax:

`./script/rspec-slow-repeat <number of times to run the spec> <spec file or example>`

for example:

`script/rspec-slow-repeat 100 ./spec/system/admin/configuration/taxonomies_spec.rb:45`

Here we run the context on line 45 of the `taxonomies_spec.rb`, on hundred times.

It will display passing runs with `Pass.`, failing runs with `!!! Fail !!!` and a pass percentage when it finishes, something like `97 of 100 passed (97%).` However, it will not display which examples are failing and why they are failing.&#x20;

To find out, we'll need to look at the gererated log file.

### Track the output with the log

The script above will generate a log file, in which the RSpec output is recorded. The log file is located at `./tmp/rspec.log` .

### Catching failing specs like a pro :muscle:

Open two terminal windows:

* On the left window run the script following the indicated syntax
* On the right window track the log file, by running `tail -f ./tmp/rspec.log`

Now you can see the specs passing/failing :popcorn:

![](https://88588569-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7NnFH2urYVKG6zlEtN%2Fuploads%2FzKKTGTH1xDCvvdYYr9bb%2Fimage.png?alt=media\&token=5f20fb63-415a-4038-8bdb-417c9cbfe050)

Happy debugging!