From 8d71330158fa8ff79b4061b7b7af35dd20c299e4 Mon Sep 17 00:00:00 2001 From: Christopher McCulloh Date: Tue, 13 Jun 2017 15:55:37 -0400 Subject: [PATCH] (GH1986) sets threshold for regression test errors to eight pixels --- test/REGRESSION_TESTING.md | 11 +++++++++++ test/mocha.js | 4 +++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/test/REGRESSION_TESTING.md b/test/REGRESSION_TESTING.md index 90bd7def..0c72b05e 100644 --- a/test/REGRESSION_TESTING.md +++ b/test/REGRESSION_TESTING.md @@ -1,3 +1,4 @@ +# Overview We use [Niffy](https://segment.com/blog/perceptual-diffing-with-niffy/) for regression testing. Niffy is built on top of Nightmare, and Mocha + Chai. Niffy compares the current code with the contents of the reference folder. @@ -10,6 +11,7 @@ The test/regression folder has a separate file for each Fuel UX asset. The regression tests will look at either the local dev code (from `js/*` or `dist/css/fuelux.css`) or will look at the reference code in `reference/dist/`. Ideally the reference code will match the last prod release. Upon Fuel UX release the code from `/dist/` is copied into `/reference/dist/`. +# Updating reference with intentional changes If a change intends to change the look of an asset, you will need: 1. Make a build (`npm build`) @@ -20,7 +22,16 @@ If a change intends to change the look of an asset, you will need: Further changes tested against this new output will now ensure that the new change does not revert. +# False Positives +Because of "visual noise" you can not set the Threshold to `0`. If you do, 2/3 of the time it will "fail" simply due to a few errant pixels of noise (around shadows or gradients etc). +Currently the visual buffer in TravisCI is set to 1024x768 (see `.travis.yml` file for this setting), which is 786,432 pixels. 0.001% of that is slightly less than eight pixels. Most visual noise we observed was about 3 pixels. Running a test on `http://localhost:8000/checkbox` where we set the top-left border-radius to 0px instead of 4px resulted in a 0.01% (79px) difference. + +If you get false positives more than 1% of the time, look at the number of pixels by which it is failing (When it fails it will tell you by what amount, something like "Error: 0.01% different, open /tmp/niffy/checkbox/diff.png") and change the threshold to the absolute minimum you can without risking false negatives. + +Do not ever set the threshold over `0.009%` as this will result in false negatives of the worst kind. You will have a change, it will tell you you don't have a change, and you won't be likely to ever notice the change unless you are specifically looking for it. Arrow buttons on spinboxes have previously had mis-rounded corners that no one noticed (but looked horrible _if_ you were actually looking at them) for over 6 months. It was just these sorts of subtle errors that we implemented visual regression testing to catch. The worst possible thing that could happen is that we set up visual regression testing intended to catch imperceptible errors and then _tell it to ignore imperceptible errors_. You have been warned. + +# Debugging Travis CI When TravisCI was erring out on Niffy, Woodward was able to run Travis locally through Docker: 1. [Install and setup Docker + Travis](https://docs.travis-ci.com/user/common-build-problems/#Troubleshooting-Locally-in-a-Docker-Image) diff --git a/test/mocha.js b/test/mocha.js index 2001e29d..188852ab 100644 --- a/test/mocha.js +++ b/test/mocha.js @@ -10,6 +10,8 @@ var server = require('./regression/server.js'); // var should = require('chai').should(); var Niffy = require('niffy'); +var EIGHT_PIXELS = 0.00102; + describe('Fuel UX 3', function testFuelUX3 () { var basehost = 'http://localhost:8000'; var testhost = 'http://localhost:8013'; @@ -20,7 +22,7 @@ describe('Fuel UX 3', function testFuelUX3 () { before(function instantiateNiffy () { devServer = server.getServer(8000); refServer = server.getServer(8013); - niffy = new Niffy(basehost, testhost, { show: true, threshold: 0 }); + niffy = new Niffy(basehost, testhost, { show: true, threshold: EIGHT_PIXELS }); }); after(function* stopAll () { devServer.close();