While running the tests three times might make a lot of sence in CI
environments, it certainly slows things down in development, especially
when you've failed a test on purpose, but still have to wait for it to
run three times to get the output.
This not only implements the `--retries` flag, but also activates it in
both CircleCI and AppVeyor test suites.
While this was necessary in previous versions of Meteor, as of 1.6 the
`self-test` tooling will automatically install the dependencies it needs
at runtime.
Not only does this allow the installation to be avoided when tests using
those facilities aren't invoked, it also allows us to reliably install
the same version of these npms, rather than falling back to the
`latest` npm tag on each CI run.
Currently, three 'dynamic import' tests exist in the self-test suite:
* dynamic import(...) in development
* dynamic import(...) in production
* dynamic import(...) with cache
These tests take quite some time to run; each test takes ~5 minutes.
While it's great to test them in a faster environment (Unix), which
Meteor will continue to do, the overall value of the three separate
tests is not as great in a slower testing environment, like Windows.
Even with $ErrorActionPreference, PowerShell won't automatically fail
when an external command fails with an error code. This explicitly
checks those exit codes and throws an error when that occurs.
Hopefully prevents false successes like that shown in this AppVeyor
test run: https://goo.gl/xxRsF9.
Though it was thought to be reliable when running through 'self-test' on
Windows, it's yet to be seen how reliable.
The worst thing that could come of adding Windows testing would be that
we have test failures again after such a string of green checkmarks and
confidence.
Also, reordered.
This implements a first generation of Windows CI testing. Presently,
this only runs valuable, hand-picked tests which have been known to work
in the past, and whose failure would indicate a critical problem.
A test which isn't passing doesn't mean that the feature being tested is
not working. For example, the 'create' test fails ostentatiously,
though the 'meteor create' command certainly works in practice. This
points to problems some compatibility problems with the 'self-test'
harness itself, some of which I'm aware of. Though, it likely will
highlight some legitimate problems which Windows users experience too.
There are a number of additional tests which should be enabled which
likely pass already, and many more which are failing and we should fix.
Additional tests can be added to the scripts/windows/appveyor/test.ps1
file as they've been deemed working.
Altogether, this will take extensive work to achieve the same level of
coverage our Unix test suite enjoy, but we've got to start somewhere!
cc @benjamn
