* Add a self-test skip option
Meteor's CI infrastructure is configured to exclude certain
`self-test`'s on each run. These excludes are specified in
each CI environment's config file, and included when running
`meteor self-test`. Developers running `meteor self-test`
locally however are not using these excludes by default,
so developer's have to manually look up the current exclude
list from one of the CI configs, then add these excludes to
their own `meteor self-test` call manually.
This commit adds a new `skip` option to Meteor's `self-test`
system, that can be used to skip adding/running a defined
`self-test` (similar in concept to Mocha's `skip` feature).
This provides a way to skip the running of older
`self-test`'s that are no longer needed, but allows them to
be preserved in the `self-test` suite, for future reference.
With this functionality in place, and the older test suites
updated to use it, Meteor's base CI excludes no longer need
to be maintained in their respective config files. The
excludes are all managed at the source (the test definition),
and can be leveraged by anyone/anything calling
`meteor self-test`.
* Log message describing skipped test
* Add manually-ignored count to self-test summary
* Small comment correction
* History.md entry with PR link
Store the result of any npm error logs which may prove useful
during investigation of random Windows failures, exhibited in
the Windows `dynamic-import` tests during npm installation:
https://ci.appveyor.com/project/meteor/meteor/build/368
AppVeyor does provide us quite a bit of value at no cost since we're
open-source software, but with only a single container at our disposal
under that plan, our build queue is often maxed-out for hours.
Since most Meteor developers are using 64-bit platforms, the 64-bit
tests provide the most value. Meteor doesn't have much (if any?) in the
way of 32-bit/64-bit exceptions so any functionality difference would be
at the OS level. Some day, we might bring this back.
These take precedence over the UI, and I'm not sure the UI is taking
effect at the moment.
We don't need to build branches which start with 'dev-bundle-' since
those dev bundles won't be built yet when the tests are kicked off.
This implements a first generation of Windows CI testing. Presently,
this only runs valuable, hand-picked tests which have been known to work
in the past, and whose failure would indicate a critical problem.
A test which isn't passing doesn't mean that the feature being tested is
not working. For example, the 'create' test fails ostentatiously,
though the 'meteor create' command certainly works in practice. This
points to problems some compatibility problems with the 'self-test'
harness itself, some of which I'm aware of. Though, it likely will
highlight some legitimate problems which Windows users experience too.
There are a number of additional tests which should be enabled which
likely pass already, and many more which are failing and we should fix.
Additional tests can be added to the scripts/windows/appveyor/test.ps1
file as they've been deemed working.
Altogether, this will take extensive work to achieve the same level of
coverage our Unix test suite enjoy, but we've got to start somewhere!
cc @benjamn
