* Add a self-test skip option
Meteor's CI infrastructure is configured to exclude certain
`self-test`'s on each run. These excludes are specified in
each CI environment's config file, and included when running
`meteor self-test`. Developers running `meteor self-test`
locally however are not using these excludes by default,
so developer's have to manually look up the current exclude
list from one of the CI configs, then add these excludes to
their own `meteor self-test` call manually.
This commit adds a new `skip` option to Meteor's `self-test`
system, that can be used to skip adding/running a defined
`self-test` (similar in concept to Mocha's `skip` feature).
This provides a way to skip the running of older
`self-test`'s that are no longer needed, but allows them to
be preserved in the `self-test` suite, for future reference.
With this functionality in place, and the older test suites
updated to use it, Meteor's base CI excludes no longer need
to be maintained in their respective config files. The
excludes are all managed at the source (the test definition),
and can be leveraged by anyone/anything calling
`meteor self-test`.
* Log message describing skipped test
* Add manually-ignored count to self-test summary
* Small comment correction
* History.md entry with PR link
Much of the infrastructure to run "self test" tests (those which test
core functionality of the `meteor` tool itself) in PhantomJS and
BrowserStack was already in place, though the BrowserStack portion had
been disabled some time ago, though the exact reason isn't entirely
clear.
BrowserStack could play an important roll in Meteor's future as Meteor
works to ensure that the bundle delivered to the client is appropriate
for that client's capabilities, including appropriate polyfills to
implement functionality not natively available in the browser (e.g.
ECMAScript features, SockJS, etc.).
Since tests themselves already keep track of their failure, and the test
list (i.e. the `TestList` class) tracks other concepts of grouped tests,
such as "all tests", "filtered tests", etc. (like a `TestList` class
would!) I believe it makes sense for it to also maintain a collection of
"failed tests"... (in list form, of course!)
This will allow the `Run.runTest` logic to move into `run.js` file without
needing to pass back a separate `failed` variable in the event of a
failure, since that is already being tracked through the `notifyFailed`
facility.
While running the tests three times might make a lot of sence in CI
environments, it certainly slows things down in development, especially
when you've failed a test on purpose, but still have to wait for it to
run three times to get the output.
This not only implements the `--retries` flag, but also activates it in
both CircleCI and AppVeyor test suites.
The single file which represented the bulk of the `meteor self-test`
functionality had got a bit heavy and it stood to benefit from some
dissemination.
I embarked on this change originally when looking into replacing
PhantomJS with Chrome Headless (and a new `ChromeClient` class) within
self-test. Unfortunately, I didn't have time to take this the last
step of actually implementing Chrome, but this should hopefully
facilitate that change in the future by providing what I believe to be
better compartmentalization of this logic.
I apologize for the (likely) difficulty of reviewing this commit. Due
to heavy intertwining of existing code it was hard to arrange these
changes in an easy-to-review manner. I believe a reviewer will
find that it's mainly copy and pasting into different files and
careful adjusting of those files (new) module dependencies.
By my calculations, the sum of the sizes of the individual isopackets was
152MB, and the size of the combined isopacket is now just 36MB. That
remarkable difference goes to show how much duplication of transitive
dependencies was happening before this change.
That's a savings of 116MB for the (uncompressed) size of the meteor-tool
package. In Meteor 1.5.x, the meteor-tool package is about 544MB, but in
Meteor 1.6 it's considerably smaller: 373MB. In other words, this change
should reduce those sizes to 428MB (-21%) and 257MB (-31%), respectively.
* Auto-install `meteor self-test` dependencies upon use.
This uses the same new facilities which were created for auto-installing
Cordova (#8976) to also auto-install PhantomJS and BrowserStack WebDriver npms
into their appropriate home in the dev bundle when they're needed for running
self-tests.
* Use a more descriptive name for the reference to the `require`-d npm module.
