Should fix#8988 and #8942.
Now that #8866 is the default behavior, it can take up to 5000ms for
changes to files modified during the build process to be noticed.
Before #8866, when we called e.g. files.writeFile(path), a native file
watcher would notice the change immediately, almost always before the
build process read the file again. This was definitely racy, but we were
getting away with it consistently... until #8866.
I was able to reproduce the problem in #8988 by running
echo some-local-package-name >> .meteor/packages
in an app with a local package of the given name. After debugging the
endless rebuild cycle, I found that .meteor/versions was being rewritten
by files.writeFile during the build process, but the file watching system
was not noticing the change in time to prevent watch.isUpToDate from
returning true. The change was finally detected when restarting the
Watcher responsible for .meteor/versions, which of course triggered
another rebuild, so the same problem kept happening again and again.
This is a bit different from the previous strategy of invalidating
optimistic functions for specific npm package names.
Now, whenever we make changes to the contents of a specific node_modules
directory, or whenever the developer independently modifies an app's
node_modules directory, all optimistic results derived from paths
contained within that node_modules directory will be marked as dirty, and
thus may need to be recomputed.
This strategy prioritizes starting fewer watchers (just one per
node_modules directory) while still allowing npm packages to be added or
removed while the app is running:
https://github.com/meteor/meteor/pull/7668#issuecomment-255120373
The drawback is that changes within subdirectories of node_modules will
not be detected until the server is fully restarted, but that seems like
an acceptable tradeoff, since npm packages change much less often than
application code.
Healthy competition among fs.watch wrappers appears to have produced a
clear winner: https://www.npmjs.com/package/chokidar
This wrapper is better for Meteor than pathwatcher was, because it can
watch directory trees recursively, and it has no trouble watching
nonexistent file paths, whereas pathwatcher would throw an exception.
As of optimism@0.3.0, the value of `this` within subscribe functions is no
longer the optimistic wrapper function object (instead: null or global, as
if called with no receiver object), so we need to retain a reference to
the optimistic function so that the watcher can call .dirty(...args).
I originally added the exception-caching functionality in order to avoid
calling files.stat repeatedly for files known to be missing, but now that
we're using statOrNull, knowledge of missing files (as indicated by
statOrNull returning null) is being properly cached.
The reason it's dangerous to cache exceptions is that (for example) when
an ENOENT exception indicates the file is missing, there will be no more
change events for that file, effectively making the exception permanent,
even if the file comes to exist at a later time.
