Commit Graph

6036 Commits

Author SHA1 Message Date
Ben Newman
6c501b0148 Respect $METEOR_DISABLE_OPTIMISTIC_CACHING for debugging purposes. 2016-10-18 14:25:07 -04:00
Ben Newman
320874989f Implement and use optimisticReadJsonOrNull for better caching.
The problem with optimisticReadFile is that it doesn't cache anything when
the file is missing, because files.readFile throws an ENOENT exception.
2016-10-18 14:05:52 -04:00
Ben Newman
6999bdd99e Revert "Use files.* methods instead of optimistic functions in meteor-npm.js."
This reverts commit b69716d778.

Now that we have a better system for invalidating optimistic results
derived from node_modules paths, these optimizations are safe (and yes
fourseven:scss rebuilds successfully).
2016-10-18 14:05:51 -04:00
Ben Newman
af626a5ed6 Dirty relevant optimistic functions after npm updates. 2016-10-18 14:05:51 -04:00
Ben Newman
30d9a57f0d Allow dirtying optimistic functions by npm package name or path. 2016-10-18 13:26:06 -04:00
Ben Newman
b69716d778 Use files.* methods instead of optimistic functions in meteor-npm.js.
This fixes a bug that prevents fourseven:scss from properly rebuilding,
because the new .../node_modules/node-sass/vendor/<platform>/binding.node
file is not found by Builder#copyDirectory, because the cached results of
optimisticReaddir are the same as before the rebuild.

Unfortunately, this change introduces a small performance regression
(hundreds of milliseconds at worst), because these files.* methods are
called many times.

I think we can continue using optimistic functions here if we are more
careful about invalidating their results, especially after calling
meteorNpm.rebuildIfNonPortable, but I'll save that for a future commit.
2016-10-18 13:02:20 -04:00
Ben Newman
90ece32013 Fix problems in request callback that broke getUrlWithResuming tests. 2016-10-17 15:19:21 -04:00
Ben Newman
30aec9f345 Report an error when HTTP request body is incomplete.
When a download aborts prematurely, the status code is often 200 OK, even
though we probably should not proceed with any further processing of the
downloaded information.

This silent failure leads to problems like the dreaded "Error: ENOENT: no
such file or directory, open... os.json" (#7806 and others), which were
hard to diagnose properly because the failure occurred only later, when
extracting a buffer that downloaded incompletely.

The getUrlWithResuming helper should be able to retry after this error is
thrown, which will result in a more helpful warning, even if in the most
common case, i.e. MaxCDN failure, it will never actually succeed.

Note that this change will not help until Meteor 1.4.2 is officially
released and becomes the implementation used to download later releases.

Mitigates #7806.
2016-10-17 13:52:53 -04:00
Ben Newman
c33c68f817 Avoid scanning app node_modules for global variable assignments.
File#computeAssignedVariables is one of the most expensive methods called
during initial startup. This change significantly reduces the number of
times it needs to call through to findAssignedGlobals, which saves quite a
bit of parsing time. The exact savings are hard to quantify, of course,
because they depend entirely on how many modules you have in your app's
node_modules directory.

Scanning for global variable assignments is only really useful in Meteor
packages, where we sometimes need to intercept assignments for the
purposes of api.export, though this is increasingly unnecessary now that
you can (and should) just import values from node_modules.

In the app, the only possible value of intercepting global variable
assignments was to prevent polluting the global scope, but we don't even
create a private scope for the app when useGlobalNamespace is true, so
there really was no point to scanning app node_modules.
2016-10-17 11:51:52 -04:00
Ben Newman
592e2d68d4 Merge branch 'devel' into release-1.4.2 2016-10-17 10:40:26 -04:00
Ben Newman
2a14061c6e Merge pull request #7919 from abernix/feature/c9s-package-dir-rename
Add METEOR_PACKAGE_DIRS support and deprecate PACKAGE_DIRS.
2016-10-17 09:50:12 -04:00
Ben Newman
7a18012975 Consult babel-runtime package when discarding helper modules.
The most fool-proof way to tell if a module is provided by babel-runtime
is to load babel-runtime as an isopacket and ask it.

This should fix any remaining issues like this one:
https://github.com/meteor/meteor/pull/7668#pullrequestreview-4379559
2016-10-16 16:40:11 -04:00
Ben Newman
8d4ef3f1f7 Only filter out node_modules/babel-runtime/{helpers,regenerator} modules.
This is a refinement of my previous commit 56c041c858.

The Meteor babel-runtime package does not provide substitutes for
babel-runtime/core-js/* modules, so we should not be discarding them from
the client bundle.

Fixes #7930.
2016-10-15 12:01:02 -04:00
Ben Newman
e94d2ad4fc Remove references to node_modules "reference-directly" mode. 2016-10-14 17:01:47 -04:00
Ben Newman
91058b53a7 Allow (junction) symlinking node_modules on Windows.
Hopefully fixes #7830 by ensuring that server node_modules are physically
present in the programs/server/npm directory.
2016-10-14 16:29:35 -04:00
Ben Newman
c572b12f26 Allow Builder#copyDirectory to make junction links on Windows. 2016-10-14 16:25:07 -04:00
Ben Newman
9039f98b30 Make findMongoPort return null if net.connect throws. 2016-10-14 16:24:27 -04:00
Ben Newman
56c041c858 Filter out node_modules/babel-runtime/* when Meteor provides babel-runtime.
Should fix #7872 and others (e.g. #7833).
2016-10-13 19:24:34 -04:00
Ben Newman
5de9070c9c Implement ImportScanner#isWeb convenience method. 2016-10-13 19:24:33 -04:00
Ben Newman
7ea95063b0 Ignore non-top-level identifiers early in Resolver#_resolveNodeModule. 2016-10-13 19:24:33 -04:00
Ben Newman
ec34a4ecc6 Fix tests by making sure to close optimistic file watchers.
If a test process does not explicitly call process.exit, pathwatcher
watchers may keep it alive indefinitely (either that, or there's a bug
with the persistent:false option to fs.watchFile).

This accidental immortality can be prevented by explicitly closing all
watchers when we no longer have any interest in file change notifications.
2016-10-13 14:09:16 -04:00
Jesse Rosenberger
815e32e68b Add METEOR_PACKAGE_DIRS support and deprecate PACKAGE_DIRS
This is a refactor/finishing-move of @c9s's original PR meteor/meteor#7586.

This maintains backward compatibility with PACKAGE_DIRS using the original separator that it expected (':').  Users of the new METEOR_PACKAGE_DIRS variable should use the correct path separator for their platform (`:` on BSD/Linux and `;` on Windows).

Fixes meteor/meteor#7585
Fixes meteor/meteor#4204
2016-10-13 14:26:23 +03:00
Ben Newman
56de90e440 Switch back some optimistic functions in tools/fs/watch.js.
In particular, optimisticReaddir was getting called before relevant dirty
callbacks were firing, and thus failing to notice actual changes on the
file system.

In general, the tools/fs/watch.js code is the one place where we really
need an up-to-date view of the file system. Put another way, if optimistic
functions worked perfectly, we wouldn't need to rely so much on WatchSet
logic, but for now it's a balance of equally important strategies, and we
shouldn't be compromising one by intermingling it with the other.
2016-10-12 18:12:16 -04:00
Ben Newman
4300d261f3 Support meteor <command> for any dev_bundle/bin/<command> executable.
This will make it easier to use tools like https://yarnpkg.com/ with the
right version of Node, etc.

With this commit, here's all you have to do:

  meteor npm install -g yarnpkg

Then test that it works:

  meteor yarn info

Note that any commands registered by Meteor itself will not be honored.
2016-10-12 13:22:26 -04:00
c9s
3c5059b72a Add METEOR_PACKAGE_DIRS support
- Use path.delimiter to support windows platform
- Add console.warn message when PACAKGE_DIRS is defined
2016-10-12 18:52:58 +03:00
Ben Newman
ebecfeeb3e Prevent exceptional optimisticReadFile from invalidating Resolver caching. 2016-10-12 11:41:34 -04:00
Ben Newman
1d375ffb70 Hold off on using optimistic functions in watch.readAndWatchFileWithHash.
This partially reverts commit 96c95629eb.

When running `meteor update`, we call writeReleaseFileAndDevBundleLink
immediately before reading .meteor/release, so there's no time for a file
change notification to invalidate the cached contents of that file.

In the future, we could perhaps call optimisticReadFile.dirty(path) as a
consequence of files.writeFile(path, ...), but that may be tricky.
2016-10-11 11:25:36 -04:00
Ben Newman
b594b460a4 Go back to using pathwatcher.watch, with new rewatching logic.
Judging from the variety and extent of test failures, switching to
chokidar.watch was too drastic a change for this late-beta stage of the
release cycle.

The problem with pathwatcher.watch was that watches don't survive the
deletion of the watched file, because (like fs.watch) it watches files
based on inodes, not paths. This problem can be solved in a relatively
narrow way, by attempting to rewatch the file after any "delete" or
"rename" events.
2016-10-11 10:02:06 -04:00
Ben Newman
b814311266 Catch async chokidar.watch errors and suggest raising watch limit. 2016-10-10 17:39:17 -04:00
Ben Newman
0311e76673 Bring back fs.watchFile fallback when chokidar.watch fails.
Especially on Linux, chokidar.watch tends to throw ENOSPC errors when too
many files are being watched.

I removed this logic earlier because I thought chokidar could handle such
failures by itself, but that was wishful thinking.
2016-10-10 17:12:53 -04:00
Ben Newman
96c95629eb Use optimistic functions throughout tools/fs/watch.js. 2016-10-10 15:22:53 -04:00
Ben Newman
6a66368e09 Cache optimistic stat objects in InputFile#findControlFile. 2016-10-10 15:22:53 -04:00
Ben Newman
117b1a8525 Use chokidar for file watching instead of pathwatcher.
Healthy competition among fs.watch wrappers appears to have produced a
clear winner: https://www.npmjs.com/package/chokidar

This wrapper is better for Meteor than pathwatcher was, because it can
watch directory trees recursively, and it has no trouble watching
nonexistent file paths, whereas pathwatcher would throw an exception.
2016-10-10 14:12:51 -04:00
Ben Newman
1e5b414e48 Prepare for breaking API change affecting optimistic subscribe functions.
As of optimism@0.3.0, the value of `this` within subscribe functions is no
longer the optimistic wrapper function object (instead: null or global, as
if called with no receiver object), so we need to retain a reference to
the optimistic function so that the watcher can call .dirty(...args).
2016-10-10 11:47:19 -04:00
Ben Newman
817876f7b3 Use optimistic functions in PackageSource#initFromPackageDir. 2016-10-07 19:31:17 -04:00
Ben Newman
a1cfbc7ab2 Add profiling and use optimistic functions in LocalCatalog. 2016-10-07 19:31:07 -04:00
Ben Newman
2553958c38 Merge branch 'devel' into release-1.4.2 2016-10-07 19:30:30 -04:00
Ben Newman
dfd477568f Fix test failure by tolerating different line/column numbers. 2016-10-07 18:56:13 -04:00
Ben Newman
d3d60fc475 Merge pull request #7886 from abernix/feature/add-cordova-dependency-test
Add Cordova Platform Dependency Tests
2016-10-07 11:21:58 -04:00
Ben Newman
c969cde1af Merge pull request #7814 from engelgabriel/patch-1
Add --headless option to the build command.
2016-10-07 10:03:38 -04:00
Ben Newman
a24448e7a7 Merge pull request #7885 from abernix/feature/fix-console-success
`Console.success` should provide an indicator when pretty is disabled
2016-10-07 09:58:36 -04:00
Ben Newman
16e9e0c0f9 Merge pull request #7884 from abernix/feature/fix-console-fail
Fix for `Console.failInfo` and `Console.failWarn` when `pretty` is false.
2016-10-07 09:56:25 -04:00
Jesse Rosenberger
8c948d2ebc Test: Ensure message when platform already added
If the platform is already added, it should provide the appropriate message.
2016-10-07 14:17:11 +03:00
Jesse Rosenberger
3359d2397c Add test which verifies the Cordova dependency messages are working
Previously, the Cordova message was failing in CI.  It wasn't caught on CircleCI since they have Cordova installed by default so this path was never tested.  I found this when I ran this through Semaphore CI who does not have Cordova available by default.

This test fails without meteor/meteor#7884 and passes with it.
2016-10-07 14:16:48 +03:00
Jesse Rosenberger
932750f77d Fix redeclared var & change to let 2016-10-07 14:07:54 +03:00
Jesse Rosenberger
58a6d743c8 Console.success should provide an indicator when pretty is disabled
`Console.success` is currently called in only one place but it relies on the fact that `pretty` console mode is enabled for it to make any sense.  This adds the ability to pass an "ugly" (less pretty?) message to `Console.success` (default: "success") and changes the Cordova dependency checker to use "installed" when showing a met-dependency.

Fixes meteor/meteor#7883
2016-10-07 13:41:39 +03:00
Jesse Rosenberger
1d3d6c2bda Fix for Console.failInfo and Console.failWarn when pretty is false
These functions were both using the `Console._fail` function which is passed a severity of failure as a string (e.g. "warn", "error", etc.) which corresponds to the appropriate `Console` function.  Previously, the string was just being called as if it was a function.

Fixes meteor/meteor#7882
2016-10-07 13:36:59 +03:00
Ben Newman
2d18c25f7e Be more defensive when terminating watchers. 2016-10-06 18:09:22 -04:00
Ben Newman
14ee8775c3 Decompose PackageNamespace, PackageNpm, and PackageCordova classes.
Similar to the treatment given to PackageAPI in my commit
af50b4cc5b.

This clear separation of concerns will be helpful for optimizing
PackageSource#initFromPackageDir.
2016-10-06 17:39:23 -04:00
Ben Newman
9608506ce5 Merge pull request #7835 from sebakerckhof/optimize-ignorefiles
Optimize ignoreFiles patterns.
2016-10-06 15:07:29 -04:00