Commit Graph

570 Commits

Author SHA1 Message Date
Ben Newman
7e1d4ba6a1 Don't skip reinstalling unchanged Npm.depends-style dependencies. (#8787)
Fixes #8732.
Should also fix #8747.
2017-06-12 16:59:34 -04:00
Ben Newman
a2f26cbe6e Fix #8761 in three different ways.
The root of the problem was that the es5-ext npm package contains
directories called '#', e.g.
https://github.com/medikoo/es5-ext/tree/master/array/%23

These directory names were being sanitized to '' and thus ignored when
reserving paths in the Builder, which led to reservation conflicts later.

This commit fixes the problem in three different and independently
sufficient ways:

* Use files.mkdir_p instead of files.mkdir when creating parent
  directories of written files.

* Replace illegal characters in sanitized paths with '_' instead of ''.

* Allow '#' in sanitized paths (only needs to be escaped in the shell, not
  actually forbidden in paths).
2017-06-06 12:25:33 -04:00
Ben Newman
da097e209d Bump LINKER_CACHE_SALT and compiler.BUILT_BY for good measure. 2017-05-30 14:16:46 -04:00
Ben Newman
0fbb97432c Fix ENOTEMPTY and EPERM errors on Windows.
Windows began suffering from cryptic ENOTEMPTY and EPERM errors between
1.5-beta.20 and 1.5-rc.0. After a tricky `git bisect` adventure, I tracked
the problem down to my commit b6694b2f5d,
which caused dynamic modules to be written more than once by the bundler.
Though I don't understand exactly why Windows complained in this way, I'm
glad that it did, because otherwise this mistake would merely have been a
performance problem, and might not have been noticed before the release.
2017-05-18 11:48:40 -04:00
Ben Newman
16a8a3ae3c Tweaks to linker caching.
No longer using a RegExp when we know what the old file wildcard path
should be, and no longer using Fiber when we can just use a Promise
callback (since all Promise callbacks run in a Fiber).
2017-05-16 13:43:38 -04:00
Hugh Willson
b8a8833ac6 Moved LINKER_CACHE_SALT into cacheKeySuffix part of cache filename hash. 2017-05-16 13:41:40 -04:00
Hugh Willson
7dd14e723d Adjusted bundler-cache saved file approach to remove out of date cache files. 2017-05-16 13:41:39 -04:00
Ben Newman
2e057d7119 Call requestGarbageCollection in Isopack#_writeTool.
This method appears to be causing large spikes in memory consumption on
Circle CI during the `meteor --get-ready` preparation step, which often
leads to the test process being killed.

Also added a call in IsopackCache#_loadLocalPackage for good measure.

We're now calling requestGarbageCollection pretty frequently when
we run Node with --expose-gc, but that currently only happens during
Circle CI tests, so I don't think we need to implement the improvements
suggested in tools/utils/gc.js, yet.

Previously: 35f488e140, f6df21ff1e
2017-05-15 16:17:20 -04:00
Ben Newman
9e32c00428 Give dynamic module manifest items a different type: "dynamic js".
Now that dynamic modules are part of the manifest that determines which
files are served over HTTP, I'm a bit paranoid about them somehow ending
up as <script> tags in the initial HTML of the application.

This commit adds another safety measure to prevent that, just in case the
boilerplate-generator package for some reason fails to skip items whose
.path starts with "dynamic/" (see my previous commit).
2017-05-10 12:30:19 -04:00
Ben Newman
b6694b2f5d Expose dynamic modules via HTTP, but do not load them eagerly.
This allows fetching the compiled code of dynamic modules via HTTP,
without generating <script> tags for those resources in the intial
boilerplate HTML of the application.

The URL for a dynamic module should be formed by taking its absolute
module identifier, prepending "/dynamic" and appending "?hash=<version>".
Appropriate version hashes can be obtained from the tree exported by the
meteor/dynamic-import/dynamic-versions.js module, though the hashes are
used only for cache busting, so they could be anything at all.

A good place to do this fetching would be the meteorInstall.fetch
callback, as defined (for example) in meteor/dynamic-import/client.js.
That implementation still uses a WebSocket rather than HTTP, but this
commit will allow us to experiment with HTTP in the future.

Because the code returned for these dynamic modules is wrapped as an
anonymous function expression, you'll need to fetch them using an
XMLHttpRequest, the HTTP fetch() function, or some similar utility, rather
than using a <script> tag, because executing the unmodified code as JS
will likely throw a syntax error.
2017-05-10 12:11:10 -04:00
Ben Newman
fba9e9dfe9 Restrict the dynamic-import package to Meteor 1.5+ apps. 2017-05-10 11:31:39 -04:00
Ben Newman
75ada39092 Bump LINKER_CACHE_SALT and compiler.BUILT_BY for good measure.
Recent updates to meteor-babel and reify make it particularly important to
force recompilation of ECMAScript modules and package/app bundles.
2017-05-09 15:21:42 -04:00
Ben Newman
b5b76a850e Include minifier package name and version in .stats.json file. 2017-05-09 13:17:01 -04:00
Ben Newman
1c8bc7353e Serve <hash>.stats.json for every <hash>.js bundle. 2017-05-09 13:17:00 -04:00
Ben Newman
084801237a Report per-bundle stats from minifier. 2017-05-09 13:16:59 -04:00
Ben Newman
269e10caa6 Remove support for pkg.module in client bundles.
This partially reverts commit d7cae3c939.

This functionality was problematic without providing any clear benefits to
Meteor developers, as reported by @fermuch and @markoshust:
https://github.com/meteor/meteor/pull/8327#issuecomment-298723203
2017-05-04 16:09:45 -04:00
Ben Newman
9c7778da36 Let the install npm package implement Module.prototype.prefetch.
Now anyone can define meteorInstall.fetch however they see fit, and the
install.js implementation will handle everything else.

This separation of concerns leads to significantly less code, too.
2017-05-01 23:55:00 -04:00
Ben Newman
d7cae3c939 Support pkg.module in client bundles, even for node_modules.
As proposed here: https://github.com/rollup/rollup/wiki/pkg.module

By supporting ECMAScript module entry points for npm packages in Meteor
1.5, we will be well-positioned to do more effective import/export-based
tree shaking in future versions of Meteor.

We can't do the same thing on the server because we can't change how
native Node resolves package entry points based on the "main" field of the
package.json module.

On the other hand, all npm packages have to work in Node using the "main"
field, and client bundles stand to benefit the most from tree shaking, so
this client/server difference should not be problematic.

Note that the "jsnext:main" property is also supported as a legacy synonym
for "module".
2017-04-27 17:34:34 -04:00
Ben Newman
b023f5c67a Merge branch 'release-1.4.4.2' into release-1.5 2017-04-27 17:16:05 -04:00
Bartosz Wojtkowiak
5a4d5035c3 Implement CORDOVA_COMPATIBILITY_VERSION_IOS/ANDROID and EXCLUDE (#8581)
* Implement CORDOVA_COMPATIBILITY_VERSION_EXCLUDE and CORDOVA_COMPATIBILITY_VERSION_IOS/ANDROID

CORDOVA_COMPATIBILITY_VERSION_IOS or CORDOVA_COMPATIBILITY_VERSION_ANDROID allows to override compatibility version for a specified platform.

CORDOVA_COMPATIBILITY_VERSION_EXCLUDE provides a way of excluding a certain plugin from compatibility version calculation. You can pass several plugin names with ';'. For example: `CORDOVA_COMPATIBILITY_VERSION_EXCLUDE='cordova-plugin-crosswalk-webview;cordova-plugin-device'`

* Changes after review
2017-04-26 13:23:22 -07:00
Ben Newman
f4fc2a904d Use final hashes of dynamic modules for browser caching.
Previously, the linker included a "version" hash in each stub entry
corresponding to a dynamic module in the meteorInstall bundle. These stub
entries also contain information about any module identifiers ("deps")
imported by the module, and various other metadata.

This hash was derived from the source code of the dynamic module, instead
of its fully-processed generated code, which created a small risk that the
source hash would remain the same when anything changed in the post-linker
processing logic.

This new implementation uses the same hashes found in program.json
manifest files, which more reliably reflect changes in the actual final
contents of the modules.

Because these hashes are not known at link time, they are now injected as
one blob into the meteor/dynamic-import/dynamic-versions.js module, rather
than appearing in the meteorInstall bundles for their containing packages.
2017-04-21 16:15:09 -04:00
Ben Newman
b057415048 Merge branch 'devel' into release-1.5 2017-04-20 16:46:03 -04:00
Jesse Rosenberger
27b2a04f6f Remove experimental trailing comma to avoid problems with jsdoc.
The comma in question: a trailing comma a rest-parameter, within a
function argument's parameter de-structuring:

	function a({
	  a = false,
	  ...b,
	}) {
	  // ...
	}

Espree, the parser used by `jsdoc` (used in Meteor docs) previously
allowed this with `experimentalObjectRestSpread` enabled but now throws
an error with the addition of 652990a7bf.

It might have been argued at one point that the trailing-comma could
allow for the easy, future addition of another parameter, ala:

	function a({
	  a = false,
	  ...b,
	  c = true,
	}) {
	  // ...
	}

Having the rest-parameter in the last position is certainly more clear
(it is the "rest", after-all, and there can be only one) but I'm not
going to have that discussion at the cost of docs not deploying!
2017-04-12 09:59:09 -07:00
Ben Newman
43d339d087 Merge branch 'devel' into release-1.5 2017-04-07 12:45:29 -04:00
Jesse Rosenberger
948ec2b05f Merge branch 'master' into devel 2017-04-07 18:50:17 +03:00
Ben Newman
337a61c426 Disable dynamic module fetching in Cordova apps.
Instead of fetching a bundle of JavaScript from the server, Cordova apps
include the complete bundle in the installed app, so there's no compelling
reason to allow dynamic module fetching.

The dynamic import(...) function will still work on Cordova, of course.
It will just never have to fetch anything from the server.
2017-04-04 19:09:26 -04:00
Ben Newman
5cf603ce82 Merge branch 'release-1.4.3.x' into release-1.5 2017-03-27 15:37:11 -04:00
Ben Newman
0eb52a1fd8 Upgrade meteor-babel to v0.19.1 and reify to v0.6.6. 2017-03-25 11:12:52 -04:00
Ben Newman
d0f179b2f5 Merge pull request #8502 from hwillson/issue-6945
Updated the builder whitelist to allow "@" in filenames.
2017-03-22 12:50:28 -04:00
Hugh Willson
39748d82f1 Updated the builder whitelist to allow "@" in filenames. 2017-03-20 13:33:57 -04:00
Ben Newman
e79a4e9272 Merge branch 'devel' into release-1.5 2017-03-14 15:36:06 -04:00
Jesse Rosenberger
2f4e0124b0 Merge branch '20170303-dev-bundle-upgrades' into release-1.4.3.x 2017-03-09 17:33:28 +02:00
Jesse Rosenberger
511d43fe52 Upgrade versions of node-pre-gyp and node-gyp in the built bundle.
As a finishing move for ba06bf6eed, this
updates the versions of `node-pre-gyp` and `node-gyp` included in the
`meteor-dev-bundle`.
2017-03-09 17:19:04 +02:00
Ben Newman
db968ad975 Merge branch 'devel' into release-1.5 2017-03-08 17:33:45 -05:00
Jesse Rosenberger
c2c92b9b01 Allow malformed package.json when determining binary dependencies. (#8468)
We now check `package.json` in order to help make an educated decision
as to whether or not a package has binary dependencies which need to be
rebuilt.  In some cases, such as the `npmconf` npm which is included
as a dependency of `flow-router, the `package.json` is invalid (i.e.
empty), and we should silently permit this.

Fixes meteor/meteor#8427
2017-03-08 15:12:58 -05:00
Ben Newman
f6df21ff1e Force garbage collection more aggressively during meteor --get-ready.
This is an effort to keep the `meteor --get-ready` command from running
out of memory during Circle CI tests, as mentioned in this comment:
https://github.com/meteor/meteor/pull/8327#issuecomment-284431618
2017-03-06 20:48:53 -05:00
Ben Newman
ab0f9ba290 Force garbage collection more aggressively during meteor --get-ready.
This is an effort to keep the `meteor --get-ready` command from running
out of memory during Circle CI tests, as mentioned in this comment:
https://github.com/meteor/meteor/pull/8327#issuecomment-284431618
2017-03-06 20:34:56 -05:00
dr.dimitru
dd548d998d Use process.version for generated README (#8379)
* Update generated README

 - Update node version in generated README file by bundler, according
to version from `.node_version.txt` file

* Use `process.version`

 - Minor file linting, one missed semicolon, one unnecessary semicolon
2017-02-23 12:08:09 +02:00
Ben Newman
7031ea1dd5 Bump LINKER_CACHE_SALT for good measure.
Recent changes to the way the linker works might not be reflected unless
we invalidate .meteor/local/bundler-cache.
2017-02-21 13:25:58 -05:00
Ben Newman
41e9ee857e Evaluate dynamic module code in package scope.
This should elegantly address the issues described in this comment:
https://github.com/meteor/meteor/pull/8327#issuecomment-280881830

I toyed with the possibility of turning package variables (both imports
from other packages and intercepted "global" variable assignments) into
properties on a shared namespace object, but that would have been a major
breaking change for existing package code, because it would have required
automatically rewriting variable references in package modules.
2017-02-21 11:47:31 -05:00
Ben Newman
2c172db31a Merge branch 'devel' into release-1.5 2017-02-21 11:33:00 -05:00
Ben Newman
c2e4b77636 Protect against edge case when resolved is a string.
In particular, if the "main" field of the package.json file cannot be
resolved, then the `resolved` variable will be "missing", which is a
truthy value that silently rejects property assignments. Ugh!
2017-02-15 14:09:36 -05:00
Ben Newman
0b26890d97 Protect against edge case when resolved is a string.
In particular, if the "main" field of the package.json file cannot be
resolved, then the `resolved` variable will be "missing", which is a
truthy value that silently rejects property assignments. Ugh!
2017-02-15 14:07:27 -05:00
Ben Newman
6b89bed42d Merge branch 'devel' into release-1.5 2017-02-14 12:28:59 -05:00
Eric Dobbertin
005c521f54 Fix comment about unordered imports (#8362) 2017-02-14 09:03:25 -08:00
Ben Newman
434116aa84 Fix installing from shrinkwrap.
Although we thought the upgrade from `npm` 3.10.9 to 4.1.2 was worthwhile
and safe, this breaking change proved problematic:
https://github.com/npm/npm/blob/latest/CHANGELOG.md#no-more-partial-shrinkwraps-breaking

Specifically, if a Meteor package calls `Npm.depends` in a way that
disagrees with the contents of `.npm/package/npm-shrinkwrap.json` file,
Meteor will create a partial shrinkwrap file in order to install the
correct top-level npm dependencies, but transitive dependencies of the
package will no longer be installed.

This was fixed in Meteor 1.4.2.7 by reverting the upgrade of npm, but
Meteor 1.4.3.1 will keep npm@4.1.2 and fix the consequences.
2017-02-13 18:11:11 -05:00
Ben Newman
cda9ba1d41 Merge branch 'release-1.4.3' into release-1.5 2017-02-10 14:37:43 -05:00
Ben Newman
d9184808cb Avoid calling ImportScanner#_getRelativeImportId with bad parentPath.
Fixes #8336.
2017-02-10 14:33:20 -05:00
Ben Newman
844e2c34ad Consider npm packages with {,pre,post}install scripts non-portable.
Inspired by analysis from @danstiner:
https://github.com/meteor/meteor/issues/8225#issuecomment-275044900

Fixes #8225, as well as the tests I added in 672c4f338a.

I changed the name of the .meteor-portable file to .meteor-portable-1.json
in order to invalidate previous .meteor-portable files. This naming scheme
will be more sustainable, because we can keep incrementing the version
number whenever we change this logic.
2017-02-09 16:06:55 -05:00
Ben Newman
bcffe53d14 Consider npm packages with {,pre,post}install scripts non-portable.
Inspired by analysis from @danstiner:
https://github.com/meteor/meteor/issues/8225#issuecomment-275044900

Fixes #8225, as well as the tests I added in 672c4f338a.

I changed the name of the .meteor-portable file to .meteor-portable-1.json
in order to invalidate previous .meteor-portable files. This naming scheme
will be more sustainable, because we can keep incrementing the version
number whenever we change this logic.
2017-02-09 16:04:23 -05:00