Case in point: @babel/runtime/helpers/esm/typeof.js uses ECMAScript module
syntax (import, export), but must not be compiled with transforms like
@babel/plugin-transform-typeof-symbol, since it's part of the runtime
library depended upon by that transform.
This logic is an extreme implementation detail for sure, but at least
babel-compiler is the only code that needs to know about this complexity.
Now that we have the ability to invoke Babel via compiler plugins for
individual modules encountered by the ImportScanner, it's possible the
ImportScanner will try to compile @babel/runtime/helpers/* modules.
These modules are not safe to recompile because they use native syntax
(such as typeof) in ways that do not need additional transformation or
simulation, and would be broken by applying the usual Babel transforms.
It may bother you that this list of packages is hard-coded, or that it
might grow over time. To ease those concerns, I would say:
1. We can release new versions of the babel-compiler and ecmascript
packages whenever we need to.
2. These particular npm packages belong in this list because Babel
itself assumes they have already been compiled, so there shouldn't be
(m)any other packages that fit that narrow criterion.
In other words, this is just a list of packages that must be left
untouched in order to bootstrap the Babel compiler system, and the
babel-compiler package is where the details of that system primarily
reside, so that's where we should put this list, until/unless we find a
better solution.
Instead of merely supporting ECMAScript module syntax via Reify, we should
really be compiling unanticipated modules (typically within node_modules)
using the same logic that the rest of the application uses.
Note: this processing applies only to .js files for now, since that's what
the ImportScanner works with.
Should help with #10563.
* Update ecmascript-runtime-{client,server} to core-js@3.1.4.
Also added a polyfill for Symbol.asyncIterator to server, modern, and
legacy, which should fix#9897.
* Add a test of for-await-of async iteration.
This should verify that #9897 is fixed.
We ended up with two different versions of the reify package (0.19.0 and
0.19.1) in the last build of the dev bundle, which seems to have caused
the test failures.
Although I hoped we could be clever about which npm packages we compiled,
there are already too many exceptions to the rules (for example, not all
npm packages that contain ESM code have a "module" entry point in
package.json).
It seems safer simply to compile all modules imported from node_modules
that have not already been handled by compiler plugins, and trust that our
disk+memory caching system will provide acceptable build performance.
Should help with #10547, #10544, and #10546.
Sometimes (very rarely) an npm package may contain package.json files
other than the one found in the root package directory. For example, the
date-fns@2.0.0-alpha.27 package includes a small package.json file for
each of its functions (date-fns/someDateFn/package.json) that contains
only { sideEffects, typings }, for whatever reason. These package.json
files clearly do not serve the same purpose as date-fns/package.json, and
it would be convenient to ignore them in optimisticLookupPackageJson. The
easiest way I can see to accomplish that is to ignore package.json files
that do not have a "name" property, since any package.json file that
governs an actual package must have a name.
Should fix#10547.
Instead of compiling ESM syntax in node_modules using compiler plugins,
the ImportScanner can provide "native" support for ESM syntax by using
Reify to quickly compile just the import/export syntax in any imported
modules that were not already handled by compiler plugins.
Since this code runs every time the app is built, it should not matter
which version of Meteor was used to publish a package. Compared to the
previous implementation based on PackageSource#_findSources and unibuild
JSON files (#10545), this implementation should have far fewer
compatibility concerns, as well as being faster thanks to not processing
or compiling modules until the ImportScanner determines that they are
actually imported.
Though the number of files that get compiled by this system should be
relatively small for now, to maintain good performance, the results of the
compilation are cached on disk and in memory.
After much thought, I believe this implementation (#10545) would have
caused severe compatibility problems when using packages published with
earlier versions of Meteor in a Meteor 1.8.2 app, or when publishing
packages with Meteor 1.8.2 for use with earlier Meteor versions.
Specifically, this implementation relied on writing the additional
.npm/package/node_modules resources found by _findSources into the
unibuild JSON file(s), and there just wasn't any good way to make sure the
new JSON format could be safely consumed by previous Meteor versions.
Even if we found a way to hide the new resources from older versions of
Meteor, perhaps by putting them in a new/different property of the
unibuild JSON file, packages published with older Meteor versions might
try to load an npm package with a "module" field without realizing the
code must be compiled, which would likely cause a syntax error in Meteor
1.8.2, since the "module" field always gets preference over the "main"
field of package.json (in Meteor 1.8.2).
When I implemented support for the "module" entry point in package.json
files for client code in #10541, I modified PackageSource#_findSources to
include files found in node_modules that need to be compiled, but my
implementation considered only "local" node_modules directories, like the
one in the application root directory, while neglecting the private
.npm/package/node_modules directories that many Meteor packages have.
This commit includes .npm/**/node_modules when _findSources is scanning a
Meteor package, which should solve issues like #10544, where a Meteor
package imports an npm package that was installed with Npm.depends, and
that npm package has a "module" field in its package.json file, pointing
to an ESM entry point module, but the ESM syntax was not appropriately
compiled, leading to parse errors like "Unexpected token export".
Before lazy compilation was introduced in Meteor 1.7 (#9983), including
the node_modules directories of Meteor packages would likely have been a
big problem for build performance, since there would be that many more
modules to compile. It's still worth making sure this change doesn't
regress build performance for other reasons, but I'm reasonably confident
lazy compilation will save us here, unless there are just too many npm
packages installed via Npm.depends that export ESM modules.
