Using fs.writeFileSync in a serial style becomes especially costly when
we're writing a lot of files. In a recent profiling exercise I did on
Windows, nearly 80% of the time taken by Builder#_copyDirectory was spent
just closing the written files. By using the asynchronous fs.writeFile
function, we should be able to parallelize at least some of this work, and
await all the promises at the very end of copying the directory.
* Avoid modifying unibuild.watchSet in PackageSourceBatch._watchOutputFiles.
Should fix#10736 by preventing old hashes from remaining in
unibuild.watchSet, which was sometimes causing isUpToDate to fail
immediately upon restart.
* Regression test for issue #10736.
The previous implementation simply avoided calling watchSet.addFile for
potentially unused files, trusting that addFile would be called later if
the file was eventually used. However, this strategy left the contents of
watchSet.files incomplete for tasks such as IsopackCache._checkUpToDate,
which require full information about all files, even the ones that might
not be used by the bundle. The new strategy maintains metadata about
potentially unused files in a separate data structure, which will be
merged/cloned/serialized/deserialized along with other WatchSet data.
Unfortunately, this conversion triggered an error due to one of the
shortcomings of the Babel implementation of TypeScript:
SyntaxError: /tools/tool-env/profile.ts: Namespace not marked type-only declare. Non-declarative namespaces are only supported experimentally in Babel. To enable and review caveats see: https://babeljs.io/docs/en/babel-plugin-transform-typescript
278 | }
279 |
> 280 | export namespace Profile {
| ^
281 | export let enabled = !! process.env.METEOR_PROFILE;
282 |
283 | export function time<TResult>(bucket: string, f: () => TResult) {
at File.buildCodeFrameError (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/core/lib/transformation/file/file.js:261:12)
at transpileNamespace (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/namespace.js:25:25)
at PluginPass.TSModuleDeclaration (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/index.js:271:32)
at newFn (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/visitors.js:193:21)
at NodePath._call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:53:20)
at NodePath.call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:40:17)
at NodePath.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:88:12)
at TraversalContext.visitQueue (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:118:16)
at TraversalContext.visitMultiple (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:85:17)
at TraversalContext.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:144:19)
The meteor/tools/isobuild/resolver.js changes are the static half of the
puzzle. The runtime half was implemented in install@0.13.0 with this
commit: 233aa75ce3
This partially reverts commit 4dfc74197a.
Some server packages, especially those that rely on __dirname or
__filename (e.g. puppeteer), simply cannot be included in the server
bundle, and must be evaluated natively by Node.
As long as we have a Module.prototype._compile hook to process natively
evaluated modules with Reify, module.useNode() can still benefit from ESM
import/export syntax.
This partially reverts f0d39b86e6 by simply
including .js and .mjs modules in the server bundle, rather than
delegating to pure Node evaluation. In practice, whether or not the
package has a "module" entry point (or "type":"module") in its
package.json was not a perfect indicator of whether it should be compiled
with Reify and bundled, or left untouched and handled by Node.
Truly native modules (such as those with a .node file extension) should
always be handled by Node, so module.useNode() definitely still has a role
to play in the server bundle.
In PR #10585, I tried enabling full compiler plugin processing for any
modules imported from `node_modules` that were not otherwise handled by
compiler plugins, but doing that for all packages was just too slow, not
to mention potentially dangerous for modules whose code cannot be safely
recompiled by Babel.
The primary reasons for wanting to recompile node_modules are
1. to enable "native" ECMAScript `import`/`export` syntax (given that
Node.js still does not fully support ESM syntax yet), and
2. to compile standard syntax like `const`, `let`, classes, and arrow
functions for legacy browsers.
The first goal can be achieved by compiling unanticipated `node_modules`
code with Reify, which is much faster than Babel, in part because Reify
can avoid doing any parsing when the source contains no `import` or
`export` identifiers. This compilation is also cached on disk, so its
impact should only be felt during cold builds after a `meteor reset`.
The second goal can be accomplished using the new `package.json`
configuration option `meteor.nodeModules.recompile`, which causes the
recompiled packages to be handled by the normal compiler plugins system,
so that the `ImportScanner`'s fallback compilation is not necessary.
Before this option was introduced, this recompilation behavior could be
achieved by symlinking application directories into `node_modules`, but
that always felt like an esoteric hack.
Example:
"meteor": {
"mainModule": ...,
"testModule": ...,
"nodeModules": {
"recompile": {
"very-modern-package": ["client", "server"],
"alternate-notation-for-client-and-server": true,
"somewhat-modern-package": "legacy",
"another-package": ["legacy", "server"]
}
}
}
The keys of the meteor.nodeModules.recompile configuration object are npm
package names, and the values specify for which bundles those packages
should be recompiled using the Meteor compiler plugins system, as if the
packages were part of the Meteor application.
For example, if an npm package uses const/let syntax or arrow functions,
that's fine for modern and server code, but you would probably want to
recompile the package when building the legacy bundle. To accomplish this,
specify "legacy" or ["legacy"] as the value of the package's property,
similar to somewhat-modern-package above. These strings and arrays of
strings have the same meaning as the second argument to
api.addFiles(files, where) in a package.js file.
This configuration serves pretty much the same purpose as symlinking an
application directory into node_modules, but without any symlinking:
https://forums.meteor.com/t/litelement-import-litelement-html/45042/8?u=benjamn
The meteor.nodeModules.recompile configuration currently applies to the
application node_modules directory only (not to Npm.depends dependencies
in Meteor packages). Recompiled packages must be direct dependencies of
the application.
Instead of merely supporting ECMAScript module syntax via Reify, we should
really be compiling unanticipated modules (typically within node_modules)
using the same logic that the rest of the application uses.
Note: this processing applies only to .js files for now, since that's what
the ImportScanner works with.
Should help with #10563.
Although I hoped we could be clever about which npm packages we compiled,
there are already too many exceptions to the rules (for example, not all
npm packages that contain ESM code have a "module" entry point in
package.json).
It seems safer simply to compile all modules imported from node_modules
that have not already been handled by compiler plugins, and trust that our
disk+memory caching system will provide acceptable build performance.
Should help with #10547, #10544, and #10546.
Instead of compiling ESM syntax in node_modules using compiler plugins,
the ImportScanner can provide "native" support for ESM syntax by using
Reify to quickly compile just the import/export syntax in any imported
modules that were not already handled by compiler plugins.
Since this code runs every time the app is built, it should not matter
which version of Meteor was used to publish a package. Compared to the
previous implementation based on PackageSource#_findSources and unibuild
JSON files (#10545), this implementation should have far fewer
compatibility concerns, as well as being faster thanks to not processing
or compiling modules until the ImportScanner determines that they are
actually imported.
Though the number of files that get compiled by this system should be
relatively small for now, to maintain good performance, the results of the
compilation are cached on disk and in memory.
After much thought, I believe this implementation (#10545) would have
caused severe compatibility problems when using packages published with
earlier versions of Meteor in a Meteor 1.8.2 app, or when publishing
packages with Meteor 1.8.2 for use with earlier Meteor versions.
Specifically, this implementation relied on writing the additional
.npm/package/node_modules resources found by _findSources into the
unibuild JSON file(s), and there just wasn't any good way to make sure the
new JSON format could be safely consumed by previous Meteor versions.
Even if we found a way to hide the new resources from older versions of
Meteor, perhaps by putting them in a new/different property of the
unibuild JSON file, packages published with older Meteor versions might
try to load an npm package with a "module" field without realizing the
code must be compiled, which would likely cause a syntax error in Meteor
1.8.2, since the "module" field always gets preference over the "main"
field of package.json (in Meteor 1.8.2).
