* Avoid modifying unibuild.watchSet in PackageSourceBatch._watchOutputFiles.
Should fix#10736 by preventing old hashes from remaining in
unibuild.watchSet, which was sometimes causing isUpToDate to fail
immediately upon restart.
* Regression test for issue #10736.
The previous implementation simply avoided calling watchSet.addFile for
potentially unused files, trusting that addFile would be called later if
the file was eventually used. However, this strategy left the contents of
watchSet.files incomplete for tasks such as IsopackCache._checkUpToDate,
which require full information about all files, even the ones that might
not be used by the bundle. The new strategy maintains metadata about
potentially unused files in a separate data structure, which will be
merged/cloned/serialized/deserialized along with other WatchSet data.
Unfortunately, this conversion triggered an error due to one of the
shortcomings of the Babel implementation of TypeScript:
SyntaxError: /tools/tool-env/profile.ts: Namespace not marked type-only declare. Non-declarative namespaces are only supported experimentally in Babel. To enable and review caveats see: https://babeljs.io/docs/en/babel-plugin-transform-typescript
278 | }
279 |
> 280 | export namespace Profile {
| ^
281 | export let enabled = !! process.env.METEOR_PROFILE;
282 |
283 | export function time<TResult>(bucket: string, f: () => TResult) {
at File.buildCodeFrameError (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/core/lib/transformation/file/file.js:261:12)
at transpileNamespace (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/namespace.js:25:25)
at PluginPass.TSModuleDeclaration (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/index.js:271:32)
at newFn (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/visitors.js:193:21)
at NodePath._call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:53:20)
at NodePath.call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:40:17)
at NodePath.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:88:12)
at TraversalContext.visitQueue (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:118:16)
at TraversalContext.visitMultiple (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:85:17)
at TraversalContext.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:144:19)
In PR #10585, I tried enabling full compiler plugin processing for any
modules imported from `node_modules` that were not otherwise handled by
compiler plugins, but doing that for all packages was just too slow, not
to mention potentially dangerous for modules whose code cannot be safely
recompiled by Babel.
The primary reasons for wanting to recompile node_modules are
1. to enable "native" ECMAScript `import`/`export` syntax (given that
Node.js still does not fully support ESM syntax yet), and
2. to compile standard syntax like `const`, `let`, classes, and arrow
functions for legacy browsers.
The first goal can be achieved by compiling unanticipated `node_modules`
code with Reify, which is much faster than Babel, in part because Reify
can avoid doing any parsing when the source contains no `import` or
`export` identifiers. This compilation is also cached on disk, so its
impact should only be felt during cold builds after a `meteor reset`.
The second goal can be accomplished using the new `package.json`
configuration option `meteor.nodeModules.recompile`, which causes the
recompiled packages to be handled by the normal compiler plugins system,
so that the `ImportScanner`'s fallback compilation is not necessary.
Before this option was introduced, this recompilation behavior could be
achieved by symlinking application directories into `node_modules`, but
that always felt like an esoteric hack.
Instead of merely supporting ECMAScript module syntax via Reify, we should
really be compiling unanticipated modules (typically within node_modules)
using the same logic that the rest of the application uses.
Note: this processing applies only to .js files for now, since that's what
the ImportScanner works with.
Should help with #10563.
Instead of compiling ESM syntax in node_modules using compiler plugins,
the ImportScanner can provide "native" support for ESM syntax by using
Reify to quickly compile just the import/export syntax in any imported
modules that were not already handled by compiler plugins.
Since this code runs every time the app is built, it should not matter
which version of Meteor was used to publish a package. Compared to the
previous implementation based on PackageSource#_findSources and unibuild
JSON files (#10545), this implementation should have far fewer
compatibility concerns, as well as being faster thanks to not processing
or compiling modules until the ImportScanner determines that they are
actually imported.
Though the number of files that get compiled by this system should be
relatively small for now, to maintain good performance, the results of the
compilation are cached on disk and in memory.
Fixes#10393.
Bumping compiler.BUILT_BY and LINKER_CACHE_SALT because
PR #10414 changes the behavior of the build system in a subtle way that
does not automatically trigger recompilation.
Hashes have a number of overlapping but not entirely redundant or
equivalent purposes within the build system.
Hashes of source code are important because they can be computed before
compilation and processing, and thus are useful as keys for caching that
expensive work. Source hashes remain useful even after compilation, as a
way of reflecting the contributions of source-code-sensitive assets like
source maps.
However, source hashes do not tell the whole story, and using them as
cache keys can be risky if the work that's being cached depends on
generated code rather than source code, as we recently discovered with the
findImportedModuleIdentifiers function. The preliminary fix for that
problem (#10330) was to cache findImportedModuleIdentifiers using a hash
of the generated code rather than the source hash.
PR #10330 swung a bit too far in the direction of ignoring source hashes
and considering only hashes of generated code. For example, the URLs of
source maps share the hash of the corresponding resource, but source maps
can change (because of superficial changes in the source code) without
changing the generated code of the resource. Ignoring the source hash when
computing source map URLs resulted in stale source maps with incorrect
line numbers.
A better solution seems to be to propagate the source hash (along with any
hashes of intermediate generated artifacts) all the way through bundling,
so that the final hash of any static resource reflects all information
that could/should change the behavior of that static resource, including
its source map, which embeds the exact source code of all contributing
files in the sourcesContent property. At every step of the way, we merge
all the input hashes into a single hash, so we don't have to keep juggling
multiple hashes, thankfully.
Sub-Resource Integrity (SRI) hashes still need to be computed from just
the final contents of a given asset, so that the browser can verify those
contents without knowing anything about the Meteor build system, but
that's handled separately.
With the introduction of lazy compilation in Meteor 1.8, calling
inputFile.addJavaScript({
...
hash: inputFile.getSourceHash(),
...
}, function () {
return compiler.processFilesForTarget(inputFile);
});
becomes problematic, since inputFile.getSourceHash() is usually different
from compiler.processFilesForTarget(inputFile).hash, because the latter is
computed from the compiled code, whereas the former is computed from the
source code.
For example, when we use file.hash to cache imported module identifiers in
ImportScanner#_findImportedModuleIdentifiers, we really need to be using
the hash of the compiled code, since a single source module can be
compiled in different ways. If we cache based on the source hash, there's
a risk of reusing the scanned imports from the web.browser version for the
web.browser.legacy version, which can lead to all sorts of problems that
are only apparent in legacy browsers.
The quick fix is easy enough: BabelCompiler can simply stop including a
hash in the eager options to inputFile.addJavaScript. This fix can be
published as a minor update to the babel-compiler and ecmascript packages.
The remaining changes in this commit add another layer of defense against
this problem, by ignoring any hash options provided by compiler plugins,
in favor of simply computing the hash from the compiled data buffer.
These additional changes will become available in the next release of
Meteor (likely 1.8.1).
I recently noticed a bug whereby modules transferred from the application
bundle to the modules bundle would lose their application-specific import
extensions, since all modules installed in the modules bundle have only
.js and .json as import extensions, matching default Node behavior.
This commit fixes that bug by emitting one meteorInstall call per distinct
meteorInstallOptions object. This logic would work if every module had a
different meteorInstallOptions object, but in practice the modules bundle
should end up with exactly two meteorInstall calls, because a single
options object is shared among all modules from the same source batch.
This should be a better fix for the problem I tried to fix with
479e505d71.
If we're going to be using compileOneFileLater by default, that's what we
should be testing in the compiler plugins self-tests.
The concept of a "root" file is specific to MultiFileCachingCompiler, so
we need to normalize it into a representation that makes sense to the rest
of the compiler plugin system.
Should help with #10014.
When a CSS (or compiled-to-CSS) module is lazy (e.g., in `imports/` or
`node_modules`) and not otherwise imported by another CSS module, Meteor
automatically turns it into a JS module so that it can be handled by the
`ImportScanner`, and imported dynamically by other JS modules.
Until now, there were two problems with CSS handled in this way: it did
not have proper source maps, and the CSS text was not minified in
production.
This commit introduces a special minification step for dynamic CSS, which
must take place before we create the dynamic JS module. Waiting for the
usual minification of CSS would be a mistake, since that happens long
after the `ImportScanner` and `Linker` have already processed JavaScript
modules. Modifying the contents of JS modules at that point would be
impossible without recomputing source maps, etc.
Since the JS module dynamically creates a `<style>` tag and appends it to
the `<head>` of the document, the code of the JS module has no meaningful
relationship to the lines of CSS text that are actually evaluated by the
browser, so it would be a mistake to give the JS module the same source
map as the original CSS resource.
Instead, when there is a source map, we write it out as an asset that can
be requested at runtime, and append a sourceMappingURL comment to the end
of the CSS text referring to this asset URL. Note that this only happens
in development, which makes sense because minification in production
invalidates the source map, and we don't want to expose source code in
production anyway.
One limitation of Meteor's current compiler plugins system is that every
file we *might* use must be compiled before we know whether it *will* be
used by the application (which is something we only find out when we later
run the `ImportScanner`). More specifically, when inputFile.addJavaScript
is called, any information relevant to the current file must already have
been computed, even if the file will never be used.
This commit begins the process of supporting a lazy version of the
`inputFile.addJavaScript` method. For consistency, this interface is
supported by other methods like `inputFile.addStylesheet`, though it may
not make as much sense for non-JavaScript resources.
In this very basic initial implementation, the `lazyFinalizer` function is
called immediately, so that subsequent code can keep pretending that all
relevant information was eagerly provided.
The next step will be waiting to call `lazyFinalizer` until the last
possible moment, so that we can skip a potentially huge amount of
unnecessary compilation time.
Now that getResolver takes options, it seems inappropriate to cache only
the first resolver created, since multiple resolvers might be created with
different options.
Fortunately, caching Resolver objects is not strictly necessary (even for
performance), since Resolver.getOrCreate already caches based on the
options it receives.
The only other meaningful work PackageSourceBatch#getResolver was doing
was computing the nodeModulesPaths option, so I've cached that.
Setting meteor.testModule is a great way to specify test entry points
explicitly, rather than relying on Meteor's isTestFilePath heuristics:
https://github.com/meteor/meteor/blob/devel/tools/isobuild/test-files.js
The syntax is identical to meteor.mainModule, so you can set an explicit
meteor.testModule.{client,server,...} for each platform. If a testModule
is not specified for a platform, then Meteor's existing rules about test
file paths apply for that platform, as before.
If a testModule is specified, that module will always be loaded eagerly
when running `meteor test`, in addition to any other modules that load
eagerly because of meteor.mainModule or other rules regarding module
loading. If you run `meteor test` without the `--full-app` option, then no
application JS modules other than the testModule (and any modules imported
by it) will be loaded eagerly.
