Instead of compiling ESM syntax in node_modules using compiler plugins,
the ImportScanner can provide "native" support for ESM syntax by using
Reify to quickly compile just the import/export syntax in any imported
modules that were not already handled by compiler plugins.
Since this code runs every time the app is built, it should not matter
which version of Meteor was used to publish a package. Compared to the
previous implementation based on PackageSource#_findSources and unibuild
JSON files (#10545), this implementation should have far fewer
compatibility concerns, as well as being faster thanks to not processing
or compiling modules until the ImportScanner determines that they are
actually imported.
Though the number of files that get compiled by this system should be
relatively small for now, to maintain good performance, the results of the
compilation are cached on disk and in memory.
After much thought, I believe this implementation (#10545) would have
caused severe compatibility problems when using packages published with
earlier versions of Meteor in a Meteor 1.8.2 app, or when publishing
packages with Meteor 1.8.2 for use with earlier Meteor versions.
Specifically, this implementation relied on writing the additional
.npm/package/node_modules resources found by _findSources into the
unibuild JSON file(s), and there just wasn't any good way to make sure the
new JSON format could be safely consumed by previous Meteor versions.
Even if we found a way to hide the new resources from older versions of
Meteor, perhaps by putting them in a new/different property of the
unibuild JSON file, packages published with older Meteor versions might
try to load an npm package with a "module" field without realizing the
code must be compiled, which would likely cause a syntax error in Meteor
1.8.2, since the "module" field always gets preference over the "main"
field of package.json (in Meteor 1.8.2).
When I implemented support for the "module" entry point in package.json
files for client code in #10541, I modified PackageSource#_findSources to
include files found in node_modules that need to be compiled, but my
implementation considered only "local" node_modules directories, like the
one in the application root directory, while neglecting the private
.npm/package/node_modules directories that many Meteor packages have.
This commit includes .npm/**/node_modules when _findSources is scanning a
Meteor package, which should solve issues like #10544, where a Meteor
package imports an npm package that was installed with Npm.depends, and
that npm package has a "module" field in its package.json file, pointing
to an ESM entry point module, but the ESM syntax was not appropriately
compiled, leading to parse errors like "Unexpected token export".
Before lazy compilation was introduced in Meteor 1.7 (#9983), including
the node_modules directories of Meteor packages would likely have been a
big problem for build performance, since there would be that many more
modules to compile. It's still worth making sure this change doesn't
regress build performance for other reasons, but I'm reasonably confident
lazy compilation will save us here, unless there are just too many npm
packages installed via Npm.depends that export ESM modules.
Supporting "module" in package.json for server code is not advisable
because Node.js will be adopting the "type":"module" convention instead,
and in the meantime we need to maintain consistency with Node's module
resolution rules, which only currently pay attention to "main":
https://medium.com/@nodejs/announcing-a-new-experimental-modules-1be8d2d6c2ff
On Linux, child processes that have exited may remain as <defunct>
"zombie" processes, which prevents process.kill(childPid, 0) from
throwing, so we need a different trick for detecting whether the child
process is still alive.
The SIGKILL self-test in tools/tests/run.js has been failing recently
because @babel/runtime can't be found when the app-prints-pid app starts
up, which prevents the app from polling the parent process correctly.
Should help prevent noYieldsAllowed errors due to the Promise#await call
in the removed copyFileHelper function, which is what caused 1.8.2-beta.0
to fail to publish on Linux (reported in #10540).
This should have been done when PR #10299 was first merged. Thankfully,
@macrozone discovered the problem while diagnosing issue #10530.
Should fix#10530.
These changes pave the way for incrementally converting the implementation
of Meteor's command-line tool to TypeScript, which should have profound
benefits for self-documentation via types, as well as substantially
improving navigability and approachability for community contributors.
Just imagine being able to auto-complete the fields of the various
File-like classes currently floating around the codebase, instead of
having to track down their implementations every time. TypeScript was
designed with large projects like Meteor in mind, and it seems
increasingly irresponsible to forgo the benefits of a type system by
relying on the expertise of a few core contributors who know the codebase
inside and out. I am one of those few people, and I am very excited to
have the assistance of a type system, so I can only imagine how
transformative and empowering it will be for everyone else.
If you've ever wanted to get involved in core Meteor development, picking
a few meteor/tools modules to convert to TypeScript is a great way to get
to know that part of the codebase, while also making things easier for
everyone else who interacts with that code in the future.
Because we already compile meteor/tools using Babel, it makes the most
sense to use Babel's @babel/preset-typescript to compile .ts files:
https://babeljs.io/docs/en/next/babel-preset-typescript.html
Using Babel also means we get to keep all of our current advanced
compilation strategies, such as using Reify to compile module syntax:
https://www.npmjs.com/package/reify
Since we're using Babel, the meteor/tools/tsconfig.json file exists mostly
for the benefit of external tools like VSCode, rather than as a source of
truth for compilation behavior.
Despite our existing convention of including explicit .js file extensions
when importing modules, TypeScript and VSCode strongly encourage omitting
the file extension, so the import can be resolved to a .ts file in
development or a .js file when compiled. Although I find this ambiguity
somewhat unfortunate, it makes sense to follow community norms, at least
until Node.js begins supporting .ts modules by default.
The promise package needs a weak dependency on this package (on the
server) because meteor-promise saves a reference to Fiber.yield, so we
need to have wrapped Fiber.yield with noContext before that happens.
The optimism package no longer knows anything about Fibers, but it does
export various helpers for managing execution contexts, one of which
(noContext) allows us to censor the current context for the duration of a
function call. By wrapping Fiber.yield with noContext, we keep distinct
Fibers from accidentally registering cache dependencies on one another.
