87bc54fb9f
On the server, Meteor attempts to avoid bundling node_modules code by replacing entry point modules with a stub that calls module.useNode() (see packages/modules-runtime/server.js). This trick allows evaluating server node_modules natively in Node.js, faithfully preserving all Node-specific behaviors, such as module.id being an absolute file system path, the __dirname and __filename variables, the ability to import binary .node modules, and so on. However, starting in Node.js 12.16.0 (Meteor 1.9.1+), modules evaluated natively by Node are considered ECMAScript modules (ESM) if the closest package.json file has "type": "module" (or has an .mjs file extension). This poses a problem for the module.useNode() trick, because ESM modules cannot be imported synchronously using require (which is currently how module.useNode() works). To work around this new error, this commit checks package.json for "type": "module" in ImportScanner#shouldUseNode to determine whether it's safe to use the module.useNode() trick. The good news is that ESM modules don't have access to nearly as many Node.js-specific quirks: no module, require, or exports variables; no __dirname, no __filename; no ability to import JSON or other non-ESM file types (at least right now). So it seems somewhat less important for ESM code (compared to CommonJS code) to bail out into native Node.js execution using module.useNode(). In other words, bundling server code should not affect its execution in nearly as many cases, if that code is ESM rather than legacy CommonJS. If this good news turns out to be overly optimistic, we can consider using a different kind of bailout stub that's capable of importing ESM using dynamic import(). For now, making sure we avoid bailing out for ESM code like @babel/runtime/helpers/esm/* is the priority.
Isobuild
Isobuild is the build system used by the Meteor Tool. See the high level description for more.
Terms
The terms Isobuild operates on often have two names: internal names and public concepts.
packageSource- an abstract representation of a package/app source with metadataisopack- a compiled version of a package/appunibuild- a part of an isopack for a specific target (browser, server, tool, etc)isopackCache- an abstract representation of cached isopacks on diskbuild plugin- a part of an isopack that plugs into the build processlinked file- a wrapped file by linkerproject context- an object that has lots of metadata, mostly about the packages, catalogs, it is responsible for catalogs refreshes, pulling the relevant packages, resolving dependencies, etc.js-analyze- some piece of software that implements a set of JS parser features
How an app is built
The app is built by Bundler, "bundling" is the most high-level process that happens to the application code once it is loaded. Here is a more detailed time-line:
- Create the "Project Context" out of the app directory. Project Context makes sure we know what package versions we need to use and prepares them for us.
- The Bundler is given the Project Context and now it is Bundler's job to create the Package Source for the app, compile app's parts as they were packages, and then put everything together.
- The Builder is ran to write the output files to disk.
- While all the building was done, a WatchSet of the whole app is collected.
It's returned to the caller. Also, files
star.jsonandprogram.jsonare written.
Compiler
Takes care of compiling an individual package and returning an Isopack.
Compiler has a dependency on Bundler, as it needs to bundle the build plugins used to compile the input package source.
Compiler runs various compiler batch plugins and "source handlers" (the old-style build plugins) on the source. If there is a need to build any build plugins to use them, it would recursively build those.
Bundler
Builds an individual app or a build plugin (that appears to be just an app that is run in the context of the build).
Bundler introduces additional terms:
JsImage- is a representation of a built App or a build plugin.ClientTargetandServerTargetare representations of two separate types of "programs" in a built App.
There are commonly two important entry-points for Bundler:
buildJsImage- build a build pluginbundle- the main function to bundle the application from a Project Context
For the actual compilation, Bundler often calls into Compiler. Other tasks that Bundler performs include:
- initiate a Package Source for the app
- run linters on the app
- run the application files through Linker in a special mode that allows the use of global variables
- add a special file called "global-imports" that explicitly puts all used packages' symbols into the global namespace
- write compiled files to the right location with multiple Builder's
- write a
star.jsonfile with metadata for the "star" (the app) and aprogram.jsonfile for every target
Builder
Manages the files written to the filesystem.
Since the rebuilds of an app is something that occurs over and over again (in the development cycle), it makes sense to reuse the information about the files that didn't change. Builder tries not to spend too much time writing files that remained the same over and over again.
It is even OK to do when an app process serving these files is still running, on Unix, the process can retain files by their inodes (not by file paths) and then once the process release them, the FS will clean up unlinked files.
Linker
A Meteor-specific transform. Wraps every file into a closure, creates "package
local variables" and sets up the "global imports" to look like
var Minimongo = Package.minimongo.Minimongo;.
The process of linking is Meteor's substitute to other module loading solutions
like r.js or require.js, or ES2015-style modules. Eventually, Meteor wants
to transition to ES2015 modules for everything.
The way linker works for individual modules (packages in Meteor's case), is several things. Roughly, it can be separated into two phases "prelink" and "link" (both combined define "fullLink").
Prelink is something that can be done with a package in Isolation:
- create a closure around each file
- concatenate all files
- add comments of the original line number, file-headers - the metadata to be read by humans when viewed in a browser that doesn't support source-maps
- run a JS parser and figure out all the global variables used, out of those,
pick the ones that are assigned in the package and form "Package-level" global
variables by adding a
varfor them.
In the second part "link", the linker caller already knows what versions of linked module's dependencies are and what exports they provide for the module:
- create import strings for globals that come from dependencies' exports:
var Minimongo = Package.minimongo.Minimongo;, if the module referencesMinimongo.
Historically, Meteor used to ship "prelinked" files in packages and then "link" them in the bundle time. Starting with Meteor 1.2 and Batch Plugins API, Meteor distributes source files, so they are "fullLinked" in bunlde time together.
Batch Build Plugins
In Meteor 1.2, the new Build Plugin APIs have been introduced. You can read more about them here: wiki.
The Build Plugins APIs register compilers, minifiers and linters. All of them
are applied on different stages of the build process. Compilers are used by
Compiler and Isopack methods. Linters are ran per Unibuild in
compiler.lint and for the App in Bundler. Minifiers are ran by Bundler on its
last stage.
The previous generation of build plugins (File Handlers) used to run on package publish (i.e., built before published). In the world of Build Plugins, they are compiled as part of the bundle process.
WatchSet
WatchSets are collected throughout the build process. The idea is: the WatchSet represents all the files that participate in the build process and also this list is used to notice any changes for rebuilds.
Working on the Isobuild codebase, it is important to understand the common usage pattern: the common WatchSet is passed down to functions as an argument. The functions on the bottom, add files to the WatchSet, mutating the passed argument.
In the end, the final WatchSet that contains merged information about every used file, is returned to the caller.