Commit 646fa4e3eefixed#10547 by
restricting optimisticLookupPackageJson to package.json files with a
"name" property, which effectively skipped over intermediate package.json
files with additional properties.
However, in Node.js 12.16.0 (Meteor 1.9.1+), modules evaluated natively by
Node are considered ECMAScript modules if the closest package.json file
has "type": "module" (or has an .mjs file extension). This poses a problem
for the module.useNode() trick (see packages/modules-runtime/server.js),
because ESM modules cannot be imported using require.
For example, recent versions of the @babel/runtime package have a
@babel/runtime/helpers/esm/package.json file for the ESM versions of its
helpers (which specifies "type": "module"), but that package.json file
does not have a "name" property, because it is not the root package.json
file representing the entire @babel/runtime package.
I considered making the "name" restriction configurable, but that would
have fragmented the caching of optimisticLookupPackageJson. Instead, I
made it return an array of all potentially relevant package.json objects,
which can be safely cached.
This means that the caller has to iterate over the array, but there is
only one call site for this function (in tools/isobuild/package-source.js)
right now, so that wasn't too much work.
Unfortunately, this conversion triggered an error due to one of the
shortcomings of the Babel implementation of TypeScript:
SyntaxError: /tools/tool-env/profile.ts: Namespace not marked type-only declare. Non-declarative namespaces are only supported experimentally in Babel. To enable and review caveats see: https://babeljs.io/docs/en/babel-plugin-transform-typescript
278 | }
279 |
> 280 | export namespace Profile {
| ^
281 | export let enabled = !! process.env.METEOR_PROFILE;
282 |
283 | export function time<TResult>(bucket: string, f: () => TResult) {
at File.buildCodeFrameError (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/core/lib/transformation/file/file.js:261:12)
at transpileNamespace (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/namespace.js:25:25)
at PluginPass.TSModuleDeclaration (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/plugin-transform-typescript/lib/index.js:271:32)
at newFn (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/visitors.js:193:21)
at NodePath._call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:53:20)
at NodePath.call (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:40:17)
at NodePath.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/path/context.js:88:12)
at TraversalContext.visitQueue (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:118:16)
at TraversalContext.visitMultiple (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:85:17)
at TraversalContext.visit (/Users/ben/meteor/dev_bundle/lib/node_modules/@babel/traverse/lib/context.js:144:19)
Example:
"meteor": {
"mainModule": ...,
"testModule": ...,
"nodeModules": {
"recompile": {
"very-modern-package": ["client", "server"],
"alternate-notation-for-client-and-server": true,
"somewhat-modern-package": "legacy",
"another-package": ["legacy", "server"]
}
}
}
The keys of the meteor.nodeModules.recompile configuration object are npm
package names, and the values specify for which bundles those packages
should be recompiled using the Meteor compiler plugins system, as if the
packages were part of the Meteor application.
For example, if an npm package uses const/let syntax or arrow functions,
that's fine for modern and server code, but you would probably want to
recompile the package when building the legacy bundle. To accomplish this,
specify "legacy" or ["legacy"] as the value of the package's property,
similar to somewhat-modern-package above. These strings and arrays of
strings have the same meaning as the second argument to
api.addFiles(files, where) in a package.js file.
This configuration serves pretty much the same purpose as symlinking an
application directory into node_modules, but without any symlinking:
https://forums.meteor.com/t/litelement-import-litelement-html/45042/8?u=benjamn
The meteor.nodeModules.recompile configuration currently applies to the
application node_modules directory only (not to Npm.depends dependencies
in Meteor packages). Recompiled packages must be direct dependencies of
the application.
After much thought, I believe this implementation (#10545) would have
caused severe compatibility problems when using packages published with
earlier versions of Meteor in a Meteor 1.8.2 app, or when publishing
packages with Meteor 1.8.2 for use with earlier Meteor versions.
Specifically, this implementation relied on writing the additional
.npm/package/node_modules resources found by _findSources into the
unibuild JSON file(s), and there just wasn't any good way to make sure the
new JSON format could be safely consumed by previous Meteor versions.
Even if we found a way to hide the new resources from older versions of
Meteor, perhaps by putting them in a new/different property of the
unibuild JSON file, packages published with older Meteor versions might
try to load an npm package with a "module" field without realizing the
code must be compiled, which would likely cause a syntax error in Meteor
1.8.2, since the "module" field always gets preference over the "main"
field of package.json (in Meteor 1.8.2).
When I implemented support for the "module" entry point in package.json
files for client code in #10541, I modified PackageSource#_findSources to
include files found in node_modules that need to be compiled, but my
implementation considered only "local" node_modules directories, like the
one in the application root directory, while neglecting the private
.npm/package/node_modules directories that many Meteor packages have.
This commit includes .npm/**/node_modules when _findSources is scanning a
Meteor package, which should solve issues like #10544, where a Meteor
package imports an npm package that was installed with Npm.depends, and
that npm package has a "module" field in its package.json file, pointing
to an ESM entry point module, but the ESM syntax was not appropriately
compiled, leading to parse errors like "Unexpected token export".
Before lazy compilation was introduced in Meteor 1.7 (#9983), including
the node_modules directories of Meteor packages would likely have been a
big problem for build performance, since there would be that many more
modules to compile. It's still worth making sure this change doesn't
regress build performance for other reasons, but I'm reasonably confident
lazy compilation will save us here, unless there are just too many npm
packages installed via Npm.depends that export ESM modules.
Supporting "module" in package.json for server code is not advisable
because Node.js will be adopting the "type":"module" convention instead,
and in the meantime we need to maintain consistency with Node's module
resolution rules, which only currently pay attention to "main":
https://medium.com/@nodejs/announcing-a-new-experimental-modules-1be8d2d6c2ff
Fixes#10393.
Bumping compiler.BUILT_BY and LINKER_CACHE_SALT because
PR #10414 changes the behavior of the build system in a subtle way that
does not automatically trigger recompilation.
New Meteor apps have the following meteor.testModule in their package.json
files by default
"meteor": {
"testModule": "tests/main.js"
}
When meteor.testModule is defined, it determines the test entry point when
running the `meteor test` command, ignoring legacy file naming conventions
like *.tests.js or *.app-tests.js.
The package-source.js code changed by this commit was incorrect because it
ignored those specially-named test files even when running tests, which
was a problem if the meteor.testModule tried to import them explicitly,
because they would not be properly compiled.
If you're using meteor.testModule, the distinction between `meteor test`
and `meteor test --full-app` matters a bit less, since the test entry
point will be the same for both modes, though you can still check
Meteor.isTest and Meteor.isAppTest at runtime to control test behavior.
According to traditional Meteor file loading rules, tests/ directories are
completely ignored: https://guide.meteor.com/structure.html#special-directories
However, if you specify a meteor.testModule in your package.json that
refers to a file inside a tests/ directory, Meteor should permit modules
to be loaded from tests/, as well as any modules that are imported by the
meteor.testModule entry point.
Fixes#9991.
This reverts commit c5302bb2ba.
Based on conversation with @KoenLav, it seems this optimization was more
of a breaking change than anticipated, and the workaround of creating a
symbolic link to induce compilation is not just annoying on Windows but
also does not satisfy the goal of being able to import from node_modules
(as before) rather than importing via the symbolic link.
We will need to revisit this pitfall in Meteor 1.7.1, but it's important
to unbreak it ASAP in Meteor 1.7.0.1 (since 1.7 is unfortunately final).
https://github.com/meteor/meteor/pull/9826#issuecomment-392596129https://github.com/mozfet/meteor-autoform-materialize/issues/43
For apps that are accidentally compiling too many LESS or SCSS files, the
recommended workaround (for now) is to add the offending directories
and/or files to a .meteorignore file.
This functionality was originally intended to allow importing
compiled-to-JS modules from `node_modules`, by precompiling any modules
found in top-level npm packages, as long as a Meteor compiler plugin was
registered for the module's file extension.
As discussed in #9800, this extra compilation burden can be non-trivial,
especially if you happen to install an npm package such as `less`, which
contains hundreds of `.less` files in the `node_modules/less/test/`
directory.
More generally, this functionality was an early attempt to enable
selective compilation of `node_modules` directories, but it was not a good
solution to that problem, because in almost all cases the extra
compilation was unwanted.
Meteor 1.7 (formerly known as 1.6.2) will give full control over selective
compilation of `node_modules` back to the application developer (#9771),
which should afford a much better solution to this problem. If you've
installed some `.less` or `.scss` or `.ts` files from npm into your
`node_modules` directory, just create a symlink to the package directory
within your application, and those modules will be compiled and become
importable by other JS modules, as if they were part of the application.
Although there was a comment in this code about not applying .meteorignore
files to the contents of node_modules directories, I'm pretty sure that
was the wrong decision, because .meteorignore merely limits what Meteor
tries to compile as application code, and does not actually modify or hide
node_modules files from other parts of Meteor (or Node).
In other words, there's no harm in letting .meteorignore apply to
node_modules, and there may be a LOT of benefit, because it allows the
developer to fight back when compilation descends unexpectedly into an npm
package that contains non-.js[on] files for which a compiler plugin has
been registered, an obscure but not uncommon behavior originally intended
to allow importing CSS assets from npm packages:
* https://github.com/meteor/meteor/issues/6037
* 43659ff561
* a073280e3f
* https://github.com/meteor/meteor/issues/5242
* https://github.com/meteor/meteor/issues/6846
* https://github.com/meteor/meteor/issues/7406
However, we now have a much more powerful tool for selectively compiling
specific npm packages: #9771. In light of this new approach, we should
probably remove the promiscuous node_modules compilation behavior
altogether, as it might speed up rebuild times for many applications whose
developers don't know or care that this behavior exists. For example,
abandoning this behavior would prevent the problem reported here:
https://github.com/meteor/meteor/issues/6950
In the meantime, this commit makes .meteorignore work for node_modules.
Setting meteor.testModule is a great way to specify test entry points
explicitly, rather than relying on Meteor's isTestFilePath heuristics:
https://github.com/meteor/meteor/blob/devel/tools/isobuild/test-files.js
The syntax is identical to meteor.mainModule, so you can set an explicit
meteor.testModule.{client,server,...} for each platform. If a testModule
is not specified for a platform, then Meteor's existing rules about test
file paths apply for that platform, as before.
If a testModule is specified, that module will always be loaded eagerly
when running `meteor test`, in addition to any other modules that load
eagerly because of meteor.mainModule or other rules regarding module
loading. If you run `meteor test` without the `--full-app` option, then no
application JS modules other than the testModule (and any modules imported
by it) will be loaded eagerly.
If meteor.mainModule.{client,server,...} === false, no modules will be
loaded eagerly for that architecture. This is useful if you have an app
with no special app/{client,server} directory structure and you want to
specify an entry point for just the client (or just the server), without
accidentally loading everything on the other architecture.
Determining if Meteor package files should be loaded lazily or eagerly is
a lot easier than doing so for application files, so we should just do
that separately, to avoid any risk of application directory layout logic
interfering with package behavior.
https://github.com/meteor/meteor-feature-requests/issues/135
This change allows applications to specify specific entry points for each
architecture, without relying on `imports` directories to determine the
eagerness/laziness of modules. In other words, it will finally be possible
to build a Meteor app without a special `imports` directory.
Specifically, if `packageJson.meteor.mainModule[architecture]` is defined,
all modules for that architecture will be lazy except for the specified
module, which will be loaded eagerly.
Possible values for `architecture` include "client", "server", "web",
"web.browser", "web.cordova", "os", and so on, just like the second
argument to `api.mainModule(file, where)` in Meteor packages.
In order to match existing behavior, a Meteor application might include
the following in its `package.json` file:
"meteor": {
"mainModule": {
"client": "client/main.js",
"server": "server/main.js"
}
}
These architectures are handled independently, so omitting the "client" or
"server" property would cause that architecture to revert to standard
Meteor loading semantics. In other words, Meteor developers must opt into
this functionality, which is crucial for backwards compatibility.
Note that this functionality applies only to application modules, since
modules in Meteor packages are already lazy by default, and Meteor
packages can already specify entry points by calling `api.mainModule` in
their `package.js` files.
Also note that the loading behavior of non-JavaScript resources is *not*
affected by `packageJson.meteor.mainModule`. Only resources added by
compiler plugins via `addJavaScript` are subject to the new configuration
option. If a compiler plugin calls `addStylesheet` or `addHtml`, those
resources will still be included unconditionally in the HTML document
rendered by the web server. While you could try to import these resources
from JavaScript, you would only be importing any JavaScript resources the
compiler plugin registered using `addJavaScript`, and not the actual HTML
or CSS resources. I welcome feedback on this decision, but if there's no
meaningful way to import a resource, making it lazy just means it won't be
loaded at all.
An ulterior motive for this feature is to enable Meteor apps to have
directory layouts that developers who are not familiar with Meteor can
immediately understand. The special meaning of the `imports` directory and
the surprising eagerness of modules outside of `imports` have always
required some explanation, so this change should reduce that surprise.
Because Meteor strives to be a zero-configuration tool, this is currently
the only supported option in the "meteor" section of `package.json`,
though the available options may be expanded in the future if that's the
best/only way to solve important problems. This would involve adding
additional methods to the `MeteorConfig` class in `project-context.js`,
and then using those methods elsewhere in the `meteor/tools` codebase.
Many npm based packages provide supporting assets that need
to be made available, when used on the web. For example, to
use the `font-awesome` package properly, the
`node_modules/font-awesome/fonts` files need to be made accessible
to incoming web requests.
With Meteor, an easy way to handle this would be to create a
symlink to `node_modules/font-awesome/fonts` from within an
application's `/public` directory. This would then allow all
of `font-awesome`'s font files to be accessed directly by
incoming clients. Unfortunately, while this approach does work
in development when using the Meteor Tool, it does not work when
production bundles are created.
Meteor's isobuild process uses a helper class called
`SymlinkLoopChecker`, to make sure the build process doesn't get
caught up in an infinite loop trying to follow circular symlinks.
Currently, a shared `SymlinkLoopChecker` instance is used to watch
for symlink loops during both the `_findSoures` and `_findAssets`
parts of the isobuild process. `_findSources` is called first, and
covers source files that an application uses from the `node_modules`
directory. The `SymlinkLoopChecker` tracks all of the `node_modules`
directories covered, so they can be watched for in the future (to
prevent duplicate inclusions). Next, `_findAssets` is called using
the same `SymlinkLoopChecker`. This means that if there are any
`node_modules` symlinks used in the `public` or `private`
directories, they will be marked as being duplicates (and stripped),
since they were already covered in the `_findSources` run.
This commit changes things a bit so that both `_findSources` and
`_findAssets` use their own `SymlinkLoopChecker` instance. This
opens up an applications symlink capabilities a bit, while still
preserving some circular symlink safeguards. By doing this, a
production application bundle can now maintain the contents of
`node_modules` based symlinks, used in `public` and `private`.
So in the case of `font-awesome` for example
```
public/fonts --> ../node_modules/font-awesome/fonts
```
becomes the following in the production application bundle
```
bundle/programs/web.browser/app/fonts/[all fonts files]
```
Fixes#7013.
Similar to the treatment given to PackageAPI in my commit
af50b4cc5b.
This clear separation of concerns will be helpful for optimizing
PackageSource#initFromPackageDir.
