This will be useful when we want to be smart with windows file paths later
Also, all of the file calls are asynchronous with fibers now, which comes with
many benefits.
This is a combination of 23 commits. Original messages:
Wrap a large number of fs calls inside files.*
Convert a few more fs calls to files.*
More moving fs.* to files
Implement read/write streams and open/read/close
Get rid of fs from auth.js
Remove fs and unused imports from catalog-local and catalog-remote
Remove unused imports from catalog.js
Replace a whole lot of fs calls
Fix error
Migrate a lot more fs. calls to files.
Add a temporary symlink method
Convert old test to files.*
Use files.pathX instead of path.x everywhere
Replace path.x to files.pathX in tests
Small fixes to files.js and one rename
Make cleanup run in a fiber
Make wrapping functions take function name in case we need it
Add some timeouts and stuff to HCP tests
wrapFsFunc also makes a sync version of the function
Sometimes you just don't want to yield!
Make sure JsImage readFromDisk doesn't yield
Remove unused imports from npm test
Change order of test now that some things don't yield
Fix missing files import, and add a debug error printout
Includes the following changes to Console.js:
- Console.info, Console.warn, Console.debug and Console.error now automatically
line-wrap the output to 80 characters, or the width of the terminal screen (if
known). This is in line with our current style guide on how things should be wrapped!
- Sometimes, there are parts of text that we don't want to line-wrap. For example, if we are
telling the user to run 'meteor long-command --with --options' we don't want to
have a newline in the middle of that! Wrap those commands in Console.command, like
this:
Console.info("something and then run", Console.command(command), "and then");
This also makes them bold if chalk is on, as a nice bonus. So, if we ever turn
chalk back on, the bolding of commands will be more consistent.
- Sometimes, there is bulkier output that we don't want to format at all, including
line-wrapping: log snippets, stack traces, JSON output, etc. In that case, we can use
Console.rawInfo, Console.rawError, Console.rawWarn and Console.rawDebug. Don't use
Console.command inside the raw* functions! It won't be processed (at all).
- There are fancier things that we can do, other than just simply wrapping things.
We can indent:
" Start here and then when wrapping
continue over here".
We frequently do this for commands, for example. In the past, we did this manually --
but we can't do this for long messages that might get wrapped, and anyway, it is
good to codify this instead of counting spaces. Allows us to be better about consistency,
for example.
- We can also add a bulletPoint, which is a small notice in the beginning that looks like
this:
" => Start here and then when wrapping
continue below the bulletPoint".
Since it is a elss intuitive option, I have wrapped most of the time that we use a
bulletPoint into helper functions on the Console.js.
- Some common bulletpoints that we use are:
ASCII Checkboxes (Console.success)
ASCII X-s (Console.failWarn and Console.failInfo)
=> (Console.arrowError, Console.arrowWarn, Console.arrowInfo)
WARNING (Console.labelWarn)
The => are sometimes indented, so they take an optional indent argument, showing how
many spaces to indent by.
The wrapper interface would be less complicated, if there was a more unified conceit behind our
terminal messages. If there is one, it is not documented. My hope is that, in many cases,
moving these to Console will make it easier for someone with great product sense to
clean up our terminal messages. It will also make it easier to write such messages, since
it will be easier to follow an accepted standard.
In the codebase outside of Console:
- Went through and looked at our use of Console.error/info/etc, replacing with rawError/etc
whenever approporiate.
- Went through and modified most of 'stdout' and 'stderr' calls to use the new functions.
I made an exception for stuff that doesn't want a new line at the end, or otherwise does
weird things (ex: print user logs directly), on the basis that, at this juncture, it is
better to be safe than to be sorry.
- Long messages no longer need to break the code style guide by ignoring indentation rules.
Fixed that where approporiate.
- Fixed the tests! A number of our stock messages are actually longer than 80 chars.
- Personal favourite: The Android license agreement is now line wrapped! Much better experience.
- There is some more work to do on:
- longform help (currently comes with built-in linebreaks, would have to change the entire
mechanism for how that works)
- Buildmessage sometimes has headers that start with =>, but they are short. I didn't want to
pass wrapper options all the way to main.captureOrExit before merging the rest of this and
making sure that we like it. Since these messages are fairly short, I don't think that's
likely to be a serious problem.
I hope that this makes life easier for us in the future! No more counting chars, no more breaking
the style guide. Better experience for users with wider terminals (or even shorter terminals!).
Let's give this a try.
You can only request a named set of packages, not a random assortment.
In future commits, we will pre-build these packages into JsImages and
load them from that. Building packages for uniload will eventually not
involve the .build.foo directories at all. (All saved packages will be
built in app context, eventually.)
Now ServiceConnection's guarantee is that once a DDP connection is
successfully negotiated, it won't restart. This relies on the assumption
that the only use of reconnect({_force: true}) is DDP protocol
negotiation!
Drop some unnecessary (and flawed, for this application) `disconnect`
stream events.
Also, remove some unnecessary `new` calls.
Fixes 'meteor mongo some-galaxy app'.
_.once has the problem that if you call the once'd function while it is
still in progress (re-entrantly or in another Fiber), it returns
undefined immediately. That's bad for uniload! uniload already has a
cache, so just use that. (In the future, perhaps detect an attempt to
uniload something that's currently in the process of being uniloaded in
another fiber and block until the other fiber is ready.)
Using instanceof with things you've uniloaded is a little sketchy: maybe
two different uniload calls will end up with two different copies of
Package.meteor.Meteor.Error, and it seems kind of hairy to ensure you're
not mixing and matching copies. However, Meteor.Errors are all tagged
with a string errorType, which fills me with much less fear,
uncertainty, and doubt than instanceof.
- ServiceConnection should never try to reconnect. It's already the case
that we don't hold open ServiceConnections over long periods while
idle; it makes the class much simpler if it corresponds to a single
TCP connection. This also means that as soon as we have one connection
failure (eg you're offline) we can fail instantly instead of retrying
pointlessly.
- Drop the explicit timeout code in ServiceConnection. There's already
timeout handling in stream_client, and now that we don't retry, it
actually takes effect.
- Be more rigorous about uses of Future in ServiceConnection. Ensure
that each Future is only used once (ie, avoid "Future resolved more
than once" errors). Hopefully fixes#2390.
- ServiceConnection constructor now blocks until it's connected (and
throws if there's a connection failure). Maybe this introduces a tiny
bit more latency to the connection, but it makes it much easier to
handle errors properly.
- In packageClient.handlePackageServerConnectionError, show the error
message corresponding to the connection failure.
- In Node, the (newish) error passed to the Stream callback is now a
"DDP.ConnectionError" object. We can detect this in the tool (and we
don't even need to do some complex uniload/instanceof dance, since
error classes made with Meteor.makeErrorType label themselves with a
string errorType). We also no longer have a special
ServiceConnection.ConnectionTimeoutError.
We're going to make uniload use a different flavor of "complete" catalog
soon. So we need to reduce the number of singleton-ish references to
it.
Also, we need one PackageCache per catalog, so stop it from being a
singleton too.
Also eliminated arguments that weren't used anywhere,
and removed an XXX comment that was false (recordPackages
doesn't use buildmessage to report connection failures)
Port a simplified version of Meteor.EnvironmentVariable and
Meteor.bindEnvironment to fiber-helpers.js to deal with this.
Identify uses of fiberHelpers.inFiber and switch them to either
fiberHelpers.bindEnvironment (if the callback they are wrapping is
semantically "part of" the context that creates the callback) or
fiberHelpers.inBareFiber (otherwise).
Without this, concurrency was causing the wrong buildmessage message
sets and jobs to be active when builds yielded.
Moving towards a world where all things that might invoke buildmessage.error are
encouraged to be in a buildmessage.capture.
This commit is the answer to the question "how many small changes need to be
made to add buildmessage.assertInCapture to PackageCache.loadPackageAtPath?"
Next steps include:
- Making catalog.resolveConstraints ALWAYS buildmessage.assertInCapture
(not just when ignoreProjectDeps isn't passed)
- Then changing resolveConstraints to complain using buildmessage
- Removing the process.exit(1) in _ensureDepsUpToDate
- Adding a more structured way to ensure that most commands
call _ensureDepsUpToDate at an unsurprising location
symlink as a special case for runner only
future commits on this branch will add a
package.json/npm-shrinkwrap.json that can be used by "meteor bundle"
users
In particular, this becomes especially important with packaging
since we ping the server with a DDP connection every time you
run an app. Multiple times actually.
So now there's no ECONN error messages printed
springboarding happens infinitely because of build ids
have to manually bootstrap a tropohouse
fixed some other things:
- store package server token in correct domain
- copy files (eg packages pre-publish) with +x flags
- catalog.getReleaseTrack works
- don't pass release to uniload (Meteor.release will always
end up 'UNILOAD')
- fix building meteor-tool again
- stop supporting apps without .meteor/release
- merging unipackages with tools works
springboarding to warehouse releases totally not supported
Previously, we could make a connection, do some method calls, and then
10 seconds later the connection happens to be dropped and the connection
timer fires, which not only throws an unexpected error into the future,
but also resolves the future twice. I think ServiceConnection is just
supposed to time out if you don't hear anything from the server within
10 seconds, so it now no longer times out if you hear things from the
server but then happen to be not connected when 10 seconds has elapsed.
If we set it up before `subscribeAndWait` returns, then we'll end up
with two subscriptions; we don't have the log-reader sub yet, so we
can't stop it when `onReconnect` runs the first time, so we end up with
a redundant subscription. This means that if a real reconnect happens
later, we'll stop the sub that we set up inside `onReconnect`, but not
the initial sub, so we've leaked a sub and end up with duplicate
messages after reconnect.
