This is intended to reduce the number of critical secrets stored in the
INI file. An initial subset of secrets is moved into the vault to test
things out.
This model will initially be used to transfer subreddit images (used in
the stylesheet) off to a new system, but is intended to be used for
per-wikipage images in the future as well.
The static files S3 bucket has been getting a lot larger recently,
between subreddit stylesheets being in there and the static file cleaner
being disabled due to a bug. This is causing the deploy process to take
upwards of 3 minutes just to determine that no files need to be uploaded
to the bucket.
As a short-term workaround, this changes the uploader to check each key
individually with an S3 HEAD request rather than listing the whole
bucket. This is slower than best case of listing the bucket, but is
significantly faster than the current condition (~25 second runtime
now).
The dynamic configuration system has two components: the app, which
reads, and the writer script. The latter is meant for use by humans, and
converts a [live_config] section of the INI file into JSON for storage
in ZooKeeper. The app will read this data on startup and place a watch
on the node to be notified, by ZooKeeper, of changes. This means that
running the writer script with new data will automatically propagate the
changes to every app very quickly, without restart.
The writer script relies on a human-entered password to authenticate
with ZooKeeper. The reddit app uses a different set of credentials
(specified in the INI file) to obtain read-only access to the
configuration data.
Also adds a new "live_config" spec to reddit and plugins. This spec is
parsed at write-time only and the parsed values are stored as JSON in
ZooKeeper.
NOTE: if you're using an old-style install with symlinks to any of these
files, please make sure to copy them out of the repo before merging this
commit.
We need onload to be able to get a client-IP based
tracking validation hash, but it doesn't actually need
to do any database work to make those hashes. It also
fails a huge number of times per day. This moves the hash
calculation out to an external app with no db dependencies.
