The write_live_config script often gives really strange results when
trying to display diffs of changes to dict values, since the ordering of
a dict is not defined. So key/value pairs will sometimes be rearranged
between the old and new versions, creating a confusing diff.
This changes to use the pprint module to generate string representations
for dicts, because it sorts the dict by key before outputting it, so we
should get consistent representations that can be compared more easily.
Turns out that SequenceMatcher is not quite as neat as it appears. When
diffing a string change like:
old: "this is the old string value"
new: "on"
It was displayed as:
- "this is the "
- "ld stri"
- "g value"
Since the "o" and "n" were kept. This just displays it as a wholesale
key removal/addition unless the strings are at least 50% similar.
The script would previously dump out the entire new parsed live
configuration, which (now that we have a lot of fields in it) made it
difficult to find the ones that had actually changed. This fetches the
existing live config, then compares it to the new one and only outputs
any data that has changed for confirmation.
This takes our current config payload from 4700 bytes to 1700. The goal
is to reduce zookeeper network load during config changes as well as app
restarts during deploys.
Ok, now I'm getting some angst in my commit messages like my
predecessors had. I understand now. It's a terrible burden. Why must
the calendar progress? Why must numbers increment? The world is
forever turning.
The future is here.
It is 2014.
The dynamic configuration system has two components: the app, which
reads, and the writer script. The latter is meant for use by humans, and
converts a [live_config] section of the INI file into JSON for storage
in ZooKeeper. The app will read this data on startup and place a watch
on the node to be notified, by ZooKeeper, of changes. This means that
running the writer script with new data will automatically propagate the
changes to every app very quickly, without restart.
The writer script relies on a human-entered password to authenticate
with ZooKeeper. The reddit app uses a different set of credentials
(specified in the INI file) to obtain read-only access to the
configuration data.
Also adds a new "live_config" spec to reddit and plugins. This spec is
parsed at write-time only and the parsed values are stored as JSON in
ZooKeeper.
