Omitting `keep_blank_values` was dropping blank query parameters.
Furthermore, converting the output of `parse_qsl` to a dictionary
was unnecessarily modifying the order of parameters since dicts
are not ordered. Fortunately `urllib.urlencode` also accepts a
sequence of two-element tuples and the order of parameters in
the encoded string will match the order of parameter tuples in the
sequence.
Endpoint for retrieving subreddits to make new
users aware of without actually subscribing
them while doing the onboarding process.
The list of featured subreddits will be managed
separately from the default ones.
For testing purposes, we add a subreddit to show
up via the inject_test_data script.
Complete iteration of our transition to an auction model. Non-sponsor
users can only create auction campaigns; sponsors can create either.
All other users are forced to create legacy fixed CPM campaigns.
* The constructor for `SSTableSimpleUnsortedWriter` changed
* `sstableloader` changed its directory structure. Now we just don't manage it at all in `tuples_to_sstables` and make the caller do the work
**Explanation**: compute_time_listings is slow. Really slow. At a quick glance, here are the jobs running right now:
date: Sun Jan 17 20:04:56 PST 2016
-rw-rw-r-- 1 ri ri 1.2G Jan 17 12:37 comment-week-data.dump
-rw-rw-r-- 1 ri ri 683M Jan 17 12:25 comment-week-thing.dump
-rw-rw-r-- 1 ri ri 53G Jan 16 07:13 comment-year-data.dump
-rw-rw-r-- 1 ri ri 31G Jan 16 04:37 comment-year-thing.dump
-rw-rw-r-- 1 ri ri 276M Jan 17 17:04 link-week-data.dump
-rw-rw-r-- 1 ri ri 70M Jan 17 17:03 link-week-thing.dump
So the currently running top-comments-by-year listing has been running for nearly 37 hours and isn't done. top-comments-by-week has been running for 8 hours. top-links-by-week has been running for 3 hours. And this is just me checking on currently running jobs, not actual completion times.
The slow bit is the actual writing to Cassandra in `write_permacache`. This is mostly because `write_permacache` is extremely naive and blocks waiting for individual writes with no batching or parallelisation. There are a lot of ways to work around this and some of them will become easier when we're not longer writing out to the permacache at all, but until then (and even after that) this approach lets us keep doing the simple-to-understand thing while parallelising some of the work.
**The approach**: `compute_time_listings` is written as a mapreduce job in our `mr_tools` toolkit, with `write_permacache` as the final reducer. In `mr_tools`, you can run multiple reducers as long as a given reducer can be guaranteed to receive all of the keys for the same key. So this patch adds `hashdist.py`, a tool that runs multiple copies of a target job and distributes lines to them from stdin using their first tab-delimited field to meet this promise. (The same script could apply to mappers and sorts too but in my tests for this job the gains were minimal because `write_permacache` is still the bottleneck up to a large number of reducers.)
**Numbers**: A top-links-by-hour listing in prod right now takes 1m46.387s to run. This patch reduces that to 0m43.960s using 2 jobs (a 60% savings). That top-links-by-week job that before I killed after 3 hours completed in 56m47.329s. The top-links-by-year job that I killed last week at over 36 hours finished in 19 hours.
**Downsides**: It costs some additional RAM: roughly 10mb for hashdist.py and 100mb in memory for each additional copy of the job. It multiplies the effective load on Cassandra by the number of jobs (although I have no reason to believe that it's practical to overload Cassandra this way right now; I've tested up to 5 jobs).
**Further work**: with this we could easily do sort|reducer fusion to significantly reduce the work required by the sorter. `hashdist.py` as written is pretty slow and is only acceptable because `write_permcache` is even slower; a non-Python implementation would be straight forward and way faster.
By default, Linux's sort(1) uses locale-based sorting. Normally this is what
humans want, but for mapreduce it breaks the guarantee that the same reducer
always sees each instance of the same key. Here's an example:
user/comment/top/week/1102922 26 1453516098.92 t1_cz8jgq9
user/comment/top/week/1102922 3 1453527927.97 t1_cz8ovzj
user/comment/top/week/11029224 1 1453662674.45 t1_cza98tb
user/comment/top/week/1102922 4 1453515976.97 t1_cz8jee8
user/comment/top/week/1102922 4 1453519790.67 t1_cz8lavb
user/comment/top/week/11029224 2 1453827188.31 t1_czcotf1
user/comment/top/week/1102922 7 1453521946.74 t1_cz8mb50
user/comment/top/week/1102922 7 1453524230.93 t1_cz8ncj2
user/comment/top/week/1102922 7 1453527760.32 t1_cz8otkx
user/comment/top/week/1102922 7 1453528700.96 t1_cz8p6u3
user/comment/top/week/11029228 1 1453285875.44 t1_cz525gu
user/comment/top/week/11029228 1 1453292202.65 t1_cz53ulm
user/comment/top/week/11029228 1 1453292232.55 t1_cz53uxe
According to sort(1) using the default locale, this is already sorted.
Unfortunately, that means that to a reducer this list represents 6 different
listings (each of which will overwrite the previous runs of the same listing).
But that's not what we want. It's actually two listings, like:
user/comment/top/week/1102922 26 1453516098.92 t1_cz8jgq9
user/comment/top/week/1102922 3 1453527927.97 t1_cz8ovzj
user/comment/top/week/1102922 4 1453515976.97 t1_cz8jee8
user/comment/top/week/1102922 4 1453519790.67 t1_cz8lavb
user/comment/top/week/1102922 7 1453521946.74 t1_cz8mb50
user/comment/top/week/1102922 7 1453524230.93 t1_cz8ncj2
user/comment/top/week/1102922 7 1453527760.32 t1_cz8otkx
user/comment/top/week/1102922 7 1453528700.96 t1_cz8p6u3
user/comment/top/week/11029224 1 1453662674.45 t1_cza98tb
user/comment/top/week/11029224 2 1453827188.31 t1_czcotf1
user/comment/top/week/11029228 1 1453285875.44 t1_cz525gu
user/comment/top/week/11029228 1 1453292202.65 t1_cz53ulm
user/comment/top/week/11029228 1 1453292232.55 t1_cz53uxe
To do this, we need to set the enviroment variable LC_ALL=C when running sort(1)
to indicate that the sorting should operate only on the raw bytes.
It looks like this has been broken since the Trusty Tahr upgrade.
The cause of the bug is that if we fail to start because someone else has already started, we still delete their files.
From job-02 right now:
write_permacache() # comment ("day", "week+
write_permacache() # link ("day","week")
write_permacache() # link ("day","week")
write_permacache() # comment ("day", "week+
write_permacache() # link ("month","year")
write_permacache() # comment ("month", "ye+
* Make them lockless, because mr_top is the only one that ever writes to it.
This avoids a lot of memcached round trips
* Don't set them to permacache_memcaches. Delete from it instead.
This keeps us from blowing out that whole cache with listings that will never
be read out of every time mr_top runs.
* Fix a performance bug in _mr_tools.mr_reduce_max_per_key
This is a rewrite of much of the code related to voting. Some of the
improvements include:
- Detangling the whole process of creating and processing votes
- Creating an actual Vote class to use instead of dealing with
inconsistent ad-hoc voting data everywhere
- More consistency with naming and other similar things like vote
directions (previously had True/False/None in some places,
1/-1/0 in others, etc.)
- More flexible methods in determining and applying the effects of votes
- Improvement of modhash generation/validation
- Removing various obsolete/unnecessary code
The write_live_config script often gives really strange results when
trying to display diffs of changes to dict values, since the ordering of
a dict is not defined. So key/value pairs will sometimes be rearranged
between the old and new versions, creating a confusing diff.
This changes to use the pprint module to generate string representations
for dicts, because it sorts the dict by key before outputting it, so we
should get consistent representations that can be compared more easily.
http://pylons-webframework.readthedocs.org/en/latest/upgrading.html
This requires several code changes:
* pylons `config` option must be explicitly passed during setup
* the pylons global has been renamed from `g` to `app_globals`
* the pylons global has been renamed from `c` to `tmpl_context`
* set pylons.strict_tmpl_context = False (instead of pylons.strict_c)
* redirect_to() has been swapped for redirect()
* must implement `ErrorDocuments` middleware ourselves
pylons 1.0 also required an upgrade of routes from 1.11 to 1.12. This
required the following changes:
* set Mapper.minimization = True (the default value changed)
* set Mapper.explicit = False (the default value changed)
Turns out that SequenceMatcher is not quite as neat as it appears. When
diffing a string change like:
old: "this is the old string value"
new: "on"
It was displayed as:
- "this is the "
- "ld stri"
- "g value"
Since the "o" and "n" were kept. This just displays it as a wholesale
key removal/addition unless the strings are at least 50% similar.
The script would previously dump out the entire new parsed live
configuration, which (now that we have a lot of fields in it) made it
difficult to find the ones that had actually changed. This fetches the
existing live config, then compares it to the new one and only outputs
any data that has changed for confirmation.
There's no index on SRMember.c.rel_id so instead sort the query
by SRMember.c.thing2_id (the user's id). Also the timestamp for
writing to C* is updated to an integer timestamp corresponding to
when the dual write was deployed.
We've changed the url structure of image previews a number of times, which
breaks everything uploaded prior to the latest version. This script should
find all preview images that have been uploaded thus far, move them to the
appropriate place, and save an updated and correct storage url in every Link
that uses them.
See http://docs.python-requests.org/en/latest/api/#migrating-to-1-x
for the rationale,
`.json()` also differs from `.json` in that it `raise`s instead of
returning `None` on a decoding error, but that shouldn't affect us
anywhere.
Conflicts:
r2/r2/lib/media.py
If you go to a userpage and sort by top (in either the overview or comments
tabs), and restrict the time range to anything other than "all time", no
comments will be shown.
The data in these listings is built from functions in `lib/db/queries.py`
(specifically from `get_comments()` down). This ends up trying to pull the
query results from permacache (in `CachedResults.fetch_multi()`), defaulting to
an empty list if no cache entry is found.
Now, the cache entry is supposed to be populated periodically by a cronjob that
calls `scripts/compute_time_listings`. This script (and its Python helpers in
`lib/mr_top.py` and `lib/mr_tools/`) generates a dump of data from Postgresql,
then reads through that and builds up entries to insert into the cache. As
with many scripts of this sort, it expects to get in some bad data, and so
performs some basic sanity checks.
The problem is that the sanity checks have been throwing out all comments.
With no new comments, there's nothing new to put into the cache!
The root of this was a refactoring in reddit/reddit@3511b08 that combined
several different scripts that were doing similar things. Unfortunately, we
ended up requiring the `url` field on comments, which doesn't exist because,
well, comments aren't links.
Now we have two sets of fields that we expect to get, one for comments and one
for links, and all is good.
We also now have a one-line summary of processed/skipped entries printed out,
which will help to make a problem like this more obvious in the future.
This new script attempts to generate some subreddits that are more like
production data. It first pulls down data from reddit.com, then uses
markov chains to generate new data for insertion into the databases.
* configuration now comes from the command line so it's easier to use
for multiple projects.
* the bucket is now an s3 url allowing a path prefix to be added to
files.
* authentication now comes from boto's credential system which allows
us to use IAM roles.
This takes our current config payload from 4700 bytes to 1700. The goal
is to reduce zookeeper network load during config changes as well as app
restarts during deploys.
This adds in two redirects - `event_click` and `event_redirect` - `event_click`
to allow appending in a user ID to an event before redirect, if we require one,
and `event_redirect` to service a local evented redirect, similar to ad clicks.
`event_click` is necessary for tracking clicks from users on embeds, which are
served via redditmedia, and therefore are always anonymous. When a user clicks
through, we want to know who they were and redirect them on their way. Because
of the way we're using nginx to store events as an access log right now, this
means we'll need to use two redirects: one to append the session ID and
another to store the event with the proper session ID.
