We've changed the url structure of image previews a number of times, which
breaks everything uploaded prior to the latest version. This script should
find all preview images that have been uploaded thus far, move them to the
appropriate place, and save an updated and correct storage url in every Link
that uses them.
See http://docs.python-requests.org/en/latest/api/#migrating-to-1-x
for the rationale,
`.json()` also differs from `.json` in that it `raise`s instead of
returning `None` on a decoding error, but that shouldn't affect us
anywhere.
Conflicts:
r2/r2/lib/media.py
If you go to a userpage and sort by top (in either the overview or comments
tabs), and restrict the time range to anything other than "all time", no
comments will be shown.
The data in these listings is built from functions in `lib/db/queries.py`
(specifically from `get_comments()` down). This ends up trying to pull the
query results from permacache (in `CachedResults.fetch_multi()`), defaulting to
an empty list if no cache entry is found.
Now, the cache entry is supposed to be populated periodically by a cronjob that
calls `scripts/compute_time_listings`. This script (and its Python helpers in
`lib/mr_top.py` and `lib/mr_tools/`) generates a dump of data from Postgresql,
then reads through that and builds up entries to insert into the cache. As
with many scripts of this sort, it expects to get in some bad data, and so
performs some basic sanity checks.
The problem is that the sanity checks have been throwing out all comments.
With no new comments, there's nothing new to put into the cache!
The root of this was a refactoring in reddit/reddit@3511b08 that combined
several different scripts that were doing similar things. Unfortunately, we
ended up requiring the `url` field on comments, which doesn't exist because,
well, comments aren't links.
Now we have two sets of fields that we expect to get, one for comments and one
for links, and all is good.
We also now have a one-line summary of processed/skipped entries printed out,
which will help to make a problem like this more obvious in the future.
This new script attempts to generate some subreddits that are more like
production data. It first pulls down data from reddit.com, then uses
markov chains to generate new data for insertion into the databases.
* configuration now comes from the command line so it's easier to use
for multiple projects.
* the bucket is now an s3 url allowing a path prefix to be added to
files.
* authentication now comes from boto's credential system which allows
us to use IAM roles.
This takes our current config payload from 4700 bytes to 1700. The goal
is to reduce zookeeper network load during config changes as well as app
restarts during deploys.
This adds in two redirects - `event_click` and `event_redirect` - `event_click`
to allow appending in a user ID to an event before redirect, if we require one,
and `event_redirect` to service a local evented redirect, similar to ad clicks.
`event_click` is necessary for tracking clicks from users on embeds, which are
served via redditmedia, and therefore are always anonymous. When a user clicks
through, we want to know who they were and redirect them on their way. Because
of the way we're using nginx to store events as an access log right now, this
means we'll need to use two redirects: one to append the session ID and
another to store the event with the proper session ID.
Thanks to Nathanael A. Hoyle for the report! Some of these may have
been exploitable due to pointer arithmetic before reads / writes.
Just bail out if we can't allocate.
Double checking in the click app and in the processing scripts was
difficult. Just trust the click app and assume any request that got
a 302 response is valid.
Some advertisers set their ad's url to an intermediate tracker so
they can independently track clicks. This results in a series of
redirects like this:
reddit tracker > intermediate tracker > final destination
The ad's url is communicated to the reddit tracker through a query
parameter which is urlencoded on reddit.com and then unquoted when
being handled by the reddit tracker. This unquoting causes problems
if there is an intermediate tracker with its own query string
that needs to be urlencoded. This commit adds handling for those query
strings.
The queries are used to find the ids of PromoCampaign or Link objects
and we don't need the many (one per campaign per subreddit target per day)
PromotionWeights objects.
Ok, now I'm getting some angst in my commit messages like my
predecessors had. I understand now. It's a terrible burden. Why must
the calendar progress? Why must numbers increment? The world is
forever turning.
The future is here.
It is 2014.
Previously, the subreddit/domain and account precomputers were separate.
This merges the two and improves their portability in the process.
Because of the increased portability, the precomputer can now be added
to the install script by default.
This attribute can serve as a handy indicator that a user is a moderator
somewhere and can therefore replace the more costly modship lookup in
reddit_base.
