The dynamic configuration system has two components: the app, which
reads, and the writer script. The latter is meant for use by humans, and
converts a [live_config] section of the INI file into JSON for storage
in ZooKeeper. The app will read this data on startup and place a watch
on the node to be notified, by ZooKeeper, of changes. This means that
running the writer script with new data will automatically propagate the
changes to every app very quickly, without restart.
The writer script relies on a human-entered password to authenticate
with ZooKeeper. The reddit app uses a different set of credentials
(specified in the INI file) to obtain read-only access to the
configuration data.
Also adds a new "live_config" spec to reddit and plugins. This spec is
parsed at write-time only and the parsed values are stored as JSON in
ZooKeeper.
NOTE: if you're using an old-style install with symlinks to any of these
files, please make sure to copy them out of the repo before merging this
commit.
We need onload to be able to get a client-IP based
tracking validation hash, but it doesn't actually need
to do any database work to make those hashes. It also
fails a huge number of times per day. This moves the hash
calculation out to an external app with no db dependencies.
Subreddit name search is used for recommending
subreddits to users as they type a name on the
submission page.
Also adds a missing script that is needed
to load up the subreddit names into cassandra.
This script should be run occasionally (we do it
once per day) to fill/update the search cache.
* Show "this post is archived" message for things we won't allow votes on.
* Don't mark messages read when pulled via RSS
* Make compact (smartphone) interface respect user's toolbar
* preferences.
* Make mobile interface more friendly for kindles.
* Fix bug that caused comment tree corruption.
* Use cachebuster on traffic pixel for more accurate tracking.
* Make apps restart themselves after a configurable number of requests.
* Move to pycassa 1.0.8.
* Fix bug in calculations for "best" sort.
* Fixes for Firefox Mobile
* Add a global flag to disable editing of the wiki.
* Move the child-comment collapse button to the left.
* Updated list of disallows in robots.txt to save needless hits from
crawlers.
* Fix vote_q by splitting it into vote_link_q and vote_comment_q.
* Fix bug where /reddits crashes due to 'promos' subreddit.
* Add a new Cassandra CF for comment-sort caching
* Add JSONP support
* Support non-auto-renewing PayPal IPNs
* Gold accounting
* Gold features: comments tracking, larger comment limit
* Autorenewing gold
* Google checkout support
* Profile-page sorting for all
* Title-text on the logo
* Hardcache sharding and profiling
* Self serve enhancements
* Add /r/foo/faq
* Make self-centred reddits allow for more verbose selftext
* Much better down-handling of databases
* Add the ability to take a thread-dump from a running process
* Remove the comscore tracker
* Add comments to modqueue (does not back-populate)
* Use Flaptor's Indextank product for search, including santip's patch
* for their new API
* Add Cassandra ConsistencyLevels to the ini file, and storage-conf.xml
* to the public repo
* Patch contributed by umbrae in ticket #929: Add jumpToContent support
* for Keyboard Accessibility
* reddit gold
- paypal/postcard support
- friends with benefits
- profile-page sorting for gold members
- move domain listings into the permacache
* no-repeat on some background images in the sprite (#797)
* fix deleted comments on the mobile site -- preserve author anonymity (#624
* faulty permalinks on pages with all unicode title (#776)
* no more spreadshirt
* reorganize comment_tree and _builder.pyx to clean up cache handling
* fix styling for Bug #78
* Improve handling of /r/all+all (bug #699), etc (not always a 400 now)
* Fix the time listings: sometimes we get passed unicode for self.time
* Don't allow private reddits to show in /r/random
* Try to allow remote sites to show their own favicons in the reddit
toolbar
Speed improvements:
* CommentBuilder refactor rount 1:
* create new sort and parent permacache entries so that we don't need to loa
* update sorts and parents when a new comment comes in
* update non-date sorts when a new comment vote comes in
* add more trace info to timeouts in CommentBuilder
* Some misc. performance hacks (incl. adding _utils.pyx)
* Increase SR description box limit from 1k to 5k
* Fix a bug where we weren't properly allowing hidden items to be hidden
on time-filtered listings
* Make Subreddit._by_name take a list of names like byID
* Upload thumbs to S3 with reduced_redundancy==True
* make rss feeds without a 'feed' get parameter act as if the user is not logged in
* Require a boto that knows about reduced_redundancy
* remove fast_queries from Message.add_props and put the onus on fetching the unread messages from the permacache
* Store the list of popular subreddits in the permacache
* make SubredditTopBar cacheable per-user.
* add (as safe as we can make it) annotation for sql selects to track down requests hitting the db when they shouldn't be.
User submitted features:
* Merge and clean up Phire's selfs-only/no-selfs patch.
* Don't show expired items on time-filtered controversy listings
* Also add the 'hide' button to the toolbar just for breakneckridge.
Next time someone calls me prickly, breakneckridge had better step
up to the plate for me or I'm rolling it back.
====
* Uninstall the python Cassandra package that we previously depended on (it has a namespace conflict with the new package we depend on). To find it:
$ python -c "import cassandra; print cassandra.__file__"
and rm -r the .egg directory after "site-packages/"
* This version relies on cython, so if "make" fails, you may have to install cython via your distro's package manager.
$ cd reddit/r2
$ python setup.py develop # possibly with "sudo" depending on your install
$ make
* Cassandra is now required for the caching layer. An example storage-conf.xml can be found in reddit/srv/cassandra. Make sure that the additional <Keyspace> items are included in your conf file.
* remove the query_queue_reader services if they are running. add new gen_time_listings.sh instead. Suggested cron:
0 */2 * * * $SCRIPTS/gen_time_listings.sh year '("month","year")'
*/5 * * * * $SCRIPTS/gen_time_listings.sh week '("day","week")'
* * * * * $SCRIPTS/gen_time_listings.sh hour '("hour",)'
where $SCRIPTS is the location of this script directory
Features and Bugfixes
====
* Mobile reddit:
* templates are in r2/templates of the form *.compact
* css is r2/public/static/css/compact.css
* beginning of a sass-based (http://sass-lang.com/) compact.scss
* reachable via .compact extension or from the "i" subdomain.
* Cassandra is now *required*, and votes are currently written out to both cassandra and postgres (part of an eventual migration).
* attempt to make the db connection code a little smarter.
* A dropped DB connection will mark the connection dead and randomly attempt to reconnect.
* A dropped db connection on start will permanently mark the connection as dead.
* Calculate the time-filtered top/controversy listings using mapreduce instead of prec_links (new cron job in reddit/scripts)
* allow default user/pass for database to be specified with '*' to fallback on db_user and db_pass in the INI file
* Search feedback buttons
* make deleted comments not show up in your inbox.
* move last_visited into cassandra
* Swallow rare, race-conditiony POST_save/hide/subscribe problems
* Apparently we haven't been breaking properly for the past few weeks.
* The error pages have been broken for the past week.
* Fix /r/all+all
* Provides about.json to the api for subreddits (it already works for users)
* Make sure Accounts have data before checking their name and msgtime
* new bigger arrows for mobile site and a the alien is fingerless now
* Try to update the _fast_query_timestamps less often to alleviate some of the load it causes on the Accounts DB
* Swallow rare, race-conditiony POST_save/hide/subscribe problems
* Two new 404 aliens
* enable a comment page cache if the user isn't author of any of the comments and the page is a non-permalink. votes and friends are updated by javascript
Additions
* descriptive example.ini
* create "trusted_sponsor" who get auto-accepted
* split off url cache from the permacache, with major clean-up to app_globals
* Move is_banned_domain() call into POST_submit() so it can take the sanitized url
* Redirect /r/asdfasdfasdf to the search page, not the create page.
* updates to draw_load.py
* orangered envelope goes to unread, not all.
* Remove "enable whitelisting" checkbox, and switch to new failiens
* Put verification link on email prefs page
Bugfixes:
* split off a minimal api controller for onload calls (which don't care about c.site or c.user, making them cheaper)
* Assume admins want to keep their cookies forever just like everyone else
* /randomrising bugfix
* JSON error page was rendering in html. it now returns "{ error: ${code} }"
* Cassandra
* Add new cassandra libraries that we'll need to setup.py
* Select Cassandra seeds at random rather than in order
* Bugfix in CassandraCache.delete and a faster permacache migration function
* Like other caches, CassandraCaches need to be able to take (and ignore) a 'time' parameter
* add Cassandra to the permacache chain
* beginning of jury duty (later called deputy moderation)
Additions:
* Make /r/friends much cheaper at the expense of sorting
* Add Jury.delete_old(), which removes Account-Trial relations > 3 days old
* Make the pretty_button() template function's callback optional, so that
actionless pretty-buttons can be used on the admin details page
* make .embed listings work for permalink pages (think of this as a first pass to getting blog comments working). Adds 'limit' and 'depth' parameter to permalink pages
* Added final redditheader.html pretty-button class
* new iframe ads; also make button.js static
* Usage sampling
Bugfixes:
* Stop adding batched time query recalculations to the queue at all except through the catch_up_batch_queries function
* Superflous comma might be causing IE7 to barf
* Change the byurl keys again, to fit in memcaches 251 character limit
* Indentation error causing non-sponsors to be able to get to the advert listing
* Move to a custom build of pylibmc that doesn't hold the GIL during blocking operations
* Convert some cache.gets to cache.get_multis, and implement our own thread-safety around pylibmc's client
* Make search caching a little smarter for time searches
* Make the ads not be cached for 30 seconds each, ie. more random
* fix deleted things on profile pages
* Discount 1.6.1
* Lines beginning with spaces are considered code. I don't know why markdown.py didn't trigger this.
* tables in mark down: why not?
* validation of resulting HTML vial libxml to prevent hax.
* private RSS and JSON feeds
* optional whitelists for subreddits
* Moderator messaging
Additions:
* destination sanitization to cut down on XSRF
* cosmetic fix to spam and reported listing
* make the rss feeds on messages useful
* /admin/errors
* Of the types of listings hitting the precomputers (top/controversy by hour/day/week/month/year), the ones over long periods of time don't change often. So we can try to run them at most once per day, and then merge in the day's listings.
* google analytics
* logging queue
* Created empty commentspanel.xml, errorpage.xml, login.xml
* add subreddit rules/info box to submit page
* add 'via' link on messages in moderator inbox
* add a show=all get parameter to link listings to optionally ignore hiding preferences.
* Raise edited timelimit to three mins
* Remove UI that makes it look like you can edit deleted selftexts
* Make it clearer to admins when a link is deleted
* Fix [S] leak on deleted comments
* Fix /user/[deleted] misrendering
* New house ads system
* updated so that minimalcontrollers actually can use the page cache.
* Added /admin/usage
Bugfixes:
* Reduce the number of results that we request from Solr and simplify that caching a bit
* Require a secret key to shut down app-servers
* Make get_title a little more resilient to malformed documents and slow remote servers
* Cause the SearchBuilder to only byID the results that it's going to render instead of all 1000
* Remove ability for an author to XSS himself
* fix spam listings and an xsrf
* More verbose VDestination
* Fixing the famous ?limit=0.1 error, and one last password-validation one
* distinguish deleted comments' and deleted links' error messages
* Don't allow ridiculously long log lines to widen the page
* Bug with HardCache.add() when existing key is expired
* Add adminbox next to domain
