reddit relies on pycassa/pycassa@91a5887def
but at time of writing there still isn't a release that contains this
patch. Until that's available, we'll add a custom download link.
This has a couple of advantages:
* interoperability with Cassandra tools (cassandra-cli / cqlsh)
* smaller by 5-6 bytes on every column
From here on out, date columns will be written in the Cassandra
standard 8-byte integer (number of milliseconds since epoch) format.
Old-style stringified epoch seconds will be read properly.
This relies on Pycassa's new UTC-based behaviour in 1.7.
* Fixes app segfaults when memcache not available.
* Remove "time" parameter from r2.lib.cache.CMemcache.delete_multi
since it never worked anyway and was removed from pylibmc.
* Show "this post is archived" message for things we won't allow votes on.
* Don't mark messages read when pulled via RSS
* Make compact (smartphone) interface respect user's toolbar
* preferences.
* Make mobile interface more friendly for kindles.
* Fix bug that caused comment tree corruption.
* Use cachebuster on traffic pixel for more accurate tracking.
* Make apps restart themselves after a configurable number of requests.
* Move to pycassa 1.0.8.
* Fix bug in calculations for "best" sort.
* Fixes for Firefox Mobile
* Add a global flag to disable editing of the wiki.
* Move the child-comment collapse button to the left.
* Updated list of disallows in robots.txt to save needless hits from
crawlers.
* Fix vote_q by splitting it into vote_link_q and vote_comment_q.
* Fix bug where /reddits crashes due to 'promos' subreddit.
* Upgrade discount to 1.6.8
* Start reading Votes, Saves, and Hides from Cassandra (while still dual-writing all three)
* Some XSS fixes
* Significant Gold buying and gifting improvements
- Move /api/ipn to /ipn
* Allow non-US countries to buy sponsored links
* Increase embed.ly scope
* redd.it support
* Allow postgres port number to be specified in ini file (this changes the format of the .ini file)
* Upgrade Cassandra to 0.7
- Change g.urlcache to LinksByURL
- Translate storage-conf.xml to cassandra.yaml
- TTL support (and enable on Hides)
- Move permacache keyspace to inside reddit keyspace
* The stalecache: a local memcached that contains slightly old information to speed up some lookups
* Switch to patched Paste that is hopefully download.gz-proof
* Don't store votes on things > 30 days old
* Many many bugfixes/small features
* Add a new Cassandra CF for comment-sort caching
* Add JSONP support
* Support non-auto-renewing PayPal IPNs
* Gold accounting
* Gold features: comments tracking, larger comment limit
* Autorenewing gold
* Google checkout support
* Profile-page sorting for all
* Title-text on the logo
* Hardcache sharding and profiling
* Self serve enhancements
* Add /r/foo/faq
* Make self-centred reddits allow for more verbose selftext
* Much better down-handling of databases
* Add the ability to take a thread-dump from a running process
* Remove the comscore tracker
* Add comments to modqueue (does not back-populate)
* Use Flaptor's Indextank product for search, including santip's patch
* for their new API
* Add Cassandra ConsistencyLevels to the ini file, and storage-conf.xml
* to the public repo
* Patch contributed by umbrae in ticket #929: Add jumpToContent support
* for Keyboard Accessibility
* reddit gold
- paypal/postcard support
- friends with benefits
- profile-page sorting for gold members
- move domain listings into the permacache
* no-repeat on some background images in the sprite (#797)
* fix deleted comments on the mobile site -- preserve author anonymity (#624
* faulty permalinks on pages with all unicode title (#776)
* no more spreadshirt
* reorganize comment_tree and _builder.pyx to clean up cache handling
* fix styling for Bug #78
* Improve handling of /r/all+all (bug #699), etc (not always a 400 now)
* Fix the time listings: sometimes we get passed unicode for self.time
* Don't allow private reddits to show in /r/random
* Try to allow remote sites to show their own favicons in the reddit
toolbar
Speed improvements:
* CommentBuilder refactor rount 1:
* create new sort and parent permacache entries so that we don't need to loa
* update sorts and parents when a new comment comes in
* update non-date sorts when a new comment vote comes in
* add more trace info to timeouts in CommentBuilder
* Some misc. performance hacks (incl. adding _utils.pyx)
* Increase SR description box limit from 1k to 5k
* Fix a bug where we weren't properly allowing hidden items to be hidden
on time-filtered listings
* Make Subreddit._by_name take a list of names like byID
* Upload thumbs to S3 with reduced_redundancy==True
* make rss feeds without a 'feed' get parameter act as if the user is not logged in
* Require a boto that knows about reduced_redundancy
* remove fast_queries from Message.add_props and put the onus on fetching the unread messages from the permacache
* Store the list of popular subreddits in the permacache
* make SubredditTopBar cacheable per-user.
* add (as safe as we can make it) annotation for sql selects to track down requests hitting the db when they shouldn't be.
User submitted features:
* Merge and clean up Phire's selfs-only/no-selfs patch.
* Don't show expired items on time-filtered controversy listings
* Also add the 'hide' button to the toolbar just for breakneckridge.
Next time someone calls me prickly, breakneckridge had better step
up to the plate for me or I'm rolling it back.
====
* Uninstall the python Cassandra package that we previously depended on (it has a namespace conflict with the new package we depend on). To find it:
$ python -c "import cassandra; print cassandra.__file__"
and rm -r the .egg directory after "site-packages/"
* This version relies on cython, so if "make" fails, you may have to install cython via your distro's package manager.
$ cd reddit/r2
$ python setup.py develop # possibly with "sudo" depending on your install
$ make
* Cassandra is now required for the caching layer. An example storage-conf.xml can be found in reddit/srv/cassandra. Make sure that the additional <Keyspace> items are included in your conf file.
* remove the query_queue_reader services if they are running. add new gen_time_listings.sh instead. Suggested cron:
0 */2 * * * $SCRIPTS/gen_time_listings.sh year '("month","year")'
*/5 * * * * $SCRIPTS/gen_time_listings.sh week '("day","week")'
* * * * * $SCRIPTS/gen_time_listings.sh hour '("hour",)'
where $SCRIPTS is the location of this script directory
Features and Bugfixes
====
* Mobile reddit:
* templates are in r2/templates of the form *.compact
* css is r2/public/static/css/compact.css
* beginning of a sass-based (http://sass-lang.com/) compact.scss
* reachable via .compact extension or from the "i" subdomain.
* Cassandra is now *required*, and votes are currently written out to both cassandra and postgres (part of an eventual migration).
* attempt to make the db connection code a little smarter.
* A dropped DB connection will mark the connection dead and randomly attempt to reconnect.
* A dropped db connection on start will permanently mark the connection as dead.
* Calculate the time-filtered top/controversy listings using mapreduce instead of prec_links (new cron job in reddit/scripts)
* allow default user/pass for database to be specified with '*' to fallback on db_user and db_pass in the INI file
* Search feedback buttons
* make deleted comments not show up in your inbox.
* move last_visited into cassandra
* Swallow rare, race-conditiony POST_save/hide/subscribe problems
* Apparently we haven't been breaking properly for the past few weeks.
* The error pages have been broken for the past week.
* Fix /r/all+all
* Provides about.json to the api for subreddits (it already works for users)
* Make sure Accounts have data before checking their name and msgtime
* new bigger arrows for mobile site and a the alien is fingerless now
* Try to update the _fast_query_timestamps less often to alleviate some of the load it causes on the Accounts DB
* Swallow rare, race-conditiony POST_save/hide/subscribe problems
* Two new 404 aliens
* enable a comment page cache if the user isn't author of any of the comments and the page is a non-permalink. votes and friends are updated by javascript
Additions
* descriptive example.ini
* create "trusted_sponsor" who get auto-accepted
* split off url cache from the permacache, with major clean-up to app_globals
* Move is_banned_domain() call into POST_submit() so it can take the sanitized url
* Redirect /r/asdfasdfasdf to the search page, not the create page.
* updates to draw_load.py
* orangered envelope goes to unread, not all.
* Remove "enable whitelisting" checkbox, and switch to new failiens
* Put verification link on email prefs page
Bugfixes:
* split off a minimal api controller for onload calls (which don't care about c.site or c.user, making them cheaper)
* Assume admins want to keep their cookies forever just like everyone else
* /randomrising bugfix
* JSON error page was rendering in html. it now returns "{ error: ${code} }"
* Make the new targeted self-serve promotion the default
* add TheOatmeal's comic to our source. (http://www.theoatmeal.com)
* add targetting
* add rerunning or suplementing existing campaigns
* move all sponsored link listings to be precomputed
* deputy moderation
* /about/trials and /about/modqueue
* Have trials train the spam filter
* Allow admins to see all trials, site-wide, at /r/all/about/trials
* Allow trials to be ended while votes are still coming in when the jury opinion is near-unanimous
* offload scraping to embed.ly
* Use boto for s3cp instead of forking curl
Additions:
* migrated all graphs to use Flot
* Implement per-user suspiciousness quota and print over-quota trace
* Keep track of how many questionable links a user has outstanding at any given time
* add links to the FAQ and walkthru
* add a differnt thumbnail for self posts (thanks to licensplate)
* Add some visual tracking for comments trees
* Relabel: ban / unban => remove / approve
* Use big remove / approve buttons when the listing relates to moderation,
or when the item has reports or is on trial
* Merge unban, ignore, and a new approve this non-banned link into one
* Be a little smarter about the way we allow caching of subreddit stylesheets
* make cassandra's cache chain cache negative results (and yes that commit message is bigger than the patch)
* add All, friends, random, and mod to topbar
* add a local render precache (optional)
* better noimage.png from paradox460
* Add a 'legacy' mode to CMemcache that will allow us to use it as a drop-in replacement for testing purposes. Modifies python-memcached to use the same 'crc' hashing algorithm that pylibmc does, and removes a feature in python-memcached that we aren't using. Contributed by Joe Terranova <joeterranova@gmail.com>
* Everyone sees cake
* Added g.system_user
Bugfixes:
* improve validation of media embed, and remove a todo from link.py by putting all of the child div stuff in pages
* Feed/JSON updates
* Add selftext to RSS feeds
* Add subreddit names to Comments in JSON
* add is_self param to indicate self posts
* Don't shade the background of child comments of spam
* shorted the cache lifetime for AllSR to 1 min
* Tell everyone that we actually own the trademarks we claim to own (footer update)
* You no longer get an orange alien when one of your reddits sends an automated message to a user
* Cassandra
* Add new cassandra libraries that we'll need to setup.py
* Select Cassandra seeds at random rather than in order
* Bugfix in CassandraCache.delete and a faster permacache migration function
* Like other caches, CassandraCaches need to be able to take (and ignore) a 'time' parameter
* add Cassandra to the permacache chain
* beginning of jury duty (later called deputy moderation)
Additions:
* Make /r/friends much cheaper at the expense of sorting
* Add Jury.delete_old(), which removes Account-Trial relations > 3 days old
* Make the pretty_button() template function's callback optional, so that
actionless pretty-buttons can be used on the admin details page
* make .embed listings work for permalink pages (think of this as a first pass to getting blog comments working). Adds 'limit' and 'depth' parameter to permalink pages
* Added final redditheader.html pretty-button class
* new iframe ads; also make button.js static
* Usage sampling
Bugfixes:
* Stop adding batched time query recalculations to the queue at all except through the catch_up_batch_queries function
* Superflous comma might be causing IE7 to barf
* Change the byurl keys again, to fit in memcaches 251 character limit
* Indentation error causing non-sponsors to be able to get to the advert listing
* Move to a custom build of pylibmc that doesn't hold the GIL during blocking operations
* Convert some cache.gets to cache.get_multis, and implement our own thread-safety around pylibmc's client
* Make search caching a little smarter for time searches
* Make the ads not be cached for 30 seconds each, ie. more random
* fix deleted things on profile pages
* Discount 1.6.1
* Lines beginning with spaces are considered code. I don't know why markdown.py didn't trigger this.
* tables in mark down: why not?
* validation of resulting HTML vial libxml to prevent hax.
* private RSS and JSON feeds
* optional whitelists for subreddits
* Moderator messaging
Additions:
* destination sanitization to cut down on XSRF
* cosmetic fix to spam and reported listing
* make the rss feeds on messages useful
* /admin/errors
* Of the types of listings hitting the precomputers (top/controversy by hour/day/week/month/year), the ones over long periods of time don't change often. So we can try to run them at most once per day, and then merge in the day's listings.
* google analytics
* logging queue
* Created empty commentspanel.xml, errorpage.xml, login.xml
* add subreddit rules/info box to submit page
* add 'via' link on messages in moderator inbox
* add a show=all get parameter to link listings to optionally ignore hiding preferences.
* Raise edited timelimit to three mins
* Remove UI that makes it look like you can edit deleted selftexts
* Make it clearer to admins when a link is deleted
* Fix [S] leak on deleted comments
* Fix /user/[deleted] misrendering
* New house ads system
* updated so that minimalcontrollers actually can use the page cache.
* Added /admin/usage
Bugfixes:
* Reduce the number of results that we request from Solr and simplify that caching a bit
* Require a secret key to shut down app-servers
* Make get_title a little more resilient to malformed documents and slow remote servers
* Cause the SearchBuilder to only byID the results that it's going to render instead of all 1000
* Remove ability for an author to XSS himself
* fix spam listings and an xsrf
* More verbose VDestination
* Fixing the famous ?limit=0.1 error, and one last password-validation one
* distinguish deleted comments' and deleted links' error messages
* Don't allow ridiculously long log lines to widen the page
* Bug with HardCache.add() when existing key is expired
* Add adminbox next to domain
* Activate negative-result caching for HardCache chain
* begin migration to pylibmc:
* Add pylibmc to the list of required packages in preparation for the replacement of the memcached library
* Start using pylibmc for the rendercaches
* Tweak the computation of the normalized hot page to be a bit faster when the precomputer is available, by relying on the precomputer's internal permacached structure.
* Default to a SelfEmptyingCache for scripts run from `paster run'. Note that processes that run forever are still responsible for resetting their local-caches, but this can now be done with g.reset_caches()
* threaded messaging patch part 1: backend changes only. This will allow migrate.py to be run to move new onto inbox and will start tracking message trees for users.
* Specify some queries to run at most once per day
* Refactored safemarkdown() and added soup testing
Additions:
* Added _byID_rel()
* Made error messages more verbose for:
1. byID lookups of too-big thing_ids
2. memcache failures
3. Solr Nones
* Award._all_awards() now sorts by date
* Trophy.by_{account,award}() now cache properly
* new feedback page with helpful links
* Try to reduce the length of the query-queue by not adding known-long queries at all, rather than adding them and skipping them
* whitespace clean up
* simplify the 'why did my CC get denied' email checking.
* added missing translation strings and users now get PMs when they are added as translators
Bugfixes:
* Fix a bug in unsaving
* BeautifulSoup stopped hosting 3.0.7a, but 3.0.8 still uses the good parser
* Better search error handling
* Properly reset the cache-chains (incl. the hardcache; d'oh!) per-request
* Fix an attribute error on listings where some items have author_ids and some don't
* Bug when forcing recalculation of memoized functions
* the subreddit creation and edit form aren't dealing with errors properly
* buttons fix
===
- add confidence sorting to comments
* common values are precomputed for speedier response
* best is made the default sort on comment pages
- messages will now be delivered once one is moderator/contributor/banned
- UI updates to messaging page, including added show parent functionality to messages
- Remove the rate-limit on comments on your own self-posts
- Give users some leeway in editing their comments: don't show an edit star if the edit is within the first few minutes of a comment's lifetime
- Office Assistant will help users when they write to admins
Backend
===
- Replace the postgres-based query_queue with an AMQP based one
* Set up amqp queues for async tasks such as search updates and the scrapers
* service monitor updates, adding queue-tracking support
- Allow find_recent_broken_things to specify both from_time and to_time
- add a ini file parameter to disallow db writes (to create read-only reddit instances for crawlers)
New features
===
- self-serve advertisement:
* complete overhaul of sponsored link code
* functions for talking with authorize.net
* added pay domain and https support
* added ability to share traffic from sponsored links
* auto-reject promotions that are too old and unpaid for
- awards
- allow widget to have its links to have a target (in case it is iframed)
- automatic_reddits:
* Don't show automatic_reddits in the horizontal topbar
- Listing numbers are always in order with no gaps
- add support for sprites for common (r2.lib.contrib.nymph)
Admin
===
- added a takedown page for dealing with DMCA requests properly
* status code 404 on takedown pages
* JSON returns same string as in the explanation text
* nofollow on markdown in explanation
* title and image optional
- Added /c/(comment_id) for admins
- updates to JS to rate-limit voting, commenting, and anything else that could be just as easily done by a script-kiddie to cheat.
- make ad frame dynamic and add tracking pixel
- add the ability to add a sponsored banner to the rightbox of a reddit
- add the ability to show custom css on cnamed and/or non-cnamed versions of a reddit
- allow us to ignore reports from report-spammers.
Bugfixes
===
- Fix sorting of duplicate links (patch by Chromakode)
- fix traffic bug on main traffic page when it is the first of the month.
- toolbar redirects to comments page on self posts rather than generating the frame
- half-assed unicode handling in menus giving us bugs again. Switched to the whole-ass approach
- added Thing._byID36
- Support /help/foo/bar
* start showing embedded videos inside of an <iframe>. This allows us to
embed videos from external sites without having to trust their DOM security
Note that the media_object property on new links is now a dictionary,
but old ones will still be strings
the listing for that reddit
2. Allow a reddit to have a custom CSS stylesheet that appears to
visitors
3. Allow a reddit to upload a custom reddit alien logo
