New features:

* Activate negative-result caching for HardCache chain * begin migration to pylibmc: * Add pylibmc to the list of required packages in preparation for the replacement of the memcached library * Start using pylibmc for the rendercaches * Tweak the computation of the normalized hot page to be a bit faster when the precomputer is available, by relying on the precomputer's internal permacached structure. * Default to a SelfEmptyingCache for scripts run from `paster run'. Note that processes that run forever are still responsible for resetting their local-caches, but this can now be done with g.reset_caches() * threaded messaging patch part 1: backend changes only. This will allow migrate.py to be run to move new onto inbox and will start tracking message trees for users. * Specify some queries to run at most once per day * Refactored safemarkdown() and added soup testing Additions: * Added _byID_rel() * Made error messages more verbose for: 1. byID lookups of too-big thing_ids 2. memcache failures 3. Solr Nones * Award._all_awards() now sorts by date * Trophy.by_{account,award}() now cache properly * new feedback page with helpful links * Try to reduce the length of the query-queue by not adding known-long queries at all, rather than adding them and skipping them * whitespace clean up * simplify the 'why did my CC get denied' email checking. * added missing translation strings and users now get PMs when they are added as translators Bugfixes: * Fix a bug in unsaving * BeautifulSoup stopped hosting 3.0.7a, but 3.0.8 still uses the good parser * Better search error handling * Properly reset the cache-chains (incl. the hardcache; d'oh!) per-request * Fix an attribute error on listings where some items have author_ids and some don't * Bug when forcing recalculation of memoized functions * the subreddit creation and edit form aren't dealing with errors properly * buttons fix
2026-01-23 05:48:11 -05:00 · 2010-05-03 16:53:52 -07:00
parent 5ef76b96c7
commit 2869eaf8b9
340 changed files with 6443 additions and 2171 deletions
--- a/r2/Makefile
+++ b/r2/Makefile
@@ -16,7 +16,7 @@
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
 # 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

@@ -94,4 +94,4 @@ md5:    	$(MD5S)
 rtl:		$(RTLCSS)

 clean:
-	rm $(JSTARGETS) $(CSSTARGETS) $(MD5S) $(INIS)
+	rm $(JSTARGETS) $(CSSTARGETS) $(MD5S) $(INIS)
--- a/r2/babel.cfg
+++ b/r2/babel.cfg
@@ -16,7 +16,7 @@
 ## The Original Developer is the Initial Developer.  The Initial Developer of
 ## the Original Code is CondeNet, Inc.
 ## 
-## All portions of the code written by CondeNet are Copyright (c) 2006-2009
+## All portions of the code written by CondeNet are Copyright (c) 2006-2010
 ## CondeNet, Inc. All Rights Reserved.
 ################################################################################

--- a/r2/example.ini
+++ b/r2/example.ini
@@ -117,8 +117,6 @@ allowed_css_linked_domains = my.domain.com, my.otherdomain.com
 css_killswitch = False
 max_sr_images = 20

-show_awards = False
-
 takedown_sr = _takedowns
 login_cookie = reddit_session
 domain = localhost
@@ -139,6 +137,8 @@ profanity_wordlist =
 solr_url =  
 solr_cache_time = 300

+markdown_backend = py
+
 SECRET    = abcdefghijklmnopqrstuvwxyz0123456789
 MODSECRET = abcdefghijklmnopqrstuvwxyz0123456789
 tracking_secret = abcdefghijklmnopqrstuvwxyz0123456789
--- a/r2/ez_setup/init.py
+++ b/r2/ez_setup/init.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 #!python
--- a/r2/r2/init.py
+++ b/r2/r2/init.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 """r2
--- a/r2/r2/commands.py
+++ b/r2/r2/commands.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import paste.deploy.config
@@ -34,7 +34,7 @@ from pylons.wsgiapp import PylonsApp
 import os, sys
 #
 # commands that will be available by running paste with this app
-# 
+#

 class RunCommand(command.Command):
    max_args = 2
@@ -55,6 +55,7 @@ class RunCommand(command.Command):
        here_dir = os.getcwd()

        conf = appconfig(config_name, relative_to=here_dir)
+        conf.global_conf['running_as_script'] = True
        conf.update(dict(app_conf=conf.local_conf,
                         global_conf=conf.global_conf))
        paste.deploy.config.CONFIG.push_thread_config(conf)
@@ -67,7 +68,7 @@ class RunCommand(command.Command):
        # Load the wsgi app first so that everything is initialized right
        wsgiapp = RegistryManager(PylonsApp())
        test_app = paste.fixture.TestApp(wsgiapp)
-                        
+
        # Query the test app to setup the environment
        tresponse = test_app.get('/_test_vars')
        request_id = int(tresponse.body)
--- a/r2/r2/config/init.py
+++ b/r2/r2/config/init.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.cache import sgm
--- a/r2/r2/config/admin_routes.py
+++ b/r2/r2/config/admin_routes.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 def add(mc):
--- a/r2/r2/config/environment.py
+++ b/r2/r2/config/environment.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import os
--- a/r2/r2/config/middleware.py
+++ b/r2/r2/config/middleware.py
@@ -16,7 +16,7 @@
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
 #
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 """Pylons middleware initialization"""
--- a/r2/r2/config/rewrites.py
+++ b/r2/r2/config/rewrites.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import re
--- a/r2/r2/config/routing.py
+++ b/r2/r2/config/routing.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 """
@@ -150,6 +150,8 @@ def make_map(global_conf={}, app_conf={}):
       requirements = dict(sort = 'top|controversial'))
    
    mc('/message/compose', controller='message', action='compose')
+    mc('/message/messages/:mid', controller='message', action='listing',
+       where = "messages")
    mc('/message/:where', controller='message', action='listing')
    
    mc('/:action', controller='front',
--- a/r2/r2/config/templates.py
+++ b/r2/r2/config/templates.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.manager import tp_manager
--- a/r2/r2/config/utils.py
+++ b/r2/r2/config/utils.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 iters = (list, tuple, set)
--- a/r2/r2/controllers/init.py
+++ b/r2/r2/controllers/init.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from listingcontroller import ListingController
--- a/r2/r2/controllers/admin.py
+++ b/r2/r2/controllers/admin.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.controllers.reddit_base import RedditController
--- a/r2/r2/controllers/api.py
+++ b/r2/r2/controllers/api.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController, set_user_cookie
@@ -31,7 +31,7 @@ from r2.models.subreddit import Default as DefaultSR

 from r2.lib.utils import get_title, sanitize_url, timeuntil, set_last_modified
 from r2.lib.utils import query_string, link_from_url, timefromnow
-from r2.lib.utils import timeago
+from r2.lib.utils import timeago, tup
 from r2.lib.pages import FriendList, ContributorList, ModList, \
    BannedList, BoringPage, FormPage, CssError, UploadedImage, \
    ClickGadget
@@ -202,7 +202,8 @@ class ApiController(RedditController):
            l._commit()
            l.set_url_cache()

-        queries.queue_vote(c.user, l, True, ip)
+        queries.queue_vote(c.user, l, True, ip,
+                           cheater = (errors.CHEATER, None) in c.errors)
        if save:
            r = l._save(c.user)
            queries.new_savehide(r)
@@ -258,22 +259,33 @@ class ApiController(RedditController):
    @validatedForm(VRatelimit(rate_ip = True, prefix = 'login_',
                              error = errors.WRONG_PASSWORD),
                   user = VLogin(['user', 'passwd']),
+                   username = VLength('user', max_length = 100),
                   dest   = VDestination(),
                   rem    = VBoolean('rem'),
                   reason = VReason('reason'))
-    def POST_login(self, form, jquery, user, dest, rem, reason):
+    def POST_login(self, form, jquery, user, username, dest, rem, reason):
        if reason and reason[0] == 'redirect':
            dest = reason[1]

        hc_key = "login_attempts-%s" % request.ip

-        recent_attempts = g.hardcache.get(hc_key, 0)
+        # TODO: You-know-what (not mentioning it, just in case
+        # we accidentally release code with this comment in it)

+        # Cache lifetime for login_attmempts
+        la_expire_time = 3600 * 8
+
+        recent_attempts = g.hardcache.add(hc_key, 0, time=la_expire_time)
+
+        fake_failure = False
        if recent_attempts >= 25:
-            raise NotImplementedError("need proper fail msg")
-        elif form.has_errors("passwd", errors.WRONG_PASSWORD):
+            g.log.error ("%s failed to login as %s (attempt #%d)"
+                         % (request.ip, username, recent_attempts))
+            fake_failure = True
+
+        if fake_failure or form.has_errors("passwd", errors.WRONG_PASSWORD):
            VRatelimit.ratelimit(rate_ip = True, prefix = 'login_', seconds=1)
-            g.hardcache.set(hc_key, recent_attempts + 1, 3600 * 8)
+            g.hardcache.incr(hc_key, time = la_expire_time)
        else:
            self._login(form, user, dest, rem)

@@ -635,7 +647,8 @@ class ApiController(RedditController):
            else:
                item, inbox_rel = Comment._new(c.user, link, parent_comment,
                                               comment, ip)
-                queries.queue_vote(c.user, item, True, ip)
+                queries.queue_vote(c.user, item, True, ip,
+                                   cheater = (errors.CHEATER, None) in c.errors)

                #update last modified
                set_last_modified(link, 'comments')
@@ -653,6 +666,7 @@ class ApiController(RedditController):

            # insert the new comment
            jquery.insert_things(item)
+            
            # remove any null listings that may be present
            jquery("#noresults").hide()

@@ -721,7 +735,7 @@ class ApiController(RedditController):
            if should_ratelimit:
                VRatelimit.ratelimit(rate_user=True, rate_ip = True,
                                     prefix = "rate_share_")
-            
+
    @noresponse(VUser(),
                VModhash(),
                vote_type = VVotehash(('vh', 'id')),
@@ -731,13 +745,21 @@ class ApiController(RedditController):
    def POST_vote(self, dir, thing, ip, vote_type):
        ip = request.ip
        user = c.user
+        store = True
+
        if not thing or thing._deleted:
            return

+        if vote_type == "rejected":
+            g.log.error("POST_vote: rejected vote (%s) from '%s' on %s (%s)"%
+                        (request.params.get('dir'), c.user.name,
+                         thing._fullname, request.ip))
+            store = False
+
        # TODO: temporary hack until we migrate the rest of the vote data
        if thing._date < datetime(2009, 4, 17, 0, 0, 0, 0, g.tz):
-            g.log.debug("POST_vote: ignoring old vote on %s" % thing._fullname)
-            return
+            g.log.error("POST_vote: ignoring old vote on %s" % thing._fullname)
+            store = False

        # in a lock to prevent duplicate votes from people
        # double-clicking the arrows
@@ -745,16 +767,18 @@ class ApiController(RedditController):
            dir = (True if dir > 0
                   else False if dir < 0
                   else None)
+
            organic = vote_type == 'organic'
-            queries.queue_vote(user, thing, dir, ip, organic)
+            queries.queue_vote(user, thing, dir, ip, organic, store = store,
+                               cheater = (errors.CHEATER, None) in c.errors)
+            if store:
+                #update relevant caches
+                if isinstance(thing, Link):
+                    set_last_modified(c.user, 'liked')
+                    set_last_modified(c.user, 'disliked')

-            #update relevant caches
-            if isinstance(thing, Link):
-                set_last_modified(c.user, 'liked')
-                set_last_modified(c.user, 'disliked')
-
-            # flag search indexer that something has changed
-            changed(thing)
+                # flag search indexer that something has changed
+                changed(thing)

    @validatedForm(VUser(),
                   VModhash(),
@@ -967,12 +991,13 @@ class ApiController(RedditController):
                   ip = ValidIP(),
                   ad_type = VOneOf('ad', ('default', 'basic', 'custom')),
                   ad_file = VLength('ad-location', max_length = 500),
+                   sponsor_text =VLength('sponsorship-text', max_length = 500),
                   sponsor_name =VLength('sponsorship-name', max_length = 500),
                   sponsor_url = VLength('sponsorship-url', max_length = 500),
                   css_on_cname = VBoolean("css_on_cname"),
                   )
    def POST_site_admin(self, form, jquery, name, ip, sr, ad_type, ad_file,
-                        sponsor_url, sponsor_name,  **kw):
+                        sponsor_text, sponsor_url, sponsor_name,  **kw):
        # the status button is outside the form -- have to reset by hand
        form.parent().set_html('.status', "")

@@ -1004,6 +1029,7 @@ class ApiController(RedditController):
        elif (form.has_errors(None, errors.INVALID_OPTION) or
              form.has_errors('description', errors.TOO_LONG)):
            pass
+
        #creating a new reddit
        elif not sr:
            #sending kw is ok because it was sanitized above
@@ -1032,6 +1058,7 @@ class ApiController(RedditController):
                if ad_type != "custom":
                    ad_file = Subreddit._defaults['ad_file']
                sr.ad_file = ad_file
+                sr.sponsorship_text = sponsor_text or ""
                sr.sponsorship_url = sponsor_url or None
                sr.sponsorship_name = sponsor_name or None

@@ -1053,7 +1080,10 @@ class ApiController(RedditController):
            changed(sr)
            form.parent().set_html('.status', _("saved"))

-        if redir:
+        # don't go any further until the form validates
+        if form.has_error():
+            return
+        elif redir:
            form.redirect(redir)
        else:
            jquery.refresh()
@@ -1107,6 +1137,32 @@ class ApiController(RedditController):
        if r:
            queries.new_savehide(r)

+    @noresponse(VUser(),
+                VModhash(),
+                thing = VByName('id', multiple = True))
+    def POST_collapse_message(self, thing):
+        if not thing:
+            return
+        for t in tup(thing):
+            if hasattr(t, "to_id") and c.user._id == t.to_id:
+                t.to_collapse = True
+            elif hasattr(t, "author_id") and c.user._id == t.author_id:
+                t.author_collapse = True
+            t._commit()
+
+    @noresponse(VUser(),
+                VModhash(),
+                thing = VByName('id', multiple = True))
+    def POST_uncollapse_message(self, thing):
+        if not thing:
+            return
+        for t in tup(thing):
+            if hasattr(t, "to_id") and c.user._id == t.to_id:
+                t.to_collapse = False
+            elif hasattr(t, "author_id") and c.user._id == t.author_id:
+                t.author_collapse = False
+            t._commit()
+
    @noresponse(VUser(),
                VModhash(),
                thing = VByName('id'))
@@ -1115,8 +1171,7 @@ class ApiController(RedditController):
            return
        if hasattr(thing, "to_id") and c.user._id != thing.to_id:
            return 
-        thing.new = True
-        thing._commit()
+        queries.set_unread(thing, True)

    @noresponse(VUser(),
                VModhash(),
@@ -1125,10 +1180,7 @@ class ApiController(RedditController):
        if not thing: return
        if hasattr(thing, "to_id") and c.user._id != thing.to_id:
            return 
-        thing.new = False
-        thing._commit()
-
-
+        queries.set_unread(thing, False)

    @noresponse(VUser(),
                VModhash(),
@@ -1148,6 +1200,24 @@ class ApiController(RedditController):
            queries.new_savehide(r)


+    @validatedForm(VUser(),
+                   parent = VByName('parent_id'))
+    def POST_moremessages(self, form, jquery, parent):
+        if not parent.can_view():
+            return self.abort(403,'forbidden')
+
+        builder = MessageBuilder(c.user, parent = parent, skip = False)
+        listing = Listing(builder).listing()
+        a = []
+        for item in listing.things:
+            a.append(item)
+            for x in item.child.things:
+                a.append(x)
+        for item in a:
+            if hasattr(item, "child"):
+                item.child = None
+        jquery.things(parent._fullname).parent().replace_things(a, False, True)
+
    @validatedForm(link = VByName('link_id'),
                   sort = VMenu('where', CommentSortMenu),
                   children = VCommentIDs('children'),
@@ -1221,7 +1291,8 @@ class ApiController(RedditController):
                    #vote up all of the links
                    for link in links:
                        queries.queue_vote(c.user, link,
-                                           action == 'like', request.ip)
+                                           action == 'like', request.ip,
+                                           cheater = (errors.CHEATER, None) in c.errors)
                elif action == 'save':
                    link = max(links, key = lambda x: x._score)
                    r = link._save(c.user)
@@ -1285,7 +1356,7 @@ class ApiController(RedditController):
        jquery("body").captcha(get_iden())

    @noresponse(VAdmin(),
-                tr = VTranslation("id"), 
+                tr = VTranslation("lang"), 
                user = nop('user'))
    def POST_deltranslator(self, tr, user):
        if tr:
@@ -1398,8 +1469,8 @@ class ApiController(RedditController):
        else:
            cup_expiration = None

-        t = Trophy._new(recipient, award, description=description,
-                        url=url, cup_expiration=cup_expiration)
+        t = Trophy._new(recipient, award, description=description, url=url,
+                        cup_info=dict(expiration=cup_expiration))

        form.set_html(".status", _('saved'))

--- a/r2/r2/controllers/awards.py
+++ b/r2/r2/controllers/awards.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from pylons import request, g
--- a/r2/r2/controllers/buttons.py
+++ b/r2/r2/controllers/buttons.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController
@@ -41,6 +41,7 @@ class ButtonsController(RedditController):

    def get_wrapped_link(self, url, link = None, wrapper = None):
        try:
+            links = []
            if link:
                links = [link]
            else:
@@ -48,7 +49,7 @@ class ButtonsController(RedditController):
                try:
                    links = tup(Link._by_url(url, sr))
                except NotFound:
-                    links = []
+                    pass

            if links:
                kw = {}
@@ -80,10 +81,9 @@ class ButtonsController(RedditController):
              vote = VBoolean('vote', default=True),
              newwindow = VBoolean('newwindow'),
              width = VInt('width', 0, 800),
-              link = VByName('id'))
-    def GET_button_content(self, url, title, css, vote, newwindow, width, link):
-            
-        
+              l = VByName('id'))
+    def GET_button_content(self, url, title, css, vote, newwindow, width, l):
+             
        # no buttons on domain listings
        if isinstance(c.site, DomainSR):
            c.site = Default
@@ -94,17 +94,20 @@ class ButtonsController(RedditController):
            css != 'http://www.wired.com/css/redditsocial.css'): 
            css = None 

-        if link:
-            url = link.url
-            title = link.title 
+        if l:
+            url = l.url
+            title = l.title 
+        kw = {}
+        if title:
+            kw = dict(title = title)
        wrapper = make_wrapper(Button if vote else ButtonNoBody,
                               url = url, 
                               target = "_new" if newwindow else "_parent",
-                               title = title, vote = vote, bgcolor = c.bgcolor,
+                               vote = vote, bgcolor = c.bgcolor,
                               width = width, css = css,
-                               button = self.buttontype())
+                               button = self.buttontype(), **kw)

-        l = self.get_wrapped_link(url, link, wrapper)
+        l = self.get_wrapped_link(url, l, wrapper)
        res = l.render()
        c.response.content = spaceCompress(res)
        return c.response
--- a/r2/r2/controllers/captcha.py
+++ b/r2/r2/controllers/captcha.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController
--- a/r2/r2/controllers/embed.py
+++ b/r2/r2/controllers/embed.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController, proxyurl
--- a/r2/r2/controllers/error.py
+++ b/r2/r2/controllers/error.py
@@ -6,20 +6,21 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import os.path
+from mako.filters import url_escape

 import paste.fileapp
 from pylons.middleware import error_document_template, media_path
@@ -117,7 +118,12 @@ class ErrorController(RedditController):
        if 'usable_error_content' in request.environ:
            return request.environ['usable_error_content']
        if c.site._spam and not c.user_is_admin:
-            message = (strings.banned_subreddit % dict(link = '/feedback'))
+            subject = ("the subreddit /r/%s has been incorrectly banned" %
+                       c.site.name)
+            message = (strings.banned_subreddit %
+                       dict(link = '/message/compose?to=%s&subject=%s' %
+                            (g.admin_message_acct,
+                             url_escape(subject))))

            res = pages.RedditError(_('this reddit has been banned'),
                                    unsafe(safemarkdown(message)))
@@ -125,6 +131,19 @@ class ErrorController(RedditController):
        else:
            return pages.Reddit404().render()

+    def send503(self):
+        c.response.status_code = 503
+        if 'retry_after' in request.environ:
+            c.response.headers['Retry-After'] = request.environ['retry_after']
+        else:
+            c.response.headers['Retry-After'] = 1
+
+        if 'usable_error_content' in request.environ:
+            return request.environ['usable_error_content']
+        else:
+            c.response.content = toofast
+            return c.response
+
    def GET_document(self):
        try:
            #no cookies on errors
@@ -145,10 +164,7 @@ class ErrorController(RedditController):
            elif code == '500':
                return redditbroke % rand_strings.sadmessages
            elif code == '503':
-                c.response.status_code = 503
-                c.response.headers['Retry-After'] = 1
-                c.response.content = toofast
-                return c.response
+                return self.send503()
            elif code == '304':
                if request.GET.has_key('x-sup-id'):
                    c.response.headers['x-sup-id'] = request.GET.get('x-sup-id')
--- a/r2/r2/controllers/errors.py
+++ b/r2/r2/controllers/errors.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.utils import Storage, tup
--- a/r2/r2/controllers/feedback.py
+++ b/r2/r2/controllers/feedback.py
@@ -6,23 +6,23 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController
 from pylons import c, request
 from pylons.i18n import _
-from r2.lib.pages import FormPage, Feedback, Captcha, PaneStack, SelfServeBlurb
+from r2.lib.pages import FormPage, Feedback, Captcha, PaneStack, SelfServeBlurb, FeedbackBlurb

 class FeedbackController(RedditController):

@@ -32,13 +32,12 @@ class FeedbackController(RedditController):
                        loginbox = False).render()

    def GET_feedback(self):
-        title = _("send reddit feedback")
        return FormPage('feedback',
-                        content = Feedback(title=title, action='feedback'),
+                        content = FeedbackBlurb(),
                        loginbox = False).render()

    def GET_i18n(self):
        title = _("help translate reddit into your language")
-        return FormPage('help translate',
+        return FormPage(_('help translate'),
                        content = Feedback(title=title, action='i18n'),
                        loginbox = False).render()
--- a/r2/r2/controllers/front.py
+++ b/r2/r2/controllers/front.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from validator import *
@@ -31,16 +31,18 @@ from r2.lib.menus import *
 from r2.lib.utils import to36, sanitize_url, check_cheating, title_to_url
 from r2.lib.utils import query_string, UrlParser, link_from_url, link_duplicates
 from r2.lib.template_helpers import get_domain
+from r2.lib.filters import unsafe
 from r2.lib.emailer import has_opted_out, Email
 from r2.lib.db.operators import desc
 from r2.lib.db import queries
 from r2.lib.strings import strings
 from r2.lib.solrsearch import RelatedSearchQuery, SubredditSearchQuery, LinkSearchQuery
+from r2.lib.contrib.pysolr import SolrError
 from r2.lib import jsontemplates
 from r2.lib import sup
 import r2.lib.db.thing as thing
 from listingcontroller import ListingController
-from pylons import c, request
+from pylons import c, request, request, Response

 import random as rand
 import re
@@ -228,7 +230,7 @@ class FrontController(RedditController):
                                 comment, context)
        listing = NestedListing(builder, num = num,
                                parent_name = article._fullname)
-        
+
        displayPane = PaneStack()

        # if permalink page, add that message first to the content
@@ -238,11 +240,12 @@ class FrontController(RedditController):
        # insert reply box only for logged in user
        if c.user_is_loggedin and can_comment_link(article) and not is_api():
            #no comment box for permalinks
+            display = not bool(comment)
            displayPane.append(UserText(item = article, creating = True,
                                        post_form = 'comment',
-                                        display = not bool(comment),
+                                        display = display,
                                        cloneable = True))
-            
+
        # finally add the comment listing
        displayPane.append(listing.listing())

@@ -469,12 +472,13 @@ class FrontController(RedditController):
                                      SearchSortMenu(default=sort)],
                         search_params = dict(sort = sort, t = time),
                         infotext = infotext).render()
-        
+
        return res
-        
+
    def _search(self, query_obj, num, after, reverse, count=0):
        """Helper function for interfacing with search.  Basically a
        thin wrapper for SearchBuilder."""
+
        builder = SearchBuilder(query_obj,
                                after = after, num = num, reverse = reverse,
                                count = count,
@@ -484,7 +488,31 @@ class FrontController(RedditController):

        # have to do it in two steps since total_num and timing are only
        # computed after fetch_more
-        res = listing.listing()
+        try:
+            res = listing.listing()
+        except SolrError, e:
+            errmsg = "SolrError: %r %r" % (e, query_obj)
+
+            if (str(e) == 'None'):
+                # Production error logs only get non-None errors
+                g.log.debug(errmsg)
+            else:
+                g.log.error(errmsg)
+
+            sf = SearchFail()
+            sb = SearchBar(prev_search = query_obj.q)
+
+            us = unsafe(sb.render() + sf.render())
+
+            errpage = pages.RedditError(_('search failed'), us)
+
+            c.response = Response()
+            c.response.status_code = 503
+            request.environ['usable_error_content'] = errpage.render()
+            request.environ['retry_after'] = 60
+
+            abort(503)
+
        timing = time_module.time() - builder.start_time

        return builder.total_num, timing, res
--- a/r2/r2/controllers/i18n.py
+++ b/r2/r2/controllers/i18n.py
@@ -6,22 +6,23 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from pylons import request, g
 from reddit_base import RedditController
-from r2.lib.pages import UnfoundPage, AdminTranslations, AdminPage
+from r2.lib.pages import UnfoundPage, AdminTranslations, \
+    AdminPage, Translator_Message

 from r2.lib.translation import Translator, TranslatorTemplate, get_translator
 from validator import *
@@ -49,10 +50,18 @@ class I18nController(RedditController):
              lang = nop('lang'),
              a = VExistingUname('name'))
    def POST_adduser(self, lang, a):
+        from r2.lib.db import queries
        if a and Translator.exists(lang):
            tr = get_translator(locale = lang)
            tr.author.add(a.name)
            tr.save()
+
+            # send the user a message
+            body = Translator_Message(lang, a).render("html")
+            subject = "Thanks for offering to help translate!"
+            m, inbox_rel = Message._new(c.user, a, subject, body, request.ip)
+            queries.new_message(m, inbox_rel)
+
        return self.redirect("/admin/i18n")


@@ -60,7 +69,8 @@ class I18nController(RedditController):
              VAdmin())
    def GET_list(self):
        res = AdminPage(content = AdminTranslations(),
-                        title = 'translate reddit').render()
+                        title = 'translate reddit',
+                        show_sidebar = False).render()
        return res


@@ -85,7 +95,8 @@ class I18nController(RedditController):
        else:
            content = UnfoundPage()
        res = AdminPage(content = content, 
-                        title = 'translate reddit').render()
+                        title = 'translate reddit',
+                        show_sidebar = False).render()
        return res

    @validate(VTranslationEnabled(),
--- a/r2/r2/controllers/listingcontroller.py
+++ b/r2/r2/controllers/listingcontroller.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController, base_listing
@@ -39,10 +39,12 @@ from r2.lib.solrsearch import SearchQuery
 from r2.lib.utils import iters, check_cheating, timeago
 from r2.lib import sup
 from r2.lib.promote import PromoteSR
+from r2.lib.contrib.pysolr import SolrError

 from admin import admin_profile_query

 from pylons.i18n import _
+from pylons import Response

 import random

@@ -147,7 +149,29 @@ class ListingController(RedditController):
        if c.site.path == PromoteSR.path and not c.user_is_sponsor:
            abort(403, 'forbidden')
        listing = LinkListing(self.builder_obj, show_nums = self.show_nums)
-        return listing.listing()
+        try:
+            return listing.listing()
+        except SolrError, e:
+            errmsg = "SolrError: %r %r" % (e, self.builder_obj)
+
+            if (str(e) == 'None'):
+                # Production error logs only get non-None errors
+                g.log.debug(errmsg)
+            else:
+                g.log.error(errmsg)
+
+            sf = SearchFail()
+
+            us = unsafe(sf.render())
+
+            errpage = pages.RedditError(_('search failed'), us)
+
+            c.response = Response()
+            c.response.status_code = 503
+            request.environ['usable_error_content'] = errpage.render()
+            request.environ['retry_after'] = 60
+
+            abort(503)

    def title(self):
        """Page <title>"""
@@ -229,7 +253,7 @@ class HotController(FixListing, ListingController):
              and not isinstance(c.site, FakeSubreddit)
              and self.after is None
              and self.count == 0):
-            return [l._fullname for l in get_hot(c.site)]
+            return get_hot(c.site, only_fullnames = True)
        else:
            return c.site.get_links('hot', 'all')

@@ -483,8 +507,10 @@ class MessageController(ListingController):

    @property
    def menus(self):
-        if self.where in ('inbox', 'messages', 'comments', 'selfreply'):
+        if self.where in ('inbox', 'messages', 'comments',
+                          'selfreply', 'unread'):
            buttons = (NavButton(_("all"), "inbox"),
+                       NavButton(_("unread"), "unread"),
                       NavButton(plurals.messages, "messages"),
                       NavButton(_("comment replies"), 'comments'),
                       NavButton(_("post replies"), 'selfreply'))
@@ -509,14 +535,36 @@ class MessageController(ListingController):
            w.permalink, w._fullname = p, f
        else:
            w = ListingController.builder_wrapper(thing)
-            
-        if c.user.pref_mark_messages_read and thing.new:
-            w.new = True
-            thing.new = False
-            thing._commit()

        return w

+    def builder(self):
+        if self.where == 'messages':
+            if self.message:
+                if self.message.first_message:
+                    parent = Message._byID(self.message.first_message)
+                else:
+                    parent = self.message
+                return MessageBuilder(c.user, parent = parent,
+                                      skip = False, 
+                                      focal = self.message, 
+                                      wrap = self.builder_wrapper,
+                                      num = self.num)
+            elif c.user.pref_threaded_messages:
+                skip = (c.render_style == "html")
+                return MessageBuilder(c.user, wrap = self.builder_wrapper,
+                                      skip = skip, 
+                                      num = self.num,
+                                      after = self.after,
+                                      reverse = self.reverse)
+        return ListingController.builder(self)
+
+    def listing(self):
+        if (self.where == 'messages' and 
+            (c.user.pref_threaded_messages or self.message)):
+            return Listing(self.builder_obj).listing()
+        return ListingController.listing(self)
+
    def query(self):
        if self.where == 'messages':
            q = queries.get_inbox_messages(c.user)
@@ -524,24 +572,28 @@ class MessageController(ListingController):
            q = queries.get_inbox_comments(c.user)
        elif self.where == 'selfreply':
            q = queries.get_inbox_selfreply(c.user)
-        if self.where == 'inbox':
+        elif self.where == 'inbox':
            q = queries.get_inbox(c.user)
+        elif self.where == 'unread':
+            q = queries.get_unread_inbox(c.user)
+        elif self.where == 'sent':
+            q = queries.get_sent(c.user)

+        if self.where != 'sent':
            #reset the inbox
            if c.have_messages and self.mark != 'false':
                c.user.msgtime = False
                c.user._commit()

-        elif self.where == 'sent':
-            q = queries.get_sent(c.user)
-
        return q

    @validate(VUser(),
+              message = VMessageID('mid'),
              mark = VOneOf('mark',('true','false'), default = 'true'))
-    def GET_listing(self, where, mark, **env):
+    def GET_listing(self, where, mark, message, **env):
        self.where = where
        self.mark = mark
+        self.message = message
        c.msg_location = where
        return ListingController.GET_listing(self, **env)

@@ -627,7 +679,7 @@ class MyredditsController(ListingController):
        return stack

    @validate(VUser())
-    def GET_listing(self, where, **env):
+    def GET_listing(self, where = 'inbox', **env):
        self.where = where
        return ListingController.GET_listing(self, **env)

@@ -636,3 +688,4 @@ class CommentsController(ListingController):

    def query(self):
        return queries.get_all_comments()
+
--- a/r2/r2/controllers/mediaembed.py
+++ b/r2/r2/controllers/mediaembed.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from validator import *
--- a/r2/r2/controllers/post.py
+++ b/r2/r2/controllers/post.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.pages import *
@@ -108,6 +108,8 @@ class PostController(ApiController):
              pref_label_nsfw = VBoolean('label_nsfw'),
              pref_show_promote = VBoolean('show_promote'),
              pref_mark_messages_read = VBoolean("mark_messages_read"),
+              pref_threaded_messages = VBoolean("threaded_messages"),
+              pref_collapse_read_messages = VBoolean("collapse_read_messages"),
              all_langs = nop('all-langs', default = 'all'))
    def POST_options(self, all_langs, pref_lang, **kw):
        #temporary. eventually we'll change pref_clickgadget to an
--- a/r2/r2/controllers/promotecontroller.py
+++ b/r2/r2/controllers/promotecontroller.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from validator import *
--- a/r2/r2/controllers/reddit_base.py
+++ b/r2/r2/controllers/reddit_base.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import r2.lib.helpers as h
@@ -463,7 +463,7 @@ class RedditController(BaseController):
    def pre(self):
        c.start_time = datetime.now(g.tz)

-        g.cache.caches = (LocalCache(),) + g.cache.caches[1:]
+        g.reset_caches()

        c.domain_prefix = request.environ.get("reddit-domain-prefix", 
                                              g.domain_prefix)
--- a/r2/r2/controllers/redirect.py
+++ b/r2/r2/controllers/redirect.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from pylons.controllers.util import redirect_to
--- a/r2/r2/controllers/template.py
+++ b/r2/r2/controllers/template.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.base import *
--- a/r2/r2/controllers/toolbar.py
+++ b/r2/r2/controllers/toolbar.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from reddit_base import RedditController
--- a/r2/r2/controllers/validator/init.py
+++ b/r2/r2/controllers/validator/init.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from validator import *
--- a/r2/r2/controllers/validator/validator.py
+++ b/r2/r2/controllers/validator/validator.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from pylons import c, request, g
@@ -308,6 +308,18 @@ class VCommentID(Validator):
            except (NotFound, ValueError):
                pass

+class VMessageID(Validator):
+    def run(self, cid):
+        if cid:
+            try:
+                cid = int(cid, 36)
+                m = Message._byID(cid, True)
+                if not m.can_view():
+                    abort(403, 'forbidden')
+                return m
+            except (NotFound, ValueError):
+                pass
+
 class VCount(Validator):
    def run(self, count):
        if count is None:
--- a/r2/r2/i18n/init.py
+++ b/r2/r2/i18n/init.py
@@ -6,16 +6,16 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
--- a/r2/r2/i18n/r2.pot
+++ b/r2/r2/i18n/r2.pot
--- a/r2/r2/lib/init.py
+++ b/r2/r2/lib/init.py
@@ -6,16 +6,16 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
--- a/r2/r2/lib/amqp.py
+++ b/r2/r2/lib/amqp.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

@@ -174,9 +174,9 @@ def handle_items(queue, callback, ack = True, limit = 1, drain = False):
            time.sleep(1)
            continue

+        g.reset_caches()
+
        items = []
-        #reset the local cache
-        g.cache.caches = (LocalCache(),) + g.cache.caches[1:]

        while msg:
            items.append(msg)
--- a/r2/r2/lib/app_globals.py
+++ b/r2/r2/lib/app_globals.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from __future__ import with_statement
@@ -24,6 +24,7 @@ from pylons import config
 import pytz, os, logging, sys, socket, re, subprocess
 from datetime import timedelta, datetime
 from r2.lib.cache import LocalCache, Memcache, HardCache, CacheChain
+from r2.lib.cache import SelfEmptyingCache
 from r2.lib.db.stats import QueryStats
 from r2.lib.translation import get_active_langs
 from r2.lib.lock import make_lock_factory
@@ -63,7 +64,6 @@ class Globals(object):
                  'enable_doquery',
                  'use_query_cache',
                  'write_query_queue',
-                  'show_awards',
                  'css_killswitch',
                  'db_create_tables',
                  'disallow_db_writes',
@@ -77,6 +77,7 @@ class Globals(object):
                   'sponsors',
                   'monitored_servers',
                   'automatic_reddits',
+                   'skip_precompute_queries',
                   'agents',
                   'allowed_css_linked_domains']

@@ -119,12 +120,19 @@ class Globals(object):
                    v = tuple(self.to_iter(v))
                setattr(self, k, v)

-        # initialize caches
+        self.running_as_script = global_conf.get('running_as_script', False)
+
+        self.skip_precompute_queries = set(self.skip_precompute_queries)
+
+        # initialize caches. Any cache-chains built here must be added
+        # to reset_caches so that they can properly reset their local
+        # components
        mc = Memcache(self.memcaches, pickleProtocol = 1)
        self.memcache = mc
        self.cache = CacheChain((LocalCache(), mc))
        self.permacache = Memcache(self.permacaches, pickleProtocol = 1)
        self.rendercache = Memcache(self.rendercaches, pickleProtocol = 1)
+
        self.make_lock = make_lock_factory(mc)

        self.rec_cache = Memcache(self.rec_cache, pickleProtocol = 1)
@@ -140,7 +148,10 @@ class Globals(object):
        self.dbm = self.load_db_params(global_conf)

        # can't do this until load_db_params() has been called
-        self.hardcache = CacheChain((LocalCache(), mc, HardCache(self)))
+        self.hardcache = CacheChain((LocalCache(), mc, HardCache(self)),
+                                    cache_negative_results = True)
+
+        self.reset_caches()

        #make a query cache
        self.stats_collector = QueryStats()
@@ -190,8 +201,6 @@ class Globals(object):
        else:
            self.log.setLevel(logging.WARNING)

-        if self.log_start:
-            self.log.error("reddit app started on %s" % datetime.now())
        # set log level for pycountry which is chatty
        logging.getLogger('pycountry.db').setLevel(logging.CRITICAL)

@@ -241,6 +250,15 @@ class Globals(object):
            self.log.info("Couldn't read source revision (%r)" % e)
            self.version = self.short_version = '(unknown)'

+        if self.log_start:
+            self.log.error("reddit app started %s at %s" % (self.short_version, datetime.now()))
+
+    def reset_caches(self):
+        for ca in ('cache', 'hardcache'):
+            cache = getattr(self, ca)
+            new_cache = SelfEmptyingCache() if self.running_as_script else LocalCache()
+            cache.caches = (new_cache,) + cache.caches[1:]
+
    @staticmethod
    def to_bool(x):
        return (x.lower() == 'true') if x else None
--- a/r2/r2/lib/authorize/init.py
+++ b/r2/r2/lib/authorize/init.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from interaction import *
--- a/r2/r2/lib/authorize/api.py
+++ b/r2/r2/lib/authorize/api.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 """
--- a/r2/r2/lib/authorize/interaction.py
+++ b/r2/r2/lib/authorize/interaction.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from api import *
--- a/r2/r2/lib/base.py
+++ b/r2/r2/lib/base.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

--- a/r2/r2/lib/c/filters.c
+++ b/r2/r2/lib/c/filters.c
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/
 #include <Python.h>
@@ -179,7 +179,7 @@ filters_websafe(PyObject * self, PyObject *args)
      buffer[ib++] = 'u';
      buffer[ib++] = 'o';
      buffer[ib++] = 't';
-      buffer[ib]   = ';';      
+      buffer[ib]   = ';';
    }
    else {
      buffer[ib] = input_buffer[ic];
--- a/r2/r2/lib/c/recommendations/Database.cpp
+++ b/r2/r2/lib/c/recommendations/Database.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/Database.h
+++ b/r2/r2/lib/c/recommendations/Database.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/Dictionary.h
+++ b/r2/r2/lib/c/recommendations/Dictionary.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/ModsTable.cpp
+++ b/r2/r2/lib/c/recommendations/ModsTable.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/ModsTable.h
+++ b/r2/r2/lib/c/recommendations/ModsTable.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/Recommender.cpp
+++ b/r2/r2/lib/c/recommendations/Recommender.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/Recommender.h
+++ b/r2/r2/lib/c/recommendations/Recommender.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/SparseMatrix.h
+++ b/r2/r2/lib/c/recommendations/SparseMatrix.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/User.cpp
+++ b/r2/r2/lib/c/recommendations/User.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/User.h
+++ b/r2/r2/lib/c/recommendations/User.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/articles.cpp
+++ b/r2/r2/lib/c/recommendations/articles.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/articles.h
+++ b/r2/r2/lib/c/recommendations/articles.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/main.cpp
+++ b/r2/r2/lib/c/recommendations/main.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/recommend_memcache.cpp
+++ b/r2/r2/lib/c/recommendations/recommend_memcache.cpp
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/recommend_memcache.h
+++ b/r2/r2/lib/c/recommendations/recommend_memcache.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c/recommendations/recommender_py.h
+++ b/r2/r2/lib/c/recommendations/recommender_py.h
@@ -17,7 +17,7 @@
 * The Original Developer is the Initial Developer.  The Initial Developer of the
 * Original Code is CondeNet, Inc.
 * 
-* All portions of the code written by CondeNet are Copyright (c) 2006-2009
+* All portions of the code written by CondeNet are Copyright (c) 2006-2010
 * CondeNet, Inc. All Rights Reserved.
 *******************************************************************************/

--- a/r2/r2/lib/c_markdown.py
+++ b/r2/r2/lib/c_markdown.py
@@ -0,0 +1,2 @@
+def c_markdown(text, nofollow=False, target=None):
+    raise NotImplementedError()
--- a/r2/r2/lib/cache.py
+++ b/r2/r2/lib/cache.py
@@ -6,31 +6,33 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from threading import local

-from utils import lstrips
+from utils import lstrips, in_chunks
 from contrib import memcache

 from r2.lib.hardcachebackend import HardCacheBackend

+class NoneResult(object): pass
+
 class CacheUtils(object):
-    def incr_multi(self, keys, amt=1, prefix=''):
+    def incr_multi(self, keys, delta=1, time=0, prefix=''):
        for k in keys:
            try:
-                self.incr(prefix + k, amt)
+                self.incr(prefix + k, time=time, delta=delta)
            except ValueError:
                pass

@@ -92,11 +94,34 @@ class HardCache(CacheUtils):
        return category, ids

    def set(self, key, val, time=0):
+        if val is NoneResult:
+            # NoneResult caching is for other parts of the chain
+            return
+
        category, ids = self._split_key(key)
-        if time <= 0:
-            raise ValueError ("HardCache.set() *must* have an expiration time")
        self.backend.set(category, ids, val, time)

+    def simple_get_multi(self, keys):
+        results = {}
+        category_bundles = {}
+        for key in keys:
+            category, ids = self._split_key(key)
+            category_bundles.setdefault(category, []).append(ids)
+
+        for category in category_bundles:
+            idses = category_bundles[category]
+            chunks = in_chunks(idses, size=50)
+            for chunk in chunks:
+                new_results = self.backend.get_multi(category, chunk)
+                results.update(new_results)
+
+        return results
+
+    def set_multi(self, keys, prefix='', time=0):
+        for k,v in keys.iteritems():
+            if v is not NoneResult:
+                self.set(prefix+str(k), v, time=time)
+
    def get(self, key, default=None):
        category, ids = self._split_key(key)
        r = self.backend.get(category, ids)
@@ -104,9 +129,20 @@ class HardCache(CacheUtils):
        return r

    def delete(self, key, time=0):
+        # Potential optimization: When on a negative-result caching chain,
+        # shove NoneResult throughout the chain when a key is deleted.
        category, ids = self._split_key(key)
        self.backend.delete(category, ids)

+    def add(self, key, value, time=0):
+        category, ids = self._split_key(key)
+        return self.backend.add(category, ids, value, time=time)
+
+    def incr(self, key, delta=1, time=0):
+        category, ids = self._split_key(key)
+        return self.backend.incr(category, ids, delta=delta, time=time)
+
+
 class LocalCache(dict, CacheUtils):
    def __init__(self, *a, **kw):
        return dict.__init__(self, *a, **kw)
@@ -129,16 +165,17 @@ class LocalCache(dict, CacheUtils):
        return out

    def set(self, key, val, time = 0):
+        # time is ignored on localcache
        self._check_key(key)
        self[key] = val

    def set_multi(self, keys, prefix='', time=0):
        for k,v in keys.iteritems():
-            self.set(prefix+str(k), v)
+            self.set(prefix+str(k), v, time=time)

    def add(self, key, val, time = 0):
        self._check_key(key)
-        self.setdefault(key, val)
+        return self.setdefault(key, val)

    def delete(self, key):
        if self.has_key(key):
@@ -149,11 +186,11 @@ class LocalCache(dict, CacheUtils):
            if self.has_key(key):
                del self[key]

-    def incr(self, key, amt=1):
+    def incr(self, key, delta=1, time=0):
        if self.has_key(key):
-            self[key] = int(self[key]) + amt
+            self[key] = int(self[key]) + delta

-    def decr(self, key, amt=1): 
+    def decr(self, key, amt=1):
        if self.has_key(key):
            self[key] = int(self[key]) - amt

@@ -173,13 +210,15 @@ class LocalCache(dict, CacheUtils):
        self.clear()

 class CacheChain(CacheUtils, local):
-    def __init__(self, caches):
+    def __init__(self, caches, cache_negative_results=False):
        self.caches = caches
+        self.cache_negative_results = cache_negative_results

    def make_set_fn(fn_name):
        def fn(self, *a, **kw):
            for c in self.caches:
-                getattr(c, fn_name)(*a, **kw)
+                ret = getattr(c, fn_name)(*a, **kw)
+            return ret
        return fn

    set = make_set_fn('set')
@@ -193,20 +232,60 @@ class CacheChain(CacheUtils, local):
    delete = make_set_fn('delete')
    delete_multi = make_set_fn('delete_multi')
    flush_all = make_set_fn('flush_all')
+    cache_negative_results = False
+
+    def add(self, key, val, time=0):
+        authority = self.caches[-1]
+        added_val = authority.add(key, val, time=time)
+        for cache in self.caches[:-1]:
+            # Calling set() rather than add() to ensure that all caches are
+            # in sync and that de-syncs repair themselves
+            cache.set(key, added_val, time=time)
+        return added_val
+
+    def accrue(self, key, time=0, delta=1):
+        auth_value = self.caches[-1].get(key)
+
+        if auth_value is None:
+            self.caches[-1].set(key, 0, time)
+            auth_value = 0
+
+        try:
+            auth_value = int(auth_value)
+        except ValueError:
+            raise ValueError("Can't accrue %s; it's a %s (%r)" %
+                             (key, auth_value.__class__.__name__, auth_value))
+
+        for c in self.caches:
+            c.set(key, auth_value, time=time)
+
+        self.incr(key, time=time, delta=delta)

    def get(self, key, default = None, local = True):
        for c in self.caches:
            if not local and isinstance(c,LocalCache):
                continue
-            val = c.get(key, default)
+
+            val = c.get(key)
+
            if val is not None:
                #update other caches
                for d in self.caches:
                    if c == d:
-                        break;
+                        break # so we don't set caches later in the chain
                    d.set(key, val)
-                return val
+
+                if self.cache_negative_results and val is NoneResult:
+                    return None
+                else:
+                    return val
+
        #didn't find anything
+
+        if self.cache_negative_results:
+            for c in self.caches:
+                c.set(key, NoneResult)
+
        return default

    def simple_get_multi(self, keys):
@@ -220,13 +299,32 @@ class CacheChain(CacheUtils, local):
            if r:
                for d in self.caches:
                    if c == d:
-                        break;
+                        break # so we don't set caches later in the chain
                    d.set_multi(r)
                r.update(out)
                out = r
                need = need - set(r.keys())
+
+        if need and self.cache_negative_results:
+            d = dict( (key,NoneResult) for key in need)
+            for c in self.caches:
+                c.set_multi(d)
+
+        if self.cache_negative_results:
+            filtered_out = {}
+            for k,v in out.iteritems():
+                if v is not NoneResult:
+                    filtered_out[k] = v
+            out = filtered_out
+
        return out

+    def debug(self, key):
+        print "Looking up [%r]" % key
+        for i, c in enumerate(self.caches):
+            print "[%d] %10s has value [%r]" % (i, c.__class__.__name__,
+                                                c.get(key))
+
 #smart get multi
 def sgm(cache, keys, miss_fn, prefix='', time=0):
    keys = set(keys)
@@ -274,7 +372,7 @@ def test_cache(cache):
 # a cache that occasionally dumps itself to be used for long-running
 # processes
 class SelfEmptyingCache(LocalCache):
-    def __init__(self,max_size=100*1000):
+    def __init__(self, max_size=10*1000):
        self.max_size = max_size

    def maybe_reset(self):
@@ -284,5 +382,6 @@ class SelfEmptyingCache(LocalCache):
    def set(self,key,val,time = 0):
        self.maybe_reset()
        return LocalCache.set(self,key,val,time)
+
    def add(self,key,val):
        return self.set(key,val)
--- a/r2/r2/lib/captcha.py
+++ b/r2/r2/lib/captcha.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import random, string
--- a/r2/r2/lib/comment_tree.py
+++ b/r2/r2/lib/comment_tree.py
@@ -6,24 +6,24 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from __future__ import with_statement

 from pylons import g
-
-from r2.models import *
+from itertools import chain
+from utils import tup

 def comments_key(link_id):
    return 'comments_' + str(link_id)
@@ -98,6 +98,7 @@ def link_comments(link_id):
        return r

 def load_link_comments(link_id):
+    from r2.models import Comment
    q = Comment._query(Comment.c.link_id == link_id,
                       Comment.c._deleted == (True, False),
                       Comment.c._spam == (True, False),
@@ -135,3 +136,140 @@ def load_link_comments(link_id):
        num_children[cm_id] = num

    return cids, comment_tree, depth, num_children
+
+# message conversation functions
+def messages_key(user_id):
+    return 'message_conversations_' + str(user_id)
+
+def messages_lock_key(user_id):
+    return 'message_conversations_lock_' + str(user_id)
+
+def add_message(message):
+    # add the message to the author's list and the recipient
+    with g.make_lock(messages_lock_key(message.author_id)):
+        add_message_nolock(message.author_id, message)
+    with g.make_lock(messages_lock_key(message.to_id)):
+        add_message_nolock(message.to_id, message)
+
+def add_message_nolock(user_id, message):
+    from r2.models import Account, Message
+    key = messages_key(user_id)
+    trees = g.permacache.get(key)
+    if not trees:
+        # no point computing it now.  We'll do it when they go to
+        # their message page.
+        return
+
+    # if it is a new root message, easy enough
+    if message.first_message is None:
+        trees.insert(0, (message._id, []))
+    else:
+        tree_dict = dict(trees)
+
+        # if the tree already has the first message, update the list
+        if message.first_message in tree_dict:
+            if message._id not in tree_dict[message.first_message]:
+                tree_dict[message.first_message].append(message._id)
+                tree_dict[message.first_message].sort()
+        # we have to regenerate the conversation :/
+        else:
+            m = Message._query(Message.c.first_message == message.first_message,
+                               data = True)
+            new_tree = compute_message_trees(m)
+            if new_tree:
+                trees.append(new_tree[0])
+        trees.sort(key = tree_sort_fn, reverse = True)
+
+    # done!
+    g.permacache.set(key, trees)
+
+
+def conversation(user, parent):
+    from r2.models import Message
+    trees = dict(user_messages(user))
+
+    if parent._id in trees:
+        convo = trees[parent._id]
+        if convo:
+            m = Message._byID(convo[0], data = True)
+        if not convo or m.first_message == m.parent_id:
+            return [(parent._id, convo)]
+
+    # if we get to this point, either we didn't find the conversation,
+    # or the first child of the result was not the actual first child.
+    # To the database!
+    m = Message._query(Message.c.first_message == parent._id,
+                       data = True)
+    return compute_message_trees([parent] + list(m))
+
+def user_messages(user):
+    key = messages_key(user._id)
+    trees = g.permacache.get(key)
+    if trees is None:
+        trees = user_messages_nocache(user)
+        g.permacache.set(key, trees)
+    return trees
+
+def user_messages_nocache(user):
+    """
+    Just like user_messages, but avoiding the cache
+    """
+    from r2.lib.db import queries
+    from r2.models import Message
+
+    inbox = queries.get_inbox_messages(user)
+    if hasattr(inbox, 'prewrap_fn'):
+        inbox = [inbox.prewrap_fn(i) for i in inbox]
+    else:
+        inbox = list(inbox)
+
+    sent = queries.get_sent(user)
+    if hasattr(sent, 'prewrap_fn'):
+        sent = [sent.prewrap_fn(i) for i in sent]
+    else:
+        sent = list(sent)
+    
+    m = {}
+    ids = [x for x in chain(inbox, sent) if not isinstance(x, Message)]
+    if ids:
+        m = Message._by_fullname(ids, return_dict = True, data = True)
+
+    messages = [m.get(x, x) for x in chain(inbox, sent)]
+
+    return compute_message_trees(messages)
+
+def compute_message_trees(messages):
+    from r2.models import Message
+    roots = set()
+    threads = {}
+    mdict = {}
+    messages = sorted(messages, key = lambda m: m._date, reverse = True)
+
+    for m in messages:
+        if not m._loaded:
+            m._load()
+        mdict[m._id] = m
+        if m.first_message:
+            roots.add(m.first_message)
+            threads.setdefault(m.first_message, set()).add(m._id)
+        else:
+            roots.add(m._id)
+
+    # load any top-level messages which are not in the original list
+    missing = [m for m in roots if m not in mdict]
+    if missing:
+        mdict.update(Message._byID(tup(missing),
+                                   return_dict = True, data = True))
+
+    # sort threads in chrono order
+    for k in threads:
+        threads[k] = list(sorted(threads[k]))
+
+    tree = [(root, threads.get(root, [])) for root in roots]
+    tree.sort(key = tree_sort_fn, reverse = True)
+
+    return tree
+
+def tree_sort_fn(tree):
+    root, threads = tree
+    return threads[-1] if threads else root
--- a/r2/r2/lib/contrib/gprof2dot.py
+++ b/r2/r2/lib/contrib/gprof2dot.py
--- a/r2/r2/lib/contrib/markdown.py
+++ b/r2/r2/lib/contrib/markdown.py
@@ -129,7 +129,7 @@ class _Markdown:
    [ ]{0,%d}
    <(hr)        # start tag = $2
    \b           # word break
-    ([^<>])*?    # 
+    ([^<>])*?    #
    /?>          # the matching end tag
    [ \t]*
    (?=\n{2,}|\Z)# followed by a blank line or end of document
@@ -264,6 +264,7 @@ class _Markdown:
                url = self.urls[link_id]
                url = url.replace("*", self.escapetable["*"])
                url = url.replace("_", self.escapetable["_"])
+                url = url.replace("[", self.escapetable["["])
                res = '<a href="%s"' % htmlquote(url)

                if title:
@@ -273,6 +274,7 @@ class _Markdown:
                res += ">%s</a>" % htmlquote(link_text)
            else:
                res = whole_match
+
            return res

        def handler2(m):
@@ -283,6 +285,7 @@ class _Markdown:

            url = url.replace("*", self.escapetable["*"])
            url = url.replace("_", self.escapetable["_"])
+            url = url.replace("[", self.escapetable["["])
            res = '''<a href="%s"''' % htmlquote(url)
            
            if title:
--- a/r2/r2/lib/contrib/memcache.py
+++ b/r2/r2/lib/contrib/memcache.py
@@ -256,6 +256,8 @@ class Client(local):
        #        return server, key
        #    serverhash = serverHashFunction(str(serverhash) + str(i))

+        print ("Couldn't connect to any of the %d memcache servers" %
+               len(self.buckets))
        return None, key

    def disconnect_all(self):
@@ -355,7 +357,7 @@ class Client(local):
            return 0
        return 1

-    def incr(self, key, delta=1):
+    def incr(self, key, delta=1, time=0):
        """
        Sends a command to the server to atomically increment the value for C{key} by
        C{delta}, or by 1 if C{delta} is unspecified.  Returns None if C{key} doesn't
@@ -379,8 +381,10 @@ class Client(local):
        @rtype: int
        """
        return self._incrdecr("incr", key, delta)
+        # Note: cachechain throws away this return value, so it's almost
+        # pointless to return anything

-    def decr(self, key, delta=1):
+    def decr(self, key, delta=1, time=0):
        """
        Like L{incr}, but decrements.  Unlike L{incr}, underflow is checked and
        new values are capped at 0.  If server value is 1, a decrement of 2
--- a/r2/r2/lib/contrib/nymph.py
+++ b/r2/r2/lib/contrib/nymph.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import re, sys, Image, os, hashlib, StringIO
--- a/r2/r2/lib/count.py
+++ b/r2/r2/lib/count.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

--- a/r2/r2/lib/cssfilter.py
+++ b/r2/r2/lib/cssfilter.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from __future__ import with_statement
@@ -329,11 +329,14 @@ def find_preview_links(sr):
    from r2.lib.normalized_hot import get_hot

    # try to find a link to use, otherwise give up and return
-    links = get_hot(c.site)
+    links = get_hot(c.site, only_fullnames = True)
    if not links:
        sr = Subreddit._by_name(g.default_sr)
        if sr:
-            links = get_hot(sr)
+            links = get_hot(sr, only_fullnames = True)
+
+    if links:
+        links = Link._by_fullname(links, data=True, return_dict=False)

    return links

--- a/r2/r2/lib/db/init.py
+++ b/r2/r2/lib/db/init.py
@@ -6,16 +6,16 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
--- a/r2/r2/lib/db/alter_db.py
+++ b/r2/r2/lib/db/alter_db.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import tdb_sql
--- a/r2/r2/lib/db/operators.py
+++ b/r2/r2/lib/db/operators.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 class BooleanOp(object):
--- a/r2/r2/lib/db/queries.py
+++ b/r2/r2/lib/db/queries.py
@@ -124,15 +124,13 @@ class CachedResults(object):
    def delete(self, items):
        """Deletes an item from the cached data."""
        self.fetch()
-        did_change = False
+        fnames = set(self.filter(x)._fullname for x in tup(items))

-        for item in tup(items):
-            t = self.make_item_tuple(item)
-            while t in self.data:
-                self.data.remove(t)
-                did_change = True
-            
-        if did_change:
+        data = filter(lambda x: x[0] not in fnames,
+                      self.data)
+
+        if data != self.data:
+            self.data = data
            query_cache.set(self.iden, self.data)
        
    def update(self):
@@ -270,8 +268,8 @@ def get_submitted(user, sort, time):
 def get_overview(user, sort, time):
    return merge_results(get_comments(user, sort, time),
                         get_submitted(user, sort, time))
-    
-def user_rel_query(rel, user, name):
+
+def user_rel_query(rel, user, name, filters = []):
    """General user relationship query."""
    q = rel._query(rel.c._thing1_id == user._id,
                   rel.c._t2_deleted == False,
@@ -280,6 +278,8 @@ def user_rel_query(rel, user, name):
                   eager_load = True,
                   thing_data = not g.use_query_cache
                   )
+    if filters:
+        q._filter(*filters)

    return make_results(q, filter_thing2)

@@ -301,13 +301,25 @@ inbox_message_rel = Inbox.rel(Account, Message)
 def get_inbox_messages(user):
    return user_rel_query(inbox_message_rel, user, 'inbox')

+def get_unread_messages(user):
+    return user_rel_query(inbox_message_rel, user, 'inbox', 
+                          filters = [inbox_message_rel.c.new == True])
+
 inbox_comment_rel = Inbox.rel(Account, Comment)
 def get_inbox_comments(user):
    return user_rel_query(inbox_comment_rel, user, 'inbox')

+def get_unread_comments(user):
+    return user_rel_query(inbox_comment_rel, user, 'inbox', 
+                          filters = [inbox_comment_rel.c.new == True])
+
 def get_inbox_selfreply(user):
    return user_rel_query(inbox_comment_rel, user, 'selfreply')

+def get_unread_selfreply(user):
+    return user_rel_query(inbox_comment_rel, user, 'selfreply', 
+                          filters = [inbox_comment_rel.c.new == True])
+
 def get_inbox(user):
    return merge_results(get_inbox_comments(user),
                         get_inbox_messages(user),
@@ -319,6 +331,11 @@ def get_sent(user):
                       sort = desc('_date'))
    return make_results(q)

+def get_unread_inbox(user):
+    return merge_results(get_unread_comments(user),
+                         get_unread_messages(user),
+                         get_unread_selfreply(user))
+
 def add_queries(queries, insert_items = None, delete_items = None):
    """Adds multiple queries to the query queue. If insert_items or
    delete_items is specified, the query may not need to be recomputed at
@@ -420,6 +437,8 @@ def new_comment(comment, inbox_rel):
        else:
            add_queries([get_inbox_selfreply(inbox_owner)],
                        insert_items = inbox_rel)
+        set_unread(comment, True)
+


 def new_subreddit(sr):
@@ -451,14 +470,29 @@ def new_vote(vote):
    else:
        add_queries([get_liked(user)], delete_items = vote)
        add_queries([get_disliked(user)], delete_items = vote)
-    
+
 def new_message(message, inbox_rel):
+    from r2.lib.comment_tree import add_message
+
    from_user = Account._byID(message.author_id)
    to_user = Account._byID(message.to_id)

    add_queries([get_sent(from_user)], insert_items = message)
    add_queries([get_inbox_messages(to_user)], insert_items = inbox_rel)

+    add_message(message)
+    set_unread(message, True)
+
+def set_unread(message, unread):
+    for i in Inbox.set_unread(message, unread):
+        kw = dict(insert_items = i) if unread else dict(delete_items = i)
+        if i._name == 'selfreply':
+            add_queries([get_unread_selfreply(i._thing1)], **kw)
+        elif isinstance(message, Comment):
+            add_queries([get_unread_comments(i._thing1)], **kw)
+        else:
+            add_queries([get_unread_messages(i._thing1)], **kw)
+
 def new_savehide(rel):
    user = rel._thing1
    name = rel._name
@@ -653,14 +687,19 @@ def run_new_comments():
 #    amqp.handle_items('newpage_q', _run_new_links, limit=100)


-def queue_vote(user, thing, dir, ip, organic = False):
-    if g.amqp_host:
-        key = "registered_vote_%s_%s" % (user._id, thing._fullname)
-        g.cache.set(key, '1' if dir is True else '0' if dir is None else '-1')
-        amqp.add_item('register_vote_q',
-                      pickle.dumps((user._id, thing._fullname, dir, ip, organic)))
-    else:
-        handle_vote(user, thing, dir, ip, organic)
+def queue_vote(user, thing, dir, ip, organic = False,
+               cheater = False, store = True):
+    # set the vote in memcached so the UI gets updated immediately
+    key = "registered_vote_%s_%s" % (user._id, thing._fullname)
+    g.cache.set(key, '1' if dir is True else '0' if dir is None else '-1')
+    # queue the vote to be stored unless told not to
+    if store:
+        if g.amqp_host:
+            amqp.add_item('register_vote_q',
+                          pickle.dumps((user._id, thing._fullname,
+                                        dir, ip, organic, cheater)))
+        else:
+            handle_vote(user, thing, dir, ip, organic)

 def get_likes(user, items):
    if not user or not items:
@@ -688,15 +727,23 @@ def get_likes(user, items):

    return res

-def handle_vote(user, thing, dir, ip, organic):
+def handle_vote(user, thing, dir, ip, organic, cheater = False):
    from r2.lib.db import tdb_sql
    from sqlalchemy.exc import IntegrityError
    try:
-        v = Vote.vote(user, thing, dir, ip, organic)
+        v = Vote.vote(user, thing, dir, ip, organic, cheater = cheater)
    except (tdb_sql.CreationError, IntegrityError):
        g.log.error("duplicate vote for: %s" % str((user, thing, dir)))
        return

+    # keep track of upvotes in the hard cache by subreddit
+    sr_id = getattr(thing, "sr_id", None)
+    if (sr_id and dir > 0 and getattr(thing, "author_id", None) != user._id
+        and v.valid_thing):
+        now = datetime.now(g.tz).strftime("%Y/%m/%d")
+        g.hardcache.add("subreddit_vote-%s_%s_%s" % (now, sr_id, user._id),
+                        sr_id, time = 86400 * 7) # 1 week for now
+
    if isinstance(thing, Link):
        new_vote(v)
        if v.valid_thing:
@@ -732,17 +779,21 @@ def process_votes(drain = False, limit = 100):
        uids = set()
        tids = set()
        for x in msgs:
-            uid, tid, dir, ip, organic = pickle.loads(x.body)
-            print (uid, tid, dir, ip, organic)
+            r = pickle.loads(x.body)
+            uid, tid, dir, ip, organic, cheater = r
+
+            print (uid, tid, dir, ip, organic, cheater)
+
            uids.add(uid)
            tids.add(tid)
-            to_do.append((uid, tid, dir, ip, organic))
+            to_do.append((uid, tid, dir, ip, organic, cheater))

        users = Account._byID(uids, data = True, return_dict = True)
        things = Thing._by_fullname(tids, data = True, return_dict = True)

-        for uid, tid, dir, ip, organic in to_do:
-            handle_vote(users[uid], things[tid], dir, ip, organic)
+        for uid, tid, dir, ip, organic, cheater in to_do:
+            handle_vote(users[uid], things[tid], dir, ip, organic,
+                        cheater = cheater)

    amqp.handle_items('register_vote_q', _handle_votes, limit = limit,
                      drain = drain)
--- a/r2/r2/lib/db/query_queue.py
+++ b/r2/r2/lib/db/query_queue.py
@@ -12,6 +12,9 @@ TIMEOUT = 600
 def add_query(cached_results):
    amqp.add_item('prec_links', pickle.dumps(cached_results, -1))

+def _skip_key(iden):
+    return 'skip_precompute_queries-%s' % iden
+
 def run():
    def callback(msgs, chan):
        for msg in msgs: # will be len==1
@@ -19,6 +22,11 @@ def run():
            cr = pickle.loads(msg.body)
            iden = cr.query._iden()

+            if (iden in g.skip_precompute_queries
+                and g.hardcache.get(_skip_key(iden))):
+                print 'skipping known query', iden
+                continue
+
            working_key = working_prefix + iden
            key = prefix + iden

@@ -34,13 +42,17 @@ def run():
                print 'skipping, someone else is working', working_key
                return

-            cr = pickle.loads(msg.body)
-
            print 'working: ', iden, cr.query._rules, cr.query._sort
            start = datetime.now()
            try:
                cr.update()
                g.memcache.set(key, datetime.now())
+
+                if iden in g.skip_precompute_queries:
+                    print 'setting to be skipped for 6 hours', iden
+                    g.hardcache.set(_skip_key(iden), start,
+                                    60*60*6)
+
            finally:
                g.memcache.delete(working_key)

--- a/r2/r2/lib/db/sorts.py
+++ b/r2/r2/lib/db/sorts.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from math import log, sqrt
--- a/r2/r2/lib/db/stats.py
+++ b/r2/r2/lib/db/stats.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 cache_key = 'popular_queries'
--- a/r2/r2/lib/db/tdb_lite.py
+++ b/r2/r2/lib/db/tdb_lite.py
@@ -6,21 +6,22 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

 import sqlalchemy as sa
+import cPickle as pickle

 class tdb_lite(object):
    def __init__(self, gc):
--- a/r2/r2/lib/db/tdb_sql.py
+++ b/r2/r2/lib/db/tdb_sql.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from datetime import datetime
@@ -307,12 +307,12 @@ def get_read_table(tables):
            load, avg_load, conns, avg_conns, max_conns = ip_loads[ip]

            #prune high-connection machines
-            if conns < .9 * max_conns:
-                max_load = max(load, avg_load)
-                total_load += max_load
-                have_loads.append((ip, max_load))
-            else:
-                no_connections.append(ip)
+            #if conns < .9 * max_conns:
+            max_load = max(load, avg_load)
+            total_load += max_load
+            have_loads.append((ip, max_load))
+            #else:
+            #    no_connections.append(ip)

    if total_load:
        avg_load = total_load / max(len(have_loads), 1)
@@ -390,6 +390,11 @@ def make_thing(type_id, ups, downs, date, deleted, spam, id=None):
        transactions.add_engine(t.bind)
        r = t.insert().execute(**params)
        new_id = r.last_inserted_ids()[0]
+        new_r = r.last_inserted_params()
+        for k, v in params.iteritems():
+            if new_r[k] != v:
+                raise CreationError, ("There's shit in the plumbing. " +
+                                      "expected %s, got %s" % (params,  new_r))
        return new_id

    try:
@@ -401,7 +406,7 @@ def make_thing(type_id, ups, downs, date, deleted, spam, id=None):
            raise
        # wrap the error to prevent db layer bleeding out
        raise CreationError, "Thing exists (%s)" % str(params)
-        
+

 def set_thing_props(type_id, thing_id, **props):
    table = get_thing_table(type_id, action = 'write')[0]
@@ -554,7 +559,12 @@ def get_data(table, thing_id):
    for row in r:
        val = db2py(row.value, row.kind)
        stor = res if single else res.setdefault(row.thing_id, storage())
+        if single and row.thing_id != thing_id:
+            raise ValueError, ("tdb_sql.py: there's shit in the plumbing." 
+                               + " got %s, wanted %s" % (row.thing_id,
+                                                         thing_id))
        stor[row.key] = val
+
    return res

 def set_thing_data(type_id, thing_id, **vals):
@@ -583,6 +593,11 @@ def get_thing(type_id, thing_id):
                       spam = row.spam)
        if single:
            res = stor
+            # check that we got what we asked for
+            if row.thing_id != thing_id:
+                raise ValueError, ("tdb_sql.py: there's shit in the plumbing." 
+                                    + " got %s, wanted %s" % (row.thing_id,
+                                                              thing_id))
        else:
            res[row.thing_id] = stor
    return res
--- a/r2/r2/lib/db/thing.py
+++ b/r2/r2/lib/db/thing.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 #TODO byID use Things?
@@ -131,7 +131,13 @@ class DataThing(object):

    def _other_self(self):
        """Load from the cached version of myself. Skip the local cache."""
-        return cache.get(self._cache_key(), local = False)
+        l = cache.get(self._cache_key(), local = False)
+        if l and l._id != self._id:
+            g.log.error("thing.py: Doppleganger on read: got %s for %s",
+                        (l, self))
+            cache.delete(self._cache_key())
+            return 
+        return l

    def _cache_myself(self):
        cache.set(self._cache_key(), self)
@@ -270,8 +276,7 @@ class DataThing(object):
        prefix = thing_prefix(cls.__name__)

        if not all(x <= tdb.MAX_THING_ID for x in ids):
-            g.log.debug("Attempted to look up too-big thing_id?")
-            raise NotFound('thing_id greater than MAX_THING_ID')
+            raise NotFound('huge thing_id in %r' % ids)

        def items_db(ids):
            items = cls._get_item(cls._type_id, ids)
@@ -283,9 +288,16 @@ class DataThing(object):
        bases = sgm(cache, ids, items_db, prefix)

        #check to see if we found everything we asked for
-        if any(i not in bases for i in ids):
-            missing = [i for i in ids if i not in bases]
-            raise NotFound, '%s %s' % (cls.__name__, missing)
+        for i in ids:
+            if i not in bases:
+                missing = [i for i in ids if i not in bases]
+                raise NotFound, '%s %s' % (cls.__name__, missing)
+            if bases[i] and bases[i]._id != i:
+                g.log.error("thing.py: Doppleganger on byID: %s got %s for %s" %
+                            (cls.__name__, bases[i]._id, i))
+                bases[i] = items_db([i]).values()[0]
+                bases[i]._cache_myself()
+

        if data:
            need = [v for v in bases.itervalues() if not v._loaded]
@@ -549,6 +561,27 @@ def Relation(type1, type2, denorm1 = None, denorm2 = None):
        _incr_data = staticmethod(tdb.incr_rel_data)
        _type_prefix = Relation._type_prefix

+        @classmethod
+        def _byID_rel(cls, ids, data=False, return_dict=True, extra_props=None,
+                      eager_load=False, thing_data=False):
+
+            ids, single = tup(ids, True)
+
+            bases = cls._byID(ids, data=data, return_dict=True,
+                              extra_props=extra_props)
+
+            values = bases.values()
+
+            if values and eager_load:
+                load_things(values, thing_data)
+
+            if single:
+                return bases[ids[0]]
+            elif return_dict:
+                return bases
+            else:
+                return filter(None, (bases.get(i) for i in ids))
+
        def __init__(self, thing1, thing2, name, date = None, id = None, **attrs):
            DataThing.__init__(self)

--- a/r2/r2/lib/db/userrel.py
+++ b/r2/r2/lib/db/userrel.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.memoize import memoize, clear_memo
--- a/r2/r2/lib/emailer.py
+++ b/r2/r2/lib/emailer.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from email.MIMEText import MIMEText
--- a/r2/r2/lib/filters.py
+++ b/r2/r2/lib/filters.py
@@ -6,19 +6,21 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
+from BeautifulSoup import BeautifulSoup
+
 from pylons import c

 import cgi
@@ -29,6 +31,8 @@ from wrapped import Templated, CacheStub
 SC_OFF = "<!-- SC_OFF -->"
 SC_ON = "<!-- SC_ON -->"

+MD_START = '<div class="md">'
+MD_END = '</div>'


 def python_websafe(text):
@@ -118,59 +122,79 @@ def edit_comment_filter(text = ''):
        text = unicode(text)
    return url_escape(text)

-#TODO is this fast?
-r_url = re.compile('(?<![\(\[])(http://[^\s\'\"\]\)]+)')
-jscript_url = re.compile('<a href="(?!http|ftp|mailto|/).*</a>', re.I | re.S)
-img = re.compile('<img.*?>', re.I | re.S)
-href_re = re.compile('<a href="([^"]+)"', re.I)
-code_re = re.compile('<code>([^<]+)</code>')
-a_re    = re.compile('>([^<]+)</a>')
-fix_url = re.compile('&lt;(http://[^\s\'\"\]\)]+)&gt;')
+def markdown_souptest(text, nofollow=False, target=None, lang=None):
+    ok_tags  = {
+        'div': ('class'),
+        'a': ('href', 'title', 'target', 'nofollow'),
+        }
+
+    boring_tags = ( 'p', 'em', 'strong', 'br', 'ol', 'ul', 'hr', 'li',
+                    'pre', 'code', 'blockquote',
+                    'h1', 'h2', 'h3', 'h4', 'h5', 'h6', )
+
+    for bt in boring_tags:
+        ok_tags[bt] = ()
+
+    smd = safemarkdown (text, nofollow, target, lang)
+    soup = BeautifulSoup(smd)
+
+    for tag in soup.findAll():
+        if not tag.name in ok_tags:
+            raise ValueError("<%s> tag found in markdown!" % tag.name)
+        ok_attrs = ok_tags[tag.name]
+        for k,v in tag.attrs:
+            if not k in ok_attrs:
+                raise ValueError("<%s %s='%s'> attr found in markdown!"
+                                 % (tag.name, k,v))
+            if tag.name == 'a' and k == 'href':
+                lv = v.lower()
+                if lv.startswith("http:"):
+                    pass
+                elif lv.startswith("https:"):
+                    pass
+                elif lv.startswith("ftp:"):
+                    pass
+                elif lv.startswith("mailto:"):
+                    pass
+                elif lv.startswith("/"):
+                    pass
+                else:
+                    raise ValueError("Link to '%s' found in markdown!" % v)
+

 #TODO markdown should be looked up in batch?
 #@memoize('markdown')
-def safemarkdown(text, nofollow=False, target=None):
+def safemarkdown(text, nofollow=False, target=None, lang=None):
+    from r2.lib.c_markdown import c_markdown
+    from r2.lib.py_markdown import py_markdown
+    from pylons import g
+
    from contrib.markdown import markdown
+
    if c.user.pref_no_profanity:
        text = profanity_filter(text)
-    if text:
-        # increase escaping of &, < and > once
-        text = text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
-        #wrap urls in "<>" so that markdown will handle them as urls
-        text = r_url.sub(r'<\1>', text)
-        try:
-            text = markdown(text)
-        except RuntimeError:
-            text = "<p><em>Comment Broken</em></p>"
-        #remove images
-        text = img.sub('', text)
-        #wipe malicious javascript
-        text = jscript_url.sub('', text)
-        def href_handler(m):
-            url = m.group(1).replace('&amp;', '&')
-            link = '<a href="%s"' % url

-            if target:
-                link += ' target="%s"' % target
-            elif c.cname:
-                link += ' target="_top"'
+    if not text:
+        return None

-            if nofollow:
-                link += ' rel="nofollow"'
-            return link
-        def code_handler(m):
-            l = m.group(1)
-            return '<code>%s</code>' % l.replace('&amp;','&')
-        #unescape double escaping in links
-        def inner_a_handler(m):
-            l = m.group(1)
-            return '>%s</a>' % l.replace('&amp;','&')
-        # remove the "&" escaping in urls
-        text = href_re.sub(href_handler, text)
-        text = code_re.sub(code_handler, text)
-        text = a_re.sub(inner_a_handler, text)
-        text = fix_url.sub(r'\1', text)
-        return SC_OFF + '<div class="md">' + text + '</div>' + SC_ON
+    if c.cname and not target:
+        target = "_top"
+
+    if lang is None:
+        # TODO: lang should respect g.markdown_backend
+        lang = "py"
+
+    try:
+        if lang == "c":
+            text = c_markdown(text, nofollow, target)
+        elif lang == "py":
+            text = py_markdown(text, nofollow, target)
+        else:
+            raise ValueError("weird lang")
+    except RuntimeError:
+        text = "<p><em>Comment Broken</em></p>"
+
+    return SC_OFF + MD_START + text + MD_END + SC_ON


 def keep_space(text):
--- a/r2/r2/lib/find_tz.py
+++ b/r2/r2/lib/find_tz.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.models.link import Link
--- a/r2/r2/lib/hardcachebackend.py
+++ b/r2/r2/lib/hardcachebackend.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

@@ -26,6 +26,11 @@ from datetime import datetime
 import sqlalchemy as sa
 from r2.lib.db.tdb_lite import tdb_lite

+def expiration_from_time(time):
+    if time <= 0:
+        raise ValueError ("HardCache items *must* have an expiration time")
+    return datetime.now(g.tz) + timedelta(0, time)
+
 class HardCacheBackend(object):
    def __init__(self, gc):
        self.tdb = tdb_lite(gc)
@@ -50,10 +55,10 @@ class HardCacheBackend(object):

        self.delete(category, ids) # delete it if it already exists

-        expiration = datetime.now(g.tz) + timedelta(0, time)
-
        value, kind = self.tdb.py2db(val, True)

+        expiration = expiration_from_time(time)
+
        self.table.insert().execute(
            category=category,
            ids=ids,
@@ -62,6 +67,53 @@ class HardCacheBackend(object):
            expiration=expiration
            )

+    def add(self, category, ids, val, time=0):
+        expiration = expiration_from_time(time)
+
+        value, kind = self.tdb.py2db(val, True)
+
+        try:
+            rp = self.table.insert().execute(
+                category=category,
+                ids=ids,
+                value=value,
+                kind=kind,
+                expiration=expiration
+                )
+
+            return value
+
+        except sa.exceptions.IntegrityError, e:
+            return self.get(category, ids)
+
+    def incr(self, category, ids, time=0, delta=1):
+        expiration = expiration_from_time(time)
+
+        rp = self.table.update(sa.and_(self.table.c.category==category,
+                                       self.table.c.ids==ids,
+                                       self.table.c.kind=='num'),
+                               values = {
+                                         self.table.c.value:
+                                         sa.cast(
+                                                 sa.cast(self.table.c.value,
+                                                          sa.Integer) + delta,
+                                                 sa.String),
+                                         self.table.c.expiration: expiration
+                                         }
+                               ).execute()
+        if rp.rowcount == 1:
+            return self.get(category, ids)
+        elif rp.rowcount == 0:
+            existing_value = self.get(category, ids)
+            if existing_value is None:
+                raise ValueError("[%s][%s] can't be incr()ed -- it's not set" %
+                                 (category, ids))
+            else:
+                raise ValueError("[%s][%s] has non-integer value %r" %
+                                 (category, ids, existing_value))
+        else:
+            raise ValueError("Somehow %d rows got updated" % rp.rowcount)
+
    def get(self, category, ids):
        s = sa.select([self.table.c.value,
                       self.table.c.kind,
@@ -77,6 +129,25 @@ class HardCacheBackend(object):
        else:
            return self.tdb.db2py(rows[0].value, rows[0].kind)

+    def get_multi(self, category, idses):
+        s = sa.select([self.table.c.ids,
+                       self.table.c.value,
+                       self.table.c.kind,
+                       self.table.c.expiration],
+                      sa.and_(self.table.c.category==category,
+                              sa.or_(*[self.table.c.ids==ids
+                                       for ids in idses])))
+        rows = s.execute().fetchall()
+
+        results = {}
+
+        for row in rows:
+          if row.expiration >= datetime.now(g.tz):
+              k = "%s-%s" % (category, row.ids)
+              results[k] = self.tdb.db2py(row.value, row.kind)
+
+        return results
+
    def delete(self, category, ids):
        self.table.delete(
            sa.and_(self.table.c.category==category,
--- a/r2/r2/lib/helpers.py
+++ b/r2/r2/lib/helpers.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 """
--- a/r2/r2/lib/html_source.py
+++ b/r2/r2/lib/html_source.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from HTMLParser import HTMLParser
--- a/r2/r2/lib/jsonresponse.py
+++ b/r2/r2/lib/jsonresponse.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.lib.utils import tup
--- a/r2/r2/lib/jsontemplates.py
+++ b/r2/r2/lib/jsontemplates.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from utils import to36, tup, iters
@@ -178,11 +178,11 @@ class ThingJsonTemplate(JsonTemplate):
            return self.rendered_data(thing)
        else:
            return self.raw_data(thing)
-        
+
    def render(self, thing = None, action = None, *a, **kw):
        return ObjectTemplate(dict(kind = self.kind(thing),
                                   data = self.data(thing)))
-        
+
 class SubredditJsonTemplate(ThingJsonTemplate):
    _data_attrs_ = ThingJsonTemplate.data_attrs(subscribers  = "score",
                                                title        = "title",
@@ -193,7 +193,15 @@ class SubredditJsonTemplate(ThingJsonTemplate):
 class AccountJsonTemplate(ThingJsonTemplate):
    _data_attrs_ = ThingJsonTemplate.data_attrs(name = "name",
                                                link_karma = "safe_karma",
-                                                comment_karma = "comment_karma")
+                                                comment_karma = "comment_karma",
+                                                has_mail = "has_mail")
+
+    def thing_attr(self, thing, attr):
+        if attr == "has_mail":
+            if c.user_is_loggedin and thing._id == c.user._id:
+                return bool(c.have_messages)
+            return None
+        return ThingJsonTemplate.thing_attr(self, thing, attr)

 class LinkJsonTemplate(ThingJsonTemplate):
    _data_attrs_ = ThingJsonTemplate.data_attrs(ups          = "upvotes",
@@ -316,12 +324,15 @@ class MessageJsonTemplate(ThingJsonTemplate):
    _data_attrs_ = ThingJsonTemplate.data_attrs(new          = "new",
                                                subject      = "subject",
                                                body         = "body",
+                                                replies      = "child",
                                                body_html    = "body_html",
                                                author       = "author",
                                                dest         = "dest",
                                                was_comment  = "was_comment",
                                                context      = "context", 
-                                                created      = "created")
+                                                created      = "created",
+                                                parent_id    = "parent_id",
+                                                first_message= "first_message")

    def thing_attr(self, thing, attr):
        if attr == "was_comment":
--- a/r2/r2/lib/lock.py
+++ b/r2/r2/lib/lock.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

--- a/r2/r2/lib/logger.py
+++ b/r2/r2/lib/logger.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from __future__ import with_statement
--- a/r2/r2/lib/manager/init.py
+++ b/r2/r2/lib/manager/init.py
@@ -6,16 +6,16 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
--- a/r2/r2/lib/manager/db_manager.py
+++ b/r2/r2/lib/manager/db_manager.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import sqlalchemy as sa
--- a/r2/r2/lib/manager/tp_manager.py
+++ b/r2/r2/lib/manager/tp_manager.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 import pylons, sha
--- a/r2/r2/lib/media.py
+++ b/r2/r2/lib/media.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################

--- a/r2/r2/lib/memoize.py
+++ b/r2/r2/lib/memoize.py
@@ -6,23 +6,22 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from r2.config import cache
 from r2.lib.filters import _force_utf8
-
-class NoneResult(object): pass
+from r2.lib.cache import NoneResult

 def memoize(iden, time = 0):
    def memoize_fn(fn):
--- a/r2/r2/lib/menus.py
+++ b/r2/r2/lib/menus.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 from wrapped import CachedTemplate, Styled
--- a/r2/r2/lib/migrate.py
+++ b/r2/r2/lib/migrate.py
@@ -6,17 +6,17 @@
 # software over a computer network and provide for limited attribution for the
 # Original Developer. In addition, Exhibit A has been modified to be consistent
 # with Exhibit B.
-# 
+#
 # Software distributed under the License is distributed on an "AS IS" basis,
 # WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
 # the specific language governing rights and limitations under the License.
-# 
+#
 # The Original Code is Reddit.
-# 
+#
 # The Original Developer is the Initial Developer.  The Initial Developer of the
 # Original Code is CondeNet, Inc.
-# 
-# All portions of the code written by CondeNet are Copyright (c) 2006-2009
+#
+# All portions of the code written by CondeNet are Copyright (c) 2006-2010
 # CondeNet, Inc. All Rights Reserved.
 ################################################################################
 """
@@ -135,3 +135,74 @@ def subscribe_to_blog_and_annoucements(filename):
                print ("%d: subscribed %s to %s" % (i, account.name, sr.name))
            else:
                print ("%d: didn't subscribe %s to %s" % (i, account.name, sr.name))
+
+
+def upgrade_messages(update_comments = True, update_messages = True,
+                     update_trees = True):
+    from r2.lib.db import queries
+    from r2.lib import comment_tree, cache
+    from r2.models import Account
+    from pylons import g
+    accounts = set()
+
+    def batch_fn(items):
+        g.reset_caches()
+        return items
+    
+    if update_messages or update_trees:
+        q = Message._query(Message.c.new == True,
+                           sort = desc("_date"),
+                           data = True)
+        for m in fetch_things2(q, batch_fn = batch_fn):
+            print m,m._date
+            if update_messages:
+                accounts = accounts | queries.set_unread(m, m.new)
+            else:
+                accounts.add(m.to_id)
+    if update_comments:
+        q = Comment._query(Comment.c.new == True,
+                           sort = desc("_date"))
+        q._filter(Comment.c._id < 26152162676)
+
+        for m in fetch_things2(q, batch_fn = batch_fn):
+            print m,m._date
+            queries.set_unread(m, True)
+
+    print "Precomputing comment trees for %d accounts" % len(accounts)
+
+    for i, a in enumerate(accounts):
+        if not isinstance(a, Account):
+            a = Account._byID(a)
+        print i, a
+        comment_tree.user_messages(a)
+
+def recompute_unread(min_date = None):
+    from r2.models import Inbox, Account, Comment, Message
+    from r2.lib.db import queries
+
+    def load_accounts(inbox_rel):
+        accounts = set()
+        q = inbox_rel._query(eager_load = False, data = False,
+                             sort = desc("_date"))
+        if min_date:
+            q._filter(inbox_rel.c._date > min_date)
+
+        for i in fetch_things2(q):
+            accounts.add(i._thing1_id)
+
+        return accounts
+
+    accounts_m = load_accounts(Inbox.rel(Account, Message))
+    for i, a in enumerate(accounts_m):
+        a = Account._byID(a)
+        print "%s / %s : %s" % (i, len(accounts_m), a)
+        queries.get_unread_messages(a).update()
+        queries.get_unread_comments(a).update()
+        queries.get_unread_selfreply(a).update()
+
+    accounts = load_accounts(Inbox.rel(Account, Comment)) - accounts_m
+    for i, a in enumerate(accounts):
+        a = Account._byID(a)
+        print "%s / %s : %s" % (i, len(accounts), a)
+        queries.get_unread_comments(a).update()
+        queries.get_unread_selfreply(a).update()
--- a/Show More
+++ b/Show More