Files
reddit/r2
xiongchiamiov 94984b78af Robots: allow crawling of mobile pages
We got a message from Google Webmaster Tools letting us know they'd like to
pretend to be a mobile device and crawl our mobile pages, but can't, because we
disallow it in our `robots.txt` rules.  We have canonicalization set up to
point towards desktop pages, so we shouldn't be hurt by allowing this.

We should also at some point provide two-way annotations to let them go from
desktop -> mobile[0], but for now this should suffice.

[0]: https://developers.google.com/webmasters/mobile-sites/mobile-seo/configurations/separate-urls
2015-01-06 12:03:41 -08:00
..
2015-01-06 12:03:41 -08:00
2014-05-02 16:26:31 -04:00
2015-01-05 14:06:46 -08:00
2014-12-02 11:04:31 -08:00
2014-05-02 16:26:31 -04:00
2014-05-02 16:26:31 -04:00