Commit Graph

56 Commits

Author SHA1 Message Date
Geoff Schmidt
5661a590f2 Move spiderable docs from docs.meteor.com to readme 2014-10-13 18:37:23 -07:00
Sashko Stubailo
36da5aecb4 Make all version numbers not have rc 2014-10-13 14:09:37 -07:00
Sashko Stubailo
698fbedb10 Bump all of the version numbers to rc.0 2014-10-07 17:10:48 -07:00
Sashko Stubailo
1e53f6b598 Bump all version numbers again after cherry-picking unipackage fix 2014-10-01 17:06:57 -07:00
Sashko Stubailo
0129c3f5ac Bump all of the version numbers again, to republish with the unipackage.json fix 2014-10-01 15:43:05 -07:00
Sashko Stubailo
eecc272e4a Bump every single package version number due to a change in compilation 2014-09-29 23:44:50 -07:00
Avital Oliver
c23408bdd7 versions for 0.9.1 2014-09-04 15:41:39 -07:00
Avital Oliver
864fffc9e9 Bump package versions in preparation for 0.9.1-rc0 2014-08-28 11:00:35 -07:00
Sashko Stubailo
41910019e1 Renamed Deps to Tracker in all packages 2014-08-27 20:05:22 -07:00
Emily Stark
b0e0ccb5a4 Bump package versions for 0.9.0.1 2014-08-27 15:16:49 -07:00
Emily Stark
1b639e5853 Bump package versions for 0.9.0.1-rc1 2014-08-27 14:28:03 -07:00
Justin SB
f1a7455d4c Spiderable now uses a flag to determine when the initial JS has loaded
Other approaches seemed to be heuristics that broke down in edge cases:
a timeout wasn't guaranteed if your JS was slow; page.load in PhantomJS
wasn't firing reliably because of long polling.
2014-08-27 12:25:45 -07:00
David Glasser
563860c769 Version number bumps, this time for real 2014-08-05 19:03:16 -07:00
David Glasser
845286b925 Merge branch 'devel' into packaging 2014-08-05 13:13:27 -07:00
MaximDubrovin
6345d92f07 move phantom_script.js setInterval into page.open callback
Until I made this I always got only ```<head/>``` with it's content, ```<body>``` was empty.

It seems setInterval script was finished earlier then url content was loaded to the page. Maybe because I have subscriptions with response time lower then 100ms so they were ready very quickly — database server in the same data center.

http://phantomjs.org/api/webpage/method/open.html
2014-08-01 16:44:04 -07:00
David Glasser
996d347139 Merge branch 'devel' into packaging
Conflicts:
	docs/client/api.html
	docs/client/concepts.html
	docs/client/introduction.html
	meteor
	packages/html-tools/package.js
	packages/spacebars-common/package.js
	packages/webapp/webapp_server.js
	tools/bundler.js
2014-07-21 16:18:25 -07:00
David Glasser
39fc2ac12c Move spiderable phantom script to an asset 2014-07-18 16:45:16 -07:00
ekatek
fb709c9563 remove versions files from core packages 2014-06-16 22:11:31 -07:00
David Glasser
c5e5305a3b Merge branch 'devel' into packaging
Conflicts:
	tools/bundler.js
	tools/packages.js
2014-06-11 12:02:57 -07:00
Emily Stark
560ba659e1 Use ROOT_URL instead of Host header in spiderable 2014-05-21 20:40:40 -07:00
ekatek
beb0e59b9d new version files from teh new solver? Also, package.js correct with tests 2014-05-09 17:36:25 -07:00
ekatek
031890d2c5 don't store buildtime deps in the catalog 2014-05-04 16:56:58 -07:00
ekatek
23ad149a78 better versions files 2014-04-25 15:06:45 -07:00
ekatek
10d2047d12 clean up 2014-04-25 15:01:20 -07:00
ekatek
c84c8d6097 alphabetize package version lock files 2014-04-24 18:18:21 -07:00
Geoff Schmidt
66b7253695 Take all packages to 1.0.0 (that aren't already there) 2014-03-07 12:00:20 -08:00
David Glasser
06267daa93 Comment: we only partially do AJAX crawling spec
Fixes #487
2014-03-03 20:55:37 -08:00
Nick Martin
fbfcb4859b Increase buffer size for phantomjs in spiderable. Our docs were too big. 2013-10-18 13:28:25 -07:00
Nick Martin
57c87a403d OCD 2013-10-08 20:03:02 -07:00
Nick Martin
30b08fc012 Export Spiderable symbol, so people can override user agents. Also add twitterbot to agents. Fixes #1300. 2013-08-19 19:51:01 -07:00
Geoff Schmidt
3d1c09794f Comprehensive namespace cleanup. 2013-07-25 18:54:40 -07:00
David Glasser
053db8e9e9 Begin to refactor webapp package.
Move stuff from __meteor_bootstrap__ to WebApp exported symbol.

Note that the WebApp API continues to be subject to change.
2013-07-02 16:15:12 -07:00
David Glasser
2d827e74fc Factor webapp-specific stuff (and keepalive) out of boot.js into webapp package.
Replace type: "bare" with type: "server" in attributes.json.  This now just
means "only make one server process" rather than controlling the boot script.

Programs that are type: "traditional" that want to be a webapp now need to
explicitly depend on the webapp package.
2013-05-13 11:58:58 -07:00
Naomi Seyfer
3aa28e54ab some changes necessary to make various packages work together 2013-05-13 11:58:57 -07:00
David Glasser
8dbee56e77 Add .gitignore to package directories to match linker branch.
This way switching between linker and devel doesn't leave you with lots of junk
in git status.
2013-05-13 10:57:37 -07:00
David Glasser
308395da46 Spiderable.userAgentRegExps can now be modified to change what user agents are
treated as spiders by the spiderable package.
2013-04-09 16:03:57 -07:00
David Glasser
5a5204e3a4 Remove closures around package files. (hint: git blame -w)
The bundler now adds closures around client files and the server adds closures
around server files.
2013-03-19 15:13:53 -07:00
Avital Oliver
78ca755d3d Npm.require unified requireNpm and __meteor_bootstrap__.require 2013-03-19 15:11:04 -07:00
David Glasser
70f1412c0a Avoid crash on spiderable errors. Fixes #832. 2013-03-18 10:22:37 -07:00
David Greenspan
975454d4e4 Merge branch 'devel' into deps-radical 2013-03-04 13:46:50 -08:00
Morten Henriksen
a60f4eeddd Fixed #721
Testing for bots should be case insensitive - facebook bots are not all
lowercase - they adapted CamelCase on some servers. I've added the
linkedin bot just for the sake of it.
2013-02-28 14:37:23 -08:00
David Greenspan
f6aabffab3 port all packages to new Deps (tests pass) 2013-02-25 17:52:51 -08:00
David Glasser
aec2e586d8 spiderable: Correctly kill phantomjs after timeout. Fixes #571.
The previous version only killed the wrapper bash script, not the child. We now
work around the broken (socketpair) stdin presented to Node's child processes by
using a bash heredoc rather than "cat |"; this means we can use exec, which only
gives us one process to clean up.

While we're at it, simplify code by switching from child_process.spawn to
child_process.execFile, which does the work of capturing stdout and setting a
timeout for us automatically.
2013-01-08 22:51:40 -08:00
Nick Martin
cb0d7238a1 Convert to use regexps for user agents, so we can match multiple Facebook agents. 2012-10-29 19:14:22 -07:00
ayalgelles
2b72391ccf using a list of agents to serve 2012-10-29 19:14:22 -07:00
ayalgelles
40010607af serve facebookexternalhit user agent with spiderable 2012-10-29 19:14:22 -07:00
David Glasser
6c6fa5a6b1 Fix race condition in spiderable.
If the "check to see if page is ready" interval ran when "Meteor" is defined but
"Meteor.status" is not yet defined, an error message about trying to call an
undefined function would be dumped into the output source before the page
itself.

This can be easily reproduced by changing the timer interval from 100 to 1.
2012-09-24 10:41:14 -07:00
David Glasser
e27304c999 Work around Node 0.8 brokenness with using /dev/stdin in subprocesses. 2012-09-19 13:50:06 -07:00
Nick Martin
0397945fd8 Add hack in livedata_connection to support knowing when all data is received on all connections. 2012-08-03 22:00:25 -07:00
Nick Martin
07c703270c Use subscription readiness instead of timeout.
This only looks at default_connection, though, which isn't good. May need to patch livedata.
2012-08-03 20:59:38 -07:00