parislemon:

Q. Does Google need a deal to crawl Twitter? A. No. Google’s web bots crawl Twitter 120M+ times a day; 3 billion pages indexed & counting.

— Twitter Comms (@twittercomms) January 23, 2012

With all the Google Search+ talk over the past few weeks, there seems to be something very simple…

I’ve seen this addressed in multiple places. Google had real-time search set up with Twitter last year, and invested a fair amount of engineering and resources in it. Then, with basically no notice, Twitter pulled the plug on the deal and the Google engineers had to work around the clock for a couple of days to dismantle the thing.

My main source for this is Matt Cutts, on TWiG. Cutts said that since then, Google’s been hesitant to invest the resources in something that would be heavily reliant on inputs from Twitter or Facebook because the plug could be pulled at any moment. Let’s say that Google went ahead and integrated all public data they crawl from Facebook and Twitter into personalized search, and the very day that they announced it, Facebook and Twitter changed their robot.txt files and told Google not to crawl any of their pages any more. It would be real-time search all over again.

That’s the situation as I understand it. Not saying Google’s made the best decisions, just saying that pretty much all companies involved have left us with this mess.

We would all be better off if they could get over themselves and just strike deals and stick to them. You know Google would pay big bucks to both companies to inform search results with their data. We’d get better search results, Twitter and Facebook would get a big payday, Google would improve its service, everybody wins.