There's a new firestorm brewing in web services architectures.
Cloud services are being talked up as a fundamental shift in web architecture that promises to move us from interconnected silos to a collaborative network of services whose sum is greater than its parts. The problem is that the protocols powering current cloud services; SOAP and a few other assorted HTTP-based protocols are all one way information exchanges. Therefore cloud services aren't real-time, won't scale, and often can't clear the firewall. So, it's time we blow up those barriers and come to Jesus about the protocol that will fuel the SaaS models of tomorrow--that solution is XMPP (also called Jabber) . Never heard of it?
In just a couple of years Google, Apple, AOL, IBM, Livejournal and Jive have all jumped on board.
Sounds good, right? So, what's the hold up? Why aren't we building out cloud services with XMPP now? And, if people are already providing cloud services without XMPP, what's the motivation to switch? The rest of this post will shed some light on the current landscape and provide some answers to those questions.
Polling isn't working anymore
Since the beginning of the Internet, if you wanted to sync services between two servers the most common solution was to have the client ping the host at regular intervals, which his known as polling. Polling is how most of us check our email. We ping our email server every few minutes to see if we got new mail. It's also how nearly all web services APIs work.
Take, for example, Twitter. High Scalability recently covered the load stats on Twitter reporting that they average 200-300 connections per second with spikes that climb to 800 connections per second. Their MySQL server handles 2,400 requests per second! Recently, the Macworld keynote became the most recent culprit for causing Twitter to cut off its API, which has 10x the load of their website. While Twitter is not a cloud service, nor the largest demand service on the internet (with a paltry 350,000ish users, which pales in comparison to a MySpace or Yahoo!), they do illustrate the kind of frustration a user experiences with polling based services. And, that's just Twitter! Imagine the impact on overall Internet traffic congestion polling creates worldwide.
Interestingly, the recent Twitter outage lead some influencers, like Dave Winer, to suggest that Twitter move to XMPP which we've already begun experimenting with 
Some companies are trying to address the polling problem with existing protocols. I think that move is largely motivated by a significant investment in legacy systems that makes moving to another protocol difficult. Salesforce is a perfect example of a company attempting to address the polling problem with creative applications of the old one way protocols.
The latest version of Salesforce will send notifications back to your own webservice to avoid polling. But, that's a pain to setup for developers. Worse, its very difficult to wire up reverse webservices calls through a corporate firewall.
The hold up
XMPP's largest hurdle is that its not HTTP, and common wisdom states everything new that's built must be web-based. That means we won't see a widespread application of XMPP in cloud services until a few more brave pioneers clear the path for the rest of us.
I've been heavily involved in the XMPP world as a developer of Smack (client library) and Openfire (server) and have also helped craft the standard as a member of the XMPP Standards Foundation. XMPP was invented for instant messaging and presence, and is the dominant open protocol in that space. Instant messaging? Yep, it turns out that all of the problems that had to be solved for instant messaging make the protocol perfect for cloud computing:
It allows for easy two-way communication, so bye bye polling. It even has rich pub-sub functionality built-in.
It's XML-based and easily extensible, perfect for both new instant messaging features and custom cloud services.
It's efficient and proven to scale to millions of concurrent users on a single service (such as Google's GTalk). It also has a built-in worldwide federation model.
I'm not the only one to notice that XMPP is a great fit for cloud computing. Tivo is switching to XMPP as a more efficient alternative to their old architecture:
<div class="jive-quote">Today each TiVo polls TiVo’s severs roughly every 15 minutes to check for new scheduled recordings, TiVoCast downloads, Unbox downloads, etc. That’s highly inefficient - nearly all of those polling calls are for nothing. There is nothing waiting to be done. And it introduces a lag when you want to start a download - up to 15 minutes. And it doesn’t scale well as TiVo’s user base keeps growing.
So what’s changed? The polling system is gone. TiVo is using XMPP now instead. (...) Yep, TiVo is basically using instant messaging for real- time communication. Now when the TiVo server has a new recording to schedule, it will IM the TiVo to tell it. Or if there is a download to pull, it will IM the TiVo to tell it to do so. This is a much more efficient system and it eliminates latency. It is really a clever idea.
</div>
Fixing the polling and scaling problems with XMPP as Tivo has done is compelling, but the built-in presence functionality also offers tantalizing possibilities. Presence includes basic availability information, but is extensible and can also include things like geo-location. Imagine cloud services taking different actions based on where the client is connecting from.
More people, us included, will make the shift to XMPP, which will provide the missing evidence to create momentum toward a tipping point. In fact, I'm happy to announce that Clearspace 2.0 will include a feature that's powered by an XMPP-based cloud service. We'll be publishing a series of blog entries in the near future to discuss how we built it.
Resources for XMPP cloud service developers
There are a few places you can turn for help building cloud services around XMPP. Here is a list of a few:
I'm pleased to announce that Jive Software has acquired Jotlet. Jotlet has built some amazing calendar technology that we'll be incorporating into Clearspace in a future release. The two super-talented guys behind Jotlet are joining our team in Portland from Texas.
