Last night's Google London Open Source Jam (also here) was on the subject of the 'Web' (didn't they invent that? Oh no, that was Microsoft).
This event has been getting better and better each time I've attended. There were some very interesting lightning talks held together with a tight structure and plenty of chance to chat, drink cold Leffe and eat cold pizza. And nick [transatlantic translation: 'steal'] the Green & Black's chocolate.
An ideal Micro Conference... ...
Last night's Google London Open Source Jam (also here) was on the subject of the 'Web' (didn't they invent that? Oh no, that was Microsoft).
This event has been getting better and better each time I've attended. There were some very interesting lightning talks held together with a tight structure and plenty of chance to chat, drink cold Leffe and eat cold pizza. And nick [transatlantic translation: 'steal'] the Green & Black's chocolate.
An ideal Micro Conference...
I arrived late (it starts at 6pm) and spent some time catching up with ex-Thoughtworks colleagues, so I missed Dion "Ajaxian" Almaer's Google Gears slideset from FOWA. Go there now and check it out.
Thus the first talk I saw was a nifty piece of widgetry by Steven Goodwin called WARP. In WARP, interacting with a page of 'applets' changed the URL to encode those applets' current state. If you link to the current page, it will always show that state. Very long URLs, you can imagine. None of that fancy Ajax stuff. RESTful, dare I say. Nice API server-side for unpacking your applet params.
A trip to the lavatories [transatlantic translation: 'restroom'/'bathroom'] revealed that they are, indeed, doing that Testing in the Toilet project in Google. It works, too! I learned something. Other intelligence on Google's Inner Workings include confirmation of the beanbags and of the high quality, free grub to which I have already alluded.
A nice bloke from Yahoo! (Tom Hughes-Croucher: another spy?) came along to sell his idea that, in the collaborative world of open-minded hackers, we who run websites could help each other with our 404s. If I get a 404, I use the referrer link to tell you, via some RESTful POST, that your link to me is bust (assuming I don't intend to fix it myself).
I think the world is a little more selfish, so you need to decide who hurts more - the site who sends their visitors to a dead-end, or the site delivering that dead-end to a new visitor. I suspect the latter, by a small margin, as it's not exactly a nice welcome. So it's up to them to let the new visitor down more gently, and to notify the publisher of the broken link with little or no cost to them. For example, a really sociable 404-ing site could just redirect the hapless visitor back to the referring page, adding '?broken=links' to the URL - hopefully to be picked up by log scanning scripts at the referring site.
Next up, yours truly taking yet another chance to promote his excellent Micro Web thingy. Couple of people asked about it afterwards - including that nice chap from Yahoo! Also, a smart - and nice - chap called Toby (this one?) got me into a deep discussion on imperative vs. event-driven vs. state-driven programming. He was apparently an old-timer like me, as he was able to engage in dewy-eyed Functional Programming recollections. I managed to give out about four full colour printouts about the Micro Web, and to collect some good calling cards.
However, Joe Walnes, even a pint down in the pub afterwards, still refused to sign up for Micro Web duties. This in spite of over three years of intensive lobbying, including eight months of me working Trojan-horse-like in his kitchen, on The 2005 Implementation.
Another ex-Thoughtworks colleague, Simon Stewart took yet another chance to promote his promising Webdriver thingy. And a very interesting project it is becoming. Still needs more work - on IE support, etc - but I'll probably be using it in my new job at the Financial Times.
Another ex-Thoughtworks colleague, Chris Matts took a chance to promote his and Andy Pols' interesting new Dream Machine thingy. Perhaps a bit like Cambrian House - you put your dreams and ideas into it and people expand on them. Chris is a natural on-stage - and even used the age-old trick of promising lots of money for no effort, to get our attention at the start.
All I could come up with for the Micro Web was 'Cheaper, Wider, Faster'...
Updated: added reference to Dion Almaer, details about WARP, swapped in the picture of TWers that I was waiting for and fixed a minor blunder thanks to that ever-sharp ThoughtWorker, Dan Bodart..
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 4: Inter-Enterprise REST Integration ...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 4: Inter-Enterprise REST Integration
Duncan Cragg: OK - I've demonstrated how you can replace imperative, function-call API-driving with a clean, declarative, RESTful interaction, driven by simple business rules.
We had servers run by eBay and clients run by the public, in the same way your SOAP API is used.
eBay Architect: Ah: that's something SOA has that REST doesn't!
DC: What? What's that?
eA: Services are all about Enterprise Integration: about servers talking to servers. In REST you're all about clients talking to servers. The Web is essentially only browser clients talking to Web servers. With Web Services, you can do more serious Enterprise Integration.
DC: You never give up do you? So you want 'serious' integration. Is that within or between enterprises?
eA: Let's say between.
DC: Fine. We'll use the same example as before: it's just a variation on the Patterns used.
We can standardise a more general version of the eBay schemas for Items, Offers, ResponseToBestOffers and so on. Anyone can put their own Items, Offers, etc. up on their own servers, or on some public auction service site. Everyone can do auctions with eBay and with anyone else who decides to set up.
Even, say, a new Google auction site: let's call it 'gBay'!
eA: Ha! OK, let's go through this slowly: you have eBay and 'gBay' sites, with sets of users on each. Now Ernie wants to sell his old laptop on eBay, so creates a new Item for it. Gordon is registered to gBay and needs a cheap laptop.
DC: Great - well the first thing is search. As an interoperable site, gBay offers a broad search across both gBay sale Items and eBay ones - cached and indexed internally. The gBay search database would be filled by crawling eBay URIs and even by running queries on eBay.
eA: Mm. Have to check the T's & C's...
DC: So Gordon on gBay finds Ernie's laptop on eBay. The presentation of this eBay sale item will be given the gBay style, but calling out directly to the eBay data and images.
eA: OK, now let's say Gordon decides to make an offer.
DC: So an Offer resource is created on gBay referring to the laptop on eBay. Then through a notification, the Item on eBay is alerted to this Offer.
eA: What's notified, to where?
DC: There's a number of possible patterns. Before, we had the pattern of POSTing a resource to a server that then creates the GETable version.
However, now gBay is hosting the Offer, so the internal mechanisms for notification are no longer available.
So gBay could suggest an update through APP or a simpler POST to a collection of Offer entries within the eBay Item to point to this, now remote, Offer.
Perhaps the gBay Offer can simply be POSTed wholesale to the eBay Item.
Or just a link to it.
Or eBay may poll, read a feed or search gBay for new Offer URIs, putting them into Offer lists as they come up.
An unusual approach (thanks to Joe Gregorio) would be for gBay to GET the eBay Item, with the Offer marked in a Referer: header.
eA: Plenty of patterns to choose from. So there are some Offers on eBay, some on gBay. The Item lists its Offers in a rank as before, as they appear through this notification.
Now, let's say Ernie wants to accept Gordon's Offer on gBay.
DC: OK, assuming he can see the Offers the same regardless of host, he just chooses Gordon's Offer on the offer listing for his Item and accepts it.
eA: So we need to create a ResponseToBestOffer on eBay.
DC: Yes. Now the patterns are reversed, because eBay needs to notify gBay this time - of its ResponseToBestOffer.
Pub-Sub and Observer Pattern
DC: Again, it can do this by POSTing the ResponseToBestOffer to each Offer on gBay in turn, or can POST the actual Item itself to each Offer, where the Item has a link to the ResponseToBestOffer.
That would implement a logical subscription to the Item from each of the Offers on it.
eA: It sounds to me like POSTing several times to implement this pub-sub pattern is physically inefficient, even if it's logically correct. Especially when it's the same information repeated from eBay to gBay servers.
DC: Yes, indeed: a single notification to gBay would be better, letting gBay handle the propagation of subscription responses. This would in effect treat gBay as a proxy cache, and the notification as a cache invalidation event on gBay's copy of the eBay Item.
eA: What URI on gBay would you POST this eBay Item to?
DC: Something like http://gbay.com/ebay.com/item/4243 - to a
copy of itself. You could also GET this cached copy if you wanted.
eA: OK, what next?
DC: In gBay the losing Offers get updated on receipt of this ResponseToBestOffer state. Gordon's Offer gets set to 'won'. In eBay, all the losing Offers are updated to 'lost'. The laptop Item gets marked 'sold', with a link to the ResponseToBestOffer, which links to the Offer that won.
It is possible to implement this internally in eBay (and that pub-sub cache invalidation propagation in gBay) using the Observer Pattern. and an event-driven server.
eA: Makes sense - you mean something like SEDA?
DC: Yep.
So the Offers all subscribe to the Item to watch for its status switching to 'sold' and to see if they won. Conversely, the Item can subscribe to the Offers: maybe the Offers could change or be withdrawn, and the Item needs to keep itself updated accordingly.
eA: Wow - symmetric subscription - the two-way Observer Pattern!
OK, what next?
DC: The eBay laptop Item resource will be further updated by its owner with paid, shipped, refunded, etc., as it currently is within eBay.
eA: Hold on, you're mixing patterns: you had the Observer Pattern on the Item just now: the Item observes the Offers. The Offers' state can be POSTed to the Item, whose own state may then change according to its rules.
But you then mix patterns by allowing a POST directly to the Item from the Item's owner, to update a couple of fields.
In one, the Item chooses what its state will be according to the state of its peers, and in the other, it's told, not according to a peer state, but some POST content type.
That doesn't seem neat or symmetric.
DC: It's true that these interaction styles differ: the Observer Pattern or pub-sub approach is peer-to-peer (resource-to-resource as equals watching each other); and in this scenario it's also server-to-server.
The direct edit request is more a client-server pattern, where the server resource - the Item - is considered under the control of a client.
However, the Item is always in control of its own state, and can even ignore a request by its owner if that request doesn't match its internal integrity rules.
The Item supporting both styles at the same time is absolutely fine.
Actually, you could see these two styles as aspects of the same peer-to-peer pattern: introduce a resource in the client that holds edit requests, to which the Item subscribes. It all ends up being much the same.
Transactions, Trust
eA: Right, now what if you have a race, where the ResponseToBestOffer is created at the same time as an Offer is changed or withdrawn?
Don't you need some kind of two-phase commit or distributed transaction logic?
DC: Of course not. It's the same as in the real world: as long as it all settles in the end and the rules are followed. The ResponseToBestOffer cites what state of the Offer it is accepting. If that changes for any reason, the ResponseToBestOffer is void.
It's about state and state consistency in REST, as opposed to the SOA style of maintaining total control at all times.
There will be temporary states that trigger the rules and that need to be resolved. That's the programming and distribution model. Tolerance of transient states is what makes this model so robust.
eA: Surely there are some legal and contract issues? How is this exchange legally binding?
DC: You can digitally sign the Item, Offer and ResponseToBestOffer resources, and each side needs to keep records of the history. Then it's down to agreements between eBay and gBay and the local laws in force.
eA: What about buyer and seller ratings and feedback?
DC: Ernie in eBay and Gordon in gBay can happily publish feedback about each other, and Ernie will be able to see Gordon's rating via eBay's interface, or directly on gBay.
As for aggregated ratings from several buyer/seller interactions: a person's rating is a function of the ratings of all those they have dealt with. These ratings can be fetched by GET from remote sites, and combined with internally-held ratings, depending on the trust of one site over another site's ratings.
eA: So how do we trust these ratings across sites?
DC: We have to trust eBay that it trusts gBay. This is one of the basics of distributed systems. In a monolithic system you have a single trust domain: all parts can trust each other.
Split the application up across multiple trust domains and you need authentication and crypto. You can't get way from needing peer trust structures built up explicitly through crypto, agreement and contract and/or implicitly through past successful experience.
eA: Can you be more specific?
DC: Normally, a GET for a resource or a POST of some data comes with a header identifying the GETer or POSTer. The resource can also be signed by a user on a site or by the site itself as a proxy.
Or, if you have an agreement with the site, you just need to use https to ensure you've got a secure connection with that site, then needn't have individual signatures.
eA: Where's the Single Sign On and Identity in all this? We've got users working across multiple sites.
DC: Well, gBay is the holder of the Gordon identity or persona - and it manages his world view. Gordon on gBay needs his identity to mean something on eBay, but we don't want him to have to create an account on eBay or to have to tell gBay his eBay login details to work on both sites. So he expects gBay and eBay to have come to some agreements about technology and policy.
In REST, we don't have sessions and logins - we have identity, which implies asymmetric (private/public key) crypto for signatures and security. We have a number of tools available to us, including OpenID and https, as well as resource signing.
eA: Here's a question for you: how would you manage a single shopping trolley for Gordon on gBay, containing and allowing payment for eBay goods?
DC: ShoppingTrolley resource, links to eBay and gBay items. At checkout, smaller eBay-Items-only ShoppingTrolley resource POSTed to eBay along with CreditCard resource (again, you can sign the ShoppingTrolley and encrypt the data).
eA: So, as eBay, why should we integrate the seller ratings of someone on gBay? Or get gBay's for-sale items coming up in our searches? Or accept Offers and ShoppingTrolleys from gBay? We don't control or trust them, and don't want to send traffic or business over to them.
DC: Fair enough, for now. I'm only describing what's technically possible. Like I said before, you may revisit your stance on interoperability and mutual agreements one day soon.
Also, what if your business decides this year to set up a commercial partnership with another similar business and the managers come to you asking how it's all going to work together internally?
You'll find having good REST interoperability already in place a huge asset for internal integration! You'll also find that an interop-friendly approach makes developing internal 'mashups' much easier.
Better Than SOA
eA: I still can't see why all this is better than our SOAP approach, though: it just seems like the same things are happening at the end of the day - that it's only a change of perspective.
DC: Well, a minute ago, you were challenging using REST for anything other than simple data manipulation. Now I've shown you the power of a REST approach can be easily extended to a clean, simple, scalable, interoperable, general, declarative programming model. And you're still not satisfied!
eA: Ha! OK. So tell me why this programming model is so scalable and interoperable compared with the SOAP API and normal function calls.
DC: It's scalable because of all the reasons I mentioned before: the cacheability of the basic data operations and their parallelisability through URI partitioning. updated - I meant data partitions not operation partitions!
Plus now we have parallelisability of the application of the business rules. There's nothing more parallelisable than a declarative system.
eA: If you say so! OK, perhaps you could elaborate on that; it sounds like a new point.
DC: It is: when you're leading the computer step-by-step through a process, you have to handle concurrency yourself. That's the 'How' of 'What not How'.
Conversely, when you simply declare 'What' the rules are, the computer is free to go off and do things as concurrently as the rules and the data separation allow.
eA: Mm. OK. Interoperability?
DC: It's interoperable again for the reasons I mentioned before. Firstly, the power of the URI; this scenario is a full player in the Web: you can share links to Items around and go fetch your Offers and Feedbacks with a simple HTTP GET. You can make things happen by POSTing to the relevant URI, given its content type.
There's also the expectation of standard Content-Types, sub-types and schemas in GET and POST, rather than custom eBay WSDLs and schemas, that I mentioned before.
eA: Like you said, you already mentioned these things. Anything to add now that we're doing business rules?
DC: Yes; when data is your interface and resource transformation your basic programming model, resource data types become part of your 'programming language'. As such, there is great benefit in sharing data types to allow such programming across multiple domain boundaries.
SOA, on the other hand, encourages inventing your own 'programming language' every time. It's a much more brittle model and mind-set.
You can't GET your RespondToBestOffer function call, but I can GET the ResponseToBestOffer! It's basically a more mashable approach to distributed programming.
(c) 2006-2007 Duncan Cragg
In Part 5: The Distributed Observer Pattern.
Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.
Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...
There is something about the Internet that nurtures open data, and something about computers that nurtures closed. It is often necessary, but often painful, to make the jump from local, closed data to global, open data. ...
There is something about the Internet that nurtures open data, and something about computers that nurtures closed. It is often necessary, but often painful, to make the jump from local, closed data to global, open data.
The Internet is all about data; open data and open data formats in particular. HTML, the fabric of the Web, links up vast amounts of open data created in an open data format. Email is sent either in the same HTML or (the most open of formats) simple text. Feed publishing shows that opening data can become an unstoppable flow. Indeed, Peer-to-Peer technology shows that opening data can become a torrent...
However, on our computers, data is often much less free. Take a closed format such as Microsoft Word: such a document can only properly be unlocked after paying for Microsoft Office. Take copyright data under DRM (Digital Restrictions Management): such music, etc., has to be unlocked by paying for the key.
Even with nominally open formats, there's a tendency to tie one format to one application. You still need that application to unlock the data. And even object-oriented programmers follow the closed data rule: data must be wrapped in a class interface.
Application user interfaces, class interfaces and service interfaces whose implementation processes mediate, control and restrict data and their formats are the bread-and-butter of the non-Internet computer domain. They work best on the more tightly-controllable computer hosts.
They Just Don't Mix
But the processes and the people that like to control data don't mix too well with the Internet. Here are some examples of the incongruity (or outright clash) that can result:
Now, these processes and people that like to control data obviously can't just avoid the Internet. So they'd be better off embracing the open data philosophy if they're going to do things here.
Embracing open data can take courage and open-mindedness. It means letting go of proprietary lock-in, copyright obsession, RPC, RMI and Service-Oriented Architectures. The lock-step imperative, procedural, workflow and object-oriented programming that works locally doesn't translate well onto the 'Net. The rules change.
Open Data Can Work
In fact, it is possible to make a living in this wild frontier, even when everyone can see, understand and copy your data! (For example, watch this blog for ways to make money on the Internet even when unlimited copying is allowed, based on originality or novelty backed up by a little crypto technology!)
I call the flip from closed data to open data the 'Imperative to Declarative Inversion'. It's the flip from 'How' to 'What' - from process-centric to data-centric programming.
In this upside-down world of What not How, the data format and its update formats are the public 'interface', and process animates things behind the scenes. Reading about REST will provide an excellent grounding in this way of thinking.
Showing the benefits of inverting (scalability, robustness, interoperability, value for money, speed and flexibility of development) and helping overcome the understandable fears of inverting are the main goals of this blog. In fact, I will also show how the Declarative approach works pretty well off-Net too - in the domain traditionally owned by process...
Declarative Architectures focus on the What, not the How, of programming. The How has dominated the field - perhaps 80% of programming is done in the traditional Imperative style, where we tell the computer How to do a task in explicit steps.
I'd like to show in this blog how Declarative Architectures and technologies are not just an interesting sideshow to the main Imperative attraction, but a complete and powerful programming alternative in their own right - indeed, one which has already dominated certain fields.
Imagine being able to simply express What we want the computer to do - to give it constraints and rules - then let it work out for itself How to achieve our goals.
I believe that saying What, not How, will become the dominant paradigm in programming. ...
Declarative Architectures focus on the What, not the How, of programming. The How has dominated the field - perhaps 80% of programming is done in the traditional Imperative style, where we tell the computer How to do a task in explicit steps.
I'd like to show in this blog how Declarative Architectures and technologies are not just an interesting sideshow to the main Imperative attraction, but a complete and powerful programming alternative in their own right - indeed, one which has already dominated certain fields.
Imagine being able to simply express What we want the computer to do - to give it constraints and rules - then let it work out for itself How to achieve our goals.
I believe that saying What, not How, will become the dominant paradigm in programming.
What?
Declarative Architectures are about data: documents, assertions, truths, rules, tables, structures, state, bits. Data storage, changes, constraints, transformation, sharing, querying, fetching, interaction, presentation, subscription, tagging.
In summary, we talk about stuff, changes to stuff and rules governing stuff. Not processes, actions, jobs or functions.
It's amazing how many interesting Declarative technologies are actually out there already - especially given that most programming is Imperative. They tend to go unnoticed; subservient to their Imperative masters.
Here are some examples from our daily work life (assuming you work in computing!):
What all these have in common is that they are purely data- and event-driven or oriented. Not a thread in sight (if used properly). No method calls. Just querying, fetching, subscribing, tagging, declaring truths and rules, making changes to stuff, etc.
On the 'Net, all the dominant technologies are actually Declarative, not Imperative:
Again, the same things are happening: querying, fetching, subscribing, tagging, declaring truths and rules, making changes to stuff.
Email (SMTP) is a great example of a Declarative Architecture, with its routing and transformation rules and self-addressed store-and-forward transmission. And, in fact, much the same applies to the rest of this list, where the Declarative aspects may include network querying and data events.
Imperative protocols such as RPC, RMI, CORBA, SOAP, etc. haven't had anywhere near the impact of their Declarative cousins.
Now, consider the 'Web 2.0' technologies:
This is a wonderful list of Declarative approaches (as long as we use REST APIs and talk about Resources, not SOAP ones talking about functions and jobs!).
So far, I've mentioned technologies that have attached themselves to the periphery of Imperative programming, where the actual behaviour is still driven procedurally, or How-like.
However, Declarative programming itself has a history at least as long as Imperative programming, and includes:
We can, if we choose, write 100% Declarative systems (even down to asynchronous or event-driven operating systems and hardware).
Wherefore?
In this blog, I'll be pointing out opportunities where we can benefit from the Declarative approach when it's ready.
Perhaps the biggest example of a Declarative Architecture that is ready to benefit us now is the Web. In contrast, the Imperative Service-Oriented Architecture model and so-called 'Web' Services are still weaning; still trying to get a life free of their huge standards consortia. REST, in its simplicity, ubiquity and straightforward usefulness is orders of magnitude more deployed than SOAP will likely ever be.
This blog is for discussing the common problems and solutions in Declarative technologies; seeing if ideas in one area can be used in another and seeing if we can distill their lessons and their approaches.
Ultimately, heading towards a unified, distributed Declarative programming technology landscape.
I hope you'll join me in my explorations into Declarative Architectures, in particular if you're interested in the currently most promising segment I mentioned: Web 2.0, and its application in Social Software.
Or the currently massive but little-mentioned segment: Peer-to-Peer, and its application to the disruption of, well you know, all those big technological and commercial hierarchies...