In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 9: Web Objects Ask, They Never Tell ...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 9: Web Objects Ask, They Never Tell
eBay Architect: You've pushed a lot of responsibility up to the business logic and away from the distribution technologies - away from the HTTP level.
As I understand it, you only want to use HTTP to implement a distributed Observer Pattern, where clients can become servers?
Duncan Cragg: Indeed. REST can be symmetric when used in integration outside of the Web. Further, these server-clients can have resources that do their own GET-ing, with POST callbacks, in order to Observe other resources. I've given many examples of this style.
eA: So how do you program such a distributed system at the business logic level?
DC: Well you need to be able to easily express that the state of a resource depends on the latest state - the intentions and declarations - of other resources it is Observing.
My vision is that you would express this business logic in a simple, powerful and expressive declarative language.
eA: Hmmm.. Just try getting all declarative on someone who just wants his share prices on time! No-one in business understands such abstract concepts.
DC: Oh no? The favourite programming tool of business is the spreadsheet!
And HTML is a form of declarative programming: imagine if you had to build your DOM from the top using Javascript! Or if you had to set all your styles in Javascript instead of using declarative CSS!
eA: Well I've seen some pretty ghastly examples of those two crafts!
And many failed attempts to allow non-programmers or business analysts to program directly.
DC: True, but that doesn't change the fact that most non-programmers can only think declaratively - they know What they want, but they haven't got much of a clue How to get it.
Business people talk business rules and business data.
eA: We just can't program with them!
DC: Well, perhaps we just haven't found the best discipline of pragmatic formalisms and methodology that will allow those rules to easily become programs.
It's a well-supported claim, though, that declarative programming is simpler, clearer and more productive than imperative.
eA: OK - so how would you actually define these business rules?
DC: You could use a rules engine, or a DSL engine, or even XSLT - if that works for you. There are many ways to transform XML.
eA: How would you do it?
DC: I would like to have an XML rewriting and templating system: "if this XPath or XML template matches Observed XML resource A, this one matches resource B and this one matches myself, rewrite this and that bit of myself with these bits from A and B".
Web Objects Ask, They Never Tell
eA: So you want to go around re-creating huge XML documents all the time?
DC: Who said anything about huge? This is another Web thing that doesn't necessarily apply to us in REST integration: we don't need the equivalent of the giant, monolithic HTML page. We can work in much smaller chunks.
I'd even just call them 'objects' rather than resources. Or 'Web objects', if you like.
They could be little XHTML carriers of Microformats.
eA: I thought we'd given up on distributing fine-grained objects back in the CORBA days?
DC: Ah, but this isn't trying to transparently distribute zillions of method calls in an RPC model.
This is about optimising state transfer. Only send what you need, cache where you can, push when something changes. Separate your data by rate of change, timeliness, cacheability.
eA: So these Web objects don't have any methods? Because that would be RPC when distributed?
DC: Exactly. In REST integration, the Web objects Ask, they never Tell! These Web objects are reactive: Asking for public state, not Telling each other what to do.
eA: You mean the opposite of Tell Don't Ask?
DC: Yes. As an object, you don't Tell another object How to do something, you Ask for What you want by either simply Observing its public state or by it Observing yours, then letting it decide How to evolve by itself. You then watch it and react or interact.
It's the "imperative to declarative inversion": everything is turned upside-down or inside-out when you distribute things this way!
eA: Oh yes, your "inevitable inversion" thing.
DC: Another indication of this inversion from the imperative object-oriented world to the declarative ROA world is how the derided 'train wrecks' of object-orientation now become the essential XPaths of the object Web.
You could say we have no methods, only 'getters', and XPath 'train wrecks' are encouraged!
eA: Doesn't sound too safe to me - it breaks encapsulation, doesn't it?
DC: Well, it's actually safe to dig around, since the data is held in shape by a stable, open, public schema. You're expected to go traversing the tree.
And you get excellent encapsulation since Web objects are total masters of their own destiny: they privately control the evolution of that public state.
You very much retain the value of 'What not How'; in fact, in a much better-defined way since it's fundamentally baked in to the programming model.
Hyperdata as the Engine of Object State
eA: But when you write 'train wrecks' you often end up jumping from object to object. How do your object Web XPaths do that, assuming they want to?
DC: Hyperdata of course! Links to links around the Web. Objects can have their opaque UUIDs or GUID object handles encoded into their URIs. Then objects can be wired up with XHTML links.
eA: So now your XPath transparently jumps these links?!
DC: Actually, yes! That would then allow us to dynamically break up data into more manageable chunks without breaking the XPaths that traverse it.
Also, with this approach, you still get to drill down to data like in transparent URI paths, but you now use XPaths that are properly a part of the content layer, jumping transparently over those opaque inter-object URIs.
eA: Would you use these 'jumping XPaths' in the rewrite rules you said you wanted?
DC: Of course. Either linear XPaths, or XPath-like XML tree templates on the left-hand side of a rewrite rule.
You'd start a template match on yourself, then continue on to match other objects by jumping over links, then on and on from object to object.
eA: So I suppose any such jump to another object means you then need to start Observing it, right?
DC: Exactly. And if hyperlinks are the only way you can find other objects to Observe, it brings us to the following:
Your object's next public state depends only on its current public state and the states of those objects that are visible to it through hyperlinks.
eA: Sounds a bit like the "Hypertext As The Engine Of Application State" constraint of REST.
DC: Exactly! Except now that we're doing REST symmetrically - now that clients can be servers, too - client Application State can have its own URIs!
eA: So you could re-phrase this as the even more intimidating: "Hyperdata As The Engine Of Application Resource State"!
DC: Well - how about just "Hyperdata As The Engine Of Object State"?!
eA: So do I have to wait for this link-jumping XML rewrite engine of yours to express my business logic, in order to get "Hyperdata As The Engine Of Object State"?
DC: No, of course not! Use a nice, dynamic, XML-talking language, like Scala, and follow the same principle, perhaps using a DSL.
eA: Can't you have objects that aren't entirely dependent on others? Like those that represent external state?
DC: Of course. That's normal Web stuff. It's probably best to keep these 'pure': to have either fully-interdependent objects driven by Hyperdata, or fully externally-driven ones.
Class, Extension, Instance and Behaviour
eA: Right, so we've got these little Web-mapped XHTML Microformat objects all linked up and watching each other in an Observer Pattern. This object's state depends on that linked object's state according to rewrite rules or a DSL.
So taking this mapping to objects one last step, I presume object 'class' maps onto an XML schema, XHTML Microformat specification or other content type?
DC: Yes. A public grammar in some form. Domain or business classes only, of course, not low-level classes.
eA: So, does each standard Web object class or type have a standard set of rules guiding its evolution or behaviour?
DC: Yes. If you see something and recognise its type, you can attempt to interact with it according to its public specification, and it should, but needn't, react.
The public specification can define the expected behaviour in the RFC language of MUST and SHOULD, like AtomPub. Or it can define it in rewrite rules!
eA: So then, how do I add my own business rules to a content type or 'class', if its behaviour is standardised and meant to be stable and predictable?
DC: There are two ways you can make use of a generic standard, and both are forms of layering or abstraction:
You can 'mash up' standard component instances with declarative configuration - just using them as they are for your domain.
Alternatively, you can extend or subclass their standard structures and behaviour to become themselves more domain-specific. For example, rules can be overridden or their right-hand sides merged.
Again, think of how AtomPub can be used in any domain that looks like time-ordered lists of content; and how you can extend AtomPub without breaking unextended clients or servers.
The REST Observer Pattern
eA: I think you need a memorable name for your symmetric-REST, "Hyperdata As The Engine Of Object State" architectural style!
DC: How about the "REST Observer Pattern"?
eA: But a Pattern has to be a retro-fit to existing behaviour, and "REST Observer" is quite new.
DC: I mean REST .. [DC does double bunnie ears] "Observer Pattern"!
eA: Have you got any worked examples of the REST Observer Pattern?
DC: Will have soon...
(c) 2006-2009 Duncan Cragg
Coming soon: Worked examples of the REST Observer Pattern.
Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.
Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...
Mobile Monday London met last night to discuss the Mobile Web and Widgets. It was an engaging and thought-provoking evening.
Your intrepid reporter was there and, in spite of the crashing of his sad, clunky old Windows Mobile Xperia X1, losing all his notes, he brings you this hot report from right out of his memory (somewhat steamed up by subsequent socialising, but reclarified by Google).
After that, I give an explanation of why I believe that Widgets are not the solution to what Mobile 2.0 needs... ...
Mobile Monday London met last night to discuss the Mobile Web and Widgets. It was an engaging and thought-provoking evening.
Your intrepid reporter was there and, in spite of the crashing of his sad, clunky old Windows Mobile Xperia X1, losing all his notes, he brings you this hot report from right out of his memory (somewhat steamed up by subsequent socialising, but reclarified by Google).
After that, I give an explanation of why I believe that Widgets are not the solution to what Mobile 2.0 needs...
GSMA ONE
First up was Kevin Smith from Vodafone to tell us about the GSMA ONE Web API (also via here). ONE means 'Open Network Enablers'. It seems quite similar to the Web21C initiative by BT, led by my good friend Paul Downey, but which was unfortunately set aside in favour of Ribbit.
You can send an SMS, get a user's location and access billing and connection info. All through a scary STREST interface.
Scary because it looks like you can send a text using a GET... The documentation here [PDF] does suggest using POST, but even so, the design fundamentally wants Resource-Orientation in place of Service Orientation:
The message appears to be tacked on to the URL as an argument. There are references to 'soapUI', endpoints, etc. An SMS message doesn't appear to have its own URL, just an internal message ID. There are 'exceptions thrown', all with a 400 code, even for internal platform and integration faults. And so-on - in classic STREST style!
Still, nothing that can't be fixed with a little help from a REST dude, or a read of "RESTful Web Services".
OMTP BONDI and Ikivo
Next, Nick Allot told us all about OMTP BONDI, which is an attempt to ameliorate fragmentation in the mobile Web and widget space - the one which allows access to parts of the phone not normally reached in a browser: PIM, SMS, camera, GPS, etc.
Their big thing is security: take widgets out of the safety of the browser and give them access to the phone via APIs, and you have a potential security nightmare.
Seems BONDI will let you do most Mobile 2.0 native-like applets, such as LBS, photo-sharing, etc. There's a reference implementation for sad, clunky old Windows Mobile, and both Opera and LiMo are jumping aboard, too.
BONDI aims to be W3C compatible where possible. Not HTML5, mind, but rather a works-now, fit-for-purpose solution to the fragmentation problem, that includes widgets and may later converge with HTML5. There's an excellent discussion about this on WAP Review.
This was followed by a talk by Samuel Sweet from Ikivo. It seems that, starting with their successful Samsung T*Omnia widget interface, Ikivo have been overcome with standards love, and are going BONDI as well as W3C compliant, especially with SVG as a rendering technology.
Firefox for Mobile
Christian Sejersen of Mozilla gave a good intro to what is currently called Fennec - Alpha 2, but will soon be renamed the more grown-up 'Firefox'. It's got the same code in it after all: and APIs such as camera and location that get added to the common codebase will be accessible via both mobile and desktop.
Looks very swishy and touchy and in sync with the times. Available now on Maemo (and Moblin) and soon on sad, clunky old Windows Mobile. Then later this year on Symbian. Codebase is C, so no Java or Javascript phones supported, of course. Or closed walled-garden proprietary locked-down phones like, um, the iPhone. It seems that the add-on community is already fixing up their plugins for this mobile version without even being prompted. No news on the Prism widget-alike system, to compete with BONDI or Opera. It's "still in the labs"...
Panel
Then the panel session started, with the best questions from the excellent host, Dan Appelquist of Vodafone, and some good ones from the audience.
There was much elaboration and clarification on the presentations (thus included above instead), and an interesting conversation around monetisation: how do widgets make money? Three suggestions: adverts, selling widgets on an app(let) store and micro-payments. Graham Thomas of T-Mobile appeared to be happy just to get more Internet traffic for now..
There was also confirmation from Graham that Web'n'Walk 4.0 will major on widgets, but he didn't say what flavour. More hot journalism uncovers this confirmation [PDF] (page 29) that it's still going to be tied in with Opera.
The outstanding revelation of the evening (sorry Ikivo!) came when someone told us that the Palm Pre's widget system is not only proprietary, but uses tables for layout! The horror! Lots of tutting and W3C-like smugness around the room.
Following Open W3C Standards can still break the Web
The goal of all this standard widget aspiration, apart from the obvious motivation of being able to compete with the iPhone, is to allow everyone to write widgets, and for those widgets to work on all our phones.
However, just because everyone does what the W3C thinks they should do doesn't mean you automatically get interoperability. Most importantly, doing what the W3C wants doesn't mean you won't break the Web!
The W3C, don't forget, also brings you 'Web' Services - those mis-named standards responsible for much Web-breaking, including inspiring a whole generation of Web-breakers with their STREST interfaces.
Interoperability on the Web is about state transfer, content types, URLs, hypertext. No amount of API and widget standardisation will give classic Web interoperability as long as they ignore these Web basics.
A widget is an applet is an application. It's a closed structure whose data interoperability has to be hard coded each time, and whose imperative Javascript naturally wants to make function calls back on the server - probably through HTTP.
And you run either this application or that, each having its own way of pushing or pulling data around, or even the same data presented in much the same way all over again.
In the Web you run one browser application and everything is mashed within - data interoperability.
The U-Web gives us the best of both worlds
Admittedly, the basic Web isn't good enough by itself when seeking an architecture for Mobile 2.0's essential personalisation, interactivity and usability.
But that doesn't mean that we should throw away the Web's hard-won advantages of interoperability and scalability, that we perhaps take too much for granted.
Enter the U-Web! The U-Web "puts the Web back into Web 2.0 and Mobile 2.0".
It takes the best of the Web's one-way static document publishing model, and extends it to a two-way dynamic data exchange model, while keeping the interoperability and scalability of the Web.
Instead of working on widget standards that break the Web, let's standardise a fully Web-compatible Mobile 2.0 architecture that delivers the same rich, personal functionality, but adds back the seamless mashability of ever-changing people and their ever-changing stuff. Oh, and promises scalability and rapid, easy development.
I've kicked off the project with the U-Web proposal - perhaps you'd like to jump in and help? Email me (see left bar) or leave a comment.
Since the announcement by IBM and Linden Lab that OpenSim can talk to Second Life, I've been thinking again about RESTful Virtual Reality.
I'm not the first, of course. Others have been motivated by the same goal: To bring the Web's scalability, linkability and interoperability into Virtual World platforms.
Ultimately, how to use the same techniques as the Web to link Virtual Worlds together into a single, massive 'Virtual Universe'.
Here's how I would architect the Universe Web... ...
Since the announcement by IBM and Linden Lab that OpenSim can talk to Second Life, I've been thinking again about RESTful Virtual Reality.
I'm not the first, of course. Others have been motivated by the same goal: To bring the Web's scalability, linkability and interoperability into Virtual World platforms.
Ultimately, how to use the same techniques as the Web to link Virtual Worlds together into a single, massive 'Virtual Universe'.
Here's how I would architect the Universe Web...
Entities
The Universe Web needs Entities. (I'm going to be capitalising Significant Things a lot here to help the Web comparison - hope you don't mind). There's no need to make the Entity list too complicated since we're not doing online games yet - just a world like Second Life.
Here's a list of different kinds of Entity:
State, Links and animation
Entities aren't exactly the Objects of Object-Oriented programming: they have lots of public State. In a Virtual World, you can directly experience their colour, position, orientation, physical relation to each other, etc., thanks to the local render engine.
The basic structure of the Universe Web is made up of Linked Entities. A tree is Linked to its ground which is Linked to a building which is Linked to its occupants which are Linked to their clothing. Kneebones may even be Linked to thighbones.
Entities are often animated. Actually, Places are generally static, and People generally animate themselves. So, when talking in general, we should look at the animation of Things, like trees, books, birds and clocks.
There are two ways an Entity's animation can be guided: by Laws and by Rules.
Laws
Springiness is a handy Law. For example, tree branches, pages in a book and birds' wings and legs can all have springiness. We can declare a mid-air location in a bird's State and a springiness in its legs, then if the bird lands on a branch, the springiness can trigger the bird and branch waving gently to a halt.
We could even allow a single bird to become part of a flock Entity, where the flock only knows how many birds it has and the coordinates of its centre. Each bird will then be animated by global flock Laws.
The book can have a State of open or closed. If bookness was elevated to a global Law, then a book deciding that its State was now simply closed could trigger the elaborate, springy closing of its pages with a nice 'shlump' sound at the end.
The clock's current time may be set by the clock Entity itself, then the Law of time take over in between the clock updating its own time directly.
A Law can be invoked over an Entity's animation by simply adding the Law's name and parameters to its State. Such Laws can be handled by the physics engine in the same way surface texture is handled by the render engine: States such as 'red' need global meaning, Law applications such as 'springy' need global meaning.
Rules
When a clock reaches its alarm time, it rings. When its wizard master approaches, a magic book may glow and open. Birds fly away from danger (if a fox is too close, fly away from it). In between danger, birds and foxes look for food (if a bird is close, run towards it).
These are Rules. Rules are driven by the Observation of one Entity by another. The fox Observes the bird, the book Observes the wizard. Rules apply to Entity types: fox Rules, bird Rules, book Rules, etc. Rules can guide the animation of an Entity by evolving its State.
State evolution depends on the current State plus the States of the Observed Entities: Rules make an Entity change from one State to the next by taking the Entity's own State, Observing the States of the Entities around that it cares about, then deciding what the next, new State should be. If an Entity changes State, those Entities Observing it may themselves change State according to their own Rules, and so-on.
Links are part of an Entity's State, and Links may therefore change on the application of Rules. In other words, the virtual world may restructure when a Rule is run.
An Entity may either Observe another Entity, or Link to it, or both. An Entity that Links to another needn't Observe it. Trees don't bother to Observe the ground they grow from. Conversely, an Entity that Observes another needn't Link to it. A fox would only Link to a bird if she was lucky enough to catch it. Meanwhile, it is true that there's probably a chain of Links to everything that's Observed.
Unlike Laws, Rules can't or shouldn't be second-guessed, so have to be run 'at source' or 'inside the Entity', then the new State distributed to both the Observing Entities and to all the render engines.
But render engines are actually Observing People Entities..
People are First Class
In Virtual Worlds, we have People avatars: People are first class Entities. People can Observe and can change State, but it's a real user doing the animation, not a set of Rules. Observation and State change by a Person is implemented through the render engine.
Although self-animated, People can still make use of Laws. For example Laws can be invoked to allow gestures to be easier to make and to look better, and to allow People to 'run forward' rather than putting their 'left foot forward, ..'. Avatar skills become global Laws which can be triggered by an avatar's State animated by a user.
The Universe Wide Web
The Universe Web I've just described is a lot like the World Wide Web:
A crucial difference between the Resource Web and the Entity Web is that, here it's another Entity doing the Observation. One Entity transfers its State to another.
It's as if the GET or POST has another Entity 'at the client end'. GET is like an Entity subscribing or Observing; POST is like an Entity publishing or notifying its State to another (an HTML form has a content type, too).
People are just examples of that Entity 'at the client end'. Instead of a user hidden behind a client browser, we have first-class People Entities doing 'GET's and 'POST's: Observing the Virtual World and acting (i.e., changing State) within it. Other examples are foxes and clocks.
Notation and Protocol
There are two open standards we need to have in progress while implementing the Universe Web: the notation and the protocol (our HTML/URI and HTTP).
Firstly, we have to define the notation for Entities and their Rules. Both the State of Entities and the Rules that animate them will be written in the same notation: a Rule talks directly about State. To me, basing that on JSON (rather than XML) is a clear choice. Add inter-JSON Entity Links (essentially UUIDs), then work out a simple JSON Rule language. We will then need to define JSON 'schemas' for the kinds of Entity we would model. These can be written in the Rule language itself.
Secondly, to define the protocol for Entities to Observe or subscribe to one another. It will be a 'symmetric, asynchronous HTTP' - a Peer-to-Peer, Publish-Subscribe protocol. UDP and Multicast stand out for this kind of interaction. A packet will subscribe, a packet will return current State, a subsequent packet will multicast State updates. We can re-use all the HTTP cache headers, only now we can push updates into caches and into render programs. Packets will inevitably end up tunnelled through TCP and HTTP, mind...
This lighter notation, which also breaks the data up into smaller pieces, and this multicast pushing of State change into caches, along with P2P-style operation, will mean the Universe Web scales even better than its parent.
Also, programming declaratively using Rules is both highly productive and further cleanly scalable through parallel processing.
Of course, it will be pretty trivial to allow inter-Web cross-over such as referring to PNG texture images via URL and pulling Universe Web Entities into a Javascript application that subscribes to them via Comet.
Building the Universe Web
So that's my Universe Web. It's basically the same as the World Wide Web and thus as scalable, linkable and interoperable. It comes with added symmetry: Entities, including People as avatars, exchange State in each direction.
It's a two-way data Web, not a one-way document Web.
I've started work on the Universe Web: the notation, the protocol and the implementation (in C, for portability across platforms, including mobile). Contact me if you want to help!
Last night's Google London Open Source Jam (also here) was on the subject of the 'Web' (didn't they invent that? Oh no, that was Microsoft).
This event has been getting better and better each time I've attended. There were some very interesting lightning talks held together with a tight structure and plenty of chance to chat, drink cold Leffe and eat cold pizza. And nick [transatlantic translation: 'steal'] the Green & Black's chocolate.
An ideal Micro Conference... ...
Last night's Google London Open Source Jam (also here) was on the subject of the 'Web' (didn't they invent that? Oh no, that was Microsoft).
This event has been getting better and better each time I've attended. There were some very interesting lightning talks held together with a tight structure and plenty of chance to chat, drink cold Leffe and eat cold pizza. And nick [transatlantic translation: 'steal'] the Green & Black's chocolate.
An ideal Micro Conference...
I arrived late (it starts at 6pm) and spent some time catching up with ex-Thoughtworks colleagues, so I missed Dion "Ajaxian" Almaer's Google Gears slideset from FOWA. Go there now and check it out.
Thus the first talk I saw was a nifty piece of widgetry by Steven Goodwin called WARP. In WARP, interacting with a page of 'applets' changed the URL to encode those applets' current state. If you link to the current page, it will always show that state. Very long URLs, you can imagine. None of that fancy Ajax stuff. RESTful, dare I say. Nice API server-side for unpacking your applet params.
A trip to the lavatories [transatlantic translation: 'restroom'/'bathroom'] revealed that they are, indeed, doing that Testing in the Toilet project in Google. It works, too! I learned something. Other intelligence on Google's Inner Workings include confirmation of the beanbags and of the high quality, free grub to which I have already alluded.
A nice bloke from Yahoo! (Tom Hughes-Croucher: another spy?) came along to sell his idea that, in the collaborative world of open-minded hackers, we who run websites could help each other with our 404s. If I get a 404, I use the referrer link to tell you, via some RESTful POST, that your link to me is bust (assuming I don't intend to fix it myself).
I think the world is a little more selfish, so you need to decide who hurts more - the site who sends their visitors to a dead-end, or the site delivering that dead-end to a new visitor. I suspect the latter, by a small margin, as it's not exactly a nice welcome. So it's up to them to let the new visitor down more gently, and to notify the publisher of the broken link with little or no cost to them. For example, a really sociable 404-ing site could just redirect the hapless visitor back to the referring page, adding '?broken=links' to the URL - hopefully to be picked up by log scanning scripts at the referring site.
Next up, yours truly taking yet another chance to promote his excellent Micro Web thingy. Couple of people asked about it afterwards - including that nice chap from Yahoo! Also, a smart - and nice - chap called Toby (this one?) got me into a deep discussion on imperative vs. event-driven vs. state-driven programming. He was apparently an old-timer like me, as he was able to engage in dewy-eyed Functional Programming recollections. I managed to give out about four full colour printouts about the Micro Web, and to collect some good calling cards.
However, Joe Walnes, even a pint down in the pub afterwards, still refused to sign up for Micro Web duties. This in spite of over three years of intensive lobbying, including eight months of me working Trojan-horse-like in his kitchen, on The 2005 Implementation.
Another ex-Thoughtworks colleague, Simon Stewart took yet another chance to promote his promising Webdriver thingy. And a very interesting project it is becoming. Still needs more work - on IE support, etc - but I'll probably be using it in my new job at the Financial Times.
Another ex-Thoughtworks colleague, Chris Matts took a chance to promote his and Andy Pols' interesting new Dream Machine thingy. Perhaps a bit like Cambrian House - you put your dreams and ideas into it and people expand on them. Chris is a natural on-stage - and even used the age-old trick of promising lots of money for no effort, to get our attention at the start.
All I could come up with for the Micro Web was 'Cheaper, Wider, Faster'...
Updated: added reference to Dion Almaer, details about WARP, swapped in the picture of TWers that I was waiting for and fixed a minor blunder thanks to that ever-sharp ThoughtWorker, Dan Bodart..
Web 2.0's definition includes seeing the Web as an application platform. Which means it is in competition with Java and .Net, and with SOA, for both local and widely distributed applications.
If the Web is going to be a platform, the skills you need to learn to program it are the core Web 2.0 technologies such as Ajax, JSON, Atom, Microformats and OpenID.
And Ruby. This language, that's capturing the hearts of many Web 2.0 programmers, is ideal for easing the transition from the Java and .Net platforms to the Web platform, as I will show.
Even if you're part of a big company that is generally immune to the latest trends, the marriage of Ruby and the Web-as-platform may be something to prepare for. It could even displace your SOA agenda... ...
Web 2.0's definition includes seeing the Web as an application platform. Which means it is in competition with Java and .Net, and with SOA, for both local and widely distributed applications.
If the Web is going to be a platform, the skills you need to learn to program it are the core Web 2.0 technologies such as Ajax, JSON, Atom, Microformats and OpenID.
And Ruby. This language, that's capturing the hearts of many Web 2.0 programmers, is ideal for easing the transition from the Java and .Net platforms to the Web platform, as I will show.
Even if you're part of a big company that is generally immune to the latest trends, the marriage of Ruby and the Web-as-platform may be something to prepare for. It could even displace your SOA agenda...
Few would disagree that the Ruby language is riding the wave generated by Ruby-on-Rails. In turn, Rails is riding the Web 2.0 wave, coming as it does from underpinning the very Web 2.0 37signals product suite.
Rails and Ruby have tapped into the tech Zeitgeist of friendly, simple and powerful. The speed with which the Ruby and Rails communities have delivered the key components of Web 2.0 is matched by the speed at which Ruby and Rails books are leaving the shelves.
What is the ideal platform of Web 2.0? Will it be Rails and Ruby? Will Ruby ride the Web 2.0 wave into the mainstream in the same way Java rode the Web 1.0 wave?
Well, here's the problem with that question: Web 2.0 is supposed to be primarily about the Web itself as the platform, as explained first by Tim O'Reilly and then by a thousand Web 2.0 vendors and industry watchers after him.
Web-as-platform is not just vendor hype or pundit hand-waving. Let's think about what O'Reilly meant by that.
Web as Platform
Web 2.0 is about making the Web more interactive, and thus able to support applications where Java and .Net would once have been considered the sole delivery platforms.
The fact that the technologies of the Web can be turned to this use is a shift with far-reaching implications.
Broadly, the shift we are seeing is from the one-way, static document delivery of Web 1.0 towards the two-way, dynamic data exchange of Web 2.0.
This fundamental repurposing is delivering more complex, interactive applications that work inside our browsers and which fully leverage the benefits of online operation.
Web 2.0 is bringing the user and their stuff into the very Web that they hitherto only passively consumed. This network-enablement of the user in turn enables their social networking and their shared creativity and self-expression.
Web 2.0 has tapped into a deep human need - a fact reflected in the vast traffic volumes and correspondingly vast valuations of Web 2.0 startups that we're currently seeing.
But Web 2.0 is not just for the startups: Enterprise Web 2.0 is coming! The bigco.com site is going to be looking a little, well, static and lifeless when compared to the new sites that are springing up everywhere, and that most of BigCo's employees are using. Further, BigCo can gain huge benefits from Web 2.0 approaches empowering and connecting those employees on the Intranet. And that Intranet is an ideal platform for deploying company-wide, interactive applications.
This shift in the Web to two-way dynamic data is being powered by a set of technologies that a Web platform programmer is going to have to learn.
Web 2.0 Platform Technologies
Anything that claims to be an application platform must support data. Web 2.0 is above all the data Web. Web 2.0 is about semantics, not free text and font sizes. Hence, it inevitably starts with data-oriented formats such as XHTML, YAML and JSON. In Web 2.0 more than ever, we talk about data not documents and about separating data from its presentation. CSS is big in Web 2.0, for good reason (not just for gradient fills). Inside the page of a self-respecting Web 2.0 application, you'll often find Microformats - again, semantics in the page: publishing concise data of widely-understood standard formats. Some of those Microformats may be tags, and in Web 2.0 the simplest and most powerful semantics are those little pivot points in Webspace.
Again, if you're going to be a general purpose platform, you need to be able to fetch, update, notify and display that data. Web 2.0 integration usually happens via JSON data structures and REST interfaces (some of which, especially those based on AtomPub, are true REST). Following on from the data-like pages we serve to browsers, come the data-like feeds we publish to feed readers and to other applications. After feeds, the core technology that gives Web 2.0 its dynamism and interactivity is Ajax and DHTML, and increasingly Comet (server push to the browser). The core technology that gives Web 2.0 its users is increasingly OpenID.
All of the above are open technologies. You can do Web 2.0 without proprietary technologies, just like Web 1.0. Indeed, keeping to the principles that made the Web successful is also essential to the success of Web 2.0. The Web platform is the first application platform that has to consider scalability and interoperability, and will ignore them at its cost. I have written before about open data, use of standard data formats and using REST properly to avoid creating unscalable, walled-garden sites. You don't need Flash or SilverLight, you don't need vast amounts of custom Javascript, you don't need function calls tying you to your servers.
Programming the Web 2.0 Platform
So, we've got the dynamic data that you'd expect of a would-be platform. But how to drive changes in those data? How do we program the Web platform to animate all this data?
All Rails programmers will know the above technology list; it comes with the territory. The Web 2.0 Platform can be very succesfully powered by Rails and Ruby. Ruby and Rails make Web 2.0 applications simple and quick to program, addressing many of the needs of simple Web 2.0 applications out of the box. There's little doubt that Ruby and Rails will have a secure future riding the Web 2.0 wave.
However, for many Web 2.0 applications, programming may not even be necessary, at least not in the procedural or imperative style programmers expect.
Look back to the early 90's: 'Web 1.0' made a whole class of applications easy to write without programming: applications for navigating information. You just wrote in HTML, declaratively.
Now look back at the long path of evolution of Java, through J2EE, Spring, AOP, IoC, Domain Driven Design, POJOs. All trying to achieve the simple goal of 'remove all that MVC and persistence stuff and let us concentrate on business or domain objects'. But they never quite seemed to get it right.
But then Rails comes along, and has succeeded by simple virtue of concentrating on easy manipulation of the 'Intel Inside' of Web 2.0 - data.
It's reminiscent of the 'Naked Objects' approach to application building with minimal programming (just business or domain code in POJOs that expose state into the GUI and are transparently persisted). The Streamlined project takes Rails even further down this path. Rails' nearest competitor, Django, has an admin interface that works in a similar way, automatically generating edit pages based on the data model.
Web 2.0 is about data, about semantics. Web 2.0 is inherently declarative. So Web 2.0 applications can be written declaratively - Web 2.0 mashups can be just wired together and their data animated by business rules. A bit like programming spreadsheets.
Teqlo, Coghead, Pipes, DabbleDB, LongJump, Popfly, AppExchange and Wyaworks are all examples of the different ways to program the new Web 2.0 platform without imperative code.
That's what we mean by Web-as-platform - not only is the underlying programming language irrelevant, it will often not even be needed, certainly for simple data manipulation applications and for many simple mashups. Being RESTful gives you a massive head start in this, of course.
While Rails is already in the game with its innate understanding of Web 2.0 techniques and philosophies, Ruby itself has a huge amount to offer the would-be declarative programmer, who is making the transition to this new Web platform from their traditional Java or .Net platform. In particular, it is easy to write your domain logic in a declarative style in Ruby: they call them 'DSLs' these days, but the idea is the same in most examples I've seen.
Web 2.0 - The Web Redux
Now, if you've been following this blog, you'll know I have a few opinions on Declarative Web 2.0 and on patterns for programming REST. Essentially I argue that, if you want to play in the Web 2.0 platform game, you don't want to be writing screeds of Javascript functions that call more functions on your servers.
I recently presented some ideas along these lines at the WWW2007 conference, entitled 'The Micro Web: putting the Web back into Web 2.0', where I also showed a demo written in Python.
This approach combines my Distributed Observer Pattern with Comet push to enable highly dynamic Web 2.0 applications to be coded RESTfully and declaratively, with zero Javascript. The Distributed Observer Pattern offers a clean programming model for animating the Web 2.0 dynamic-data technology set I described above.
I believe the Observer Pattern is core to the way we'll be programming when the Web 2.0 Platform hits mainstream. It enables the kind of event- and rule-driven programming that matches the characteristics of the Web 2.0 dynamic data platform. As a further killer benefit, it also directly addresses the optimal utilisation of multicore processors.
I am currently porting my Python implementation of this approach to Ruby, in the Redux project on Rubyforge. Redux stands for 'Ruby Event-Driven Update Exchange'. It uses the highly scalable EventMachine epoll-based event loop to power its event-driven architecture. This will be essential when Redux is asked to scale up a Comet-based application.
Like Rails, Redux will be a Web (2.0) application framework, but unlike Rails, it puts the Observer Pattern and event- and rule-driven programming at its core.
Redux's headline is 'Web 2.0 in-a-box' or 'Naked Objects on the Web'.
Conclusion
If you're in BigCo, and are responsible for setting BigCo's technical strategy, then train your Java devs up on Web 2.0 core technologies such as Ajax, JSON, Atom, Microformats and OpenID.
And fire up their enthusiasm by tapping into Ruby (perhaps via JRuby) on your way to the Web 2.0 platform.
Learn patterns for mashing and integrating. Learn about REST and event- and rule-driven programming, including declarative DSLs.
When this Web platform hits BigCo, you will probably find that its REST or ROA style make your SOA integration strategy look rather complex and unweildy.
Check out the Distributed Observer Pattern, and download Redux when it's done (I'll let you know if you subscribe here!).
In 2007 and beyond, its the Web itself that's the platform, not Java or .Net. But if you want to get there via a language-based platform, Ruby could be the best way to transition to it.
Note: Everything I said about Ruby and Rails applies equally in technical terms to Python and Django, but regardless of the significant benefits of the latter, Ruby and Rails have the Web 2.0 market and mindshare. I'll probably switch this blog from Django to Redux sometime this year..
(c) 2007 Duncan Cragg
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 4: Inter-Enterprise REST Integration ...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 4: Inter-Enterprise REST Integration
Duncan Cragg: OK - I've demonstrated how you can replace imperative, function-call API-driving with a clean, declarative, RESTful interaction, driven by simple business rules.
We had servers run by eBay and clients run by the public, in the same way your SOAP API is used.
eBay Architect: Ah: that's something SOA has that REST doesn't!
DC: What? What's that?
eA: Services are all about Enterprise Integration: about servers talking to servers. In REST you're all about clients talking to servers. The Web is essentially only browser clients talking to Web servers. With Web Services, you can do more serious Enterprise Integration.
DC: You never give up do you? So you want 'serious' integration. Is that within or between enterprises?
eA: Let's say between.
DC: Fine. We'll use the same example as before: it's just a variation on the Patterns used.
We can standardise a more general version of the eBay schemas for Items, Offers, ResponseToBestOffers and so on. Anyone can put their own Items, Offers, etc. up on their own servers, or on some public auction service site. Everyone can do auctions with eBay and with anyone else who decides to set up.
Even, say, a new Google auction site: let's call it 'gBay'!
eA: Ha! OK, let's go through this slowly: you have eBay and 'gBay' sites, with sets of users on each. Now Ernie wants to sell his old laptop on eBay, so creates a new Item for it. Gordon is registered to gBay and needs a cheap laptop.
DC: Great - well the first thing is search. As an interoperable site, gBay offers a broad search across both gBay sale Items and eBay ones - cached and indexed internally. The gBay search database would be filled by crawling eBay URIs and even by running queries on eBay.
eA: Mm. Have to check the T's & C's...
DC: So Gordon on gBay finds Ernie's laptop on eBay. The presentation of this eBay sale item will be given the gBay style, but calling out directly to the eBay data and images.
eA: OK, now let's say Gordon decides to make an offer.
DC: So an Offer resource is created on gBay referring to the laptop on eBay. Then through a notification, the Item on eBay is alerted to this Offer.
eA: What's notified, to where?
DC: There's a number of possible patterns. Before, we had the pattern of POSTing a resource to a server that then creates the GETable version.
However, now gBay is hosting the Offer, so the internal mechanisms for notification are no longer available.
So gBay could suggest an update through APP or a simpler POST to a collection of Offer entries within the eBay Item to point to this, now remote, Offer.
Perhaps the gBay Offer can simply be POSTed wholesale to the eBay Item.
Or just a link to it.
Or eBay may poll, read a feed or search gBay for new Offer URIs, putting them into Offer lists as they come up.
An unusual approach (thanks to Joe Gregorio) would be for gBay to GET the eBay Item, with the Offer marked in a Referer: header.
eA: Plenty of patterns to choose from. So there are some Offers on eBay, some on gBay. The Item lists its Offers in a rank as before, as they appear through this notification.
Now, let's say Ernie wants to accept Gordon's Offer on gBay.
DC: OK, assuming he can see the Offers the same regardless of host, he just chooses Gordon's Offer on the offer listing for his Item and accepts it.
eA: So we need to create a ResponseToBestOffer on eBay.
DC: Yes. Now the patterns are reversed, because eBay needs to notify gBay this time - of its ResponseToBestOffer.
Pub-Sub and Observer Pattern
DC: Again, it can do this by POSTing the ResponseToBestOffer to each Offer on gBay in turn, or can POST the actual Item itself to each Offer, where the Item has a link to the ResponseToBestOffer.
That would implement a logical subscription to the Item from each of the Offers on it.
eA: It sounds to me like POSTing several times to implement this pub-sub pattern is physically inefficient, even if it's logically correct. Especially when it's the same information repeated from eBay to gBay servers.
DC: Yes, indeed: a single notification to gBay would be better, letting gBay handle the propagation of subscription responses. This would in effect treat gBay as a proxy cache, and the notification as a cache invalidation event on gBay's copy of the eBay Item.
eA: What URI on gBay would you POST this eBay Item to?
DC: Something like http://gbay.com/ebay.com/item/4243 - to a
copy of itself. You could also GET this cached copy if you wanted.
eA: OK, what next?
DC: In gBay the losing Offers get updated on receipt of this ResponseToBestOffer state. Gordon's Offer gets set to 'won'. In eBay, all the losing Offers are updated to 'lost'. The laptop Item gets marked 'sold', with a link to the ResponseToBestOffer, which links to the Offer that won.
It is possible to implement this internally in eBay (and that pub-sub cache invalidation propagation in gBay) using the Observer Pattern. and an event-driven server.
eA: Makes sense - you mean something like SEDA?
DC: Yep.
So the Offers all subscribe to the Item to watch for its status switching to 'sold' and to see if they won. Conversely, the Item can subscribe to the Offers: maybe the Offers could change or be withdrawn, and the Item needs to keep itself updated accordingly.
eA: Wow - symmetric subscription - the two-way Observer Pattern!
OK, what next?
DC: The eBay laptop Item resource will be further updated by its owner with paid, shipped, refunded, etc., as it currently is within eBay.
eA: Hold on, you're mixing patterns: you had the Observer Pattern on the Item just now: the Item observes the Offers. The Offers' state can be POSTed to the Item, whose own state may then change according to its rules.
But you then mix patterns by allowing a POST directly to the Item from the Item's owner, to update a couple of fields.
In one, the Item chooses what its state will be according to the state of its peers, and in the other, it's told, not according to a peer state, but some POST content type.
That doesn't seem neat or symmetric.
DC: It's true that these interaction styles differ: the Observer Pattern or pub-sub approach is peer-to-peer (resource-to-resource as equals watching each other); and in this scenario it's also server-to-server.
The direct edit request is more a client-server pattern, where the server resource - the Item - is considered under the control of a client.
However, the Item is always in control of its own state, and can even ignore a request by its owner if that request doesn't match its internal integrity rules.
The Item supporting both styles at the same time is absolutely fine.
Actually, you could see these two styles as aspects of the same peer-to-peer pattern: introduce a resource in the client that holds edit requests, to which the Item subscribes. It all ends up being much the same.
Transactions, Trust
eA: Right, now what if you have a race, where the ResponseToBestOffer is created at the same time as an Offer is changed or withdrawn?
Don't you need some kind of two-phase commit or distributed transaction logic?
DC: Of course not. It's the same as in the real world: as long as it all settles in the end and the rules are followed. The ResponseToBestOffer cites what state of the Offer it is accepting. If that changes for any reason, the ResponseToBestOffer is void.
It's about state and state consistency in REST, as opposed to the SOA style of maintaining total control at all times.
There will be temporary states that trigger the rules and that need to be resolved. That's the programming and distribution model. Tolerance of transient states is what makes this model so robust.
eA: Surely there are some legal and contract issues? How is this exchange legally binding?
DC: You can digitally sign the Item, Offer and ResponseToBestOffer resources, and each side needs to keep records of the history. Then it's down to agreements between eBay and gBay and the local laws in force.
eA: What about buyer and seller ratings and feedback?
DC: Ernie in eBay and Gordon in gBay can happily publish feedback about each other, and Ernie will be able to see Gordon's rating via eBay's interface, or directly on gBay.
As for aggregated ratings from several buyer/seller interactions: a person's rating is a function of the ratings of all those they have dealt with. These ratings can be fetched by GET from remote sites, and combined with internally-held ratings, depending on the trust of one site over another site's ratings.
eA: So how do we trust these ratings across sites?
DC: We have to trust eBay that it trusts gBay. This is one of the basics of distributed systems. In a monolithic system you have a single trust domain: all parts can trust each other.
Split the application up across multiple trust domains and you need authentication and crypto. You can't get way from needing peer trust structures built up explicitly through crypto, agreement and contract and/or implicitly through past successful experience.
eA: Can you be more specific?
DC: Normally, a GET for a resource or a POST of some data comes with a header identifying the GETer or POSTer. The resource can also be signed by a user on a site or by the site itself as a proxy.
Or, if you have an agreement with the site, you just need to use https to ensure you've got a secure connection with that site, then needn't have individual signatures.
eA: Where's the Single Sign On and Identity in all this? We've got users working across multiple sites.
DC: Well, gBay is the holder of the Gordon identity or persona - and it manages his world view. Gordon on gBay needs his identity to mean something on eBay, but we don't want him to have to create an account on eBay or to have to tell gBay his eBay login details to work on both sites. So he expects gBay and eBay to have come to some agreements about technology and policy.
In REST, we don't have sessions and logins - we have identity, which implies asymmetric (private/public key) crypto for signatures and security. We have a number of tools available to us, including OpenID and https, as well as resource signing.
eA: Here's a question for you: how would you manage a single shopping trolley for Gordon on gBay, containing and allowing payment for eBay goods?
DC: ShoppingTrolley resource, links to eBay and gBay items. At checkout, smaller eBay-Items-only ShoppingTrolley resource POSTed to eBay along with CreditCard resource (again, you can sign the ShoppingTrolley and encrypt the data).
eA: So, as eBay, why should we integrate the seller ratings of someone on gBay? Or get gBay's for-sale items coming up in our searches? Or accept Offers and ShoppingTrolleys from gBay? We don't control or trust them, and don't want to send traffic or business over to them.
DC: Fair enough, for now. I'm only describing what's technically possible. Like I said before, you may revisit your stance on interoperability and mutual agreements one day soon.
Also, what if your business decides this year to set up a commercial partnership with another similar business and the managers come to you asking how it's all going to work together internally?
You'll find having good REST interoperability already in place a huge asset for internal integration! You'll also find that an interop-friendly approach makes developing internal 'mashups' much easier.
Better Than SOA
eA: I still can't see why all this is better than our SOAP approach, though: it just seems like the same things are happening at the end of the day - that it's only a change of perspective.
DC: Well, a minute ago, you were challenging using REST for anything other than simple data manipulation. Now I've shown you the power of a REST approach can be easily extended to a clean, simple, scalable, interoperable, general, declarative programming model. And you're still not satisfied!
eA: Ha! OK. So tell me why this programming model is so scalable and interoperable compared with the SOAP API and normal function calls.
DC: It's scalable because of all the reasons I mentioned before: the cacheability of the basic data operations and their parallelisability through URI partitioning. updated - I meant data partitions not operation partitions!
Plus now we have parallelisability of the application of the business rules. There's nothing more parallelisable than a declarative system.
eA: If you say so! OK, perhaps you could elaborate on that; it sounds like a new point.
DC: It is: when you're leading the computer step-by-step through a process, you have to handle concurrency yourself. That's the 'How' of 'What not How'.
Conversely, when you simply declare 'What' the rules are, the computer is free to go off and do things as concurrently as the rules and the data separation allow.
eA: Mm. OK. Interoperability?
DC: It's interoperable again for the reasons I mentioned before. Firstly, the power of the URI; this scenario is a full player in the Web: you can share links to Items around and go fetch your Offers and Feedbacks with a simple HTTP GET. You can make things happen by POSTing to the relevant URI, given its content type.
There's also the expectation of standard Content-Types, sub-types and schemas in GET and POST, rather than custom eBay WSDLs and schemas, that I mentioned before.
eA: Like you said, you already mentioned these things. Anything to add now that we're doing business rules?
DC: Yes; when data is your interface and resource transformation your basic programming model, resource data types become part of your 'programming language'. As such, there is great benefit in sharing data types to allow such programming across multiple domain boundaries.
SOA, on the other hand, encourages inventing your own 'programming language' every time. It's a much more brittle model and mind-set.
You can't GET your RespondToBestOffer function call, but I can GET the ResponseToBestOffer! It's basically a more mashable approach to distributed programming.
(c) 2006-2007 Duncan Cragg
In Part 5: The Distributed Observer Pattern.
Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.
Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 3: Business Functions ...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 3: Business Functions
Duncan Cragg: So, where did we get to? Oh yes: the REST recipe for scalable and interoperable distributed systems!
We can read data at a URI with GET. We will usually understand that data when we get it, because it has a standard content type at a number of layers - perhaps from character set up to Microformat via XML and XHTML.
We can cache the data if the response allows it.
Then we can POST back our own content to the same URI - if we believe that this resource is interested and that it will do something we want.
Finally, the content we GET has more URIs that we can use in the same way.
eBay Architect: Simple enough.
DC: In fact, these are essentially the constraints of REST as formulated by Roy Fielding. Note that Fielding hardly mentions actual verbs in his thesis: he, in fact, gives special mention to GET and PUT, which is consistent with the basic 'state transfer' concept - one verb for each direction. The Web (and I) prefer POST to PUT!
Although we expect the resource to change in some way - perhaps change, create more resources or delete resources, the resource can actually do what it likes when POSTed to, as long as it conforms to its declared standard.
Don't forget that REST doesn't require direct resource editing as either a necessary or sufficient mode of interaction, and that in general a resource can do what it likes on receipt of incoming content, as long as both sides have agreed in advance.
eA: How do you know what a resource will do, then?
DC: The standard to which the POST target conforms will declare the type of the POSTed content and set the POSTer's expectations of the consequence of such a POST.
HTML, of course, is a bit vague and low-level about what you can POST - it's either name/value pairs or files, with no promise about what will happen afterwards. This is because a human is directly involved in the interaction. In contrast, we're discussing the general REST integration case here, where higher-level, more complex, machine-generated POST types are expected.
eA: This all sounds fine to me. But, like I said, our API has more complex business functions in it that go beyond simple data read and write.
DC: What sort of functions were you referring to?
eA: There's many of them. Functions like PlaceOffer, RespondToBestOffer, CompleteSale, SendInvoice.
These are real functions, not getting, setting or adding data.
DC: Indeed, and as such form an excellent exercise to demonstrate the general power of REST!
eA: Show me how. What's the trick?
DC: The trick is simply to identify your resources, then to discover or define how their state depends on related resources and POSTed content, using transformation rules, or 'animation' code.
Indeed, if the state of a resource depends on the state of other resources - or POSTed data - via a transformation, then you have a complete, general programming model.
eA: You mean my 'real functions'?
DC: Yes - a resource can do any 'real functions' just by watching resource changes via GET or receiving resource data via POST on its URI and then mechanically transforming itself according to its internal rules - as defined by standard, convention or agreement.
Data operations are enough to enable much more than just data operations, as long as you have the internal transformation rules in place that animate the data in the face of current state.
eA: What does this mean to an eBay Architect? How does that help with placing offers and responding to offers? How do I send an invoice?
A REST eBay Transaction
DC: OK, first identify your resources: let's look at what resources are implied by your list. You mentioned PlaceOffer, RespondToBestOffer, CompleteSale, SendInvoice.
So, it looks like we have this list of resource types: User, Item, Offer, ResponseToBestOffer, Feedback, Invoice.
Notice how, when we go from function (RespondToBestOffer!) to resource (ResponseToBestOffer), we change the names from an imperative instruction style to a declarative state style. This is a crucial change in mind-set.
eA: Um - the CompleteSale function isn't just for adding feedback, it allows paid and shipped to be set on an Item.
DC: Exactly: paid and shipped status are attributes of the actual Item being sold; the Item would be updated directly to complete a sale. However, it makes sense to put Feedback in its own separate resource.
Admittedly, this is just my initial analysis. I'm sure it could get more complex on closer inspection; I'm just trying to be illustrative. And I don't necessarily understand the whole eBay auction process, so please forgive any slip-ups there!
eA: OK. Now you said that we should define how these resources depend on each other and on POSTed content, using transformation rules. Take me through that.
DC: Sure. There are two API Users: Sam the Seller and Bill the Buyer.
Sam POSTs his Item resource, and eBay's servers create a linkable copy server-side, returning the URI in the 'Location:' header of the POST response.
Note that a server doesn't always need to create a linkable copy of what's POSTed to it, but that's the pattern I'll use in these examples.
Let's say Sam the Seller is posting a Fixed Price Item with Best Offers switched on - since you've mentioned RespondToBestOffer. Here, people can suggest a near, lower offer and a bit of negotiating can take place. Not the usual eBay auction but a simpler interaction that is perhaps more generally-applicable in e-commerce.
eA: Let Bill do a PlaceOffer, then: any transformations now?
DC: Yes. A number of people, including Bill, POST Offers linking to the Item and see these Offers created on the server.
eA: So how does Sam get to know about the Offers on his Item?
DC: Ah - that's your first transformation! Each time an Offer is POSTed referring to an Item, the target Item itself is updated. In particular, the Item has a sub-container with a list of current Offers on it, ordered by value.
At any time, anyone can fetch their own resources, and some owned by others, with a GET. Sam can GET his Item and its list of current Offers; Bill can also see the Item, and GET his Offer. Items and Offers thus refer to one another by internal links that everyone can understand because they understand URIs.
eA: Is this transformation really that big a deal?
DC: Not hugely. This transformation rule encapsulates the commonly-occurring concept in commerce of a range of values of multiple bids on some sale item - such as found in real estate and in financial markets.
A further business transformation rule may perhaps have Offers running a 'best-offer' status flag that is kept consistent with that Offer's relative state.
As you can see, and as is often the case in REST and declarative programming generally, these business rules are very simple.
eA: So, Sam sees a number of Offers. Let's say Bill has the highest and Sam wants to take it.
DC: Sam then POSTs a ResponseToBestOffer, accepting, and linking to, Bill's Offer. Again, Sam gets the server-side URI of his ResponseToBestOffer for future reference.
eA: This is where I'd call the RespondToBestOffer function.
DC: Yep. Now, you have this state: an Item from Sam and a list of Offers on that Item, with a Best Offer from Bill. Onto that overall state arrives Sam's ResponseToBestOffer referring to and accepting Bill's Offer.
This new state is 'in tension': it's not yet mutually self-consistent with the business rules - which say that we have a sale.
To resolve the tension, something's got to change. So all the losing Offers get updated to 'lost', and Bill's to 'won'. The Item gets updated to 'sold', with a link to Bill's Offer.
Now we're back in harmony again after some simple transformations on our resources.
eA: Mm. That's an interesting perspective.
DC: Yes, instead of calling functions, we're asserting state and then applying rules to bring the state into a configuration consistent with those rules.
eA: OK. So let's go back to my SOAP function list: CompleteSale. You said paid and shipped from CompleteSale are on the Item - but they still need to be actioned. And we still need to add Feedback.
DC: Clearly, you can set the paid and shipped status of an Item using POST to the Item's URI, just like in the simpler data setting examples before.
Always remember, though, that an Item is responsible for its own integrity - it may change spontaneously according to its internal transformation rules, or may change directly via User POSTs - but only if it will be consistent with its internal integrity rules.
eA: Feedback?
DC: Adding Feedback happens by the seller POSTing a Feedback resource to the URI of the buyer User's Feedback collection, which then returns a URI for the server-created copy.
A number of Feedback resources can be aggregated into a rating on a User resource; another simple transformation, where the rating is dependent on the state of the collection of Feedback resources.
eA: That's a point - what URI did you POST your Items and Offers to before, to get them created on the server?
DC: A seller User can keep a collection of current Items to POST to; a buyer User can keep a collection of current Offers.
It's just like in the Atom Publishing Protocol: new Items are POSTed to the seller's Item collection, new Offers either to a buyer's Offer collection or directly to the relevant Item - either way should cause the Offer to be added to the other Offer list.
eA: OK. SendInvoice? This has the side-effect of sending an email - now there's an imperative!
DC: Not when you think declarative!
To send an Invoice, just POST it to the buyer User's Invoice collection to get it created on the server with its URI.
Creation of the Invoice resource then triggers notification to the recipient by email. This email can either include an entire copy of the Invoice, or have a nice RESTful link back to the actual Invoice resource.
Email notification in a REST perspective falls into the same category as Web Feeds and POST - it's a proactive way to get resource state or state change out to an interested party.
Note that, like all interactions in this approach, it's not event-driven - it's state-driven. If you keep re-POSTing the same Invoice, or Item or Offer, it only gets created once, and the email only needs to be sent once!
eA: All these resources floating around - how do you manage them? You mentioned the Item and Offer collections.
DC: A seller's User resource would collect resources together like Items and ResponseToBestOffers. A buyer's User resource would have links to their collections of Offers, Invoices and Feedbacks. Buyers can also be sellers, of course.
These collections all form good candidates for viewing through a feed reader, or feed reading widget. For example, you could subscribe to your Item's collection of Offers to see new offers come in. You might subscribe to the Item collection of a seller whose goods often interest you. You would almost certainly have a subscription to the collection of Feedbacks about you...
This kind of thing falls naturally out of the REST approach, as long as collections have their own URI. Note that, unlike Web Feeds, such collections should contain links to their contents, perhaps alongside some summary information, not entire copies of their contents' text!
(c) 2006-2007 Duncan Cragg
In Part 4: Inter-Enterprise REST Integration.
Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.
Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...