Duncan Cragg: Now, let's look at those calls in eBay's SOAP API that are expected to change data. There is often a corresponding 'Get' function.

Examples are GetTaxTable, SetTaxTable and GetMyMessages, ReviseMyMessages, ReviseMyMessagesFolders, DeleteMyMessages, etc.

Here, there is an implicit model of data (or lists of data) that you can look at, modify, delete, etc.

eBay Architect: It's not just data: Tax Tables and Messages have meaning and behaviour internally to the server.

DC: Yes, but we're talking about what the API is telling us, here. It says things like GetTaxTable and SetTaxTable!

We'll come on to business processes later, but just look at the simple stuff first.

eA: OK, so what's the benefit of doing these calls in HTTP, through POST or whatever?

DC: Again, scalability and interoperability.

In the same way as with GET, you can partition your POST handling across the URI space. Have your Tax Tables and Message folders handled by many machines, split by their URIs. This is an example of the inherent parallelisability of the Web's architecture.

eA: We do partition by function: search, display, sell.

DC: Sure, but you had to 'hard-code' this partition: to use specific heuristics and code to achieve it: URI-based partitioning is much more generic and flexible, and can be much finer-grained. You seldom get single-point of failure or bottleneck issues with good REST architectures.

eA: If you say so.

DC: Interoperability again derives from the standardisation of the Content-Types and the schemas of POSTed data. When you know the meaning of the data you fetch, you also know how to try and change it, if you can.

eA: In HTML, you actually get presented a form in some GET data which explicitly tells you what to POST back - what the 'schema' is.

What's the equivalent in a general REST integration context where it's machines that are doing the POSTing?

DC: Well, the same applies, insofar as the GET data effectively tells you what any return data should look like. However, this time it's in an implicit way.

Where, in the read context, you understood what to do with a given content type you'd fetched, now, in the write context, you further understand what you can send back to it again.

eA: Can you give examples, please? The Tax Table perhaps?

DC: Straightforward data updating is the simplest case - it's the same content type coming out as going back in. You see a Tax Table at some URI, then you just PUT a new Tax Table back at that URI to try and replace it.

All the API function calls beginning 'Set' should probably be implemented this way.

eA: OK, so how does REST implement our Message API functions?

DC: An example of a data format that implies its edit capabilities by reference to a standard is Atom syndication.

You can GET a list of Atom entries, then POST a new one according to the Atom Publishing Protocol (APP) specification. You POST, not to a 'service' or handler, but to the URI of the actual collection you're adding to.

eA: So what's that got to do with our Messages?

DC: The eBay Message lists are a great candidate for using this approach, with the benefit that any (recent) feed reader can be used to view them, and an APP-compliant client used to manage them.

All the API function calls beginning 'Add' should probably be implemented in the same way APP adds new entries, or you should consider using APP itself to implement them.

eA: OK, so we can replace some calls with a standard set of REST interactions. But clients still have to know about Tax Table schemas, User Preferences schemas, Message schemas, etc.

It's not enough to simply know about HTTP and HTML to be interoperable any more, like a browser does. It's not even enough to just know APP.

Surely interoperability stops when the content types and schemas start to proliferate like this?

DC: Schema proliferation is SOA's problem - so I'm glad you brought it up! It's baked in to the SOA mindset that it's OK to design your own interfaces and schemas from scratch each time.

Conversely, the expectation and culture in the Web and especially in the REST-aware community is to constantly look out for opportunities to conform and to standardise schemas and interactions, and to build on layers below that are already standardised. It's a side-effect of the shareability of URIs.

With REST you get interoperability at many levels above the byte-transfer of HTTP. Content type understanding and standardisation can occur all the way up from characters to Tax Tables.

eA: I'm not sure I get this.

DC: OK: there is some code that understands basic HTTP GET and PUT, and UTF-8 characters - and doesn't need to know what a Tax Table is - some that understands UTF-8 plus XML, some that understands that plus Atom, or plus XHTML, then XHTML tables and Microformats like hCalendar, hCard (and hAtom!), then conference schedules using hCalendar and hCard. Tax Tables can perhaps use XHTML tables.

Clients may or may not need to know what the schema of a Message looks like internally in order to be able to do useful work with Messages at their level of understanding - from character stream up to APP.

eA: I'm not sure how general auction schemas fit in there.

DC: You never know, your eBay schemas might be taken into account when coming up with new standard content types for e-commerce!

eA: Hopefully. We could always put everything into XHTML tables for you!

DC: Ha! Go for it. Having done some HTML scraping myself, I'd welcome such semantics!

eA: Right, it's OK when we're just talking data that you can read and write - HTTP, APP, REST, whatever may be great for that - but our API has much more complex business functions in it that don't fit that simplistic model.

There's a whole load of functions in our API that aren't just about getting and setting data or collections of data. Some are eBay- or auction-specific.

DC: This is one of the Myths of REST - that it's just for simplistic reading and writing of data.

It's a actually a myth that's often propagated by the REST community itself, especially with their over-emphasis on the Four HTTP Verbs, which seem to map so conveniently onto Create, Read, Update and Delete.

eA: But they do map so well - I thought that was 'good' REST...

DC: It is often good, but can detract from the whole goodness!

One consequence of this mapping it that people inevitably go on to see an analogy between REST and databases, and then start to expect transactions and other database features.

Another consequence of the database analogy is that resources are seen as lifeless servants of the active client: it takes away responsibility from a resource to be master of its own destiny.

This then causes confusion (even within the REST community) about the power of the client and even the very meaning of such basics as the PUT method.

The fact is, GET and POST are more than enough to be REST compliant. This cut-down pair also help focus the mind on URIs, two-way content transfer, content type or schema and on the responsibilities of each resource as active players in an integration scenario.

eA: Sounds like a minority REST view - any support from a REST luminary for that?

DC: Tim Bray has a history of suggesting that GET and POST are enough.

And recently, there has been further high-level support from Sam Ruby and Leonard Richardson in their manifesto for an upcoming book.

eA: I bet you still get told off, but it sounds reasonable to me.

DC: So, to summarise: use GET to read data, then POST back to the same URI to suggest changes to the resource there.

All based on a given level of understanding and interpretation of the content type and any corresponding interaction standards.

Interoperability and scalability in a nutshell!

Now I will explain how there's more to REST than simple reading and (attempted) writing of resources. We're still only two-ninths done...

(c) 2006 Duncan Cragg

In Part 3: Business Functions.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Duncan Cragg: So - let's get straight to my argument: I claim that your SOAP APIs, as instances of the SOA style, won't scale or interoperate as well as they would if they were implemented in the REST style. Which, in the form of the Web, has largely proven scalability and interoperability.

eBay Architect: SOA won't scale or interoperate and REST will? You've got some explaining to do there: the whole industry is going SOA crazy, and we're following that trend. You'll need to give specific examples.

DC: OK.

eA: And even if it is true, we don't care about scalability and interoperability: we have quite low traffic on our APIs - certainly compared to port 80; plus we're eBay so we don't need to interoperate with anyone.

DC: That's true now, but if the Web 2.0 vision comes together, you may care: your API traffic could increase dramatically. It would be better to be the one prepared for the scale of the API-Web!

Can you really argue in your company that you don't need to be scalable? What if your port 80 traffic needs to be routed to your APIs for some reason?

eA: Maybe.

DC: As for interoperability, you could be excluded from Web 2.0 industry-boosting consortia, or excluded from perhaps hugely popular Web 2.0 applications in the future...

Interoperability raises the level of the market as a whole. Market players shouldn't differentiate on what's common to them, they should differentiate on the level above.

It also depends on the value you place on having happy customers who don't have to do the same thing multiple ways or multiple times.

eA: Ha! I'm sure there's a name for that style of argument!

So, let's say we want to be good scalable and interoperable citizens in a future Web 2.0 world: show me how REST can help us more than SOA and SOAP do!

Getting Data

DC: OK, let's look at your SOAP API. There are 72 function calls in there that begin with 'Get'. Each one specifies a particular piece of data that you can fetch.

eA: Yes, our API is extremely rich.

DC: Sure, but you don't need a new function call for everything you can get from your system: you can just use HTTP GET!

eA: OK, but it's just data going in to request the information, then data coming back out again. What difference does it make whether we use HTTP or SOAP to carry those messages? You've just moved the name onto your URI!

DC: It's not just any 'data going in': the URI can be passed around for anyone to re-use. This URI is more interoperable because so much deployed software understands it. No-one understands 'GetSearchResults()'!

eA: OK.

DC: Another example of how the URI can glue things together is that the data returned from your GETs can have more URIs in them, ready to go! You won't get data from your Web Service with 'GetItem()' in it..

eA: Heh - I s'pose not.

DC: REST also talks about the formats of the data behind a URI. In a GET, the response data is given a Content-Type, and there's an expectation that clients will understand the types of data being returned: interoperability comes from broad standardisation of return data.

eA: But we return XML, and our schemas are published. They're specific to our application anyway. That wouldn't change with REST.

DC:The explicit statement of Content-Type reflects a culture of agreement forced by the sharability of URIs: your URIs are more sharable when more clients understand the data they dereference to.

On the other hand, the culture of SOA is to declare custom WSDL and custom XML schemas.

Like I said, one day you may care about interoperability, and having an architecture that puts a high value on content type and schema standardisation, as REST does, puts you one step ahead.

eA: We'll see.

DC: You can also gain scalability by partitioning on those URIs.

eA: We already do plenty of partitioning of our data and functions.

DC: Yes, but URI partitioning cuts right through the system in a very simple way: your partitioning is an application-specific optimisation which has to be hand-coded behind the SOAP interface.

eA: Mmm. Maybe.

DC: Another benefit of using HTTP over using SOAP is that you get cacheing built in to the architecture, which you can start using as soon as you ask for it in the headers. This boosts scalability.

eA: OK, that's fair. Our clients would have to write their own caches, should they need them. But in general we advise against cacheing our data and don't put expiry on them or have not-modified checks.

DC: Which is where you're potentially inefficient.

eA: Maybe. We internally cache things like the data behind GetCategories, and advise our users how to cache it themselves.

DC: Again - it's application-specific.

So - even in the simple cases of fetching data, REST has given you much greater scalability and interoperability than your SOAP interface - as well as a simpler, more generic approach.

And we're only one-ninth of the way through our conversation!

eA: How do you know that??

DC: Aaah...

(c) 2006 Duncan Cragg

In Part 2: Setting Data.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

This is the last of my XTech 2006-inspired articles. (I know: XTech was some weeks ago now, but I'm sure you'll have noticed that this is a 'contemporary article' blog, not a news blog!)

The theme for XTech 2006 was 'Building Web 2.0', and as a Web 2.0 Conference (don't tell O'Reilly), there was tons of Ajax... And as this is essentially an XML Conference, there was also a strong Declarative theme.

And the right way to do Ajax is Declaratively!

Ajax Lightning Demos

Simon Willison presided over this session. The demo that impressed me most was Hijax: Progressive Enhancement with Ajax, whose paper is here.

Hijax takes the view that you should design for no Ajax, then hijack the links, form posts and page construction in Ajax to speed it up, rather than designing for Ajax in the first place and excluding users without a capable browser. This is related to the AHAH idea.

So you don't build your application in Javascript, but simply enhance the existing Web model, running all the page's behaviour back on the server which owns it. Much more Declarative!

Web 2.0 On Speed

Mark Nottingham is one of those people whose blogs are top of my feed list. He currently works for Yahoo! This talk was about using some basic techniques to ensure Web 2.0 doesn't grind to a halt with all those dynamic and personalised pages. Simple fixes can get you very far: faster servers like lighttpd, various cacheing tricks, etc. Yahoo! should be implementing his ideas and insights on their sites (hopefully Mark will then go on to tell their API designers how to do REST...)

The main idea that interested me was using Ajax for client-side page construction (think client-side SiteMesh) and for inserting personalised URLs into the page.

It's a bit like Hijax, that I mentioned above. Mark says:

What I would like to see is for common JS functions like this to be sifted out and standardised as declarative markup...

This is good stuff (assuming you agree with us that Declarative is good!). See more in the summary and slides, and in Mark's hInclude proposal.

Ditching the database: XML and the PHP webapp

David Megginson basically said we should work in XML throughout our web apps: cache XML as your datastore, keeping all the little snippets that make up your pages - dynamic and static - cached in memory, then just send XML snippets from the server cache to the Ajax client to render, etc. This is rather like Mark Nottingham's tip above on client-side page assembly to achieve greater cacheability.

Paper here.

Introduction to XHTML2 and XForms

The flamboyant-looking and entertaining Steven Pemberton of the W3C presented a fast-paced and in-depth overview of XHTML2 and XForms.

Unless I missed something, it looks to me like XHTML2 has taken a perfect trajectory between backwards-compatible concepts, semantics and syntax and abstract idealism. XHTML2 appears to cleanly fix all the outstanding problems with HTML once and for all.

Now, part of XHTML2 is XForms. XForms does the same cleaning up job on HTML forms that XHTML2 does on HTML.

The important thing to know is that it works as an XML 'schema constraints engine'. That is, it returns to the server a fully-semantically-valid chunk of XML on a form submission. The 'schema' can include some static data pulled in from a separate URL to populate drop-downs, etc.

Yet it is more than this: I asked if it was Turing Complete, and Steven said yes. You can program an application using it.

And that application will be programmed Declaratively!

I can define the constraint that some output element be dependent on some input, then if the user changes the input, the output changes accordingly.

It's like Declarative Ajax. There were further talks on XForms which I cover just below.

Of course, I asked if it was RESTful, and it seems it was, mostly. The issue was that a URL may bring you in an editor XForms page, which then itself brings in some data underneath.

The editor XForms page is the bookmarkable thing but you don't see the URL of the populating data, so that's not bookmarkable. The page may change according to this data, but bookmarking what you see still only bookmarks the editor, not the editor plus the current visible data.

Also, even though XForms is amazingly programmable, it is still a forms system at heart. For example, you have explicit input and output elements; you can't get input events off the DOM and rewrite the DOM as output.

Slides here.

Read more about XHTML2 here.

Read more on XForms from Mark Birbeck (a Pemberton collaborator) starting here.

Building Rich, Encapsulated Widgets Using XBL, XForms and SVG

Mark Birbeck showed us how to create a Declarative client-side mash of Flickr. OK, there's a tiny bit of script in there, but I'm sure it's unnecessary, as it only does some simple DOM manipulation and event handling. Perhaps it can be replaced somehow, or made part of a generic library.

Paper here.

XForms: an alternative to Ajax?

Since XForms is so amazingly programmable, you can use it instead of Ajax.

Instead of coding a special script on top of an Ajax library, you can bring in a generic (Ajax) script that gives you XForms today, and write your application Declaratively on top of that.

Alternatively, you can do some pre-processing on the server, which is the approach Erik Bruchez explained here.

Paper here

The Power of Declarative Thinking

This talk was more from Steven Pemberton giving us a little history of his work and showing us how XForms can be used for Declarative programming.

Some good quotes from the slides:

.. a program that is 10 times longer is 32 times harder to write.

Or put another way: a program that is 10 times smaller needs only 3% of the effort.

And of course, the point is that Declarative programs are usually 10 times smaller.

.. no one writes applications except programmers.

Interesting exception: spreadsheets

Mostly because they use a declarative programming model.

The nice part about declarative programming is that the computer takes care of all the boring fiddly detail.

Now I'm as aware as anyone of the dangers of giving non-technical people the power to program. Spreadsheets are, let's say, a mixed blessing. There's probably a long way to go before we can give the business the reins over their own business rules.

But it will happen: it means splitting programmers into (Declarative) Business Analysts and (Imperative) Those That Enhance and Maintain The Underlying Processing Fabric. I can tell you now which group will get paid more.

Standardising Web Applications: Rich Web Clients at W3C

Dean Jackson gave a talk about what the W3C are doing to play catch-up with all the latest goings-on in the Wide World of Web 2.0.

Here's a quick list of things they're standardising: XMLHttpRequest, Networking, Timers, Client-Side Storage, File Upload, the Window class, DOM Level 3 Events and XPath, Drag'n'Drop, Remote Events for XML (REX); XBL2, XUL (possibly alongside XAML, MXML and LZX); Compound Document Formats.

Again, the magic word Declarative came up in the talk.

The Declarative aspects of Rich Web Clients centre around the DOM Level 3 work, Drag'n'Drop, Remote Events for XML (REX), XBL2 and XUL and Compound Document Formats. SVG is a common theme between many of these (once again, Sam Ruby is on the case).

The WHATWG is an independent collaboration of browser builders (Opera, Mozilla, Apple) that has its own initiatives in this space - including HTML 5 and Web Forms 2.0, the Canvas object, server-sent DOM events, network connections, client-side state, drag'n'drop, etc. They are happy to work with the W3C to ensure compatibility and W3C standardisation. Here is an article covering some of these WHATWG ideas.

Other Declarative Browser-Driving Work

When you look around outside of the Conference, you suddenly discover numerous further examples of Declarative browser-driving:

Tim Bray has been stitching pages together with Ajax.

Sam Ruby says Declarative is more mashable.

AHAH is 'Asychronous HTML and HTTP', an apparent attempt to make a big deal out of rewriting the DOM on receipt of an Ajax HTML response. Same ideas as above.

Jemplate looks like a similar kind of thing.

XAP (Extensible Ajax Platform) is an interesting project that was given by Nexaweb Technologies to Apache. Here's the elevator pitch:

• Declarative rich user interface via XML;
• Data binding for connecting UI to data;
• Incremental update of web pages declaratively and programmatically;

Widgets intersect with this whole dynamic, client-side page-assembly space. There are now numerous examples of commercial Widget approaches. Some Widgets may be driven by a Web Feed, some may get their data from a Microformat or from a Web API.

Opera Widgets are DHTML packages that break free from the browser and run on the desktop. The WHATWG could even end up taking them as the basis of their play in this game.

Of course, we've already seen how to do Widgets right, in Mark Birbeck's talk above. In general, Compound Documents and dynamic, POSTable Microformats are the (Declarative) future of Widgets...

Declarative Meme

We need the 'Declarative' meme to consolidate and create a movement out of all this, otherwise apparently disparate, work. Once we have a meme to gather around, we gain collaborative and cognitive power from its mere existence. This could work in the same way that the label 'Ajax' itself consolidated (and launched into orbit) techniques that a large number of us had already been doing for months or years.

The 'Declarative' meme also includes and extrapolates the 'REST' meme. REST is Declarative, but not something you may normally associate with Ajax application programming. Yet programming a dynamic Web interface Declaratively goes hand-in-hand with the linkability, cacheability and mashability of a RESTful back-end interaction, as Mark Nottingham and others above have shown.

The Right Way to do Web Applications is Declaratively

Specialist client-side Javascripts are a mistake. They break the generic Web client model: hand-crafted thick clients are hard to make cross-browser compatible, they add to download times, they are tied to specific application servers, they break the browsing experience (bookmarking, linking, history) and miss the benefits of the Web's cacheing architecture.

We should be writing Declarative pages to exploit generic Javascripts and plugins or extensions (such as SVG, XForms and other Declarative technologies). These generic facilities will effectively extend the concept of the browser, from one that deals primarily with one-way static documents, to one that deals with two-way dynamic data.

After some time, the browsers themselves will be written to new standards that make those browser-redefining Javascripts and plugins irrelevant.

By a 'two-way dynamic data' browser, I mean a browser that constructs pages from several sources - several snippets, microcontents and widgets. It will show those pages updating in real-time as those sources update.

It will be aware of the user's key and mouse events on the visible DOM, and relay them back to the affected source, which may in turn update - for everyone viewing the page.

Programming the behaviour behind the data can be completely Declarative; rule-driven and event-driven. The programs will run both on the server, where the Resources are owned, as well as on the client, as in XForms.

I've shown many examples in this article that are heading in exactly this direction.

New Web (2.0) standards, that respect and extend Web architecture, can one day replace hand-crafted thick client scripts and generic scripts and browser extensions, allowing us to program our dynamic applications Declaratively - directly to a Web 2.0 browser.

Such a Web 2.0 architecture and platform will still be as scalable and interoperable as the current Web architecture - more so, in fact. And it will be much more fun to program for!