eBay Architect: So, can you summarise your argument that 'REST isn't just about reading and writing data', and explain your view on RESTful business logic?

Duncan Cragg: OK. The whole collection of related resources determines where things stand at any given time.

Resources are masters of their own destiny - guided by rules declared in the standard to which their content type conforms.

These rules, or business logic, run on notification of any declarations of the state of peer resources, or on arrival of any state via POST. Such peer states and POSTs are not commands, although it is possible to go ahead and define a special command or edit command content type.

The rules aim to satisfy the business or domain constraints on the mutual states of these resources - updating and creating resources accordingly and causing appropriate side-effects outside the server, such as financial transactions and emails.

These transformations are state driven. Even though the 'tension' in unresolved rules may be detected by events, that tension exists, not in those events as such, but in resource state.

eA: That sounds like a core difference to SOA.

DC: Indeed. It's a Resource-Oriented Architecture. And ROAs are declarative, not imperative like SOAs.

We have a world of resources declaring their current state, and resources settling into new states depending on the current state of related resources. These state changes can be driven by hard-coded resource animation logic, or by simpler, clearer, more scalable, declarative state transformation rules.

eA: Remind me of those patterns for notifying state change.

DC: Resource states are either polled via GET or actively notified via POST. Such actively POSTed state could be from a resource that also happens to be GETable, could be simply a link to such a resource, or could cause such a GETable resource to be created on the target server. Alternatively, the POSTed state could be considered too transient to record in a GETable resource, but can still trigger transformation in its target resource.

The above eBay examples used the pattern of 'server creates GETable copy of POSTed resource', and also 'second server hosts GETable copy of POST-notified resource'.

What I have described is a general programming model because, in general, such simple, declarative, transformational mechanisms are Turing Complete.

eA: I'm sure it's a novel perspective - even to RESTians! Again, do you have any high-level RESTian support for this?

DC: Any web resource that is a derivative of, or is dependent on, one or more other resources is using this approach.

Like I said before, there is an example of a similar approach by Joe Gregorio on his 'Well-Formed Web' site for alerting resources to peer resources of mutual interest.

Every time you would POST some data, consider making that data GETable and POST its URI instead, as a notification of the data existing.

eA: GETable POST data? You sure that's REST-compliant?

DC: In REST integration, things become more symmetric than in the client-server Web, or rather, the 'client-resource' Web. We can start to talk about the 'resource-resource' Web!

But anyway, we're already halfway to the symmetric resource-resource Web when we POST - not to a service, but to a URI. Resources can already both issue and receive state, which is a pretty symmetric state of affairs.

eA: I never thought of it that way - I keep forgetting that you can POST right back to a resource you just fetched.

DC: But think one step on: the POSTed data has a Content-Type but no URI!

Why not close the loop and have this POSTed data be a first-class resource (with a URI) that POSTs itself to the target. And it can itself GET that target or be POSTed to by that target in return.

That really is a Resource-Oriented Architecture. Once resources are seen as equal and active participants in RESTful integration, it becomes irrelevant whether their state is transferred by GET or by POST.

eA: I'm still having trouble with this pattern of POST just being a pro-active GET.

DC: Making POSTed data GETable more correctly moves the responsibility to the target resource to fetch the incoming resource state when its ready (rather than being bombarded by state it hasn't asked for).

Once the target is interested, updates can be POSTed directly as they happen, to prevent the target polling, or notification of an updated URI POSTed to trigger the target to re-GET the changed resource when it wants (thereby updating the caches).

eA: Hmm - makes clients look like servers..

DC: Since our 'clients' in REST integration are also 'servers' in other contexts, it is easier to set up client-side resources than on the browser-based Web. One objection to cookies on the Web is that they are state or resource that has no URI. So give your 'client' state a URI! And put any client-specific server resources on your own 'client' host.

eA: Is anyone doing this sort of thing?

DC: Well, in fact there are many examples of this POST-notification of a GETable resource already happening between web sites. Like submitting a link to your site to an indexing engine and letting it crawl (or poll) it.

Trackback pings are another example: POST a URI along with a sample of your page. And the Microformat rel-tag adds your article to Technorati's tag index when you ping their servers with the URI of the article.

Further, imagine POSTing to some new site a link to your hCard on your own server, to save you having to type your name and address again. And you'd never need to manually update sites when your address changes: just ping 'em all.

eA: Ah - but I thought all URIs should be GETable. The ping URI you're POSTing to in these examples isn't always one that you can also GET!

DC: Indeed - so think how much more powerful it would be if we did close the loop and provide or create a GETable resource to POST these notifications to.

For example, imagine a page containing an hCalendar event. Now point to it with a rel="attending" link. When the hCalendar discovers your intention (using a direct POST ping of your page's URI to the hCalendar page's URI - or perhaps through the referrer trick from people clicking through), it adds your referring page to a list of attendees inside the hCalendar. The hCalendar could either contain lists of backlinks to the attendee's pages, which may in turn carry hCards, or it could contain lists of complete hCards copied over.

eA: Sounds like a good use of Microformats.

DC: These examples make crawling and polling (even with If-Modified-Since et al) look like a clumsy version of the more proactive POST.

Web Feeds and general publish-subscribe are further examples where POST may be used to notify changes on a resource - giving the feed consumer first-class resource status with their own URI.

eA: I'd never think of using HTTP in this way.

DC: Obviously this only applies where the feed consumer is a visible and POSTable server and where timeliness is crucial. And probably where the number of subscribers is relatively small, unless asynchronous I/O and an event-driven architecture are employed, and you don't wait for the response to each POST.

This isn't done now simply because of the asymmetry of the current Web, an asymmetry which we are free of in REST integration.

eA: What about all those REST rules about idempotent and unsafe methods?

DC: We're not mixing GET and POST in that sense, just turning the tables on the asymmetric Web. GET is still cacheable, and we can POST a link to cause a cached GET.

I believe this is a more-constrained REST style, not disjoint to REST. It is at least an ROA! It may fall foul of REST's client-server constraint, since we're now in server-server territory with integration applications. Also, the concept of 'Hypertext as the Engine of Application State' is something that may take some refitting to the mutual state dependency model. However, I believe it's most important to focus on maintaining the benefits of REST and its key elements of standard content types at URIs.

I call this symmetric REST integration style the 'Distributed Observer Pattern'.

eA: Quickly summarise the 'Distributed Observer Pattern'.

DC: OK, the Distributed Observer Pattern is 'symmetric REST'. A resource subscribes to a peer resource via a GET that supplies its own URI, and is notified of subsequent state changes in that resource through a POST back.

eA: That was too quick. Tell me the details!

DC: OK, here are four. First, a POST can be either the whole new state or the fact of the change, allowing the subscriber to GET the resource when it's ready (and thereby fill any caches).

Secondly, you can use either the Referer header or perhaps the Content-Location header in POST and GET requests to indicate the origin POSTer or GETter URI. Alternatively, you can send this origin resource URI using the Cookie header, echoing its use in the normal browser client-server case to identify the pseudo-resource of a browser user.

POSTed state notifications may be unsolicited by a prior GET subscription, when the POST target is clearly open to them (as in the ping notification examples). These can be seen as 'subscribe to anyone', and may be combined with a corresponding 'GET anyone' crawling process, without explicit subscription.

Finally, POST notifications may be targetted to single resources to ask them to update: the Distributed Observer Pattern way of achieving the client-server editing function. These now become 'edit suggestions' of the POSTer resource - putting the target back in control of its own destiny and integrity.

eA: And why should I use the Distributed Observer Pattern?

DC: The Distributed Observer Pattern supports the programming model of inter-dependent resources whose own state is a function of their peers' state, driven by declarative rules. It's a very general ROA programming model.

(c) 2006-2007 Duncan Cragg

In Part 6: Content-Types and URIs.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Microformats from the Ground Up

Ryan King and Brian Suda from Technorati presented a half-day tutorial.

You can describe social networks, calendars and various other semantic relationships by annotating page elements or embedding small XML structures into your page according to a Microformat spec.

Microformats take Declarative or semantic markup (e.g. in XHTML) to the next logical level. Microformats have been called the 'lower case Semantic Web', because they fill the need for a lightweight set of conventions for data in Web pages that transcends the merely renderable.

HTML has a number of extension points that make this kind of thing easier. For example, the 'rel' and 'rev' attributes on a link (say what type of link it is - maybe a tag); the 'class' attributes (used for CSS but can also carry add-on semantics); the 'profile' element (say what kind of thing the whole document represents).

Microformats examples include rel-tag (for tagging your blog entries in Technorati, etc), hCard (like vCard-on-a-page), hCalendar (like iCalendar-on-a-page), XOXO (outlines) and XFN (social networking). Technorati have an hCalendar subscription service.

However, the one that stood out for me was hAtom, with which you can publish a blog and then let feed readers poll the actual page. They can read the hAtom markup and treat it as an Atom feed.

Slides here.

The Intelligent Design of Microformats

Ryan King also presented a talk on the motivation and style of the Microformats effort.

There is no official standards body defining Microformats, just an "Open Source"-style engineering effort, which follows the following list of Worthy Engineering Principles: "Rough Consensus and Working Code", "Paving the Cowpath", "Keep It Simple and Stupid", "You Ain't Gonna Need It' and "Don't Repeat Yourself".

Paper here.

Microsummaries in Firefox and on the Web

Myk Melez gave a presentation about Microsummaries in the up-coming Firefox releases.

To quote:

Microsummaries are regularly-updated compilations of the most important and timely information on web pages. For example:

• current stock price and movement for a company profile: "GOOG: 406.74 + 0.58"
• latest headline for a news site: "BBC: US dismisses Iran attack claims"
• highest bid and time remaining for an auction item: "Godzilla DVD: $15 / 2 minutes left"

One way to create this information is to link ('rel="microsummary"') from a page to its associated Microsummary document. You can also use XSLT to build the summary on the client. Both approaches struck me as clumsy.

I commented that it would be more Microformat-like to allow embedded markup (just annotate the Microsummary information right in the page; perhaps call it 'hSummary'?).

Myk used the bandwidth and page generation cost argument against that, but good XHTML pages should be very lightweight, and there are various AJAX techniques that can assemble heavier pages.

The current manifestation of the idea is to write this summary on a corresponding bookmark button in the chrome.

Allowing just bookmark buttons seems rather restricted to me. I'd be happy with a whole page in its own tab (or a widget in an Ajax desktop).

A drag-and-drop prototype was also shown, where you visually mark up the parts of the source page you want summarised. I commented that a good extension of that would be dragging across elements to a whole new page, whose construction would be dependent on the information in one or more source pages.

Myk didn't jump up and down with joy at my suggestions, so I guess we'll have to wait before we get Microsummaries written into Web pages that generate more Web pages.

Slides here; Paper here.

RDFa: The Easy Way to Publish Your Metadata

Of course, the Semantic Web folk have their own Microformat approaches, including this one.

To quote from their paper:

The approach taken by RDFa is that ultimately any RDF structure should be representable. This means that instead of having to 'codify' each format to describe how it must be marked up, we simply provide a set of rules that explain how any RDF can be marked up, and than any RDF 'language' can be used.

The 'a' stands for 'attribute': RDF encoded as attributes in XML.

The example shown was FoaF, whose Microformat competition is XFN and hCard. You can use attributes like <link rel="foaf:mbox">, <span property="foaf:name">, <span property="foaf:phone"> to encode RDF triple information at various points around your document.

RDFa has some arguable benefits over their competition, but I don't see why both sides shouldn't get along and share ideas. And, indeed, why the other alternatives (structured blogging, embedded RDF) shouldn't all find a place, too. There is a comparison here. Another speaker at XTech, Uche Ogbuji, has also discussed some of these issues.

Web 2.0 Centraal

Microformats not only challenge the Semantic Web, they even challenge such basic Web 2.0 technologies as Web feeds and Web APIs.

Shouldn't we be subscribing to and interacting with semantic Web pages (note the lower case 's'!), not using invisible Web Feeds and Web APIs? Technorati think so: they are leading the way in searching and subscribing to Microformats.

The redundancy of Atom itself (and feeds in general) has been remarked upon before. Further, since we're using REST in the Atom Publishing Protocol, why not use the same protocol to publish directly to an hAtom annotated site? (e.g., just POST a new article to your own blog's main page!) Web 'API's are redundant when our pages are just XML data anyway.

Writing data pages instead of document pages, then allowing those pages to be updated and then subscribed to, is the future of the Web: it's Web 2.0 Centraal.

The new lightweight, Declarative, Microformat-rich XHTML pages we're going to be generating can be polled just as Web feeds are, and events made from their changes. Then re-presented via Ajax in dynamic pages. Such standardised page formats and micro-formats should also accept standardised POSTs where appropriate.

Microformats represent some first steps towards the Data Web replacing the Document Web. Although there has been some thought given to extracting full-blown Semantic Web data from Microformats, there's a significant chance that this bottom-up, simple, minimal, change-aware (i.e., updateable, subscribable) approach may teach the Semantic Web practitioners a thing or two about the Worthy Engineering Principles that enable the social and technical proliferation of a good idea.

Standardising common formats and structures is a precursor to this two-way, dynamic Data Web. Both the bottom-up practitioners of Microformats and the top-down practitioners of the Semantic Web have much to learn from each other on this path.