What Not How - Posts tagged 'microformats'

Content-Types and URIs | The REST Dialogues

2008-02-16T23:44:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 6: Content-Types and URIs

eBay Architect: OK, enough fancy REST or ROA interaction patterns! Let's get back to REST basics.

Duncan Cragg: That'll be content types and URIs, then.

eA: OK - I've a question about your standard content types. In particular, about Microformats.

DC: Go on.

eA: You keep on mentioning Microformats when offering examples of content type standardisation. But Microformats are always embedded into HTML (or, preferably XHTML).

Are you suggesting somehow taking them out and using them separately? Or do we have to always use them through a, possibly inappropriate, document model schema?

DC: Microformats currently 'tunnel' through mechanisms in XHTML such as the class attribute. They themselves have distinct model schemas, that ride on the document model schema.

eA: So can you use them apart from HTML?

DC: No, not really, although you can carry a single Microformat in a single document carrier, then squeeze out any document model content that isn't supporting the Microformat, perhaps leaving enough that it still renders sensibly in a browser if anyone looks.

The Microformats people won't like you doing it, I'm quite sure, but if you and I want to exchange a pure hCard, hCalendar, hResume or hReview, and nothing else, then we can use the minimal document model carrier, and have just one Microformat per resource.

eA: But why not use the original data schema, before it was Microformatted? Why not just use vCard and vCalendar?

DC: Or use Atom instead of hAtom! Of course, if vCard has an XML representation, you could use that - as long as the constituency of your clients is the right one and is big enough. There may be more code out there that 'gets' hCard than an XML vCard. And some Microformats - such as hResume and hReview - don't have an original schema and are based on abstracting from common or prior behaviour.

This is REST integration we're talking about, where data, not documents, are native, and we aim to search out the most popular and most widely understood data schemas - even if carried over documents - to maximise interoperability.

eA: OK, that seems fine. Although I'd point out that REST doesn't have the monopoly on interoperability. SOA does that too.

DC: Interoperability is best acheived by sharing millions of URIs dereferencing to a handful of standard content types, with interlinks across the Web of resources. ROAs (Resource-Oriented Architectures) do that. SOA doesn't.

eA: REST APIs don't always have to do it. In the previous example you went through, eBay and gBay could offer REST interfaces but not talk the same schemas and not allow cross-linking in the way you described. Or talk the same schemas but not recognise each other's Items and Offers.

DC: That would be walled-garden, silo-thinking. It's also 'API' thinking. Just opening up a port to your application, even one with correct use of GET and POST on well-organised domain URIs, isn't in the spirit of REST, and certainly isn't good enough for REST integration.

In REST we always aim to adopt the same schemas, to aim explicitly for interoperability. And linking between those resources, even cross-site, is fundamental to the REST way of thinking. If someone offers you a 'REST API' that uses unnecessary proprietary schemas that miss obvious interlinking opportunities, especially across to other sites, run away!

eA: Are there any real-world examples?

DC: A good, and ironic, example of this is Google's Open Social, at least in its earlier releases, which fails to achieve true cross-site openness even with a 'REST API' and shared schemas, because sites don't cross-link or actually allow data sharing. Also, the schemas are a strange extension of Atom, rather than using, for example, vCard as the basis for 'People Data'.

This hopefully will be fixed as the 'REST API' evolves and with the work going on in groups such as DataPortability, with agreement from the major operators.

eA: So much for the interoperability of that REST interface.

DC: The heart of good REST interoperability is the acceptance of standardised data at a 'foreign' URI, and the re-publishing of that foreign URI in your own standardised resources. It happens on the Web all the time, of course. We just need to copy the model for REST integration.

Hypermedia and (more importantly, here) 'hyperdata' is baked into REST, but is an afterthought in SOA. ROAs create an interlinked hyperdata landscape across sites and domains. I'm using 'hyperdata' here in the sense of interlinked data resources in REST integration, by analogy with hypermedia, not in its Semantic Web sense.

eA: Ah! But how do your little pure Microformat resources link up into this hyperdata landscape? Microformats can't link to each other, can they?

DC: It's true you may have to go and get involved in the Microformats movement in order to help define how to link an hCalendar event to a list of hCards of people attending. Or the hCard of a company to a list of hCards of its board members. Or an hReview to the hCalendar event being reviewed and the hCard of its author. Or to include the XFN list of links to friends' hCards inside a person's own hCard.

One indication that there's something not ideal in Microformats is the fact that you have to write someone's hCard out again and again for every page or site they appear on. If you could just link to a single hCard for that person it would be more efficient.

eA: But Microformats have a narrow charter: to decorate the document model with semantics. Any links are just part of the hypertext Web. It sounds like you're trying to make some kind of domain model out of them, with their own interlinks!

DC: Yup. When you start to think the data of REST integration, the document carrier of Microformats and it's often superfluous links can be a distraction. If the document links are relevant to the Microformat, of if people would use links within the Microformat if they were told what value it has, it would be worth pulling them out into the Microformat definition itself. Then enhancing in-browser Microformat parsers to follow links will greatly enhance their utility.

All you have to do is find real-world examples, and propose it on the Microformat lists! Meantime, reuse the schemas and keep all your extensions public and backwards-compatible.

eA: What about all those 'rel-' decorations? You know, rel-tag, XFN, etc.

DC: Well, hAtom is the only Microformat that specifies nested rel-links: rel-tag, rel-bookmark and rel-enclosure. Otherwise, each Microformat is independent, and the rel-links are independent. Like I said, it may be worth going to the Microformat community and suggesting more such rel-links beyond hAtom.

URI Opacity

eA: So this RESTful data landscape of data wired up with URIs: it sounds a bit hard-wired: where do URIs as queries (and URI templates) fit into that tight mesh?

DC: URI templates fall into exactly the same category as standardised content types and schemas in terms of their level of abstraction and location in the stack. In other words, the right thing to do, if it's transparent URIs you want, is to standardise search URI templates across sites of a type.

eA: This is getting complicated. It's hard enough to get agreement on the content types of resources, never mind on URI formats as well!

DC: Indeed, and in fact, I believe that URIs should be opaque: they already are to HTTP, but also in our data landscape, a URI should point to a single, predictable resource.

The mechanism of querying that dataspace should be separated out from the mechanism of linking it up.

eA: A bit like GUIDs?

DC: Exactly. In Enterprise applications, you often see GUIDs (globally unique ids) being used, and never see them mixed up with search strings!

Transparent, query or template URIs are either used to be helpful or decorative, or are an acceptable optimisation, as long as you know that it's tunnelling through or hijacking the URI for a quick query string.

eA: Tunnelling? Hijacking? You've dismissed a long-standing convention, in the Web at least! How else do you do query fetches?

DC: A better solution is the query-POST-redirect pattern: the client POSTs their query, then the server redirects them to a linkable results resource on an opaque URI.

The POST query schema can then be properly standardised in a content type, or 'templated' in the REST integration equivalent of an HTML form.

It's an extra round trip, but only one IP packet in each direction; a redirect or a GET can fit into a single IP packet - the cost is only in the connection latency.

eA: Why not just return the state of the resource you're redirecting to in the body of the redirect, to save even this round-trip?

DC: Yes, you could do that. It's not something seen in the hypermedia Web as far as I know, but this is REST integration, where we're able to come up with new sub-protocols like this - where HTTP response codes are often given much thought.

Further, the server can offer the option to snapshot this results resource, so that it's still exactly the same whenever the link is dereferenced - something you can't do with a query URI.

eA: What would Tim Berners-Lee say about this? Is it in the spirit or letter of his vision for how HTTP and URIs should be used?

DC: I've no idea! However, in my opinion, when Tim didn't separate the concepts of a globally unique identifier returning exactly one resource from a query string returning maybe none, one or many resources (in a list), he started a good deal of unnecessary confusion, even if non-fatal in practical terms.

The phrase 'hackable URIs' sums up the situation. We may have been forced into creating slightly better user interfaces if the URI textbox were taken away from browsers.

Make your interface your content and have good search and information architecture to allow your (opaque) links to be discovered. If you know that human users - or search engines - will be interested in reading some links at the top of your information architecture, then go ahead and use just a few simple, meaningful addresses.

eA: You're venturing into controversy again! I'm sure I keep reading about designing nice URLs being good practice.

DC: There was a time when transparent URLs were considered important, but now everyone just uses Google! All the energy that's put into URL good manners and systems of URI templating and naming is just a distraction from the bigger effort of standardising content and defining schemas.

Opaque URIs keep content in the body where it can be given a Content-Type, instead of the headers - the URL line.

This is related to my preference to put 'write methods' such as PUT and DELETE into the body instead of the URL line.

eA: How exactly?

DC: The URL line should have a definite target - an opaque, globally-unique URI - and a content transfer direction - GET or POST.

The rest of the application-level interaction, including anything that will affect state and any searching and querying, should be in transferred bodies with standardised content types.

In Part 7: Business Conversations

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Google Micro Conference

2007-10-05T11:22:00Z

Last night's Google London Open Source Jam (also here) was on the subject of the 'Web' (didn't they invent that? Oh no, that was Microsoft).

This event has been getting better and better each time I've attended. There were some very interesting lightning talks held together with a tight structure and plenty of chance to chat, drink cold Leffe and eat cold pizza. And nick [transatlantic translation: 'steal'] the Green & Black's chocolate.

An ideal Micro Conference...

I arrived late (it starts at 6pm) and spent some time catching up with ex-Thoughtworks colleagues, so I missed Dion "Ajaxian" Almaer's Google Gears slideset from FOWA. Go there now and check it out.

Thus the first talk I saw was a nifty piece of widgetry by Steven Goodwin called WARP. In WARP, interacting with a page of 'applets' changed the URL to encode those applets' current state. If you link to the current page, it will always show that state. Very long URLs, you can imagine. None of that fancy Ajax stuff. RESTful, dare I say. Nice API server-side for unpacking your applet params.

A trip to the lavatories [transatlantic translation: 'restroom'/'bathroom'] revealed that they are, indeed, doing that Testing in the Toilet project in Google. It works, too! I learned something. Other intelligence on Google's Inner Workings include confirmation of the beanbags and of the high quality, free grub to which I have already alluded.

A nice bloke from Yahoo! (Tom Hughes-Croucher: another spy?) came along to sell his idea that, in the collaborative world of open-minded hackers, we who run websites could help each other with our 404s. If I get a 404, I use the referrer link to tell you, via some RESTful POST, that your link to me is bust (assuming I don't intend to fix it myself).

I think the world is a little more selfish, so you need to decide who hurts more - the site who sends their visitors to a dead-end, or the site delivering that dead-end to a new visitor. I suspect the latter, by a small margin, as it's not exactly a nice welcome. So it's up to them to let the new visitor down more gently, and to notify the publisher of the broken link with little or no cost to them. For example, a really sociable 404-ing site could just redirect the hapless visitor back to the referring page, adding '?broken=links' to the URL - hopefully to be picked up by log scanning scripts at the referring site.

Next up, yours truly taking yet another chance to promote his excellent Micro Web thingy. Couple of people asked about it afterwards - including that nice chap from Yahoo! Also, a smart - and nice - chap called Toby (this one?) got me into a deep discussion on imperative vs. event-driven vs. state-driven programming. He was apparently an old-timer like me, as he was able to engage in dewy-eyed Functional Programming recollections. I managed to give out about four full colour printouts about the Micro Web, and to collect some good calling cards.

However, Joe Walnes, even a pint down in the pub afterwards, still refused to sign up for Micro Web duties. This in spite of over three years of intensive lobbying, including eight months of me working Trojan-horse-like in his kitchen, on The 2005 Implementation.

Another ex-Thoughtworks colleague, Simon Stewart took yet another chance to promote his promising Webdriver thingy. And a very interesting project it is becoming. Still needs more work - on IE support, etc - but I'll probably be using it in my new job at the Financial Times.

Another ex-Thoughtworks colleague, Chris Matts took a chance to promote his and Andy Pols' interesting new Dream Machine thingy. Perhaps a bit like Cambrian House - you put your dreams and ideas into it and people expand on them. Chris is a natural on-stage - and even used the age-old trick of promising lots of money for no effort, to get our attention at the start.

All I could come up with for the Micro Web was 'Cheaper, Wider, Faster'...

Updated: added reference to Dion Almaer, details about WARP, swapped in the picture of TWers that I was waiting for and fixed a minor blunder thanks to that ever-sharp ThoughtWorker, Dan Bodart..

How Ruby can enable the Web 2.0 Platform

2007-06-26T15:17:00Z

Web 2.0's definition includes seeing the Web as an application platform. Which means it is in competition with Java and .Net, and with SOA, for both local and widely distributed applications.

If the Web is going to be a platform, the skills you need to learn to program it are the core Web 2.0 technologies such as Ajax, JSON, Atom, Microformats and OpenID.

And Ruby. This language, that's capturing the hearts of many Web 2.0 programmers, is ideal for easing the transition from the Java and .Net platforms to the Web platform, as I will show.

Even if you're part of a big company that is generally immune to the latest trends, the marriage of Ruby and the Web-as-platform may be something to prepare for. It could even displace your SOA agenda...

Few would disagree that the Ruby language is riding the wave generated by Ruby-on-Rails. In turn, Rails is riding the Web 2.0 wave, coming as it does from underpinning the very Web 2.0 37signals product suite.

Rails and Ruby have tapped into the tech Zeitgeist of friendly, simple and powerful. The speed with which the Ruby and Rails communities have delivered the key components of Web 2.0 is matched by the speed at which Ruby and Rails books are leaving the shelves.

What is the ideal platform of Web 2.0? Will it be Rails and Ruby? Will Ruby ride the Web 2.0 wave into the mainstream in the same way Java rode the Web 1.0 wave?

Well, here's the problem with that question: Web 2.0 is supposed to be primarily about the Web itself as the platform, as explained first by Tim O'Reilly and then by a thousand Web 2.0 vendors and industry watchers after him.

Web-as-platform is not just vendor hype or pundit hand-waving. Let's think about what O'Reilly meant by that.

Web as Platform

Web 2.0 is about making the Web more interactive, and thus able to support applications where Java and .Net would once have been considered the sole delivery platforms.

The fact that the technologies of the Web can be turned to this use is a shift with far-reaching implications.

Broadly, the shift we are seeing is from the one-way, static document delivery of Web 1.0 towards the two-way, dynamic data exchange of Web 2.0.

This fundamental repurposing is delivering more complex, interactive applications that work inside our browsers and which fully leverage the benefits of online operation.

Web 2.0 is bringing the user and their stuff into the very Web that they hitherto only passively consumed. This network-enablement of the user in turn enables their social networking and their shared creativity and self-expression.

Web 2.0 has tapped into a deep human need - a fact reflected in the vast traffic volumes and correspondingly vast valuations of Web 2.0 startups that we're currently seeing.

But Web 2.0 is not just for the startups: Enterprise Web 2.0 is coming! The bigco.com site is going to be looking a little, well, static and lifeless when compared to the new sites that are springing up everywhere, and that most of BigCo's employees are using. Further, BigCo can gain huge benefits from Web 2.0 approaches empowering and connecting those employees on the Intranet. And that Intranet is an ideal platform for deploying company-wide, interactive applications.

This shift in the Web to two-way dynamic data is being powered by a set of technologies that a Web platform programmer is going to have to learn.

Web 2.0 Platform Technologies

Anything that claims to be an application platform must support data. Web 2.0 is above all the data Web. Web 2.0 is about semantics, not free text and font sizes. Hence, it inevitably starts with data-oriented formats such as XHTML, YAML and JSON. In Web 2.0 more than ever, we talk about data not documents and about separating data from its presentation. CSS is big in Web 2.0, for good reason (not just for gradient fills). Inside the page of a self-respecting Web 2.0 application, you'll often find Microformats - again, semantics in the page: publishing concise data of widely-understood standard formats. Some of those Microformats may be tags, and in Web 2.0 the simplest and most powerful semantics are those little pivot points in Webspace.

Again, if you're going to be a general purpose platform, you need to be able to fetch, update, notify and display that data. Web 2.0 integration usually happens via JSON data structures and REST interfaces (some of which, especially those based on AtomPub, are true REST). Following on from the data-like pages we serve to browsers, come the data-like feeds we publish to feed readers and to other applications. After feeds, the core technology that gives Web 2.0 its dynamism and interactivity is Ajax and DHTML, and increasingly Comet (server push to the browser). The core technology that gives Web 2.0 its users is increasingly OpenID.

All of the above are open technologies. You can do Web 2.0 without proprietary technologies, just like Web 1.0. Indeed, keeping to the principles that made the Web successful is also essential to the success of Web 2.0. The Web platform is the first application platform that has to consider scalability and interoperability, and will ignore them at its cost. I have written before about open data, use of standard data formats and using REST properly to avoid creating unscalable, walled-garden sites. You don't need Flash or SilverLight, you don't need vast amounts of custom Javascript, you don't need function calls tying you to your servers.

Programming the Web 2.0 Platform

So, we've got the dynamic data that you'd expect of a would-be platform. But how to drive changes in those data? How do we program the Web platform to animate all this data?

All Rails programmers will know the above technology list; it comes with the territory. The Web 2.0 Platform can be very succesfully powered by Rails and Ruby. Ruby and Rails make Web 2.0 applications simple and quick to program, addressing many of the needs of simple Web 2.0 applications out of the box. There's little doubt that Ruby and Rails will have a secure future riding the Web 2.0 wave.

However, for many Web 2.0 applications, programming may not even be necessary, at least not in the procedural or imperative style programmers expect.

Look back to the early 90's: 'Web 1.0' made a whole class of applications easy to write without programming: applications for navigating information. You just wrote in HTML, declaratively.

Now look back at the long path of evolution of Java, through J2EE, Spring, AOP, IoC, Domain Driven Design, POJOs. All trying to achieve the simple goal of 'remove all that MVC and persistence stuff and let us concentrate on business or domain objects'. But they never quite seemed to get it right.

But then Rails comes along, and has succeeded by simple virtue of concentrating on easy manipulation of the 'Intel Inside' of Web 2.0 - data.

It's reminiscent of the 'Naked Objects' approach to application building with minimal programming (just business or domain code in POJOs that expose state into the GUI and are transparently persisted). The Streamlined project takes Rails even further down this path. Rails' nearest competitor, Django, has an admin interface that works in a similar way, automatically generating edit pages based on the data model.

Web 2.0 is about data, about semantics. Web 2.0 is inherently declarative. So Web 2.0 applications can be written declaratively - Web 2.0 mashups can be just wired together and their data animated by business rules. A bit like programming spreadsheets.

Teqlo, Coghead, Pipes, DabbleDB, LongJump, Popfly, AppExchange and Wyaworks are all examples of the different ways to program the new Web 2.0 platform without imperative code.

That's what we mean by Web-as-platform - not only is the underlying programming language irrelevant, it will often not even be needed, certainly for simple data manipulation applications and for many simple mashups. Being RESTful gives you a massive head start in this, of course.

While Rails is already in the game with its innate understanding of Web 2.0 techniques and philosophies, Ruby itself has a huge amount to offer the would-be declarative programmer, who is making the transition to this new Web platform from their traditional Java or .Net platform. In particular, it is easy to write your domain logic in a declarative style in Ruby: they call them 'DSLs' these days, but the idea is the same in most examples I've seen.

Web 2.0 - The Web Redux

Now, if you've been following this blog, you'll know I have a few opinions on Declarative Web 2.0 and on patterns for programming REST. Essentially I argue that, if you want to play in the Web 2.0 platform game, you don't want to be writing screeds of Javascript functions that call more functions on your servers.

I recently presented some ideas along these lines at the WWW2007 conference, entitled 'The Micro Web: putting the Web back into Web 2.0', where I also showed a demo written in Python.

This approach combines my Distributed Observer Pattern with Comet push to enable highly dynamic Web 2.0 applications to be coded RESTfully and declaratively, with zero Javascript. The Distributed Observer Pattern offers a clean programming model for animating the Web 2.0 dynamic-data technology set I described above.

I believe the Observer Pattern is core to the way we'll be programming when the Web 2.0 Platform hits mainstream. It enables the kind of event- and rule-driven programming that matches the characteristics of the Web 2.0 dynamic data platform. As a further killer benefit, it also directly addresses the optimal utilisation of multicore processors.

I am currently porting my Python implementation of this approach to Ruby, in the Redux project on Rubyforge. Redux stands for 'Ruby Event-Driven Update Exchange'. It uses the highly scalable EventMachine epoll-based event loop to power its event-driven architecture. This will be essential when Redux is asked to scale up a Comet-based application.

Like Rails, Redux will be a Web (2.0) application framework, but unlike Rails, it puts the Observer Pattern and event- and rule-driven programming at its core.

Redux's headline is 'Web 2.0 in-a-box' or 'Naked Objects on the Web'.

Conclusion

If you're in BigCo, and are responsible for setting BigCo's technical strategy, then train your Java devs up on Web 2.0 core technologies such as Ajax, JSON, Atom, Microformats and OpenID.

And fire up their enthusiasm by tapping into Ruby (perhaps via JRuby) on your way to the Web 2.0 platform.

Learn patterns for mashing and integrating. Learn about REST and event- and rule-driven programming, including declarative DSLs.

When this Web platform hits BigCo, you will probably find that its REST or ROA style make your SOA integration strategy look rather complex and unweildy.

Check out the Distributed Observer Pattern, and download Redux when it's done (I'll let you know if you subscribe here!).

In 2007 and beyond, its the Web itself that's the platform, not Java or .Net. But if you want to get there via a language-based platform, Ruby could be the best way to transition to it.

Note: Everything I said about Ruby and Rails applies equally in technical terms to Python and Django, but regardless of the significant benefits of the latter, Ruby and Rails have the Web 2.0 market and mindshare. I'll probably switch this blog from Django to Redux sometime this year..

The Distributed Observer Pattern | The REST Dialogues

2007-06-20T22:42:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 5: The Distributed Observer Pattern

eBay Architect: So, can you summarise your argument that 'REST isn't just about reading and writing data', and explain your view on RESTful business logic?

Duncan Cragg: OK. The whole collection of related resources determines where things stand at any given time.

Resources are masters of their own destiny - guided by rules declared in the standard to which their content type conforms.

These rules, or business logic, run on notification of any declarations of the state of peer resources, or on arrival of any state via POST. Such peer states and POSTs are not commands, although it is possible to go ahead and define a special command or edit command content type.

The rules aim to satisfy the business or domain constraints on the mutual states of these resources - updating and creating resources accordingly and causing appropriate side-effects outside the server, such as financial transactions and emails.

These transformations are state driven. Even though the 'tension' in unresolved rules may be detected by events, that tension exists, not in those events as such, but in resource state.

eA: That sounds like a core difference to SOA.

DC: Indeed. It's a Resource-Oriented Architecture. And ROAs are declarative, not imperative like SOAs.

We have a world of resources declaring their current state, and resources settling into new states depending on the current state of related resources. These state changes can be driven by hard-coded resource animation logic, or by simpler, clearer, more scalable, declarative state transformation rules.

eA: Remind me of those patterns for notifying state change.

DC: Resource states are either polled via GET or actively notified via POST. Such actively POSTed state could be from a resource that also happens to be GETable, could be simply a link to such a resource, or could cause such a GETable resource to be created on the target server. Alternatively, the POSTed state could be considered too transient to record in a GETable resource, but can still trigger transformation in its target resource.

The above eBay examples used the pattern of 'server creates GETable copy of POSTed resource', and also 'second server hosts GETable copy of POST-notified resource'.

What I have described is a general programming model because, in general, such simple, declarative, transformational mechanisms are Turing Complete.

eA: I'm sure it's a novel perspective - even to RESTians! Again, do you have any high-level RESTian support for this?

DC: Any web resource that is a derivative of, or is dependent on, one or more other resources is using this approach.

Like I said before, there is an example of a similar approach by Joe Gregorio on his 'Well-Formed Web' site for alerting resources to peer resources of mutual interest.

Every time you would POST some data, consider making that data GETable and POST its URI instead, as a notification of the data existing.

eA: GETable POST data? You sure that's REST-compliant?

DC: In REST integration, things become more symmetric than in the client-server Web, or rather, the 'client-resource' Web. We can start to talk about the 'resource-resource' Web!

But anyway, we're already halfway to the symmetric resource-resource Web when we POST - not to a service, but to a URI. Resources can already both issue and receive state, which is a pretty symmetric state of affairs.

eA: I never thought of it that way - I keep forgetting that you can POST right back to a resource you just fetched.

DC: But think one step on: the POSTed data has a Content-Type but no URI!

Why not close the loop and have this POSTed data be a first-class resource (with a URI) that POSTs itself to the target. And it can itself GET that target or be POSTed to by that target in return.

That really is a Resource-Oriented Architecture. Once resources are seen as equal and active participants in RESTful integration, it becomes irrelevant whether their state is transferred by GET or by POST.

eA: I'm still having trouble with this pattern of POST just being a pro-active GET.

DC: Making POSTed data GETable more correctly moves the responsibility to the target resource to fetch the incoming resource state when its ready (rather than being bombarded by state it hasn't asked for).

Once the target is interested, updates can be POSTed directly as they happen, to prevent the target polling, or notification of an updated URI POSTed to trigger the target to re-GET the changed resource when it wants (thereby updating the caches).

eA: Hmm - makes clients look like servers..

DC: Since our 'clients' in REST integration are also 'servers' in other contexts, it is easier to set up client-side resources than on the browser-based Web. One objection to cookies on the Web is that they are state or resource that has no URI. So give your 'client' state a URI! And put any client-specific server resources on your own 'client' host.

eA: Is anyone doing this sort of thing?

DC: Well, in fact there are many examples of this POST-notification of a GETable resource already happening between web sites. Like submitting a link to your site to an indexing engine and letting it crawl (or poll) it.

Trackback pings are another example: POST a URI along with a sample of your page. And the Microformat rel-tag adds your article to Technorati's tag index when you ping their servers with the URI of the article.

Further, imagine POSTing to some new site a link to your hCard on your own server, to save you having to type your name and address again. And you'd never need to manually update sites when your address changes: just ping 'em all.

eA: Ah - but I thought all URIs should be GETable. The ping URI you're POSTing to in these examples isn't always one that you can also GET!

DC: Indeed - so think how much more powerful it would be if we did close the loop and provide or create a GETable resource to POST these notifications to.

For example, imagine a page containing an hCalendar event. Now point to it with a rel="attending" link. When the hCalendar discovers your intention (using a direct POST ping of your page's URI to the hCalendar page's URI - or perhaps through the referrer trick from people clicking through), it adds your referring page to a list of attendees inside the hCalendar. The hCalendar could either contain lists of backlinks to the attendee's pages, which may in turn carry hCards, or it could contain lists of complete hCards copied over.

eA: Sounds like a good use of Microformats.

DC: These examples make crawling and polling (even with If-Modified-Since et al) look like a clumsy version of the more proactive POST.

Web Feeds and general publish-subscribe are further examples where POST may be used to notify changes on a resource - giving the feed consumer first-class resource status with their own URI.

eA: I'd never think of using HTTP in this way.

DC: Obviously this only applies where the feed consumer is a visible and POSTable server and where timeliness is crucial. And probably where the number of subscribers is relatively small, unless asynchronous I/O and an event-driven architecture are employed, and you don't wait for the response to each POST.

This isn't done now simply because of the asymmetry of the current Web, an asymmetry which we are free of in REST integration.

eA: What about all those REST rules about idempotent and unsafe methods?

DC: We're not mixing GET and POST in that sense, just turning the tables on the asymmetric Web. GET is still cacheable, and we can POST a link to cause a cached GET.

I believe this is a more-constrained REST style, not disjoint to REST. It is at least an ROA! It may fall foul of REST's client-server constraint, since we're now in server-server territory with integration applications. Also, the concept of 'Hypertext as the Engine of Application State' is something that may take some refitting to the mutual state dependency model. However, I believe it's most important to focus on maintaining the benefits of REST and its key elements of standard content types at URIs.

I call this symmetric REST integration style the 'Distributed Observer Pattern'.

eA: Quickly summarise the 'Distributed Observer Pattern'.

DC: OK, the Distributed Observer Pattern is 'symmetric REST'. A resource subscribes to a peer resource via a GET that supplies its own URI, and is notified of subsequent state changes in that resource through a POST back.

eA: That was too quick. Tell me the details!

DC: OK, here are four. First, a POST can be either the whole new state or the fact of the change, allowing the subscriber to GET the resource when it's ready (and thereby fill any caches).

Secondly, you can use either the Referer header or perhaps the Content-Location header in POST and GET requests to indicate the origin POSTer or GETter URI. Alternatively, you can send this origin resource URI using the Cookie header, echoing its use in the normal browser client-server case to identify the pseudo-resource of a browser user.

POSTed state notifications may be unsolicited by a prior GET subscription, when the POST target is clearly open to them (as in the ping notification examples). These can be seen as 'subscribe to anyone', and may be combined with a corresponding 'GET anyone' crawling process, without explicit subscription.

Finally, POST notifications may be targetted to single resources to ask them to update: the Distributed Observer Pattern way of achieving the client-server editing function. These now become 'edit suggestions' of the POSTer resource - putting the target back in control of its own destiny and integrity.

eA: And why should I use the Distributed Observer Pattern?

DC: The Distributed Observer Pattern supports the programming model of inter-dependent resources whose own state is a function of their peers' state, driven by declarative rules. It's a very general ROA programming model.

In Part 6: Content-Types and URIs.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Lighter-than Wins in 2007

2007-01-18T11:12:00Z

What do all the MAJOR Web 2.0 technologies of 2007 have in common?

Let me list them first:

    M.icroformats (including tags)
    A.jax (including Comet)
    J.SON (plus YAML)
    O.penID (plus SXIP, LID, Yadis)
    R.EST (including Atom, APP)

What these technologies have in common is that they're all lighter than their competitors:

Microformats	Lighter than the Semantic Web
Ajax	Lighter than Fat Client (!)
JSON	Lighter than XML
OpenID	Lighter than SAML/Liberty Alliance
REST	Lighter than SOA

I'm reminded of Tim Bray's Technology Predictor Success Matrix and his appreciation of the 80:20 phenomenon. If hitting the sweet spot of achieving a lot with only a little is the main test of potential, then these five Web 2.0 technologies are all set for success.

So it's no surprise to see Tim supporting Microformats once or twice and even JSON, in spite of his, um, XML background. He also understands and uses Ajax and backs REST (in particular via Atom). Tim has not yet mentioned OpenID, instead referring to SAML and Liberty Alliance (please don't be crass by leaving a comment speculating on the reason for this...).

These MAJOR Web 2.0 technologies should form the basis of the two-way, interactive data Web or the Web-as-a-platform vision that we've all been promised.

However, they will need taking a little further this year, in order to bring them closer together and to make this vision happen in a seamless way.

A synergy between these 80:20 technologies can potentially produce much more than just the 'next version of the Web', if engineered well.

I have already discussed the Subversive Microformat, I've promoted Declarative Ajax and currently I'm slowly working up to Symmetric REST.

Next up, I'll be asking you to consider a web of JSON Microformats and ways to merge hCard and OpenID into active user personas - even avatars...

I'll tag this acronym as 'rajmo' instead of 'major', for obvious reasons.

Setting Data | The REST Dialogues

2006-11-15T23:37:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for one of the many function calls that they make available via SOAP (GetSearchResults).

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 2: Setting Data

Duncan Cragg: Now, let's look at those calls in eBay's SOAP API that are expected to change data. There is often a corresponding 'Get' function.

Examples are GetTaxTable, SetTaxTable and GetMyMessages, ReviseMyMessages, ReviseMyMessagesFolders, DeleteMyMessages, etc.

Here, there is an implicit model of data (or lists of data) that you can look at, modify, delete, etc.

eBay Architect: It's not just data: Tax Tables and Messages have meaning and behaviour internally to the server.

DC: Yes, but we're talking about what the API is telling us, here. It says things like GetTaxTable and SetTaxTable!

We'll come on to business processes later, but just look at the simple stuff first.

eA: OK, so what's the benefit of doing these calls in HTTP, through POST or whatever?

DC: Again, scalability and interoperability.

In the same way as with GET, you can partition your POST handling across the URI space. Have your Tax Tables and Message folders handled by many machines, split by their URIs. This is an example of the inherent parallelisability of the Web's architecture.

eA: We do partition by function: search, display, sell.

DC: Sure, but you had to 'hard-code' this partition: to use specific heuristics and code to achieve it: URI-based partitioning is much more generic and flexible, and can be much finer-grained. You seldom get single-point of failure or bottleneck issues with good REST architectures.

eA: If you say so.

DC: Interoperability again derives from the standardisation of the Content-Types and the schemas of POSTed data. When you know the meaning of the data you fetch, you also know how to try and change it, if you can.

eA: In HTML, you actually get presented a form in some GET data which explicitly tells you what to POST back - what the 'schema' is.

What's the equivalent in a general REST integration context where it's machines that are doing the POSTing?

DC: Well, the same applies, insofar as the GET data effectively tells you what any return data should look like. However, this time it's in an implicit way.

Where, in the read context, you understood what to do with a given content type you'd fetched, now, in the write context, you further understand what you can send back to it again.

eA: Can you give examples, please? The Tax Table perhaps?

DC: Straightforward data updating is the simplest case - it's the same content type coming out as going back in. You see a Tax Table at some URI, then you just PUT a new Tax Table back at that URI to try and replace it.

All the API function calls beginning 'Set' should probably be implemented this way.

eA: OK, so how does REST implement our Message API functions?

DC: An example of a data format that implies its edit capabilities by reference to a standard is Atom syndication.

You can GET a list of Atom entries, then POST a new one according to the Atom Publishing Protocol (APP) specification. You POST, not to a 'service' or handler, but to the URI of the actual collection you're adding to.

eA: So what's that got to do with our Messages?

DC: The eBay Message lists are a great candidate for using this approach, with the benefit that any (recent) feed reader can be used to view them, and an APP-compliant client used to manage them.

All the API function calls beginning 'Add' should probably be implemented in the same way APP adds new entries, or you should consider using APP itself to implement them.

eA: OK, so we can replace some calls with a standard set of REST interactions. But clients still have to know about Tax Table schemas, User Preferences schemas, Message schemas, etc.

It's not enough to simply know about HTTP and HTML to be interoperable any more, like a browser does. It's not even enough to just know APP.

Surely interoperability stops when the content types and schemas start to proliferate like this?

DC: Schema proliferation is SOA's problem - so I'm glad you brought it up! It's baked in to the SOA mindset that it's OK to design your own interfaces and schemas from scratch each time.

Conversely, the expectation and culture in the Web and especially in the REST-aware community is to constantly look out for opportunities to conform and to standardise schemas and interactions, and to build on layers below that are already standardised. It's a side-effect of the shareability of URIs.

With REST you get interoperability at many levels above the byte-transfer of HTTP. Content type understanding and standardisation can occur all the way up from characters to Tax Tables.

eA: I'm not sure I get this.

DC: OK: there is some code that understands basic HTTP GET and PUT, and UTF-8 characters - and doesn't need to know what a Tax Table is - some that understands UTF-8 plus XML, some that understands that plus Atom, or plus XHTML, then XHTML tables and Microformats like hCalendar, hCard (and hAtom!), then conference schedules using hCalendar and hCard. Tax Tables can perhaps use XHTML tables.

Clients may or may not need to know what the schema of a Message looks like internally in order to be able to do useful work with Messages at their level of understanding - from character stream up to APP.

eA: I'm not sure how general auction schemas fit in there.

DC: You never know, your eBay schemas might be taken into account when coming up with new standard content types for e-commerce!

eA: Hopefully. We could always put everything into XHTML tables for you!

DC: Ha! Go for it. Having done some HTML scraping myself, I'd welcome such semantics!

eA: Right, it's OK when we're just talking data that you can read and write - HTTP, APP, REST, whatever may be great for that - but our API has much more complex business functions in it that don't fit that simplistic model.

There's a whole load of functions in our API that aren't just about getting and setting data or collections of data. Some are eBay- or auction-specific.

DC: This is one of the Myths of REST - that it's just for simplistic reading and writing of data.

It's a actually a myth that's often propagated by the REST community itself, especially with their over-emphasis on the Four HTTP Verbs, which seem to map so conveniently onto Create, Read, Update and Delete.

eA: But they do map so well - I thought that was 'good' REST...

DC: It is often good, but can detract from the whole goodness!

One consequence of this mapping it that people inevitably go on to see an analogy between REST and databases, and then start to expect transactions and other database features.

Another consequence of the database analogy is that resources are seen as lifeless servants of the active client: it takes away responsibility from a resource to be master of its own destiny.

This then causes confusion (even within the REST community) about the power of the client and even the very meaning of such basics as the PUT method.

The fact is, GET and POST are more than enough to be REST compliant. This cut-down pair also help focus the mind on URIs, two-way content transfer, content type or schema and on the responsibilities of each resource as active players in an integration scenario.

eA: Sounds like a minority REST view - any support from a REST luminary for that?

DC: Tim Bray has a history of suggesting that GET and POST are enough.

And recently, there has been further high-level support from Sam Ruby and Leonard Richardson in their manifesto for an upcoming book.

eA: I bet you still get told off, but it sounds reasonable to me.

DC: So, to summarise: use GET to read data, then POST back to the same URI to suggest changes to the resource there.

All based on a given level of understanding and interpretation of the content type and any corresponding interaction standards.

Interoperability and scalability in a nutshell!

Now I will explain how there's more to REST than simple reading and (attempted) writing of resources. We're still only two-ninths done...

In Part 3: Business Functions.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Getting Data | The REST Dialogues

2006-11-14T00:05:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for one of the many function calls that they make available via SOAP (GetSearchResults).

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 1: Getting Data

Duncan Cragg: So - let's get straight to my argument: I claim that your SOAP APIs, as instances of the SOA style, won't scale or interoperate as well as they would if they were implemented in the REST style. Which, in the form of the Web, has largely proven scalability and interoperability.

eBay Architect: SOA won't scale or interoperate and REST will? You've got some explaining to do there: the whole industry is going SOA crazy, and we're following that trend. You'll need to give specific examples.

DC: OK.

eA: And even if it is true, we don't care about scalability and interoperability: we have quite low traffic on our APIs - certainly compared to port 80; plus we're eBay so we don't need to interoperate with anyone.

DC: That's true now, but if the Web 2.0 vision comes together, you may care: your API traffic could increase dramatically. It would be better to be the one prepared for the scale of the API-Web!

Can you really argue in your company that you don't need to be scalable? What if your port 80 traffic needs to be routed to your APIs for some reason?

eA: Maybe.

DC: As for interoperability, you could be excluded from Web 2.0 industry-boosting consortia, or excluded from perhaps hugely popular Web 2.0 applications in the future...

Interoperability raises the level of the market as a whole. Market players shouldn't differentiate on what's common to them, they should differentiate on the level above.

It also depends on the value you place on having happy customers who don't have to do the same thing multiple ways or multiple times.

eA: Ha! I'm sure there's a name for that style of argument!

So, let's say we want to be good scalable and interoperable citizens in a future Web 2.0 world: show me how REST can help us more than SOA and SOAP do!

Getting Data

DC: OK, let's look at your SOAP API. There are 72 function calls in there that begin with 'Get'. Each one specifies a particular piece of data that you can fetch.

eA: Yes, our API is extremely rich.

DC: Sure, but you don't need a new function call for everything you can get from your system: you can just use HTTP GET!

eA: OK, but it's just data going in to request the information, then data coming back out again. What difference does it make whether we use HTTP or SOAP to carry those messages? You've just moved the name onto your URI!

DC: It's not just any 'data going in': the URI can be passed around for anyone to re-use. This URI is more interoperable because so much deployed software understands it. No-one understands 'GetSearchResults()'!

eA: OK.

DC: Another example of how the URI can glue things together is that the data returned from your GETs can have more URIs in them, ready to go! You won't get data from your Web Service with 'GetItem()' in it..

eA: Heh - I s'pose not.

DC: REST also talks about the formats of the data behind a URI. In a GET, the response data is given a Content-Type, and there's an expectation that clients will understand the types of data being returned: interoperability comes from broad standardisation of return data.

eA: But we return XML, and our schemas are published. They're specific to our application anyway. That wouldn't change with REST.

DC:The explicit statement of Content-Type reflects a culture of agreement forced by the sharability of URIs: your URIs are more sharable when more clients understand the data they dereference to.

On the other hand, the culture of SOA is to declare custom WSDL and custom XML schemas.

Like I said, one day you may care about interoperability, and having an architecture that puts a high value on content type and schema standardisation, as REST does, puts you one step ahead.

eA: We'll see.

DC: You can also gain scalability by partitioning on those URIs.

eA: We already do plenty of partitioning of our data and functions.

DC: Yes, but URI partitioning cuts right through the system in a very simple way: your partitioning is an application-specific optimisation which has to be hand-coded behind the SOAP interface.

eA: Mmm. Maybe.

DC: Another benefit of using HTTP over using SOAP is that you get cacheing built in to the architecture, which you can start using as soon as you ask for it in the headers. This boosts scalability.

eA: OK, that's fair. Our clients would have to write their own caches, should they need them. But in general we advise against cacheing our data and don't put expiry on them or have not-modified checks.

DC: Which is where you're potentially inefficient.

eA: Maybe. We internally cache things like the data behind GetCategories, and advise our users how to cache it themselves.

DC: Again - it's application-specific.

So - even in the simple cases of fetching data, REST has given you much greater scalability and interoperability than your SOAP interface - as well as a simpler, more generic approach.

And we're only one-ninth of the way through our conversation!

eA: How do you know that??

DC: Aaah...

In Part 2: Setting Data.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...