eBay Architect: Now, I have a few more questions about the areas where REST is generally considered to be weaker than SOA. I'll start with a big list, then we can go through each one.

Duncan Cragg: OK, go ahead!

eA: Right - it's basically the list of techniques you need to architect complex distributed applications: Service Discovery, Service Description, Client/Server Session State, Business Process Coordination, Workflow, Orchestration, Choreography; Security, Reliable Messaging and Transactions.

DC: Phew! Well, some of these beg the question; by assuming you need them, you end up concluding that you need SOA!

Also, seeing WS-standards for these things, some people think that SOA has them covered and that their work is done. But you still have to design and architect your system on top of these standards, to meet the business need.

In REST or ROA, we're talking about a different way of seeing distributed systems that requires a break from Service- (and even Object-) Oriented thinking and an embrace of Resource-Oriented thinking.

eA: So what's the REST way of approaching my list?

DC: Do things the REST way and there are three possibilities: (a) that it's easy to implement these functions in REST; (b) that you don't need these things anyway, or (c) that it's already your job to do them as an application architect; it's part of your business process, so you do work that you needed to do anyway in SOA, but do it in a simpler and more powerful way.

eA: Already my job?

DC: It's your job while you're designing your resource interactions at the application level: the resource-type or business level. Or maybe while you're making use of some existing resource types and 'resource-animating' code that does your business logic for you. Having WS-* only clouds and complicates this task.

eA: You've just hand-waved away a billion-dollar standardisation and tooling industry!

DC: Standards and tools are essential - we need them. But simple standards and tools are best. HTTP, URI, XML and standard schemas are perfectly good enough standards to get started with, guided by the REST or ROA mind-set.

I'll grant that we do need some extra help to support RESTful resource interaction in our domains and applications. I would like a few more standards and tools, over and above HTTP and XML, to help me build these complex distributed applications. Some help configuring and programming the way an application's resources interact at a business level.

eA: What exact support would you like for REST integration?

DC: We would benefit from standards, tools and frameworks that help define resource URIs, mapping to GETable and POSTable URIs, easy cache control, easy support for various response codes, HTTP Auth-based and URI-based identity, inter-resource links, collections, content types, Microformat support, useful schemas, structures, sets of simple default behaviours, APP libraries, business rule engines, etc.

It is still early days for the adoption of REST - an adoption that's undoubtedly been slowed by the industry's obsession with SOA. We do have Restlet and the JSR 311 work, as well as some help from Microsoft in WCF and the ADO.NET Data Services Framework, but there is still much more that could be done.

eA: Indeed. There's lots of work to do to catch up with all the SOA standards, tools and frameworks!

DC: I really don't want ROA to compete with SOA in bureaucracy and vendor in-fighting! But REST needs much less than SOA. Some things are simply part of REST: you discover stuff by following links; you know what something can do for you because of its content type or schema - we went over this sort of thing before.

Service Discovery and Description

eA: So, following links and seeing content types cover REST 'Service Discovery' and 'Service Description'?

DC: Yes. When you start thinking in terms of resources not actions, it all becomes much clearer:

In real life you might walk up the street, ask someone where the nearest shoe shop is or consult a directory, then see a likely shop, decide it may have what you want, then go in to investigate. Inside, you may consult more directories or just wander around. You see the cues around you (shoes on offer, checkout desk), and act or react accordingly.

You engage at a chosen level of understanding: there is a generic concept of street layout, then generic shop and shop assistant understanding, then an understanding of how to view and interact in shoe shops in particular.

eA: And in 'virtual life'?

DC: The Web is exactly the same: you explore through links or by asking a search engine, see what you want, enter the online shop, follow the cues, etc. It's no accident that the cyberspace metaphor is often applied to the flat, 2D Web.

eA: And in REST integration?

DC: In REST integration, the metaphor can be extended for machine interaction: get a list of items by direct reference or by query, go through them finding something that matches, interact according to standard business types and follow links between business resources.

eA: So no service contracts for you, then, just exploring and hoping for the best. Sounds a bit sloppy!

DC: You can do anything with anything in computing, it's just a matter of what metaphor is the most powerful or expressive.

You can design your 'machine cyberspace' to include contracts if that's what you need, just like in real life, and just like we went over with the eBay/gBay example.

But in general - especially when building distributed systems - it's best to start with loose arrangements and established conventions; adding constraints, contracts and Central Control only where absolutely necessary. Ensure central control only over schemas.

Make your Enterprise 'mashable' and everyone will thank you. Expose the UUIDs and GUIDs of your data in URLs. Transform and enrich your data within standard content types. Link it all up with your own data and data from elsewhere. Allow two-way interaction. Then publish it for future generations to discover and re-use!

Client State and Sessions

eA: Right, now how do you model complex state transitions, state evolution and conversations within a changing context, in a supposedly stateless architecture?

DC: There's a common misunderstanding about REST that its statelessness should extend above the protocol (e.g., HTTP) into the application. On the contrary - once above HTTP and into a Resource-Oriented application, it's all about state - as long as that state has a URI!

All the stateless requirement is saying in practical terms is that each HTTP request and response exchange is a one-off as far as HTTP is concerned. It means that you don't need to tie successive exchanges together at the protocol level, which makes implementing HTTP easier. HTTP just wants a URI to be a URI, and the content there to be manageable via headers; and HTTP doesn't and shouldn't introspect either the URI or the content.

eA: So where do you keep the conversation state, then?

DC: Above HTTP in the world of URI-tagged state. You can have an ongoing 'conversation' between a client and a server resource - as long as those resources are linkable and fetchable.

You don't need sessions below the resource level; below the business level. If your application truly demands the concept of a sequence of interactions with a start point and an end point, then go ahead and implement it - at the application or domain level, not at the framework level.

Most applications can use ad hoc, asynchronous interactions, where the client identifies itself each time with Authorization headers.

eA: Can each client machine have assigned to it dedicated server resources as part of this conversation, sort of like an explicit session state?

DC: Yes. Cacheability and linkability are clearly affected, but if it truly makes sense in your domain model to have client-specific resources, then just do it - by definition their scalability and findability is limited to that client! The client can still cache its own view, as in a browser, and even intermediate proxies can, although it's of benefit to only one client.

eA: Presumably it's a bad idea to put client state in a Cookie header?

DC: Yup: it's not got a URI, so its state is hidden. Put it on a URI in a server.

Hidden state is a red flag.

You know you're on the right track as long as you are exposing your statefulness in URIs. It should always be obvious where things stand from inspecting public state alone, to know what is possible and what will happen next.

If you find yourself hiding state in sessions and cookies, or returning different data dedicated to that client from the same URI by setting no-cache, or if you tie up successive interactions through sessions, you are going down the wrong track.

eA: Are cookies always bad?

DC: A Cookie header should be used only to identify the client, not a session. You'd probably use the Authorization header in REST integration anyway. Just use Cookie headers along with an auth scheme if you need a more elaborate, perhaps multi-layered or proxied, authentication system.

eA: What about the Vary response header and its use with Authorization or Cookie request headers?

DC: Mechanically, from the cache perspective, adding Vary on client-identifying headers like these is the same as using per-client URIs; setting the Vary header to allow the URI to be cached on a per-client basis is a hidden way of effectively adding the identifying data from the Authorization or Cookie header to the URI itself.

Actual per-client resources are explicit; these Vary'd resources are implicit. You choose which to use at the application level based on the importance of the resources being identified in your domain model. Either way, the identity of the requestor can be used to determine read permissions.

The explicit way means you can pass links to your personal resources to others, modulo read permissions. A client may or may not be able to or want to pass the link to its dedicated resources to other players - it probably isn't a link they want anyway - but they can still pass around general jumping-off links, that make sense for other recipients.

eA: To be honest, I just don't like the idea of the server saving and distributing a ton of this client-specific resource data!

DC: There are two answers to this: firstly, now that we're using REST for business data integration, not generating entire pages of HTML, the chunks of data can have much finer grain. The actual amount of client-specific data can be much smaller! You can cache and share links to all the generic chunks, and only transfer small amounts of less cacheable and linkable per-client data.

Secondly, our clients in REST integration are also often servers, so can in fact expose their own state. We may be better off just putting those client-specific resources on the client itself, to get around this whole issue. The client isn't a browser any more!

We're back to the symmetric Distributed Observer Pattern.

Business Processes

eA: OK, now what about Business Process Coordination, Workflow, Orchestration and Choreography in that sloppy cyberspace of yours?

DC: We've already discussed the auction business process, both single-site and cross-site, which was quite a good example that covers a number of business activities.

Another good feature of REST is the way it naturally maps onto declarative business rules. When you switch from process-thinking to resource-thinking, you also switch from imperative thinking to declarative. This manifests as inter-resource dependency and transformation.

eA: Eh?

DC: A spreadsheet is an example of this style, where you declare the way cells depend on each other and then let the hidden magic take care of satisfying your constraints. We've discussed this style of programming before.

Again, tools will be essential to help with this. Managing and testing event-driven business rules over shared and distributed state is somewhat new territory, even to most REST integrators!

eA: Business Processes and Business Rules seem like different things to me.

DC: Well, as we saw, the business process of an auction can emerge from the local application of business rules. The thing is to let go of the myth that you benefit from specifying and controlling some over-arching vision of your business processes - that it makes sense to centralise control of what actually works best when delegated and decentralised.

Just write the local What, not the global How, and let the process emerge!

eA: Might be a hard sell to control freaks.

DC: Yeah, true enough. But it more closely maps onto the reality of the way businesses operate and interoperate. It's more about the actual peer-to-peer business interactions and visible state evolution, and less about central controllers that know the intimate, inscrutable details of the Web Services involved.

Instead of coordinating the import and export of data from one hand-coded interface to another, you can just link it all up and expect everyone to dereference and recognise your data. Instead of coordinating sessions with implicit state, you can just react to standard, public data types.

(c) 2006-2009 Duncan Cragg

In Part 8: WS-Are-You-Sure (Security, Reliable Messaging and Transactions).

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...