Setting Data | The REST Dialogues | What Not How

About All Things...

Declarative, Mobile 2.0, REST, Cloud, Web 2.0, Ajax, Publish / Subscribe, Event-Driven Architectures, JSON, Atom, Microformats, Linked Data, P2P, Identity, Copyright, Multimedia, Cyberspace.

...taking programming beyond:
Threads, Message Queues, Client-Server, CORBA, Web Services, SOAs, Agents, Synchronous Architectures, Imperative Programming - and even Applications, Desktops and Documents

Duncan Cragg...

...works for ThoughtWorks UK; originally from April 2002 to July 2007 and now recently re-joined. Previously worked as a Web Architect for the Financial Times.

...went to both UCL and Imperial College of the University of London (in the Eighties); specialising in Logic during his MSc.

...wonders when his LinkedIn Account will be useful

...has a phone-cam, and used it on himself once, just before his weekly shave:
Photo of Duncan Cragg

...can be contacted by and followed on Twitter.

Setting Data | The REST Dialogues

November 15, 2006 23:37

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for one of the many function calls that they make available via SOAP (GetSearchResults).

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 2: Setting Data

Duncan Cragg: Now, let's look at those calls in eBay's SOAP API that are expected to change data. There is often a corresponding 'Get' function.

Examples are GetTaxTable, SetTaxTable and GetMyMessages, ReviseMyMessages, ReviseMyMessagesFolders, DeleteMyMessages, etc.

Here, there is an implicit model of data (or lists of data) that you can look at, modify, delete, etc.

eBay Architect: It's not just data: Tax Tables and Messages have meaning and behaviour internally to the server.

DC: Yes, but we're talking about what the API is telling us, here. It says things like GetTaxTable and SetTaxTable!

We'll come on to business processes later, but just look at the simple stuff first.

eA: OK, so what's the benefit of doing these calls in HTTP, through POST or whatever?

DC: Again, scalability and interoperability.

In the same way as with GET, you can partition your POST handling across the URI space. Have your Tax Tables and Message folders handled by many machines, split by their URIs. This is an example of the inherent parallelisability of the Web's architecture.

eA: We do partition by function: search, display, sell.

DC: Sure, but you had to 'hard-code' this partition: to use specific heuristics and code to achieve it: URI-based partitioning is much more generic and flexible, and can be much finer-grained. You seldom get single-point of failure or bottleneck issues with good REST architectures.

eA: If you say so.

DC: Interoperability again derives from the standardisation of the Content-Types and the schemas of POSTed data. When you know the meaning of the data you fetch, you also know how to try and change it, if you can.

eA: In HTML, you actually get presented a form in some GET data which explicitly tells you what to POST back - what the 'schema' is.

What's the equivalent in a general REST integration context where it's machines that are doing the POSTing?

DC: Well, the same applies, insofar as the GET data effectively tells you what any return data should look like. However, this time it's in an implicit way.

Where, in the read context, you understood what to do with a given content type you'd fetched, now, in the write context, you further understand what you can send back to it again.

eA: Can you give examples, please? The Tax Table perhaps?

DC: Straightforward data updating is the simplest case - it's the same content type coming out as going back in. You see a Tax Table at some URI, then you just PUT a new Tax Table back at that URI to try and replace it.

All the API function calls beginning 'Set' should probably be implemented this way.

eA: OK, so how does REST implement our Message API functions?

DC: An example of a data format that implies its edit capabilities by reference to a standard is Atom syndication.

You can GET a list of Atom entries, then POST a new one according to the Atom Publishing Protocol (APP) specification. You POST, not to a 'service' or handler, but to the URI of the actual collection you're adding to.

eA: So what's that got to do with our Messages?

DC: The eBay Message lists are a great candidate for using this approach, with the benefit that any (recent) feed reader can be used to view them, and an APP-compliant client used to manage them.

All the API function calls beginning 'Add' should probably be implemented in the same way APP adds new entries, or you should consider using APP itself to implement them.

eA: OK, so we can replace some calls with a standard set of REST interactions. But clients still have to know about Tax Table schemas, User Preferences schemas, Message schemas, etc.

It's not enough to simply know about HTTP and HTML to be interoperable any more, like a browser does. It's not even enough to just know APP.

Surely interoperability stops when the content types and schemas start to proliferate like this?

DC: Schema proliferation is SOA's problem - so I'm glad you brought it up! It's baked in to the SOA mindset that it's OK to design your own interfaces and schemas from scratch each time.

Conversely, the expectation and culture in the Web and especially in the REST-aware community is to constantly look out for opportunities to conform and to standardise schemas and interactions, and to build on layers below that are already standardised. It's a side-effect of the shareability of URIs.

With REST you get interoperability at many levels above the byte-transfer of HTTP. Content type understanding and standardisation can occur all the way up from characters to Tax Tables.

eA: I'm not sure I get this.

DC: OK: there is some code that understands basic HTTP GET and PUT, and UTF-8 characters - and doesn't need to know what a Tax Table is - some that understands UTF-8 plus XML, some that understands that plus Atom, or plus XHTML, then XHTML tables and Microformats like hCalendar, hCard (and hAtom!), then conference schedules using hCalendar and hCard. Tax Tables can perhaps use XHTML tables.

Clients may or may not need to know what the schema of a Message looks like internally in order to be able to do useful work with Messages at their level of understanding - from character stream up to APP.

eA: I'm not sure how general auction schemas fit in there.

DC: You never know, your eBay schemas might be taken into account when coming up with new standard content types for e-commerce!

eA: Hopefully. We could always put everything into XHTML tables for you!

DC: Ha! Go for it. Having done some HTML scraping myself, I'd welcome such semantics!

eA: Right, it's OK when we're just talking data that you can read and write - HTTP, APP, REST, whatever may be great for that - but our API has much more complex business functions in it that don't fit that simplistic model.

There's a whole load of functions in our API that aren't just about getting and setting data or collections of data. Some are eBay- or auction-specific.

DC: This is one of the Myths of REST - that it's just for simplistic reading and writing of data.

It's a actually a myth that's often propagated by the REST community itself, especially with their over-emphasis on the Four HTTP Verbs, which seem to map so conveniently onto Create, Read, Update and Delete.

eA: But they do map so well - I thought that was 'good' REST...

DC: It is often good, but can detract from the whole goodness!

One consequence of this mapping it that people inevitably go on to see an analogy between REST and databases, and then start to expect transactions and other database features.

Another consequence of the database analogy is that resources are seen as lifeless servants of the active client: it takes away responsibility from a resource to be master of its own destiny.

This then causes confusion (even within the REST community) about the power of the client and even the very meaning of such basics as the PUT method.

The fact is, GET and POST are more than enough to be REST compliant. This cut-down pair also help focus the mind on URIs, two-way content transfer, content type or schema and on the responsibilities of each resource as active players in an integration scenario.

eA: Sounds like a minority REST view - any support from a REST luminary for that?

DC: Tim Bray has a history of suggesting that GET and POST are enough.

And recently, there has been further high-level support from Sam Ruby and Leonard Richardson in their manifesto for an upcoming book.

eA: I bet you still get told off, but it sounds reasonable to me.

DC: So, to summarise: use GET to read data, then POST back to the same URI to suggest changes to the resource there.

All based on a given level of understanding and interpretation of the content type and any corresponding interaction standards.

Interoperability and scalability in a nutshell!

Now I will explain how there's more to REST than simple reading and (attempted) writing of resources. We're still only two-ninths done...

In Part 3: Business Functions.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Permalink for "Setting Data | The REST Dialogues"

Reader Comments

From Daryl

Duncan-

Subscribed.

Excellent posts, glad I found your blog. Please keep it up.

Does your title refer to Date's "What Not How?" I try to reread it periodically- it's an awesome reality check against so much of what we do in the IT world still to this day.

Best Regards.

November 17, 2006 23:44

From Alex James

Good stuff...

November 18, 2006 08:04

From Brad Neuberg

Your series has me hooked! Email me when you get the third installment going.

Best, Brad Neuberg

November 18, 2006 10:46

From Pete Lacey

Would you mind educating me a bit?

I've seen the advice for reusing well known formats before, and it makes sense. But when confronted with the need to present information for which no suitable content type exists, why bother wrapping things up in, say, XHTML. In other words, what makes this:

<table>
  <tr>
    <td>First Name</td>
    <td>Pete</td>
  </tr>
</table>

better than this:

<FirstName>Pete</FirstName>

November 18, 2006 15:50

From Duncan Cragg

Daryl:

'What not How' comes from the general declarative tradition, and is not a reference to that book.

I'm now off to read it, of course. Thanks!

November 18, 2006 20:29

From Duncan Cragg

Pete:

As programmers we're used to doing things 'our way' and having things nice and pure, so the second form looks and feels much more natural to us.

However, when we open the door to the cold, cruel outdoors, we can't always have things our way. Or, if we choose to, we are simultaneously choosing to cut ourselves off from our peers out there. So we sometimes have to accept ugliness for the sake of interoperability.

The obvious example is actually illustrative: in a cold, cruel browser you'd get this:

Pete

instead of this:

FirstName

Pete

I don't know if you've been following the Microformats movement, but they are facing this kind of thing straight on, and setting standards of behaviour we'd all do well to emulate. Using class= seems clumsy compared to having a new element tag, for example, but it's Good Behaviour.

I'd say as a rule of thumb, if XHTML strict complains, you're probably losing interoperability. I hate XHMTL strict, but I use it for this blog - and for ThoughtWorks projects when I can. I also hate coding conventions, but stick ruthlessly to them when I arrive on a project.

This point strikes at the heart of the appeal of the SOA approach: write your own WSDLs and schemas! Have it your way! It's appealing because it gives us what we've always had: it maintains the illusion that we can keep control, even when talking across the wire. Sometimes that's actually true. But the Web has shown that, to survive in the bigger, wider world, it's not.

With your permission, I may weave this exchange into one of the dialogue articles. Please push back on what I've said as hard as you like, to make this stuff as robust as possible in the face of argument in the real world...

November 18, 2006 20:58

From Ben Askins

Fantastic series, I'm looking forward to the next installment.

November 18, 2006 23:16

From Pete Lacey

I'm going to push back, but gently.

Here's what doesn't work for me: the choice of XHTML seems arbitrary; a last ditch attempt to use a content type that's at least one level away from XML. The thing is, for representing the resource in question, it's been (in our fictional example) determined that no perfect format pre-exists, and that would include XHTML, which is for display purposes, not for "FirstName" purposes. Besides, it's not like there isn't a text/xml MIME type.

Admittedly, if the resource were retrieved by a browser, XHTML has the positive effect of rendering properly. However, when retrieved by a Perl script, the XHTML is just cruft that has to be scraped away, and, arguably, defeats the self-describing nature of XML while complicating the code.

Finally, in the context of your post, the XHTML doesn't imply what I can POST back (unless it includes a form). Or is my choice of example just bad?

I would recommend a middle ground of providing multiple representations of a resource based on theURL. Something like this:

whatever.com/whatsitz.xml whatever.com/whatsitz.xhtml whatever.com/whatsitz.json whatever.com/whatsitz.jpg

And, of course, if any of this is valuable in your future dialogs, feel free to use it.

Pete

November 19, 2006 00:44

From Mr Coffee

My god I've never seen so much theoretical blah in a blog posting. There are so many things wrong in this article that I don't know where to start.

I'll pick one because my wifi credits are about to run out.

eA: OK, so what's the benefit of doing these calls in HTTP, through POST or whatever? DC: Again, scalability and interoperability.

You seriously think a GET or POST is more scalable? Maybe you can explain how and then think it over and come to the conclusion that REST is scalable in EXACTLY the same way as SOAP or XML-RPC. Think not? Then post your ideas about scalability. I'll show you how to do exactly the same with WS-* or XML-RPC based services.

Interoperability is also an intesting one. I wonder if you have ever used a SOAP client because the claim is so ridiculous that it makes me wonder about how much real world code you have seen.

Interoperability is reached by using standard ways of calling remote methods and using standard encoding for sending data back and forth. SOAP is all about setting a standard while REST is all about reinventing the wheel over and over again.

You claim that 'Content Types will arise' for your particular application. This is nonsense. Just count how many there actually are now.

But back to reinventing the wheel ... aren't you tired of implementing a new XML parser each and every time some silly Web 2.0 startup 'goes REST'? I sure am. Why can't these fragging companies simply settle with something simple and standard like XML-RPC or SOAP. (The first clearly having my personal preference though). All they have to do is publish WSDL or simple description of their XML-RPC API and everybody will be happy.

Take Amazon .. they publish WSDL for their SOAP based services. It is two seconds of work to create an endpoint for those services. You generate Java, Python, Ruby or Cocoa endpoint classes within seconds. Done. Compare that to hours or days to support some kind of new xml format that some 'bright' Web 2.0 kid invented.

I love REST for doing simple things like PUT a file in Amazon's S3 and then GET it back through an AJAX call. But reinventing RPC every time is not only plain silly, it also a big waste of time. My time, your time, everybody's time.

Why is there no coffee on this airport.

Mr C.

November 19, 2006 17:49

From Duncan Cragg

Pete:

When I said 'The obvious example is actually illustrative: in a cold, cruel browser..', I obviously wasn't very clear about my intentions! I was using the 'obvious example' of pushing your table into a browser to show how some common code out in the cold, cruel world gives you better results when you do things it understands already.

Another example would be my spreadsheet, which can read an HTML table directly into cells.

The point is, it's just data, and if it's a table, use a standard table format.

XHTML isn't just a presentation language on top of XML - the presentation part is supposed to be added by CSS.

If XHTML gives you a structure you can use, use it.

If Microformats on top of XHTML gives it to you - use it!

In fact, use an hCard for your example!

If you can find a better example, then I'd love to discuss that. That is, where no widely-deployed format exists either at a basic, structural level (it's a special type of table, but it is a table, so may as well use HTML tables), or with some quite domain-specific standard that a reasonably experienced practitioner in that field would immediately expect you to use.

But basically, if there's no Microformat and it isn't actually a table and you've run out of schemas to conform to - you're in a minority and you're entirely welcome to write your data in XML using your own tags. We've got a rare case where we simply have to write our very own schema. And hope everyone else from now on uses it!

At least by using XML as our common jumping-off point, we achieve a fair bit of interoperability with existing tools. Granted, no more interoperability than SOA - but like I said, it's going to be quite rare - especially as Microformats and REST take off.

As to the POSTability of standard content types: this is definitely specific to a few examples as yet - I am simply selling the principle of interoperability of POSTs once more people start doing things the REST way. We'll see how this pans out, but I'd watch the Microformats and Widgets people for setting standards in two-way REST. Especially if I have my way!

In the meantime, Atom Publishing Protocol is a reasonable place to start for many things, and can be used as a model for new work.

See my article here for the tiny list of RESTful two-way APIs that I discovered earlier this year. We still have much work to do!

But this shouldn't stop you from doing your own integration RESTfully and publishing the standards as widely as you can for comment, improvement and general adoption.

Finally, yes, you can have multiple URIs with different derivations of the same data, as long as there is one canonical form, of which the others are all straightforward transformations.

But having the raw data version and presentation versions is not the same partitioning as a set of more semantic derivations. You should keep them separate, and even consider why you're doing it. Like I said, XHTML is not a presentation format, so you probably don't need XML, JSON and XHTML. And the image derivations (where's SVG?!) would probably be done by different user groups according to their own need.

November 19, 2006 21:55

From Pete Lacey

Well, that sums it up very nicely, thank you. If there's room in the ongoing dialogs for this level of information, make sure it's included so as to surface this out of the comments. You might also want to make sure that Leonard Richardson is aware of it, for inclusion in the REST book he's writing with Sam Ruby.

Pete

November 19, 2006 23:25

From Gregory Brown

This series is wonderful. I'm doing a bunch of SOAP / REST work lately and this is really hitting on a lot of the key points about the architectures. Keep it going! :)

November 20, 2006 16:00

From Mr Coffee

Dude, where exactly did you get the idea that XHTML is not a presentation format?

Mr C.

November 20, 2006 20:12

From Duncan Cragg

A quick note for the above discussion: I just rediscovered the article Don’t Invent XML Languages by the essential Tim Bray.

Also, see Namespaces Considered Harmful.

November 20, 2006 22:12

From Dan Pritchett

As a real eBay architect, I have risen to the challenge of discussing REST with Duncan. You can catch part 2 of the interview with me at my blog.

November 22, 2006 04:17

From Duncan Cragg

In case anyone wondered:

• I am in touch with Dan Pritchett and fully support (in fact, I'm delighted to get) the additional feedback!

• I have part 3 (and parts 4-9!) ready, but have held off a bit to allow Dan time to produce his own part 2.

• I've no idea how I'm going to respond to his responses (mechanically, not rhetorically)!!! =0)

Duncan

November 22, 2006 17:28

From Ximon Eighteen

Nice one Duncan.

November 25, 2006 20:14

From Bjorn B.

Really catching, I'm looking very much forward to the continuing story

November 26, 2006 22:49

From Smies

Great, finally a clear story about how to do REST :D.

November 27, 2006 18:29

From Bruce Stockwell

Duncan, get off the can and start writing! I need more of this!

December 5, 2006 18:34

From Philippe Lachaise

Well, I'll say like the other did : I can't wait for the REST of the story !

Thanks.

December 8, 2006 12:46

From bex

@Philippe: that hurt my brain.

Maybe you folks would like to see what a real EBay architect has to say about this:

link

December 11, 2006 20:32

From an anonymous correspondent

Great series! Looking forward the the next installment when you have a moment to breathe :-)...

December 13, 2006 20:53

From David, biologeek

Thank you Duncan (and Dan) for those interesting dialogues! I had to implement some kind of API soon and it saves me a lot of time, really. I hope one of the next dialogue will give some concrete example. Maybe eBay will engage you Duncan, who knows :-).

PS : Duncan, why don't you answer Mr Coffee? (first comment) I think it can be intersting too.

January 19, 2007 13:43

From Bernd

The fact is, GET and POST are more than enough to be REST compliant. This cut-down pair also help focus the mind on URIs, two-way content transfer, content type or schema and on the responsibilities of each resource as active players in an integration scenario.

I am wondering about that statement. In my understanding, one advantage from REST over SOAP is, that HTTP operations are used for what they are intended to. However you suggest to use POST for a delete, too. Somehow you have to include your delete intent than in the message body, maybe with XML. This ignores completely the scalability advantage of REST, that you don't have to send long XML descriptions every time.

May 24, 2007 08:33

From Duncan Cragg

@Bernd:

However you suggest to use POST for a delete, too. Somehow you have to include your delete intent than in the message body, maybe with XML. This ignores completely the scalability advantage of REST, that you don't have to send long XML descriptions every time.

How about:

POST /target HTTP/1.1 
:
<editor name="Bernd" action="delete"/>

Not too long!! And don't forget PUT:

POST /target HTTP/1.1
Content-Type: multipart/form-data; boundary=xxx ..
:
<editor name="Bernd" action="put" /> 
– –xxx 
[data]

I elaborate more on my patterns using just POST in later articles. Tim Bray has occasionally suggested this style, as I linked to in the article above.

If you don't use a POST-only pattern like these, then use go ahead and use DELETE (and PUT). One of my arguments against is that the server is in more control of resources than the database-like DELETE/PUT commands imply.

So start up a conversation with the resource, using the more flexible POST, and put richer information about yourself (the Client Who Would Edit) in the content body.

Also, of course, PUT and DELETE are much less well supported.

July 24, 2007 20:29

From Aaron Oman

Wow, this was bad. It makes REST sound like an overcomplicated set of buzzwords with little practicality. I was pretty sure REST was the choice over SOAP before coming to this page, but now I'm a little less sure. REST is probably the way to go, but this fake conversation only serves to muddy the waters and make REST supporters come off as pretentious.

April 20, 2010 23:16

Please add your own comments to this post:

By submitting a comment, you agree to its being published under the same copyright conditions as the site itself. Copyright in comments belongs to the submitter. The owner of the site disclaims any responsibility for the content of comments. Comments may be deleted at any time for any reason.