In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 9: Web Objects Ask, They Never Tell ...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 9: Web Objects Ask, They Never Tell
eBay Architect: You've pushed a lot of responsibility up to the business logic and away from the distribution technologies - away from the HTTP level.
As I understand it, you only want to use HTTP to implement a distributed Observer Pattern, where clients can become servers?
Duncan Cragg: Indeed. REST can be symmetric when used in integration outside of the Web. Further, these server-clients can have resources that do their own GET-ing, with POST callbacks, in order to Observe other resources. I've given many examples of this style.
eA: So how do you program such a distributed system at the business logic level?
DC: Well you need to be able to easily express that the state of a resource depends on the latest state - the intentions and declarations - of other resources it is Observing.
My vision is that you would express this business logic in a simple, powerful and expressive declarative language.
eA: Hmmm.. Just try getting all declarative on someone who just wants his share prices on time! No-one in business understands such abstract concepts.
DC: Oh no? The favourite programming tool of business is the spreadsheet!
And HTML is a form of declarative programming: imagine if you had to build your DOM from the top using Javascript! Or if you had to set all your styles in Javascript instead of using declarative CSS!
eA: Well I've seen some pretty ghastly examples of those two crafts!
And many failed attempts to allow non-programmers or business analysts to program directly.
DC: True, but that doesn't change the fact that most non-programmers can only think declaratively - they know What they want, but they haven't got much of a clue How to get it.
Business people talk business rules and business data.
eA: We just can't program with them!
DC: Well, perhaps we just haven't found the best discipline of pragmatic formalisms and methodology that will allow those rules to easily become programs.
It's a well-supported claim, though, that declarative programming is simpler, clearer and more productive than imperative.
eA: OK - so how would you actually define these business rules?
DC: You could use a rules engine, or a DSL engine, or even XSLT - if that works for you. There are many ways to transform XML.
eA: How would you do it?
DC: I would like to have an XML rewriting and templating system: "if this XPath or XML template matches Observed XML resource A, this one matches resource B and this one matches myself, rewrite this and that bit of myself with these bits from A and B".
Web Objects Ask, They Never Tell
eA: So you want to go around re-creating huge XML documents all the time?
DC: Who said anything about huge? This is another Web thing that doesn't necessarily apply to us in REST integration: we don't need the equivalent of the giant, monolithic HTML page. We can work in much smaller chunks.
I'd even just call them 'objects' rather than resources. Or 'Web objects', if you like.
They could be little XHTML carriers of Microformats.
eA: I thought we'd given up on distributing fine-grained objects back in the CORBA days?
DC: Ah, but this isn't trying to transparently distribute zillions of method calls in an RPC model.
This is about optimising state transfer. Only send what you need, cache where you can, push when something changes. Separate your data by rate of change, timeliness, cacheability.
eA: So these Web objects don't have any methods? Because that would be RPC when distributed?
DC: Exactly. In REST integration, the Web objects Ask, they never Tell! These Web objects are reactive: Asking for public state, not Telling each other what to do.
eA: You mean the opposite of Tell Don't Ask?
DC: Yes. As an object, you don't Tell another object How to do something, you Ask for What you want by either simply Observing its public state or by it Observing yours, then letting it decide How to evolve by itself. You then watch it and react or interact.
It's the "imperative to declarative inversion": everything is turned upside-down or inside-out when you distribute things this way!
eA: Oh yes, your "inevitable inversion" thing.
DC: Another indication of this inversion from the imperative object-oriented world to the declarative ROA world is how the derided 'train wrecks' of object-orientation now become the essential XPaths of the object Web.
You could say we have no methods, only 'getters', and XPath 'train wrecks' are encouraged!
eA: Doesn't sound too safe to me - it breaks encapsulation, doesn't it?
DC: Well, it's actually safe to dig around, since the data is held in shape by a stable, open, public schema. You're expected to go traversing the tree.
And you get excellent encapsulation since Web objects are total masters of their own destiny: they privately control the evolution of that public state.
You very much retain the value of 'What not How'; in fact, in a much better-defined way since it's fundamentally baked in to the programming model.
Hyperdata as the Engine of Object State
eA: But when you write 'train wrecks' you often end up jumping from object to object. How do your object Web XPaths do that, assuming they want to?
DC: Hyperdata of course! Links to links around the Web. Objects can have their opaque UUIDs or GUID object handles encoded into their URIs. Then objects can be wired up with XHTML links.
eA: So now your XPath transparently jumps these links?!
DC: Actually, yes! That would then allow us to dynamically break up data into more manageable chunks without breaking the XPaths that traverse it.
Also, with this approach, you still get to drill down to data like in transparent URI paths, but you now use XPaths that are properly a part of the content layer, jumping transparently over those opaque inter-object URIs.
eA: Would you use these 'jumping XPaths' in the rewrite rules you said you wanted?
DC: Of course. Either linear XPaths, or XPath-like XML tree templates on the left-hand side of a rewrite rule.
You'd start a template match on yourself, then continue on to match other objects by jumping over links, then on and on from object to object.
eA: So I suppose any such jump to another object means you then need to start Observing it, right?
DC: Exactly. And if hyperlinks are the only way you can find other objects to Observe, it brings us to the following:
Your object's next public state depends only on its current public state and the states of those objects that are visible to it through hyperlinks.
eA: Sounds a bit like the "Hypertext As The Engine Of Application State" constraint of REST.
DC: Exactly! Except now that we're doing REST symmetrically - now that clients can be servers, too - client Application State can have its own URIs!
eA: So you could re-phrase this as the even more intimidating: "Hyperdata As The Engine Of Application Resource State"!
DC: Well - how about just "Hyperdata As The Engine Of Object State"?!
eA: So do I have to wait for this link-jumping XML rewrite engine of yours to express my business logic, in order to get "Hyperdata As The Engine Of Object State"?
DC: No, of course not! Use a nice, dynamic, XML-talking language, like Scala, and follow the same principle, perhaps using a DSL.
eA: Can't you have objects that aren't entirely dependent on others? Like those that represent external state?
DC: Of course. That's normal Web stuff. It's probably best to keep these 'pure': to have either fully-interdependent objects driven by Hyperdata, or fully externally-driven ones.
Class, Extension, Instance and Behaviour
eA: Right, so we've got these little Web-mapped XHTML Microformat objects all linked up and watching each other in an Observer Pattern. This object's state depends on that linked object's state according to rewrite rules or a DSL.
So taking this mapping to objects one last step, I presume object 'class' maps onto an XML schema, XHTML Microformat specification or other content type?
DC: Yes. A public grammar in some form. Domain or business classes only, of course, not low-level classes.
eA: So, does each standard Web object class or type have a standard set of rules guiding its evolution or behaviour?
DC: Yes. If you see something and recognise its type, you can attempt to interact with it according to its public specification, and it should, but needn't, react.
The public specification can define the expected behaviour in the RFC language of MUST and SHOULD, like AtomPub. Or it can define it in rewrite rules!
eA: So then, how do I add my own business rules to a content type or 'class', if its behaviour is standardised and meant to be stable and predictable?
DC: There are two ways you can make use of a generic standard, and both are forms of layering or abstraction:
You can 'mash up' standard component instances with declarative configuration - just using them as they are for your domain.
Alternatively, you can extend or subclass their standard structures and behaviour to become themselves more domain-specific. For example, rules can be overridden or their right-hand sides merged.
Again, think of how AtomPub can be used in any domain that looks like time-ordered lists of content; and how you can extend AtomPub without breaking unextended clients or servers.
The REST Observer Pattern
eA: I think you need a memorable name for your symmetric-REST, "Hyperdata As The Engine Of Object State" architectural style!
DC: How about the "REST Observer Pattern"?
eA: But a Pattern has to be a retro-fit to existing behaviour, and "REST Observer" is quite new.
DC: I mean REST .. [DC does double bunnie ears] "Observer Pattern"!
eA: Have you got any worked examples of the REST Observer Pattern?
DC: Will have soon...
(c) 2006-2009 Duncan Cragg
Coming soon: Worked examples of the REST Observer Pattern.
Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.
Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...
Microformats are subversive: they not only challenge the approach of full-blown Semantic Web approaches, but even question fundamental Web 2.0 building blocks such as Web Feeds and Web APIs.
I recently attended XTech 2006, where there were a few talks related to Microformats.
After summarising these talks, I'll finish with my shocking revelations about the subversive nature of Microformats! ...
Microformats are subversive: they not only challenge the approach of full-blown Semantic Web approaches, but even question fundamental Web 2.0 building blocks such as Web Feeds and Web APIs.
I recently attended XTech 2006, where there were a few talks related to Microformats.
After summarising these talks, I'll finish with my shocking revelations about the subversive nature of Microformats!
Microformats from the Ground Up
Ryan King and Brian Suda from Technorati presented a half-day tutorial.
You can describe social networks, calendars and various other semantic relationships by annotating page elements or embedding small XML structures into your page according to a Microformat spec.
Microformats take Declarative or semantic markup (e.g. in XHTML) to the next logical level. Microformats have been called the 'lower case Semantic Web', because they fill the need for a lightweight set of conventions for data in Web pages that transcends the merely renderable.
HTML has a number of extension points that make this kind of thing easier. For example, the 'rel' and 'rev' attributes on a link (say what type of link it is - maybe a tag); the 'class' attributes (used for CSS but can also carry add-on semantics); the 'profile' element (say what kind of thing the whole document represents).
Microformats examples include rel-tag (for tagging your blog entries in Technorati, etc), hCard (like vCard-on-a-page), hCalendar (like iCalendar-on-a-page), XOXO (outlines) and XFN (social networking). Technorati have an hCalendar subscription service.
However, the one that stood out for me was hAtom, with which you can publish a blog and then let feed readers poll the actual page. They can read the hAtom markup and treat it as an Atom feed.
The Intelligent Design of Microformats
Ryan King also presented a talk on the motivation and style of the Microformats effort.
There is no official standards body defining Microformats, just an "Open Source"-style engineering effort, which follows the following list of Worthy Engineering Principles: "Rough Consensus and Working Code", "Paving the Cowpath", "Keep It Simple and Stupid", "You Ain't Gonna Need It' and "Don't Repeat Yourself".
Microsummaries in Firefox and on the Web
Myk Melez gave a presentation about Microsummaries in the up-coming Firefox releases.
To quote:
Microsummaries are regularly-updated compilations of the most important and timely information on web pages. For example:
- current stock price and movement for a company profile: "GOOG: 406.74 + 0.58"
- latest headline for a news site: "BBC: US dismisses Iran attack claims"
- highest bid and time remaining for an auction item: "Godzilla DVD: $15 / 2 minutes left"
One way to create this information is to link ('rel="microsummary"') from a page to its associated Microsummary document. You can also use XSLT to build the summary on the client. Both approaches struck me as clumsy.
I commented that it would be more Microformat-like to allow embedded markup (just annotate the Microsummary information right in the page; perhaps call it 'hSummary'?).
Myk used the bandwidth and page generation cost argument against that, but good XHTML pages should be very lightweight, and there are various AJAX techniques that can assemble heavier pages.
The current manifestation of the idea is to write this summary on a corresponding bookmark button in the chrome.
Allowing just bookmark buttons seems rather restricted to me. I'd be happy with a whole page in its own tab (or a widget in an Ajax desktop).
A drag-and-drop prototype was also shown, where you visually mark up the parts of the source page you want summarised. I commented that a good extension of that would be dragging across elements to a whole new page, whose construction would be dependent on the information in one or more source pages.
Myk didn't jump up and down with joy at my suggestions, so I guess we'll have to wait before we get Microsummaries written into Web pages that generate more Web pages.
RDFa: The Easy Way to Publish Your Metadata
Of course, the Semantic Web folk have their own Microformat approaches, including this one.
To quote from their paper:
The approach taken by RDFa is that ultimately any RDF structure should be representable. This means that instead of having to 'codify' each format to describe how it must be marked up, we simply provide a set of rules that explain how any RDF can be marked up, and than any RDF 'language' can be used.
The 'a' stands for 'attribute': RDF encoded as attributes in XML.
The example shown was FoaF, whose Microformat competition is
XFN and hCard.
You can use attributes like <link rel="foaf:mbox">,
<span property="foaf:name">, <span property="foaf:phone">
to encode RDF triple information at various points around your
document.
RDFa has some arguable benefits over their competition, but I don't see why both sides shouldn't get along and share ideas. And, indeed, why the other alternatives (structured blogging, embedded RDF) shouldn't all find a place, too. There is a comparison here. Another speaker at XTech, Uche Ogbuji, has also discussed some of these issues.
Web 2.0 Centraal
Microformats not only challenge the Semantic Web, they even challenge such basic Web 2.0 technologies as Web feeds and Web APIs.
Shouldn't we be subscribing to and interacting with semantic Web pages (note the lower case 's'!), not using invisible Web Feeds and Web APIs? Technorati think so: they are leading the way in searching and subscribing to Microformats.
The redundancy of Atom itself (and feeds in general) has been remarked upon before. Further, since we're using REST in the Atom Publishing Protocol, why not use the same protocol to publish directly to an hAtom annotated site? (e.g., just POST a new article to your own blog's main page!) Web 'API's are redundant when our pages are just XML data anyway.
Writing data pages instead of document pages, then allowing those pages to be updated and then subscribed to, is the future of the Web: it's Web 2.0 Centraal.
The new lightweight, Declarative, Microformat-rich XHTML pages we're going to be generating can be polled just as Web feeds are, and events made from their changes. Then re-presented via Ajax in dynamic pages. Such standardised page formats and micro-formats should also accept standardised POSTs where appropriate.
Microformats represent some first steps towards the Data Web replacing the Document Web. Although there has been some thought given to extracting full-blown Semantic Web data from Microformats, there's a significant chance that this bottom-up, simple, minimal, change-aware (i.e., updateable, subscribable) approach may teach the Semantic Web practitioners a thing or two about the Worthy Engineering Principles that enable the social and technical proliferation of a good idea.
Standardising common formats and structures is a precursor to this two-way, dynamic Data Web. Both the bottom-up practitioners of Microformats and the top-down practitioners of the Semantic Web have much to learn from each other on this path.
Declarative Architectures focus on the What, not the How, of programming. The How has dominated the field - perhaps 80% of programming is done in the traditional Imperative style, where we tell the computer How to do a task in explicit steps.
I'd like to show in this blog how Declarative Architectures and technologies are not just an interesting sideshow to the main Imperative attraction, but a complete and powerful programming alternative in their own right - indeed, one which has already dominated certain fields.
Imagine being able to simply express What we want the computer to do - to give it constraints and rules - then let it work out for itself How to achieve our goals.
I believe that saying What, not How, will become the dominant paradigm in programming. ...
Declarative Architectures focus on the What, not the How, of programming. The How has dominated the field - perhaps 80% of programming is done in the traditional Imperative style, where we tell the computer How to do a task in explicit steps.
I'd like to show in this blog how Declarative Architectures and technologies are not just an interesting sideshow to the main Imperative attraction, but a complete and powerful programming alternative in their own right - indeed, one which has already dominated certain fields.
Imagine being able to simply express What we want the computer to do - to give it constraints and rules - then let it work out for itself How to achieve our goals.
I believe that saying What, not How, will become the dominant paradigm in programming.
What?
Declarative Architectures are about data: documents, assertions, truths, rules, tables, structures, state, bits. Data storage, changes, constraints, transformation, sharing, querying, fetching, interaction, presentation, subscription, tagging.
In summary, we talk about stuff, changes to stuff and rules governing stuff. Not processes, actions, jobs or functions.
It's amazing how many interesting Declarative technologies are actually out there already - especially given that most programming is Imperative. They tend to go unnoticed; subservient to their Imperative masters.
Here are some examples from our daily work life (assuming you work in computing!):
What all these have in common is that they are purely data- and event-driven or oriented. Not a thread in sight (if used properly). No method calls. Just querying, fetching, subscribing, tagging, declaring truths and rules, making changes to stuff, etc.
On the 'Net, all the dominant technologies are actually Declarative, not Imperative:
Again, the same things are happening: querying, fetching, subscribing, tagging, declaring truths and rules, making changes to stuff.
Email (SMTP) is a great example of a Declarative Architecture, with its routing and transformation rules and self-addressed store-and-forward transmission. And, in fact, much the same applies to the rest of this list, where the Declarative aspects may include network querying and data events.
Imperative protocols such as RPC, RMI, CORBA, SOAP, etc. haven't had anywhere near the impact of their Declarative cousins.
Now, consider the 'Web 2.0' technologies:
This is a wonderful list of Declarative approaches (as long as we use REST APIs and talk about Resources, not SOAP ones talking about functions and jobs!).
So far, I've mentioned technologies that have attached themselves to the periphery of Imperative programming, where the actual behaviour is still driven procedurally, or How-like.
However, Declarative programming itself has a history at least as long as Imperative programming, and includes:
We can, if we choose, write 100% Declarative systems (even down to asynchronous or event-driven operating systems and hardware).
Wherefore?
In this blog, I'll be pointing out opportunities where we can benefit from the Declarative approach when it's ready.
Perhaps the biggest example of a Declarative Architecture that is ready to benefit us now is the Web. In contrast, the Imperative Service-Oriented Architecture model and so-called 'Web' Services are still weaning; still trying to get a life free of their huge standards consortia. REST, in its simplicity, ubiquity and straightforward usefulness is orders of magnitude more deployed than SOAP will likely ever be.
This blog is for discussing the common problems and solutions in Declarative technologies; seeing if ideas in one area can be used in another and seeing if we can distill their lessons and their approaches.
Ultimately, heading towards a unified, distributed Declarative programming technology landscape.
I hope you'll join me in my explorations into Declarative Architectures, in particular if you're interested in the currently most promising segment I mentioned: Web 2.0, and its application in Social Software.
Or the currently massive but little-mentioned segment: Peer-to-Peer, and its application to the disruption of, well you know, all those big technological and commercial hierarchies...