What Not How - Posts tagged 'declarative'

Introducing Fjord

2009-12-11T08:22:00Z

Following on from my recent article where I derived FOREST, this article offers the beginnings of a JSON unification and rewriting language that can be used in a FOREST architecture.

Why JSON, not XHTML, now? Well, I recently discovered that JSON is overtaking XHTML in interest, and I was further inspired by Kris Zyp's recent announcement of his JSON Schema Internet-Draft.

Fjord is a language for describing how the state of a JSON resource at any time depends on both its current state and on the state of other JSON resources that it links to via hyperlinks.

Fjord is a Norwegian word, probably pronounced 'fiyourd', and might stand for some combination of the words: 'Functional JSON Object/Observer Resource/Rule/Rewrite Dependencies/Declarations'. Or maybe it's just because they're truly awesome, an' I wanna go.

Fjord also gives me an opportunity to show some examples of the "end-user" view of a FOREST interaction; starting with a simple finance example.

Instrument, Bids, Asks

Let's take the simplest possible view of a financial market.

There's an Instrument, and there are Bids and Asks on that Instrument. The Instrument summarises all the Bids and Asks into the 'bid-ask spread', taking the highest bid price and the lowest ask price. Forget volumes, trades and any other detail.

The Instruments, Bids and Asks are on various servers across various organisations. That's the 'symmetric REST' part.

Here's a JSON Instrument resource at the URL shown. It holds three Bid or Ask objects:

http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json
{
    "tags": [ "equity", "instrument" ],
    "long-name": "Acme Co., Inc",
    "buyers": [ 
        "@http://a-bank.com/fjord/equity-bid-9ac0d1-88ce1.json"
    ],
    "sellers": [ 
        "@http://c-bank.com/fjord/equity-ask-510efb-cca62.json",
        "@http://d-bank.com/fjord/equity-ask-8560ae-33eff.json"
    ],
    "bid-ask-spread": {
        "high-bid": "10.00",
        "low-ask":  "14.00"
    }
}

The '@' character is the flag that this string is a link or a URL. The meaningful names in those links are really just for humans and the layers below us - Fjord itself won't let you see inside them.

So now that we've introduced a JSON extension making it a hypermedia or hyperdata type, perhaps we should announce this convention in a media type string. How about "application/fjord+json"? This media type will also serve to set expectations on everything else about Fjord JSON objects, once the notation is stable.

Notice that, instead of 'class' or 'type' or a top-level tag, we have a list of tags. In Fjord, class is in the eye of the beholder - but, of course, must be a stable grammar in the eye of the generator!

Right, so here is a new Bid object held at 'B' Bank, that the Instrument hasn't seen yet:

http://b-bank.com/fjord/equity-bid-0043ee-419ac.json
{
    "tags": [ "equity", "bid" ],
    "on": "@http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json",
    "price": "",
    "by": "@http://b-bank.com/fjord/hcard-b-bank-4990cd-232b0.json"
}

It has a pointer to the Instrument it is a bid on, a bid price that isn't yet set, and a link to the hCard object of the bank offering the bid and serving this Bid object.

So Bids and Asks can see Instruments and vice-versa, and each can set their own state accordingly, as part of a FOREST interaction.

The 'animation' of Bid, Ask and Instrument JSON object resources is driven by three simple Fjord rules...

Rule One: Bid Sets Price From Instrument

Here is a rule setting the price of a new Bid to be 10% more than the highest current Bid on the instrument, hopefully ensuring this Bid is now the high bid:

{
    "tags": [ "equity", "bid" ],
    "on": { "tags": [ "equity", "instrument" ],
            "bid-ask-spread": { "high-bid": "/$hibid;decimal/" }
          },
    "price": "/null/( $hibid * 1.10 )/"
}

This is a template that matches the Bid object and its dependencies and rewrites part of the Bid.

Notice how the Bid object's link to the Instrument object is jumped transparently: the Bid can 'see' right into the Instrument in order to do its matching. However, it cannot itself rewrite the Instrument - only Instrument rules can do that.

In Fjord, matching is written with the '/ ; /' notation: each part of this semicolon-separated list must match. In this case, the high bid has to bind to the identifier '$hibid' and has to be a decimal. The price must not be set yet.

The keys 'decimal' and 'null' are, of course, special reserved words for matching in patterns. The 'null' matches empty strings, lists and hashes.

The rewrite part follows at the end of a match and is also wrapped in slashes. Here, the price is being set to the high-bid plus 10%. The parentheses introduce an expression.

This rule only matches if the price is empty - once set, the rule will not refire.

For more complex rewrites, the following forms can also be used:

    "price": { "/null//": "( $hibid * 1.10 )" }
    "price": { "when": "/null/", "then": "( $hibid * 1.10 )" }

which allow full hashes and lists to be both matched and set.

Rule Two: Instrument Puts New Bid into its Buyers List

The Instrument adds the new Bid to its buyers list with this rule:

{   "#url": "/$bid/",
    "tags": [ "equity", "bid" ],
    "on": { "#url": "/this/",
            "tags": [ "equity", "instrument" ],
            "buyers": "/list/has($bid)/",
    }
}

This is a rule on the Instrument, that 'looks up to' and matches the enclosing Bid.

The '#url' is an optional, reserved-word entry that can only appear at the top of a JSON resource and holds that resource's URL. Here, it is bound to '$bid', picking up the URL of the Bid object that refers to and encloses this Instrument.

Further down, the "#url" reserved-word value of "this" indicates the start of the object that this rule applies to. If "this" is absent, then, of course, this is a rule on the top-level object, as in the first rule above.

The rewrite part matches the buyers list of the Instrument and applies the 'has()' operator which ensures the list now contains the wrapping Bid; it rewrites the list to include its argument if it doesn't already.

If this rule matches two wrapping Bids simultaneously, then '$bid' is set to a 'match set' - a simultaneous binding to the URLs of all of those Bids, and 'has()' must ensure that all of these Bids are now in the list at once.

If there is no change after a rule is applied, the resource's Etag isn't incremented, and dependent rules won't fire. Fjord may re-apply any rule at any time, usually on an incoming state change indicated by Etag, so matches have to be designed accordingly.

That's why we say 'has', not 'add', which would add the Bid again and again each time the rule is applied. We could say "/list;hasNo($bid)/add($bid)/", but that's uglier and more imperative than declarative.

Rule Three: Instrument Sets Bid-Ask Spread from Bids and Asks

This is what the instrument looks like, now, with the new link to our Bid:

{
    "tags": [ "equity", "instrument" ],
    "long-name": "Acme Co., Inc",
    "buyers": [ 
        "@http://a-bank.com/fjord/equity-bid-9ac0d1-88ce1.json",
        "@http://b-bank.com/fjord/equity-bid-0043ee-419ac.json"
    ],
    "sellers": [ 
        "@http://c-bank.com/fjord/equity-ask-510efb-cca62.json",
        "@http://d-bank.com/fjord/equity-ask-8560ae-33eff.json"
    ],
    "bid-ask-spread": {
        "high-bid": "10.00",
        "low-ask":  "14.00"
    }
}

And here is what the new B-Bank Bid looks like with its competitive price:

{
    "tags": [ "equity", "bid" ],
    "on": "@http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json",
    "price": "11.00",
    "by": "@http://b-bank.com/fjord/hcard-b-bank-4990cd-232b0.json"
}

So now the Instrument has to adjust its bid-ask spread to accomodate the new Bid. This happens when the following rule matches, which it always does - but only fires when a change is detected:

{
    "tags": [ "equity", "instrument" ],
    "buyers":  { "price": "/$bids;decimal/" },
    "sellers": { "price": "/$asks;decimal/" },
    "bid-ask-spread": {
        "high-bid": "/decimal/max($bids)/",
        "low-ask":  "/decimal/min($asks)/"
    }
}

This rule matches as many times as there are Bids and Asks, meaning that the local identifiers $bids and $asks are each bound to match sets.

Notice how the '{ price .. }' patterns match elements of a list, in the same way an XPath can match either one or several elements in a sequence of XML elements.

The 'max()' and 'min()' operators take the highest and lowest values in a match set (or in a physical list, come to that). Once again, this rewrite means 'set this value and increment the Etag, but only if there's a change'.

Here, the high-bid becomes our new Bid's '11.00'.

Fjord

So that's a quick intro to some of my ideas for JSON matching and rewriting in a FOREST architecture.

Fjord is a declarative, JSON-encoded language for expressing how the state of a JSON resource is a function of both its own state and the state of peer JSON resources that it can see via links in a hyperdata Web.

Fjord objects are 'masters of their own destiny', not allowing other objects to tell them what to do except via declarative intention.

Further articles will give more proposals for the syntax and semantics of Fjord, including true unification, and put it into the set of languages that are ideal for programming parallel and distributed systems, as well as for integration.

Since Fjord is very young and I'm still implementing it (in Java), I would certainly welcome comments about it, or about the FOREST style that it supports. You can chat with me on Twitter, too!

Web Objects Ask, They Never Tell | The REST Dialogues

2009-08-13T11:43:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 9: Web Objects Ask, They Never Tell

eBay Architect: You've pushed a lot of responsibility up to the business logic and away from the distribution technologies - away from the HTTP level.

As I understand it, you only want to use HTTP to implement a distributed Observer Pattern, where clients can become servers?

Duncan Cragg: Indeed. REST can be symmetric when used in integration outside of the Web. Further, these server-clients can have resources that do their own GET-ing, with POST callbacks, in order to Observe other resources. I've given many examples of this style.

eA: So how do you program such a distributed system at the business logic level?

DC: Well you need to be able to easily express that the state of a resource depends on the latest state - the intentions and declarations - of other resources it is Observing.

My vision is that you would express this business logic in a simple, powerful and expressive declarative language.

eA: Hmmm.. Just try getting all declarative on someone who just wants his share prices on time! No-one in business understands such abstract concepts.

DC: Oh no? The favourite programming tool of business is the spreadsheet!

And HTML is a form of declarative programming: imagine if you had to build your DOM from the top using Javascript! Or if you had to set all your styles in Javascript instead of using declarative CSS!

eA: Well I've seen some pretty ghastly examples of those two crafts!

And many failed attempts to allow non-programmers or business analysts to program directly.

DC: True, but that doesn't change the fact that most non-programmers can only think declaratively - they know What they want, but they haven't got much of a clue How to get it.

Business people talk business rules and business data.

eA: We just can't program with them!

DC: Well, perhaps we just haven't found the best discipline of pragmatic formalisms and methodology that will allow those rules to easily become programs.

It's a well-supported claim, though, that declarative programming is simpler, clearer and more productive than imperative.

eA: OK - so how would you actually define these business rules?

DC: You could use a rules engine, or a DSL engine, or even XSLT - if that works for you. There are many ways to transform XML.

eA: How would you do it?

DC: I would like to have an XML rewriting and templating system: "if this XPath or XML template matches Observed XML resource A, this one matches resource B and this one matches myself, rewrite this and that bit of myself with these bits from A and B".

Web Objects Ask, They Never Tell

eA: So you want to go around re-creating huge XML documents all the time?

DC: Who said anything about huge? This is another Web thing that doesn't necessarily apply to us in REST integration: we don't need the equivalent of the giant, monolithic HTML page. We can work in much smaller chunks.

I'd even just call them 'objects' rather than resources. Or 'Web objects', if you like.

They could be little XHTML carriers of Microformats.

eA: I thought we'd given up on distributing fine-grained objects back in the CORBA days?

DC: Ah, but this isn't trying to transparently distribute zillions of method calls in an RPC model.

This is about optimising state transfer. Only send what you need, cache where you can, push when something changes. Separate your data by rate of change, timeliness, cacheability.

eA: So these Web objects don't have any methods? Because that would be RPC when distributed?

DC: Exactly. In REST integration, the Web objects Ask, they never Tell! These Web objects are reactive: Asking for public state, not Telling each other what to do.

eA: You mean the opposite of Tell Don't Ask?

DC: Yes. As an object, you don't Tell another object How to do something, you Ask for What you want by either simply Observing its public state or by it Observing yours, then letting it decide How to evolve by itself. You then watch it and react or interact.

It's the "imperative to declarative inversion": everything is turned upside-down or inside-out when you distribute things this way!

eA: Oh yes, your "inevitable inversion" thing.

DC: Another indication of this inversion from the imperative object-oriented world to the declarative ROA world is how the derided 'train wrecks' of object-orientation now become the essential XPaths of the object Web.

You could say we have no methods, only 'getters', and XPath 'train wrecks' are encouraged!

eA: Doesn't sound too safe to me - it breaks encapsulation, doesn't it?

DC: Well, it's actually safe to dig around, since the data is held in shape by a stable, open, public schema. You're expected to go traversing the tree.

And you get excellent encapsulation since Web objects are total masters of their own destiny: they privately control the evolution of that public state.

You very much retain the value of 'What not How'; in fact, in a much better-defined way since it's fundamentally baked in to the programming model.

Hyperdata as the Engine of Object State

eA: But when you write 'train wrecks' you often end up jumping from object to object. How do your object Web XPaths do that, assuming they want to?

DC: Hyperdata of course! Links to links around the Web. Objects can have their opaque UUIDs or GUID object handles encoded into their URIs. Then objects can be wired up with XHTML links.

eA: So now your XPath transparently jumps these links?!

DC: Actually, yes! That would then allow us to dynamically break up data into more manageable chunks without breaking the XPaths that traverse it.

Also, with this approach, you still get to drill down to data like in transparent URI paths, but you now use XPaths that are properly a part of the content layer, jumping transparently over those opaque inter-object URIs.

eA: Would you use these 'jumping XPaths' in the rewrite rules you said you wanted?

DC: Of course. Either linear XPaths, or XPath-like XML tree templates on the left-hand side of a rewrite rule.

You'd start a template match on yourself, then continue on to match other objects by jumping over links, then on and on from object to object.

eA: So I suppose any such jump to another object means you then need to start Observing it, right?

DC: Exactly. And if hyperlinks are the only way you can find other objects to Observe, it brings us to the following:

Your object's next public state depends only on its current public state and the states of those objects that are visible to it through hyperlinks.

eA: Sounds a bit like the "Hypertext As The Engine Of Application State" constraint of REST.

DC: Exactly! Except now that we're doing REST symmetrically - now that clients can be servers, too - client Application State can have its own URIs!

eA: So you could re-phrase this as the even more intimidating: "Hyperdata As The Engine Of Application Resource State"!

DC: Well - how about just "Hyperdata As The Engine Of Object State"?!

eA: So do I have to wait for this link-jumping XML rewrite engine of yours to express my business logic, in order to get "Hyperdata As The Engine Of Object State"?

DC: No, of course not! Use a nice, dynamic, XML-talking language, like Scala, and follow the same principle, perhaps using a DSL.

eA: Can't you have objects that aren't entirely dependent on others? Like those that represent external state?

DC: Of course. That's normal Web stuff. It's probably best to keep these 'pure': to have either fully-interdependent objects driven by Hyperdata, or fully externally-driven ones.

Class, Extension, Instance and Behaviour

eA: Right, so we've got these little Web-mapped XHTML Microformat objects all linked up and watching each other in an Observer Pattern. This object's state depends on that linked object's state according to rewrite rules or a DSL.

So taking this mapping to objects one last step, I presume object 'class' maps onto an XML schema, XHTML Microformat specification or other content type?

DC: Yes. A public grammar in some form. Domain or business classes only, of course, not low-level classes.

eA: So, does each standard Web object class or type have a standard set of rules guiding its evolution or behaviour?

DC: Yes. If you see something and recognise its type, you can attempt to interact with it according to its public specification, and it should, but needn't, react.

The public specification can define the expected behaviour in the RFC language of MUST and SHOULD, like AtomPub. Or it can define it in rewrite rules!

eA: So then, how do I add my own business rules to a content type or 'class', if its behaviour is standardised and meant to be stable and predictable?

DC: There are two ways you can make use of a generic standard, and both are forms of layering or abstraction:

You can 'mash up' standard component instances with declarative configuration - just using them as they are for your domain.

Alternatively, you can extend or subclass their standard structures and behaviour to become themselves more domain-specific. For example, rules can be overridden or their right-hand sides merged.

Again, think of how AtomPub can be used in any domain that looks like time-ordered lists of content; and how you can extend AtomPub without breaking unextended clients or servers.

The REST Observer Pattern

eA: I think you need a memorable name for your symmetric-REST, "Hyperdata As The Engine Of Object State" architectural style!

DC: How about the "REST Observer Pattern"?

eA: But a Pattern has to be a retro-fit to existing behaviour, and "REST Observer" is quite new.

DC: I mean REST .. [DC does double bunnie ears] "Observer Pattern"!

eA: Have you got any worked examples of the REST Observer Pattern?

DC: Will have soon...

Coming soon: Worked examples of the REST Observer Pattern.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

WS-Are-You-Sure | The REST Dialogues

2009-07-16T16:16:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 8: WS-Are-You-Sure (Security, Reliable Messaging and Transactions)

Duncan Cragg: So, back to your list of Enterprise functions. We're on to what I'm going to call the 'WS-Are-You-Sure': Security, Reliable Messaging and Transactions.

Let's attack these Starting with the Web!

eBay Architect: We could start with Security: authentication, authorisation and encryption. For example, you have to keep some information secret on eBay. Like Invoices, Offer details. Reserve price on an Item. And you have to ensure only the owners of data can change it.

DC: The simplest pattern for read security is to use HTTP Basic Authentication over TLS - following HTTPS. Simple, but well-supported and usually good enough.

But with HTTPS you lose some of the benefits of using intermediaries, such as cacheing. If those intermediaries are untrustworthy, then you can use message-level rather than transport-level security: encrypt the resource state being transferred.

eA: Can't I use WS-Security for this?

DC: Possibly(PDF)! However, the benefits of cacheing may be lost in the time taken to package and unpackage each resource in turn. You may prefer a more lightweight approach as suggested in the Atom and AtomPub specs.

eA: How does REST handle authorisation: such as read and write permissions?

DC: As I keep saying, REST is about much more than simple data read/write services. In REST we don't have the generic concept of authorisation on a specific process execution, such as a command that could cause state change.

REST infrastructure is about state transfer, which is thus really only about 'read permissions'.

Everything else is business logic: it's up to the target resource to manage its reaction to incoming non-GETs and to decide if or how it should change in response, according to internal integrity constraints and the identity of the source. Resources are masters of their own destiny and must be aware of the identity of interacting parties at that level.

eA: What can you do to secure the infrastructure level below the business logic?

DC: The department managing the infrastructure can see data going either out (GET) or in (POST), and can see the target URIs. They can thus do both server- (URI) and client- (request header) based security and partitioning.

For read permission, it's possible to implement a low-level lookup from the identity in the request header to whatever URIs they can GET. They can enforce simple rules at that level like 'only GETs are allowed on these URIs unless the client is in this list'. They can groom more and less sensitive traffic to different servers.

eA: Any more Security advice?

DC: Paul Prescod has written some notes on REST security.

Finally, remember to keep sensitive data out of those highly-propagatable unencrypted URIs by using POST instead of GET when submitting queries; another reason to use URIs that are literally opaque, not just treated as opaque operationally.

Reliable Messaging

eA: Another of the WS-* specifications deals with Reliable Messaging. How does REST give me the assurances I need that an important message - such as a new Offer on an Item or a ResponseToBestOffer, or an Invoice - will be delivered? In the right order? I can't just rely on POST, as you suggested before, if I really care about this.

DC: In REST, there are no command messages that have to make it through. There's only state that may or may not need to be reliably transferred - or that may or may not need to be notified in a timely manner.

In the eBay example, as I described it before, "if you keep re-POSTing the same Invoice, or Item or Offer, it only gets created once".

eA: Ah! Define 'same'!

DC: If, as in this eBay example, the successful POST creates a server-side copy with its own new URI, then the Item, Invoice, etc, must have some uniquely identifying information on it. It could perhaps have a Message-ID header or get cheap, unique URIs minted for it from the server in advance. Alternatively, when the POSTed resource already has a URI itself on the 'client', then it's obviously the same each time it's POSTed.

When used as state notification, POST must be idempotent; repeatable.

So if the initial POST fails, just keep POSTing until you can see the appropriate response, whatever that may be in business terms. On the pull or poll side, keep GETing until you see what you expect.

eA: So that's another issue you're side-stepping by dumping it into the business logic?!

DC: Only the business logic knows the following things: what signifies receipt of the notification; if it matters that the state didn't get through; how frequently to push or poll; whether it matters that state is out of date and by how much; and when to give up and tell someone.

Set the push or pull frequency and total number according to the business logic's view of the importance of that state transfer. Set cache control according to your domain's tolerance of stale data.

It's just like in real life: if something I sent doesn't get a response - in a form that is completely dependent on the type of recipient - then, after a time - which is also completely dependent on the type of recipient - I'll chase it up.

eA: Can't REST give any support here at all?

DC: Well, it would be easy enough to write a REST support library that implemented a simple API for specifying your constraints on a successful state transfer.

Transactions

eA: Now, when you're a site like eBay, dealing with money all the time, you need the assurance that transactions give you. You need to make sure accounts are always consistent. But I suppose, like before, you're going to tell me that it'll all be fine in the end, right?

DC: Hold on. Let's not mix up financial transactions and database transactions! We'll first talk about the need for atomic units of work. Then see how to support financial transaction business logic.

Also, we're talking about units of work in public view, not hidden behind resources. Inside, it's up to a resource to ensure that its integrity and consistency are maintained through its interactions with others, and it's free to use transactions to achieve that internally if it wants, without exposing that to its clients.

eA: OK - so now say that it'll all be fine in the end!

DC: In a distributed system, you have to decide on what to give up out of Consistency, Availability and Partition Tolerance.

I have to say that eBay are actually fully clued here: that was a paper about 'BASE' by Dan Pritchett, Technical Fellow at eBay, in which he discusses the benefits of Eventual Consistency - i.e., knowing that it'll all be fine in the end! Especially if you tidy things up eventually.

eA: Gah! Ya got me there!

DC: Essentially, the rule of thumb is, use ACID internally, use BASE externally.

We're back to the inevitable inversion from internal imperative thinking to external declarative thinking.

As an imperative programmer you're inclined to want to take your internal programming style out into the distributed world - to think single-thread, central control: 'begin - do work - commit'.

But the importance of Availability and Partition Tolerance in distributed systems usually outweighs the importance of Consistency, leading the wise architect to a more relaxed, less imperative, more declarative approach.

eA: Such as REST.

DC: Indeed. REST without transaction support.

REST isn't a database model: in the same way REST doesn't imply simple read/write services, it also doesn't imply inert data that needs locking. And resources in REST should model active domain data, not low-level, domain-independent transaction paraphenalia.

eA: How does REST without transactions work, BASE-ically, then?

DC: A handy phrase that sums it up is intention puts the system in tension.

You start by declaring your intention that some state be true, which puts the system in tension - a tension that can only be resolved by the application of business logic constraints over each player in parallel, until the whole system settles or resolves into a new, consistent state.

eA: Examples, please!

DC: Think about how you'd do the classic transfer of funds between accounts, in the real world of loosely interacting, self-determined parties. Say inside a big company before computers came along, between an office that handles one account and an office that handles the other.

Your key resource is a signed declaration (the intention) by the payer that they are happy to have funds passed to the payee. As long as this fact doesn't appear in one account or the other, you have work to do (there is tension in the business rules).

eA: You've got to run around real quick with a piece of paper.

DC: It doesn't even need to happen all at the same time: you can visit one office, check the funds are available and deduct the amount, then wander over to the other office and tell them to increase the payee's balance. If you get waylaid and the auditors come, there is always the signed declaration and the account history available to resolve the situation.

You can enforce the constraint that no money appears to be in two places with the business rule that the payee account is only increased if the payer's account has an entry corresponding to the signed declaration.

eA: Mmm. Sounds a bit too loosely coupled to me.

DC: It's life outside of Central Control.

Consider hotel and flight booking: you don't lock the hotel and the flight while telling them all in a two-phase commit what your itinerary will be. You do 'optimistic locking' with compensation: if things don't work out, you cancel a booking. A system may tell you something is available, but when it comes to booking it may have just been taken.

The real, distributed, reactive world doesn't work in a lock-step fashion, so our distributed, reactive systems don't need to work that way to model it. Reality is much more like optimistic locking with the possibility of compensation or merge on conflict that, again, can only be defined at the business level.

eA: Why not do your optimistic locking below that? HTTP has support for it, right?

DC: In the same way that REST can support read permissions but is at the wrong level for write permissions, which are a business level concern, there is an asymmetry in read versioning versus write versioning.

While using Etags is great for optimising the reading and cacheing of data, I wouldn't use them in the optimistic locking pattern for writes that is supported by HTTP. The proper place for handling a mismatch of versions in an interaction is not in the HTTP headers.

REST should be about state declaration and intention, not absolute write commands. Only the business logic governing the evolution of a resource knows if, for example, it can go ahead and respond anyway to an edit request, even though it's possible that the sender has an out-of-date copy of it.

In Part 9: Web Objects Ask, They Never Tell

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

Mobile Widgets aren't the Mobile Web

2009-02-11T16:20:00Z

Mobile Monday London met last night to discuss the Mobile Web and Widgets. It was an engaging and thought-provoking evening.

Your intrepid reporter was there and, in spite of the crashing of his sad, clunky old Windows Mobile Xperia X1, losing all his notes, he brings you this hot report from right out of his memory (somewhat steamed up by subsequent socialising, but reclarified by Google).

After that, I give an explanation of why I believe that Widgets are not the solution to what Mobile 2.0 needs...

GSMA ONE

First up was Kevin Smith from Vodafone to tell us about the GSMA ONE Web API (also via here). ONE means 'Open Network Enablers'. It seems quite similar to the Web21C initiative by BT, led by my good friend Paul Downey, but which was unfortunately set aside in favour of Ribbit.

You can send an SMS, get a user's location and access billing and connection info. All through a scary STREST interface.

Scary because it looks like you can send a text using a GET... The documentation here [PDF] does suggest using POST, but even so, the design fundamentally wants Resource-Orientation in place of Service Orientation:

The message appears to be tacked on to the URL as an argument. There are references to 'soapUI', endpoints, etc. An SMS message doesn't appear to have its own URL, just an internal message ID. There are 'exceptions thrown', all with a 400 code, even for internal platform and integration faults. And so-on - in classic STREST style!

Still, nothing that can't be fixed with a little help from a REST dude, or a read of "RESTful Web Services".

OMTP BONDI and Ikivo

Next, Nick Allot told us all about OMTP BONDI, which is an attempt to ameliorate fragmentation in the mobile Web and widget space - the one which allows access to parts of the phone not normally reached in a browser: PIM, SMS, camera, GPS, etc.

Their big thing is security: take widgets out of the safety of the browser and give them access to the phone via APIs, and you have a potential security nightmare.

Seems BONDI will let you do most Mobile 2.0 native-like applets, such as LBS, photo-sharing, etc. There's a reference implementation for sad, clunky old Windows Mobile, and both Opera and LiMo are jumping aboard, too.

BONDI aims to be W3C compatible where possible. Not HTML5, mind, but rather a works-now, fit-for-purpose solution to the fragmentation problem, that includes widgets and may later converge with HTML5. There's an excellent discussion about this on WAP Review.

This was followed by a talk by Samuel Sweet from Ikivo. It seems that, starting with their successful Samsung T*Omnia widget interface, Ikivo have been overcome with standards love, and are going BONDI as well as W3C compliant, especially with SVG as a rendering technology.

Firefox for Mobile

Christian Sejersen of Mozilla gave a good intro to what is currently called Fennec - Alpha 2, but will soon be renamed the more grown-up 'Firefox'. It's got the same code in it after all: and APIs such as camera and location that get added to the common codebase will be accessible via both mobile and desktop.

Looks very swishy and touchy and in sync with the times. Available now on Maemo (and Moblin) and soon on sad, clunky old Windows Mobile. Then later this year on Symbian. Codebase is C, so no Java or Javascript phones supported, of course. Or closed walled-garden proprietary locked-down phones like, um, the iPhone. It seems that the add-on community is already fixing up their plugins for this mobile version without even being prompted. No news on the Prism widget-alike system, to compete with BONDI or Opera. It's "still in the labs"...

Panel

Then the panel session started, with the best questions from the excellent host, Dan Appelquist of Vodafone, and some good ones from the audience.

There was much elaboration and clarification on the presentations (thus included above instead), and an interesting conversation around monetisation: how do widgets make money? Three suggestions: adverts, selling widgets on an app(let) store and micro-payments. Graham Thomas of T-Mobile appeared to be happy just to get more Internet traffic for now..

There was also confirmation from Graham that Web'n'Walk 4.0 will major on widgets, but he didn't say what flavour. More hot journalism uncovers this confirmation [PDF] (page 29) that it's still going to be tied in with Opera.

The outstanding revelation of the evening (sorry Ikivo!) came when someone told us that the Palm Pre's widget system is not only proprietary, but uses tables for layout! The horror! Lots of tutting and W3C-like smugness around the room.

Following Open W3C Standards can still break the Web

The goal of all this standard widget aspiration, apart from the obvious motivation of being able to compete with the iPhone, is to allow everyone to write widgets, and for those widgets to work on all our phones.

However, just because everyone does what the W3C thinks they should do doesn't mean you automatically get interoperability. Most importantly, doing what the W3C wants doesn't mean you won't break the Web!

The W3C, don't forget, also brings you 'Web' Services - those mis-named standards responsible for much Web-breaking, including inspiring a whole generation of Web-breakers with their STREST interfaces.

Interoperability on the Web is about state transfer, content types, URLs, hypertext. No amount of API and widget standardisation will give classic Web interoperability as long as they ignore these Web basics.

A widget is an applet is an application. It's a closed structure whose data interoperability has to be hard coded each time, and whose imperative Javascript naturally wants to make function calls back on the server - probably through HTTP.

And you run either this application or that, each having its own way of pushing or pulling data around, or even the same data presented in much the same way all over again.

In the Web you run one browser application and everything is mashed within - data interoperability.

The U-Web gives us the best of both worlds

Admittedly, the basic Web isn't good enough by itself when seeking an architecture for Mobile 2.0's essential personalisation, interactivity and usability.

But that doesn't mean that we should throw away the Web's hard-won advantages of interoperability and scalability, that we perhaps take too much for granted.

Enter the U-Web! The U-Web "puts the Web back into Web 2.0 and Mobile 2.0".

It takes the best of the Web's one-way static document publishing model, and extends it to a two-way dynamic data exchange model, while keeping the interoperability and scalability of the Web.

Instead of working on widget standards that break the Web, let's standardise a fully Web-compatible Mobile 2.0 architecture that delivers the same rich, personal functionality, but adds back the seamless mashability of ever-changing people and their ever-changing stuff. Oh, and promises scalability and rapid, easy development.

I've kicked off the project with the U-Web proposal - perhaps you'd like to jump in and help? Email me (see left bar) or leave a comment.

The Mobile 2.0 Killer App is the App Killer

2008-12-19T17:05:00Z

Mobiles are unique - if you want to miss out on the opportunity they represent, you could choose to see them as just slow computers with tiny interfaces and dodgy Internet connections. Then try to squeeze in your traditional applications; try squeezing the office desktop metaphor with its sedentary documents into a device the size of a mouse!

Alternatively, see them as the most personal, social and dynamic of devices that are becoming connected to the Internet. Now a multi-billion-scale global opportunity opens up to you. That's customers and dollars! In trying to grasp this, some are calling it 'Mobile 2.0', by analogy with its sibling, Web 2.0.

In that light, the Killer App for Mobile 2.0 is the sharer, masher and updater of People, Things, Times and Places... The key to getting Mobile 2.0 right is for it to merge seamlessly into our lives. That means the handling of dynamic and shared data becomes the top priority, even above the handling of applications.

This article describes a Mobile 2.0 platform that makes people and their stuff first class - not applications.

Mobile 2.0 should be about effortless delivery of dynamic data such as local weather, personalised news feeds, timelines off Facebook and feeds off Twitter, messages, notes, videos, photos, calendar entries, map locations and meetup plans.

Mobile 2.0 should easily allow us to two-way-share this dynamic data with family, friends, contacts, third parties and the world.

People don't actually want applications on their mobiles!

What Mobile 2.0 does not need is traditional applications, because applications just get in the way of this sharing, mashing and updating of live and personal data. What most people want on their mobiles is not the applications, but the stuff they animate.

People only accept the concept of applications (whether a native app or a Web app) because that's all they've been offered, and it's largely good enough. But no-one actually wants to download and launch and register and log in to a local find-your-friends application - they just want to find their friends in the area - now! And they shouldn't then have to flip between the find-your-friends map owned by that application and the restaurant review map owned by another.

They don't want Facebook videos and YouTube videos and phone videos. They just want to share videos. They shouldn't have to think about whether to send a picture by MMS or to use an upload app, after remembering the login. They don't want multiple ways of sending messages: IM, SMS, Twitter, Facebook, etc. They shouldn't have to think about how to tell their friends about some news item - whether to post a TinyURL link on Twitter or copy the text manually into Facebook.

They only want one shared calendar, not the phone calendar and a Google calendar and events on Upcoming.org, that need two more logins. They shouldn't have to think about how to synchronise music or contacts lists on the phone, the iPod, the PC, some memory card and online.

Applications - both native and Web, and including Applets and Widgets - just get in the way of life! They try to own or control our People, Things, Times and Places, breaking them up with arbitrary application boundaries and making things unnecessarily hard to do.

People just want to share new stuff through their mobiles

What most people would prefer is for their friends and messages from their phonebook, their Facebook, their Twitter, their IM, plus their Flickr photos, their Facebook photos, their restaurant and film reviews and their meeting places all to appear on the same map or in the same calendar... along with the local weather forecast for now and for any dates being planned.

And for that map or calendar, those photos, messages, interesting news, etc., to be actively shared amongst those friends, or even the entire world, simply by flicking the 'share with friends' or the 'make public' switch. Any member of a group should be able to post photos into a shared gallery, or scribble onto a shared whiteboard.

The Killer App for Mobile 2.0 is the sharer, masher and updater of People, Things, Times and Places. This Mobile 2.0 platform would seamlessly merge, update, synchronise, upload, download and save your latest life stuff, including that of your friends and family, so that you no longer have to think about doing it yourself.

Your identity and your stuff would be owned and controlled by you through your personal handset. They would not be merely transferrable - they'd simply be transferred, to where they're needed, when they're needed, using an open Internet protocol and standard formats.

What this Mobile 2.0 platform must do

This platform will have a tactile 3D interface, with touchable, draggable areas for People (friends, family, organisations), Things (photos, videos, messages, feeds, news, notes), Times (appointments, plans, weather) and Places (maps, galleries). It will seamlessly access and present this interactive content from the phone itself - representing you - from friends, and from third parties.

This interactive content will be findable by tags, geo-tags and timestamps - it's semantic data, not document text. You'll be able to pick a Person, then see their related Things, Times and Places; or pick a Thing, to see it's associated People, Times and Places. Stuff will also be findable by recommendation and trust, or at worst by well-targetted advertising.

Loss of Internet will be handled gracefully - stuff from others will still be visible, but simply not updated. There are no central servers, so loss of a server only affects the particular content it sources and 'animates' for you.

Third parties will easily be able to add functionality and interactive content such as map overlays, media, news feeds, etc.. They just expose them using the open protocol and notation - they won't have to ask anyone in order to do so. Some contributors can adapt the Facebook, Flickr, Twitter and various IM APIs in the same way. This will require a system for reconciling the multiple identities of users and their contacts.

There will need to be a virtual property system backed up by crypto keys and supporting payment - just because we don't have an App Store, doesn't mean we can't charge for interactive content! It will also be possible to charge for premium services, such as remote storage space for over-enthusiastic photo-snappers.

Blueprint for the Mobile 2.0 Killer Application

Right, we need to describe some more important things with capital letters, because we're going to talk about the best open notation and Internet protocol for sharing, mashing and updating...

We've already introduced People, Things, Times and Places. Call these Entities. Entities are open data - they have a current State. We'll use a text-based notation to describe their public State, because it's easy to read and write. Each Entity will conform to an open standard public schema or syntax.

Entities can Link up because they will each have a UID (unique identifier). Some friends-People met at the last-week-Time and the Tower-of-London-Place and took these photo-Things. I have a collection Thing of People and both personal and public collection Things of photo-Things. You get the idea. That's a form of mashing. Oh - photos on the Web can be wrapped in Entities that pull out their metadata, and Web URLs are valid Links.

Any Entity can Observe another. Well, it's mostly People Entities doing the Observing and being Observed, via the mobile device: my Person Observes your Person to see what you're up to, then Observes your collection Thing of latest photos and may watch you on the map Place as you head towards me snapping more. Sharing and updating.

But some Things can be Observers, too; the more interactive ones that are affected by Entities around them. A simple example may be a restaurant review summarising Thing, that Observes and averages the scores off several reviewers' submission Things. Each time a score is added or changed, the final score adjusts accordingly. That's sharing and updating, too, and Entities that depend on other Entities is another form of mashing.

Entities have read permissions, which control which People and other Entities can see their State. Crypto tech may be needed to support this in some cases. Entities don't have write permissions as such, because they control their own State as a result of what they Observe.

Of course, these Observations occur over the Internet. This will use an open standard publish-subscribe, peer-to-peer protocol. An Internet packet will subscribe to an Entity's Link, another packet will return current Entity State, subsequent packets will broadcast State updates to current subscribers. State is transferred into servers and handsets as required, by a combination of pull and push - sharing and updating. The protocol will ultimately also have to support things like streaming and real-time media; over Wifi at least, so as not to scare off the carriers...

The U-Web Mobile 2.0 platform

This Mobile 2.0 Killer Application for sharing, mashing and updating People, Things, Times and Places ... will have a shorter name! I'm calling it the 'U-Web' Mobile 2.0 platform at the moment, reflecting its affinity to the Universe Web I described before. The U-Web platform application will be free and the protocols and formats outlined above will be open.

The typical handset specification and APIs we're aiming for are: at least 320x320 touch screen with Embedded 3D; GPS, Wifi and good 3G; front- and back-facing cameras.

There's a large number of Windows Mobile devices that match this, including phones from HTC, Sony-Ericsson, Samsung, etc. There's currently one Symbian (the Nokia 5800/Tube with the N97 to come next year). One iPhone. One Android. One Blackberry without Wifi (the Storm).

Although Nokia's plans are somewhat unclear, their S60v5/Qt/Ovi offering looks like it may preclude the U-Web, especially since the U-Web wants to be the top app... The U-Web certainly wouldn't be allowed onto the Apple App Store - the clue is in the name! Writing it in C also rules out Java phones, such as Android - which in any case is another app-oriented platform - and any Blackberry handsets.

The U-Web should thus be written for Windows Mobile 6+ Professional handsets. At least their users will have a ready appetite for anything integrated and easy to use, that keeps them clear of the .. clumsy .. operating system. It will also be worth keeping an eye on the various Linux MID (Mobile Internet Device) platforms, including OpenMoko, Maemo and Moblin, and maybe LiMo. Similarly, we should write for Windows and Linux on the server side - the code will be essentially the same. It'll need to be written in C as the lowest common denominator, most efficient language.

The U-Web will be the top or idle application: on Windows Mobile it would replace the Today screen, or more likely TouchFLO 3D or Spb Mobile Shell. Else it could be an Xperia X1 Panel. Like these, it will give access to phone functions such as calls, texts, camera, speaker mute, Bluetooth and Wifi enable, lock screen, brightness, etc.

Of course, it won't actually be the only application while it is in its beta phase - people may still want to access a browser, email and various settings and utilities. The need for browser and email should reduce significantly when functions such as Web 2.0, messaging and file sharing are drawn up into the mobile-native U-Web, leaving just actual documents fitting awkwardly.

Opportunity

It is a mistake to see Mobile as just a second-class computing platform or as a miniature office desktop onto which you must squeeze applications and documents.

Mobile represents a whole different world of personal interaction and almost 100% presence - especially now that mobiles are becoming Internet-connected and location-aware and are being furnished with top-quality cameras and touch screens. The mobile device represents you, here, now; what you're seeing and doing, to your social network.

A Mobile 2.0 platform needs to drop closed applications and go instead with the flow of People, Things, Times and Places. It needs to transparently share, mash and update people and their stuff.

The U-Web Mobile 2.0 platform described here is designed to seamlessly merge your real and virtual lives, starting with intuitive, 3D touch interfaces that give an almost tangible feel to virtual property, then expanding out to touch the 3D lives of others.

If you take a photo, your friends can instantly 'join you on the map' where you stand and chat about it - and the weather conditions you're experiencing. No need to upload to photo sharing sites with logins, or to launch isolated chat, map and weather apps.

The opportunity is billion-scale - increasingly capable mobile devices will become the world's primary connected computing device, outstripping PCs and making current Internet and Web usage look like just a warm-up.

We should be in there with appropriate software when that happens. We can stumble into this step-by-painful-step via applications and browsers, or we can just let go and meet the actual requirements - with a platform for mashing dynamic, shared data.

I'd be delighted to hear from you if you're interested in helping or getting involved in any way with building this platform! Email me - link's on the left - or leave a comment.

Business Conversations | The REST Dialogues

2008-12-11T11:45:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 7: Business Conversations

eBay Architect: Now, I have a few more questions about the areas where REST is generally considered to be weaker than SOA. I'll start with a big list, then we can go through each one.

Duncan Cragg: OK, go ahead!

eA: Right - it's basically the list of techniques you need to architect complex distributed applications: Service Discovery, Service Description, Client/Server Session State, Business Process Coordination, Workflow, Orchestration, Choreography; Security, Reliable Messaging and Transactions.

DC: Phew! Well, some of these beg the question; by assuming you need them, you end up concluding that you need SOA!

Also, seeing WS-standards for these things, some people think that SOA has them covered and that their work is done. But you still have to design and architect your system on top of these standards, to meet the business need.

In REST or ROA, we're talking about a different way of seeing distributed systems that requires a break from Service- (and even Object-) Oriented thinking and an embrace of Resource-Oriented thinking.

eA: So what's the REST way of approaching my list?

DC: Do things the REST way and there are three possibilities: (a) that it's easy to implement these functions in REST; (b) that you don't need these things anyway, or (c) that it's already your job to do them as an application architect; it's part of your business process, so you do work that you needed to do anyway in SOA, but do it in a simpler and more powerful way.

eA: Already my job?

DC: It's your job while you're designing your resource interactions at the application level: the resource-type or business level. Or maybe while you're making use of some existing resource types and 'resource-animating' code that does your business logic for you. Having WS-* only clouds and complicates this task.

eA: You've just hand-waved away a billion-dollar standardisation and tooling industry!

DC: Standards and tools are essential - we need them. But simple standards and tools are best. HTTP, URI, XML and standard schemas are perfectly good enough standards to get started with, guided by the REST or ROA mind-set.

I'll grant that we do need some extra help to support RESTful resource interaction in our domains and applications. I would like a few more standards and tools, over and above HTTP and XML, to help me build these complex distributed applications. Some help configuring and programming the way an application's resources interact at a business level.

eA: What exact support would you like for REST integration?

DC: We would benefit from standards, tools and frameworks that help define resource URIs, mapping to GETable and POSTable URIs, easy cache control, easy support for various response codes, HTTP Auth-based and URI-based identity, inter-resource links, collections, content types, Microformat support, useful schemas, structures, sets of simple default behaviours, APP libraries, business rule engines, etc.

It is still early days for the adoption of REST - an adoption that's undoubtedly been slowed by the industry's obsession with SOA. We do have Restlet and the JSR 311 work, as well as some help from Microsoft in WCF and the ADO.NET Data Services Framework, but there is still much more that could be done.

eA: Indeed. There's lots of work to do to catch up with all the SOA standards, tools and frameworks!

DC: I really don't want ROA to compete with SOA in bureaucracy and vendor in-fighting! But REST needs much less than SOA. Some things are simply part of REST: you discover stuff by following links; you know what something can do for you because of its content type or schema - we went over this sort of thing before.

Service Discovery and Description

eA: So, following links and seeing content types cover REST 'Service Discovery' and 'Service Description'?

DC: Yes. When you start thinking in terms of resources not actions, it all becomes much clearer:

In real life you might walk up the street, ask someone where the nearest shoe shop is or consult a directory, then see a likely shop, decide it may have what you want, then go in to investigate. Inside, you may consult more directories or just wander around. You see the cues around you (shoes on offer, checkout desk), and act or react accordingly.

You engage at a chosen level of understanding: there is a generic concept of street layout, then generic shop and shop assistant understanding, then an understanding of how to view and interact in shoe shops in particular.

eA: And in 'virtual life'?

DC: The Web is exactly the same: you explore through links or by asking a search engine, see what you want, enter the online shop, follow the cues, etc. It's no accident that the cyberspace metaphor is often applied to the flat, 2D Web.

eA: And in REST integration?

DC: In REST integration, the metaphor can be extended for machine interaction: get a list of items by direct reference or by query, go through them finding something that matches, interact according to standard business types and follow links between business resources.

eA: So no service contracts for you, then, just exploring and hoping for the best. Sounds a bit sloppy!

DC: You can do anything with anything in computing, it's just a matter of what metaphor is the most powerful or expressive.

You can design your 'machine cyberspace' to include contracts if that's what you need, just like in real life, and just like we went over with the eBay/gBay example.

But in general - especially when building distributed systems - it's best to start with loose arrangements and established conventions; adding constraints, contracts and Central Control only where absolutely necessary. Ensure central control only over schemas.

Make your Enterprise 'mashable' and everyone will thank you. Expose the UUIDs and GUIDs of your data in URLs. Transform and enrich your data within standard content types. Link it all up with your own data and data from elsewhere. Allow two-way interaction. Then publish it for future generations to discover and re-use!

Client State and Sessions

eA: Right, now how do you model complex state transitions, state evolution and conversations within a changing context, in a supposedly stateless architecture?

DC: There's a common misunderstanding about REST that its statelessness should extend above the protocol (e.g., HTTP) into the application. On the contrary - once above HTTP and into a Resource-Oriented application, it's all about state - as long as that state has a URI!

All the stateless requirement is saying in practical terms is that each HTTP request and response exchange is a one-off as far as HTTP is concerned. It means that you don't need to tie successive exchanges together at the protocol level, which makes implementing HTTP easier. HTTP just wants a URI to be a URI, and the content there to be manageable via headers; and HTTP doesn't and shouldn't introspect either the URI or the content.

eA: So where do you keep the conversation state, then?

DC: Above HTTP in the world of URI-tagged state. You can have an ongoing 'conversation' between a client and a server resource - as long as those resources are linkable and fetchable.

You don't need sessions below the resource level; below the business level. If your application truly demands the concept of a sequence of interactions with a start point and an end point, then go ahead and implement it - at the application or domain level, not at the framework level.

Most applications can use ad hoc, asynchronous interactions, where the client identifies itself each time with Authorization headers.

eA: Can each client machine have assigned to it dedicated server resources as part of this conversation, sort of like an explicit session state?

DC: Yes. Cacheability and linkability are clearly affected, but if it truly makes sense in your domain model to have client-specific resources, then just do it - by definition their scalability and findability is limited to that client! The client can still cache its own view, as in a browser, and even intermediate proxies can, although it's of benefit to only one client.

eA: Presumably it's a bad idea to put client state in a Cookie header?

DC: Yup: it's not got a URI, so its state is hidden. Put it on a URI in a server.

Hidden state is a red flag.

You know you're on the right track as long as you are exposing your statefulness in URIs. It should always be obvious where things stand from inspecting public state alone, to know what is possible and what will happen next.

If you find yourself hiding state in sessions and cookies, or returning different data dedicated to that client from the same URI by setting no-cache, or if you tie up successive interactions through sessions, you are going down the wrong track.

eA: Are cookies always bad?

DC: A Cookie header should be used only to identify the client, not a session. You'd probably use the Authorization header in REST integration anyway. Just use Cookie headers along with an auth scheme if you need a more elaborate, perhaps multi-layered or proxied, authentication system.

eA: What about the Vary response header and its use with Authorization or Cookie request headers?

DC: Mechanically, from the cache perspective, adding Vary on client-identifying headers like these is the same as using per-client URIs; setting the Vary header to allow the URI to be cached on a per-client basis is a hidden way of effectively adding the identifying data from the Authorization or Cookie header to the URI itself.

Actual per-client resources are explicit; these Vary'd resources are implicit. You choose which to use at the application level based on the importance of the resources being identified in your domain model. Either way, the identity of the requestor can be used to determine read permissions.

The explicit way means you can pass links to your personal resources to others, modulo read permissions. A client may or may not be able to or want to pass the link to its dedicated resources to other players - it probably isn't a link they want anyway - but they can still pass around general jumping-off links, that make sense for other recipients.

eA: To be honest, I just don't like the idea of the server saving and distributing a ton of this client-specific resource data!

DC: There are two answers to this: firstly, now that we're using REST for business data integration, not generating entire pages of HTML, the chunks of data can have much finer grain. The actual amount of client-specific data can be much smaller! You can cache and share links to all the generic chunks, and only transfer small amounts of less cacheable and linkable per-client data.

Secondly, our clients in REST integration are also often servers, so can in fact expose their own state. We may be better off just putting those client-specific resources on the client itself, to get around this whole issue. The client isn't a browser any more!

We're back to the symmetric Distributed Observer Pattern.

Business Processes

eA: OK, now what about Business Process Coordination, Workflow, Orchestration and Choreography in that sloppy cyberspace of yours?

DC: We've already discussed the auction business process, both single-site and cross-site, which was quite a good example that covers a number of business activities.

Another good feature of REST is the way it naturally maps onto declarative business rules. When you switch from process-thinking to resource-thinking, you also switch from imperative thinking to declarative. This manifests as inter-resource dependency and transformation.

eA: Eh?

DC: A spreadsheet is an example of this style, where you declare the way cells depend on each other and then let the hidden magic take care of satisfying your constraints. We've discussed this style of programming before.

Again, tools will be essential to help with this. Managing and testing event-driven business rules over shared and distributed state is somewhat new territory, even to most REST integrators!

eA: Business Processes and Business Rules seem like different things to me.

DC: Well, as we saw, the business process of an auction can emerge from the local application of business rules. The thing is to let go of the myth that you benefit from specifying and controlling some over-arching vision of your business processes - that it makes sense to centralise control of what actually works best when delegated and decentralised.

Just write the local What, not the global How, and let the process emerge!

eA: Might be a hard sell to control freaks.

DC: Yeah, true enough. But it more closely maps onto the reality of the way businesses operate and interoperate. It's more about the actual peer-to-peer business interactions and visible state evolution, and less about central controllers that know the intimate, inscrutable details of the Web Services involved.

Instead of coordinating the import and export of data from one hand-coded interface to another, you can just link it all up and expect everyone to dereference and recognise your data. Instead of coordinating sessions with implicit state, you can just react to standard, public data types.

In Part 8: WS-Are-You-Sure (Security, Reliable Messaging and Transactions).

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

The Universe Web

2008-07-18T19:49:00Z

Since the announcement by IBM and Linden Lab that OpenSim can talk to Second Life, I've been thinking again about RESTful Virtual Reality.

I'm not the first, of course. Others have been motivated by the same goal: To bring the Web's scalability, linkability and interoperability into Virtual World platforms.

Ultimately, how to use the same techniques as the Web to link Virtual Worlds together into a single, massive 'Virtual Universe'.

Here's how I would architect the Universe Web...

Entities

The Universe Web needs Entities. (I'm going to be capitalising Significant Things a lot here to help the Web comparison - hope you don't mind). There's no need to make the Entity list too complicated since we're not doing online games yet - just a world like Second Life.

Here's a list of different kinds of Entity:

Places: buildings, streets, hills, lakes
Things: trees, books, birds, clocks
People

State, Links and animation

Entities aren't exactly the Objects of Object-Oriented programming: they have lots of public State. In a Virtual World, you can directly experience their colour, position, orientation, physical relation to each other, etc., thanks to the local render engine.

The basic structure of the Universe Web is made up of Linked Entities. A tree is Linked to its ground which is Linked to a building which is Linked to its occupants which are Linked to their clothing. Kneebones may even be Linked to thighbones.

Entities are often animated. Actually, Places are generally static, and People generally animate themselves. So, when talking in general, we should look at the animation of Things, like trees, books, birds and clocks.

There are two ways an Entity's animation can be guided: by Laws and by Rules.

Laws

Springiness is a handy Law. For example, tree branches, pages in a book and birds' wings and legs can all have springiness. We can declare a mid-air location in a bird's State and a springiness in its legs, then if the bird lands on a branch, the springiness can trigger the bird and branch waving gently to a halt.

We could even allow a single bird to become part of a flock Entity, where the flock only knows how many birds it has and the coordinates of its centre. Each bird will then be animated by global flock Laws.

The book can have a State of open or closed. If bookness was elevated to a global Law, then a book deciding that its State was now simply closed could trigger the elaborate, springy closing of its pages with a nice 'shlump' sound at the end.

The clock's current time may be set by the clock Entity itself, then the Law of time take over in between the clock updating its own time directly.

A Law can be invoked over an Entity's animation by simply adding the Law's name and parameters to its State. Such Laws can be handled by the physics engine in the same way surface texture is handled by the render engine: States such as 'red' need global meaning, Law applications such as 'springy' need global meaning.

Rules

When a clock reaches its alarm time, it rings. When its wizard master approaches, a magic book may glow and open. Birds fly away from danger (if a fox is too close, fly away from it). In between danger, birds and foxes look for food (if a bird is close, run towards it).

These are Rules. Rules are driven by the Observation of one Entity by another. The fox Observes the bird, the book Observes the wizard. Rules apply to Entity types: fox Rules, bird Rules, book Rules, etc. Rules can guide the animation of an Entity by evolving its State.

State evolution depends on the current State plus the States of the Observed Entities: Rules make an Entity change from one State to the next by taking the Entity's own State, Observing the States of the Entities around that it cares about, then deciding what the next, new State should be. If an Entity changes State, those Entities Observing it may themselves change State according to their own Rules, and so-on.

Links are part of an Entity's State, and Links may therefore change on the application of Rules. In other words, the virtual world may restructure when a Rule is run.

An Entity may either Observe another Entity, or Link to it, or both. An Entity that Links to another needn't Observe it. Trees don't bother to Observe the ground they grow from. Conversely, an Entity that Observes another needn't Link to it. A fox would only Link to a bird if she was lucky enough to catch it. Meanwhile, it is true that there's probably a chain of Links to everything that's Observed.

Unlike Laws, Rules can't or shouldn't be second-guessed, so have to be run 'at source' or 'inside the Entity', then the new State distributed to both the Observing Entities and to all the render engines.

But render engines are actually Observing People Entities..

People are First Class

In Virtual Worlds, we have People avatars: People are first class Entities. People can Observe and can change State, but it's a real user doing the animation, not a set of Rules. Observation and State change by a Person is implemented through the render engine.

Although self-animated, People can still make use of Laws. For example Laws can be invoked to allow gestures to be easier to make and to look better, and to allow People to 'run forward' rather than putting their 'left foot forward, ..'. Avatar skills become global Laws which can be triggered by an avatar's State animated by a user.

The Universe Wide Web

The Universe Web I've just described is a lot like the World Wide Web:

An Entity is like a Web Resource
The State of an Entity is like an HTML Web Representation of a Web Resource
A Link in an Entity's State is like a Web URL inside an HTML Web Representation
Laws are like Javascript, Flash, link hover, blink tags and animated GIFs
Rule sets are like PHP scripts or Java servers
A State Observation is like a Web Representation Transfer

A crucial difference between the Resource Web and the Entity Web is that, here it's another Entity doing the Observation. One Entity transfers its State to another.

It's as if the GET or POST has another Entity 'at the client end'. GET is like an Entity subscribing or Observing; POST is like an Entity publishing or notifying its State to another (an HTML form has a content type, too).

People are just examples of that Entity 'at the client end'. Instead of a user hidden behind a client browser, we have first-class People Entities doing 'GET's and 'POST's: Observing the Virtual World and acting (i.e., changing State) within it. Other examples are foxes and clocks.

Notation and Protocol

There are two open standards we need to have in progress while implementing the Universe Web: the notation and the protocol (our HTML/URI and HTTP).

Firstly, we have to define the notation for Entities and their Rules. Both the State of Entities and the Rules that animate them will be written in the same notation: a Rule talks directly about State. To me, basing that on JSON (rather than XML) is a clear choice. Add inter-JSON Entity Links (essentially UUIDs), then work out a simple JSON Rule language. We will then need to define JSON 'schemas' for the kinds of Entity we would model. These can be written in the Rule language itself.

Secondly, to define the protocol for Entities to Observe or subscribe to one another. It will be a 'symmetric, asynchronous HTTP' - a Peer-to-Peer, Publish-Subscribe protocol. UDP and Multicast stand out for this kind of interaction. A packet will subscribe, a packet will return current State, a subsequent packet will multicast State updates. We can re-use all the HTTP cache headers, only now we can push updates into caches and into render programs. Packets will inevitably end up tunnelled through TCP and HTTP, mind...

This lighter notation, which also breaks the data up into smaller pieces, and this multicast pushing of State change into caches, along with P2P-style operation, will mean the Universe Web scales even better than its parent.

Also, programming declaratively using Rules is both highly productive and further cleanly scalable through parallel processing.

Of course, it will be pretty trivial to allow inter-Web cross-over such as referring to PNG texture images via URL and pulling Universe Web Entities into a Javascript application that subscribes to them via Comet.

Building the Universe Web

So that's my Universe Web. It's basically the same as the World Wide Web and thus as scalable, linkable and interoperable. It comes with added symmetry: Entities, including People as avatars, exchange State in each direction.

It's a two-way data Web, not a one-way document Web.

I've started work on the Universe Web: the notation, the protocol and the implementation (in C, for portability across platforms, including mobile). Contact me if you want to help!