What Not How - Posts tagged 'forest'

JSON-Mash

2010-03-18T16:58:00Z

Around the middle of February I completed a basic persistence and networking implementation for Fjord, then had to do other things for a month. Just recently I fixed Fjord to work with the latest version of the Node.js APIs.

Next project: I'm going to use Fjord in a Web Framework to be called "JSON-Mash".

I intend to show that JSON-Mash will be a great framework for rapidly building truly interoperable and truly scalable online and distributed functionality.

Here's how JSON-Mash will work.

Juggling Data Formats

In traditional Web Frameworks, the developer has to juggle up to five different data formats.

At the front end, you may be working in HTML or HTML chunks sent via Ajax. Then your domain logic may use object classes. You'll no doubt map those objects to relations in your database layer. That's three data representations (also known as Circles, Triangles and Rectangles).

But there may be even more: if you call out to integration points such as a search service or other API, you'll probably be using XML. And if you did application-level cacheing, you would probably consider casting your data into key-value form.

This is not unrealistic - at a current ThoughtWorks client, I was faced with exactly these five data formats - in an extremely simple application.

JSON Everywhere

JSON-Mash will do away with all that. It uses JSON objects everywhere.

JSON-Mash sends JSON data to a script in the browser that builds the page declaratively from its understanding of a number of stable, generic JSON types. It uses Ajax and Comet to pull JSON chunks in, to composite into the page.

JSON-Mash uses Fjord to process its JSON: transforms, map-reduce, etc. It uses Fjord to persist objects in a JSON NoSQL database, and Fjord to provide shared JSON cacheing functionality with cache updating via Fjord's FOREST integration style.

Finally, if there's any external system to be integrated, that too will be accessed via an adaptor that converts its API, etc., into JSON-Mash stabilised JSON. There will be Twitter and Flickr adaptors at first, I should imagine.

JSON-Mash and Fjord

Using Fjord means that JSON-Mash treats its JSON as a first-class hyperdata type: lots of little JSON chunks all with URLs and all linked up. I suggested that one day it may have the media type of "application/fjord+json". JSON-Mash Nodes communicate in the FOREST style.

Of course, the main benefit of using Fjord is that you get a great, declarative mashup language to describe your logic in.

Fjord is still very new, of course, so some goodness is yet to be rounded out. For example, it comes with a very basic, but highly improvable, NoSQL JSON datastore and already allows some powerful map-reduce and other query and transformation modes, with more to come.

Finally, Fjord gives a dynamic view of JSON data: caches get updates pushed into them and Comet pull will also come as part of the package.

CSI and SSI

Now, one of the great things about Node.js is that you can run the same code on the server as you run in the browser.

My previous attempts at doing things the JSON-Mash way have met with some resistance when I suggested building duplicate logic server-side to render pages statically, for accessibility and search engines.

But with Node, this problem goes away: you can use the exact same Javascript to convert JSON to HTML on the server side as you do on the browser.

I use the phrases 'Client-Side Include' and 'Server-Side Include' or CSI and SSI as easily-understood shorthand for this kind of page assembly from JSON elements.

Settling on a number of stable JSON types means that - once the types and their rendering are stable and coded - you never need to write any Javascript yourself in JSON-Mash. It's all done declaratively, and looks the same whether generated by an SSI Node or the CSI in the browser. For example, you can trigger calendars and maps in the browser view, with just small JSON declarations.

This approach differs from "progressive enhancement" as long as you have both full CSI and full SSI to render your site. But you could also have a 'CSI-SSI slider', where an initial page is generated by SSI, then CSI takes over to fill in the funky stuff: the dynamic widgets and Comet-driven elements.

Users Exploring a Dynamic Web of JSON

The way JSON-Mash works is that you get a view into a hyperlinked Web of JSON data, and can explore it from any point outwards. In the CSI version, each JSON chunk is loaded lazily as it is encountered, then reloaded when it changes. Even the SSI version of the site would allow this form of exploration, although in a less efficient and dynamic way, of course.

In JSON-Mash, the user is also a first-class object; the stable JSON types also include a type representing the user. This user is driving their browser, subscribing to events on visible JSON page components via Comet, and POSTing their own user state to a server-side cache copy.

Hence, if that user state contains a message, you can get simple chat functionality, without any code, just by linking to a user and viewing them.

Conclusion

JSON-Mash is a Web Framework that will use, re-use, assemble, mash, transform, store, share, push, pull and render JSON from end to end and top to bottom; from Ajax to NoSQL.

Most domain-logic programming will be done in simple, powerful declarative functions, with adaptors, drivers or renderers written in Javascript for Node.js or the browser. Cross-host exchange is done in the FOREST style.

SSI or CSI page assembly are very flexible ways of mashing up new functionality from multiple back-end sources of RESTfully-sourced, interlinked JSON data. And with cache updates telling SSI and CSI when the back-end data has changed, you get a very dynamic yet scalable system.

JSON-Mash will be a framework for rapidly building truly interoperable and truly scalable online and distributed functionality.

I'll write more on the benefits once I've got some code to prove it! Follow me on Twitter and I'll let you know when I've got something up on GitHub.

Fjord in Memory

2010-01-26T13:46:00Z

Right, I'm pleased to say that I've now implemented enough of the Fjord language on Node.js to be able to run the Instrument example that I introduced it with. As yet, this runs in memory only - i.e., no disk, no network.

Here's the code on GitHub with tests that show how it works. The language has changed a little so I'll show the example here again, copied over from the test code, in order to explain the differences.

Starting with the instrument rules:

inrl1=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "%owid": "/$this/",'+
                       '  "%refs": { "tags": [ "equity", "bid" ],'+
                       '             "%owid": "/$bid/",'+
                       '             "on": "/$this/" },'+
                       '  "buyers": "/array/has($bid)/" }');
// -------------------------------------------------------------------
inrl2=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "%owid": "/$this/",'+
                       '  "%refs": { "tags": [ "equity", "ask" ],'+
                       '             "%owid": "/$ask/",'+
                       '             "on": "/$this/" },'+
                       '  "sellers": "/array/has($ask)/" }');

These are the rules that add - or, rather, ensure the presence of - referring bids and asks in the instrument's buyers and sellers lists. The big difference here is that referrers are now accessed through the '%refs' tag. Both '%refs' and '%owid' are available on all Fjord objects: %owid is the Object Web ID of the object, and %refs has a list of the OWIDs of all currently referring objects.

But, of course, you can't see the list or the OWIDs under %refs! This rule shows how Fjord can match single items to lists of items, and how it transparently jumps any OWIDs it comes across, to continue matching. In this case there is a pattern for a bid or ask object that refers to the instrument - notice the "on": "/$this/" match.

Every object in %refs must have some such link to the referred-to object that triggered the referral in the first place. This referrer becomes an observer: its rules are run again if any object it refers to changes.

More instrument rules:

inrl3=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "buyers":  { "price": "/$bids;number/" },'+
                       '  "bid-ask-spread": { "high-bid": "/number/max($bids)/" } }');
// -------------------------------------------------------------------
inrl4=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "sellers": { "price": "/$asks;number/" },'+
                       '  "bid-ask-spread": { "low-ask":  "/number/min($asks)/" } }');

These are the rules that set the bid-ask-spread of the instrument from the prices of the bids or asks in each list. It shows how, when a single match instance binds to a list, it can give a match set in any bound variables - here, '$bids' and '$asks' - which can then be reduced - here, using 'max()' and 'min()'.

The rules are now separated into two, as there was a historic first ever bug in public Fjord code: the original rule wouldn't match unless at least one buyer and one seller were present, and we're adding them one by one from scratch this time!

Here is the actual instrument object:

inst=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                      '  "long-name": "Acme Co., Inc",'+
                      '  "buyers": [ ],'+
                      '  "sellers": [ ],'+
                      '  "bid-ask-spread": { "high-bid": "10.0", "low-ask":  "20.0" } }',
                      [ inrl1, inrl2, inrl3, inrl4 ] );

Notice the list of rules that the object will be animated by: inrl1, etc., which are the rule object OWIDs, not the actual rule objects. The bid-ask-spread is set to some seed values.

On to bid and ask rules:

bidrule=WebObject.create('{ "tags": [ "equity", "bid" ],'+
                         '  "on": { "tags": [ "equity", "instrument" ],'+
                         '          "bid-ask-spread": { "high-bid": "/$hibid;number/" } },'+
                         '  "price": "/null/fix(2, $hibid * 1.10 )/" }');
// -------------------------------------------------------------------
askrule=WebObject.create('{ "tags": [ "equity", "ask" ],'+
                         '  "on": { "tags": [ "equity", "instrument" ],'+
                         '          "bid-ask-spread": { "low-ask": "/$loask;number/" } },'+
                         '  "price": "/null/fix(2, $loask * 0.90 )/" }');

Here we have rules for setting the price of a bid or an ask as a ratio of the instrument's bid-ask-spread numbers. These are not meant to be realistic business rules of course - they're for illustration purposes. The only real difference here is fixing the result to 2 decimal places, since Javascript was adding ugly rounding errors.

Now we go ahead and create two bids and two asks, pointing at the instrument by simply including its OWID:

bid1=WebObject.create('{ "tags": [ "equity", "bid" ], "on": "'+inst+'", "price": "" }', [ bidrule ]);
bid2=WebObject.create('{ "tags": [ "equity", "bid" ], "on": "'+inst+'", "price": "" }', [ bidrule ]);
ask1=WebObject.create('{ "tags": [ "equity", "ask" ], "on": "'+inst+'", "price": "" }', [ askrule ]);
ask2=WebObject.create('{ "tags": [ "equity", "ask" ], "on": "'+inst+'", "price": "" }', [ askrule ]);

This fires off all the rules and notifications .. and rules and notifications .. giving the final result:

exp=new WebObject('{ "tags": [ "equity", "instrument" ],'+
                  '  "long-name": "Acme Co., Inc",'+
                  '  "buyers":  [ "'+bid1+'", "'+bid2+'" ],'+
                  '  "sellers": [ "'+ask1+'", "'+ask2+'" ],'+
                  '  "bid-ask-spread": { "high-bid": "12.1", "low-ask":  "16.2" } }')
// -------------------------------------------------------------------
test.objectsEqual("Instrument example works", Cache[inst], exp);

Here the buyers and sellers correctly list the referring bids and asks, and the bid-ask-spread has been set to the values you get after two ratios are applied to the original seed numbers.

Rewrite rules are run on object construction, they're run whenever an object has a new referrer in its 'refs' list, and also whenever another object that's being referred to changes. The latter isn't exercised in this example, as all events are propagated by referral alone. The tests of Fjord have examples of the full Observer Pattern in operation.

Next Up

Now, this is living code, so it will no doubt be different again, soon. Watch this space, and follow me on Twitter, to keep up to date with developments.

Next, I'll be working on simple disk persistence of Fjord JSON objects and rules, followed by distribution in the FOREST style...

Fjord in Node

2010-01-06T17:03:00Z

Well, I've put together the first few lines of Fjord, implemented on Node.js.

Here's the description on GitHub: Fjord is a language for expressing domain logic as match-rewrite functions over mashable JSON Web objects.

I'm developing Fjord very openly, in the hope someone out there will be interested in getting involved in helping guide its design and implementation. I suppose code speaks louder than blog posts.

It's going to be slow progress, with a wife, two kids, and Thoughtworks all together getting 99% of my life, but at least people will be able to keep up with what I'm trying to do... I only got the project started at all because I'm off sick right now!

Here's the GitHub link: http://github.com/DuncanCragg/Fjord.

Follow me on Twitter if you want all the latest news .. http://twitter.com/duncancragg

Oh - I was lying about FOREST being GET-only, that was just to engage the REST community, but since I got zero reaction from anyone I may as well admit that I intended FOREST to use POST all along - to push updated resources into subscriber caches as I have always done. Or to use long-GETs to pull them in. Both are tunnelling cache refreshes over HTTP. Neither are Official REST.

Cheers!

Introducing Fjord

2009-12-11T08:22:00Z

Following on from my recent article where I derived FOREST, this article offers the beginnings of a JSON unification and rewriting language that can be used in a FOREST architecture.

Why JSON, not XHTML, now? Well, I recently discovered that JSON is overtaking XHTML in interest, and I was further inspired by Kris Zyp's recent announcement of his JSON Schema Internet-Draft.

Fjord is a language for describing how the state of a JSON resource at any time depends on both its current state and on the state of other JSON resources that it links to via hyperlinks.

Fjord is a Norwegian word, probably pronounced 'fiyourd', and might stand for some combination of the words: 'Functional JSON Object/Observer Resource/Rule/Rewrite Dependencies/Declarations'. Or maybe it's just because they're truly awesome, an' I wanna go.

Fjord also gives me an opportunity to show some examples of the "end-user" view of a FOREST interaction; starting with a simple finance example.

Instrument, Bids, Asks

Let's take the simplest possible view of a financial market.

There's an Instrument, and there are Bids and Asks on that Instrument. The Instrument summarises all the Bids and Asks into the 'bid-ask spread', taking the highest bid price and the lowest ask price. Forget volumes, trades and any other detail.

The Instruments, Bids and Asks are on various servers across various organisations. That's the 'symmetric REST' part.

Here's a JSON Instrument resource at the URL shown. It holds three Bid or Ask objects:

http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json
{
    "tags": [ "equity", "instrument" ],
    "long-name": "Acme Co., Inc",
    "buyers": [ 
        "@http://a-bank.com/fjord/equity-bid-9ac0d1-88ce1.json"
    ],
    "sellers": [ 
        "@http://c-bank.com/fjord/equity-ask-510efb-cca62.json",
        "@http://d-bank.com/fjord/equity-ask-8560ae-33eff.json"
    ],
    "bid-ask-spread": {
        "high-bid": "10.00",
        "low-ask":  "14.00"
    }
}

The '@' character is the flag that this string is a link or a URL. The meaningful names in those links are really just for humans and the layers below us - Fjord itself won't let you see inside them.

So now that we've introduced a JSON extension making it a hypermedia or hyperdata type, perhaps we should announce this convention in a media type string. How about "application/fjord+json"? This media type will also serve to set expectations on everything else about Fjord JSON objects, once the notation is stable.

Notice that, instead of 'class' or 'type' or a top-level tag, we have a list of tags. In Fjord, class is in the eye of the beholder - but, of course, must be a stable grammar in the eye of the generator!

Right, so here is a new Bid object held at 'B' Bank, that the Instrument hasn't seen yet:

http://b-bank.com/fjord/equity-bid-0043ee-419ac.json
{
    "tags": [ "equity", "bid" ],
    "on": "@http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json",
    "price": "",
    "by": "@http://b-bank.com/fjord/hcard-b-bank-4990cd-232b0.json"
}

It has a pointer to the Instrument it is a bid on, a bid price that isn't yet set, and a link to the hCard object of the bank offering the bid and serving this Bid object.

So Bids and Asks can see Instruments and vice-versa, and each can set their own state accordingly, as part of a FOREST interaction.

The 'animation' of Bid, Ask and Instrument JSON object resources is driven by three simple Fjord rules...

Rule One: Bid Sets Price From Instrument

Here is a rule setting the price of a new Bid to be 10% more than the highest current Bid on the instrument, hopefully ensuring this Bid is now the high bid:

{
    "tags": [ "equity", "bid" ],
    "on": { "tags": [ "equity", "instrument" ],
            "bid-ask-spread": { "high-bid": "/$hibid;decimal/" }
          },
    "price": "/null/( $hibid * 1.10 )/"
}

This is a template that matches the Bid object and its dependencies and rewrites part of the Bid.

Notice how the Bid object's link to the Instrument object is jumped transparently: the Bid can 'see' right into the Instrument in order to do its matching. However, it cannot itself rewrite the Instrument - only Instrument rules can do that.

In Fjord, matching is written with the '/ ; /' notation: each part of this semicolon-separated list must match. In this case, the high bid has to bind to the identifier '$hibid' and has to be a decimal. The price must not be set yet.

The keys 'decimal' and 'null' are, of course, special reserved words for matching in patterns. The 'null' matches empty strings, lists and hashes.

The rewrite part follows at the end of a match and is also wrapped in slashes. Here, the price is being set to the high-bid plus 10%. The parentheses introduce an expression.

This rule only matches if the price is empty - once set, the rule will not refire.

For more complex rewrites, the following forms can also be used:

    "price": { "/null//": "( $hibid * 1.10 )" }
    "price": { "when": "/null/", "then": "( $hibid * 1.10 )" }

which allow full hashes and lists to be both matched and set.

Rule Two: Instrument Puts New Bid into its Buyers List

The Instrument adds the new Bid to its buyers list with this rule:

{   "#url": "/$bid/",
    "tags": [ "equity", "bid" ],
    "on": { "#url": "/this/",
            "tags": [ "equity", "instrument" ],
            "buyers": "/list/has($bid)/",
    }
}

This is a rule on the Instrument, that 'looks up to' and matches the enclosing Bid.

The '#url' is an optional, reserved-word entry that can only appear at the top of a JSON resource and holds that resource's URL. Here, it is bound to '$bid', picking up the URL of the Bid object that refers to and encloses this Instrument.

Further down, the "#url" reserved-word value of "this" indicates the start of the object that this rule applies to. If "this" is absent, then, of course, this is a rule on the top-level object, as in the first rule above.

The rewrite part matches the buyers list of the Instrument and applies the 'has()' operator which ensures the list now contains the wrapping Bid; it rewrites the list to include its argument if it doesn't already.

If this rule matches two wrapping Bids simultaneously, then '$bid' is set to a 'match set' - a simultaneous binding to the URLs of all of those Bids, and 'has()' must ensure that all of these Bids are now in the list at once.

If there is no change after a rule is applied, the resource's Etag isn't incremented, and dependent rules won't fire. Fjord may re-apply any rule at any time, usually on an incoming state change indicated by Etag, so matches have to be designed accordingly.

That's why we say 'has', not 'add', which would add the Bid again and again each time the rule is applied. We could say "/list;hasNo($bid)/add($bid)/", but that's uglier and more imperative than declarative.

Rule Three: Instrument Sets Bid-Ask Spread from Bids and Asks

This is what the instrument looks like, now, with the new link to our Bid:

{
    "tags": [ "equity", "instrument" ],
    "long-name": "Acme Co., Inc",
    "buyers": [ 
        "@http://a-bank.com/fjord/equity-bid-9ac0d1-88ce1.json",
        "@http://b-bank.com/fjord/equity-bid-0043ee-419ac.json"
    ],
    "sellers": [ 
        "@http://c-bank.com/fjord/equity-ask-510efb-cca62.json",
        "@http://d-bank.com/fjord/equity-ask-8560ae-33eff.json"
    ],
    "bid-ask-spread": {
        "high-bid": "10.00",
        "low-ask":  "14.00"
    }
}

And here is what the new B-Bank Bid looks like with its competitive price:

{
    "tags": [ "equity", "bid" ],
    "on": "@http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json",
    "price": "11.00",
    "by": "@http://b-bank.com/fjord/hcard-b-bank-4990cd-232b0.json"
}

So now the Instrument has to adjust its bid-ask spread to accomodate the new Bid. This happens when the following rule matches, which it always does - but only fires when a change is detected:

{
    "tags": [ "equity", "instrument" ],
    "buyers":  { "price": "/$bids;decimal/" },
    "sellers": { "price": "/$asks;decimal/" },
    "bid-ask-spread": {
        "high-bid": "/decimal/max($bids)/",
        "low-ask":  "/decimal/min($asks)/"
    }
}

This rule matches as many times as there are Bids and Asks, meaning that the local identifiers $bids and $asks are each bound to match sets.

Notice how the '{ price .. }' patterns match elements of a list, in the same way an XPath can match either one or several elements in a sequence of XML elements.

The 'max()' and 'min()' operators take the highest and lowest values in a match set (or in a physical list, come to that). Once again, this rewrite means 'set this value and increment the Etag, but only if there's a change'.

Here, the high-bid becomes our new Bid's '11.00'.

Fjord

So that's a quick intro to some of my ideas for JSON matching and rewriting in a FOREST architecture.

Fjord is a declarative, JSON-encoded language for expressing how the state of a JSON resource is a function of both its own state and the state of peer JSON resources that it can see via links in a hyperdata Web.

Fjord objects are 'masters of their own destiny', not allowing other objects to tell them what to do except via declarative intention.

Further articles will give more proposals for the syntax and semantics of Fjord, including true unification, and put it into the set of languages that are ideal for programming parallel and distributed systems, as well as for integration.

Since Fjord is very young and I'm still implementing it (in Java), I would certainly welcome comments about it, or about the FOREST style that it supports. You can chat with me on Twitter, too!

Deriving FOREST

2009-11-25T21:29:00Z

Say we want to integrate multiple applications which handle order processing. OK, that's got to be one of the dullest starts to a blog post. Never mind, bear with me...

So, we have applications on separate servers for handling and driving data such as orders, product descriptions and catalogues, stock lists, price lists, tracking, packing notes and delivery notes, invoices, payments, etc.

We may choose an SOA approach, of course. But let's say our sponsors have heard of this cheaper alternative: REST! Which to them means 'using Web technology to save money'.

Now .. suppose we push the time slider right back to before Mark Baker and the SOA -vs- REST Wars - or the 'SOAP -vs- REST Wars' as people naively called it. To when REST was simply (!) a description of the Web's architectural style...

What if we revisit the applicability of the Web, and its abstraction into REST, to the architecture of machine-to-machine distributed systems - to something like our order processing integration?

I think we'd quickly arrive at something that looks more like FOREST than, say, AtomPub...

Some pretty obvious things to notice about the Web and, indeed, REST:

The Web is essentially data on URLs of standard content types containing more URLs;
The Web is the Web because of the massive proliferation of links in that data;
REST mostly concerns itself with the consequences of GET, including cacheing;
The Web uses, I don't know, let's say 98% GET, 2% POST, around 0% other methods.

In other words, the Web, and its good qualities, are mostly based on:

GET URL -> HTML -> a.href=URL -> GET URL ..

When applying this Web/REST architectural style to our integration scenario, there are things that we can say right now with certainty will be different, but will have corresponding elements:

It's about data not documents, so HTML is probably going to be replaced by XML, although perhaps XHTML+Microformats or Atom would make a good compromise;
We have a choice of link specs: xhtml:a.href, atom:link.rel, xml:xlink; I don't think we'll be using XLink since no-one else seems to;
We'll probably use machine-generated URLs perhaps containing UUIDs, GUIDs or whatever.

In other words, we're not going to be spinning a hypermedia Web - it's more a 'hyperdata' Web.

So, in order to emulate the document Web in our hyperdata integration Web, we'll mostly be doing something like:

GET ID-URL -> XHTML -> a.href=ID-URL -> GET ID-URL ..

Oh! I've got some slides of all this on Google Docs: we're up to Slide 2! Maybe right-click, open in a new window...

Symmetry - Slide 3

But by far the biggest difference between the Web and an integration scenario is that the asymmetry on the network goes away; even for a cross-enterprise integration.

Where the Web's browser clients and site servers have always been asymmetric - clients being hidden away and only able to establish outbound connections - machine-to-machine integration is fundamentally symmetric - all servers can be made visible to each other.

Now, in order to keep the well-studied benefits described in REST, including separation of concerns, we should aim to maintain the client-server, layered structure in the use of the protocol.

But clients can be servers and vice-versa!

So, in machine-to-machine integration scenarios, we have:

Two-way GETs on machine-minted URLs pointing at XHTML+Microformats or Atom content containing more links.

In other words:

A hyperdata Web both created and consumed by the applications being integrated.

All of the dynamic data or hyperdata items in our order processing scenario will be distributed across the many applications being integrated. Each application serves its part of the hyperdata Web to the others.

And, of course, the hyperdata joins all these applications up: a payment resource in the accounting application will point to an order resource in the order processing application, etc.

Interactions and Application State

Now, each application has its own set of business rules and constraints over the hyperdata parts that it governs.

So how exactly should the applications publishing those bits of the hyperdata Web interact? How do orders interact with packing notes and stock levels, with payments and accounts?

In the Web, you go to a site and jump some links. Each page brings in CSS, Javascript, images, maybe iframes: an eager assembly of pages from links to many resources, in contrast to the lazy, on-demand fetching of links from a user jumping them.

The browser at any time has a state that depends on the page and images, etc, currently being viewed, plus the history of previous pages, bookmarks, etc.

Search engines in the Web, without a user driving things, are eager to traverse links in order to do their work indexing pages. Order processing applications will probably have more in common with search engines than with browsers.

REST describes this in terms of hypermedia - links - driving 'application state'.

So we next need to decide what 'application state' is, in our Web- and REST-driven architecture for machine-to-machine distributed systems; where the user driving hypermedia link traversals is replaced by business rules or logic driving hyperdata link traversals.

Each integrated application has its own 'application state', so, to follow REST, this application state should be driven by the surrounding hyperdata of peer applications, according to those business rules.

Application State is Linked Resources - Slide 4

In fact - and this is a consequence of the symmetry of integration - 'application state' is those very resources that the application contributes to the hyperdata Web!

A stock tracking application's state is pretty well described by a bunch of resources describing the stock levels. A fulfilment application's state could be inferred by inspecting the outstanding packing notes.

We're not limited to the asymmetric browser-server of the Web, where the browser's 'application state' is never visible except when it POSTs something back.

It's more like a search engine, where you can publically access an 'application state' that is entirely driven by the hypermedia crawled by the search bot. A search engine's application state is rendered into the results page resources you see when you do a search.

So the resources of each application in the order processing integration are driven by the surrounding, linked resources of the other applications.

You could rephrase REST's 'hypermedia as the engine of application state' when applying it to symmetric machine-to-machine integration in this neat way:

Hyperdata as the Engine of Hyperdata.

The Functional Observer Programming Model - Slide 5

So now, how do we program the hyperdata-driven-hyperdata of our integrated applications?

How do we animate the stock tracking hyperdata chunk over here in the face of today's packing notes in their hyperdata chunk over there?

That's what FOREST is all about!

The name 'FOREST' stands for 'Functional Observer REST'.

The words 'Functional Observer' describe FOREST's hyperdata-driven-hyperdata programming model. But it's much simpler than it sounds...

A FOREST resource in the hyperdata Web sets its next state as a Function of its current state plus the state of those other resources Observed by it via its links.

The best way to encode such state evolution is in rewrite rules or functions which match a resource and its linked resources on the left-hand side, then rewrite that resource's state on the right-hand side.

Not Like AtomPub, then

So, quite a different conclusion from what is now the 'conventional REST' of the four verbs - GET, POST, PUT and DELETE.

Quite different from asymmetric, one-way application protocols as modelled by AtomPub, in which clients aren't considered worthy to hold their own resources, but are allowed only an inscrutable 'application state'.

By focusing on GET and the freedom in integration to be symmetric, we've arrived at a general distributed programming model, FOREST, that allows us to express business rules that drive an application's hyperdata in the context of another application's hyperdata.

Watch this blog (and Twitter), where I'll be talking more about the benefits of FOREST, its implementation, and, above all, offering examples of how it would work (once the code is ready enough!).

FOREST: a GET-only REST Integration Pattern

2009-10-09T17:14:00Z

Since the day in 2006 that our dialogue took place with an imaginary eBay Architect, he has been promoted to imaginary Enterprise Architect in an investment bank! Convinced by the merits of REST, he took his enthusiasm for it into his new job and embarked on architecting a trading system using REST or ROA as an alternative to SOA.

Now, he hit upon a snag: he had a REST "bank server" generating bids on an instrument and POSTing them into that instrument's REST "market server". But then he had two copies of his bid! One held by the bank server on one URI, and the other in a "bid collection" held by the market server's instrument - on another URI.

He asked himself: "Which URI is the real one? Which host 'owns' the bid? Is the market's copy just a cache? If so, why does it have a new URI? Why doesn't the market host know the URI of the bank's original bid? Why can't servers become clients and just GET the data that their own data depends upon?" The server seemed to be dominating the conversation, not letting its 'client' server have a say in things.

Our worried Enterprise Architect noticed that such Service-Orientation permeated REST practice: there were "REST APIs" to Web sites, or "Web services" with a small 's'. Even AtomPub had a "service document"! Some patterns, like AtomPub, offered just simple read/write data services through the full HTTP method set. Some simply used such a read/write interface as a wrapper around more complex service functions.

He wondered: "Where's the Web in REST integration? The Web works great without PUT and DELETE: isn't using GET on its own RESTful enough?"

So, remembering something I said about "Symmetric REST", he contacted me again...

Enterprise Architect: I see we made it into Appendix A of the REST book by Richardson and Ruby!

Duncan Cragg: Indeed - even though I hadn't finished writing up our chat when it was published...

EA: So why did it take you so long to write it up?

DC: Well, I, er, got distracted by Web 2.0 and Mobile 2.0!

But I'm back now, intending to focus more on ROA's advantages over SOA.

EA: Great! Because I wanted to talk to you about that.

Where I now work, we are looking at REST or ROA as an alternative to SOA. However, all the available REST patterns still seem to see the world through Service-Oriented eyes.

I want to do REST like the Web does: to have different servers just publishing stuff that's all linked up. And "mashed up": to have that stuff, that data, "over here" depend on that data "over there": meaning that servers can be clients and vice-versa.

DC: Hyperdata that depends on someone else's hyperdata! Maybe rewrite rules over interlinked XHTML.

I called it "REST Observer" back then, but recent events on the rest-discuss mailing list have left me very wary of using the word 'REST' so openly in the name of something!

So I decided to hide it within a different word: 'FOREST'!

Here is a posting about FOREST that I recently made to the rest-discuss mailing list:

FOREST

FOREST is a GET-only REST Integration Pattern defined simply as:

A resource's state depends on the state of other resources that it links to.

This means that resource servers must also be clients in order to see those dependencies.

Common Web Pattern

FOREST is a REST Pattern derived from GET-only or polling Web use-cases, including mashups:

feed aggregators or filters
search index results pages
pages that depend on a search
Google's mobile versions of pages
sites that create summaries of other Web pages
sites that create feeds from Web pages
creating pages or feeds from REST 'APIs' (GET only)
Yahoo Pipes

Going Enterprisey

FOREST is a REST Pattern for building "Enterprise Mashups" in an ROA / WOA / SOA.

OK - those of you without Dion Hinchcliffe in your feed reader may be feeling a little queasy at this point, but I'd encourage you to read on ... Actually, I quite like the phrase "Enterprise Mashup" since it lightens the gravity of that 'Enterprise' word.

Enterprise Mashup Markup Language is the nearest thing to this that I know about, but FOREST is quite different: it is much simpler and is /only/ a REST Pattern.

Patterns can be implemented in frameworks...

A FOREST implementation would inevitably be over HTTP. It would initially be just XHTML or Atom. I imagine fetching XHTML resources within which are expected to be links to more such documents. Any XHTML could depend on any other, and they're all interlinked. If you depend on another resource, you must have found it directly or indirectly through links in your body. Alternative discovery: a resource could be told that it is being watched using an HTTP header in the GET request listing the URIs of the resources that depend on it - then it could watch and link back. Etag would be used for an automatically incremented version number.

Rough Consensus and Working Code

I would ideally see this work towards a formal description via "rough consensus and working code". I intend to knock up a prototype of FOREST in a Jetty servlet and post it to GitHub; if that code works, I may get rough consensus...

What a FOREST XHTML/HTTP formalisation would specify: Updated

use of HTTP headers (Etag, Cache-Control, Content-Location, Referer*)
API*: doc builder, XPath body set/get*, callbacks (observed, notified*)

Notes (*):

'Referer' is a possible header for the URIs of dependent resources
the API would be language-independent, but probably Java-like
the XPath 'get' would be extended to jump links from doc to doc
every doc jumped to gets observed
'notified' means being told when the GET returns with the observed state

What a FOREST Java servlet and client library would implement 'under' these specs:

a driver module loader: drivers animate resources through the API
a document cache - in memory and maybe saved to disk or database

Resource animation would either be by the application of business rules driving the API, or by adapting between external state and the API.

Amazing

EA: Wow! That's amazing! Can I help build it?

DC: Of course you can. Know any Java?

FOREST: Functional Observer REST

2009-10-06T10:28:00Z

Functional Observer REST = Hyperdata Interdependency requiring Symmetric REST
Hyperdata Interdependency = a resource's state is a Function of its linked resource context
Symmetric REST = servers become clients in order to Observe the dependencies of their resources

...