What Not How - Posts tagged 'json'

JSON-Mash

2010-03-18T16:58:00Z

Around the middle of February I completed a basic persistence and networking implementation for Fjord, then had to do other things for a month. Just recently I fixed Fjord to work with the latest version of the Node.js APIs.

Next project: I'm going to use Fjord in a Web Framework to be called "JSON-Mash".

I intend to show that JSON-Mash will be a great framework for rapidly building truly interoperable and truly scalable online and distributed functionality.

Here's how JSON-Mash will work.

Juggling Data Formats

In traditional Web Frameworks, the developer has to juggle up to five different data formats.

At the front end, you may be working in HTML or HTML chunks sent via Ajax. Then your domain logic may use object classes. You'll no doubt map those objects to relations in your database layer. That's three data representations (also known as Circles, Triangles and Rectangles).

But there may be even more: if you call out to integration points such as a search service or other API, you'll probably be using XML. And if you did application-level cacheing, you would probably consider casting your data into key-value form.

This is not unrealistic - at a current ThoughtWorks client, I was faced with exactly these five data formats - in an extremely simple application.

JSON Everywhere

JSON-Mash will do away with all that. It uses JSON objects everywhere.

JSON-Mash sends JSON data to a script in the browser that builds the page declaratively from its understanding of a number of stable, generic JSON types. It uses Ajax and Comet to pull JSON chunks in, to composite into the page.

JSON-Mash uses Fjord to process its JSON: transforms, map-reduce, etc. It uses Fjord to persist objects in a JSON NoSQL database, and Fjord to provide shared JSON cacheing functionality with cache updating via Fjord's FOREST integration style.

Finally, if there's any external system to be integrated, that too will be accessed via an adaptor that converts its API, etc., into JSON-Mash stabilised JSON. There will be Twitter and Flickr adaptors at first, I should imagine.

JSON-Mash and Fjord

Using Fjord means that JSON-Mash treats its JSON as a first-class hyperdata type: lots of little JSON chunks all with URLs and all linked up. I suggested that one day it may have the media type of "application/fjord+json". JSON-Mash Nodes communicate in the FOREST style.

Of course, the main benefit of using Fjord is that you get a great, declarative mashup language to describe your logic in.

Fjord is still very new, of course, so some goodness is yet to be rounded out. For example, it comes with a very basic, but highly improvable, NoSQL JSON datastore and already allows some powerful map-reduce and other query and transformation modes, with more to come.

Finally, Fjord gives a dynamic view of JSON data: caches get updates pushed into them and Comet pull will also come as part of the package.

CSI and SSI

Now, one of the great things about Node.js is that you can run the same code on the server as you run in the browser.

My previous attempts at doing things the JSON-Mash way have met with some resistance when I suggested building duplicate logic server-side to render pages statically, for accessibility and search engines.

But with Node, this problem goes away: you can use the exact same Javascript to convert JSON to HTML on the server side as you do on the browser.

I use the phrases 'Client-Side Include' and 'Server-Side Include' or CSI and SSI as easily-understood shorthand for this kind of page assembly from JSON elements.

Settling on a number of stable JSON types means that - once the types and their rendering are stable and coded - you never need to write any Javascript yourself in JSON-Mash. It's all done declaratively, and looks the same whether generated by an SSI Node or the CSI in the browser. For example, you can trigger calendars and maps in the browser view, with just small JSON declarations.

This approach differs from "progressive enhancement" as long as you have both full CSI and full SSI to render your site. But you could also have a 'CSI-SSI slider', where an initial page is generated by SSI, then CSI takes over to fill in the funky stuff: the dynamic widgets and Comet-driven elements.

Users Exploring a Dynamic Web of JSON

The way JSON-Mash works is that you get a view into a hyperlinked Web of JSON data, and can explore it from any point outwards. In the CSI version, each JSON chunk is loaded lazily as it is encountered, then reloaded when it changes. Even the SSI version of the site would allow this form of exploration, although in a less efficient and dynamic way, of course.

In JSON-Mash, the user is also a first-class object; the stable JSON types also include a type representing the user. This user is driving their browser, subscribing to events on visible JSON page components via Comet, and POSTing their own user state to a server-side cache copy.

Hence, if that user state contains a message, you can get simple chat functionality, without any code, just by linking to a user and viewing them.

Conclusion

JSON-Mash is a Web Framework that will use, re-use, assemble, mash, transform, store, share, push, pull and render JSON from end to end and top to bottom; from Ajax to NoSQL.

Most domain-logic programming will be done in simple, powerful declarative functions, with adaptors, drivers or renderers written in Javascript for Node.js or the browser. Cross-host exchange is done in the FOREST style.

SSI or CSI page assembly are very flexible ways of mashing up new functionality from multiple back-end sources of RESTfully-sourced, interlinked JSON data. And with cache updates telling SSI and CSI when the back-end data has changed, you get a very dynamic yet scalable system.

JSON-Mash will be a framework for rapidly building truly interoperable and truly scalable online and distributed functionality.

I'll write more on the benefits once I've got some code to prove it! Follow me on Twitter and I'll let you know when I've got something up on GitHub.

Fjord in Memory

2010-01-26T13:46:00Z

Right, I'm pleased to say that I've now implemented enough of the Fjord language on Node.js to be able to run the Instrument example that I introduced it with. As yet, this runs in memory only - i.e., no disk, no network.

Here's the code on GitHub with tests that show how it works. The language has changed a little so I'll show the example here again, copied over from the test code, in order to explain the differences.

Starting with the instrument rules:

inrl1=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "%owid": "/$this/",'+
                       '  "%refs": { "tags": [ "equity", "bid" ],'+
                       '             "%owid": "/$bid/",'+
                       '             "on": "/$this/" },'+
                       '  "buyers": "/array/has($bid)/" }');
// -------------------------------------------------------------------
inrl2=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "%owid": "/$this/",'+
                       '  "%refs": { "tags": [ "equity", "ask" ],'+
                       '             "%owid": "/$ask/",'+
                       '             "on": "/$this/" },'+
                       '  "sellers": "/array/has($ask)/" }');

These are the rules that add - or, rather, ensure the presence of - referring bids and asks in the instrument's buyers and sellers lists. The big difference here is that referrers are now accessed through the '%refs' tag. Both '%refs' and '%owid' are available on all Fjord objects: %owid is the Object Web ID of the object, and %refs has a list of the OWIDs of all currently referring objects.

But, of course, you can't see the list or the OWIDs under %refs! This rule shows how Fjord can match single items to lists of items, and how it transparently jumps any OWIDs it comes across, to continue matching. In this case there is a pattern for a bid or ask object that refers to the instrument - notice the "on": "/$this/" match.

Every object in %refs must have some such link to the referred-to object that triggered the referral in the first place. This referrer becomes an observer: its rules are run again if any object it refers to changes.

More instrument rules:

inrl3=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "buyers":  { "price": "/$bids;number/" },'+
                       '  "bid-ask-spread": { "high-bid": "/number/max($bids)/" } }');
// -------------------------------------------------------------------
inrl4=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                       '  "sellers": { "price": "/$asks;number/" },'+
                       '  "bid-ask-spread": { "low-ask":  "/number/min($asks)/" } }');

These are the rules that set the bid-ask-spread of the instrument from the prices of the bids or asks in each list. It shows how, when a single match instance binds to a list, it can give a match set in any bound variables - here, '$bids' and '$asks' - which can then be reduced - here, using 'max()' and 'min()'.

The rules are now separated into two, as there was a historic first ever bug in public Fjord code: the original rule wouldn't match unless at least one buyer and one seller were present, and we're adding them one by one from scratch this time!

Here is the actual instrument object:

inst=WebObject.create('{ "tags": [ "equity", "instrument" ],'+
                      '  "long-name": "Acme Co., Inc",'+
                      '  "buyers": [ ],'+
                      '  "sellers": [ ],'+
                      '  "bid-ask-spread": { "high-bid": "10.0", "low-ask":  "20.0" } }',
                      [ inrl1, inrl2, inrl3, inrl4 ] );

Notice the list of rules that the object will be animated by: inrl1, etc., which are the rule object OWIDs, not the actual rule objects. The bid-ask-spread is set to some seed values.

On to bid and ask rules:

bidrule=WebObject.create('{ "tags": [ "equity", "bid" ],'+
                         '  "on": { "tags": [ "equity", "instrument" ],'+
                         '          "bid-ask-spread": { "high-bid": "/$hibid;number/" } },'+
                         '  "price": "/null/fix(2, $hibid * 1.10 )/" }');
// -------------------------------------------------------------------
askrule=WebObject.create('{ "tags": [ "equity", "ask" ],'+
                         '  "on": { "tags": [ "equity", "instrument" ],'+
                         '          "bid-ask-spread": { "low-ask": "/$loask;number/" } },'+
                         '  "price": "/null/fix(2, $loask * 0.90 )/" }');

Here we have rules for setting the price of a bid or an ask as a ratio of the instrument's bid-ask-spread numbers. These are not meant to be realistic business rules of course - they're for illustration purposes. The only real difference here is fixing the result to 2 decimal places, since Javascript was adding ugly rounding errors.

Now we go ahead and create two bids and two asks, pointing at the instrument by simply including its OWID:

bid1=WebObject.create('{ "tags": [ "equity", "bid" ], "on": "'+inst+'", "price": "" }', [ bidrule ]);
bid2=WebObject.create('{ "tags": [ "equity", "bid" ], "on": "'+inst+'", "price": "" }', [ bidrule ]);
ask1=WebObject.create('{ "tags": [ "equity", "ask" ], "on": "'+inst+'", "price": "" }', [ askrule ]);
ask2=WebObject.create('{ "tags": [ "equity", "ask" ], "on": "'+inst+'", "price": "" }', [ askrule ]);

This fires off all the rules and notifications .. and rules and notifications .. giving the final result:

exp=new WebObject('{ "tags": [ "equity", "instrument" ],'+
                  '  "long-name": "Acme Co., Inc",'+
                  '  "buyers":  [ "'+bid1+'", "'+bid2+'" ],'+
                  '  "sellers": [ "'+ask1+'", "'+ask2+'" ],'+
                  '  "bid-ask-spread": { "high-bid": "12.1", "low-ask":  "16.2" } }')
// -------------------------------------------------------------------
test.objectsEqual("Instrument example works", Cache[inst], exp);

Here the buyers and sellers correctly list the referring bids and asks, and the bid-ask-spread has been set to the values you get after two ratios are applied to the original seed numbers.

Rewrite rules are run on object construction, they're run whenever an object has a new referrer in its 'refs' list, and also whenever another object that's being referred to changes. The latter isn't exercised in this example, as all events are propagated by referral alone. The tests of Fjord have examples of the full Observer Pattern in operation.

Next Up

Now, this is living code, so it will no doubt be different again, soon. Watch this space, and follow me on Twitter, to keep up to date with developments.

Next, I'll be working on simple disk persistence of Fjord JSON objects and rules, followed by distribution in the FOREST style...

Fjord in Node

2010-01-06T17:03:00Z

Well, I've put together the first few lines of Fjord, implemented on Node.js.

Here's the description on GitHub: Fjord is a language for expressing domain logic as match-rewrite functions over mashable JSON Web objects.

I'm developing Fjord very openly, in the hope someone out there will be interested in getting involved in helping guide its design and implementation. I suppose code speaks louder than blog posts.

It's going to be slow progress, with a wife, two kids, and Thoughtworks all together getting 99% of my life, but at least people will be able to keep up with what I'm trying to do... I only got the project started at all because I'm off sick right now!

Here's the GitHub link: http://github.com/DuncanCragg/Fjord.

Follow me on Twitter if you want all the latest news .. http://twitter.com/duncancragg

Oh - I was lying about FOREST being GET-only, that was just to engage the REST community, but since I got zero reaction from anyone I may as well admit that I intended FOREST to use POST all along - to push updated resources into subscriber caches as I have always done. Or to use long-GETs to pull them in. Both are tunnelling cache refreshes over HTTP. Neither are Official REST.

Cheers!

Introducing Fjord

2009-12-11T08:22:00Z

Following on from my recent article where I derived FOREST, this article offers the beginnings of a JSON unification and rewriting language that can be used in a FOREST architecture.

Why JSON, not XHTML, now? Well, I recently discovered that JSON is overtaking XHTML in interest, and I was further inspired by Kris Zyp's recent announcement of his JSON Schema Internet-Draft.

Fjord is a language for describing how the state of a JSON resource at any time depends on both its current state and on the state of other JSON resources that it links to via hyperlinks.

Fjord is a Norwegian word, probably pronounced 'fiyourd', and might stand for some combination of the words: 'Functional JSON Object/Observer Resource/Rule/Rewrite Dependencies/Declarations'. Or maybe it's just because they're truly awesome, an' I wanna go.

Fjord also gives me an opportunity to show some examples of the "end-user" view of a FOREST interaction; starting with a simple finance example.

Instrument, Bids, Asks

Let's take the simplest possible view of a financial market.

There's an Instrument, and there are Bids and Asks on that Instrument. The Instrument summarises all the Bids and Asks into the 'bid-ask spread', taking the highest bid price and the lowest ask price. Forget volumes, trades and any other detail.

The Instruments, Bids and Asks are on various servers across various organisations. That's the 'symmetric REST' part.

Here's a JSON Instrument resource at the URL shown. It holds three Bid or Ask objects:

http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json
{
    "tags": [ "equity", "instrument" ],
    "long-name": "Acme Co., Inc",
    "buyers": [ 
        "@http://a-bank.com/fjord/equity-bid-9ac0d1-88ce1.json"
    ],
    "sellers": [ 
        "@http://c-bank.com/fjord/equity-ask-510efb-cca62.json",
        "@http://d-bank.com/fjord/equity-ask-8560ae-33eff.json"
    ],
    "bid-ask-spread": {
        "high-bid": "10.00",
        "low-ask":  "14.00"
    }
}

The '@' character is the flag that this string is a link or a URL. The meaningful names in those links are really just for humans and the layers below us - Fjord itself won't let you see inside them.

So now that we've introduced a JSON extension making it a hypermedia or hyperdata type, perhaps we should announce this convention in a media type string. How about "application/fjord+json"? This media type will also serve to set expectations on everything else about Fjord JSON objects, once the notation is stable.

Notice that, instead of 'class' or 'type' or a top-level tag, we have a list of tags. In Fjord, class is in the eye of the beholder - but, of course, must be a stable grammar in the eye of the generator!

Right, so here is a new Bid object held at 'B' Bank, that the Instrument hasn't seen yet:

http://b-bank.com/fjord/equity-bid-0043ee-419ac.json
{
    "tags": [ "equity", "bid" ],
    "on": "@http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json",
    "price": "",
    "by": "@http://b-bank.com/fjord/hcard-b-bank-4990cd-232b0.json"
}

It has a pointer to the Instrument it is a bid on, a bid price that isn't yet set, and a link to the hCard object of the bank offering the bid and serving this Bid object.

So Bids and Asks can see Instruments and vice-versa, and each can set their own state accordingly, as part of a FOREST interaction.

The 'animation' of Bid, Ask and Instrument JSON object resources is driven by three simple Fjord rules...

Rule One: Bid Sets Price From Instrument

Here is a rule setting the price of a new Bid to be 10% more than the highest current Bid on the instrument, hopefully ensuring this Bid is now the high bid:

{
    "tags": [ "equity", "bid" ],
    "on": { "tags": [ "equity", "instrument" ],
            "bid-ask-spread": { "high-bid": "/$hibid;decimal/" }
          },
    "price": "/null/( $hibid * 1.10 )/"
}

This is a template that matches the Bid object and its dependencies and rewrites part of the Bid.

Notice how the Bid object's link to the Instrument object is jumped transparently: the Bid can 'see' right into the Instrument in order to do its matching. However, it cannot itself rewrite the Instrument - only Instrument rules can do that.

In Fjord, matching is written with the '/ ; /' notation: each part of this semicolon-separated list must match. In this case, the high bid has to bind to the identifier '$hibid' and has to be a decimal. The price must not be set yet.

The keys 'decimal' and 'null' are, of course, special reserved words for matching in patterns. The 'null' matches empty strings, lists and hashes.

The rewrite part follows at the end of a match and is also wrapped in slashes. Here, the price is being set to the high-bid plus 10%. The parentheses introduce an expression.

This rule only matches if the price is empty - once set, the rule will not refire.

For more complex rewrites, the following forms can also be used:

    "price": { "/null//": "( $hibid * 1.10 )" }
    "price": { "when": "/null/", "then": "( $hibid * 1.10 )" }

which allow full hashes and lists to be both matched and set.

Rule Two: Instrument Puts New Bid into its Buyers List

The Instrument adds the new Bid to its buyers list with this rule:

{   "#url": "/$bid/",
    "tags": [ "equity", "bid" ],
    "on": { "#url": "/this/",
            "tags": [ "equity", "instrument" ],
            "buyers": "/list/has($bid)/",
    }
}

This is a rule on the Instrument, that 'looks up to' and matches the enclosing Bid.

The '#url' is an optional, reserved-word entry that can only appear at the top of a JSON resource and holds that resource's URL. Here, it is bound to '$bid', picking up the URL of the Bid object that refers to and encloses this Instrument.

Further down, the "#url" reserved-word value of "this" indicates the start of the object that this rule applies to. If "this" is absent, then, of course, this is a rule on the top-level object, as in the first rule above.

The rewrite part matches the buyers list of the Instrument and applies the 'has()' operator which ensures the list now contains the wrapping Bid; it rewrites the list to include its argument if it doesn't already.

If this rule matches two wrapping Bids simultaneously, then '$bid' is set to a 'match set' - a simultaneous binding to the URLs of all of those Bids, and 'has()' must ensure that all of these Bids are now in the list at once.

If there is no change after a rule is applied, the resource's Etag isn't incremented, and dependent rules won't fire. Fjord may re-apply any rule at any time, usually on an incoming state change indicated by Etag, so matches have to be designed accordingly.

That's why we say 'has', not 'add', which would add the Bid again and again each time the rule is applied. We could say "/list;hasNo($bid)/add($bid)/", but that's uglier and more imperative than declarative.

Rule Three: Instrument Sets Bid-Ask Spread from Bids and Asks

This is what the instrument looks like, now, with the new link to our Bid:

{
    "tags": [ "equity", "instrument" ],
    "long-name": "Acme Co., Inc",
    "buyers": [ 
        "@http://a-bank.com/fjord/equity-bid-9ac0d1-88ce1.json",
        "@http://b-bank.com/fjord/equity-bid-0043ee-419ac.json"
    ],
    "sellers": [ 
        "@http://c-bank.com/fjord/equity-ask-510efb-cca62.json",
        "@http://d-bank.com/fjord/equity-ask-8560ae-33eff.json"
    ],
    "bid-ask-spread": {
        "high-bid": "10.00",
        "low-ask":  "14.00"
    }
}

And here is what the new B-Bank Bid looks like with its competitive price:

{
    "tags": [ "equity", "bid" ],
    "on": "@http://the-bank.com/fjord/equity-instrument-23ab33-2319f.json",
    "price": "11.00",
    "by": "@http://b-bank.com/fjord/hcard-b-bank-4990cd-232b0.json"
}

So now the Instrument has to adjust its bid-ask spread to accomodate the new Bid. This happens when the following rule matches, which it always does - but only fires when a change is detected:

{
    "tags": [ "equity", "instrument" ],
    "buyers":  { "price": "/$bids;decimal/" },
    "sellers": { "price": "/$asks;decimal/" },
    "bid-ask-spread": {
        "high-bid": "/decimal/max($bids)/",
        "low-ask":  "/decimal/min($asks)/"
    }
}

This rule matches as many times as there are Bids and Asks, meaning that the local identifiers $bids and $asks are each bound to match sets.

Notice how the '{ price .. }' patterns match elements of a list, in the same way an XPath can match either one or several elements in a sequence of XML elements.

The 'max()' and 'min()' operators take the highest and lowest values in a match set (or in a physical list, come to that). Once again, this rewrite means 'set this value and increment the Etag, but only if there's a change'.

Here, the high-bid becomes our new Bid's '11.00'.

Fjord

So that's a quick intro to some of my ideas for JSON matching and rewriting in a FOREST architecture.

Fjord is a declarative, JSON-encoded language for expressing how the state of a JSON resource is a function of both its own state and the state of peer JSON resources that it can see via links in a hyperdata Web.

Fjord objects are 'masters of their own destiny', not allowing other objects to tell them what to do except via declarative intention.

Further articles will give more proposals for the syntax and semantics of Fjord, including true unification, and put it into the set of languages that are ideal for programming parallel and distributed systems, as well as for integration.

Since Fjord is very young and I'm still implementing it (in Java), I would certainly welcome comments about it, or about the FOREST style that it supports. You can chat with me on Twitter, too!

The Universe Web

2008-07-18T19:49:00Z

Since the announcement by IBM and Linden Lab that OpenSim can talk to Second Life, I've been thinking again about RESTful Virtual Reality.

I'm not the first, of course. Others have been motivated by the same goal: To bring the Web's scalability, linkability and interoperability into Virtual World platforms.

Ultimately, how to use the same techniques as the Web to link Virtual Worlds together into a single, massive 'Virtual Universe'.

Here's how I would architect the Universe Web...

Entities

The Universe Web needs Entities. (I'm going to be capitalising Significant Things a lot here to help the Web comparison - hope you don't mind). There's no need to make the Entity list too complicated since we're not doing online games yet - just a world like Second Life.

Here's a list of different kinds of Entity:

Places: buildings, streets, hills, lakes
Things: trees, books, birds, clocks
People

State, Links and animation

Entities aren't exactly the Objects of Object-Oriented programming: they have lots of public State. In a Virtual World, you can directly experience their colour, position, orientation, physical relation to each other, etc., thanks to the local render engine.

The basic structure of the Universe Web is made up of Linked Entities. A tree is Linked to its ground which is Linked to a building which is Linked to its occupants which are Linked to their clothing. Kneebones may even be Linked to thighbones.

Entities are often animated. Actually, Places are generally static, and People generally animate themselves. So, when talking in general, we should look at the animation of Things, like trees, books, birds and clocks.

There are two ways an Entity's animation can be guided: by Laws and by Rules.

Laws

Springiness is a handy Law. For example, tree branches, pages in a book and birds' wings and legs can all have springiness. We can declare a mid-air location in a bird's State and a springiness in its legs, then if the bird lands on a branch, the springiness can trigger the bird and branch waving gently to a halt.

We could even allow a single bird to become part of a flock Entity, where the flock only knows how many birds it has and the coordinates of its centre. Each bird will then be animated by global flock Laws.

The book can have a State of open or closed. If bookness was elevated to a global Law, then a book deciding that its State was now simply closed could trigger the elaborate, springy closing of its pages with a nice 'shlump' sound at the end.

The clock's current time may be set by the clock Entity itself, then the Law of time take over in between the clock updating its own time directly.

A Law can be invoked over an Entity's animation by simply adding the Law's name and parameters to its State. Such Laws can be handled by the physics engine in the same way surface texture is handled by the render engine: States such as 'red' need global meaning, Law applications such as 'springy' need global meaning.

Rules

When a clock reaches its alarm time, it rings. When its wizard master approaches, a magic book may glow and open. Birds fly away from danger (if a fox is too close, fly away from it). In between danger, birds and foxes look for food (if a bird is close, run towards it).

These are Rules. Rules are driven by the Observation of one Entity by another. The fox Observes the bird, the book Observes the wizard. Rules apply to Entity types: fox Rules, bird Rules, book Rules, etc. Rules can guide the animation of an Entity by evolving its State.

State evolution depends on the current State plus the States of the Observed Entities: Rules make an Entity change from one State to the next by taking the Entity's own State, Observing the States of the Entities around that it cares about, then deciding what the next, new State should be. If an Entity changes State, those Entities Observing it may themselves change State according to their own Rules, and so-on.

Links are part of an Entity's State, and Links may therefore change on the application of Rules. In other words, the virtual world may restructure when a Rule is run.

An Entity may either Observe another Entity, or Link to it, or both. An Entity that Links to another needn't Observe it. Trees don't bother to Observe the ground they grow from. Conversely, an Entity that Observes another needn't Link to it. A fox would only Link to a bird if she was lucky enough to catch it. Meanwhile, it is true that there's probably a chain of Links to everything that's Observed.

Unlike Laws, Rules can't or shouldn't be second-guessed, so have to be run 'at source' or 'inside the Entity', then the new State distributed to both the Observing Entities and to all the render engines.

But render engines are actually Observing People Entities..

People are First Class

In Virtual Worlds, we have People avatars: People are first class Entities. People can Observe and can change State, but it's a real user doing the animation, not a set of Rules. Observation and State change by a Person is implemented through the render engine.

Although self-animated, People can still make use of Laws. For example Laws can be invoked to allow gestures to be easier to make and to look better, and to allow People to 'run forward' rather than putting their 'left foot forward, ..'. Avatar skills become global Laws which can be triggered by an avatar's State animated by a user.

The Universe Wide Web

The Universe Web I've just described is a lot like the World Wide Web:

An Entity is like a Web Resource
The State of an Entity is like an HTML Web Representation of a Web Resource
A Link in an Entity's State is like a Web URL inside an HTML Web Representation
Laws are like Javascript, Flash, link hover, blink tags and animated GIFs
Rule sets are like PHP scripts or Java servers
A State Observation is like a Web Representation Transfer

A crucial difference between the Resource Web and the Entity Web is that, here it's another Entity doing the Observation. One Entity transfers its State to another.

It's as if the GET or POST has another Entity 'at the client end'. GET is like an Entity subscribing or Observing; POST is like an Entity publishing or notifying its State to another (an HTML form has a content type, too).

People are just examples of that Entity 'at the client end'. Instead of a user hidden behind a client browser, we have first-class People Entities doing 'GET's and 'POST's: Observing the Virtual World and acting (i.e., changing State) within it. Other examples are foxes and clocks.

Notation and Protocol

There are two open standards we need to have in progress while implementing the Universe Web: the notation and the protocol (our HTML/URI and HTTP).

Firstly, we have to define the notation for Entities and their Rules. Both the State of Entities and the Rules that animate them will be written in the same notation: a Rule talks directly about State. To me, basing that on JSON (rather than XML) is a clear choice. Add inter-JSON Entity Links (essentially UUIDs), then work out a simple JSON Rule language. We will then need to define JSON 'schemas' for the kinds of Entity we would model. These can be written in the Rule language itself.

Secondly, to define the protocol for Entities to Observe or subscribe to one another. It will be a 'symmetric, asynchronous HTTP' - a Peer-to-Peer, Publish-Subscribe protocol. UDP and Multicast stand out for this kind of interaction. A packet will subscribe, a packet will return current State, a subsequent packet will multicast State updates. We can re-use all the HTTP cache headers, only now we can push updates into caches and into render programs. Packets will inevitably end up tunnelled through TCP and HTTP, mind...

This lighter notation, which also breaks the data up into smaller pieces, and this multicast pushing of State change into caches, along with P2P-style operation, will mean the Universe Web scales even better than its parent.

Also, programming declaratively using Rules is both highly productive and further cleanly scalable through parallel processing.

Of course, it will be pretty trivial to allow inter-Web cross-over such as referring to PNG texture images via URL and pulling Universe Web Entities into a Javascript application that subscribes to them via Comet.

Building the Universe Web

So that's my Universe Web. It's basically the same as the World Wide Web and thus as scalable, linkable and interoperable. It comes with added symmetry: Entities, including People as avatars, exchange State in each direction.

It's a two-way data Web, not a one-way document Web.

I've started work on the Universe Web: the notation, the protocol and the implementation (in C, for portability across platforms, including mobile). Contact me if you want to help!

Google Micro Conference

2007-10-05T11:22:00Z

Last night's Google London Open Source Jam (also here) was on the subject of the 'Web' (didn't they invent that? Oh no, that was Microsoft).

This event has been getting better and better each time I've attended. There were some very interesting lightning talks held together with a tight structure and plenty of chance to chat, drink cold Leffe and eat cold pizza. And nick [transatlantic translation: 'steal'] the Green & Black's chocolate.

An ideal Micro Conference...

I arrived late (it starts at 6pm) and spent some time catching up with ex-Thoughtworks colleagues, so I missed Dion "Ajaxian" Almaer's Google Gears slideset from FOWA. Go there now and check it out.

Thus the first talk I saw was a nifty piece of widgetry by Steven Goodwin called WARP. In WARP, interacting with a page of 'applets' changed the URL to encode those applets' current state. If you link to the current page, it will always show that state. Very long URLs, you can imagine. None of that fancy Ajax stuff. RESTful, dare I say. Nice API server-side for unpacking your applet params.

A trip to the lavatories [transatlantic translation: 'restroom'/'bathroom'] revealed that they are, indeed, doing that Testing in the Toilet project in Google. It works, too! I learned something. Other intelligence on Google's Inner Workings include confirmation of the beanbags and of the high quality, free grub to which I have already alluded.

A nice bloke from Yahoo! (Tom Hughes-Croucher: another spy?) came along to sell his idea that, in the collaborative world of open-minded hackers, we who run websites could help each other with our 404s. If I get a 404, I use the referrer link to tell you, via some RESTful POST, that your link to me is bust (assuming I don't intend to fix it myself).

I think the world is a little more selfish, so you need to decide who hurts more - the site who sends their visitors to a dead-end, or the site delivering that dead-end to a new visitor. I suspect the latter, by a small margin, as it's not exactly a nice welcome. So it's up to them to let the new visitor down more gently, and to notify the publisher of the broken link with little or no cost to them. For example, a really sociable 404-ing site could just redirect the hapless visitor back to the referring page, adding '?broken=links' to the URL - hopefully to be picked up by log scanning scripts at the referring site.

Next up, yours truly taking yet another chance to promote his excellent Micro Web thingy. Couple of people asked about it afterwards - including that nice chap from Yahoo! Also, a smart - and nice - chap called Toby (this one?) got me into a deep discussion on imperative vs. event-driven vs. state-driven programming. He was apparently an old-timer like me, as he was able to engage in dewy-eyed Functional Programming recollections. I managed to give out about four full colour printouts about the Micro Web, and to collect some good calling cards.

However, Joe Walnes, even a pint down in the pub afterwards, still refused to sign up for Micro Web duties. This in spite of over three years of intensive lobbying, including eight months of me working Trojan-horse-like in his kitchen, on The 2005 Implementation.

Another ex-Thoughtworks colleague, Simon Stewart took yet another chance to promote his promising Webdriver thingy. And a very interesting project it is becoming. Still needs more work - on IE support, etc - but I'll probably be using it in my new job at the Financial Times.

Another ex-Thoughtworks colleague, Chris Matts took a chance to promote his and Andy Pols' interesting new Dream Machine thingy. Perhaps a bit like Cambrian House - you put your dreams and ideas into it and people expand on them. Chris is a natural on-stage - and even used the age-old trick of promising lots of money for no effort, to get our attention at the start.

All I could come up with for the Micro Web was 'Cheaper, Wider, Faster'...

Updated: added reference to Dion Almaer, details about WARP, swapped in the picture of TWers that I was waiting for and fixed a minor blunder thanks to that ever-sharp ThoughtWorker, Dan Bodart..

How Ruby can enable the Web 2.0 Platform

2007-06-26T15:17:00Z

Web 2.0's definition includes seeing the Web as an application platform. Which means it is in competition with Java and .Net, and with SOA, for both local and widely distributed applications.

If the Web is going to be a platform, the skills you need to learn to program it are the core Web 2.0 technologies such as Ajax, JSON, Atom, Microformats and OpenID.

And Ruby. This language, that's capturing the hearts of many Web 2.0 programmers, is ideal for easing the transition from the Java and .Net platforms to the Web platform, as I will show.

Even if you're part of a big company that is generally immune to the latest trends, the marriage of Ruby and the Web-as-platform may be something to prepare for. It could even displace your SOA agenda...

Few would disagree that the Ruby language is riding the wave generated by Ruby-on-Rails. In turn, Rails is riding the Web 2.0 wave, coming as it does from underpinning the very Web 2.0 37signals product suite.

Rails and Ruby have tapped into the tech Zeitgeist of friendly, simple and powerful. The speed with which the Ruby and Rails communities have delivered the key components of Web 2.0 is matched by the speed at which Ruby and Rails books are leaving the shelves.

What is the ideal platform of Web 2.0? Will it be Rails and Ruby? Will Ruby ride the Web 2.0 wave into the mainstream in the same way Java rode the Web 1.0 wave?

Well, here's the problem with that question: Web 2.0 is supposed to be primarily about the Web itself as the platform, as explained first by Tim O'Reilly and then by a thousand Web 2.0 vendors and industry watchers after him.

Web-as-platform is not just vendor hype or pundit hand-waving. Let's think about what O'Reilly meant by that.

Web as Platform

Web 2.0 is about making the Web more interactive, and thus able to support applications where Java and .Net would once have been considered the sole delivery platforms.

The fact that the technologies of the Web can be turned to this use is a shift with far-reaching implications.

Broadly, the shift we are seeing is from the one-way, static document delivery of Web 1.0 towards the two-way, dynamic data exchange of Web 2.0.

This fundamental repurposing is delivering more complex, interactive applications that work inside our browsers and which fully leverage the benefits of online operation.

Web 2.0 is bringing the user and their stuff into the very Web that they hitherto only passively consumed. This network-enablement of the user in turn enables their social networking and their shared creativity and self-expression.

Web 2.0 has tapped into a deep human need - a fact reflected in the vast traffic volumes and correspondingly vast valuations of Web 2.0 startups that we're currently seeing.

But Web 2.0 is not just for the startups: Enterprise Web 2.0 is coming! The bigco.com site is going to be looking a little, well, static and lifeless when compared to the new sites that are springing up everywhere, and that most of BigCo's employees are using. Further, BigCo can gain huge benefits from Web 2.0 approaches empowering and connecting those employees on the Intranet. And that Intranet is an ideal platform for deploying company-wide, interactive applications.

This shift in the Web to two-way dynamic data is being powered by a set of technologies that a Web platform programmer is going to have to learn.

Web 2.0 Platform Technologies

Anything that claims to be an application platform must support data. Web 2.0 is above all the data Web. Web 2.0 is about semantics, not free text and font sizes. Hence, it inevitably starts with data-oriented formats such as XHTML, YAML and JSON. In Web 2.0 more than ever, we talk about data not documents and about separating data from its presentation. CSS is big in Web 2.0, for good reason (not just for gradient fills). Inside the page of a self-respecting Web 2.0 application, you'll often find Microformats - again, semantics in the page: publishing concise data of widely-understood standard formats. Some of those Microformats may be tags, and in Web 2.0 the simplest and most powerful semantics are those little pivot points in Webspace.

Again, if you're going to be a general purpose platform, you need to be able to fetch, update, notify and display that data. Web 2.0 integration usually happens via JSON data structures and REST interfaces (some of which, especially those based on AtomPub, are true REST). Following on from the data-like pages we serve to browsers, come the data-like feeds we publish to feed readers and to other applications. After feeds, the core technology that gives Web 2.0 its dynamism and interactivity is Ajax and DHTML, and increasingly Comet (server push to the browser). The core technology that gives Web 2.0 its users is increasingly OpenID.

All of the above are open technologies. You can do Web 2.0 without proprietary technologies, just like Web 1.0. Indeed, keeping to the principles that made the Web successful is also essential to the success of Web 2.0. The Web platform is the first application platform that has to consider scalability and interoperability, and will ignore them at its cost. I have written before about open data, use of standard data formats and using REST properly to avoid creating unscalable, walled-garden sites. You don't need Flash or SilverLight, you don't need vast amounts of custom Javascript, you don't need function calls tying you to your servers.

Programming the Web 2.0 Platform

So, we've got the dynamic data that you'd expect of a would-be platform. But how to drive changes in those data? How do we program the Web platform to animate all this data?

All Rails programmers will know the above technology list; it comes with the territory. The Web 2.0 Platform can be very succesfully powered by Rails and Ruby. Ruby and Rails make Web 2.0 applications simple and quick to program, addressing many of the needs of simple Web 2.0 applications out of the box. There's little doubt that Ruby and Rails will have a secure future riding the Web 2.0 wave.

However, for many Web 2.0 applications, programming may not even be necessary, at least not in the procedural or imperative style programmers expect.

Look back to the early 90's: 'Web 1.0' made a whole class of applications easy to write without programming: applications for navigating information. You just wrote in HTML, declaratively.

Now look back at the long path of evolution of Java, through J2EE, Spring, AOP, IoC, Domain Driven Design, POJOs. All trying to achieve the simple goal of 'remove all that MVC and persistence stuff and let us concentrate on business or domain objects'. But they never quite seemed to get it right.

But then Rails comes along, and has succeeded by simple virtue of concentrating on easy manipulation of the 'Intel Inside' of Web 2.0 - data.

It's reminiscent of the 'Naked Objects' approach to application building with minimal programming (just business or domain code in POJOs that expose state into the GUI and are transparently persisted). The Streamlined project takes Rails even further down this path. Rails' nearest competitor, Django, has an admin interface that works in a similar way, automatically generating edit pages based on the data model.

Web 2.0 is about data, about semantics. Web 2.0 is inherently declarative. So Web 2.0 applications can be written declaratively - Web 2.0 mashups can be just wired together and their data animated by business rules. A bit like programming spreadsheets.

Teqlo, Coghead, Pipes, DabbleDB, LongJump, Popfly, AppExchange and Wyaworks are all examples of the different ways to program the new Web 2.0 platform without imperative code.

That's what we mean by Web-as-platform - not only is the underlying programming language irrelevant, it will often not even be needed, certainly for simple data manipulation applications and for many simple mashups. Being RESTful gives you a massive head start in this, of course.

While Rails is already in the game with its innate understanding of Web 2.0 techniques and philosophies, Ruby itself has a huge amount to offer the would-be declarative programmer, who is making the transition to this new Web platform from their traditional Java or .Net platform. In particular, it is easy to write your domain logic in a declarative style in Ruby: they call them 'DSLs' these days, but the idea is the same in most examples I've seen.

Web 2.0 - The Web Redux

Now, if you've been following this blog, you'll know I have a few opinions on Declarative Web 2.0 and on patterns for programming REST. Essentially I argue that, if you want to play in the Web 2.0 platform game, you don't want to be writing screeds of Javascript functions that call more functions on your servers.

I recently presented some ideas along these lines at the WWW2007 conference, entitled 'The Micro Web: putting the Web back into Web 2.0', where I also showed a demo written in Python.

This approach combines my Distributed Observer Pattern with Comet push to enable highly dynamic Web 2.0 applications to be coded RESTfully and declaratively, with zero Javascript. The Distributed Observer Pattern offers a clean programming model for animating the Web 2.0 dynamic-data technology set I described above.

I believe the Observer Pattern is core to the way we'll be programming when the Web 2.0 Platform hits mainstream. It enables the kind of event- and rule-driven programming that matches the characteristics of the Web 2.0 dynamic data platform. As a further killer benefit, it also directly addresses the optimal utilisation of multicore processors.

I am currently porting my Python implementation of this approach to Ruby, in the Redux project on Rubyforge. Redux stands for 'Ruby Event-Driven Update Exchange'. It uses the highly scalable EventMachine epoll-based event loop to power its event-driven architecture. This will be essential when Redux is asked to scale up a Comet-based application.

Like Rails, Redux will be a Web (2.0) application framework, but unlike Rails, it puts the Observer Pattern and event- and rule-driven programming at its core.

Redux's headline is 'Web 2.0 in-a-box' or 'Naked Objects on the Web'.

Conclusion

If you're in BigCo, and are responsible for setting BigCo's technical strategy, then train your Java devs up on Web 2.0 core technologies such as Ajax, JSON, Atom, Microformats and OpenID.

And fire up their enthusiasm by tapping into Ruby (perhaps via JRuby) on your way to the Web 2.0 platform.

Learn patterns for mashing and integrating. Learn about REST and event- and rule-driven programming, including declarative DSLs.

When this Web platform hits BigCo, you will probably find that its REST or ROA style make your SOA integration strategy look rather complex and unweildy.

Check out the Distributed Observer Pattern, and download Redux when it's done (I'll let you know if you subscribe here!).

In 2007 and beyond, its the Web itself that's the platform, not Java or .Net. But if you want to get there via a language-based platform, Ruby could be the best way to transition to it.

Note: Everything I said about Ruby and Rails applies equally in technical terms to Python and Django, but regardless of the significant benefits of the latter, Ruby and Rails have the Web 2.0 market and mindshare. I'll probably switch this blog from Django to Redux sometime this year..