Around the middle of February I completed a basic persistence and networking implementation for Fjord, then had to do other things for a month. Just recently I fixed Fjord to work with the latest version of the Node.js APIs.
Next project: I'm going to use Fjord in a Web Framework to be called "JSON-Mash".
I intend to show that JSON-Mash will be a great framework for rapidly building truly interoperable and truly scalable online and distributed functionality.
Here's how JSON-Mash will work. ...
Around the middle of February I completed a basic persistence and networking implementation for Fjord, then had to do other things for a month. Just recently I fixed Fjord to work with the latest version of the Node.js APIs.
Next project: I'm going to use Fjord in a Web Framework to be called "JSON-Mash".
I intend to show that JSON-Mash will be a great framework for rapidly building truly interoperable and truly scalable online and distributed functionality.
Here's how JSON-Mash will work.
Juggling Data Formats
In traditional Web Frameworks, the developer has to juggle up to five different data formats.
At the front end, you may be working in HTML or HTML chunks sent via Ajax. Then your domain logic may use object classes. You'll no doubt map those objects to relations in your database layer. That's three data representations (also known as Circles, Triangles and Rectangles).
But there may be even more: if you call out to integration points such as a search service or other API, you'll probably be using XML. And if you did application-level cacheing, you would probably consider casting your data into key-value form.
This is not unrealistic - at a current ThoughtWorks client, I was faced with exactly these five data formats - in an extremely simple application.
JSON Everywhere
JSON-Mash will do away with all that. It uses JSON objects everywhere.
JSON-Mash sends JSON data to a script in the browser that builds the page declaratively from its understanding of a number of stable, generic JSON types. It uses Ajax and Comet to pull JSON chunks in, to composite into the page.
JSON-Mash uses Fjord to process its JSON: transforms, map-reduce, etc. It uses Fjord to persist objects in a JSON NoSQL database, and Fjord to provide shared JSON cacheing functionality with cache updating via Fjord's FOREST integration style.
Finally, if there's any external system to be integrated, that too will be accessed via an adaptor that converts its API, etc., into JSON-Mash stabilised JSON. There will be Twitter and Flickr adaptors at first, I should imagine.
JSON-Mash and Fjord
Using Fjord means that JSON-Mash treats its JSON as a first-class hyperdata type: lots
of little JSON chunks all with URLs and all linked up. I
suggested
that one day it may have the media type of "application/fjord+json". JSON-Mash Nodes
communicate in the
FOREST
style.
Of course, the main benefit of using Fjord is that you get a great, declarative mashup language to describe your logic in.
Fjord is still very new, of course, so some goodness is yet to be rounded out. For example, it comes with a very basic, but highly improvable, NoSQL JSON datastore and already allows some powerful map-reduce and other query and transformation modes, with more to come.
Finally, Fjord gives a dynamic view of JSON data: caches get updates pushed into them and Comet pull will also come as part of the package.
CSI and SSI
Now, one of the great things about Node.js is that you can run the same code on the server as you run in the browser.
My previous attempts at doing things the JSON-Mash way have met with some resistance when I suggested building duplicate logic server-side to render pages statically, for accessibility and search engines.
But with Node, this problem goes away: you can use the exact same Javascript to convert JSON to HTML on the server side as you do on the browser.
I use the phrases 'Client-Side Include' and 'Server-Side Include' or CSI and SSI as easily-understood shorthand for this kind of page assembly from JSON elements.
Settling on a number of stable JSON types means that - once the types and their rendering are stable and coded - you never need to write any Javascript yourself in JSON-Mash. It's all done declaratively, and looks the same whether generated by an SSI Node or the CSI in the browser. For example, you can trigger calendars and maps in the browser view, with just small JSON declarations.
This approach differs from "progressive enhancement" as long as you have both full CSI and full SSI to render your site. But you could also have a 'CSI-SSI slider', where an initial page is generated by SSI, then CSI takes over to fill in the funky stuff: the dynamic widgets and Comet-driven elements.
Users Exploring a Dynamic Web of JSON
The way JSON-Mash works is that you get a view into a hyperlinked Web of JSON data, and can explore it from any point outwards. In the CSI version, each JSON chunk is loaded lazily as it is encountered, then reloaded when it changes. Even the SSI version of the site would allow this form of exploration, although in a less efficient and dynamic way, of course.
In JSON-Mash, the user is also a first-class object; the stable JSON types also include a type representing the user. This user is driving their browser, subscribing to events on visible JSON page components via Comet, and POSTing their own user state to a server-side cache copy.
Hence, if that user state contains a message, you can get simple chat functionality, without any code, just by linking to a user and viewing them.
Conclusion
JSON-Mash is a Web Framework that will use, re-use, assemble, mash, transform, store, share, push, pull and render JSON from end to end and top to bottom; from Ajax to NoSQL.
Most domain-logic programming will be done in simple, powerful declarative functions, with adaptors, drivers or renderers written in Javascript for Node.js or the browser. Cross-host exchange is done in the FOREST style.
SSI or CSI page assembly are very flexible ways of mashing up new functionality from multiple back-end sources of RESTfully-sourced, interlinked JSON data. And with cache updates telling SSI and CSI when the back-end data has changed, you get a very dynamic yet scalable system.
JSON-Mash will be a framework for rapidly building truly interoperable and truly scalable online and distributed functionality.
I'll write more on the benefits once I've got some code to prove it! Follow me on Twitter and I'll let you know when I've got something up on GitHub.
Since the day in 2006 that our dialogue took place with an imaginary eBay Architect, he has been promoted to imaginary Enterprise Architect in an investment bank! Convinced by the merits of REST, he took his enthusiasm for it into his new job and embarked on architecting a trading system using REST or ROA as an alternative to SOA.
Now, he hit upon a snag: he had a REST "bank server" generating bids on an instrument and POSTing them into that instrument's REST "market server". But then he had two copies of his bid! One held by the bank server on one URI, and the other in a "bid collection" held by the market server's instrument - on another URI.
He asked himself: "Which URI is the real one? Which host 'owns' the bid? Is the market's copy just a cache? If so, why does it have a new URI? Why doesn't the market host know the URI of the bank's original bid? Why can't servers become clients and just GET the data that their own data depends upon?" The server seemed to be dominating the conversation, not letting its 'client' server have a say in things.
Our worried Enterprise Architect noticed that such Service-Orientation permeated REST practice: there were "REST APIs" to Web sites, or "Web services" with a small 's'. Even AtomPub had a "service document"! Some patterns, like AtomPub, offered just simple read/write data services through the full HTTP method set. Some simply used such a read/write interface as a wrapper around more complex service functions.
He wondered: "Where's the Web in REST integration? The Web works great without PUT and DELETE: isn't using GET on its own RESTful enough?"
So, remembering something I said about "Symmetric REST", he contacted me again... ...
Since the day in 2006 that our dialogue took place with an imaginary eBay Architect, he has been promoted to imaginary Enterprise Architect in an investment bank! Convinced by the merits of REST, he took his enthusiasm for it into his new job and embarked on architecting a trading system using REST or ROA as an alternative to SOA.
Now, he hit upon a snag: he had a REST "bank server" generating bids on an instrument and POSTing them into that instrument's REST "market server". But then he had two copies of his bid! One held by the bank server on one URI, and the other in a "bid collection" held by the market server's instrument - on another URI.
He asked himself: "Which URI is the real one? Which host 'owns' the bid? Is the market's copy just a cache? If so, why does it have a new URI? Why doesn't the market host know the URI of the bank's original bid? Why can't servers become clients and just GET the data that their own data depends upon?" The server seemed to be dominating the conversation, not letting its 'client' server have a say in things.
Our worried Enterprise Architect noticed that such Service-Orientation permeated REST practice: there were "REST APIs" to Web sites, or "Web services" with a small 's'. Even AtomPub had a "service document"! Some patterns, like AtomPub, offered just simple read/write data services through the full HTTP method set. Some simply used such a read/write interface as a wrapper around more complex service functions.
He wondered: "Where's the Web in REST integration? The Web works great without PUT and DELETE: isn't using GET on its own RESTful enough?"
So, remembering something I said about "Symmetric REST", he contacted me again...
Enterprise Architect: I see we made it into Appendix A of the REST book by Richardson and Ruby!
Duncan Cragg: Indeed - even though I hadn't finished writing up our chat when it was published...
EA: So why did it take you so long to write it up?
DC: Well, I, er, got distracted by Web 2.0 and Mobile 2.0!
But I'm back now, intending to focus more on ROA's advantages over SOA.
EA: Great! Because I wanted to talk to you about that.
Where I now work, we are looking at REST or ROA as an alternative to SOA. However, all the available REST patterns still seem to see the world through Service-Oriented eyes.
I want to do REST like the Web does: to have different servers just publishing stuff that's all linked up. And "mashed up": to have that stuff, that data, "over here" depend on that data "over there": meaning that servers can be clients and vice-versa.
DC: Hyperdata that depends on someone else's hyperdata! Maybe rewrite rules over interlinked XHTML.
I called it "REST Observer" back then, but recent events on the rest-discuss mailing list have left me very wary of using the word 'REST' so openly in the name of something!
So I decided to hide it within a different word: 'FOREST'!
Here is a posting about FOREST that I recently made to the rest-discuss mailing list:
FOREST
FOREST is a GET-only REST Integration Pattern defined simply as:
A resource's state depends on the state of other resources that it links to.
This means that resource servers must also be clients in order to see those dependencies.
Common Web Pattern
FOREST is a REST Pattern derived from GET-only or polling Web use-cases, including mashups:
Going Enterprisey
FOREST is a REST Pattern for building "Enterprise Mashups" in an ROA / WOA / SOA.
OK - those of you without Dion Hinchcliffe in your feed reader may be feeling a little queasy at this point, but I'd encourage you to read on ... Actually, I quite like the phrase "Enterprise Mashup" since it lightens the gravity of that 'Enterprise' word.
Enterprise Mashup Markup Language is the nearest thing to this that I know about, but FOREST is quite different: it is much simpler and is /only/ a REST Pattern.
Patterns can be implemented in frameworks...
A FOREST implementation would inevitably be over HTTP. It would initially be just XHTML or Atom. I imagine fetching XHTML resources within which are expected to be links to more such documents. Any XHTML could depend on any other, and they're all interlinked. If you depend on another resource, you must have found it directly or indirectly through links in your body. Alternative discovery: a resource could be told that it is being watched using an HTTP header in the GET request listing the URIs of the resources that depend on it - then it could watch and link back. Etag would be used for an automatically incremented version number.
Rough Consensus and Working Code
I would ideally see this work towards a formal description via "rough consensus and working code". I intend to knock up a prototype of FOREST in a Jetty servlet and post it to GitHub; if that code works, I may get rough consensus...
What a FOREST XHTML/HTTP formalisation would specify: Updated
Notes (*):
What a FOREST Java servlet and client library would implement 'under' these specs:
Resource animation would either be by the application of business rules driving the API, or by adapting between external state and the API.
Amazing
EA: Wow! That's amazing! Can I help build it?
DC: Of course you can. Know any Java?
...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 9: Web Objects Ask, They Never Tell ...
In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.
Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.
In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.
Part 9: Web Objects Ask, They Never Tell
eBay Architect: You've pushed a lot of responsibility up to the business logic and away from the distribution technologies - away from the HTTP level.
As I understand it, you only want to use HTTP to implement a distributed Observer Pattern, where clients can become servers?
Duncan Cragg: Indeed. REST can be symmetric when used in integration outside of the Web. Further, these server-clients can have resources that do their own GET-ing, with POST callbacks, in order to Observe other resources. I've given many examples of this style.
eA: So how do you program such a distributed system at the business logic level?
DC: Well you need to be able to easily express that the state of a resource depends on the latest state - the intentions and declarations - of other resources it is Observing.
My vision is that you would express this business logic in a simple, powerful and expressive declarative language.
eA: Hmmm.. Just try getting all declarative on someone who just wants his share prices on time! No-one in business understands such abstract concepts.
DC: Oh no? The favourite programming tool of business is the spreadsheet!
And HTML is a form of declarative programming: imagine if you had to build your DOM from the top using Javascript! Or if you had to set all your styles in Javascript instead of using declarative CSS!
eA: Well I've seen some pretty ghastly examples of those two crafts!
And many failed attempts to allow non-programmers or business analysts to program directly.
DC: True, but that doesn't change the fact that most non-programmers can only think declaratively - they know What they want, but they haven't got much of a clue How to get it.
Business people talk business rules and business data.
eA: We just can't program with them!
DC: Well, perhaps we just haven't found the best discipline of pragmatic formalisms and methodology that will allow those rules to easily become programs.
It's a well-supported claim, though, that declarative programming is simpler, clearer and more productive than imperative.
eA: OK - so how would you actually define these business rules?
DC: You could use a rules engine, or a DSL engine, or even XSLT - if that works for you. There are many ways to transform XML.
eA: How would you do it?
DC: I would like to have an XML rewriting and templating system: "if this XPath or XML template matches Observed XML resource A, this one matches resource B and this one matches myself, rewrite this and that bit of myself with these bits from A and B".
Web Objects Ask, They Never Tell
eA: So you want to go around re-creating huge XML documents all the time?
DC: Who said anything about huge? This is another Web thing that doesn't necessarily apply to us in REST integration: we don't need the equivalent of the giant, monolithic HTML page. We can work in much smaller chunks.
I'd even just call them 'objects' rather than resources. Or 'Web objects', if you like.
They could be little XHTML carriers of Microformats.
eA: I thought we'd given up on distributing fine-grained objects back in the CORBA days?
DC: Ah, but this isn't trying to transparently distribute zillions of method calls in an RPC model.
This is about optimising state transfer. Only send what you need, cache where you can, push when something changes. Separate your data by rate of change, timeliness, cacheability.
eA: So these Web objects don't have any methods? Because that would be RPC when distributed?
DC: Exactly. In REST integration, the Web objects Ask, they never Tell! These Web objects are reactive: Asking for public state, not Telling each other what to do.
eA: You mean the opposite of Tell Don't Ask?
DC: Yes. As an object, you don't Tell another object How to do something, you Ask for What you want by either simply Observing its public state or by it Observing yours, then letting it decide How to evolve by itself. You then watch it and react or interact.
It's the "imperative to declarative inversion": everything is turned upside-down or inside-out when you distribute things this way!
eA: Oh yes, your "inevitable inversion" thing.
DC: Another indication of this inversion from the imperative object-oriented world to the declarative ROA world is how the derided 'train wrecks' of object-orientation now become the essential XPaths of the object Web.
You could say we have no methods, only 'getters', and XPath 'train wrecks' are encouraged!
eA: Doesn't sound too safe to me - it breaks encapsulation, doesn't it?
DC: Well, it's actually safe to dig around, since the data is held in shape by a stable, open, public schema. You're expected to go traversing the tree.
And you get excellent encapsulation since Web objects are total masters of their own destiny: they privately control the evolution of that public state.
You very much retain the value of 'What not How'; in fact, in a much better-defined way since it's fundamentally baked in to the programming model.
Hyperdata as the Engine of Object State
eA: But when you write 'train wrecks' you often end up jumping from object to object. How do your object Web XPaths do that, assuming they want to?
DC: Hyperdata of course! Links to links around the Web. Objects can have their opaque UUIDs or GUID object handles encoded into their URIs. Then objects can be wired up with XHTML links.
eA: So now your XPath transparently jumps these links?!
DC: Actually, yes! That would then allow us to dynamically break up data into more manageable chunks without breaking the XPaths that traverse it.
Also, with this approach, you still get to drill down to data like in transparent URI paths, but you now use XPaths that are properly a part of the content layer, jumping transparently over those opaque inter-object URIs.
eA: Would you use these 'jumping XPaths' in the rewrite rules you said you wanted?
DC: Of course. Either linear XPaths, or XPath-like XML tree templates on the left-hand side of a rewrite rule.
You'd start a template match on yourself, then continue on to match other objects by jumping over links, then on and on from object to object.
eA: So I suppose any such jump to another object means you then need to start Observing it, right?
DC: Exactly. And if hyperlinks are the only way you can find other objects to Observe, it brings us to the following:
Your object's next public state depends only on its current public state and the states of those objects that are visible to it through hyperlinks.
eA: Sounds a bit like the "Hypertext As The Engine Of Application State" constraint of REST.
DC: Exactly! Except now that we're doing REST symmetrically - now that clients can be servers, too - client Application State can have its own URIs!
eA: So you could re-phrase this as the even more intimidating: "Hyperdata As The Engine Of Application Resource State"!
DC: Well - how about just "Hyperdata As The Engine Of Object State"?!
eA: So do I have to wait for this link-jumping XML rewrite engine of yours to express my business logic, in order to get "Hyperdata As The Engine Of Object State"?
DC: No, of course not! Use a nice, dynamic, XML-talking language, like Scala, and follow the same principle, perhaps using a DSL.
eA: Can't you have objects that aren't entirely dependent on others? Like those that represent external state?
DC: Of course. That's normal Web stuff. It's probably best to keep these 'pure': to have either fully-interdependent objects driven by Hyperdata, or fully externally-driven ones.
Class, Extension, Instance and Behaviour
eA: Right, so we've got these little Web-mapped XHTML Microformat objects all linked up and watching each other in an Observer Pattern. This object's state depends on that linked object's state according to rewrite rules or a DSL.
So taking this mapping to objects one last step, I presume object 'class' maps onto an XML schema, XHTML Microformat specification or other content type?
DC: Yes. A public grammar in some form. Domain or business classes only, of course, not low-level classes.
eA: So, does each standard Web object class or type have a standard set of rules guiding its evolution or behaviour?
DC: Yes. If you see something and recognise its type, you can attempt to interact with it according to its public specification, and it should, but needn't, react.
The public specification can define the expected behaviour in the RFC language of MUST and SHOULD, like AtomPub. Or it can define it in rewrite rules!
eA: So then, how do I add my own business rules to a content type or 'class', if its behaviour is standardised and meant to be stable and predictable?
DC: There are two ways you can make use of a generic standard, and both are forms of layering or abstraction:
You can 'mash up' standard component instances with declarative configuration - just using them as they are for your domain.
Alternatively, you can extend or subclass their standard structures and behaviour to become themselves more domain-specific. For example, rules can be overridden or their right-hand sides merged.
Again, think of how AtomPub can be used in any domain that looks like time-ordered lists of content; and how you can extend AtomPub without breaking unextended clients or servers.
The REST Observer Pattern
eA: I think you need a memorable name for your symmetric-REST, "Hyperdata As The Engine Of Object State" architectural style!
DC: How about the "REST Observer Pattern"?
eA: But a Pattern has to be a retro-fit to existing behaviour, and "REST Observer" is quite new.
DC: I mean REST .. [DC does double bunnie ears] "Observer Pattern"!
eA: Have you got any worked examples of the REST Observer Pattern?
DC: Will have soon...
(c) 2006-2009 Duncan Cragg
Coming soon: Worked examples of the REST Observer Pattern.
Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.
Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...