Juggling Data Formats

In traditional Web Frameworks, the developer has to juggle up to five different data formats.

At the front end, you may be working in HTML or HTML chunks sent via Ajax. Then your domain logic may use object classes. You'll no doubt map those objects to relations in your database layer. That's three data representations (also known as Circles, Triangles and Rectangles).

But there may be even more: if you call out to integration points such as a search service or other API, you'll probably be using XML. And if you did application-level cacheing, you would probably consider casting your data into key-value form.

This is not unrealistic - at a current ThoughtWorks client, I was faced with exactly these five data formats - in an extremely simple application.

JSON Everywhere

JSON-Mash will do away with all that. It uses JSON objects everywhere.

JSON-Mash sends JSON data to a script in the browser that builds the page declaratively from its understanding of a number of stable, generic JSON types. It uses Ajax and Comet to pull JSON chunks in, to composite into the page.

JSON-Mash uses Fjord to process its JSON: transforms, map-reduce, etc. It uses Fjord to persist objects in a JSON NoSQL database, and Fjord to provide shared JSON cacheing functionality with cache updating via Fjord's FOREST integration style.

Finally, if there's any external system to be integrated, that too will be accessed via an adaptor that converts its API, etc., into JSON-Mash stabilised JSON. There will be Twitter and Flickr adaptors at first, I should imagine.

JSON-Mash and Fjord

Using Fjord means that JSON-Mash treats its JSON as a first-class hyperdata type: lots of little JSON chunks all with URLs and all linked up. I suggested that one day it may have the media type of "application/fjord+json". JSON-Mash Nodes communicate in the FOREST style.

Of course, the main benefit of using Fjord is that you get a great, declarative mashup language to describe your logic in.

Fjord is still very new, of course, so some goodness is yet to be rounded out. For example, it comes with a very basic, but highly improvable, NoSQL JSON datastore and already allows some powerful map-reduce and other query and transformation modes, with more to come.

Finally, Fjord gives a dynamic view of JSON data: caches get updates pushed into them and Comet pull will also come as part of the package.

CSI and SSI

Now, one of the great things about Node.js is that you can run the same code on the server as you run in the browser.

My previous attempts at doing things the JSON-Mash way have met with some resistance when I suggested building duplicate logic server-side to render pages statically, for accessibility and search engines.

But with Node, this problem goes away: you can use the exact same Javascript to convert JSON to HTML on the server side as you do on the browser.

I use the phrases 'Client-Side Include' and 'Server-Side Include' or CSI and SSI as easily-understood shorthand for this kind of page assembly from JSON elements.

Settling on a number of stable JSON types means that - once the types and their rendering are stable and coded - you never need to write any Javascript yourself in JSON-Mash. It's all done declaratively, and looks the same whether generated by an SSI Node or the CSI in the browser. For example, you can trigger calendars and maps in the browser view, with just small JSON declarations.

This approach differs from "progressive enhancement" as long as you have both full CSI and full SSI to render your site. But you could also have a 'CSI-SSI slider', where an initial page is generated by SSI, then CSI takes over to fill in the funky stuff: the dynamic widgets and Comet-driven elements.

Users Exploring a Dynamic Web of JSON

The way JSON-Mash works is that you get a view into a hyperlinked Web of JSON data, and can explore it from any point outwards. In the CSI version, each JSON chunk is loaded lazily as it is encountered, then reloaded when it changes. Even the SSI version of the site would allow this form of exploration, although in a less efficient and dynamic way, of course.

In JSON-Mash, the user is also a first-class object; the stable JSON types also include a type representing the user. This user is driving their browser, subscribing to events on visible JSON page components via Comet, and POSTing their own user state to a server-side cache copy.

Hence, if that user state contains a message, you can get simple chat functionality, without any code, just by linking to a user and viewing them.

Conclusion

JSON-Mash is a Web Framework that will use, re-use, assemble, mash, transform, store, share, push, pull and render JSON from end to end and top to bottom; from Ajax to NoSQL.

Most domain-logic programming will be done in simple, powerful declarative functions, with adaptors, drivers or renderers written in Javascript for Node.js or the browser. Cross-host exchange is done in the FOREST style.

SSI or CSI page assembly are very flexible ways of mashing up new functionality from multiple back-end sources of RESTfully-sourced, interlinked JSON data. And with cache updates telling SSI and CSI when the back-end data has changed, you get a very dynamic yet scalable system.

JSON-Mash will be a framework for rapidly building truly interoperable and truly scalable online and distributed functionality.

I'll write more on the benefits once I've got some code to prove it! Follow me on Twitter and I'll let you know when I've got something up on GitHub.

eBay Architect: Now, I have a few more questions about the areas where REST is generally considered to be weaker than SOA. I'll start with a big list, then we can go through each one.

Duncan Cragg: OK, go ahead!

eA: Right - it's basically the list of techniques you need to architect complex distributed applications: Service Discovery, Service Description, Client/Server Session State, Business Process Coordination, Workflow, Orchestration, Choreography; Security, Reliable Messaging and Transactions.

DC: Phew! Well, some of these beg the question; by assuming you need them, you end up concluding that you need SOA!

Also, seeing WS-standards for these things, some people think that SOA has them covered and that their work is done. But you still have to design and architect your system on top of these standards, to meet the business need.

In REST or ROA, we're talking about a different way of seeing distributed systems that requires a break from Service- (and even Object-) Oriented thinking and an embrace of Resource-Oriented thinking.

eA: So what's the REST way of approaching my list?

DC: Do things the REST way and there are three possibilities: (a) that it's easy to implement these functions in REST; (b) that you don't need these things anyway, or (c) that it's already your job to do them as an application architect; it's part of your business process, so you do work that you needed to do anyway in SOA, but do it in a simpler and more powerful way.

eA: Already my job?

DC: It's your job while you're designing your resource interactions at the application level: the resource-type or business level. Or maybe while you're making use of some existing resource types and 'resource-animating' code that does your business logic for you. Having WS-* only clouds and complicates this task.

eA: You've just hand-waved away a billion-dollar standardisation and tooling industry!

DC: Standards and tools are essential - we need them. But simple standards and tools are best. HTTP, URI, XML and standard schemas are perfectly good enough standards to get started with, guided by the REST or ROA mind-set.

I'll grant that we do need some extra help to support RESTful resource interaction in our domains and applications. I would like a few more standards and tools, over and above HTTP and XML, to help me build these complex distributed applications. Some help configuring and programming the way an application's resources interact at a business level.

eA: What exact support would you like for REST integration?

DC: We would benefit from standards, tools and frameworks that help define resource URIs, mapping to GETable and POSTable URIs, easy cache control, easy support for various response codes, HTTP Auth-based and URI-based identity, inter-resource links, collections, content types, Microformat support, useful schemas, structures, sets of simple default behaviours, APP libraries, business rule engines, etc.

It is still early days for the adoption of REST - an adoption that's undoubtedly been slowed by the industry's obsession with SOA. We do have Restlet and the JSR 311 work, as well as some help from Microsoft in WCF and the ADO.NET Data Services Framework, but there is still much more that could be done.

eA: Indeed. There's lots of work to do to catch up with all the SOA standards, tools and frameworks!

DC: I really don't want ROA to compete with SOA in bureaucracy and vendor in-fighting! But REST needs much less than SOA. Some things are simply part of REST: you discover stuff by following links; you know what something can do for you because of its content type or schema - we went over this sort of thing before.

Service Discovery and Description

eA: So, following links and seeing content types cover REST 'Service Discovery' and 'Service Description'?

DC: Yes. When you start thinking in terms of resources not actions, it all becomes much clearer:

In real life you might walk up the street, ask someone where the nearest shoe shop is or consult a directory, then see a likely shop, decide it may have what you want, then go in to investigate. Inside, you may consult more directories or just wander around. You see the cues around you (shoes on offer, checkout desk), and act or react accordingly.

You engage at a chosen level of understanding: there is a generic concept of street layout, then generic shop and shop assistant understanding, then an understanding of how to view and interact in shoe shops in particular.

eA: And in 'virtual life'?

DC: The Web is exactly the same: you explore through links or by asking a search engine, see what you want, enter the online shop, follow the cues, etc. It's no accident that the cyberspace metaphor is often applied to the flat, 2D Web.

eA: And in REST integration?

DC: In REST integration, the metaphor can be extended for machine interaction: get a list of items by direct reference or by query, go through them finding something that matches, interact according to standard business types and follow links between business resources.

eA: So no service contracts for you, then, just exploring and hoping for the best. Sounds a bit sloppy!

DC: You can do anything with anything in computing, it's just a matter of what metaphor is the most powerful or expressive.

You can design your 'machine cyberspace' to include contracts if that's what you need, just like in real life, and just like we went over with the eBay/gBay example.

But in general - especially when building distributed systems - it's best to start with loose arrangements and established conventions; adding constraints, contracts and Central Control only where absolutely necessary. Ensure central control only over schemas.

Make your Enterprise 'mashable' and everyone will thank you. Expose the UUIDs and GUIDs of your data in URLs. Transform and enrich your data within standard content types. Link it all up with your own data and data from elsewhere. Allow two-way interaction. Then publish it for future generations to discover and re-use!

Client State and Sessions

eA: Right, now how do you model complex state transitions, state evolution and conversations within a changing context, in a supposedly stateless architecture?

DC: There's a common misunderstanding about REST that its statelessness should extend above the protocol (e.g., HTTP) into the application. On the contrary - once above HTTP and into a Resource-Oriented application, it's all about state - as long as that state has a URI!

All the stateless requirement is saying in practical terms is that each HTTP request and response exchange is a one-off as far as HTTP is concerned. It means that you don't need to tie successive exchanges together at the protocol level, which makes implementing HTTP easier. HTTP just wants a URI to be a URI, and the content there to be manageable via headers; and HTTP doesn't and shouldn't introspect either the URI or the content.

eA: So where do you keep the conversation state, then?

DC: Above HTTP in the world of URI-tagged state. You can have an ongoing 'conversation' between a client and a server resource - as long as those resources are linkable and fetchable.

You don't need sessions below the resource level; below the business level. If your application truly demands the concept of a sequence of interactions with a start point and an end point, then go ahead and implement it - at the application or domain level, not at the framework level.

Most applications can use ad hoc, asynchronous interactions, where the client identifies itself each time with Authorization headers.

eA: Can each client machine have assigned to it dedicated server resources as part of this conversation, sort of like an explicit session state?

DC: Yes. Cacheability and linkability are clearly affected, but if it truly makes sense in your domain model to have client-specific resources, then just do it - by definition their scalability and findability is limited to that client! The client can still cache its own view, as in a browser, and even intermediate proxies can, although it's of benefit to only one client.

eA: Presumably it's a bad idea to put client state in a Cookie header?

DC: Yup: it's not got a URI, so its state is hidden. Put it on a URI in a server.

Hidden state is a red flag.

You know you're on the right track as long as you are exposing your statefulness in URIs. It should always be obvious where things stand from inspecting public state alone, to know what is possible and what will happen next.

If you find yourself hiding state in sessions and cookies, or returning different data dedicated to that client from the same URI by setting no-cache, or if you tie up successive interactions through sessions, you are going down the wrong track.

eA: Are cookies always bad?

DC: A Cookie header should be used only to identify the client, not a session. You'd probably use the Authorization header in REST integration anyway. Just use Cookie headers along with an auth scheme if you need a more elaborate, perhaps multi-layered or proxied, authentication system.

eA: What about the Vary response header and its use with Authorization or Cookie request headers?

DC: Mechanically, from the cache perspective, adding Vary on client-identifying headers like these is the same as using per-client URIs; setting the Vary header to allow the URI to be cached on a per-client basis is a hidden way of effectively adding the identifying data from the Authorization or Cookie header to the URI itself.

Actual per-client resources are explicit; these Vary'd resources are implicit. You choose which to use at the application level based on the importance of the resources being identified in your domain model. Either way, the identity of the requestor can be used to determine read permissions.

The explicit way means you can pass links to your personal resources to others, modulo read permissions. A client may or may not be able to or want to pass the link to its dedicated resources to other players - it probably isn't a link they want anyway - but they can still pass around general jumping-off links, that make sense for other recipients.

eA: To be honest, I just don't like the idea of the server saving and distributing a ton of this client-specific resource data!

DC: There are two answers to this: firstly, now that we're using REST for business data integration, not generating entire pages of HTML, the chunks of data can have much finer grain. The actual amount of client-specific data can be much smaller! You can cache and share links to all the generic chunks, and only transfer small amounts of less cacheable and linkable per-client data.

Secondly, our clients in REST integration are also often servers, so can in fact expose their own state. We may be better off just putting those client-specific resources on the client itself, to get around this whole issue. The client isn't a browser any more!

We're back to the symmetric Distributed Observer Pattern.

Business Processes

eA: OK, now what about Business Process Coordination, Workflow, Orchestration and Choreography in that sloppy cyberspace of yours?

DC: We've already discussed the auction business process, both single-site and cross-site, which was quite a good example that covers a number of business activities.

Another good feature of REST is the way it naturally maps onto declarative business rules. When you switch from process-thinking to resource-thinking, you also switch from imperative thinking to declarative. This manifests as inter-resource dependency and transformation.

eA: Eh?

DC: A spreadsheet is an example of this style, where you declare the way cells depend on each other and then let the hidden magic take care of satisfying your constraints. We've discussed this style of programming before.

Again, tools will be essential to help with this. Managing and testing event-driven business rules over shared and distributed state is somewhat new territory, even to most REST integrators!

eA: Business Processes and Business Rules seem like different things to me.

DC: Well, as we saw, the business process of an auction can emerge from the local application of business rules. The thing is to let go of the myth that you benefit from specifying and controlling some over-arching vision of your business processes - that it makes sense to centralise control of what actually works best when delegated and decentralised.

Just write the local What, not the global How, and let the process emerge!

eA: Might be a hard sell to control freaks.

DC: Yeah, true enough. But it more closely maps onto the reality of the way businesses operate and interoperate. It's more about the actual peer-to-peer business interactions and visible state evolution, and less about central controllers that know the intimate, inscrutable details of the Web Services involved.

Instead of coordinating the import and export of data from one hand-coded interface to another, you can just link it all up and expect everyone to dereference and recognise your data. Instead of coordinating sessions with implicit state, you can just react to standard, public data types.

(c) 2006-2009 Duncan Cragg

In Part 8: WS-Are-You-Sure (Security, Reliable Messaging and Transactions).

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...