http://duncan-cragg.org/blog/ What Not How - Posts tagged 'microformats' Duncan Cragg on Declarative Architectures Duncan Cragg /favicon.gif /favicon.ico All content including photos and images by Duncan Cragg. Copyright (c) Duncan Cragg, your rights preserved: see /CXL.html A Django Production. 2012-01-20T18:07:00Z http://duncan-cragg.org/blog/post/basics-object-network/ The Basics | The Object Network 2012-01-20T18:07:00Z 2012-01-23T19:45:00Z

Right, let's get started with some basic conventions in the Object Network!

This part in the Object Network series will cover URLs, HTTP headers and some common JSON patterns.

Updated 23/1/12: I changed the URLs in the example to have one of each type.   ...

Right, let's get started with some basic conventions in the Object Network!

This part in the Object Network series will cover URLs, HTTP headers and some common JSON patterns.

Updated 23/1/12: I changed the URLs in the example to have one of each type.

Example

Here's an example interaction with an Object Network interface:

GET /events/uid-f330a3-4011e.json HTTP/1.1
Host: mobiconfs.com
If-None-Match: "32"
 :
HTTP/1.1 200 OK
Cache-Control: max-age=1800
ETag: "33"
Content-Type: application/json
                                                       . 
{ "is": "event",
  "title": "Over the Air 2011",
  "content": "Top Mobile Meetup at Bletchley Park",
  "start": "Fri, 30 Sep 2011 09:00:00 GMT",
  "end":   "Sat,  1 Oct 2011 23:00:00 GMT",
  "location": "http://bletchleypark.org/address.json",
  "attendees": [ "/staff/uid-55d3.json", "uid-06-7d5d" ]
}

So let's start from the top...

 

URLs

The full event URL is http://mobiconfs.com/events/uid-f330a3-4011e.json, which has a host part, a path prefix, a unique id for this object, and a file type suffix.

Of course, being a URL, none of this is significant from a resource identifying or locating point of view. Any sequence of characters that can be used to find a JSON object will do.

But it will help our routing and debugging to have some handy keywords in the path prefix and it will help our debugging to have a JSON suffix when playing with files.

If we use a JSON database or a relational database, we probably have a UUID or GUID to index and retrieve the object, which can be used in the URL. If we find the object by query, it's possible to pack the query terms into the URL instead.

We avoid actual query strings in the URL because we want to be sure that all caches will work! Also, we don't put auth tokens in the URL, we put them in the headers, as otherwise we fail to identify a real resource with orthogonal access control.

But the most important thing to remember is that the structure of our URLs isn't part of the domain or resource model - it is not significant to clients and we can change it at any time. There are no URL templates or query 'forms' in the Object Network. Our public resource model isn't in the URLs, it's in the JSON formats and the links between objects.

 

HTTP Headers

There's an integer ETag here, which could increment with every change in object state. Rather store the ETag with the object than calculate the ETag every time from the content, as that could slow you down. ETags are used for both saving network traffic and for change detection. Their use is expected behaviour in the Object Network.

As is the use of Cache-Control; use max-age, not Expires. Always set it, even if you set it to a month or a year, or zero. Get into good habits now and write the code you need, for this and for ETags. In other words, record when your objects change and know how often that happens.

The Content-Type - the Media Type - is application/json. The Object Network is only defined for JSON. This blog page forms the foundation of an Object Network Media Type based on JSON that we'll be defining when we reach rough consensus and running code. Individual object types, such as contact and event, are defined on top of the base Media Type, Microformat-style.

The return code is only used to guide or report on the success of data transfer; it's not used in the Object Network for anything at the domain level. So 200 if transfer succeeded, 303 if it's somewhere else, 304 if it's not changed, 404 if it's not found. We won't need much more than that for now.

 

JSON Types

Looking inside the JSON, we offer one or more tags asserting what kind of object this is intended to represent - the example above "is" a calendar "event". But clients are free to ignore that and interpret the object or its elements any way they like.

There is a selection of possible types, such as "event", "contact", "user", etc. Plus special tags: "list" and "query", for lists of objects and queries on those lists. For example:

"is": [ "user", "list" ] or "is": [ "contact", "query" ]

An object may be the union of multiple types, such as "is": [ "contact", "event" ], or a 'subclass' with an extra "is" tag and extra fields in the object. If there is only one tag after "is", as in this example, the list brackets are optional.

The semantics, if not the exact syntax, of certain fields in the JSON will usually be declared in the Object Network as being defined in some other standard - in this case iCalendar: "start" and "end" map onto "DTSTART" and "DTEND".

Most fields in the JSON will be defined within the type or format that an object "is", but many will stand alone. For example, "title" above is something that will mean the same in any format, as will "location", and other common fields, such as "created" and "author".

Almost by definition we don't have namespacing in the Object Network. The whole point is to encourage discussion and agreement, not isolation.

 

JSON Links

Above, "location" indicates a link to another object; just an absolute URL inside a string, which points to an Object Network object on another domain. The "attendees" list has two relative URLs, the first with a full path, giving http://mobiconfs.com/staff/uid-55d3.json.

You can also just put the UID there and let the client put the same relative prefix on and attach ".json" on the end, as in the second attendee above: http://mobiconfs.com/events/uid-06-7d5d.json (maybe that was the organiser, and their URL paths or routes are kept with the events).

It's at least an Object Network convention, and possibly a requirement, that links start with "http://", "/" or "uid-". It may then be useful to add "./" to that list for relative URLs not starting with "uid-". Fully-expanded absolute URLs, starting with "http://", have the advantage of allowing the containing JSON to be directly exported and used independently of its own URL. Absolute URLs can also be detected and rendered as active links in browsers, using a plugin such as JSONView, and in email clients, etc.

 

List and Link Transparency

A wrapping list isn't only optional for single "is" tags. This rule applies within all Object Network JSON. For example, one email address in a contact needn't be put into a list of one. A corollary of this is that, when pattern matching, a single item will match that item anywhere in a list.

Also, in the Object Network, a link to another object is logically the same as replacing that link with the object itself. In other words, a pattern match will transparently jump over those links and into the linked object's content. It is therefore also valid syntax to directly include, rather than link to, an object.

Links to list objects are also seen as the same as inline lists. For example:

{ .., "email": "uid-111", .. }

where uid-111 is:

{ "is": "list", "list": [ "a@b.com", "c@d.com" ] }

is the same as:

{ .., "email": [ "a@b.com", "c@d.com" ], .. }

This allows lists to be grown outside of their containing object, or reduced back in again. An object can switch at any time between these forms: single-to-list-to-single, link-to-inline-to-link.

There are no nulls or empty lists in the Object Network, or rather, null and empty list are equivalent to non-existent. So:

{ a: [ null ], b: [ null, c ] }

is the same as:

{ b: c }

 

Linking into the Object Network

Here's one way in: put something like <link rel="object-network" href="/events/uid-f330a3-4011e.json" /> into a Web page to point to the Object Network JSON version of an event, blog post, contact page, etc. Or just have an in-page link, and rely on the client responding correctly to the Media Type.

 

Next: The Object Network Contact Type.

Previous: Why we should link up our Web APIs.

Join the discussion in the Object Network Google Group!

http://duncan-cragg.org/blog/post/why-we-should-link-our-web-apis-object-network/ Why we should link up our Web APIs | The Object Network 2012-01-19T20:58:00Z 2012-01-22T16:05:00Z

OK, I'm trying to take a Big Idea and make it as Simple As Possible to grasp.

If we link our JSON data together and use the same formats, then our mobile, browser and server apps can become much simpler - through clean, stable, common, shared, re-used code - and much more powerful - through clean, stable, common, shared, linked, cached data.

This is the second part in the Object Network series, which will guide you away from building isolated Web APIs to engaging in a linked-up data landscape.   ...

OK, I'm trying to take a Big Idea and make it as Simple As Possible to grasp.

If we link our JSON data together and use the same formats, then our mobile, browser and server apps can become much simpler - through clean, stable, common, shared, re-used code - and much more powerful - through clean, stable, common, shared, linked, cached data.

This is the second part in the Object Network series, which will guide you away from building isolated Web APIs to engaging in a linked-up data landscape.

It Just Takes Two

You're about to create a Web API.

All you have to do is to use Object Network formats to render your data, and to link it all up as an interconnected graph.

When the next person does that, they can link to your data. And you can link back to theirs.

You save the effort of working out how to design your API: you just use the pre-defined formats and protocols.

You don't have to replicate others' data any more in your own API, you can just link to it directly.

You should start to discover new opportunities within your mutually linked-up data.

 

Sharing and Re-using Code

You'll find yourselves sharing and re-using the code you wrote for server-side publishing and client-side interaction, both within your own project and on GitHub.

That open source server-side code may have adaptors from existing APIs and back-end systems into linked Object Network data. It may run in EC2, or use ql.io. Maybe the JSON is stored in a NoSQL database - and the Object Network formats fill the "schema gap" there.

That re-usable client-side code may be driven by link-fetching, and have built-in on- and off-line cacheing, pre-cacheing and change detection. It may render objects as widgets, maybe nested, maybe even as an endless network of navigable, cross-site data.

You may have a page assembler 'client' of this data on the server side, building your actual website from this cross-site data. Those pages would then be animated by the same data, pulled in by Javascript.

 

Example Part 1: Lanyrd

So let's take an example.

What if Simon Willison wrote the API for Lanyrd using Object Network standards. This would cost him very little. As an early adopter, there would be a little chatter, but I would do most of the work, as I'd be delighted to have him involved..!

So, what data? Well Lanyrd basically has events and people attending them. That's like iCalendar and vCard. The Object Network JSON format for them is called 'event' and 'contact', respectively. Simple.

An event links to various contacts, via lists of speakers and known attendees, and the address of the venue. Contacts have lists of links to the events they're attending and connections in their social network represented as links to other contacts. Still simple.

Note that I'm literally talking traditional HTTP links here, right in the JSON data that you get back from the Object Network interface. Something like:

"attendees": [ "http://obnet.lanyrd.com/134-134.json", .. ]

 

Example Part 2: Meetup

Now, along comes Meetup, who are also interested in publishing into the Object Network.

They also have events and lists of attendees.

Meetup don't have to do quite so much work to get things going, now, as there's an example already up and running and some of the wrinkles have been ironed out. Perhaps Simon open sourced some stuff.

But some of those events and some of those attendees are actually exactly the same across the sites (follow those 'events' links just above).

Bingo! We can link up the sites! We can link events and contacts across the two sites.

Update/elaboration - 21/1/12: We can start by just cross-linking the respective event and contacts that are equivalent. But over time, Meetup and other event organisers can point their events directly at Lanyrd users/contacts attending who aren't signed up to Meetup for some reason. And Lanyrd can either point directly at Meetup events instead of via their own copy, or be the main host of an event pointing at both Lanyrd and Meetup attendees.

We've shared code and understanding of our data and its links, we've brought developers together, we've linked and shared data. We've empowered and freed our users.

The sum is far greater than the otherwise isolated parts!

 

Let's Get Started

If we link our JSON data together and use the same formats, then our mobile, browser and server apps can become much simpler - through clean, stable, common, shared, re-used code - and much more powerful - through clean, stable, common, shared, linked, cached data.

It seems to me to be so obviously valuable to do this, that I'm having a really hard time understanding why I'm the only one that gets it so far! Hopefully this post has made it clearer than my previous attempts.

So, while I'm publishing this series of technical blog posts that will show exactly how this will work, why not get in touch so we can get the open Object Network moving right now?

I'm talking to you, Simon Willison!

 

Next: Object Network Basics.

Previous: Introduction to the Object Network.

Join the discussion in the Object Network Google Group!

http://duncan-cragg.org/blog/post/introduction-object-network/ Introduction | The Object Network 2011-11-29T23:11:00Z 2012-01-22T16:03:00Z

It's interesting to compare the current growth of Web APIs with the early growth of the Web itself. To save you jumping those links: the Web dramatically beats the APIs.

I believe that the most likely cause of such relatively slow growth (in what should be a booming ecosystem) is that each API forms a closed silo and cannot benefit from any network effects. Every API is different and there are no links between them. There usually aren't any links within a silo. You can't even use a given API without first consulting the documentation.

The Object Network is designed to fix this, with linked-up JSON in common formats. This will allow easier mashing, sharing and cacheing of data and allow client code to be shared and reused.   ...

It's interesting to compare the current growth of Web APIs with the early growth of the Web itself. To save you jumping those links: the Web dramatically beats the APIs.

I believe that the most likely cause of such relatively slow growth (in what should be a booming ecosystem) is that each API forms a closed silo and cannot benefit from any network effects. Every API is different and there are no links between them. There usually aren't any links within a silo. You can't even use a given API without first consulting the documentation.

The Object Network is designed to fix this, with linked-up JSON in common formats. This will allow easier mashing, sharing and cacheing of data and allow client code to be shared and reused.

The Web Isn't APIs

Imagine if, back in the early 90s, you had to write a new client or download a new plugin for every website's "publishing API" - if programmers first had to read the documentation before they could get access to that particular website's content.

And imagine that there were no links between those websites - well, there'd be no point in having links if you didn't know what to expect when you got there - without first knowing the "API".

It's unlikely we'd have seen the phenomenal growth that the Web enjoyed. Google built a multi-billion dollar industry on those links between compatible sites.

 

HTML for APIs

Web APIs should be able to learn from the Web that they seem so keen to join. So how can we take these Web APIs and make them work like the Web does?

What's needed is something like an "HTML for APIs" - a format for data sharing and linking that all interested sites can publish in. This would allow easy linking of data between those sites, and clients that could work against any server they encountered by following those data links.

Now, API builders generally like to use JSON or XML for their data, but that's as high as they go up the shared semantic stack. They can do whatever they like inside that JSON or XML - but linking and re-using common types are hardly ever included in those things.

So that's where the Object Network comes in. It defines JSON formats for various common data types, and the links between them.

 

What Formats?

Well, we can start social, since that's where most of these Web APIs started. So we'll need contacts - based on vCard, of course - events based on iCalendar, news and messages in a JSON form of Atom, etc. Events will link to the contact objects of their venues.

Even though we have contacts, we'll still need objects representing users. A user is more dynamic than their linked contact card, including messaging status, GPS location and so-on.

We'll need comments linked to that news and reviews of those events. A review will link to the user reviewing and the restaurant contact or gig event being reviewed. We will need media metadata for pictures and videos, linked from any of the other types.

Finally, any of the above can be bundled into lists. In fact, a feed is just a list of news items. A gallery is a list of pictures. An address book a list of contacts.

 

Plundering and Interacting

We could also scan over some Microdata formats for ideas. We could even plunder the Semantic Web for useful types: we could derive our publication metadata formats from PRISM or Dublin Core, for example. Looking around some more, there's RFC5988 which can give us our labels for some handy JSON links: "copyright":"http://..", etc.

Once all that's in place, we may need to run searches or queries over those lists. And of course, allow events and notifications and update requests, etc. But that's looking quite far enough ahead for now...

 

How to Join the Object Network

This is the first of a series of articles discussing how we could go about building this Object Network - how we could launch a new network of shared data to replace isolated Web APIs.

 

Next: Why we should link up our Web APIs.

Join the discussion in the Object Network Google Group!

http://duncan-cragg.org/blog/post/benefits-native-web-mobile-apps-with-ots-netmash/ OTS: The Benefits of both Native and Web Mobile 2011-05-10T11:11:00Z 2011-05-10T11:11:00Z

The Web, in its purest form - declarative HTML and CSS documents, XML feeds - is mashable, linkable, sharable. It's easy to create documents that slot into the global Web and can be accessed on any device; accessed by just a simple link. Servers can easily scale through statelessness and cacheing.

Native Mobile Apps are fast and slick. They are intimate with the dynamic, interactive, tactile mobile user interface, intimate with the capabilities of the device and intimate with the domain of mobile: photos, locations, contacts, messages.

OTS is a simple, clean, powerful approach to delivering Mobile functionality and content that is designed to realise these benefits of both Native Apps and the Web.   ...

The Web, in its purest form - declarative HTML and CSS documents, XML feeds - is mashable, linkable, sharable. It's easy to create documents that slot into the global Web and can be accessed on any device; accessed by just a simple link. Servers can easily scale through statelessness and cacheing.

Native Mobile Apps are fast and slick. They are intimate with the dynamic, interactive, tactile mobile user interface, intimate with the capabilities of the device and intimate with the domain of mobile: photos, locations, contacts, messages.

OTS is a simple, clean, powerful approach to delivering Mobile functionality and content that is designed to realise these benefits of both Native Apps and the Web.

How OTS Works

OTS - Object Type Specifications - achieves this by defining open JSON object types for people, places, dates, photos, messages, feeds, lists, galleries, user interfaces, forms, maps, calendars, etc., all linked together into the Web.

These dynamic OTS objects are published or shared instead of native apps and instead of Web documents.

A native OTS mobile client for Android, iOS, etc. can then render those objects and allow the user to interact with them, using its understanding of what they mean to deliver a slick user experience.

The user will start with the link of an object to look at, then may see immediately-linked objects nested into the same view, perhaps summarised and/or with open/close sliders. Any linked object can be jumped to as the new top view. If an object being viewed updates, the view, or the corresponding part of the view, updates.

 

Your Journey

Indeed, you are an OTS object, that others can see, and that gives access to stuff you want to make public from your device, such as photos you've only just snapped, your current location if enabled, status and text messages, selected contacts, links of interest, etc. SMS and MMS will be mapped to OTS photo and message objects on your device.

Your user object and your shared stuff are automatically pushed to shared servers when you make them public or when they update. All of this could use contacts, photos, reviews, messages, etc., from your and others' Twitter, Facebook, Google, etc. accounts, adapted to OTS objects server-side.

Many applications can be built through simply wiring up some OTS objects. Indeed, you can do this yourself through the mobile client, including creating and publishing new objects and lists of existing objects. You can plot on a shared map any OTS object that has a location field, or lists of such objects all at once.

Such as lists of your contacts or friends - to watch your party on the map as it converges on the restaurant, perhaps. That restaurant may be publishing an OTS object describing its outdoor Jazz schedule this week, which can be plotted on a shared calendar along with the weather report objects, retrieved from another server. If you discover a list of review objects about the Jazz via an OTS search form, you can save the link and then instantly share it with your party.

If your friend takes a photo, you can instantly 'join them on the map' where they stand and chat about it - and the weather conditions they're experiencing. No need for them to upload to a photo-sharing site, with login and file finder dialogues, or for you to download and launch separate, unrelated chat, photo-sharing, map and weather apps.

 

Programming OTS in Fjord

None of the above would need any code to be downloaded and run on the device. It's interesting to notice just how much can be achieved simply by leveraging the client's understanding of the object semantics, and through generic declarative interactions with points in the dynamic Web of OTS objects. With object cacheing, once the objects needed have been seen once, the user can still use them when the network breaks, alongside the on-board objects.

Obviously, being able to run programs on the device would open up an even wider range of applications, including giving greater tolerance of network outages for object behaviours that would otherwise be run on remote servers. But so far, everything has been declarative; how do we avoid having to go back to writing code in imperative Java or Javascript?

Well, being based on the FOREST architecture, OTS would use Fjord as its declarative programming language. And Fjord is just another OTS JSON object type, that can be fetched along with the objects it animates, for on-device program execution.

Server-side, again, so much can be achieved simply by publishing OTS objects, and adapting existing services. For more interactive applications, behaviour could be programmed in Java, etc., against a FOREST declarative API, or again in Fjord.

 

NetMash and Fjord Development

NetMash, as well as being the Java reference implementation for FOREST - networking, persistence, declarative API and Fjord implementation - is also an Android app that forms the reference implementation for OTS.

Fjord is very much in flux and hasn't been updated to the latest way FOREST works, but a version of it I built on Node.js has worked as expected. Based on that work, NetMash should have a Fjord implementation this year, which, being in Java, will thereby be available on both the Android client and the server side.

NetMash and OTS supercede all the other variants of these ideas I've been dabbling with before, including the Node.js implementations.

So, as always, it's all coming along slowly, and I can always use help with all this. Ping me via email (see left bar) if you want to be involved in the OTS Mobile Revolution!

 

Conclusion

Applications and content built with OTS don't need to be downloaded or updated via app-stores, and have no app boundaries - all applications can build on one another. They share one user, one map, one calendar, one combined view of the on-device and on-line world of interlinked, dynamic mobile objects from multiple sources. Your, and your friends', location, message, photo, list, contact and date objects are pushed and pulled automatically when made public or when they change.

As Martin Fowler points out, there is a gulf between native and Web applications on mobile that seems like an "uncanny valley" when you attempt to work within it by patching up the differences with hybrid technologies that pull too hard away from the respective comfort zones of pure native or pure Web.

Even so, there is a great deal of activity here, with multiple, evolving technology specifications to choose from. Into this discomfort zone I place technologies such as the HTML5 and related Javascript APIs, WAC, PhoneGap and Javascript toolkits that emulate native look and feel.

OTS is a clean alternative that effectively brings native and Web closer together, thus squeezing out this "uncanny valley" discomfort zone. Being both a native mobile app and a first-class player in the Web, OTS brings out the best of both, then goes beyond either.

http://duncan-cragg.org/blog/post/deriving-forest/ Deriving FOREST 2009-11-25T21:29:00Z 2009-11-25T21:29:00Z

Say we want to integrate multiple applications which handle order processing. OK, that's got to be one of the dullest starts to a blog post. Never mind, bear with me...

So, we have applications on separate servers for handling and driving data such as orders, product descriptions and catalogues, stock lists, price lists, tracking, packing notes and delivery notes, invoices, payments, etc.

We may choose an SOA approach, of course. But let's say our sponsors have heard of this cheaper alternative: REST! Which to them means 'using Web technology to save money'.

Now .. suppose we push the time slider right back to before Mark Baker and the SOA -vs- REST Wars - or the 'SOAP -vs- REST Wars' as people naively called it. To when REST was simply (!) a description of the Web's architectural style...

What if we revisit the applicability of the Web, and its abstraction into REST, to the architecture of machine-to-machine distributed systems - to something like our order processing integration?

I think we'd quickly arrive at something that looks more like FOREST than, say, AtomPub...   ...

Say we want to integrate multiple applications which handle order processing. OK, that's got to be one of the dullest starts to a blog post. Never mind, bear with me...

So, we have applications on separate servers for handling and driving data such as orders, product descriptions and catalogues, stock lists, price lists, tracking, packing notes and delivery notes, invoices, payments, etc.

We may choose an SOA approach, of course. But let's say our sponsors have heard of this cheaper alternative: REST! Which to them means 'using Web technology to save money'.

Now .. suppose we push the time slider right back to before Mark Baker and the SOA -vs- REST Wars - or the 'SOAP -vs- REST Wars' as people naively called it. To when REST was simply (!) a description of the Web's architectural style...

What if we revisit the applicability of the Web, and its abstraction into REST, to the architecture of machine-to-machine distributed systems - to something like our order processing integration?

I think we'd quickly arrive at something that looks more like FOREST than, say, AtomPub...

Some pretty obvious things to notice about the Web and, indeed, REST:

  • The Web is essentially data on URLs of standard content types containing more URLs;
  • The Web is the Web because of the massive proliferation of links in that data;
  • REST mostly concerns itself with the consequences of GET, including cacheing;
  • The Web uses, I don't know, let's say 98% GET, 2% POST, around 0% other methods.

In other words, the Web, and its good qualities, are mostly based on:

GET URL -> HTML -> a.href=URL -> GET URL ..

When applying this Web/REST architectural style to our integration scenario, there are things that we can say right now with certainty will be different, but will have corresponding elements:

  • It's about data not documents, so HTML is probably going to be replaced by XML, although perhaps XHTML+Microformats or Atom would make a good compromise;
  • We have a choice of link specs: xhtml:a.href, atom:link.rel, xml:xlink; I don't think we'll be using XLink since no-one else seems to;
  • We'll probably use machine-generated URLs perhaps containing UUIDs, GUIDs or whatever.

In other words, we're not going to be spinning a hypermedia Web - it's more a 'hyperdata' Web.

So, in order to emulate the document Web in our hyperdata integration Web, we'll mostly be doing something like:

GET ID-URL -> XHTML -> a.href=ID-URL -> GET ID-URL ..

 

Oh! I've got some slides of all this on Google Docs: we're up to Slide 2! Maybe right-click, open in a new window...

 

Symmetry - Slide 3

But by far the biggest difference between the Web and an integration scenario is that the asymmetry on the network goes away; even for a cross-enterprise integration.

Where the Web's browser clients and site servers have always been asymmetric - clients being hidden away and only able to establish outbound connections - machine-to-machine integration is fundamentally symmetric - all servers can be made visible to each other.

Now, in order to keep the well-studied benefits described in REST, including separation of concerns, we should aim to maintain the client-server, layered structure in the use of the protocol.

But clients can be servers and vice-versa!

So, in machine-to-machine integration scenarios, we have:

  • Two-way GETs on machine-minted URLs pointing at XHTML+Microformats or Atom content containing more links.

In other words:

  • A hyperdata Web both created and consumed by the applications being integrated.

All of the dynamic data or hyperdata items in our order processing scenario will be distributed across the many applications being integrated. Each application serves its part of the hyperdata Web to the others.

And, of course, the hyperdata joins all these applications up: a payment resource in the accounting application will point to an order resource in the order processing application, etc.

 

Interactions and Application State

Now, each application has its own set of business rules and constraints over the hyperdata parts that it governs.

So how exactly should the applications publishing those bits of the hyperdata Web interact? How do orders interact with packing notes and stock levels, with payments and accounts?

In the Web, you go to a site and jump some links. Each page brings in CSS, Javascript, images, maybe iframes: an eager assembly of pages from links to many resources, in contrast to the lazy, on-demand fetching of links from a user jumping them.

The browser at any time has a state that depends on the page and images, etc, currently being viewed, plus the history of previous pages, bookmarks, etc.

Search engines in the Web, without a user driving things, are eager to traverse links in order to do their work indexing pages. Order processing applications will probably have more in common with search engines than with browsers.

REST describes this in terms of hypermedia - links - driving 'application state'.

So we next need to decide what 'application state' is, in our Web- and REST-driven architecture for machine-to-machine distributed systems; where the user driving hypermedia link traversals is replaced by business rules or logic driving hyperdata link traversals.

Each integrated application has its own 'application state', so, to follow REST, this application state should be driven by the surrounding hyperdata of peer applications, according to those business rules.

 

Application State is Linked Resources - Slide 4

In fact - and this is a consequence of the symmetry of integration - 'application state' is those very resources that the application contributes to the hyperdata Web!

A stock tracking application's state is pretty well described by a bunch of resources describing the stock levels. A fulfilment application's state could be inferred by inspecting the outstanding packing notes.

We're not limited to the asymmetric browser-server of the Web, where the browser's 'application state' is never visible except when it POSTs something back.

It's more like a search engine, where you can publically access an 'application state' that is entirely driven by the hypermedia crawled by the search bot. A search engine's application state is rendered into the results page resources you see when you do a search.

So the resources of each application in the order processing integration are driven by the surrounding, linked resources of the other applications.

You could rephrase REST's 'hypermedia as the engine of application state' when applying it to symmetric machine-to-machine integration in this neat way:

Hyperdata as the Engine of Hyperdata.

 

The Functional Observer Programming Model - Slide 5

So now, how do we program the hyperdata-driven-hyperdata of our integrated applications?

How do we animate the stock tracking hyperdata chunk over here in the face of today's packing notes in their hyperdata chunk over there?

That's what FOREST is all about!

The name 'FOREST' stands for 'Functional Observer REST'.

The words 'Functional Observer' describe FOREST's hyperdata-driven-hyperdata programming model. But it's much simpler than it sounds...

A FOREST resource in the hyperdata Web sets its next state as a Function of its current state plus the state of those other resources Observed by it via its links.

The best way to encode such state evolution is in rewrite rules or functions which match a resource and its linked resources on the left-hand side, then rewrite that resource's state on the right-hand side.

 

Not Like AtomPub, then

So, quite a different conclusion from what is now the 'conventional REST' of the four verbs - GET, POST, PUT and DELETE.

Quite different from asymmetric, one-way application protocols as modelled by AtomPub, in which clients aren't considered worthy to hold their own resources, but are allowed only an inscrutable 'application state'.

By focusing on GET and the freedom in integration to be symmetric, we've arrived at a general distributed programming model, FOREST, that allows us to express business rules that drive an application's hyperdata in the context of another application's hyperdata.

Watch this blog (and Twitter), where I'll be talking more about the benefits of FOREST, its implementation, and, above all, offering examples of how it would work (once the code is ready enough!).

http://duncan-cragg.org/blog/post/web-objects-ask-they-never-tell-rest-dialogues/ Web Objects Ask, They Never Tell | The REST Dialogues 2009-08-13T11:43:00Z 2009-08-13T11:43:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 9: Web Objects Ask, They Never Tell   ...

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP.

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 9: Web Objects Ask, They Never Tell

eBay Architect: You've pushed a lot of responsibility up to the business logic and away from the distribution technologies - away from the HTTP level.

As I understand it, you only want to use HTTP to implement a distributed Observer Pattern, where clients can become servers?

Duncan Cragg: Indeed. REST can be symmetric when used in integration outside of the Web. Further, these server-clients can have resources that do their own GET-ing, with POST callbacks, in order to Observe other resources. I've given many examples of this style.

eA: So how do you program such a distributed system at the business logic level?

DC: Well you need to be able to easily express that the state of a resource depends on the latest state - the intentions and declarations - of other resources it is Observing.

My vision is that you would express this business logic in a simple, powerful and expressive declarative language.

eA: Hmmm.. Just try getting all declarative on someone who just wants his share prices on time! No-one in business understands such abstract concepts.

DC: Oh no? The favourite programming tool of business is the spreadsheet!

And HTML is a form of declarative programming: imagine if you had to build your DOM from the top using Javascript! Or if you had to set all your styles in Javascript instead of using declarative CSS!

eA: Well I've seen some pretty ghastly examples of those two crafts!

And many failed attempts to allow non-programmers or business analysts to program directly.

DC: True, but that doesn't change the fact that most non-programmers can only think declaratively - they know What they want, but they haven't got much of a clue How to get it.

Business people talk business rules and business data.

eA: We just can't program with them!

DC: Well, perhaps we just haven't found the best discipline of pragmatic formalisms and methodology that will allow those rules to easily become programs.

It's a well-supported claim, though, that declarative programming is simpler, clearer and more productive than imperative.

eA: OK - so how would you actually define these business rules?

DC: You could use a rules engine, or a DSL engine, or even XSLT - if that works for you. There are many ways to transform XML.

eA: How would you do it?

DC: I would like to have an XML rewriting and templating system: "if this XPath or XML template matches Observed XML resource A, this one matches resource B and this one matches myself, rewrite this and that bit of myself with these bits from A and B".

 

Web Objects Ask, They Never Tell

eA: So you want to go around re-creating huge XML documents all the time?

DC: Who said anything about huge? This is another Web thing that doesn't necessarily apply to us in REST integration: we don't need the equivalent of the giant, monolithic HTML page. We can work in much smaller chunks.

I'd even just call them 'objects' rather than resources. Or 'Web objects', if you like.

They could be little XHTML carriers of Microformats.

eA: I thought we'd given up on distributing fine-grained objects back in the CORBA days?

DC: Ah, but this isn't trying to transparently distribute zillions of method calls in an RPC model.

This is about optimising state transfer. Only send what you need, cache where you can, push when something changes. Separate your data by rate of change, timeliness, cacheability.

eA: So these Web objects don't have any methods? Because that would be RPC when distributed?

DC: Exactly. In REST integration, the Web objects Ask, they never Tell! These Web objects are reactive: Asking for public state, not Telling each other what to do.

eA: You mean the opposite of Tell Don't Ask?

DC: Yes. As an object, you don't Tell another object How to do something, you Ask for What you want by either simply Observing its public state or by it Observing yours, then letting it decide How to evolve by itself. You then watch it and react or interact.

It's the "imperative to declarative inversion": everything is turned upside-down or inside-out when you distribute things this way!

eA: Oh yes, your "inevitable inversion" thing.

DC: Another indication of this inversion from the imperative object-oriented world to the declarative ROA world is how the derided 'train wrecks' of object-orientation now become the essential XPaths of the object Web.

You could say we have no methods, only 'getters', and XPath 'train wrecks' are encouraged!

eA: Doesn't sound too safe to me - it breaks encapsulation, doesn't it?

DC: Well, it's actually safe to dig around, since the data is held in shape by a stable, open, public schema. You're expected to go traversing the tree.

And you get excellent encapsulation since Web objects are total masters of their own destiny: they privately control the evolution of that public state.

You very much retain the value of 'What not How'; in fact, in a much better-defined way since it's fundamentally baked in to the programming model.

 

Hyperdata as the Engine of Object State

eA: But when you write 'train wrecks' you often end up jumping from object to object. How do your object Web XPaths do that, assuming they want to?

DC: Hyperdata of course! Links to links around the Web. Objects can have their opaque UUIDs or GUID object handles encoded into their URIs. Then objects can be wired up with XHTML links.

eA: So now your XPath transparently jumps these links?!

DC: Actually, yes! That would then allow us to dynamically break up data into more manageable chunks without breaking the XPaths that traverse it.

Also, with this approach, you still get to drill down to data like in transparent URI paths, but you now use XPaths that are properly a part of the content layer, jumping transparently over those opaque inter-object URIs.

eA: Would you use these 'jumping XPaths' in the rewrite rules you said you wanted?

DC: Of course. Either linear XPaths, or XPath-like XML tree templates on the left-hand side of a rewrite rule.

You'd start a template match on yourself, then continue on to match other objects by jumping over links, then on and on from object to object.

eA: So I suppose any such jump to another object means you then need to start Observing it, right?

DC: Exactly. And if hyperlinks are the only way you can find other objects to Observe, it brings us to the following:

Your object's next public state depends only on its current public state and the states of those objects that are visible to it through hyperlinks.

eA: Sounds a bit like the "Hypertext As The Engine Of Application State" constraint of REST.

DC: Exactly! Except now that we're doing REST symmetrically - now that clients can be servers, too - client Application State can have its own URIs!

eA: So you could re-phrase this as the even more intimidating: "Hyperdata As The Engine Of Application Resource State"!

DC: Well - how about just "Hyperdata As The Engine Of Object State"?!

eA: So do I have to wait for this link-jumping XML rewrite engine of yours to express my business logic, in order to get "Hyperdata As The Engine Of Object State"?

DC: No, of course not! Use a nice, dynamic, XML-talking language, like Scala, and follow the same principle, perhaps using a DSL.

eA: Can't you have objects that aren't entirely dependent on others? Like those that represent external state?

DC: Of course. That's normal Web stuff. It's probably best to keep these 'pure': to have either fully-interdependent objects driven by Hyperdata, or fully externally-driven ones.

 

Class, Extension, Instance and Behaviour

eA: Right, so we've got these little Web-mapped XHTML Microformat objects all linked up and watching each other in an Observer Pattern. This object's state depends on that linked object's state according to rewrite rules or a DSL.

So taking this mapping to objects one last step, I presume object 'class' maps onto an XML schema, XHTML Microformat specification or other content type?

DC: Yes. A public grammar in some form. Domain or business classes only, of course, not low-level classes.

eA: So, does each standard Web object class or type have a standard set of rules guiding its evolution or behaviour?

DC: Yes. If you see something and recognise its type, you can attempt to interact with it according to its public specification, and it should, but needn't, react.

The public specification can define the expected behaviour in the RFC language of MUST and SHOULD, like AtomPub. Or it can define it in rewrite rules!

eA: So then, how do I add my own business rules to a content type or 'class', if its behaviour is standardised and meant to be stable and predictable?

DC: There are two ways you can make use of a generic standard, and both are forms of layering or abstraction:

You can 'mash up' standard component instances with declarative configuration - just using them as they are for your domain.

Alternatively, you can extend or subclass their standard structures and behaviour to become themselves more domain-specific. For example, rules can be overridden or their right-hand sides merged.

Again, think of how AtomPub can be used in any domain that looks like time-ordered lists of content; and how you can extend AtomPub without breaking unextended clients or servers.

 

The REST Observer Pattern

eA: I think you need a memorable name for your symmetric-REST, "Hyperdata As The Engine Of Object State" architectural style!

DC: How about the "REST Observer Pattern"?

eA: But a Pattern has to be a retro-fit to existing behaviour, and "REST Observer" is quite new.

DC: I mean REST .. [DC does double bunnie ears] "Observer Pattern"!

eA: Have you got any worked examples of the REST Observer Pattern?

DC: Will have soon...

(c) 2006-2009 Duncan Cragg

 

Coming soon: Worked examples of the REST Observer Pattern.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...

http://duncan-cragg.org/blog/post/content-types-and-uris-rest-dialogues/ Content-Types and URIs | The REST Dialogues 2008-02-16T23:44:00Z 2008-02-16T23:44:00Z

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 6: Content-Types and URIs   ...

In an exclusive nine-part dialogue with an imaginary eBay Architect, we present an accessible discussion of the REST vs. SOA issue.

Although eBay have what they call a 'REST' interface, it is, in fact, a STREST interface, and only works for a few of the many function calls that they make available via SOAP (GetSearchResults, GetItem, GetCategoryListings, etc).

In this dialogue series, I argue the case for eBay to adopt a truly REST approach to their integration API.

Part 6: Content-Types and URIs

eBay Architect: OK, enough fancy REST or ROA interaction patterns! Let's get back to REST basics.

Duncan Cragg: That'll be content types and URIs, then.

eA: OK - I've a question about your standard content types. In particular, about Microformats.

DC: Go on.

eA: You keep on mentioning Microformats when offering examples of content type standardisation. But Microformats are always embedded into HTML (or, preferably XHTML).

Are you suggesting somehow taking them out and using them separately? Or do we have to always use them through a, possibly inappropriate, document model schema?

DC: Microformats currently 'tunnel' through mechanisms in XHTML such as the class attribute. They themselves have distinct model schemas, that ride on the document model schema.

eA: So can you use them apart from HTML?

DC: No, not really, although you can carry a single Microformat in a single document carrier, then squeeze out any document model content that isn't supporting the Microformat, perhaps leaving enough that it still renders sensibly in a browser if anyone looks.

The Microformats people won't like you doing it, I'm quite sure, but if you and I want to exchange a pure hCard, hCalendar, hResume or hReview, and nothing else, then we can use the minimal document model carrier, and have just one Microformat per resource.

eA: But why not use the original data schema, before it was Microformatted? Why not just use vCard and vCalendar?

DC: Or use Atom instead of hAtom! Of course, if vCard has an XML representation, you could use that - as long as the constituency of your clients is the right one and is big enough. There may be more code out there that 'gets' hCard than an XML vCard. And some Microformats - such as hResume and hReview - don't have an original schema and are based on abstracting from common or prior behaviour.

This is REST integration we're talking about, where data, not documents, are native, and we aim to search out the most popular and most widely understood data schemas - even if carried over documents - to maximise interoperability.

eA: OK, that seems fine. Although I'd point out that REST doesn't have the monopoly on interoperability. SOA does that too.

DC: Interoperability is best acheived by sharing millions of URIs dereferencing to a handful of standard content types, with interlinks across the Web of resources. ROAs (Resource-Oriented Architectures) do that. SOA doesn't.

eA: REST APIs don't always have to do it. In the previous example you went through, eBay and gBay could offer REST interfaces but not talk the same schemas and not allow cross-linking in the way you described. Or talk the same schemas but not recognise each other's Items and Offers.

DC: That would be walled-garden, silo-thinking. It's also 'API' thinking. Just opening up a port to your application, even one with correct use of GET and POST on well-organised domain URIs, isn't in the spirit of REST, and certainly isn't good enough for REST integration.

In REST we always aim to adopt the same schemas, to aim explicitly for interoperability. And linking between those resources, even cross-site, is fundamental to the REST way of thinking. If someone offers you a 'REST API' that uses unnecessary proprietary schemas that miss obvious interlinking opportunities, especially across to other sites, run away!

eA: Are there any real-world examples?

DC: A good, and ironic, example of this is Google's Open Social, at least in its earlier releases, which fails to achieve true cross-site openness even with a 'REST API' and shared schemas, because sites don't cross-link or actually allow data sharing. Also, the schemas are a strange extension of Atom, rather than using, for example, vCard as the basis for 'People Data'.

This hopefully will be fixed as the 'REST API' evolves and with the work going on in groups such as DataPortability, with agreement from the major operators.

eA: So much for the interoperability of that REST interface.

DC: The heart of good REST interoperability is the acceptance of standardised data at a 'foreign' URI, and the re-publishing of that foreign URI in your own standardised resources. It happens on the Web all the time, of course. We just need to copy the model for REST integration.

Hypermedia and (more importantly, here) 'hyperdata' is baked into REST, but is an afterthought in SOA. ROAs create an interlinked hyperdata landscape across sites and domains. I'm using 'hyperdata' here in the sense of interlinked data resources in REST integration, by analogy with hypermedia, not in its Semantic Web sense.

eA: Ah! But how do your little pure Microformat resources link up into this hyperdata landscape? Microformats can't link to each other, can they?

DC: It's true you may have to go and get involved in the Microformats movement in order to help define how to link an hCalendar event to a list of hCards of people attending. Or the hCard of a company to a list of hCards of its board members. Or an hReview to the hCalendar event being reviewed and the hCard of its author. Or to include the XFN list of links to friends' hCards inside a person's own hCard.

One indication that there's something not ideal in Microformats is the fact that you have to write someone's hCard out again and again for every page or site they appear on. If you could just link to a single hCard for that person it would be more efficient.

eA: But Microformats have a narrow charter: to decorate the document model with semantics. Any links are just part of the hypertext Web. It sounds like you're trying to make some kind of domain model out of them, with their own interlinks!

DC: Yup. When you start to think the data of REST integration, the document carrier of Microformats and it's often superfluous links can be a distraction. If the document links are relevant to the Microformat, of if people would use links within the Microformat if they were told what value it has, it would be worth pulling them out into the Microformat definition itself. Then enhancing in-browser Microformat parsers to follow links will greatly enhance their utility.

All you have to do is find real-world examples, and propose it on the Microformat lists! Meantime, reuse the schemas and keep all your extensions public and backwards-compatible.

eA: What about all those 'rel-' decorations? You know, rel-tag, XFN, etc.

DC: Well, hAtom is the only Microformat that specifies nested rel-links: rel-tag, rel-bookmark and rel-enclosure. Otherwise, each Microformat is independent, and the rel-links are independent. Like I said, it may be worth going to the Microformat community and suggesting more such rel-links beyond hAtom.

 

URI Opacity

eA: So this RESTful data landscape of data wired up with URIs: it sounds a bit hard-wired: where do URIs as queries (and URI templates) fit into that tight mesh?

DC: URI templates fall into exactly the same category as standardised content types and schemas in terms of their level of abstraction and location in the stack. In other words, the right thing to do, if it's transparent URIs you want, is to standardise search URI templates across sites of a type.

eA: This is getting complicated. It's hard enough to get agreement on the content types of resources, never mind on URI formats as well!

DC: Indeed, and in fact, I believe that URIs should be opaque: they already are to HTTP, but also in our data landscape, a URI should point to a single, predictable resource.

The mechanism of querying that dataspace should be separated out from the mechanism of linking it up.

eA: A bit like GUIDs?

DC: Exactly. In Enterprise applications, you often see GUIDs (globally unique ids) being used, and never see them mixed up with search strings!

Transparent, query or template URIs are either used to be helpful or decorative, or are an acceptable optimisation, as long as you know that it's tunnelling through or hijacking the URI for a quick query string.

eA: Tunnelling? Hijacking? You've dismissed a long-standing convention, in the Web at least! How else do you do query fetches?

DC: A better solution is the query-POST-redirect pattern: the client POSTs their query, then the server redirects them to a linkable results resource on an opaque URI.

The POST query schema can then be properly standardised in a content type, or 'templated' in the REST integration equivalent of an HTML form.

It's an extra round trip, but only one IP packet in each direction; a redirect or a GET can fit into a single IP packet - the cost is only in the connection latency.

eA: Why not just return the state of the resource you're redirecting to in the body of the redirect, to save even this round-trip?

DC: Yes, you could do that. It's not something seen in the hypermedia Web as far as I know, but this is REST integration, where we're able to come up with new sub-protocols like this - where HTTP response codes are often given much thought.

Further, the server can offer the option to snapshot this results resource, so that it's still exactly the same whenever the link is dereferenced - something you can't do with a query URI.

eA: What would Tim Berners-Lee say about this? Is it in the spirit or letter of his vision for how HTTP and URIs should be used?

DC: I've no idea! However, in my opinion, when Tim didn't separate the concepts of a globally unique identifier returning exactly one resource from a query string returning maybe none, one or many resources (in a list), he started a good deal of unnecessary confusion, even if non-fatal in practical terms.

The phrase 'hackable URIs' sums up the situation. We may have been forced into creating slightly better user interfaces if the URI textbox were taken away from browsers.

Make your interface your content and have good search and information architecture to allow your (opaque) links to be discovered. If you know that human users - or search engines - will be interested in reading some links at the top of your information architecture, then go ahead and use just a few simple, meaningful addresses.

eA: You're venturing into controversy again! I'm sure I keep reading about designing nice URLs being good practice.

DC: There was a time when transparent URLs were considered important, but now everyone just uses Google! All the energy that's put into URL good manners and systems of URI templating and naming is just a distraction from the bigger effort of standardising content and defining schemas.

Opaque URIs keep content in the body where it can be given a Content-Type, instead of the headers - the URL line.

This is related to my preference to put 'write methods' such as PUT and DELETE into the body instead of the URL line.

eA: How exactly?

DC: The URL line should have a definite target - an opaque, globally-unique URI - and a content transfer direction - GET or POST.

The rest of the application-level interaction, including anything that will affect state and any searching and querying, should be in transferred bodies with standardised content types.

(c) 2006-2008 Duncan Cragg

 

In Part 7: Business Conversations.

Note that the opinions of our imaginary eBay Architect don't necessarily represent or reflect in any way the official opinions of eBay or the opinions of anyone at eBay.

Indeed, I can't guarantee that the opinions of our real blogger necessarily represent or reflect in any way the official opinions of Roy Fielding...