What Not How - Posts tagged 'yaml'

How Ruby can enable the Web 2.0 Platform

2007-06-26T15:17:00Z

Web 2.0's definition includes seeing the Web as an application platform. Which means it is in competition with Java and .Net, and with SOA, for both local and widely distributed applications.

If the Web is going to be a platform, the skills you need to learn to program it are the core Web 2.0 technologies such as Ajax, JSON, Atom, Microformats and OpenID.

And Ruby. This language, that's capturing the hearts of many Web 2.0 programmers, is ideal for easing the transition from the Java and .Net platforms to the Web platform, as I will show.

Even if you're part of a big company that is generally immune to the latest trends, the marriage of Ruby and the Web-as-platform may be something to prepare for. It could even displace your SOA agenda...

Few would disagree that the Ruby language is riding the wave generated by Ruby-on-Rails. In turn, Rails is riding the Web 2.0 wave, coming as it does from underpinning the very Web 2.0 37signals product suite.

Rails and Ruby have tapped into the tech Zeitgeist of friendly, simple and powerful. The speed with which the Ruby and Rails communities have delivered the key components of Web 2.0 is matched by the speed at which Ruby and Rails books are leaving the shelves.

What is the ideal platform of Web 2.0? Will it be Rails and Ruby? Will Ruby ride the Web 2.0 wave into the mainstream in the same way Java rode the Web 1.0 wave?

Well, here's the problem with that question: Web 2.0 is supposed to be primarily about the Web itself as the platform, as explained first by Tim O'Reilly and then by a thousand Web 2.0 vendors and industry watchers after him.

Web-as-platform is not just vendor hype or pundit hand-waving. Let's think about what O'Reilly meant by that.

Web as Platform

Web 2.0 is about making the Web more interactive, and thus able to support applications where Java and .Net would once have been considered the sole delivery platforms.

The fact that the technologies of the Web can be turned to this use is a shift with far-reaching implications.

Broadly, the shift we are seeing is from the one-way, static document delivery of Web 1.0 towards the two-way, dynamic data exchange of Web 2.0.

This fundamental repurposing is delivering more complex, interactive applications that work inside our browsers and which fully leverage the benefits of online operation.

Web 2.0 is bringing the user and their stuff into the very Web that they hitherto only passively consumed. This network-enablement of the user in turn enables their social networking and their shared creativity and self-expression.

Web 2.0 has tapped into a deep human need - a fact reflected in the vast traffic volumes and correspondingly vast valuations of Web 2.0 startups that we're currently seeing.

But Web 2.0 is not just for the startups: Enterprise Web 2.0 is coming! The bigco.com site is going to be looking a little, well, static and lifeless when compared to the new sites that are springing up everywhere, and that most of BigCo's employees are using. Further, BigCo can gain huge benefits from Web 2.0 approaches empowering and connecting those employees on the Intranet. And that Intranet is an ideal platform for deploying company-wide, interactive applications.

This shift in the Web to two-way dynamic data is being powered by a set of technologies that a Web platform programmer is going to have to learn.

Web 2.0 Platform Technologies

Anything that claims to be an application platform must support data. Web 2.0 is above all the data Web. Web 2.0 is about semantics, not free text and font sizes. Hence, it inevitably starts with data-oriented formats such as XHTML, YAML and JSON. In Web 2.0 more than ever, we talk about data not documents and about separating data from its presentation. CSS is big in Web 2.0, for good reason (not just for gradient fills). Inside the page of a self-respecting Web 2.0 application, you'll often find Microformats - again, semantics in the page: publishing concise data of widely-understood standard formats. Some of those Microformats may be tags, and in Web 2.0 the simplest and most powerful semantics are those little pivot points in Webspace.

Again, if you're going to be a general purpose platform, you need to be able to fetch, update, notify and display that data. Web 2.0 integration usually happens via JSON data structures and REST interfaces (some of which, especially those based on AtomPub, are true REST). Following on from the data-like pages we serve to browsers, come the data-like feeds we publish to feed readers and to other applications. After feeds, the core technology that gives Web 2.0 its dynamism and interactivity is Ajax and DHTML, and increasingly Comet (server push to the browser). The core technology that gives Web 2.0 its users is increasingly OpenID.

All of the above are open technologies. You can do Web 2.0 without proprietary technologies, just like Web 1.0. Indeed, keeping to the principles that made the Web successful is also essential to the success of Web 2.0. The Web platform is the first application platform that has to consider scalability and interoperability, and will ignore them at its cost. I have written before about open data, use of standard data formats and using REST properly to avoid creating unscalable, walled-garden sites. You don't need Flash or SilverLight, you don't need vast amounts of custom Javascript, you don't need function calls tying you to your servers.

Programming the Web 2.0 Platform

So, we've got the dynamic data that you'd expect of a would-be platform. But how to drive changes in those data? How do we program the Web platform to animate all this data?

All Rails programmers will know the above technology list; it comes with the territory. The Web 2.0 Platform can be very succesfully powered by Rails and Ruby. Ruby and Rails make Web 2.0 applications simple and quick to program, addressing many of the needs of simple Web 2.0 applications out of the box. There's little doubt that Ruby and Rails will have a secure future riding the Web 2.0 wave.

However, for many Web 2.0 applications, programming may not even be necessary, at least not in the procedural or imperative style programmers expect.

Look back to the early 90's: 'Web 1.0' made a whole class of applications easy to write without programming: applications for navigating information. You just wrote in HTML, declaratively.

Now look back at the long path of evolution of Java, through J2EE, Spring, AOP, IoC, Domain Driven Design, POJOs. All trying to achieve the simple goal of 'remove all that MVC and persistence stuff and let us concentrate on business or domain objects'. But they never quite seemed to get it right.

But then Rails comes along, and has succeeded by simple virtue of concentrating on easy manipulation of the 'Intel Inside' of Web 2.0 - data.

It's reminiscent of the 'Naked Objects' approach to application building with minimal programming (just business or domain code in POJOs that expose state into the GUI and are transparently persisted). The Streamlined project takes Rails even further down this path. Rails' nearest competitor, Django, has an admin interface that works in a similar way, automatically generating edit pages based on the data model.

Web 2.0 is about data, about semantics. Web 2.0 is inherently declarative. So Web 2.0 applications can be written declaratively - Web 2.0 mashups can be just wired together and their data animated by business rules. A bit like programming spreadsheets.

Teqlo, Coghead, Pipes, DabbleDB, LongJump, Popfly, AppExchange and Wyaworks are all examples of the different ways to program the new Web 2.0 platform without imperative code.

That's what we mean by Web-as-platform - not only is the underlying programming language irrelevant, it will often not even be needed, certainly for simple data manipulation applications and for many simple mashups. Being RESTful gives you a massive head start in this, of course.

While Rails is already in the game with its innate understanding of Web 2.0 techniques and philosophies, Ruby itself has a huge amount to offer the would-be declarative programmer, who is making the transition to this new Web platform from their traditional Java or .Net platform. In particular, it is easy to write your domain logic in a declarative style in Ruby: they call them 'DSLs' these days, but the idea is the same in most examples I've seen.

Web 2.0 - The Web Redux

Now, if you've been following this blog, you'll know I have a few opinions on Declarative Web 2.0 and on patterns for programming REST. Essentially I argue that, if you want to play in the Web 2.0 platform game, you don't want to be writing screeds of Javascript functions that call more functions on your servers.

I recently presented some ideas along these lines at the WWW2007 conference, entitled 'The Micro Web: putting the Web back into Web 2.0', where I also showed a demo written in Python.

This approach combines my Distributed Observer Pattern with Comet push to enable highly dynamic Web 2.0 applications to be coded RESTfully and declaratively, with zero Javascript. The Distributed Observer Pattern offers a clean programming model for animating the Web 2.0 dynamic-data technology set I described above.

I believe the Observer Pattern is core to the way we'll be programming when the Web 2.0 Platform hits mainstream. It enables the kind of event- and rule-driven programming that matches the characteristics of the Web 2.0 dynamic data platform. As a further killer benefit, it also directly addresses the optimal utilisation of multicore processors.

I am currently porting my Python implementation of this approach to Ruby, in the Redux project on Rubyforge. Redux stands for 'Ruby Event-Driven Update Exchange'. It uses the highly scalable EventMachine epoll-based event loop to power its event-driven architecture. This will be essential when Redux is asked to scale up a Comet-based application.

Like Rails, Redux will be a Web (2.0) application framework, but unlike Rails, it puts the Observer Pattern and event- and rule-driven programming at its core.

Redux's headline is 'Web 2.0 in-a-box' or 'Naked Objects on the Web'.

Conclusion

If you're in BigCo, and are responsible for setting BigCo's technical strategy, then train your Java devs up on Web 2.0 core technologies such as Ajax, JSON, Atom, Microformats and OpenID.

And fire up their enthusiasm by tapping into Ruby (perhaps via JRuby) on your way to the Web 2.0 platform.

Learn patterns for mashing and integrating. Learn about REST and event- and rule-driven programming, including declarative DSLs.

When this Web platform hits BigCo, you will probably find that its REST or ROA style make your SOA integration strategy look rather complex and unweildy.

Check out the Distributed Observer Pattern, and download Redux when it's done (I'll let you know if you subscribe here!).

In 2007 and beyond, its the Web itself that's the platform, not Java or .Net. But if you want to get there via a language-based platform, Ruby could be the best way to transition to it.

Note: Everything I said about Ruby and Rails applies equally in technical terms to Python and Django, but regardless of the significant benefits of the latter, Ruby and Rails have the Web 2.0 market and mindshare. I'll probably switch this blog from Django to Redux sometime this year..

Lighter-than Wins in 2007

2007-01-18T11:12:00Z

What do all the MAJOR Web 2.0 technologies of 2007 have in common?

Let me list them first:

    M.icroformats (including tags)
    A.jax (including Comet)
    J.SON (plus YAML)
    O.penID (plus SXIP, LID, Yadis)
    R.EST (including Atom, APP)

What these technologies have in common is that they're all lighter than their competitors:

Microformats	Lighter than the Semantic Web
Ajax	Lighter than Fat Client (!)
JSON	Lighter than XML
OpenID	Lighter than SAML/Liberty Alliance
REST	Lighter than SOA

I'm reminded of Tim Bray's Technology Predictor Success Matrix and his appreciation of the 80:20 phenomenon. If hitting the sweet spot of achieving a lot with only a little is the main test of potential, then these five Web 2.0 technologies are all set for success.

So it's no surprise to see Tim supporting Microformats once or twice and even JSON, in spite of his, um, XML background. He also understands and uses Ajax and backs REST (in particular via Atom). Tim has not yet mentioned OpenID, instead referring to SAML and Liberty Alliance (please don't be crass by leaving a comment speculating on the reason for this...).

These MAJOR Web 2.0 technologies should form the basis of the two-way, interactive data Web or the Web-as-a-platform vision that we've all been promised.

However, they will need taking a little further this year, in order to bring them closer together and to make this vision happen in a seamless way.

A synergy between these 80:20 technologies can potentially produce much more than just the 'next version of the Web', if engineered well.

I have already discussed the Subversive Microformat, I've promoted Declarative Ajax and currently I'm slowly working up to Symmetric REST.

Next up, I'll be asking you to consider a web of JSON Microformats and ways to merge hCard and OpenID into active user personas - even avatars...

I'll tag this acronym as 'rajmo' instead of 'major', for obvious reasons.

Excuse me - did you say 'Web' Services?

2006-05-17T00:27:00Z

Distributing an application over a network isn't just a case of splitting it down a natural line and putting a network in-between. What works in-process simply doesn't work so well across the wire.

And just calling such an Internet version of application and process interfaces 'Web' Services doesn't mean it has anything at all to do with the Web, or that it in any way shares the Web's scalability, flexibility and robustness.

Indeed, I claim that you cannot distribute without also 'inverting'; you have to face what I call the 'Imperative-to-Declarative Inversion', if you really want a successful, scalable, distributed application.

Declarative Architectures such as REST (i.e. the Web, and now 'Web 2.0') dominate the broader Internet.

Take a simple application: calendar events. Let's first see what the Web Services approach is to distribution...

We'll start with an object-oriented calendar events class that hides the data and gives us methods to read from and write to it, plus some methods to add and subtract days, to calculate the number of seconds till due, etc. The calendar class interface is public and its data is encapsulated and private.

Now let's put this calendar event class on the Internet. We run up a Web Service that makes our calendar event class look much like it did before, with methods or, in fact, functions exposed by an interface. We lose a bit of object-orientation, but that is considered acceptable these days, after some bitter experiences with CORBA...

In fact, synchronicity is also frowned upon these days after similar lessons of the past. So let's help our Web Service a little by making it asynchronous. We'll drop off our commands and either come back every so often to see if they're done, or wait for a callback to tell us so. Instead of many synchronous functions, we have message schemas describing jobs, along with the needed data. Results come back as messages, too. The public message interface still talks of actions and functions that can be called or invoked. However, it acknowledges the constraints of the network by running asynchronously and by having coarser-grained functions: the messages are chunkier now, perhaps called 'documents'.

The difference with the distributed version is the noticable lag in interaction and the demise of the service when disconnected or under network outages. But it's acceptable on a good LAN connection.

Now, look at the benefits. Our calendar events application can be shared (very important for calendars). It's available and accessible to other applications on the network. Someone else can manage it, who are perhaps better at it than us. It can be accessed regardless of operating system and programming language. A number of authorized users or clients can create, manage, process calendar events.

It may even be possible to export calendar events. The Web Service might have getCalendarEvent(id), getCalendarEvents(date), etc. request messages - and may return XML documents now that it uses a chunkier, message- or document-based interface. This doesn't necessarily break encapsulation because the syntax of the exposed data is considered part of the interface, with standards, versioning, contracts, etc.

We're now pretty much up-to-date with the latest and/or best practice in the Web Services industry. Rich, versioned invocation interfaces (WSDL, Message Schemas), hidden or encapsulated data.

'Web' Services??

The above scenario is fine for small-scale, self-contained applications shared by a handful of regular clients.

But what if you actually want to publicise your calendar events? How do you advertise them and share them somewhere; somewhere like the Web?

Unfortunately the model fails us now: hidden data = outside of the Web!

Hidden data means no URLs. So how do you pass around a generally-understood reference, not to the service, but to a calendar event on the service? How do you index it? Google indexes pages, not services. How, indeed do you bookmark it?

Hidden data means no caches. Web Services standards don't concern themselves with cacheing. Cacheing is implemented in an ad-hoc manner because it's outside the model.

So scaling our calendar service for wide publishing can't be done with caches like it is on the Web (or by BitTorrent). It can only be done at the service operator, by adding fatter pipes and fatter machines.

This is where the 'Web' of 'Web Services' starts to look like a serious misnomer. There's none of the scalability and interoperability of the Web, because the data's hidden!

You can't index it, bookmark it or advertise it; you can't cache it.

Clients of the 'Web' Service may end up screaming to be able to see their locked-up data and to break the rules of encapsulation. To start to realise some of the proven benefits of the Web itself, which is hugely scalable, bookmarkable, indexable, linkable, cacheable.

Just having getCalendarEvent(id), etc. export functions doesn't make you part of the Web - and the incentive to export to a known MIME type and schema is low when the whole of the rest of the interface is defined for this particular service.

Oh - and WS-Addressing and WS-Transfer may create a parallel WS-Web, but it's not the one most of us will be using.

Public Data and Open Formats are OK!

The Web has shown us that, in contrast to the hidden data of object-oriented programming, public data and open formats are not only OK, they're incredibly desirable.

On the Web, data replaces service as our 'public interface'. And data thereby tends towards standard, stable or open formats.

Our service interface to the calendar event objects can be replaced by publicly-visible calendar event resources - perhaps in the open hCalendar XML format. (You can choose YAML instead of XML if you want, and you can restrict access if you want - I mean 'public' or 'open' data in principle.)

These calendar events are each given a URL and we tell everyone what to expect them to look like. The layout of the data is now our interface, and we have to apply the same rules of standards, versions, etc. that applied to our service interface.

They may be fetched using GET, and updated using POST. The data derivatives, such as time-till-due, can be added to our open format, retrieved by a separate query URL, or simply calculated by the client given the basic information. Many people may be authorised to edit them via POST, including adding and subtracting days or moving the time forward an hour.

Perhaps they could be 'Atomised', using the Atom Publishing Protocol to read and update them and an Atom feed for when events become due or for when they've been updated, etc.

Due Diligence

Naturally, before jumping headlong into such a Web-enabled world, we must show due diligence over the costs of this approach.

The main cost as seen by object-oriented programmers is that they have lost the ability for the class or application to change the internal data format or representation of calendar events without users seeing anything different in the application or object interface.

Of course, if at some point the data is exposed (whether exporting or serializing an object), then the need for versioning thwarts this benefit anyway in practical terms. Indeed, whatever is public needs to be controlled, agreed and versioned, whether it's our new data interface or the old service interface. We've just moved the constraint over.

Now, the data we're talking about exposing is not that inside some low-level library which has nothing to do with the domain of calendar events itself. Object-oriented tenets, and the Abstract Data Types that preceded them, undoubtedly have a value at these lower levels, where function-based APIs are ubiquitous and programmers need to leave themselves some breathing room behind the API to improve the implementation.

No, calendar events are a domain concept, not an implementation concept. Unlike users of implementation classes, domain users do care about the basic structure of their domain entities.

Indeed, users only fully understand 'what', not 'how'; they need to talk in terms of tangible things first, then the evolutions and constraints of those things second. They may never, and arguably should never, understand the mapping from those domain concepts that they understand into the threads and processes that programmers understand.

That's why there's so much business analysis that goes straight into database design, so much business analysis that goes straight into user interface design, so much domain analysis that goes into ontologies and message schemas. Business or domain level data will always be openly discussed: there's no reason not to expose and constrain domain data structures.

In fact, some of the key features of object-oriented programming map perfectly well to a purely data-oriented analysis. Inheritance, 'Duck Typing', Polymorphism, and even forms of data Encapsulation are all possible at the domain data level. This is the subject of another article, however...

The Inversion

Now that our calendar event data is opened, the data has become more important than the application. The importance of our shared data is underlined by its acquiring a URL: a universal handle on it.

You could even see the application as 'animating' the data from within (you only see the changes to the data, not how it happened, which tool was used client-side or server-side).

We have gone from public process wrapper around hidden state to public state animated by hidden process.

This is the Imperative-to-Declarative Inversion.

And moving stuff to the Internet is one of the quickest ways to feel the Force of Inversion! By joining the Web community, we get the benefits of bookmarkable, indexable, linkable, scalable and cacheable calendar events.

The fact is, the Internet is driven by data and events, not service interfaces. Distributed systems must face the Imperative-to-Declarative Inversion if they want to join the community of widely-deployed, scalable and successful systems that run over the Internet.

Imperative, Service-Oriented Architectures, 'Web' Services and SOAP will always be limited to small-scale deployments sold by the vendor consortia.

Declarative Architectures and REST will continue to dominate the broad Internet.