Open Data .. has .. recently .. been.. all .. over .. the .. blog .. o .. sphere!
Openness is a classic Us-and-Them issue. Big, nasty Apple/MySpace/Flickr is trying to control what little me/SingleStatus/Zoomr can do with my/our own stuff.
Open Data vs. Closed; Open Source vs. Proprietary; P2P vs. DRM; privacy vs. surveillance. The battles between the freedom of the pioneer, the individual and the minority against the rules and stability of the establishment and the majority form the endless shape of human history.
Us beating Them is Hollywood's favourite subject on-screen - and ironically Them fighting Us Hollywood's favourite battle off-screen.
As an Us-and-Them issue, with Us less powerful than Them, it's also tempting to give up and to follow the crowd - to do what we're told, to not ask for or sieze the privacy and open data we feel entitled to.
However, at XTech 2006 recently, there was a set of talks on the subject with a more positive approach. ...
Open Data .. has .. recently .. been.. all .. over .. the .. blog .. o .. sphere!
Openness is a classic Us-and-Them issue. Big, nasty Apple/MySpace/Flickr is trying to control what little me/SingleStatus/Zoomr can do with my/our own stuff.
Open Data vs. Closed; Open Source vs. Proprietary; P2P vs. DRM; privacy vs. surveillance. The battles between the freedom of the pioneer, the individual and the minority against the rules and stability of the establishment and the majority form the endless shape of human history.
Us beating Them is Hollywood's favourite subject on-screen - and ironically Them fighting Us Hollywood's favourite battle off-screen.
As an Us-and-Them issue, with Us less powerful than Them, it's also tempting to give up and to follow the crowd - to do what we're told, to not ask for or sieze the privacy and open data we feel entitled to.
However, at XTech 2006 recently, there was a set of talks on the subject with a more positive approach.
I'm counted amongst the 'founding members' of the UK's Open Rights Group, having signed the pledge in PledgeBank. So I was delighted to see Suw Charman's name on the XTech programme.
Suw gave a talk about the issues they/we are tackling: ISPs and Telcos tracking our traffic, DRM controlling our media and even our computers, ID cards, Patents, various government schemes to undermine our privacy, Copyright extension, Trusted Computing, etc., etc.
Read about the issues, then join the group (or the EFF, etc., depending on where you live).
Now, this was cool. I'll start right off by noting that they have a REST API, not a STREST API..
And it just gets better from there.. OpenStreetMap is like a Geo Wiki of GPS trails around the world. The centre of London was mapped by attaching GPS devices to couriers and plotting the slug-trails they left behind. Steve Coast showed an animation of a day's tracking. The trails shot off to various points out of town at the end of the day. But the animation continued..
And at pub closing time, another burst of trails as the couriers came home!
A few weeks ago, a crowd of OpenStreetMap volunteers descended on the Isle of Wight in England to walk around mapping it. The enthusiasm generated by this project seems likely to give it enough momentum that it will very soon be good enough for us to use for real. A complete, reliable dataset created by the people, for the people, unencumbered by copyright and restrictive licensing.
I suggested an idea I had about five years ago in my dot-com days: let people upload mobile-snapped pictures with GPS coordinates attached. Central servers scan the pictures for text and OCR them into a searchable database.
People would snap-and-upload their own street name signs, house name plaques, shop fronts, station name signs, company office signs, etc. Even menus in the local restaurant. We could construct a 'GeoGoogle' of public, mappable, searchable text.
However, Steve told me that OCR and mobile GPS technology still isn't up to it; that manual tagging of GPS trails is enough to make OpenStreetMap work well. Ah well, I've waited five years, so I can wait a bit longer...
Paul Hammond (ex-BBC, Yahoo!) gave a good talk on how to motivate companies to open up their data to the Mashable Data Web, hopefully through nice shiny new Web 2.0 APIs.
Here's your takeaway quote courtesy of Suw Charman's amazing typing:
- Be aware of the problems
- Demonstrate usefulness, screen scrape if you need to, but don't get yourself cease-and-desisted
- Don't assume it's a technology problem
- Target the right people, find someone on the inside who can help you
- Talk about benefits to the provider, not the consumer. If you talk about the benefits to you, they'll see you just as someone who wants something for free.
- Have patience. It is getting better every day, and it takes time for business to come round.
The Data Web will be created mostly from the edges, but the middle should be encouraged to see the benefits, too.
Native to a Web of Data: Designing a Part of the Aggregate Web
Yahoo! had a big presence at XTech - as did the BBC - and this Yahoo! - and ex-BBC - presenter was Tom Coates.
Tom's talk extended the previous one of his colleague, Paul Hammond, by going over some principles of opening data and helping build the Data Web.
Principles such as finding data sources, giving them nice representations and URLs then distributing them in a most accessible and navigable way.
A bit Web 1.0, in truth; he looked rather blankly at me when I suggested adding some concept of data events or updates to his list of data-opening principles.
Again, Suw has this written up (actually, a better-looking version of the same talk from FoWA).
Oh - here's a good article by Tom on the power of pointability
Harnessing the Collective
All of these talks had a positive spin on the Us-and-Them issue.
We can do something about the creeping imbalance of Digital Rights, tipping slowly but persistently from Us to Them.
Web 2.0 and the Social Software it underpins can actually empower those of us that are willing to use it.
As taxpayers, voters, shareholders, pensionholders, customers and employees, we really do own and control Them. We are just one Social Networking movement away from realising that - in both senses of the word.
So: stop staring at YouTube, sign up to your local Digital Rights organisation, get out on your bike with your GPS, understand and talk nicely to that unenlightened, data hoarding corporation and start a bottom-up revolution to take back control over your own data...
Above all, design your systems to put Open Data at the top of your list of priorities...
There is something about the Internet that nurtures open data, and something about computers that nurtures closed. It is often necessary, but often painful, to make the jump from local, closed data to global, open data. ...
There is something about the Internet that nurtures open data, and something about computers that nurtures closed. It is often necessary, but often painful, to make the jump from local, closed data to global, open data.
The Internet is all about data; open data and open data formats in particular. HTML, the fabric of the Web, links up vast amounts of open data created in an open data format. Email is sent either in the same HTML or (the most open of formats) simple text. Feed publishing shows that opening data can become an unstoppable flow. Indeed, Peer-to-Peer technology shows that opening data can become a torrent...
However, on our computers, data is often much less free. Take a closed format such as Microsoft Word: such a document can only properly be unlocked after paying for Microsoft Office. Take copyright data under DRM (Digital Restrictions Management): such music, etc., has to be unlocked by paying for the key.
Even with nominally open formats, there's a tendency to tie one format to one application. You still need that application to unlock the data. And even object-oriented programmers follow the closed data rule: data must be wrapped in a class interface.
Application user interfaces, class interfaces and service interfaces whose implementation processes mediate, control and restrict data and their formats are the bread-and-butter of the non-Internet computer domain. They work best on the more tightly-controllable computer hosts.
They Just Don't Mix
But the processes and the people that like to control data don't mix too well with the Internet. Here are some examples of the incongruity (or outright clash) that can result:
Now, these processes and people that like to control data obviously can't just avoid the Internet. So they'd be better off embracing the open data philosophy if they're going to do things here.
Embracing open data can take courage and open-mindedness. It means letting go of proprietary lock-in, copyright obsession, RPC, RMI and Service-Oriented Architectures. The lock-step imperative, procedural, workflow and object-oriented programming that works locally doesn't translate well onto the 'Net. The rules change.
Open Data Can Work
In fact, it is possible to make a living in this wild frontier, even when everyone can see, understand and copy your data! (For example, watch this blog for ways to make money on the Internet even when unlimited copying is allowed, based on originality or novelty backed up by a little crypto technology!)
I call the flip from closed data to open data the 'Imperative to Declarative Inversion'. It's the flip from 'How' to 'What' - from process-centric to data-centric programming.
In this upside-down world of What not How, the data format and its update formats are the public 'interface', and process animates things behind the scenes. Reading about REST will provide an excellent grounding in this way of thinking.
Showing the benefits of inverting (scalability, robustness, interoperability, value for money, speed and flexibility of development) and helping overcome the understandable fears of inverting are the main goals of this blog. In fact, I will also show how the Declarative approach works pretty well off-Net too - in the domain traditionally owned by process...