There’s an awful lot of news around at the moment, isn’t there? But away from the Groundhog Day diet of rolling news channels, with their 15 minute cycles of library footage and speculation, and the hysterical gangster-porn of our local media, where do you go for context? Where can you pull back, take stock and get the full picture?
The truth is out there. And it’s probably on the internet. But wading through its daunting signal-to-noise ratio could sap even the most news hungry of reporters.
To dig down to the real story, you’ve got to mine deeper than the Huffington Post, press releases and half truths that litter the Internet’s entry level. And that’s something Liverpool based based ScraperWiki, recently awarded a $280,000 news innovation award funded by Google knows more than anyone.
The achievement went largely unnoticed in our local media but, as the only European recipient of the Google-and Knight Foundation funded award (the Knight News Challenge – an annual media innovation contest) set up to advance quality journalism and open access to information, Liverpool’s ScraperWiki was lauded in leading publications from the US to Japan.
A big deal? You bet.
“We did get a great write up in The Guardian, though!” says ScraperWiki CEO Francis Irving of the award (he’s in the centre of the igloo. No, we’ve no idea either), which recognised the site’s groundbreaking work in enabling journalists to better decode public information, and sniff out the stories that might otherwise be – deliberately or otherwise – obfuscated. “There are journalists who know that data is a news source, and want better ways to access it. This grant will help us achieve that.”
Access to information is something Irving has spent the past eight years championing. He set up mysociety – the Freedom of Information-based web of democracy sites which includes www.theyworkforyou.com, that nemesis of shady MPs across the land.
“Julian Todd (now ScraperWiki’s CTO) wanted to know how his MP, Louise Ellman (Liverpool Riverside) voted on the Iraq war,” Irving tells SevenStreets.
“It was really hard work to click through all the pages of debates on the Parliament website, and then find long lists of names for how MPs voted. He talked to me about it, and asked me to make a website which automatically extracted all that information (by a technique known as screen scraping) and presented it in a more useful form”
The result was the site Public Whip, accessed by millions each year, and still the only place you can easily find a full record of how your MP votes.
Taking pages of raw Government information (or, for that matter, information about any corporation), reformatting it, breaking it down and serving it in database friendly chunks is increasingly being appreciated as an engine room powering tomorrow’s headlines. By helping journalists to join the dots, and track a story’s development over time, data scraping is a way to make sense of emerging news, and set it out into the world.
It’s open, collaborative, accountable and – in bringing investigative journalism kicking and screaming into the age of the internet – it’s exactly the sort of public interest-driven innovation that gives us hope, in a sea of hacking and hysteria, that the right story will eventually rise to the surface.
“After some years of scraping, Julian decided that it our ideas were difficult to maintain, and that it would be good to have an online service making it easier to make and work on them together,” Irving says.
Eventually, the two hooked up with Aidan McGuire, from Liverpool software company Blue Fountain, who’d successfully won a grant Channel 4 to build the first version of the ScraperWiki site.
Last month, when the site’s increasing reach and its role in promoting honest, open and accountable access to data was recognised, Irving and the ScraperWiki team allowed themselves a moment of celebration. But the future’s still uncertain.
“The grant will give us a good shot to make it work to help journalists to use data to find new stories, but without more investment, our business plan is too tight in terms of cashflow, which will slow us down. We’re currently looking for some more investment from angels or a VC to cover that gap.”
The grant also pays for the ScraperWiki team to run events in 12 US cities, linking together reporters and programmers, so they can learn from each other. “Our aim is to lower the barrier to entry into data driven journalism and to make ScraperWiki really easy to use for everyone,” Irving says. “What really matters is that people are using it. Data is dead if it doesn’t have someone, a journalist or a citizen, analysing it, finding stories in it and making decisions from it.”
“From a business point of view, our events in the US gives us a chance to start making sales, which is really important in terms of bringing new money into Liverpool,” Irving says, adding that he’s determined to stay Liverpool based if funding allows.
“Liverpool is very strong at this link between the physical world and the virtual world, and it is a massive new sector with lots of possibilities,” Irving says.
“Nobody talks about ‘telephone journalism’ anymore,” he adds. “ It’s assumed that journalists will use the ‘phone to talk to their sources, check facts and so on. But once, the telephone was new and innovative.
“Likewise, right now, not many journalists make good use of the Internet, and increasing amounts of open data. But they will.”
In a month where journalism has been dragged to its lowest levels, where claim and counterclaim, scandal, sleaze and blagging have dealt fatal blows to the very fabric of our ‘free’ press, ScraperWiki’s refreshingly egalitarian approach to data without borders shows that there is another way. And that, in the right hands, ‘hacking’ can mean something else entirely: something of a force for good.
“Journalism is about understanding the world in both more detail and in context, and helping people make better decisions because of that. The News of the World affair has shown the danger of letting news go out of hand the other way, and ScraperWiki is about bringing journalism back to properly sourced facts.
“It’s about making better use of information that is already available, and finding stories in it that people don’t want to be found.”
And SevenStreets is more than happy to make this particular story public.
Don’t believe everything you read. We can be proud that at least part of our local media is in the vanguard of a new, open and democratic form of investigative journalism. Which, for our money, beats guns and gangster stories any day.
If you want help gathering new data, or making better use of data, get in touch with ScraperWiki.
UPDATE: Since this story was first published, Francis has secured £1million in investment for ScraperWiki, to roll out its collaborative data platform. target=”_blank”>Read about it here.