TFL prepares to monitor big data during London Olympics 2012 [Video].
TFL prepares to monitor big data during London Olympics 2012
Reply
Big Data, Hadoop and Sports
Last week I stumbled over yet another buzz term in digital. Big Data. For sometime I’ve known that lovely people are working very hard on Hadoop projects, secretly in glamorous places. Telling me snippets of exciting news. I really didn’t get the link till I went to see my brilliant friend Hilary on a panel with Hadoop founder Doug Cutting at London Big Data.
Apparently, and the audience weren’t sure but ‘Big data’ refers to data sets whose size is beyond the ability of everyday software tools to capture, manage, and process within a reasonable time. I don’t understand this. That means when people created punch card programs in the 70′s or I waited for my game to process my scores by working with a date drive in the 80′s we were all dealing with Big Data? Those data sets seemed very small to me.
As part of work at KSL I helped a start up in this area. Using Nutch we devised a way to create a knowledge intranet for a architecture design company. They had several large servers with unstructured data such as documents and cad drawings. Like every company they also had systems of structured data such as HR and accounts systems. The idea was to dynamically generate a view of the data either based on the employee or a project all thrown up in Sharepoint. Sadly we didn’t have Hadoop and never managed to get Nutch to scale. It didn’t help that the servers were based in a bit of Manchester suffering from brown outs. So I have had first hand experience of Big Data and the problems it throws up.
However I have been thinking how useful such a system maybe after London 2012 has passed successfully and everyone has moved on to another glamorous role. The numbers for London 2012 are big. London 2012 is a group of organisations that has grown quickly as has the data. Each Games has access to large amount of data from previous Games and will generate large amounts of data for the next Games. To get a flavor check out the British Library’s Web Olympic and Paralympics archives. By the way they use a 50 node Hadoop infrastructure for the entire web archive project.
Sports events are made possible by tickets sales and Ticketmaster, people who sell a lot of sports tickets have recently invested heavily into Big Data. Last year they launched LiveAnalytics, a dedicated team of statisticians, modelers, and researchers focused on providing live event intelligence to sports teams and venues. They have exciting projects in dynamic pricing and social tickets sales.
Ticketmaster’s John Forese talks about how losing a player through injury can reduce ticket sales. Here big data has been helping too. IBM has been working with Rugby Leicester Tigers. They have been monitoring the performance of the athletes and detecting when they are past their tiredness threshold and at risk of injury.
Technology and high volume stats also permutates the field of the play. Time monitoring wrist bands for open air marathon swimming, or in pool swimming, touch pads are used to record when a swimmer dives into the pool and when they touch the edge of the pool at the race’s end. In track cycling, a radio transponder is attached to the front tyre of each bike emitting an identification code to antennae placed at the start and finish lines. All of which generating tons of data.
We are at a place in computing history where all sort of technology can generate large amounts of data. Hence the Big Data debate. However just a quick gallop through the sports industry alone and we can see that we are at an exciting early stage of what data scientists can manipulate and feed back with their resulting analysis.
Open Data – Games – Wired Magazine
Working with the Olympic Delivery Authority (ODA) it seemed important to ensure that as much road and travel changes due to London 2012 makes it into the public domain. This was done as part of an agreement that has seen the ODA’s Travel Demand Managment team working closely with Transport for London. Wired Magazine explains more.
Remote Control – ICA
Poor exhibits are upsetting. As is writing the review. You get excited about organising yourself on a quiet Easter weekend, make the effort just to be disappointed. As annoying as flicking through the hundreds of channels on Sky television to find that there is really nothing to watch. That probably is the problem with the Remote Control exhibit. Television has indeed had an enormous impact on our culture but perhaps it is to vast to cram into two small galleries in the ICA.
Since the Internet hit and I embedded myself squarely in its development I have been looking for the next revolution. In the early part of this century I really thought it would be about digital television and convergence. It’s safe to say I was wrong the media world is diverging with new devices and the applications are being built specific to medium. Although I would still love to see fiction story telling cross platforms just like following a friend’s life story via twitter and Facebook on phone and email.
The Remote Control exhibit coincides with the end of analogue broadcasting in the UK ensuring the nation is completely digital. The exhibit It is really an excuse to dust down a few artistic installments rifting on the notion that the television kit is art itself or that content can be re-cut to be installation art. Continue reading
Arduino Iris Hack Day
For a few years I have been wondering what I would make if I had electronics training rather than software. We maybe about to find out. Iris were kind enough to invite me to their Arduino Hack Day
Ref
Make a Book
Don’t just write one. Make one. In the quest to make code sometimes it is good to diversify and do something non tech. Cook. Garden. Make books ?
After all books are the lifeblood of development. When I was small I needed magazines and books for copying little bits of assembler codes. Now google has the answers. Last year when I learnt Ruby I didn’t go near a book. It was some delight that the Arduino hack day this month had books littered around.
The book making class jcn and I attended was part of Selfridges Words Words Words campaign. Continue reading
Inspire coders, value coders and make coding accessible
The lovely Dr Sue Black has created a little facebook poll asking us about our first programming language. Mine was Basic. I must have been 12. My family were on the usual Saturday drudge around central Milton Keynes shopping centre. Instead of spending it in the library we wandered into a store selling computers and walkman‘s. My dad walked up to a computer and showed me how to program the following ;
10 Print "Hello World"; 20 Goto 10;
Within a week I was locking up school computers with naughty hacks of the above. Within a year I was buying computer magazines and copying out assembler code on my Acorn Electron.
It wasn’t my first machine. That was a games console called Soundic. I was 4. Continue reading
Mac, MS Exchange and Mobile Phone Happiness
For the longest time I have wanted a Calendar that works as well as an IMAP e-mail account. Every time I change my phone or computer I spend ages getting set up. A remote access Calendar will save me oodles of time.
After the death of my iPhone, my requirement for a remote hosted calendar become urgent. The obvious choice is Google. Frankly it creeps me out Google will happily trawl your content in the hopes of better advertising. I know it’s not a sentient application but I wouldn’t let anyone read my diary or correspondence, so by extension, potentially bug ridden programs are objectionable.
Continue reading
British Library’s ‘Out of This World’ exhibition and Lectures
UFO
I noticed that the British Library had decided to trawl its vast archive and draw together a Science Fiction exhibition. It sounded phenomenal. A lecture on Lem, J G Ballard and a full exhibition. As I checked in on four square each week, I was surprised that others shared my excitement.
The truth is that this series is perhaps far to intellectual for me or perhaps too focused on the literary geek. I chose the lecture on Lem because I was sucked into a Radio 4 haunting adaptation of Solaris. I read J G Ballard Cocaine Nights when working on a feature film that was never filmed. Both authors have their deep links to film and it was wonderful learning about their lives and work from true fans and in J G Ballard case family.
JG Ballard Panel Debate
If anything changed my relationship with Science Fiction then it was the inclusion of the Bronte’s in the main exhibition. Writers, the British Library tells us, have long asked the big questions; What does it mean to be human ? Are we alone ? What is reality ? Is there a perfect world ? and What are we doing to our world ? Apparently everyone has been asking these questions making them Science Fiction writers. In Iliad (850BC) Homer describes how the god Hephaestus created female robots and mechanical tripods. The Bronte’s children created detailed imaginary worlds. Mary Shelley created Frankenstein (1818). Pullman sent Lyra through multiple universe. The exhibit showed these all (including Pullman‘s manuscript).
Continue reading
London 2012 Mascots
My life at London 2012 started when the Marketing team asked me to deliver a new section of the Mascot website. It contains a series of Flash Games and the ability to dress the Mascots in outfits inspired by sport and the UK. Happily the Mascots and their fans have been delighted. I love the Beano series out now. Anyway this week to celebrate the launch of Mascot Music Maker, Finn and I made this;