Some weeks you talk to your colleagues and watch a documentary and feel unprepared to understand the world. This was one of those weeks. Ross was back in Camden and the last time he was here he had helped me take baby cloud steps while he got started on WikiReverse.
Two and bit years later WikiReverse has been launched and this is what I learnt.
There is this thing. Its called the Common Crawl.
Really – you have crawled the internet again. Again ? Haven’t we been doing this since gopher was mistaken for a red squirrel and everything was still in black and white, or was that black and green ?
Unlike Ross I wasn’t really that excited, when, back in 2012 Common Crawl, using Nutch, started dumping everything they found on the Internet into Amazon (AWS). Making it accessible for free (yes as in freedom).
But today I am much more interested in the Common Crawl project because…
Continue reading “What WikiReverse Teaches Me”