Betting on the Twitter stream with Elasticsearch and Kibana

 Posted on January 19, 2014  |  Barnaby Gray

Sentiment analysis, Elasticsearch and Kibana The idea From the Twitter streaming API grab tweets for a live TV show whilst it’s showing, classify by contestant, analyze sentiment and provide a real-time dashboard into the outlook. Try to predict good bets on the winners. Optional: Gamble. :-) The tools Grabbing the data: Scala and the twitter4j Java library Sentiment analysis: SentiStrength Search engine: Elasticsearch Dashboard: Kibana (also from the Elasticsearch guys) The show I picked the final of the BBC’s “Strictly Come Dancing” in the UK. [Read More]
twitter  sentiment  kibana  elasticsearch 

Dynamic DNS for EC2 instances

 Posted on November 23, 2013  |  Barnaby Gray

The problem Booting instances on EC2 is easy. But once it comes to connecting to them, you end up having to copy around transient and unwieldy ec2-....compute.amazonaws.com host names. So you manually set up a CNAME in your DNS to give the instance a friendly name, but as soon as you stop and restart the instance, the ec2 name has changed so your DNS record needs updating. Rapidly gets tedious! [Read More]
aws  ec2  route53  dns  dyndns 

Finding related searches with Spark

 Posted on November 19, 2013  |  Barnaby Gray

Unsupervised learning from user behaviour When a user navigates a site they leave a valuable trail of information - what their first search was, what they followed this search with, and so on. Using this data we can learn related searches automatically by co-occurrence counting. This post takes you through the steps to get from raw search logs to results using the Spark cluster computing framework. Spark provides a natural processing language for flows of data, and can be scaled up to clusters when data growth dictates. [Read More]
spark  related searches  co-occurrence  machine learning  scala 

Scala Future and the Elasticsearch Java API

 Posted on November 16, 2013  |  Barnaby Gray

Most examples of the Elasticsearch Java API you’ll have seen follow the prepare/execute/actionGet pattern (in Scala): val response = client.prepareIndex("twitter", "tweet", "1") .setSource(jsonBuilder() .startObject() .field("user", "kimchy") .field("postDate", new Date()) .field("message", "trying out Elastic Search") .endObject() ) .execute() .actionGet() This blocks in actionGet(). If you’re developing in a reactive environment (eg. Akka or Play), blocking is a no no. And even if you’re not, it’s a win to be able to avoid blocking so you can parallelize indexing to eek out the best performance. [Read More]
scala  elasticsearch  futures  java  api 

Spark and Elasticsearch

 Posted on November 12, 2013  |  Barnaby Gray

Elastic Sparkle If you work in the Hadoop world and have not yet heard of Spark, drop everything and go check it out. It’s a really powerful, intuitive and fast map/reduce system (and some). Where it beats Hadoop/Pig/Hive hands down is it’s not a massive stack of quirky DSLs built on top of layers of clunky Java abstractions - it’s a simple, pure Scala functional DSL with all the flexibility and succinctness of Scala. [Read More]
spark  scala  elasticsearch  big data 

Migrating from GoDaddy to Amazon Route53

 Posted on September 11, 2012  |  Barnaby Gray

Due to the recent outage on GoDaddy a lot of people are reconsidering their DNS options. Amazon Route53 is a great option - cheap, flexible and well proven. To migrate you first need to export a zone file for the domain from GoDaddy. It’s been highlighted the zone files are slightly broken in CNAME records, so you may need run this fix over them: $ perl -pe 's/(CNAME .+)(?!.)$/$1./i' broken.txt > fixed. [Read More]
dns  route53  cli53  aws 

A duplicate classifier using elasticsearch

 Posted on August 20, 2012  |  Barnaby Gray

Frequently with search based, big data projects the problem of content duplication is an obstacle to having a clean data source. Here’s an approach to improving the data quality by training a classifier to spot duplicates. The Problem The data set has about 470,000 non-unique hotel descriptions (e.g. name, metadata, images) provided from 11 bed bank services. These are received in various different feed formats, and in this data is hotel name, resort name, address, country, description and image data, among other data. [Read More]
elasticsearch  machine-learning 

iboto screencast

 Posted on April 6, 2012  |  Barnaby Gray

Made a screencast (my first!) of iboto to give a demo of the Amazon EC2 multi-account coolness:

Incidentally, making screencasts on Linux was a bit of a slog until I found the right tools and workarounds, so that might make the subject of a blog post of its own.

Route 53 latency based routing

 Posted on March 27, 2012  |  Barnaby Gray

Amazon have launched a neat new Route 53 feature: latency-based routing. The idea behind this is when someone hits www.yoursite.com this resolves to the closest server to them, cutting latency. This DNS cleverness has been used by the big boys for some time, but not been available to us mortals without shelling out big bucks to someone like neustar/ultradns (shudder). The ’location’ of your server is determined by multiple DNS records for a given lookup, each with an EC2 region attached to them (us-east-1, eu-west-1, etc. [Read More]
dns  route53  cli53  aws 

Hosting a jekyll blog on Amazon S3

 Posted on March 26, 2012  |  Barnaby Gray

** Note: I’ve replaced jekyll with the equally adapt pelican now. ** This article describes how to host your own static blog/site on S3. It revolves around the evolution this site has taken. First off I started using github’s public site feature. Dead neat, nice set of features and so quick to get running. The problem is I’m impatient, and often after pushing an update the “Page build successful” can take upwards of 30 minutes. [Read More]
jekyll  s3