loads.pickle.me.uk

Open sourcing my home automation

Feb 18, 2015 homeautomation home golang

I have been meaning to open source my home automation system for a while now, and finally got around to cleaning it up a bit, documenting and removing the parts specific to my house (actually very little surprisingly). It’s the most Go code I’ve written, and I’m pretty happy with how it looks …

Elasticsearch as a smart cache

Feb 18, 2014 elasticsearch caching

A novel use of Elasticsearch in the context of holiday search, not as a traditional store of persistent documents, but as a rolling cache of transient holiday packages. This puts unique demands on Elasticsearch in terms of index and deletion rate, concurrent to a high query rate. Figures below are from February 2014, …

docker fish completion

Feb 8, 2014 docker fish shell completion

Command completion for docker in the fish shell I’ve just released a handy completion definition for the fish shell. fish? Fish whilst not as popular as zsh/bash is a far better shell. Make the switch ;-) install docker completion $ mkdir ~/.config/fish/completions $ wget …

Betting on the Twitter stream with Elasticsearch and Kibana

Jan 19, 2014 twitter sentiment kibana elasticsearch

Sentiment analysis, Elasticsearch and Kibana The idea From the Twitter streaming API grab tweets for a live TV show whilst it’s showing, classify by contestant, analyze sentiment and provide a real-time dashboard into the outlook. Try to predict good bets on the winners. Optional: Gamble. :-) The tools Grabbing …

Automatic EC2 architecture diagrams

Nov 28, 2013 cloudily ec2 graphviz devops ops

A picture is worth a thousand words - how about if you could visualize your EC2 infrastructure automagically? That’s what Cloudily does: It discovers connections between hosts using a variety of tools: arp, netstat and last. Find me on github or pypi.

Dynamic DNS for EC2 instances

Nov 23, 2013 aws ec2 route53 dns dyndns

The problem Booting instances on EC2 is easy. But once it comes to connecting to them, you end up having to copy around transient and unwieldy ec2-....compute.amazonaws.com host names. So you manually set up a CNAME in your DNS to give the instance a friendly name, but as soon as you stop and restart the instance, the …

Finding related searches with Spark

Nov 19, 2013 spark related searches co-occurrence machine learning scala

Unsupervised learning from user behaviour When a user navigates a site they leave a valuable trail of information - what their first search was, what they followed this search with, and so on. Using this data we can learn related searches automatically by co-occurrence counting. This post takes you through the steps to …

Scala Future and the Elasticsearch Java API

Nov 16, 2013 scala elasticsearch futures java api

Most examples of the Elasticsearch Java API you’ll have seen follow the prepare/execute/actionGet pattern (in Scala): val response = client.prepareIndex("twitter", "tweet", "1") .setSource(jsonBuilder() .startObject() .field("user", "kimchy") .field("postDate", new …

Spark and Elasticsearch

Nov 12, 2013 spark scala elasticsearch big data

Elastic Sparkle If you work in the Hadoop world and have not yet heard of Spark, drop everything and go check it out. It’s a really powerful, intuitive and fast map/reduce system (and some). Where it beats Hadoop/Pig/Hive hands down is it’s not a massive stack of quirky DSLs built on top of layers of clunky …

cli53

Apr 6, 2013

Command line script to administer the Amazon Route 53 DNS service. quick example: $ cli53 create example.com $ cli53 import example.com --file zonefile.txt $ cli53 rrcreate example.com www A 192.168.0.1 --ttl 3600 install: $ pip install cli53 Find me on github or pypi.