RethinkDB and CoreOS: Navigating Digital Ocean Together posted on 03 October 2014

A new generation of databases are now sailing on new seas such as Digital Ocean, transported in Docker containers, and steered by CoreOS.

CoreOS is a terrific tool to deploy applications on multiple servers. However, running a RethinkDB cluster on CoreOS is a bit more complicated than running multiple Nginx servers with a load balancer since a RethinkDB instance must be given at least one server to join. This article illustrates one way, hopefully the right way, to do it on Digital Ocean.

First, set up a fleet (or sub-fleet) of CoreOS machines; in this example, we will suppose that we are building a 6-server RethinkDB cluster. If you have never done it before, begin by getting a discovery token on

Then boot a few CoreOS instances with the following cloud-config file and your ssh keys. Make sure that you enable private networking on Digital Ocean as RethinkDB does not provide encryption/security for cluster traffic yet.


    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
    public-ip: $private_ipv4   # used for fleetctl ssh command
    metadata: group=rethinkdb
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

Note that these machines will be tagged with the metadata group=rethinkdb. I personally appreciate being able to group my servers depending on what their responsibilities are (or will be).

To create a RethinkDB cluster, we need to start the instances with the argument --join host:port where another instance of RethinkDB will be running. We will first create a discovery service where CoreOS servers will register their IP in etcd using etcdctl; we will force RethinkDB to run on these servers and provide them with all the IP addresses of the servers running the discovery service.

First, let's create a file rethinkdb-discovery@.service on one of your servers with the following content:

Description=Announce RethinkDB@%i service

ExecStart=/bin/sh -c "while true; do etcdctl set /announce/services/rethinkdb%i ${COREOS_PRIVATE_IPV4} --ttl 60; sleep 45; done"
ExecStop=/usr/bin/etcdctl rm /announce/services/rethinkdb%i


Then load the service with:

fleetctl submit rethinkdb-discovery@.service

Finally, start it with:

fleetctl start rethinkdb-discovery@{1..6}.service

Now create the service for RethinkDB:

Description=RethinkDB@%i service

ExecStartPre=-/usr/bin/docker kill rethinkdb%i
ExecStartPre=-/usr/bin/docker rm rethinkdb%i
ExecStartPre=-/usr/bin/mkdir -p /home/core/docker-volumes/rethinkdb
ExecStartPre=/usr/bin/docker pull dockerfile/rethinkdb
ExecStart=/bin/sh -c '/usr/bin/docker run --name rethinkdb%i   \
    -p ${COREOS_PRIVATE_IPV4}:8080:8080                        \
    -p ${COREOS_PRIVATE_IPV4}:28015:28015                      \
    -p ${COREOS_PRIVATE_IPV4}:29015:29015                      \
    -v /home/core/docker-volumes/rethinkdb/:/data/             \
    dockerfile/rethinkdb rethinkdb --bind all                  \
    --canonical-address ${COREOS_PRIVATE_IPV4}                 \
    $(/usr/bin/etcdctl ls /announce/services |                 \
        xargs -I {} /usr/bin/etcdctl get {} |                  \
        sed s/^/"--join "/ | sed s/$/":29015"/ |               \
        tr "\n" " ")'

ExecStop=/usr/bin/docker stop rethinkdb%i


The service is first going to fetch data from etcd and then start a Docker container with RethinkDB with the --join argument.

Note: Because you are running RethinkDB inside a container, you must provide the argument canonical-address or other instances will try to connect to the wrong IP address.


fleetctl start rethinkdb@{1..6}.service

And it's done! You now have a cluster of six machines running RethinkDB.

When RethinkDB provides auto-failover, in the event of a server failure, if you happen to have an extra CoreOS server, CoreOS will restart another RethinkDB instance and RethinkDB will automatically re-elect a master/backfill to prepare another replica without requiring any work on your end. Heavy refactoring of the clustering is being done right now, so hopefully this feature should ship soon (~2 months?).

Two things to finish this article:

  1. Thanks to @atnnn for helping me with some bash issues and Jessie for proofreading my Frenglish.
  2. Questions? Suggestions? Ping me on Twitter: @neumino.

Edit: dividuum on Hacker News pointed out that the private networking on Digital Ocean was not restricting other droplets from connecting to the cluster. I will follow up with another post to run some iptables commands to make sure that the cluster is safe.

Docker container for Firefox OS posted on 14 July 2014

I recently got a Flame, the developer reference phone for Firefox OS.

I created a Docker container to build Firefox OS, mostly because I didn't feel like installing Java 6 on my system (since it is not supported anymore). This post is about how to build such the container.

If you are interested in the image, you can find it on the Docker's hub once this issue will be solved.

Start it with

sudo docker run -t -i --privileged --expose 5037 -v /dev/bus/usb:/dev/bus/usb -v /host/data:/container/data ubuntu-firefoxos /bin/bash

Steps to create the container:

Start a Ubuntu container.

sudo docker run -t -i ubuntu:14.04 /bin/bash

Update the system.

apt-get update
apt-get upgrade

Install the dependencies as described in the docs.

dpkg --add-architecture i386
apt-get update
apt-get install --no-install-recommends autoconf2.13 bison bzip2 ccache curl flex gawk gcc g++ g++-multilib gcc-4.6 g++-4.6 g++-4.6-multilib git lib32ncurses5-dev lib32z1-dev zlib1g:amd64 zlib1g-dev:amd64 zlib1g:i386 zlib1g-dev:i386 libgl1-mesa-dev libx11-dev make zip libxml2-utils
apt-get install python
apt-get install android-tools-adb
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 1 
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 2 
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.6 1 
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 2 
update-alternatives --set gcc "/usr/bin/gcc-4.6" 
update-alternatives --set g++ "/usr/bin/g++-4.6" 

Note: All the instructions below are what I did to be able to build the OS for the Flame. Some steps may not be required for another phone (and some may be missing).

You need to specify some credential for git.

git config --global <your_email>
git config --global <your_name>

Then you have to install a few more things to be able to build, with first Java.

add-apt-repository ppa:webupd8team/java
apt-get update
install oracle-java6-installer

Building Firefox OS requires you to pull some blobs from your phone with adb.

apt-get install android-tools-adb
apt-get install libusb-1.0-0 libusb-1.0-0-dev
apt-get install usbutils # This may not be needed, I used it to debug a few things

Install a few more packages required by the build process.

apt-get install dosfstools libxrender1 libasound2 libatk1.0 libice6

Export a SHELL variable.

export SHELL=/bin/bash

Install unzip.

apt-get install unzip

Get your container id with:

sudo docker ps -a

Commit your changes.

sudo docker commit <container_id> ubuntu-firefoxos

Stop and remove the container.

sudo docker stop <container_id>
sudo docker rm <container_id>

Restart the container with a few more flags.

sudo docker run -t -i --privileged --expose 5037 -v /dev/bus/usb:/dev/bus/usb -v /host/data:/container/data ubuntu-firefoxos /bin/bash

The --privileged --expose 5037 -v /dev/bus/usb:/dev/bus/usb options are required for adb to be able to find your device.

Before building, make sure you enable the remote debugging mode on your phone.

Open the Settings app, then Device Information > More Information > Developer.
In the developer menu, check "Remote debugging".

Then you are good to go:

cd /container/data/B2G
./ flame

Rethinkdbdash for Node.js 0.10.26 posted on 29 March 2014

I just released two packages of rethinkdbdash

  • rethinkdbdash for Node.js 0.10.26
  • rethinkdbdash-unstable for Node.js 0.11.10 (and 0.11.9)

I wrote rethinkdbdash two months ago to improve the syntax of ReQL in the Node.js driver by providing

  • promises (and testing them with generators)
  • a native/automatic connection pool

While you cannot use generators with the stable version of Node.js, the connection pool is a reason good enough to make this driver available for the stable Node.js. You basically never have to deal with connections.

For those who want to know what the syntax looks like, here it is:

var r = require('rethinkdbdash')();

r.table("comments").get("eef5fa0c").run().then(function(result) {
}).error(function(error) {

Compared to the one with the official driver:

var r = require('rethinkdb');

r.connect({}, function(error, connection) {
    r.table("comments").get("eef5fa0c").run(function(error, result) {
        if (err) {
        else {

Note: If you were using rethinkdbdash with Node 0.11.10, please switch to rethinkdbdash-unstable.

First experience on Digital Ocean - Updating Archlinux posted on 14 March 2014

I have been using a dedicated server at OVH for a few years now, and the quality of their service has become worse and the last incidents prompted me to look for a new server.
Digital Ocean claims that they are user-friendly and since it is quite cheap, I just gave it a try.

Subscribing, setting up 2 factor authentification, starting a droplet was a blast. I picked Archlinux, and less than one minute after, my droplet was up and running.

The Arch image is quite old (June 2013) and updating the system is a little more tricky than just running pacman -Syu.
These instructions were written a few hours after the installation, so they may be slightly inaccurate.

First, update the whole system. Because Arch merged /bin, /sbin into /usr/bin and /lib into /usr/lib, you cannot just run pacman -Syu. Run instead:

pacman -Syu --ignore filesystem,bash
pacman -S bash
pacman -Su

Then remove netcfg and install netctl.

pacman -R netcfg
pacman -S netctl

Run ip addr to see your interface. In my case it was enp0s3

Create a config file /etc/netctl/enp0s3 with

DNS=('', '')

Enable the interface

netctl enable enp0s3

Then update the kernel via the web interface.

The network interface is going to change to something like ens3. Move /etc/netctl/enp0s3 to /etc/netctl/ens3 and change the Interface field.

Update /lib/systemd/system/sshd.service to be sure that the ssh daemon doesn't fail on boot

Description=OpenSSH Daemon

ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID


Reboot and your server should be up to date.

And that's it for updating Arch. It was not the easiest updates, but nothing impossible. It would have been nice if Digital Ocean was provided an up to date Arch image though.

Note: You can probably directly set the network interface to ens3.
In the worst case you can still access your machine with Digital Ocean's web shell and fix things there.

Archlinux on a OVH kimsufi server posted on 15 February 2014

The Archlinux distribution installed on OVH kimsufi servers come with a custom kernel.

While it is not super-old, it is still not up to date (at the time of writing, OVH is using 3.10.9 while Arch comes with 3,12.9).

Installing Arch from scratch seems to be doable, but is probably some work. Another way to have a recent kernel is just to use OVH template and then swap the kernel. For that you just need to:

  • Install Arch from the web interface
  • Update /etc/pacman.d/mirrorlist with normal values (look at mirrorlist.pacorig)
  • Generate a new grub conf with
sudo grub-mkconfig -o /boot/grub/grub.cfg
  • Make sure that the entry with the normal kernel in /boot/grub/grub.cfg is the first one.
  • Reboot

Then you get:

michel@ks******:~$ uname -r

Paving the way for the new Nodejs posted on 27 January 2014

ReQL, the RethinkDB Query Language, embeds into your programming language, making it pleasant to write queries.
The JavaScript driver is a great implementation of ReQL, but it forces users to deal with callbacks making the code cumbersome.

Recently Nodejs announced that the next stable release, 0.12.0, is imminent. The biggest feature in Nodejs 0.12.0 most developers are looking forward to is generators. Since generators remove the need for cumbersome callback code, I decided to take the opportunity to write a new callback-free RethinkDB driver from scratch.

I wrote this driver taking into consideration what people were building with the current one.

  • Promises
    A few projects wrapped the RethinkDB driver with different libraries -- rql-promise, reql-then
    The new driver works with promises natively.

  • Connection pool
    Many ORMs (reheat, thinky, etc.) and connectors (sweets-nougat, waterline-rethinkdb, etc.) are implementing their own connection pools on top of the driver.
    The new driver provides a native connection pool without having to manually acquire and release a connection.

  • More accessible
    The current JavaScript driver is buried in RethinkDB main repository, is written in CoffeeScript and its tests are designed to run for all official drivers. In the end, it is hard to contribute to the driver.
    The new driver has its own repository, is written in full JavaScript, and tests are run with mocha (and on wercker).

The result of all these considerations is rethinkdbdash, which is now, as far as I know, feature complete and working.

How is rethinkdash better than the official JavaScript driver? Let's look at a concrete example.

Suppose you have two tables with the following schemas:

// Table posts
  id: String,
  title: String,
  content: String

// Table comments
  id: String,
  idPost: String, // id of the post
  comment: String

And suppose you want to fetch all the posts of your blog with their comments and retrieve them with the following format:

// Results
  id: String,
  title: String,
  content: String,
  comments: [ {id: String, idPost: String, comment: String}, ... ]

With ReQL you can retrieve everything in one query using a subquery[1].

This is how you currently do it with the old driver.

var r = require("rethinkdb");
// Code for Express goes here...

function getPosts(req, res) {
  r.connect({}, function(error, connection) {
    if (error) return handleError(error);

    r.db("blog").table("posts").map(function(post) {
      return post.merge({
        comments: r.db("blog").table("comments").filter({idPost: post("id")}).coerceTo("array")
    }).run(connection, function(error, cursor) {
      if (error) return handleError(error);

      cursor.toArray(function(error, results) {
        if (error) return handleError(error);


Now look how wonderful the code looks with new driver.

var r = require("rethinkdbdash")();
// Code for Koa goes here...

function* getPosts() {
    var cursor = yield r.db("blog").table("posts").map(function(post) {
      return post.merge({
        comments: r.db("blog").table("comments").filter({idPost: post("id")}).coerceTo("array")
    var results = yield cursor.toArray();
    this.body = JSON.stringify(results);
  catch(error) {
    return handleError(error);

What is awesome here, is:

  • There are no callbacks.
  • You do not need to open/provide/close a connection.

Take a look at the usual Todo example built with AngularJS, Koa end Rethinkdbdash.

We have to wait for the stable release of Node before the new driver can become mainstream. In the meantime, you can build Nodejs 0.11.10 from source to play with rethinkdbdash[2].
Once people use it for a bit and it is better tested, it should become the official driver.

Feedback, pull requests and comments are welcome!

[1] The anonymous function in map is parsed and send to the server.
Read all about lambda functions in RethinkDB queries if that piques your interest.

[2] You need to build from source or you may get errors like

node: symbol lookup error: /some/path/protobuf.node: undefined symbol: _ZN4node6Buffer3NewEm

Quick hack - Sound volume on Leapcast posted on 05 January 2014

Leapcast currently does let you change the sound volume with Pandora (see the source for CastPlatform).

I did a quick/dirty hack to change the sound volume on my computer with the pactl command. That works only if you are running pulseaudio (tested only with Linux for now).

This commit let you change the sound volume (if you are running pulseaudio).

You may have to set the --pulse argument to make it work. Run pacmd list to find your running pulseaudio server.
You can provide the index (an integer) or the name of the server (like alsa_output.pci-0000_00_1b.0.analog-stereo).