Fast select and ordering with RethinkDB posted on 17 June 2017

I often get questions about ReQL on how to select all the documents that match a given value and order them by another.

Solution with an example.

Suppose your schema for the table user is:

{
  id: <string>,
  country: <string>,
  score: <number>
}

And that you want all the users in the US ordered by they score. The quick way to do it is

r.table('user').indexCreate('country_score', function(user) {
  return [user('country'), user('score')]
})
r.table('user').between(
  ['US', r.minval],
  ['US', r.maxval],
  {index: 'country_score'}
).orderBy({index: 'country_score'})

Explanation

Let’s start with a bit of background on what an index is. When you build an index, you basically build a tree with the indexed value (and a pointer to your document). For example, you may have the following tree

                 ['FR', 7]
               /           \
       ['FR', 2]           ['US', 8]
         /  \                 /  \
 ['CA', 2]  ['FR', 5]  ['US', 2] ['ZA', 8]

When you request between(['US', r.minval], ['US', r.maxval], {index: 'country_score'}, you extract the subtree

     ['US', 8]
       /
['US', 2]

This subtree still represent a proper index on a subset of your table. If your table had only these two documents, this would be the tree representing your index. You can therefore reuse it to order your keys. Because you filtered the elements to have country to be US, you basically just order by the second field, that is to say score.

Why you can’t order by another index

I’m also asked why can’t we just have

r.table('score').getAll('US', {index: 'country'}).orderBy({index: 'score'})

The main reason is that getAll would return you a subtree where all the keys are US. Each node would also give you a pointer to the document saved in the database. But then the index score is a whole different tree where the keys are just the score, so you cannot extract a subtree with the retrieved documents without scanning the whole index first, which is more or less equivalent to selecting without an index.

Hopefully this helps a bit understanding what indexes are, how they are used and what are their limitations.


Docker fails to create endpoint posted on 24 December 2016

I ran into this issue recently while trying to get tensorflow running:

[email protected]:~$ docker run  -p 8888:8888 gcr.io/tensorflow/tensorflow
docker: Error response from daemon: failed to create endpoint high_raman on network bridge: failed to add the host (veth3c0e4fe) <=> sandbox (veth21faddb) pair interfaces: operation not supported.

Nothing looked suspicious in ip addr

[email protected]:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 8c:70:5a:ff:a4:08 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.71/24 brd 192.168.1.255 scope global dynamic wlp3s0
       valid_lft 79660sec preferred_lft 79660sec
    inet6 2602:306:ce40:a090:8e70:5aff:feff:a408/64 scope global noprefixroute dynamic 
       valid_lft 2591806sec preferred_lft 604606sec
    inet6 fe80::8e70:5aff:feff:a408/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:d3:0f:49:9b brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

Looking around, it seems that Archlinux needs to reboot for Docker to properly work. I didn’t look any further, but at least this did the trick for me.

Force update ttf-dejavu posted on 09 November 2016

On archlinux, ttf-dejavu 2.37 requires a forced upgrade. If you don’t, you will be greeted with this error:

[email protected]:~$ sudo pacman -S ttf-dejavu
[sudo] password for michel: 
resolving dependencies...
looking for conflicting packages...

Packages (1) ttf-dejavu-2.37-1

Total Installed Size:  9.75 MiB
Net Upgrade Size:      0.57 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                                              [########] 100%
(1/1) checking package integrity                                                            [########] 100%
(1/1) loading package files                                                                 [########] 100%
(1/1) checking for file conflicts                                                           [########] 100%
error: failed to commit transaction (conflicting files)
ttf-dejavu: /etc/fonts/conf.d/20-unhint-small-dejavu-sans-mono.conf exists in filesystem
ttf-dejavu: /etc/fonts/conf.d/20-unhint-small-dejavu-sans.conf exists in filesystem
ttf-dejavu: /etc/fonts/conf.d/20-unhint-small-dejavu-serif.conf exists in filesystem
ttf-dejavu: /etc/fonts/conf.d/57-dejavu-sans-mono.conf exists in filesystem
ttf-dejavu: /etc/fonts/conf.d/57-dejavu-sans.conf exists in filesystem
ttf-dejavu: /etc/fonts/conf.d/57-dejavu-serif.conf exists in filesystem
Errors occurred, no packages were upgraded.

Make sure to run pacman -f for this package.

[email protected]:~$ sudo pacman -S --force ttf-dejavu

Incomplete kernel update posted on 30 October 2016

These are quick notes about fixing my system after an incomplete kernel update.

During a system update, my workstation suddenly shut down (hardware failure). I was working on something else during the update, so I had no idea when the update was aborted. When I tried to boot again, I was greeted with this error:

Warning: /lib/modules/4.8.4-1-ARCH/modules.devname not found - ignoring.
starting version 231
ERROR: device 'UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' not found, Skipping fsck.
ERROR: Unable to find root device 'UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'.
You are being dropped to a recovery shell
    Type 'exit' to try and continue booting
sh: can't access tty: job control turned off
[rootfs ]#

At that time I thought that one of my harddrive died. Digging a bit into it, the partition that couldn’t be found was an encrypted one. So I thought that maybe my fstab got somehow corrupted. I didn’t have access to a shell after booting as the keyboard was not working, so I just booted with a live usb key. My disks are all encrypted, so the first step was to decrypt them:

Confirm that they are indeed crypted:

blkid | grep crypto

Decrypt/mount the partition:

cryptsetup luksOpen /dev/sdc2/ crypt

You then can’t just mount /dev/mapper/crypt. If you try, you’ll get:

mount /dev/mapper/crypthome /mnt/crypt
mount: unknown filesystem type 'LVM2_member'

Find the group of your volume (mine was main):

vgscan

Find the name of the volume (mine was root):

lvs

Mount it:

mount /dev/main/root /mnt

In my case, /boot is not encrypted, so I also had to mount it to be able to look at my kernel.

mount /dev/sdc1 /mnt/boot

Then I just used chrooted:

arch-chroot /mnt

I thought that the issue might have been in the initial ramdisk environment, so I just ran:

mkinitcpio -p linux

I was greeted with quite a few warnings that some modules were not found. But all the modules for encryption was present, so I decided to just reboot. Some progress were made: I could decrypt my disk, but was then greeted with:

mount: unknown filesystem type 'ext4'

I booted on the usb key again, and looked a bit at my modules. That’s when I found that uname -r was giving me an old kernel version. Trying to reinstall the linux package was returning an error because the desc file was missing. I guess my update was interrupted during the kernel update.

Removing the package and re-installing it did the trick. Note that pacman -S --force was not enougth to prevent pacman from complaining about the missing files.

pacman -R linux
pacman -Sy linux
mkinitcpio -p linux