Sync Kobo highlights and annotations with other devices

I hope this post will save you the hours I lost, this is an attempt (effective) to syncronize a Kobo with Linux over wifi¹.

For a long time, I wanted a solution to sync the metadata on my EReader with my computers, here were my requirements:

• Support the following formats: PDF, EPUB, and DJVU.
• Decentralised.
• Lightweight, mostly regarding resources consumption, I want it to run on a mini laptop.
• I don’t want anything stored on a database just local metadata, possibly everything stored inside the files themselfs.
• Not Calibre: its not lightweight, it enforces a rigid hierarchy (based on authors), and it stores metadata on a database.²
• Bi-directional sync: I want to be able to add a note from my workstation, and get it back when I open the document on my EReader.
• I also want it to be reader agnostic, I want to open an EPUB on any software and get my highlights.
• Manage conflicts.
• Works over Wifi.

What KoReader got wrong… was right

I use KoReader on my Kobo, it store metadata in a lua file inside a specific directory (one per file). For example, as soon as you open a document, it will create a .sdr directory, inside of which there will be a metadata.lua file. This file contain all the document metadata (like date of modification, progression, highlights, notes, and a bunch of other metrics).

At first I’ve seen this as a bug, because why not just insert the metadata inside the files themselves?

But I was being naive, the answer is, its impossible…

The EPUB standard does not support highlights or notes

Yes you read it right, it doesn’t³

PDF and DJVU have this, but there seem to be many way to do it, and so it ends up being dependant on the software you use.

So given all that, it understand why the people behind KoReader have opted for doing things that way. It has also two main advantages:

• Format independance
• Does not alter the original file (that may be useful for hash based search).

What I ended up doing

I resigned myself to use KoReader on my Workstation, this way I would attain my needs just by synchronizing the directories containing the metadata.

KoReader is available on a variety of platforms, in my case there was a Debian package. Its optimized for Eink screens, but it does the job on other supports.

I evaluated different solutions, at first I tried using Rrsync over FSTP, it worked, but I wasn’t satisfied because it is not bi-directional, which means it will invariably create conflicts when you’re editing on both sides.

Then I tried to run Syncthing on my kobo, and it worked!

How to run syncthing on kobo

Ok. So first you obviously need to have KoReader installed (It’s just a file to copy to the Kobo).

And obviously syncthing should be installed on the device you want to sync with (I won’t cover its installation and uses here, but it’s quitte user friendly).

Then download syncthing for arm here (it’s name should be something like syncthing-linux-arm-*.tar.gz).

Connect to your kobo over ssh

In the KoReader menu, select Network and activate the Wifi

• Activate the ssh plugin.

• Run the ssh server.

• Connect to it from your other device:

  For some reason I needed to specify the full path to my OpenSSH binary, for example:

  /bin/ssh root@192.168.1.20 -p 2222

  You can find the Kobo IP in the network settings.

• Copy the syncthing binary to /usr/bin or /usr/sbin and set the permissions like others files in this dir (root user with x).

Run syncthing

Connect with SSH and run it as root (I have not tried setting a specific user) like this:

syncthing --no-browser

Now, you should be able to visit the WebUI of your Kobo (port 8384 by default for me) on your local network.

At that point the sync process is a formality if you know how to use syncthing.

Be careful: if you delete something on one side, it will be deleted on the other, I recommend to do some tests on a dummy directory beforehand.

My workflow

I personaly don’t sync the EPUBs or PDFs themselves, because I have too many files and I don’t need all of them in the Kobo, so I only sync the metadata by restricting syncthing to the .sdr files.

To do that, add this to Ignore Patterns (order matters):

!*.sdr
*

Some drawbacks

• It seems that syncthing fail to use inotify on the Kobo, the consequence is that you have to manually clic on the “Rescan” button on the web interface to make it detect changes.

  I haven’t looked for a solution yet, because I view this as a minor inconvenience, and I sync maybe once a week max.

• I have to connect over SSH to run the executable, I didn’t find a way to run it from KoReader, maybe this is possible, again that’s not really bothering me for now.

• The security can be improved by:

  • Using an SSH keypair: I failed to add the key to the SSH client (it’s not OpenSSH), but it’s probably possible.
  • By adding a user specifically for Syncthing.

Footnotes