Building a GeoJSON travel log: an introduction to Org mode and Babel

Literate programming is a technique that caught my attention after recently stumbling upon Howard Abrams’ ‘Literate Devops with Emacs’ video.The intersection of watching this awesome video, reading about GeoJSON rendering on GitHub and returning from a road trip last summer led me to building my own travel log. In my first blog post I would like to show you how it works.

Creating a travel log means entering a lot of data like dates and locations; a job that can be tedious without a user-friendly interface. It shouldn’t be too hard to beat exisiting online travel log services, such as the well-known Dutch website ‘WaarBenJij.nu’, in this aspect. Besides, what happens to your data when the online travel log company goes bankrupt? These two shortcomings are easily addressed with common programmer tools like Emacs’ Org mode and distributed version control systems like Git. While most programmers are familiar with the features of Git and GitHub, those of Org mode are less-known:

“Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.” — http://orgmode.org

In addition to Org mode, we use the Babel extension to execute source code in various languages (in this case just Shell and Ruby) in between the blog post paragraphs. By using these blocks exclusively, we will create all the code necessary to: geocode the locations in the travel log to coordinates (and install a library that helps us do this), convert the travel log to a GeoJSON file, commit and push it into a new repository, and open GitHub in a browser at the right URL. In fact, you can copy & paste this blog post into Emacs, enter org-mode and execute everything (org-babel-execute-buffer) to reproduce my steps.

Enough talking, let’s build this thing! We begin by defining the travel log, and ‘store it in a variable’ called travel-log.

#+NAME: travel-log
#+RESULTS:
| Date             | Location                  |
|------------------+---------------------------|
| <2015-08-10 Mon> | Utrecht, The Netherlands  |
| <2015-08-10 Mon> | Kožná, Prague             |
| <2015-08-12 Wed> | Prenzlauer Berg, Berlin   |
| <2015-08-13 Thu> | A&O Hamburg City, Hamburg |
| <2015-08-14 Fri> | Utrecht, The Netherlands  |

The brackets (< and >) around the dates indicate an Org mode timestamp. We can easily add and manipulate dates by using the datepicker (C-c ., which, for those unfamiliar with Emacs, means pressing the Control and C keys simultaneously before hitting the dot) and use TAB to move through the table: a user-friendly interface.

Emacs Org mode datepicker

In order to geocode the location names to coordinates we will use the geocoder Ruby gem. A shell source code block is an excellent way to install it, most importantly because the output displays the version that I used while writing the blog post, which improves reproducibility.

#+HEADER: :results output
#+BEGIN_SRC sh

gem install geocoder

#+END_SRC

#+RESULTS:
: Successfully installed geocoder-1.2.11
: Parsing documentation for geocoder-1.2.11
: Done installing documentation for geocoder after 1 seconds
: 1 gem installed

Now we will geocode the locations using the gem above. We don’t want to get rate-limited by the Google Maps API, so that’s why we create the geolocation-cache table with the distinct locations and their coordinates. For instance, Utrecht is listed twice in the travel log but only geocoded once. By the way, the #+HEADER: and #+BEGIN_SRC lines are instructions to Babel. I included them to enhance reproducibility.

#+HEADER: :var travel_log=travel-log
#+BEGIN_SRC ruby

locations = travel_log.map do |entry|
  # We only need the second column: the location
  _, location = entry
  location
end

distinct_locations = locations.uniq

require 'geocoder'
distinct_locations.map do |location|
  geo = Geocoder.search location
  coordinates = geo.first.geometry['location']

  [location, coordinates['lng'], coordinates['lat']]
end
#+END_SRC

#+NAME: geolocation-cache
#+RESULTS:
| Utrecht, The Netherlands  |  5.1214201 | 52.09073739999999 |
| Kožná, Prague             | 14.4213456 |        50.0862754 |
| Prenzlauer Berg, Berlin   |   13.44009 |          52.54114 |
| A&O Hamburg City, Hamburg |  9.9936818 |        53.5510846 |

Before we move on to the GeoJSON conversion, we have to specify a path where we can save the file.

#+NAME: geojson-file-path
#+RESULTS:
: /tmp/example_travel_log/my_trip.geojson

The following source block joins the travel-log with the geolocation-cache, builds a GeoJSON-formatted structure and saves it to the geojson-file-path.

#+HEADER: :results silent
#+HEADER: :var geojson_file_path=geojson-file-path
#+HEADER: :var geolocation_cache=geolocation-cache
#+HEADER: :var travel_log=travel-log
#+BEGIN_SRC ruby

coordinates = Hash[geolocation_cache.map do |entry|
  location, longitude, latitude = entry
  [location, [longitude, latitude]]
end]

require 'date'
geojson_features = []

# We use each_cons so we can draw lines between the locations
travel_log << nil
travel_log.each_cons(2) do |entry, next_entry|
  org_date, location = entry

  date = Date.parse org_date

  geojson_features << {
    type: 'Feature',
    geometry: {
      type: 'Point',
      coordinates: coordinates[location]
    },
    properties: {
      Location: location,
      Date: date
    }
  }

  next unless next_entry
  _, next_loc = next_entry
  geojson_features << {
    type: 'Feature',
    geometry: {
      type: 'LineString',
      coordinates: [coordinates[location], coordinates[next_loc]]
    }
  }
end

require 'fileutils'
repository_dir = File.dirname geojson_file_path
FileUtils.mkdir_p repository_dir

require 'json'
open(geojson_file_path, 'w') do |file|
  file.write JSON.pretty_generate(
    type: 'FeatureCollection',
    features: geojson_features
  )
end
#+END_SRC

Let’s verify the contents of the newly created file before we move on.

#+HEADER: :results output
#+HEADER: :var GEOJSON_FILE_PATH=geojson-file-path
#+BEGIN_SRC sh
head -n 33 '' $GEOJSON_FILE_PATH
echo etc...
#+END_SRC

#+RESULTS:
#+begin_example
==> /tmp/example_travel_log/my_trip.geojson <==
{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          5.1214201,
          52.09073739999999
        ]
      },
      "properties": {
        "Location": "Utrecht, The Netherlands",
        "Date": "2015-08-10"
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            5.1214201,
            52.09073739999999
          ],
          [
            14.4213456,
            50.0862754
          ]
        ]
      }
    },
etc...
#+end_example

The date, location and coordinates seem to match with the travel log we specified earlier. Let’s create a repository, commit, push and open GitHub to check it out!

#+HEADER: :results output silent
#+HEADER: :var GEOJSON_FILE_PATH=geojson-file-path
#+BEGIN_SRC sh
cd "$(dirname $GEOJSON_FILE_PATH)"

FILENAME="$(basename $GEOJSON_FILE_PATH)"

brew install hub
hub init
hub create
hub add $FILENAME
hub commit -m 'Update travel_log' $FILENAME
hub push origin master

GH_PATH_ROOT="$(hub remote -v | grep fetch | grep -oE '\w+\/\w+')"
hub browse "$GH_PATH_ROOT/blob/master/$FILENAME"
#+END_SRC

*Safari opens…*

It works; all the destinations from the travel log show up on the map with lines connecting them. Whenever we change the travel log or the code we can just rerun all the source code blocks and check out the new result; no slow switches between editors and command lines. All in all, the consequential higher speed of development and improved transparency of the program will most definitely make me pick Org mode and Babel for future projects.

For a more feature complete implementation that might actually be useful for real-life travel logging, please check out: https://github.com/pepijn/travel_log. Manipulating it (adding a destination) looks like this:

Happy hacking!