Getting Started with the Crux builder

Generating custom deployment artefacts

Getting Started with the Crux builder

Published
July 23, 2020
by
Daniel Mason
Categories
Crux is now XTDB

September 2021 Update: Crux has recently been renamed XTDB. This post still refers to XTDB as "Crux" but you will find newer versions of the Builder (the topic of this post) in the most recent version of XTDB.

A completely updated version of this post can be found at https://xtdb.com/blog/xtdb-build.html.

The official home for XTDB is now https://xtdb.com.

Spinning up a new Crux node is generally a case of including the relevant Crux dependencies within your project, defining a 'topology' configuration, and then making a call to start-node with that configuration. This is not a particularly difficult process but it still requires you to dip into Clojure or Java before you can get something running.

We created the crux-build tool to bypass this process and make getting started much simpler for non-Clojure users.

crux-build was initially released with Crux in 20.02-1.7.0-alpha. It offers a small range of 'batteries included' JARs and Docker containers to help you get a basic Crux implementation running, and an easy mechanism to generate your own custom Crux deployment artifacts.

Batteries included - crux-in-memory

Available on both:

This starts up a bare-bones Crux implementation, utilising three Crux modules:

  • crux-core - required by all nodes.

  • crux-http-server - each configuration starts an HTTP server (on port 3000).

  • crux-sql - starts a SQL server (on port 1501).

Docker Hub page there is a basic tutorial for getting started with the Crux REST API, and you can find out more in the docs. For communicating with the node using SQL, see the crux-sql[https://xtdb.com/reference/21.01-1.14.0/sql.html#_connecting] docs.

crux-in-memory is intended for users to get started with Crux, without requiring any prior Java/Clojure knowledge. It contains an in-memory node that does not persist any data across restarts.

We plan to provide further 'batteries included' implementations in the future (suggestions are welcome!), but let’s now take a look at how to generate custom Crux deployment artifacts.

Roll your own Crux deployment artifacts

Alongside the JARs deployed on the GitHub releases is crux-builder.tar.gz - using the scripts in this archive, we can customise and build a JAR or a Docker container.

Building a JAR (Clojure CLI tooling):

The clj-uberjar folder contains a number of files:

  • a deps.edn file to configure the Maven dependencies required

  • a crux.edn file to configure the node itself

  • a resources/logback.xml to configure logging output

  • a build-uberjar.sh script to build the JAR.

In this example, let’s add rocksdb as the KV store, storing the indexes in a folder called 'crux-indexes', and the standalone tx-log/document-store in crux-event-log. First, we add juxt/crux-rocksdb as a dependency in deps.edn.

...
juxt/crux-rocksdb {:mvn/version "21.01-1.14.0-beta"}
...

Then, in crux.edn, use this new topology:

{:crux/index-store {:kv-store {:crux/module crux.rocksdb/->kv-store, :db-dir "/tmp/crux/indexes"}}
 :crux/document-store {:kv-store {:crux/module crux.rocksdb/->kv-store, :db-dir "/tmp/crux/documents"}}
 :crux/tx-log {:kv-store {:crux/module crux.rocksdb/->kv-store, :db-dir "/tmp/crux/tx-log"}}
 :crux.http-server/server {}}

To build the JAR, run the build-uberjar.sh script. You can optionally pass the environment variable UBERJAR_NAME to the script (for example, UBERJAR_NAME=crux-rocks.jar ./build-uberjar.sh), otherwise the built uberjar will be called crux.jar.

To run the clojure uberjar, use the following command: java -jar crux.jar. This now starts up a node with both a HTTP server and persistent storage.

Building a JAR (Maven tooling):

Similarly to building a JAR using the Clojure CLI tooling, we can also build an uberjar using Maven. In the mvn-uberjar directory, we can add dependencies to the pom.xml file, update the crux.edn file as before, and then run build-uberjar.sh to create the uberjar. To run the maven generated uberjar, use the following command: java -jar crux.jar.

Building a Docker Container:

In the docker directory, there are a similar set of files to the uberjar examples above, as well as a Dockerfile and a build-docker.sh script. As with building a JAR, we can add rocksdb as the KV store - first, by adding a dependency on juxt/crux-rocksdb to deps.edn Default index and event-log directories are already configured within the sample crux.edn (allowing you to map the Docker directories and give the images persistent storage), we just need to add crux.kv.rocksdb/kv-store to the :crux.node/topology list.

{:crux/index-store {:kv-store {:crux/module crux.rocksdb/->kv-store, :db-dir "/var/lib/crux/indexes"}}
 :crux/document-store {:kv-store {:crux/module crux.rocksdb/->kv-store, :db-dir "/var/lib/crux/documents"}}
 :crux/tx-log {:kv-store {:crux/module crux.rocksdb/->kv-store, :db-dir "/var/lib/crux/tx-log"}}
 :crux.http-server/server {}}

To build your Docker container, run the build-docker.sh script. You can optionally pass an IMAGE_NAME and IMAGE_VERSION to tag the container with (by default, the custom Docker container is called crux-custom:latest).

In Summary

This is only scratching the surface of different setups that Crux’s unbundled nature allows - for more information, check out the configuration section of the docs.

Get busy building!

As always, feel free to reach out to us on Zulip, the #xtdb channel on the Clojurians' Slack or via hello@xtdb.com.