Crux Development Diary

Getting into Beta

Crux Development Diary

October 24, 2019
Jon Pither

Hot on the heels of our last Crux dev diary we have published crux-jdbc, facilitating users to quickly get going without committing to a Kafka deployment model. As wrote about in the diary, the JDBC form of Crux is aimed at that sweet spot, where users will already likely have a relational data-store setup in place, and the volumes of data may not be high enough to warrant the up-front Kafka investment.

This said, we’re still loving Kafka - Confluent’s managed cloud solution is worth considering as a good stepping stone, or even an end solution.

Kafka gives you a more authentic separation of concerns that you need with the database inside out approach, to handle vast amounts of data at scale. It is much more than just a message bus used for systems integration - it can be a golden store record of truth that is highly available and easily replayable.

To that end, the Crux team has been cooking up a Crux Kafka Connector for ease of plugging Crux into an existing Kafka installation and for data to be streamed in to produce subsequent bitemporal graph queries. More details to follow.

Community / Unbundled

It’s been great to see so many people playing around with Crux and the odd case-study emerging.

Crux is unbundled and open source, and so we always hoped that people would start building out their own modules and different ways of using it. So far, we’ve seen Crux built to use Active Objects as the event-log (mirroring the recent JDBC effort), Xodus as the node-local K/V store, and there have been experiments such as running on top of AWS SQS and using Pathom/Fulcro. There have been many other conversations about what underlying tools and platforms could support a running Crux setup and so hopefully more community contributions will continue to blossom out.

To help with this - as part of the most recent release - we have changed the API and have introduced the concept of node topologies so that Crux is more easily configurable.

Before, the api had functions such as start-jdbc-node, but now there’s a single start-node entry function that takes an options map, one of those options being crux.node/topology. Officially supported topologies are currently:

  • crux.kafka/topology

  • crux.jdbc/topology

  • crux.standalone/topology.

A topology specifies a map of modules that are needed to construct a topology. A module can now specify what options it takes, the default values, validation functions, and parse functions, if Crux is to be run from the CLI. Configuration options can also be used to override modules within a topology.

Our hope is that this configuration approach makes it easier to extend Crux and to play about with new variants of Crux topologies, embracing the unbundled approach.

Below is an example of a properties file for a Kafka node:

crux.node/topology = crux.kafka/topology
crux.node/kv-store = crux.kv.rocksdb/kv
crux.standalone/event-log-dir = data/eventlog
crux.kv/db-dir = data/db-dir
crux.kafka/bootstrap-servers = localhost:9092
crux.kafka/tx-topic = crux-transaction-log
crux.kafka/doc-topic = crux-docs
crux.kafka/create-topics = true
crux.kafka/doc-partitions = 6
crux.kafka/replication-factor = 3

On Tour

We’ve gone on tour to talk about Crux. Håkan gave a talk at ClojuTRE on the Design and Implementaton of a Bitemporal Database. The last time he had visited Helsinki was for Assembly94.

Håkan Råberg talking about the internals of Crux
Håkan Råberg talking about the internals of Crux

Johanna gave a Crux tutorial at ClojuTRE, gracefully added to the schedule by our friends at Metosin.

Jeremy and I talked about Event Streaming Architectures and Temporal Databases at Strange Loop. Jeremy also ran an unsession that was well attended.

Jeremy at Strange Loop 2019 talking about Crux
Jeremy at Strange Loop 2019 talking about Crux

The JUXT team have run training workshops at Heart of Clojure and we gave a talk to our friends at 'Scala in the City' hosted by Revolut at Canary Wharf.

Heart of Clojure Crux Walkthrough
Heart of Clojure Crux Walkthrough

It’s been a fun journey since launch, but our thoughts now are turning to the medium and long term future.

The Future

We are growing the Crux team with some exciting full-time additions and we are creating a new London based development center, that will be where future versions of Crux core will be built from. When we formally start the BETA program in the next 2-3 months, we will do so with a global support capability including North America, Europe and Australia.

We have been looking long and hard at the roadmap and we’ve got exciting plans for Crux and extending its range of functionality.

First though we will be entering a formal BETA program and we will inviting partners to join. As part of the BETA we want to see and understand how Crux is performing in more production installations and of what improvements we will need to make.

If you want to simply get started with Crux, checkout Joanna’s adventure of Time and Space or Ivan’s bitemporal tale.

Thanks for reading this journal. Come along and say hi:

Stay in Touch

We'll let you know when new blogs are updated along with the latest in JUXT news.

Thanks for signing up!
We have sent you a confirmation email
Oops! Something went wrong while submitting the form.