Curious (Clojure) Programmer Simplicity matters

Menu

  • Home
  • Archives
  • Tags
  • About
  • My Talks
  • Clojure Tip of the Day Screencast
  • (Open) Source
  • Weekly Bits & Pieces
  • RSS
October 15, 2023

Liquibase: custom database migrations with Clojure (without AOT)

Table of Contents
  • Liquibase Intro
    • Custom migrations
  • Custom migrations in Clojure
    • AOT
    • Problems with AOT
    • Getting rid of AOT (deftype)
    • Classloading gotcha
  • Credits
  • References

Liquibase Intro

Liquibase is a well-know tool for tracking, versioning, and deploying database schema changes.

It uses changelog files to list database changes in the form of changesets (SQL, XML, YAML, JSON), which consists of Change Types. Standard changsets use SQL or a DB-agnostic equivalent written in XML, YAML, or JSON.

Here’s an example of a very simple changset adding a new table with a single column:

<changeSet  id="1"  author="nvoxland">
    <createTable  tableName="company">
        <column  name="address"  type="varchar(255)"/>
    </createTable>
</changeSet>

Custom migrations

Sometimes, there’s a more complicated migration that’s very difficult or impossible to express in SQL (or its Xml/YAML/JSON equivalent). That is, you need to write actual code to perform the migration. In that case, Liquibase offers customChange Change Type.

To implement a custom migration you need to:

  1. Create a Java class that implements the liquibase.change.custom.CustomSqlChange or liquibase.change.custom.CustomTaskChange interface (showing only a subset of methods here):

    public class ExampleCustomTaskChange implements CustomTaskChange, CustomTaskRollback {
        private String helloTo;
        @SuppressWarnings({"UnusedDeclaration", "FieldCanBeLocal"})
        private ResourceAccessor resourceAccessor;
    
        @Override
        public void execute(Database database) throws CustomChangeException {
            Scope.getCurrentScope().getLog(getClass()).info("Hello "+getHelloTo());
        }
        @Override
        public void rollback(Database database) throws CustomChangeException, RollbackImpossibleException {
            Scope.getCurrentScope().getLog(getClass()).info("Goodbye "+getHelloTo());
        }
        ...
    }
  2. Compile the created class, package it into a JAR file, and then add it to a Liquibase classpath.

  3. Reference the class in your changelog:

    ...
    <changeSet id="21" author="nvoxland">
        <customChange class="liquibase.change.custom.ExampleCustomTaskChange">
            <param name="helloTo" value="world"/>
        </customChange>
    </changeSet>
    ...

Custom migrations in Clojure

Being able to write custom code for db migrations is nice, but I would really like to write them in Clojure, not Java. This must be possible!

AOT

Remember the requirement of referencing the actual class in the customChange tag definition? This is something we need to preserve.

However, Clojure is a dynamically compiled language which compiles to JVM bytecode on the fly. Moreover, the class has to implement a specific Java interface, in our case liquibase.change.custom.CustomTaskChange.

How do we do that? One way, is to use gen-class:

(ns myapp.database.migrations.mig001
  "Migration for renaming column 'value' to 'val'."
  (:require [clojure.java.jdbc :as jdbc])
  (:import (java.sql SQLException)
           (liquibase.exception ValidationErrors)
           (liquibase.database Database)
           (liquibase.structure.core Column)))

(gen-class :name "myapp.database.migrations.Mig001"
           :implements [liquibase.change.custom.CustomTaskChange
                        liquibase.change.custom.CustomTaskRollback])

(defn -getConfirmationMessage [_this] "Renamed value column to val")
(defn -setFileOpener [_this _resourceAccessor] nil)
(defn -setUp [_this] nil)
(defn -validate [_this ^Database _database] (ValidationErrors.))
(defn -rollback [_this ^Database _database] nil)
(defn -execute [_this ^Database database]
  (let [db-spec {:connection (.getUnderlyingConnection (.getConnection database))}
        quote-name (fn [n] (.quoteObject database n Column))]
    (try
      (rename-column db-spec (quote-name "VALUE"))
      (catch SQLException e
        (rename-column db-spec (quote-name "value"))))))

Again, we reference the class name in our XML-based changeset definition

    <customChange class="myapp.database.migrations.Mig001" />

Then you need to AOT-compile it. It’s not too dificult when using leiningen:

(defproject myapp ...
...
  ;; DB migrations need always be compiled because the migrator requires Java classes
  :aot [#"^myapp\.database\.migrations\..*"]
...
  ;; the AOT config above is only for development - for uberjar, we AOT-compile everything anyway
  :uberjar {:aot :all
            :omit-source true}

Notice we do not want to compile all the classes, just the migrations.

Problems with AOT

The approach described above works but it’s also annoying:

  • AOT compilation is automatically performed when we compile the project and it can take a lot of time. It’s not something that we would normally do when developing the app. It’s usually enough to do it once - unless you call lein clean or otherwise remove the classes compiled into the target/ directory.

  • Whenever we change the migrations code we need to AOT compile it again to make sure Liquibase can use the latest version of our migrations code.

  • The code using gen-class is quite repetitive and unfriendly.

Getting rid of AOT (deftype)

Wouldn’t it be nice if we could get rid of all the gen-class and AOT stuff?

In Clojure, we have mechanisms to implement interfaces via deftype, defrecord, and reify.

  • Of these, we cannot use reify because it produces an anonymous class and its name changes every time it’s invoked.

  • defrecord and deftype produce a named class.

    • defrecord is more suitable for domain entities since it also offers a map-like capabilities (see also deftype vs defrecord (StackOverflow)).

    • deftype, on the other hand, is more suitable for lower-level programming constructs where we do not need the additional offerings provided by defrecord.

The verdict, then, is to use deftype. But how exactly we do that?

We can keep our XML-based changset definition - although we may want to tweak the class names a bit, it basically stays the same.

To get rid of the duplication, we may want to introduce a macro

(ns myapp.database.migrations
  (:require
   [myapp.database.migrations.mig001 :as mig001]
   [clojure.java.jdbc :as jdbc]
  (:import
   (liquibase.database.jvm JdbcConnection)
   (liquibase Liquibase Contexts LabelExpression)
   (liquibase.resource ClassLoaderResourceAccessor)))

(defmacro define-reversible-migration
  "Define a reversible custom migration.
  Both the forward and reverse migrations are defined using the same structure,
  similar to the bodies of multi-arity Clojure functions.

  Example:

  ```clj
  (define-reversible-migration ExampleMigrationName tx
   (migration-body tx)
   (reverse-migration-body tx)))
  ```"
  [name tx-symbol migration-body reverse-migration-body]
  `(deftype ~name []
     liquibase.change.custom.CustomTaskChange
     (execute [_# database#]
       ;; Make the liquibase database object available as `database-anaphora` symbol for more advanced usage
       (let [~'database-anaphora database#]
         (jdbc/with-db-transaction [~tx-symbol {:connection (.getUnderlyingConnection (.getConnection database#))}]
           ~migration-body)))
     (getConfirmationMessage [_#]
       (str "Custom migration: " ~name))
     (setUp [_#])
     (validate [_# _database#]
       (liquibase.exception.ValidationErrors.))
     (setFileOpener [_# _resourceAccessor#])

     liquibase.change.custom.CustomTaskRollback
     (rollback [_# database#]
       (jdbc/with-db-transaction [~tx-symbol {:connection (.getUnderlyingConnection (.getConnection database#))}]
         ~reverse-migration-body))))

(defn no-op
  "No-op logging rollback function"
  [n]
  (log/info "No rollback for: " n))

(defmacro define-migration
  "Define a custom migration without a reverse migration."
  [name tx-symbol & migration-body]
  `(define-reversible-migration ~name ~tx-symbol (do ~@migration-body) (no-op ~(str name))))


;;; The custom migrations
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; note that that class names are referenced in the liquibase XML config files
(define-migration Mig001 tx (mig001/execute tx))
;; => myapp.database.migrations.Mig001

This forms the basis for defining a new custom migrations easily. To add a new migration, you simply create a new Clojure namespace with the execute function:

(ns myapp.database.migrations.mig001)

(defn execute [tx]
   ;; implement your cusotm migration here ...
  )

Classloading gotcha

There’s still one missing piece, though. To make sure Liquibase can find the classes produced by the Clojure compiler after evaluating deftype, we need to tweak the classloader it uses.

For loading custom classes, Liquibase has ClassLoaderResourceAccessor But I couldn’t make it work - it did well in the REPL, but when I ran tests in a separate process (via lein test), they would fail with ClassNotFoundException.

liquibase.exception.ChangeLogParseException: liquibase.parser.core.ParsedNodeException:
liquibase.exception.CustomChangeException: java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
  liquibase.parser.core.ParsedNodeException: liquibase.exception.CustomChangeException: java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
  liquibase.exception.CustomChangeException: java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
           java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
         jdk.internal.loader.BuiltinClassLoader.loadClass       BuiltinClassLoader.java:  641
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass             ClassLoaders.java:  188
                          java.lang.ClassLoader.loadClass              ClassLoader.java:  525
                                 java.lang.Class.forName0                     Class.java
                                  java.lang.Class.forName                    Class.java:  467
     liquibase.change.custom.CustomChangeWrapper.setClass      CustomChangeWrapper.java:   79
         liquibase.change.custom.CustomChangeWrapper.load      CustomChangeWrapper.java:  298
                   liquibase.changelog.ChangeSet.toChange                ChangeSet.java:  535
...
            myapp.database.migrations/migrate                migrations.clj:   19

Notice, how it says AppClassLoader which is different from DynamicClassLoader used by Clojure.

To fix the problem, I had to manually change (and later restore) the current thread’s ContextClassLoader to make sure it uses the loader used to compile the custom migrations' classes.

Here is the final code for constructing the Liquibase object and running the migrations:

(defn migrate
  [db-spec]
  (jdbc/with-db-transaction
    [connection db-spec]
    (let [jdbc-connection (JdbcConnection. (:connection connection))
          ;; When this runs in tests (`lein test` et al), the current thread classloader is
          ;; AppClassLoader, not Clojure's DynamicClassLoader.
          ;; AppClassLoader knows nothing about our custom migration classes produced by `deftype`,
          ;; so we need to help Liquibase by using proper classloader.
          ;; But unexpectedly, just passing `clojure-classloader` to ClassLoaderResourceAccessor IS NOT ENOUGH!
          ;; We need to `.setContextClassLoader` otherwise it keeps using AppClassLoader
          ;; for whatever reason and fails with ClassNotFoundException.
          ;; - see also https://github.com/metabase/metabase/blob/77c64754c02eb0854182b96a0c6e6b96fa3b6b2c/src/metabase/db/liquibase.clj#L63
          clojure-classloader (.getClassLoader (class migrations/->Mig001))
          original-classloader (.getContextClassLoader (Thread/currentThread))
          _ (.setContextClassLoader (Thread/currentThread) clojure-classloader)
          liquibase (Liquibase. "liquibase/migrations/master.xml"
                                ;; ideally passing clojure-classloader here would be enough, but it doesn't work (see comment above)
                                (ClassLoaderResourceAccessor. clojure-classloader) jdbc-connection)]
      (try
        (try
          ;; try to validate the change set checksums
          (.validate liquibase)
          (catch Exception _
            ;; in case of exception we try to recover by clear the check sums to force recalculate them
            (.clearCheckSums liquibase)))
        (.update liquibase (Contexts.))
        (finally
          ;; restore the original classloader
          (.setContextClassLoader (Thread/currentThread) original-classloader))))))

Credits

This was no easy feat and I struggled a lot while trying to make it work.

Having problems, I asked asked on Clojurians slack about Liquibase, ClassNotFoundException, and deftype: https://clojurians.slack.com/archives/C1Q164V29/p1694554387177539 dpsutton kindle responded with hints about how Metabase approaches custom migrations with Liquibase (and when they introduced it).

This was very helpful for me and I eventually used the same approach including copying their macro to get rid of boilerplate.

References

  • Liquibase

    • Introduction to Liquibase

    • changesets (SQL, XML, YAML, JSON)

    • Change Types

    • customChange Change Type

  • gen-class

  • AOT compilation

  • leiningen’s :aot config - project.clj

  • deftype, defrecord, and reify

  • deftype vs defrecord (StackOverflow)

    • also deftype vs defrecord (Eric Normand)

  • Leiningen, AOT compilation, and classloaders

    • contains a good insights into how different classloaders come into play

  • How Metabase approaches custom migrations with Liquibase (and when they introduced it).


Tags: clojure databases migrations leiningen

« p6spy - Spying on Your Database One SSH Key to rule them all: Or how to make sure that one and only one key is used (even if you use 1Password SSH agent and ProxyJump) »

Copyright © 2025 Juraj Martinka

Powered by Cryogen | Free Website Template by Download Website Templates