<changeSet id="1" author="nvoxland">
<createTable tableName="company">
<column name="address" type="varchar(255)"/>
</createTable>
</changeSet>Liquibase is a well-know tool for tracking, versioning, and deploying database schema changes.
It uses changelog files to list database changes in the form of changesets (SQL, XML, YAML, JSON), which consists of Change Types. Standard changsets use SQL or a DB-agnostic equivalent written in XML, YAML, or JSON.
Here’s an example of a very simple changset adding a new table with a single column:
<changeSet id="1" author="nvoxland">
<createTable tableName="company">
<column name="address" type="varchar(255)"/>
</createTable>
</changeSet>Sometimes, there’s a more complicated migration that’s very difficult or impossible to express in SQL (or its Xml/YAML/JSON equivalent). That is, you need to write actual code to perform the migration. In that case, Liquibase offers customChange Change Type.
To implement a custom migration you need to:
Create a Java class that implements the liquibase.change.custom.CustomSqlChange or liquibase.change.custom.CustomTaskChange interface
(showing only a subset of methods here):
public class ExampleCustomTaskChange implements CustomTaskChange, CustomTaskRollback {
private String helloTo;
@SuppressWarnings({"UnusedDeclaration", "FieldCanBeLocal"})
private ResourceAccessor resourceAccessor;
@Override
public void execute(Database database) throws CustomChangeException {
Scope.getCurrentScope().getLog(getClass()).info("Hello "+getHelloTo());
}
@Override
public void rollback(Database database) throws CustomChangeException, RollbackImpossibleException {
Scope.getCurrentScope().getLog(getClass()).info("Goodbye "+getHelloTo());
}
...
}Compile the created class, package it into a JAR file, and then add it to a Liquibase classpath.
Reference the class in your changelog:
...
<changeSet id="21" author="nvoxland">
<customChange class="liquibase.change.custom.ExampleCustomTaskChange">
<param name="helloTo" value="world"/>
</customChange>
</changeSet>
...Being able to write custom code for db migrations is nice, but I would really like to write them in Clojure, not Java. This must be possible!
Remember the requirement of referencing the actual class in the customChange tag definition? This is something we need to preserve.
However, Clojure is a dynamically compiled language which compiles to JVM bytecode on the fly.
Moreover, the class has to implement a specific Java interface, in our case liquibase.change.custom.CustomTaskChange.
How do we do that? One way, is to use gen-class:
(ns myapp.database.migrations.mig001
"Migration for renaming column 'value' to 'val'."
(:require [clojure.java.jdbc :as jdbc])
(:import (java.sql SQLException)
(liquibase.exception ValidationErrors)
(liquibase.database Database)
(liquibase.structure.core Column)))
(gen-class :name "myapp.database.migrations.Mig001"
:implements [liquibase.change.custom.CustomTaskChange
liquibase.change.custom.CustomTaskRollback])
(defn -getConfirmationMessage [_this] "Renamed value column to val")
(defn -setFileOpener [_this _resourceAccessor] nil)
(defn -setUp [_this] nil)
(defn -validate [_this ^Database _database] (ValidationErrors.))
(defn -rollback [_this ^Database _database] nil)
(defn -execute [_this ^Database database]
(let [db-spec {:connection (.getUnderlyingConnection (.getConnection database))}
quote-name (fn [n] (.quoteObject database n Column))]
(try
(rename-column db-spec (quote-name "VALUE"))
(catch SQLException e
(rename-column db-spec (quote-name "value"))))))Again, we reference the class name in our XML-based changeset definition
<customChange class="myapp.database.migrations.Mig001" />Then you need to AOT-compile it. It’s not too dificult when using leiningen:
(defproject myapp ...
...
;; DB migrations need always be compiled because the migrator requires Java classes
:aot [#"^myapp\.database\.migrations\..*"]
...
;; the AOT config above is only for development - for uberjar, we AOT-compile everything anyway
:uberjar {:aot :all
:omit-source true}Notice we do not want to compile all the classes, just the migrations.
The approach described above works but it’s also annoying:
AOT compilation is automatically performed when we compile the project and it can take a lot of time.
It’s not something that we would normally do when developing the app.
It’s usually enough to do it once - unless you call lein clean or otherwise remove the classes compiled into the target/ directory.
Whenever we change the migrations code we need to AOT compile it again to make sure Liquibase can use the latest version of our migrations code.
The code using gen-class is quite repetitive and unfriendly.
Wouldn’t it be nice if we could get rid of all the gen-class and AOT stuff?
In Clojure, we have mechanisms to implement interfaces via deftype, defrecord, and reify.
Of these, we cannot use reify because it produces an anonymous class and its name changes every time it’s invoked.
defrecord and deftype produce a named class.
defrecord is more suitable for domain entities
since it also offers a map-like capabilities (see also deftype vs defrecord (StackOverflow)).
deftype, on the other hand, is more suitable for lower-level programming constructs
where we do not need the additional offerings provided by defrecord.
The verdict, then, is to use deftype. But how exactly we do that?
We can keep our XML-based changset definition - although we may want to tweak the class names a bit, it basically stays the same.
To get rid of the duplication, we may want to introduce a macro
(ns myapp.database.migrations
(:require
[myapp.database.migrations.mig001 :as mig001]
[clojure.java.jdbc :as jdbc]
(:import
(liquibase.database.jvm JdbcConnection)
(liquibase Liquibase Contexts LabelExpression)
(liquibase.resource ClassLoaderResourceAccessor)))
(defmacro define-reversible-migration
"Define a reversible custom migration.
Both the forward and reverse migrations are defined using the same structure,
similar to the bodies of multi-arity Clojure functions.
Example:
```clj
(define-reversible-migration ExampleMigrationName tx
(migration-body tx)
(reverse-migration-body tx)))
```"
[name tx-symbol migration-body reverse-migration-body]
`(deftype ~name []
liquibase.change.custom.CustomTaskChange
(execute [_# database#]
;; Make the liquibase database object available as `database-anaphora` symbol for more advanced usage
(let [~'database-anaphora database#]
(jdbc/with-db-transaction [~tx-symbol {:connection (.getUnderlyingConnection (.getConnection database#))}]
~migration-body)))
(getConfirmationMessage [_#]
(str "Custom migration: " ~name))
(setUp [_#])
(validate [_# _database#]
(liquibase.exception.ValidationErrors.))
(setFileOpener [_# _resourceAccessor#])
liquibase.change.custom.CustomTaskRollback
(rollback [_# database#]
(jdbc/with-db-transaction [~tx-symbol {:connection (.getUnderlyingConnection (.getConnection database#))}]
~reverse-migration-body))))
(defn no-op
"No-op logging rollback function"
[n]
(log/info "No rollback for: " n))
(defmacro define-migration
"Define a custom migration without a reverse migration."
[name tx-symbol & migration-body]
`(define-reversible-migration ~name ~tx-symbol (do ~@migration-body) (no-op ~(str name))))
;;; The custom migrations
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; note that that class names are referenced in the liquibase XML config files
(define-migration Mig001 tx (mig001/execute tx))
;; => myapp.database.migrations.Mig001This forms the basis for defining a new custom migrations easily.
To add a new migration, you simply create a new Clojure namespace
with the execute function:
(ns myapp.database.migrations.mig001)
(defn execute [tx]
;; implement your cusotm migration here ...
)There’s still one missing piece, though.
To make sure Liquibase can find the classes produced by the Clojure compiler after evaluating deftype,
we need to tweak the classloader it uses.
For loading custom classes, Liquibase has
ClassLoaderResourceAccessor
But I couldn’t make it work - it did well in the REPL, but when I ran tests in a separate process (via lein test),
they would fail with ClassNotFoundException.
liquibase.exception.ChangeLogParseException: liquibase.parser.core.ParsedNodeException:
liquibase.exception.CustomChangeException: java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
liquibase.parser.core.ParsedNodeException: liquibase.exception.CustomChangeException: java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
liquibase.exception.CustomChangeException: java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
java.lang.ClassNotFoundException: myapp.database.migrations.Mig001
jdk.internal.loader.BuiltinClassLoader.loadClass BuiltinClassLoader.java: 641
jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass ClassLoaders.java: 188
java.lang.ClassLoader.loadClass ClassLoader.java: 525
java.lang.Class.forName0 Class.java
java.lang.Class.forName Class.java: 467
liquibase.change.custom.CustomChangeWrapper.setClass CustomChangeWrapper.java: 79
liquibase.change.custom.CustomChangeWrapper.load CustomChangeWrapper.java: 298
liquibase.changelog.ChangeSet.toChange ChangeSet.java: 535
...
myapp.database.migrations/migrate migrations.clj: 19Notice, how it says AppClassLoader which is different from DynamicClassLoader used by Clojure.
To fix the problem, I had to manually change (and later restore) the current thread’s ContextClassLoader to make sure it uses the loader used to compile the custom migrations' classes.
Here is the final code for constructing the Liquibase object and running the migrations:
(defn migrate
[db-spec]
(jdbc/with-db-transaction
[connection db-spec]
(let [jdbc-connection (JdbcConnection. (:connection connection))
;; When this runs in tests (`lein test` et al), the current thread classloader is
;; AppClassLoader, not Clojure's DynamicClassLoader.
;; AppClassLoader knows nothing about our custom migration classes produced by `deftype`,
;; so we need to help Liquibase by using proper classloader.
;; But unexpectedly, just passing `clojure-classloader` to ClassLoaderResourceAccessor IS NOT ENOUGH!
;; We need to `.setContextClassLoader` otherwise it keeps using AppClassLoader
;; for whatever reason and fails with ClassNotFoundException.
;; - see also https://github.com/metabase/metabase/blob/77c64754c02eb0854182b96a0c6e6b96fa3b6b2c/src/metabase/db/liquibase.clj#L63
clojure-classloader (.getClassLoader (class migrations/->Mig001))
original-classloader (.getContextClassLoader (Thread/currentThread))
_ (.setContextClassLoader (Thread/currentThread) clojure-classloader)
liquibase (Liquibase. "liquibase/migrations/master.xml"
;; ideally passing clojure-classloader here would be enough, but it doesn't work (see comment above)
(ClassLoaderResourceAccessor. clojure-classloader) jdbc-connection)]
(try
(try
;; try to validate the change set checksums
(.validate liquibase)
(catch Exception _
;; in case of exception we try to recover by clear the check sums to force recalculate them
(.clearCheckSums liquibase)))
(.update liquibase (Contexts.))
(finally
;; restore the original classloader
(.setContextClassLoader (Thread/currentThread) original-classloader))))))This was no easy feat and I struggled a lot while trying to make it work.
Having problems, I asked asked on Clojurians slack about Liquibase, ClassNotFoundException, and deftype: https://clojurians.slack.com/archives/C1Q164V29/p1694554387177539 dpsutton kindle responded with hints about how Metabase approaches custom migrations with Liquibase (and when they introduced it).
This was very helpful for me and I eventually used the same approach including copying their macro to get rid of boilerplate.
changesets (SQL, XML, YAML, JSON)
customChange Change Type
Leiningen, AOT compilation, and classloaders
contains a good insights into how different classloaders come into play
How Metabase approaches custom migrations with Liquibase (and when they introduced it).