The Duct Framework [Beta]

This section will cover setting up a new Duct project. You’ll first need to ensure that the Clojure CLI is installed. You can check this by running the clojure command.

Next, create a project directory. For the purposes of this example, we’ll call the project tutorial.

The Clojure CLI looks for a file called deps.edn. To use Duct, we need to add the Duct Main tool as a dependency, and setup an alias to execute it.

To achieve this, create a new deps.edn file with the following content:

Duct can now be run by invoking the :duct alias.

We’ll be using this command a lot, so it’s highly recommended that you also create an shell alias. In a POSIX shell such as Bash, this can be done using the alias command.

For the rest of this documentation, we’ll assume that this shell alias has been defined.

The final step of the setup process is to create a duct.edn file. This contains the data structure that defines your Duct application. The Duct Main tool has a flag to generate a minimal configuration file for us.

This will create a file: duct.edn.

As mentioned previously, Duct uses a file, duct.edn, to define the structure of your application. We’ll begin by adding a new component key to the system.

If we try running Duct, it will complain about a missing namespace.

Duct is searching for a definition for the component, but not finding anything. This is unsurprising, as we haven’t written any code yet. Let’s fix this.

First we’ll create the directories.

Then a minimal Clojure file at: src/tutorial/print.clj.

Now if we try to run the application, we get the expected output.

Congratulations on your first Duct application!

Duct has two ways of running your application: --main and --repl.

In the previous section we started the application with --main, which will initiate the system defined in the configuration file, and halt the system when the process terminates.

The REPL is an interactive development environment.

In the REPL environment the system will not be initiated automatically. Instead, we use the inbuilt (go) function.

The REPL can be left running while source files updated. The (reset) function will halt the running system, reload any modified source files, then initiate the system again.

You can also use the Alt-E hotkey instead of typing (reset).

The configuration defined by duct.edn can be accessed with config, and the running system can be accessed with system.

Duct includes a test runner based on Kaocha. By default it looks for test files in the test directory.

We can write a unit test for our ‘Hello World’ function.

And then run it with the --test option.

Duct also provides a test library for running tests on your entire system.

Add the dependency:

This provides a function, duct.test/run, which will start the system. Our ‘hello’ component doesn’t need cleaning up after itself, but it’s good practice to use the duct.test/with-system macro. This will halt the system after the macro’s body completes (see the Integrant section for more information on halting).

As --test uses Kaocha under the hood, you can customize how the tests are run via a tests.edn file. See the Kaocha documentation for a full explanation of how this works.

Finally, there are a few options you can use at the command line to filter which tests will be run.

So far all of our dependencies have been listed under the :deps key, including org.duct-framework/test, which is only used in tests.

While this isn’t necessarily bad — your test dependencies are unlikely to be large enough to matter — it is good practice to separate out dependencies used for developing or testing into a separate alias in your deps.edn file.

Then to run the tests you’d use:

As this is quite a lot to type, you may want to add an shell alias. For example, ductt with an extra ‘t’ for ‘test’.

For the remainder of this document we’ll only use the root-level :deps and the duct alias we defined in the Project Setup. However, it’s important to keep in mind that you can customize your own project to suit your particular requirements and preferences.

A module groups multiple components together. Duct provides a number of pre-written modules that implement common functionality. One of these modules is :duct.module/logging.

We’ll first add the new dependency:

Then we’ll add the module to the Duct configuration.

Before the components are initiated, modules are expanded. We can see what this expansion looks like by using the --show flag. This will print out the expanded configuration instead of initiating it.

The logging module has been replaced with the :duct.logger/simple component.

The --show flag also works with the --repl command.

But wait a moment, why is the expansion of the configuration different depending on how we run Duct? This is because the --main flag has an implicit :main profile, and the --repl flag has an implicit :repl profile.

The :duct.module/logging module has different behaviors depending on which profile is active. When run with the :main profile, the logs print to STDOUT, but this would be inconveniently noisy when using a REPL. So when the :repl profile is active, most of the logs are sent to a file, logs/repl.log.

In order to use this module, we need to connect the logger to our ‘hello’ component. This is done via a ref.

The #ig/ref data reader is used to give the ‘hello’ component access to the logger. We use :duct/logger instead of :duct.logger/simple, as keys have a logical hierarchy, and :duct/logger fulfils a role similar to that of an interface or superclass.

Now that we’ve connected the components together in the configuration file, it’s time to replace the println function with the Duct logger.

The duct.logger/report function is used to emit a log at the :report level. This is a high-priority level that should be used sparingly, as it also prints to STDOUT when using the REPL.

You may have noticed that we’ve replaced the "Hello World" string with a keyword and a map: ::hello {:name "World"}. This is because Duct is opinionated about logs being data, rather than human-readable strings. A Duct log message consists of an event, a qualified keyword, and a map of event data, which provides additional information.

When we run the application, we can see what this produces.

But when using the REPL, we get a more concise message.

Sometimes we want to supply options from an external source, such as an environment variable or command line option. Duct allows variables, or vars, to be defined in the duct.edn configuration.

Currently our application outputs the same log message each time it’s run. Let’s create a configuration var to customize that behavior.

Then in the source file we can add the :name option that the var is attached to.

The default ensures that the application functions the same as before.

But we can now customize the behavior via a command-line flag, --name, or via an environment variable, NAME.

Vars are defined as a map of symbols to maps of options. The following option keys are supported:

A Duct application has some number of active profiles, which are represented by unqualified keywords. When run via the --main flag, an implicit :main profile is added. When run via (go) at the REPL, an implicit :repl profile is added. When run via (duct.test/run), an implicit :test profile is added.

You can add additional profiles via the --profiles argument. Profiles are an ordered list, with preceding profiles taking priority.

Most of the modules that Duct provides use profiles to customize their behavior to the environment they’re being run under. We can also use the #ig/profile data reader to create our own profile behavior.

Let’s change our component to allow for the log level to be specified.

In duct.edn we can use a profile to change the log level depending on whether the application uses the :main or :repl profile.

So far we’ve used functions to implement components. The :tutorial.print.hello component was defined by:

But this is just convenient syntax sugar for Integrant’s init-key method. The following code is equivalent to the previous component definition:

Duct uses Integrant for its component definitions, and Integrant provides several multimethods to this end. The most common one is init-key. If no such method is found, Integrant searches for a function of the same name.

There is also halt-key!, which defines a teardown procedure for a key. This can be useful for cleaning up files, threads or connections that the init-key method (or function) opened. The return value from init-key will be passed to halt-key!.

For more information on the multimethods that can be used, refer to the Integrant documentation.

As a configuration grows, it may become useful to split it up into several smaller files. We can do this via the #duct/include reader tag.

If you tag a filepath string with #duct/include, it indicates to Duct that it should replace the tagged string with the corresponding edn file. You can place this anywhere in the your duct.edn configuration.

For example, suppose we want to factor out all of the vars into their own configuration file, and also have a separate configuration for the ‘hello’ component.

The path of the includes is always relative to the root configuration file — in this case, duct.edn.

We’ll begin by creating a new project directory.

The first thing we’ll need is a deps.edn file that to provide the project dependencies. This will include Duct main and two additional modules: logging and web.

With that done, we need to ensure that the src directory exists. This is the default directory Clojure uses to store source files.

As this is a Duct application, we’ll need a duct.edn file. This will contain the two modules we added to the project’s dependencies.

We can now start the application with --main.

The web application should now be up and running at: http://localhost:3000/

Visiting that URL will result in a ‘404 Not Found’ error page, because we have no routes defined. The error page will be in plaintext, because we haven’t specified what features we want for our web application.

We’ll fix both these issues, but before we do we should terminate the application with Ctrl-C and start a REPL. We’ll keep this running while we develop the application to avoid costly restarts and to give us a way of querying the running system.

Clojure has many excellent libraries for writing web applications, but it can be difficult to put them all together. Duct’s web module handles that for you, but like all modules, we can always override any default that we don’t like.

For now, we’ll tell the web module to configure the application for use as a webside, using the :site feature. We’ll also add in a single route to handle a web request to the root of our application.

Then we’ll create a Ring handler function for that route.

Finally, we trigger a (reset) at the REPL.

Now when we go access http://localhost:3000/ we find a HTML page instead. Congratulations on your first Duct web application!

In the previous section we set up a route and a handler function, but you may rightly wonder how the route finds the function.

In the Fundamentals section we learned that key/value pairs in the Duct configuration have definitions in the application’s source files, or from a library.

The function we defined was called todo.routes/index, and therefore we might assume that we’d have a matching key in the configuration.

This component key could then be connected to the routes via a ref. In other words:

And in fact, this is almost exactly what is going on behind the scenes.

The Duct web module expands out to a great number of components, including a web server, middleware and error handlers, all which can be customized. Amongst these components, it creates a router and a number of route handlers.

A web module configured the following routes:

Will expand out to:

The router component uses Reitit, a popular data-driven routing library for Clojure. Other routing libreries can be used, but for this documentation we’ll use the default.

Let’s take a closer look at function associated with the route.

This function returns another function, known as a Ring handler. Usually this function will return a response map, but in this case we’re returning a Hiccup vector.

Hiccup is a format for representing HTML as a Clojure data structure. Elements are represented by a vector starting with a keyword, followed by an optional attribute map and then the element body.

The :site feature of the web module adds middleware to turn Hiccup vectors into HTML response maps. If the response is a vector, it wraps the vector in response map. If the response is already a map, it checks the :body of the response for a vector.

If we wanted a custom status code or headers, then the full response map could be returned.

Or we could return the string directly:

All of these examples are equivalent, but returning a vector is the most convenient and concise.

Ring middleware are functions that transform Ring handlers. These are often used to parse information from the request map, such as encoded parameters or session data, or to transform the response map, by adding headers or formatting the response body.

In the previous section we saw how a Hiccup data structure could be directly attached to the response body. This is possible because Duct adds default middleware to look for Hiccup and format it into HTML.

Let’s create some middleware that will add a map of custom headers to every response:

Once we’ve created the middleware function, we can give it to the web module via the :middleware key:

We add a new key, :todo.middleware/wrap-headers, which configures and creates the middleware function, then we use an Integrant ref to add that function to a vector of middleware.

There three ways to apply middleware:

The previous example demonstrated how to apply middleware to all requests. However, sometimes you only want middleware to apply if at least one route matches. For example:

This will add the extra header only if the route matches. It won’t be added to the default 404 response that is returned when all routes fail to match.

Finally, you can attach middleware to specific routes, or groups of nested routes by adding the :middleware key to the route itself:

The web module adds a lot of its own middleware, depending on the :features you choose. Often this is enough, and so we’ll remove the custom middleware for now; it won’t be needed for the rest of this document.

The next step is to add a database to our application. We’ll use SQLite, which means we need the corresponding JDBC adapter as a dependency.

To give us a Clojure-friendly way of querying the database, we’ll also add a dependency on next.jdbc.

Finally, we’ll add the Duct SQL module. This will add a connection pool to the system that we can use to access the database.

Our project dependencies should now look like this:

We can load these new dependencies either by restarting the REPL, or by using the sync-deps function.

The next step is to add :duct.module/sql to our Duct configuration.

Then reset via the REPL:

Wait, what’s this about an unbound var? Where did that come from?

Modules can add vars, and the SQL module adds one called jdbc-url. This var can be set via:

We can also set a default value for this var via the configuration. As SQLite uses a local file for its database, we can add a default to be used in development.

If we want to change this in production, we can use the corresponding command-line argument or environment variable to override this default.

The SQL module adds a database connection pool under the key :duct.database.sql/hikaricp, which derives from the more general :duct.database/sql key. We can use this connection pool as a javax.sql.DataSource instance.

In order to give our route handlers access to this, we’ll use a ref. We could manually add the ref to each of the handler’s option map, as shown below.

This is useful if only some routes need to access the database. However, in this case, we expect that all routes will need database access in some fashion. To make this easier, the web module has an option, :handler-opts that applies common options to all route handlers it generates.

This will add the DataSource instance to the :db key of the component options. We can access this from the route handler function we created earlier.

Before we go further, however, we should set up the database schema via a migration.

Part of the SQL module is to add a migrator, a component that will manage database migrations. By default the Ragtime library is used, and looks for a migrations.edn file in your project directory.

Let’s create a migration for a table to store the todo list items.

When we reset the REPL, the migration is automatically applied.

If the migration is modified in any way, its ID will also change. At the REPL, this will result in the old version of the migration being rolled back, and the new version applied in its place.

Running the application via --main will also apply any new migrations to the database. However, if there is any mismatch between migrations, an error will be raised instead.

This difference reflects the environments that --main and --repl are anticipated to be used in. During development a REPL is used and mistakes are expected, so the migrator will work to sync the migrations with the database. During production migrations need to be applied with more care, and so any discrepancies should halt the migration process.

In some production environments, there may be multiple instances of the application running at any one time. In these cases, you may want to run the migrations separately. The --keys option allows you to limit the system to a subset of keys. We can use this option to run only the migrations and logging subsystems.

This will run any component with a key that derives from :duct/migrator or :duct/logger, along with any mandatory dependants.

Now that we have a database table and a web server, it’s time to put the two together. The database we pass to the index function can be used to populate an unordered list. We’ll change the index function accordingly.

We can reset via the REPL and add some test data with the sql convenience function.

If you visit http://localhost:3000/ you’ll be able to see the todo items that were added to the database table.

The next step is to allow for new todo items to be added through the web interface. This is a little more involved, as we’ll need a HTML form and a route to respond to the form’s POST.

First, we add a new handler, new-todo, to the configuration to handle the POST.

Then we need incorporate the POST handler and the form into the codebase.

There are two new additions here. The create-todo-form function creates a form for making new todo list items. You may notice that it includes a hidden field for setting an anti-forgery token. This prevents a type of attack known as a Cross-site request forgery.

The second addition is the new-todo function. This inserts a new row into the todo table, then returns a “303 See Other” response that will redirect the browser back to the index page.

If you reset via the REPL and check http://localhost:3000/, you should see a text input box at the bottom of the todo list, allowing more todo items to be added.

At this point we’re hitting the limitations of what we can do with HTML alone. JavaScript allows for more sophisticated user interaction, and in the Clojure ecosystem we have ClojureScript, a version of Clojure that compiles to JavaScript.

You’ll be unsurprised to learn that Duct has a module for compiling ClojureScript. As always we begin with our dependencies, and add the ‘cljs’ module.

As before, we can load these dependencies by either restarting the REPL, or by using the (sync-deps) command.

Next, the :duct.module/cljs key needs to be added to the Duct configuration file.

The module requires a :builds option to be set. This connects a build name to a ClojureScript namespace, or collection of namespaces. In the above example, the todo.client namespace will be compiled to the target/cljs/client.js JavaScript file. When Duct is started, this will be accessible at: http://localhost:3000/cljs/client.js.

Before todo.client can be compiled, we first need to write it. In order to check everything works, we’ll have it trigger an JavaScript alert.

In order to test this script compiles correct, we’ll add the script to our index function in the todo.routes namespace.

If you restart the REPL and check http://localhost:3000, you should see the alert.

At this point we have all the tools we need to write a web application. We can write routes that return HTML, and we write ClojureScript to augment those roots.

However, there is a common alternative to this ‘traditional’ architecture. We instead serve up a single, static HTML page, and create the UI dynamically with ClojureScript. Communication to the server will be handled by a RESTful API.

In order to demonstrate this type of web application, we’ll pivot and redesign what we have so far. First, we require a static index file. By default this should be placed in the static subdirectory.

We then need to change the routes and add the :api feature to the web module.

There are now have three RESTful API routes:

By default, these will expect either JSON or edn, depending on the type of the Content-Type and Accept headers.

The next step is to rewrite the handler functions for these routes. Instead of returning HTML, we’ll return data that will be translated into the user’s preferred format.

There are three functions for each of the three routes. The list-todos function returns a map as its body. If JSON is requested, the resulting response body will look like something like this:

The create-todo function creates a new todo item given a description, and the remove-todo function deletes a todo item. In a full RESTful application we’d have more verbs per route, but as this is just an example we’ll limit the application to the bare minimum.

The next step is to create the client code. For this we’ll use Replicant for updating the DOM, and Duct client.http for communicating with the server API.

This requires us to once again update the project dependencies:

Once we’ve run sync-deps in the REPL, we can create a ClojureScript file for the client UI.

Here we reach the edge of Duct. This ClojureScript file is not specific to our framework, but would be at home in any Clojure project. Nevertheless, for the sake of completeness we’ll provide some explanation of what this file does.

The get, post and delete functions from the Duct HTTP client simplify communication with the server. They communicate using the Transit serialization format, and automatically add headers to get around the webservers CSRF protection.

The update-todos, delete-todo and create-todo functions all update the store atom, which contains a data structure that represents the state of the UI. In this case, it’s a list of todo items.

There is a watch attached to the store atom. When the store is changed, the todos DOM element is updated accordingly, with a new unordered list of todo items. Replicant is smart enough to update only the elements that have changed, making updates efficient.

Now that we have both a server and client, we can (reset) the REPL and check the web application at: http://localhost:3000

A common pattern for using Integrant have a -main function that loads and initiates an Integrant configuration. In many cases, you can use Duct to replace this with a data-driven approach.

For example, suppose you’ve written an application with a custom server and worker queue component. You may have an Integrant configuration file that looks like this:

In order to run this configuration, you have a main function that loads in the config file and populates the it with additional values from the environment. In the example below, the server port number is pulled from the PORT environment variable.

Duct can be used to replace all this with a data-driven configuration. In your deps.edn file, add a Duct alias:

Then move your configuration into duct.edn under the :system key. Use the :vars key to define the options you want to pull from the environment or from command-line options.

To run your application, use clojure -M:duct, or the duct alias defined in the Project Setup section. If your project uses Leiningen rather than tools.deps, check out the section on Leiningen integration.

You may be using Integrant in such a way that Duct is not suitable for your project. However, you may be able to still take advantage of Duct’s libraries.

Many of the libraries Duct provides can be used independently as part of any Integrant configuration. Here are three of the most useful outside of Duct:

Babashka is a fast, native Clojure scripting runtime, and it includes a task runner that can be configured with a bb.edn file.

Duct can set up a basic bb.edn file for you with the --init-bb option.

This sets up three tasks: main, test and repl that correspond to running --main, --test and --repl.

One advantage of using a task runner like Babashka is that you can specify different dependencies for task. Another advantage is that a new developer doesn’t have to set up a duct alias as long as they have Babashka installed.

Docker is a system for running software in containers — isolated and virtual environments designed to run an application.

To create a Docker container for your Duct application, you will need a Dockerfile that describes how to build it. Duct will set one up for you with the --init-docker option.

To build the container, run:

This will create a container with all the dependencies downloaded. It will also handle any compilation from keys deriving from :duct/compiler. This means that your ClojureScript will be compiled, if you’re using the ClojureScript module.

To run the container:

This will start the Duct application and bind the container port 3000 to the host machine’s port 3000 so you can access your application at: http://localhost:3000

This container is configured to only run keys deriving from :duct/daemon (and those it references). This includes keys like :duct.server.http/jetty provided by the web module. This will exclude migrations in order avoid multiple containers behind a load balancer all trying to update the database at once.

In order to run the migrations, you’ll need to run Duct in your deployment environment with only the :duct/migrator keys. This should be part of your deployment scripts and run once each time you deploy.

Emacs is a popular editor for Clojure. CIDER extends Emacs with support for interactive programming in Clojure.

To use Emacs/CIDER with Duct, use the --init-cider option in your project directory.

This creates a hidden file, .dir-locals.el, that sets up CIDER with Duct-specific options. To connect to the project using CIDER, open a file in the project (such as duct.edn) and type: M-x clojure-jack-in RET

Once CIDER has connected, you can open a REPL with: C-c C-z

This works in a similar way to the command-line REPL. To start up Duct, you can use the (go) command:

To reset the project, you can use (reset) at the REPL, or type: M-x cider-ns-refresh RET

There’s also a key binding for this command: C-c M-n r

Hashp is a small debugging library that is integrated by default in Duct.

To use it, you can put #p in front of any expression, and it will print the expression, its value, and its location in your project.

For example, at the REPL:

However, under --main nothing will be printed. In fact, #p will be completely ignored at compile time and incur no additional performance cost. This is because --main is intended for production use, and --repl for development and debugging.

It’s recommended that you use Duct with deps.edn, however it is possible to use Duct with Leiningen.

To do so, you’ll need to update your project file with profile for Duct, and an alias to run it:

Now you can run Duct with:

Note that the startup time suffers somewhat. In local tests using Leiningen increased startup time from 900ms to 1400ms.

Visual Studio Code is a popular modern editor, and the Calva plugin gives it full Clojure support.

To setup Calva for Duct, use the --init-calva option in your project directory.

This creates a project settings file for VS Code. To connect to the project, you can either:

Then choose the ‘Duct’ project type.

You’ll be presented with a REPL where you can start the application with (go)

To reset the project, you can use (reset) at the REPL, or run the command: Calva: Refresh All Namespaces.