CloverETL is now CloverDX - Learn Why

Back to CloverDX Blog on Data Integration

Custom CTL Functions in CloverDX

Posted by Jan Sedláček on Jan 10, 2014 12:55:15 PM

A few weeks ago I published a blog post about how to create a custom CloverDX component. This is great if you have special logic that operates on whole records. But what if you have specific needs when it comes to processing single values, or more precisely, field values? This is where custom CTL functions come into play. In this post, I’ll cover how to extend CTL2, the embedded data transformation language inside CloverDX.

The instructions will be very similar to building a custom component so I recommend skimming through it. However, in the following step-by-step guide, I will provide references whenever needed.

As an example, we’ll implement a “Lorem Ipsum” function that will produce English-like bogus text for testing. For this purpose, we’ll use an external loremipsum Java library by Sven Jacobs. Eventually we will create two CTL functions – one without arguments and another with the word count as a single integer argument.

Brief description

This is what we’ll do.

  1. First, we’ll implement a CloverDX Engine plugin:
    • A Java class extending org.jetel.ctl.extensions.TLFunctionLibrary and implementing the desired CTL functions
    • A plugin.xml with ctlfunction extension definition
    • We’ll use org.jetel.ctl.extensions.TLFunctionAnnotation annotation to mark methods that serve as implementation of our CTL functions
  2. Second, we’ll create a CloverDX Designer plugin:
    • Put plugin.xml into it with the engineplugin extension definition
    • Import the engine plugin to a directory specified by the engineplugin extension definition

Once finished, the new function will be available in Designer dialogs (like Source tab in Transform Editor of a Reformat) and also it will work properly in runtime, including linking the extra library. So let’s get to it!

Step-by-step Creation of Custom CTL Functions

The prerequisites are identical to building a custom component (see Complete instructions)

  1. Creating the Engine Plugin:
    • Start a new Java project com.cloveretl.custom.loremipsum
    • Put the external library to lib/loremipsum-1.0.jar and make sure it’s on classpath
    • Create a new java class LoremIpsumCTLLibrary extending org.jetel.ctl.extensions.TLFunctionLibrary
      package com.cloveretl.customctl;
      import org.jetel.ctl.Stack;
      import org.jetel.ctl.extensions.TLFunctionAnnotation;
      import org.jetel.ctl.extensions.TLFunctionCallContext;
      import org.jetel.ctl.extensions.TLFunctionLibrary;
      import org.jetel.ctl.extensions.TLFunctionPrototype;
      import de.svenjacobs.loremipsum.LoremIpsum;
      public class LoremIpsumCTLLibrary extends TLFunctionLibrary {
          LoremIpsum loremIpsum = new LoremIpsum();
          public TLFunctionPrototype getExecutable(String functionName)
                  throws IllegalArgumentException {
              TLFunctionPrototype ret =
                      "getWords".equals(functionName) ? new GetWordsFunction() : null;
              return ret;
          public String getName() {
              return "LoremIpsumCTLLibrary";
          @TLFunctionAnnotation("Generates Lorem Ipsum text")
          public String getWords(TLFunctionCallContext context) {
              return this.loremIpsum.getWords();
          @TLFunctionAnnotation("Generates Lorem Ipsum text")
          public String getWords(TLFunctionCallContext context, Integer count) {
              if(count==null) {
                  return this.getWords(context);
              return this.loremIpsum.getWords(count);
          class GetWordsFunction implements TLFunctionPrototype {
              public void execute(Stack stack, TLFunctionCallContext context) {
                  if (context.getParams().length > 0) {
                      stack.push(getWords(context, stack.popInt()));
              public void init(TLFunctionCallContext context) { }
    • Create plugin.xml in root of the engine plugin project:
      <?xml version="1.0"?>
              <library path="bin/"/>
              <library path="lib/loremipsum-1.0.jar"/>
          <extension point-id="ctlfunction">
              <parameter id="libraryName" value="LoremIpsumCTLLibrary"/>
              <parameter id="className"
    • Note one thing: If you export the engine plugin to a .jar outside of Eclipse, the compiled class files will not be in the bin folder. Instead, they’ll be placed directly in the root of the .jar. If that’s the case change <library path="bin/"/> to <library path="./"/>.

  2. Designer plugin:
    • Again, the instructions are basically identical to creating a Designer plugin for a custom component – please follow the referenced steps in the other blog post:
    • Step 10 – Plugin Project Creation
    • Step 11 – Configuring the New Plugin Project
    • Step 14 – Create a Class Loader Provider Class
    • Step 15 – Only 1, 2, 6 and 7 – i.e. only those related to com.cloveretl.gui.enginePlugin
    • Steps 16 through 23 – Identical
    • For step 16 – It’s a best practice to put engine plugins into a special folder named by plugin ID. So in our case, you would create a directory com.cloverdx.custom.loremipsum in the plugins directory and perform filesystem import into it.
    • When editing plugin.xml in Step 20, set the classpath to the following three folders: the Designer plugin folder (“.”), engine plugin classes and libraries to classpath (in our case: ., plugins/com.cloverdx.custom.loremipsum/bin/, plugins/com.cloverdx.custom.loremipsum/lib/loremipsum-1.0.jar) Custom CTL Functions

The final structure of the Designer plugin with provided engine plugin should look like this:

Custom CTL Functions

The engine plugin classes are located in plugins/com.cloveretl.custom.loremipsum/bin. This is mainly for simplicity. When you develop complex plugins, we recommend packing the engine classes into a .jar file and putting it directly to the plugin (to the same level as engine plugin.xml file).

After the steps described in this tutorial, you should be able to use your own CTL functions. You can check the attached example projects, which contain the sources and exported plugin .jar, as well as an example graph.

Download CloverDX Custom CTL Function Example

Data integration software and ETL tools provided by the CloverDX platform (formerly known as CloverETL) offer solutions for data management tasks such as data integration, data migration, or data quality. CloverDX is a vital part of enterprise solutions such as data warehousing, business intelligence (BI) or master data management (MDM). CloverDX Designer (formerly known as CloverETL Designer) is a visual data transformation designer that helps define data flows and transformations in a quick, visual, and intuitive way. CloverDX Server (formerly known as CloverETL Server) is an enterprise ETL and data integration runtime environment. It offers a set of enterprise features such as automation, monitoring, user management, real-time ETL, data API services, clustering, or cloud data integration.