Creating and Deploying Custom Components in CloverDX: A Step-by-Step Guide

Written by
Jan Sedláček
Jan Sedláček

CloverDX is designed to be an extensible system. Although it already contains a lot of components, there can be cases where you need to create your own – be it a specific and often-used task, use of an external library, your own connector, etc. The full guide for creating custom component is described in the documentation, but for those of you who want a quick look at the process without the lengthy details, I’ve prepared some brief instructions. If you follow this guide step by step, you’ll be able to create custom components in no time.

Hello World

Let’s create a component called “HelloWorld” that works just like SimpleCopy, but additionally adds a “Hello” message to every string field. Then we’ll deploy that component in the CloverDX Designer so it can be used in a transformation graph.

We’ll start with a plain CloverDX Designer installation, then add the required tools, implement the component in Java, bundle it, and finally, deploy.

Complete instructions for creating custom component

Taking the fast track...

  • Start with steps 1-3 below
  • Download these pre-built projects: CloverDX-Custom-Component-Template.zip
  • Import projects from the template archive into your workspace.
    File → Import... and select General / Existing Projects into Workspace.
    Creating Custom Component in CloverDX Step By Step: New Java Project wizard Creating Custom Component in CloverDX Step By Step: New Java Project wizard
  • Finish with step 21 below. Done.

If you want to make changes...

  • If you want to change the component behavior, edit HelloWorldComponent.java.
    Remember, in this case you will need to perform steps 17 through 21
  • Component descriptor for Engine: com.CloverDX.customcomponent\plugin.xml.
    Remember, if you change this file you will need to perform steps 17 through 21
  • Component descriptor for Designer: com.CloverDX.designer.customcomponent\customcomponent.xml.
    Remember, if you change only this file you will just need to perform step 21

Step By Step

Install the Designer and plugins

  1. Download, install and run CloverDX Designer (http://www.cloverdx.com/download)
  2. Install Eclipse Plug-in Development Environment:
    1. Go to Help → Install new software.
    2. Select Indigo – http://download.eclipse.org/releases/indigo update site
    3. Search for General Purpose Tools / Eclipse Plug-in Development Environment.
  3. Switch to Java perspective. You are now ready to deploy Eclipse plugins.

Implement the component Java class

  1. Create new Java project com.CloverDX.customcomponent in your current workspace. (File → New → Other > Java/Java Project)
    Creating Custom Component in CloverDX Step By Step: New Java Project wizard Creating Custom Component in CloverDX Step By Step: New Java Project wizard step 2
  2. Hit Next and add CloverDX Engine to the project build path.
    Creating Custom Component in CloverDX Step By Step: Add Library - CloverDX Engine
  3. Create a new java class
    • Name: HelloWorldComponent
    • Package: com.CloverDX.customcomponent

    You don't need to set anything else here – you will paste the full source code in the next step anyway.

    Creating Custom Component in CloverDX Step By Step: Create new class
  4. Copy & paste the following full class implementation. Later, you can modify this to suit your needs. This one class implements all the functionality of the component.
    package com.CloverDX.customcomponent;
    import org.jetel.data.DataRecord;
    import org.jetel.data.DataRecordFactory;
    import org.jetel.exception.AttributeNotFoundException;
    import org.jetel.exception.ConfigurationStatus;
    import org.jetel.exception.XMLConfigurationException;
    import org.jetel.graph.InputPort;
    import org.jetel.graph.Node;
    import org.jetel.graph.Result;
    import org.jetel.graph.TransformationGraph;
    import org.jetel.metadata.DataFieldType;
    import org.jetel.util.SynchronizeUtils;
    import org.jetel.util.property.ComponentXMLAttributes;
    import org.w3c.dom.Element;
    
    /**
    * A sample custom component
    *
    * Copies records from input to output, adding a "Hello" message to every string field
    *
    */
    public class HelloWorldComponent extends Node {
    
    /**
    * Component internal type ID, see getType()
    */
    public final static String COMPONENT_TYPE = "HELLO_WORLD";
    
    /**
    * The message to say.
    * Can be overridden by setting "Message to say" attribute of the component.
    *
    * @see customcomponent.xml and "defaultHint" attribute - the default value here MUST be the same as in defaultHint
    */
    private String message = "Hello";
    
    /**
    * Name of the "Message to say" attribute in the XML
    *
    * @see fromXML() and customcomponent.xml
    */
    public final static String XML_MESSAGE_ATTRIBUTE = "message";
    
    /**
    * Standard constructor
    *
    * @param id
    * Component ID in the graph
    */
    public HelloWorldComponent(String id) {
    super(id);
    }
    
    /**
    * Our constructor with custom message
    *
    * @param id
    * Component ID in the graph
    *
    * @param message
    * Custom message to say
    */
    public HelloWorldComponent(String id, String message) {
    super(id);
    
    if (message != null && message.trim().length() > 0) {
    this.message = message;
    }
    }
    
    @Override
    public Result execute() throws Exception {
    
    /**
    * Initialize input port and a bucket input record that will be field with records coming to input port
    */
    InputPort inPort = (InputPort) getInputPort(0); // ports are numbered from 0
    DataRecord inRecord = DataRecordFactory.newRecord(inPort.getMetadata()); // always use DataRecordFactory to create record instances
    int numFields = inRecord.getNumFields();
    inRecord.init(); // initialize records only once
    inRecord.reset(); // just to make sure all field values are reset
    
    /**
    * Initialize output and "bucket" output records that will be filled as "copy-by-name" from the input with additional message
    */
    int numOutputPorts=getOutPorts().size();
    DataRecord[] outRecord = new DataRecord[numOutputPorts];
    for (int i = 0; i < numOutputPorts; i++) {
    outRecord[i] = DataRecordFactory.newRecord(getOutputPort(i).getMetadata());
    outRecord[i].init();
    outRecord[i].reset();
    }
    
    /**
    * MAIN LOOP
    */
    while (inPort.readRecord(inRecord) != null && runIt) { // always check internal "runIt" state
    
    /**
    * Add "message" to every field in the input which has "string" data type
    */
    for(int fieldNo = 0; fieldNo < numFields; fieldNo++) {
    if (inRecord.getField(fieldNo).getMetadata().getDataType().equals(DataFieldType.STRING)) {
    inRecord.getField(fieldNo).setValue(message + " " + String.valueOf(inRecord.getField(fieldNo).getValue()));
    }
    }
    
    /**
    * Send output to all out ports
    * For each output port, copy the modified input record (by name) and then send it out
    */
    for (int i=0; i < numOutputPorts; i++) {
    outRecord[i].reset(); // to make sure we have clean record before copying
    outRecord[i].copyFieldsByName(inRecord);
    writeRecord(i, outRecord[i]); // sends out record to the output edge
    }
    
    /**
    * Mandatory yield after every record
    */
    SynchronizeUtils.cloverYield();
    }
    
    return runIt ? Result.FINISHED_OK : Result.ABORTED;
    }
    
    /**
    * Factory method that creates the component from transformation graph source XML
    */
    public static Node fromXML(TransformationGraph graph, Element xmlElement) throws XMLConfigurationException, AttributeNotFoundException {
    
    ComponentXMLAttributes xattribs = new ComponentXMLAttributes(xmlElement, graph);
    
    String id = xattribs.getString(XML_ID_ATTRIBUTE);
    String message = xattribs.exists(XML_MESSAGE_ATTRIBUTE) ? xattribs.getString(XML_MESSAGE_ATTRIBUTE) : null;
    
    return new HelloWorldComponent(id, message);
    }
    
    /**
    * checkConfig checks configuration of the component for Designer and Engine.
    * Here, you would check connected input and output ports, required metadata etc.
    */
    @Override
    public ConfigurationStatus checkConfig(ConfigurationStatus status) {
    super.checkConfig(status);
    
    /**
    * Check that we have one input port and at least one output
    */
    if(!checkInputPorts(status, 1, 1) || !checkOutputPorts(status, 1, Integer.MAX_VALUE, false)) {
    return status;
    }
    
    return status;
    }
    
    @Override
    public String getType(){
    return COMPONENT_TYPE;
    }
    
    }
    
  5. Create a blank file plugin.xml in the project root.
    Creating Custom Component in CloverDX Step By Step
  6. Put this XML content into plugin.xml:
    <?xml version="1.0" encoding="UTF-8"?>
    <?xml version="1.0" encoding="UTF-8"?>
    <plugin id="custom.component" version="1.0.0" provider-name="Company">
    <runtime><library path="bin/"/></runtime>
    <requires engine-version="3.3.0"></requires>
    <extension point-id="component">
    <parameter
    id="className"
    value="com.CloverDX.customcomponent.HelloWorldComponent"/>
    <parameter id="type" value="HELLO_WORLD"/>
    </extension>
    </plugin>

Create CloverDX Designer plugin

  1. Create new Plug-in project com.CloverDX.designer.customcomponent in your current workspace. (File > New > Plug-in Development > Plug-in Project)
    Creating Custom Component in CloverDX Step By Step: Create Plugin project A
  2. On the second page of the new project wizard:
    • uncheck Generate an activator, a Java class that controls the plug-in's lifecycle
    • uncheck This plug-in will make contributions to the UI
    Creating Custom Component in CloverDX Step By Step: Create Plugin project B Creating Custom Component in CloverDX Step By Step: Create Plugin project C
  3. Create a blank file customcomponent.xml in the project root
    Creating Custom Component in CloverDX Step By Step
  4. Put this content into customcomponent.xml:
    <?xml version="1.0" encoding="UTF-8"?>
    <ETLComponentList>
    <ETLComponent category="Custom" name="Hello World"
    type="HELLO_WORLD" iconPath="platform:/plugin/com.CloverDX.gui/icons/transformers/SimpleCopy"
    passThrough="true">
    <shortDescription>Hello World</shortDescription>
    <description>A sample custom component. Greets the world.</description>
    <inputPorts>
    <singlePort name="0" required="true" />
    </inputPorts>
    <outputPorts>
    <multiplePort required="true" />
    </outputPorts>
    <properties>
    <property category="basic" displayName="Message to say" modifiable="true" name="message" nullable="true" defaultHint="Hello">
    <singleType name="string" />
    </property>
    </properties>
    </ETLComponent>
    </ETLComponentList>
  5. Create a new class
    • Class name: CLProvider
    • Package: com.CloverDX.designer.customcomponent
    • Superclass: com.CloverDX.gui.plugin.engine.AbstractClassLoaderProvider

    Leave the class body empty. (AbstractClassLoaderProvider cannot be resolved yet, but you can ignore it)

    Creating Custom Component in CloverDX Step By Step: Create class CLProvider
  6. Open META-INF/MANIFEST.MF
    1. Go to Dependencies tab
    2. Add com.CloverDX.gui plugin to the list of Required Plug-ins
      Creating Custom Component in CloverDX Step By Step
    3. Move to Extensions tab
    4. Add com.CloverDX.gui.component extension
    5. Set the following:
      • file to customcomponent.xml
      Creating Custom Component in CloverDX Step By Step
    6. Add com.CloverDX.gui.enginePlugin extension
    7. Set the following:
      • directory to plugins
      • ClassLoaderProvider to com.CloverDX.designer.customcomponent.CLProvider
      Creating Custom Component in CloverDX Step By Step
  7. Create a new folder plugins in com.CloverDX.designer.customcomponent project.
    Creating Custom Component in CloverDX Step By Step: Create plugins folder
  8. Right-click plugins folder and select Import..., General → File system.
    Creating Custom Component in CloverDX Step By Step: Import to plugins
  9. Use com.CloverDX.customcomponent project’s location in From directory field.
    • Check plugin.xml
    • Check bin
    Creating Custom Component in CloverDX Step By Step: Creating Custom Component
  10. Click Finish and verify that files have been copied to plugins folder.
    Verify plugins import
  11. Open plugin.xml in the com.CloverDX.designer.customcomponent project
    1. Switch to Build tab:
    2. Make sure that the following items are checked:
      • META-INF
      • customcomponents.xml
      • plugin.xml
      • plugins
      Creating Custom Component in CloverDX Step By Step: Creating Custom Component
  12. Right click com.CloverDX.designer.customcomponent and select Export... from the context menu.
    1. Select Plug-in development → Deployable plug-ins and fragments
      Creating Custom Component in CloverDX Step By Step: Creating Custom Component
    2. Select Destination / Install into host (no need to change anything else).
      Creating Custom Component in CloverDX Step By Step: Creating Custom Component
  13. Click Finish and restart Eclipse when asked to finish the installation of the new plugin.
  14. Congratulations! From here on out, you should be able to use the new component in your graphs.

 

Deploying to CloverDX Server

The steps above should guide you through the whole process to implement your own component and how to integrate it into the CloverDX Designer.

However, if you use the CloverDX server and want to be able to use your component there, you need to go one step further. Now, I’ll show you how to deploy the plugin into the Server so that your componenets also work in transformations living on the Server.

For the purpose of this demonstration, I'm going to use the component we created in the previous blog post. You can download the project template (you will only need the project com.CloverDX.customcomponent) or follow steps 1 through 9 to create your own component plugin. Please note that you only need the "engine" core plugin for the Server. For now, you can ignore steps 10-23 which are only needed to make the component usable in the Designer. However, in the future, you'll always want to deploy your components into both.

Once you have the plugin ready, pick a location on the Server machine where your active plugins will live. The location must be readable by the CloverDX Server process. Also, I recommend choosing a place outside the Server installation folder.

Now, let's configure the CloverDX Server. It's fairly easy. For this example, I'll use the config file [tomcat_home]/conf/Catalina/localhost/clover.xml. Although, keep in mind there are several other ways to configure the Server.

The engine.plugins.additional.src config property specifies the location you chose for your plugins. Each plugin located there has to be in its own directory.

Simply add this XML element to the config file:

<Parameter name="engine.plugins.additional.src" value="[Path-To-Your-Engine-Plugin-Location]" override="false" />  

After saving the changes, you need to restart the CloverDX Server. From now on, you should be able to use the new components in the Server.

To verify that all your plugins are available in the Server, log into the Server Console and go to Configuration > CloverDX Info > Plugins. You should find something like this:

Deploying a custom component plugin to the CloverDX Server

Upon startup, the Server deploys all the configured custom plugins into its working directory. Every time you restart the Server, all plugins from the working directory are removed and redeployed. Thus, if you remove your custom plugin from the configured path above, it will also be removed from the Server working directory after restart.

If you chose to use the sample plugin from our previous blog post, you can take the HelloWordTest project, put it to your Server and execute testHelloWorld.grf. It should run successfully and you should find this output in the log in Execution History of the graph.

2013-12-16 09:49:56,800 INFO  1703936 [TRASH_1703936] +-------+------------------+-------------------------+------+
2013-12-16 09:49:56,801 INFO  1703936 [TRASH_1703936] |Record |name              |city                     |age   |
2013-12-16 09:49:56,802 INFO  1703936 [TRASH_1703936] +-------+------------------+-------------------------+------+
2013-12-16 09:49:56,802 INFO  1703936 [TRASH_1703936] |# 1    |Hello Sonya       |Hello Ottawa-Carleton    |49    |
2013-12-16 09:49:56,802 INFO  1703936 [TRASH_1703936] |# 2    |Hello Fleur       |Hello Newtonmore         |26    |
2013-12-16 09:49:56,803 INFO  1703936 [TRASH_1703936] |# 3    |Hello Addison     |Hello Pictou             |58    |
2013-12-16 09:49:56,803 INFO  1703936 [TRASH_1703936] |# 4    |Hello Ariel       |Hello Plymouth           |40    |
2013-12-16 09:49:56,803 INFO  1703936 [TRASH_1703936] |# 5    |Hello John        |Hello Cheyenne           |24    |
2013-12-16 09:49:56,803 INFO  1703936 [TRASH_1703936] |# 6    |Hello Hasad       |Hello Deschambault       |61    |
2013-12-16 09:49:56,803 INFO  1703936 [TRASH_1703936] |# 7    |Hello Abel        |Hello Roosbeek           |23    |
2013-12-16 09:49:56,804 INFO  1703936 [TRASH_1703936] |# 8    |Hello Xyla        |Hello Rycroft            |37    |
2013-12-16 09:49:56,804 INFO  1703936 [TRASH_1703936] |# 9    |Hello Keelie      |Hello Matlock            |79    |
2013-12-16 09:49:56,804 INFO  1703936 [TRASH_1703936] |# 10   |Hello Trevor      |Hello Rodengo/Rodeneck   |70    |
2013-12-16 09:49:56,804 INFO  1703936 [TRASH_1703936] |# 11   |Hello Yardley     |Hello Diepenbeek         |26    |
2013-12-16 09:49:56,805 INFO  1703936 [TRASH_1703936] |# 12   |Hello Steven      |Hello Werbomont          |43    |
2013-12-16 09:49:56,805 INFO  1703936 [TRASH_1703936] |# 13   |Hello Patience    |Hello Kirkwall           |43    |
2013-12-16 09:49:56,805 INFO  1703936 [TRASH_1703936] |# 14   |Hello Charlotte   |Hello Labro              |47    |
2013-12-16 09:49:56,805 INFO  1703936 [TRASH_1703936] |# 15   |Hello Ashely      |Hello Madrid             |32    |
2013-12-16 09:49:56,806 INFO  1703936 [TRASH_1703936] +-------+------------------+-------------------------+------+

 

PLEASE NOTE: We have observed a few issues with deploying plugins into Eclipse.
If your component is not working after steps above, please check again the settings of the “Build” tab when editing plugin file (Step 18). This seems to be an Eclipse bug.
For more information, please check our documentation
Posted on November 08, 2013.
DISCOVER  CloverDX Data Integration Platform  Design, automate and operate data jobs at scale. Learn more <>

Subscribe to our blog

Related Content


Where to go next