Workflow development primer

Following the instructions on this page, you should be able to setup and run a basic workflow within ten minutes. At the end of this page you’ll find all commands in one code block. This guide assumes that you will be developing for Roddy version 3.0.x and that you will create a JVM based workflow.

1. Setup a plugins folder

The plugins folder is the folder, where you will store your (self-created) plugins.

mkdir ~/RoddyPlugins

2. Prepare the plugin folder

Now create a folder in which you will store your new plugin.

cd ~/RoddyPlugins
mkdir NewPlugin
cd NewPlugin

3. Create the first files and folders

This will create the basic structure which is necessary for your plugin. See the Plugin developers guide for more information about plugin structures. We are

mkdir -p resources/analysisTools/workflowTools
mkdir -p resources/configurationFiles

echo 0.0 > buildversion.txt
echo 0 >> buildversion.txt

echo "dependson=PluginBase:1.0.29" > buildversion.txt
echo "dependson=DefaultPlugin:1.0.34" > buildversion.txt
echo "RoddyAPIVersion=3.0" > buildversion.txt
echo "JDKVersion=1.8" >> buildversion.txt
echo "GroovyVersion=2.4" >> buildversion.txt

4. Create the src folder and the inital java package

We’ll use our package structure for this example, change it as you need it. You’ll need the src structure, if you want to compile the plugin using Roddy.

mkdir -p src/de/dkfz/roddy/newplugin
cd src/de/dkfz/roddy/newplugin

In this directory, create the file NewPlugin.java and put in the following code.

package de.dkfz.roddy.newplugin;

import de.dkfz.roddy.plugins.BasePlugin;

public class TestPlugin extends BasePlugin {
    public static final String CURRENT_VERSION_STRING = "0.0.0";
    public static final String CURRENT_VERSION_BUILD_DATE = "NotBuildYet";

    @Override
    public String getVersionInfo() {
        return "Roddy plugin: " + this.getClass().getName() + ", V " + CURRENT_VERSION_STRING + " built at " + CURRENT_VERSION_BUILD_DATE;
    }
}

There you are, next step is…

5. Create a workflow class

In this directory, create the file NewWorkflow.java and put in the following code.

package de.dkfz.roddy.newplugin;

import de.dkfz.roddy.core.ExecutionContext;
import de.dkfz.roddy.core.Workflow;

public class NewWorkflow extends Workflow {
    @Override
    public boolean execute(ExecutionContext context) {
        return true;
    }
}

6. Create your analysis XML file

The next step is the creation of your analysis XML file, which will make the workflow available to Roddy. If the XML file is setup properly, you can import the analysis in your project configuration or call it in configuration-free mode.

cd ~/RoddyPlugins/NewPlugin/resources/configurationFiles
<configuration name='newAnalysis' description=''
           configurationType='analysis'
           class='de.dkfz.roddy.core.Analysis'
           workflowClass='de.dkfz.roddy.newplugin.NewWorkflow'
           runtimeServiceClass="de.dkfz.roddy.core.RuntimeService"
           listOfUsedTools="testScript" usedToolFolders="workflowTools">
  <configurationvalues>
    <cvalue name="firstValue" value="FillIt" type="string" />
    <cvalue name="testOutputDirectory" value="${outputAnalysisBaseDirectory}/testfiles" type="path"/>
  </configurationvalues>
  <processingTools>
    <tool name='testScript' value='testScriptSleep.sh' basepath='workflowTools'>
      <resourcesets>
        <rset size="l" memory="1" cores="1" nodes="1" walltime="5"/>
      </resourcesets>
      <input type="file" typeof="SimpleTestTextFile" scriptparameter="FILENAME_IN"/>
      <output type="file" typeof="SimpleTestTextFile" scriptparameter="FILENAME_OUT"/>
    </tool>
  </processingTools>
  <filenames package='de.dkfz.roddy.knowledge.examples' filestagesbase='de.dkfz.roddy.knowledge.examples.SimpleFileStage'>
    <filename class='SimpleTestTextFile' onTool='testScript' pattern='${testOutputDirectory}/test_onScript_1.txt'/>
  </filenames>
</configuration>

There you are. You now have a tool which you can call from your workflow.

7. Extend the workflow

Open up the workflow class again and change the execute method so that it calls the tool “testScript”. For that to work, you need to load one SimpleTestTextFile.

public boolean execute(ExecutionContext context) {

    SimpleTestTextFile textFile = (SimpleTestTextFile)loadSourceFile("/tmp/someTextFile.txt");
    SimpleTestTextFile result = call("testScript", textFile);
    return true;
}

Successful Roddy workflows will return true. If you detect an error, you can return false or throw an exception. Only one thing is missing, before you try out your new workflow.

8. Create the first script

cd ~/RoddyPlugins/NewPlugin/resources/analysisTools/workflowTools

echo 'sleep 10' > testScriptSleep.sh
echo 'cat $FILENAME_IN > $FILENAME_OUT' > testScriptSleep.sh

chmod 770 testScriptSleep.sh

9. Create a new properties file for Roddy

There is a skeleton application properties file in your Roddy folder. Copy the file [RODDY]/dist/bin/develop/helperScripts/skeletonAppProperties.ini to a location of your choice. Open it and add the folder ~/RoddyPlugins to the pluginDirectories entry. Also change the jobManager class to DirectSynchronousExecutedJobManager. Just comment the currently active line and uncomment the new jobManager.

10. Last steps

The last step you need to do is to compile and run the script.

[RODDY_DIRECTORY]/roddy.sh compileplugin NewPlugin --c=[YOUR_INI_FILE]

If you stuck to the example code, everything should be fine now and you can call it. Note that in this call the workflow version will automatically get bumped at the patch level, i.e. $major.$minor.$patch => $major.$minor.($patch + 1). If you want to increase the major or minor level, you have to do this manually in the buildversion.txt in the plugin-root directory. Furthermore, you should then turn of patch-level bumping using the INCREASE_BUILD_VERSION environment variable, otherwise the patch-version will get again incremented automatically:

INCREASE_BUILD_VERSION=false [RODDY_DIRECTORY]/roddy.sh compileplugin NewPlugin --c=[YOUR_INI_FILE]

We’ll use to the configurations-free mode here. Therefore we call the testrun mode with the pattern

[RODDY_DIRECTORY]/roddy.sh testrun [PluginName]_[PluginVersion]:[ConfigurationName]Analysis

Note that the “ConfigurationName” is the name attribute in the workflow configuration in the plugin, however without the “Analysis” suffix. The suffix is re-added by Roddy.

Project configuration files are explained in Configuration files. If you use a project configuration file, put in a directory of your choice (e.g. where you put your ini file from the step before).

[RODDY_DIRECTORY]/roddy.sh testrun NewPlugin_develop:test --c=[YOUR_INI_FILE]

Command code block