Jenkins Tutorial: Creating Jobs with the Jenkins Job DSL

In the first part of this three-part article series, I showed you how to install and configure Jenkins as Docker containers using Ansible – and how to create build and release jobs for Maven projects using Jenkins' user interface.

In this second part, you'll learn how to create the same jobs with the Jenkins Job DSL as program code and how to import them into Jenkins. You'll also learn how to manually and programmatically create views for a clearly arranged job list.

You can find the code created in this article in my GitLab repository at https://gitlab.com/SvenWoltmann/jenkins-tutorial-demo.

Introduction to the Jenkins Job DSL

In this chapter, I'll explain what the Jenkins Job DSL is. I show you how to configure IntelliJ to make writing a job more comfortable with auto-completion, how to set up a minimal job as code, and how to import it into Jenkins.

What is the Jenkins Job DSL?

The Jenkins Job DSL enables the programmatic creation of Jenkins jobs using Groovy code. You can store this code in your Git repository and thus make changes traceable and generate Jenkins jobs automatically. The DSL is provided via the Job DSL Plugin and is documented in detail in the Job DSL API Viewer.

Enabling Job DSL support in IntelliJ IDEA

IntelliJ IDEA supports Groovy by default. With a so-called GroovyDSL script, we can also make IntelliJ aware of concrete DSLs. This allows auto-completion and better syntax highlighting.

The documentation provided for this purpose at https://github.com/jenkinsci/job-dsl-plugin/wiki/IDE-Support is not very comprehensible and requires an installation of Gradle. Now I show you how to do it without Gradle:

First, we create a new Maven project (or module) in IntelliJ. Since I have worked with a Monorepo in my demo code so far, I create the project in the jenkins-jobs/ subdirectory. Using the pom.xml, we define a dependency on the job-dsl-core library, which is located in the public Jenkins repository:

<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>eu.happycoders.jenkins-tutorial-demo</groupId>
  <artifactId>jenkins-jobs</artifactId>
  <version>1.0-SNAPSHOT</version>

  <repositories>
    <repository>
      <id>jenkins</id>
      <url>https://repo.jenkins-ci.org/public/</url>
    </repository>
  </repositories>

  <dependencies>
    <dependency>
      <groupId>org.jenkins-ci.plugins</groupId>
      <artifactId>job-dsl-core</artifactId>
      <version>1.76</version>
    </dependency>
  </dependencies>

</project>Code language: HTML, XML (xml)

Under src/main/resources/, we create a file idea.gdsl with the following content:

def jobPath = /.*/jobs/.*.groovy/

def ctx = context(pathRegexp: jobPath)
contributor(ctx, {
  delegatesTo(findClass('javaposse.jobdsl.dsl.DslFactory'))
})Code language: Groovy (groovy)

You can delete the directory src/test/ created by IntelliJ's project generator; we won't need it.

Instead, we create a new directory, src/jobs/. In this directory, we will store the Jenkins jobs. To test the setup, I create a groovy file Test.groovy and enter "maven". The auto-completion shows me two mavenJob() methods from the MavenJob class of the Job DSL plugin; thus, we've completed the setup.

The basic structure of the code for a Jenkins job

First, I show you a minimal example of how to write a job and import it into Jenkins. In the jobs/ directory created in the previous section, we create a file SimpleDSLJob.groovy with the following content:

mavenJob('Jenkins Tutorial Demo - Simple DSL Job') {
  description 'A very simple demo for the Jenkins Job DSL'

}Code language: Groovy (groovy)

The mavenJob() method is the entry point through which the Jenkins job is generated. If you do not know Groovy, here is a short explanation of the syntax: The code in braces is a closure (what is a lambda expression in Java), a function that is passed as a parameter. To where is this function passed? Now it gets a bit crazy: The closure is the second parameter of the mavenJob() method after the job name. If the last parameter of a method call is a closure, it can be passed outside the parentheses. Parentheses can also be omitted altogether, as is the case with the description() method. The code above is, therefore, the same as the following code:

mavenJob('Jenkins Tutorial Demo - Simple DSL Job', {
  description('A very simple demo for the Jenkins Job DSL')

})Code language: Groovy (groovy)

The Java equivalent of this code would be:

mavenJob("Jenkins Tutorial Demo - Simple DSL Job", () -> {
  description("A very simple demo for the Jenkins Job DSL");

});Code language: Groovy (groovy)

So much for that little Groovy exercise. If you consider what the Groovy code would look like in Java at the end of this tutorial, you find that the Groovy representation is much more readable.

Importing a Job programmed with the Job DSL into Jenkins

Please check first if you have installed the Jenkins plugins "Job DSL" and "Authorize Project" and make sure that you have done so. I initially didn't list these plugins in the first part of the article and only added them later.

To import the job into Maven, first click "New Item" on the home page:

It is essential to know that you do not create the actual job, but a job that imports the job written in the DSL. As the name, we enter "Project Jenkins Tutorial Demo - Simple DSL Job Creator". As project type, we select "Freestyle project" and click on "OK":

On the following page, we scroll down to "Build" (or click on the corresponding tab) and select "Process Job DSLs" from the "Add build step" dropdown:

In the opening form, we select "Use the provided DSL script" and copy the short piece of code into the corresponding field. You might be surprised that the field is so small. This is because we don't usually import DSL jobs this way. Instead, we automate them, as I will describe later in this tutorial. At this point, we just want to see that we get our minimal job up and running.

We click on "Save" and then – on the job page – on "Build Now" to start the job:

After the job has finished, we can click on the build in the lower-left corner of the build history:

We now see the status of the creator job: when it was executed, how long it took, and we get a link to the newly created job under "Generated Items":

We follow the link to the newly created job and see here the title and description as we defined them in the program code. We click on "Configure" to see the configuration:

In the configuration, we see nothing except name and description. That' s all we have defined in the DSL code.

The Jenkins home page should now show the creator job as well as the created "Simple DSL Job" next to the jobs from the first article:

Job configuration via Jenkins Job DSL for Maven projects

In this chapter, I show you how to convert the manually created build and release jobs from the first part into Jenkins Job DSL program code.

Programming a Jenkins build job via the Job DSL

For the Maven project "library1" we create a file BuildJobLibrary1.groovy in the jobs/ directory with the following initial content as known from the previous chapter.

mavenJob('Jenkins Tutorial Demo - Library 1 (DSL)') {
  description 'Build job for Jenkins Tutorial / Library 1'

}Code language: Groovy (groovy)

With the following code, we add the setting for keeping only the last five builds:

logRotator {
  numToKeep 5
}Code language: Groovy (groovy)

Next, we add the query for the Git branch parameter. We can represent the user interface settings that required two screenshots in the last article with only five lines of code:

parameters {
  gitParam('Branch') {
    description 'The Git branch to checkout'
    type 'BRANCH'
    defaultValue 'origin/master'
  }
}Code language: Groovy (groovy)

Next, we need to implement the sparse checkout for the repository "[email protected]:SvenWoltmann/jenkins-tutorial-demo.git" on the branch "$Branch" and the directory "library1/". This time the code gets a bit longer; this is because we have to configure the sparse checkout path and the polling filter via extensions. The Jenkins Job DSL does not natively support these features because they were added by the Git plugin. If you are not using a multirepo and want to check out the entire repository, you can omit the block marked with "// Add extensions…".

scm {
  git {
    remote {
      url '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'
    }

    branch '$Branch'

    // Add extensions 'SparseCheckoutPaths' and 'PathRestriction'
    def nodeBuilder = NodeBuilder.newInstance()
    def sparseCheckout = nodeBuilder.createNode(
          'hudson.plugins.git.extensions.impl.SparseCheckoutPaths')
    sparseCheckout
          .appendNode('sparseCheckoutPaths')
          .appendNode('hudson.plugins.git.extensions.impl.SparseCheckoutPath')
          .appendNode('path', 'library1/')
    def pathRestrictions = nodeBuilder.createNode(
          'hudson.plugins.git.extensions.impl.PathRestriction')
    pathRestrictions.appendNode('includedRegions', 'library1/.*')
    extensions {
      extensions << sparseCheckout
      extensions << pathRestrictions
    }
  }
}Code language: Groovy (groovy)

There would also be a more elegant and comfortable way to read variant for configuring the extensions:

configure {
  it / 'extensions' / 'hudson.plugins.git.extensions.impl.SparseCheckoutPaths' / 'sparseCheckoutPaths' {
    'hudson.plugins.git.extensions.impl.SparseCheckoutPath' {
      path 'library1/'
    }
  }
  it / 'extensions' / 'hudson.plugins.git.extensions.impl.PathRestriction' {
    includedRegions 'library1/.*'
  }
}Code language: Groovy (groovy)

Unfortunately, this does not work within the so-called "Groovy Sandbox," which we will need later in this tutorial.

The Git configuration is followed by the build trigger, which checks every 15 minutes whether the code in GitLab has changed and rebuilds the project if necessary:

triggers {
  scm 'H/15 * * * *'
}Code language: Groovy (groovy)

Finally, we specify the root POM path and Maven build parameters:

rootPOM 'library1/pom.xml'
goals 'clean install'Code language: Groovy (groovy)

The job is now complete. Here's the full code again (which you can also find in the GitLab repository):

mavenJob('Jenkins Tutorial Demo - Library 1 (DSL)') {
  description 'Build job for Jenkins Tutorial / Library 1'

  logRotator {
    numToKeep 5
  }

  parameters {
    gitParam('Branch') {
      description 'The Git branch to checkout'
      type 'BRANCH'
      defaultValue 'origin/master'
    }
  }

  scm {
    git {
      remote {
        url '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'
      }

      branch '$Branch'

      // Add extensions 'SparseCheckoutPaths' and 'PathRestriction'
      def nodeBuilder = NodeBuilder.newInstance()
      def sparseCheckout = nodeBuilder.createNode(
            'hudson.plugins.git.extensions.impl.SparseCheckoutPaths')
      sparseCheckout
            .appendNode('sparseCheckoutPaths')
            .appendNode('hudson.plugins.git.extensions.impl.SparseCheckoutPath')
            .appendNode('path', 'library1/')
      def pathRestrictions = nodeBuilder.createNode(
            'hudson.plugins.git.extensions.impl.PathRestriction')
      pathRestrictions.appendNode('includedRegions', 'library1/.*')
      extensions {
        extensions << sparseCheckout
        extensions << pathRestrictions
      }
    }
  }

  triggers {
    scm 'H/15 * * * *'
    snapshotDependencies true
  }

  rootPOM 'library1/pom.xml'
  goals 'clean install'
}Code language: Groovy (groovy)

Here is a screenshot from my IDE. As you can see, IntelliJ offers syntax highlighting and displays the variable names for method calls (which is often unnecessary, though, since many parameters have the same names as the methods):

We import the code in the same way as I showed it in the section Importing a job programmed with the Job DSL into Jenkins. The Creator job is called "Jenkins Tutorial Demo - Library 1 (DSL) Creator". We run it and get the new job "Jenkins Tutorial Demo - Library 1 (DSL)". On the screenshot you can see at the blue dot that this has already been executed successfully (via the 15-minute trigger):

Now, when you open two browser windows, one with the manually created job and one with the automatically created one, and click on "Configure" in both, you can compare the jobs line by line, and you can see that they are identical:

Likewise, we create a BuildJobApplication1.groovy file for the Maven project "application1", by copying the previously created BuildJobLibrary1.groovy file and replacing "library" with "application" in the code. This way, we need much less time to create the job for the application than we did with the manual configuration.

Programming a Jenkins release job via the Job DSL

Since the release job has many similarities with the build job, we copy the file BuildJobLibrary1.groovy to ReleaseJobLibrary1.groovy. We adjust the job name and the description. Since we want to create releases only from the master branch, we remove the parameters block and replace the parameter value $Branch in the branch() method with origin/master. Since we always start released manually, we remove the trigger block. The code now looks like this:

mavenJob('Jenkins Tutorial Demo - Library 1 - Release (DSL)') {
  description 'Release job for Jenkins Tutorial / Library 1'

  logRotator {
    numToKeep 5
  }

  scm {
    git {
      remote {
        url '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'
      }

      branch 'origin/master'

      // Add extensions 'SparseCheckoutPaths' and 'PathRestriction'
      def nodeBuilder = NodeBuilder.newInstance()
      def sparseCheckout = nodeBuilder.createNode(
            'hudson.plugins.git.extensions.impl.SparseCheckoutPaths')
      sparseCheckout
            .appendNode('sparseCheckoutPaths')
            .appendNode('hudson.plugins.git.extensions.impl.SparseCheckoutPath')
            .appendNode('path', 'library1/')
      def pathRestrictions = nodeBuilder.createNode(
            'hudson.plugins.git.extensions.impl.PathRestriction')
      pathRestrictions.appendNode('includedRegions', 'library1/.*')
      extensions {
        extensions << sparseCheckout
        extensions << pathRestrictions
      }
    }
  }

  rootPOM 'library1/pom.xml'
  goals 'clean install'
}Code language: Groovy (groovy)

Instead of the removed Branch parameter, we add the parameters releaseVersion and nextSnapshotVersion (below logRotator):

parameters {
  stringParam('releaseVersion', '',
        'The release version for the artifact. If you leave this empty, ' +
              'the current SNAPSHOT version will be used with the ' +
              '"-SNAPSHOT" suffix removed (example: if the current version ' +
              'is "1.0-SNAPSHOT", the release version will be "1.0").')
  stringParam('nextSnapshotVersion', '',
        'The snapshot version to be used after the release. If you leave ' +
              'this empty, the minor version of the release will be ' +
              'incremented by one (example: if the release is "1.0", the ' +
              'next snapshot version will be "1.1-SNAPSHOT").')
}Code language: Groovy (groovy)

The following code adds the source code management action "Check out to specific local branch" with the branch name "master". Without it, the Maven goal scm:checkin would fail. We insert the code block at the end of the git block, directly after configure:

extensions {
  localBranch('master')
}Code language: Groovy (groovy)

Now we add the pre-build step of the type "Execute system Groovy script", which sets the variables releaseVersion and nextSnapshotVersion if the user did not fill them when starting the job. We insert the following code after the scm block:

preBuildSteps {
  systemGroovyCommand '''
    import hudson.model.StringParameterValue
    import hudson.model.ParametersAction

    def env = build.getEnvironment(listener)
    String releaseVersion = env.get('releaseVersion')
    String nextSnapshotVersion = env.get('nextSnapshotVersion')

    if (!releaseVersion) {
      String pomPath = build.workspace.toString() + '/library1/pom.xml'
      def pom = new XmlSlurper().parse(new File(pomPath))
      releaseVersion = pom.version.toString().replace('-SNAPSHOT', '')
      println "releaseVersion (calculated) = $releaseVersion"
      def param = new StringParameterValue('releaseVersion', releaseVersion)
      build.replaceAction(new ParametersAction(param))
    }

    if (!nextSnapshotVersion) {
      def tokens = releaseVersion.split('\\.')
      nextSnapshotVersion =
            tokens[0] + '.' + (Integer.parseInt(tokens[1]) + 1) + '-SNAPSHOT'
      println "nextSnapshotVersion (calculated) = $nextSnapshotVersion"
      def param1 = new StringParameterValue('releaseVersion', releaseVersion)
      def param2 = new StringParameterValue('nextSnapshotVersion',
            nextSnapshotVersion)
      build.replaceAction(new ParametersAction(param1, param2))
    }
    '''.stripIndent()
}Code language: Groovy (groovy)

The three pre-build steps for setting the version number, switching to release dependencies and checking for SNAPSHOT versions in the pom.xml follow right after systemGroovyCommand within the preBuildSteps block.

maven {
  mavenInstallation 'Latest'
  goals 'versions:set ' +
        '-DnewVersion=${releaseVersion} ' +
        '-DgenerateBackupPoms=false'
  rootPOM "library1/pom.xml"
}

maven {
  mavenInstallation 'Latest'
  goals 'versions:use-releases ' +
        '-DgenerateBackupPoms=false ' +
        '-DprocessDependencyManagement=true'
  rootPOM "library1/pom.xml"
}

shell '''
      if find library1/ -name 'pom.xml' | xargs grep -n "SNAPSHOT"; then
        echo 'SNAPSHOT versions not allowed in a release'
        exit 1
      fi
      '''.stripIndent()Code language: Groovy (groovy)

Lastly, we create the four post-build steps: committing the release version to Git, tagging the release version, changing the version number to the next snapshot version, and committing the snapshot version to Git. We insert the posts-build steps at the very end of the file, behind goals:

postBuildSteps {
  maven {
    mavenInstallation 'Latest'
    goals 'scm:checkin ' +
          '-Dmessage="Release version ' +
                '${project.artifactId}:${releaseVersion}" ' +
          '-DdeveloperConnectionUrl=scm:git:' +
                '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'
    rootPOM "library1/pom.xml"
  }

  maven {
    mavenInstallation 'Latest'
    goals 'scm:tag ' +
          '-Dtag=${project.artifactId}-${releaseVersion} ' +
          '-DdeveloperConnectionUrl=scm:git:' +
                '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'
    rootPOM "library1/pom.xml"
  }

  maven {
    mavenInstallation 'Latest'
    goals 'versions:set ' +
          '-DnewVersion=${nextSnapshotVersion} ' +
          '-DgenerateBackupPoms=false'
    rootPOM "library1/pom.xml"
  }

  maven {
    mavenInstallation 'Latest'
    goals 'scm:checkin ' +
          '-Dmessage="Switch to next snapshot version: ' +
                '${project.artifactId}:${nextSnapshotVersion}" ' +
          '-DdeveloperConnectionUrl=scm:git:' +
                '[email protected]:SvenWoltmann/jenkins-tutorial-demo.git'
    rootPOM "library1/pom.xml"
  }
}Code language: Groovy (groovy)

Since the finished code of the ReleaseJobLibrary1.groovy is quite long, I do not display it here. You can find it in my GitLab-Repository. We import and execute the code in the same way as the previous projects. Jenkins creates a new job "Jenkins Tutorial Demo - Library 1 - Release (DSL)":

We put two browser windows next to each other again and compare the manually with the automatically created job. We are satisfied that the jobs are almost identical. Only the "Additional Behaviours" in Source Code Management have a different order. However, that is irrelevant.

Likewise, we create a file ReleaseJobApplication1.groovy for the "application1" project, by copying the previously created file and replacing "library" with "application" in the code. All in all, it takes me about a minute to create the release job for the application, import it into Jenkins, and run it.

Interim conclusion after creating Jenkins jobs as code

We have created a total of four Groovy files. However, it is noticeable that we duplicated many lines of code – a fact that violates the "Don't Repeat Yourself" principle of programming. How can we prevent this? We can't do that with the way we imported the code into Jenkins. We now want to put this method behind us – it was only an interim solution.

In the following chapters, I'll show you how to get Jenkins to load the code from the Git repository and how to extract shared code.

Organizing Jenkins jobs in views

The job list has become quite long and cluttered. Fortunately, Jenkins offers the possibility to organize the jobs in so-called "views". These views are displayed as tabs at the top of the job list.

Manually creating a Jenkins view

So that we did it once by hand, we manually create a view for the four jobs from the first article of the series. To do this, we click on the plus sign next to the "All" tab:

We enter "Part I" as the name and select "List View" as type. We continue to the next step with "OK".

We select the four manually created jobs from the first article and click "OK":

The first view, "Part I" is now finished:

Creating a Jenkins view as code

Of course, we also create the view for the jobs created in this article as code. We create a file ViewDSLJobs.groovy with the following content in the jobs/ directory of the Git repository:

listView('Part II - DSL') {
  jobs {
    regex '.+\(DSL\).*'
  }

  columns {
    status()
    weather()
    name()
    lastSuccess()
    lastFailure()
    lastDuration()
    buildButton()
  }
}Code language: Groovy (groovy)

We do not specify any concrete jobs in this view. Instead, we use a regular expression that filters all jobs that contain the string "(DSL)". The columns() function specifies which columns we want to display in the view. If you omit this function, you won't see a standard selection of columns – you would rather see none at all. We create another creator job, call it "View Creator (DSL)", copy the DSL code into it and execute it. With a few lines of code and a few clicks the new View "Part II - DSL" is ready:

Summary and outlook

In this part, we created the build and release jobs from the first part programmatically with the Jenkins Job DSL. We also made the list of jobs clearer by creating views – first manually and later programmatically.

In the third and last part, I will show you how to extract repeatedly used Groovy code into utility classes (and how to configure the script security system to allow this). I will also show you how to create a so-called "seed job" that generates Jenkins jobs for all Java projects in the Git repository (including new ones) fully automatically.

If you liked this article, feel free to leave a comment and to share it via one of the share buttons at the end.

Do you want to be informed when new articles are published on HappyCoders.eu? Then click here to sign up for HappyCoders.eu newsletter.