CI/CD with Bitbucket Pipelines. An introduction

Continuous Integration and Continuous Deployment have become almost ubiquitous in software development. Here at WATA Factory, we have set up almost all projects to use at least some kind of automated integration and deployment mechanism. As these practices evolve, understanding how to structure and automate your workflows effectively becomes increasingly important.

In this post, we’ll walk through what you need to know before setting up CI/CD with Bitbucket Pipelines: we’ll clarify the core terminology, explain how Pipelines is structured, and show both simple and more advanced configuration examples that you can apply to your own projects. In short, this article will give you a practical, grounded overview of how Bitbucket Pipelines works and what you can achieve with it.

But first, let us clear up some of the terminology.

What is CI?

CI (Continuous Integration) refers to the practice of automatically building, running tests and static code analysis, as well as (possibly) merging the code.

The magic word here is “automatically”. Executing all the steps to build and test usually takes some time, and having to execute these repetitive steps (or at least a part of them) manually is not only inefficient, but also introduces another possible source for human errors (because us humans just aren’t very good at doing tedious repetitive tasks).

Instead, you set up a process (usually called a “pipeline”) that gets executed automatically every time you push your code to the repository. The developer then gets notified if any of the steps fail and can then fix the problem as soon as possible.

You can also set up different pipelines for different branches. For example, for your feature branch you might not want to execute all the time-consuming integration or end-to-end front-end tests with every push because in a big project, these might take hours to run. To avoid that, you could set up the system in such a way that the time-consuming test steps are only executed when something is merged into the master/release branch (depending on how the branching model is set up in your project).

Generally, the output of the CI stage would typically be some kind of artifact that can then be deployed in an environment (such as a .jar or .war file or a Docker image).

What is CD?

CD stands for Continuous Delivery or Continuous Deployment, and it can be considered an extension of CI. Generally speaking, it is what happens with the code after the CI (building and testing) stage has completed successfully, although it is not always possible to draw a clear line between the two (especially since both processes are typically defined in the same place).

Your CD pipeline would usually take the artifact that has been created in the CI-step and then deploy it in a test or staging environment to make it accessible for the testers. This includes copying files, setting configuration, or restarting the server. It might also include scripts to automatically increase version numbers or deploying artifacts to common repositories. You could even go further and automatically deploy your application in production, if you wish.

How to set up your CI/CD pipelines always depends on the needs of the project, and no two projects are the same.

In summary, although it is a bit more work up front to set up a CI/CD pipeline, it will pay off over time, because it automates some of the repetitive tasks, and will also catch bugs earlier because important tests can be run on every push.

Some Alternative CI/CD platforms

At WATA Factory, we use Bitbucket Pipelines mostly because it integrates seamlessly with other Atlassian products like Jira and Confluence that we use every day.

There are, however, quite a few different CI/CD frameworks besides Bitbucket Pipeline. Let’s look at some of the most popular alternatives:

• GitHub Actions
  • Native to GitHub repositories
  • Big community and easy to use
  • Good for open source projects
• GitLab CI/CD
  • Fully integrated with GitLab
  • Built-in container registry and security scanning
  • Used heavily in enterprises and DevOps-centric teams
• Jenkins
  • Completely open source
  • Probably one of the oldest participants in this list
  • Self-Hosted
  • Highly customizable, but more complex
  • Often used for complex, self-hosted enterprise setups
• CircleCI
  • Uses Docker containers or virtual machines
  • Very fast parallelization and caching
  • Good Docker and Kubernetes support
• Azure DevOps Pipelines
  • Deep integration with Microsoft and Azure ecosystem -> good for Microsoft stacks
  • Supports YAML piplines, but also offers a GUI pipeline

Setting up a Simple Bitbucket Pipeline

To use Bitbucket Pipelines, you obviously need a Bitbucket account, and your code uploaded to it.

Enable Pipelines

In order to use pipelines for a repository, you first have to enable them through the Bitbucket web interface. Go to your Bitbucket repository, click on Repository settings, find the Pipelines section in the left-hand menu, click Settings and toggle Enable pipelines. Until you do this, none of the following steps will have any effect.

The bitbucket-pipelines.yml

All the configuration for a Bitbucket Pipeline resides in a single YAML file, bitbucket-pipelines.yml, which has to be located at the root of your repository. That means, of course, that if you have your project divided over several repositories (Frontend and Backend, for example), you will have a separate pipeline and a bitbucket-pipelines.yml for each of your repositories.

All bitbucket-pipelines.yml files have a certain structure that defines the steps, environments, and conditions of the CI/CD process. The key elements are as following:

• image – the Docker image that the steps run inside of (could be a standard image for the runtime environment or a custom image that emulates the dev- or prod-environment)
• pipelines – the main block and top-level-element that defines when and how steps are run. Some of the useful properties are:
  • branches – will be run on every commit for branches that have a certain name or follow a certain naming pattern (e.g. “master” or “*/feature*”)
  • pull-requests – runs when a pull-request for a certain branch is created. Also allows restricting to certain naming patterns, similar to “branches”
  • tags – runs for certain tags in the Git repository
  • default – As the name suggests, this is a fallback option. Will run for any commit on any branch that is not covered by any of the other properties above
  • custom – use for manually or scheduled pipelines
• step – a unit of execution (for example “build and test”) that is defined by shell commands. This is where the meat is. A step may contain the following (among many others):
  • script – the actual commands to be executed in a step
  • caches – directories (for example “node_modules”) to be cached between runs, which can significantly speed up runs, because not all dependencies have to be downloaded very time
  • artifacts – files generated in a step (for example a .jar file) that can then be used in another
  • deployment – marks a step as a deployment (it will appear in Bitbucket’s “Deployments” UI)
  • pipe – a custom Docker image for a container, which contains a script to perform a task. Think of them as prebuilt “parts” for your pipeline that are particularly useful for integrating with third-party tools.

A Simple Example

The Bitbucket web interface actually provides some templates for typical configurations, but for this article, we will create the bitbucket-pipelines.yml by hand. Since I am a Java guy, the following example will be for a Java Maven project. This is about the simplest example possible:

image: maven:3.9.6-eclipse-temurin-17
pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - maven
        script:
          - mvn -B clean install

Let’s look at what this does:

• The first line instructs the pipeline to use JDK 17 with Maven 3.9
• The default keyword tells us that this pipeline is going to run on every commit, for every branch
• We cache the “maven” folder, so that future builds will be much faster
• Finally, we run the maven command that cleans, downloads dependencies (if necessary), compiles, and runs unit tests

So, even with this little code we already have a pipeline that builds the code and runs the tests automatically on every commit. It will also email you when any of the steps fail.

A More Complex Example

The following example shows a more complex pipeline that has separate and independent steps for building, running unit tests, running integration tests, packaging, and deploying the code. While building and running unit tests will be performed on every commit, integration tests, packaging and deployment will only be done for release branches.

image: maven:3.9.6-eclipse-temurin-17

pipelines:
  default:
    - step:
        name: Build (Compile Only)
        caches:
          - maven
        script:
          - mvn -B -DskipTests compile
        artifacts:
          - target/**
    - step:
        name: Unit Tests
        caches:
          - maven
        script:
          - mvn -B test
        artifacts:
          - target/**
  
  branches:
    'release/*':
      - step:
          name: Build (Compile Only)
          caches:
            - maven
          script:
            - mvn -B -DskipTests compile
          artifacts:
            - target/**
      - step:
          name: Unit Tests
          caches:
            - maven
          script:
            - mvn -B test
          artifacts:
            - target/**
      - step:
          name: Integration Tests
          caches:
            - maven
          script:
            - mvn -B verify -DskipUnitTests
          artifacts:
            - target/**
      - step:
          name: Package as JAR
          caches:
            - maven
          script:
            - mvn -B package -DskipTests
          artifacts:
            - target/*.jar
      - step:
          name: Deploy
          deployment: Development
          script:
            - pipe: atlassian/scp-deploy:1.6.0
              variables:
                USER: $DEPLOY_USER
                SERVER: $DEPLOY_HOST
                REMOTE_PATH: "/var/deploy"
                LOCAL_PATH: "target/*.jar"

You will first notice that we have two blocks below pipelines: a default block (that gets executed on every commit) and a branches block that is only executed for release branches (which we have defined to be all in the release/ directory).

Also note that the Build (Compile only) and the Unit tests steps are repeated (because we want to execute them for release branches as well).

Besides the build and unit test steps, the branches section also executes the integration tests and packages the application in a JAR file. These steps will only be executed for release branches.

Most interesting is the last step which uses the scp-deploy pipe. This pipe (as the name suggests) takes the artifacts passed in from the previous step and copies them to a remote server using scp. For that to work, it needs the information provided in the variables section. The values prefixed with a ‘$’ are variables that are defined as repository variables in the bitbucket web interface (because you don’t want to upload sensible data to your repository).

Conclusion

CI/CD is an indispensable tool in today’s development. At WATA factory, we use Bitbucket Pipelines for this purpose. As shown in this article, it is easy to set up a CI/CD pipeline (although in some projects, they can become very complex).
Besides all the mentioned advantages of using CI/CD, Bitbucket Pipelines also integrates very well with other Atlassian products, and id therefore our tool of choice.