GitLab CI CD Include

GitLab CI/CD – Include

by

This article covers the include keyword in GitLab CI/CD, detailing its purpose in breaking down large .gitlab-ci.yml files into smaller, manageable, and reusable components. We will cover various inclusion methods like local, remote, template, and project includes, along with best practices for structuring your CI/CD configuration to enhance readability, maintainability, and reusability.

Understanding the include Keyword in GitLab CI/CD

The include keyword in GitLab CI/CD that allows you to compose your .gitlab-ci.yml configuration from multiple files. Instead of having one monolithic file, include enables you to break down your CI/CD logic into smaller, more focused, and reusable components. This significantly improves the readability, maintainability, and scalability of your pipelines, especially for large or complex projects.

Why Use the include Keyword?

The primary motivations for using include are:

  • Modularity: Decompose a complex CI/CD configuration into smaller, independent files, each responsible for a specific aspect (e.g., build, test, deploy).
  • Reusability: Define common job definitions, templates, or stages once and reuse them across multiple projects or even within the same project.
  • Maintainability: Easier to update and debug specific parts of the pipeline without affecting the entire configuration.
  • Readability: Smaller files are easier to understand and navigate.
  • Version Control: Include files can be version-controlled independently, allowing for better collaboration and tracking changes.

Types of include

GitLab CI/CD supports several ways to include external YAML files, catering to different use cases:

Local Include

This is the simplest form, used to include files that reside within the same project repository as your main .gitlab-ci.yml file.

# .gitlab-ci.yml
include:
  - local: '/.gitlab-ci/build_template.yml'
  - local: '/.gitlab-ci/test_jobs.yml'

stages:
  - build
  - test
  - deploy

# Other jobs defined directly in this file
# .gitlab-ci/build_template.yml
.build_template:
  stage: build
  script:
    - echo "Running build script from template"
  tags:
    - docker
# .gitlab-ci/test_jobs.yml
unit_test:
  stage: test
  script:
    - echo "Running unit tests"

integration_test:
  stage: test
  script:
    - echo "Running integration tests"
  • Paths: The local path is relative to the root of your project directory.
  • Use Cases: Ideal for breaking down a large project’s CI/CD configuration into domain-specific files (e.g., frontend, backend, database migrations) or separating common job definitions.

Remote Include

This allows you to include files from any publicly accessible URL. This is particularly useful for including common CI/CD configurations shared across different projects, possibly hosted outside your GitLab instance, or from raw URLs of files in other version control systems.

# .gitlab-ci.yml
include:
  - remote: 'https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/library/-/raw/main/gitlab_ci_templates/Deploy-ECS.gitlab-ci.yml'
  - remote: 'https://example.com/ci-configs/common_security_checks.yml' # Example of external config
  • URLs: The URL must point directly to the raw content of the YAML file.
  • Security: Be cautious when using remote includes from untrusted sources, as they can execute arbitrary commands in your pipeline.
  • Version Control: It is recommended to include files with a specific commit SHA or tag in the URL to ensure immutability and prevent unexpected changes, rather than using branch names like main.

Template Include

GitLab provides a set of predefined CI/CD templates for common use cases (e.g., Node.js, Maven, Docker, SAST). These templates are maintained by GitLab and are excellent starting points or reusable components.

# .gitlab-ci.yml
include:
  - template: Auto-DevOps.gitlab-ci.yml
  - template: Security/SAST.gitlab-ci.yml # Include Static Application Security Testing
  • Syntax: Just specify the template name. GitLab automatically resolves the path to its internal templates.
  • Discovery: You can find available templates by exploring the GitLab UI when creating a new CI/CD configuration or in the GitLab documentation.
  • Extending Templates: Templates often define base jobs that you can extend or override in your main .gitlab-ci.yml using the extends keyword.

Project Include (Cross-Project Include)

This is the most common and powerful method for sharing CI/CD configurations across multiple projects within the same GitLab instance. You can include files from other projects in your GitLab instance, specifying the project path and the file path within that project.

# .gitlab-ci.yml
include:
  - project: 'my-group/ci-templates' # The path to the project containing the CI configs
    ref: 'main' # The branch, tag, or commit SHA to use
    file: '/templates/build-template.yml'
  - project: 'shared-ci/common-tests'
    file: '/tests/default.yml' # If 'ref' is omitted, it defaults to the project's default branch
  • project: The full path to the GitLab project containing the included file (e.g., group/subgroup/project-name).
  • file: The path to the YAML file within that project, relative to its root.
  • ref: (Optional) The Git reference (branch name, tag, or commit SHA) from which to fetch the file. Using a specific tag or commit SHA is highly recommended for stability.
  • Use Cases: Centralizing an organization’s CI/CD best practices, standardizing pipeline stages, or sharing complex deployment logic across services.

Include with rules (Conditional Include)

You can conditionally include files based on rules. This is useful for dynamically adjusting the pipeline configuration based on specific conditions, like branch names or variable presence.

# .gitlab-ci.yml
include:
  - local: '/.gitlab-ci/feature_branch_jobs.yml'
    rules:
      - if: '$CI_COMMIT_BRANCH =~ /feature\/.*/'
  - project: 'shared-ci/prod-deploy'
    file: '/deploy/prod.yml'
    rules:
      - if: '$CI_COMMIT_BRANCH == "main"'
        when: manual # Only include if on 'main' and requires manual trigger
  • Flexibility: Allows for highly dynamic pipeline configurations based on a wide range of CI/CD variables and logic.

Overriding and Extending Included Configurations

When you include multiple files, GitLab merges them. If there are conflicting keywords (e.g., two jobs with the same name), the last defined keyword takes precedence.

A common pattern is to define base job configurations in included files and then extend or override them in your main .gitlab-ci.yml using the extends keyword or by redefining parts of a job.

# .gitlab-ci/base_jobs.yml
.base_job:
  image: alpine/git
  script:
    - echo "Running base script"
  tags:
    - default-runner

# .gitlab-ci.yml
include:
  - local: '/.gitlab-ci/base_jobs.yml'

my_custom_job:
  extends: .base_job # Inherit properties from .base_job
  stage: build
  script:
    - echo "Running custom script, overriding base script" # Overrides the script
    - echo "Additional custom step"
  variables:
    MY_VAR: "value" # Add new variables

Best Practices for Using include

  • Structure Logically: Group related jobs or configuration sections into separate files. For example, build.yml, test.yml, deploy.yml, security.yml.
  • Use Specific Refs for Project/Remote Includes: Always specify a ref (commit SHA or tag) for project and remote includes to ensure pipeline stability and prevent unexpected failures due to changes in the included files. Avoid using main or master branches unless you explicitly intend for your pipelines to pick up the latest changes.
  • Version Included Files: Treat your included CI/CD files as first-class code, version-controlling them appropriately.
  • Avoid Deep Nesting: While you can include files that in turn include other files, excessive nesting can make the configuration hard to follow and debug. Aim for a flat or shallow hierarchy.
  • Consider a Central CI/CD Project: For organizations, creating a dedicated GitLab project to host all shared CI/CD templates and configurations is a common and effective strategy.
  • Use Variables for Customization: Instead of hardcoding values in included files, use CI/CD variables that can be overridden by individual projects, making the templates more flexible.
  • Test Included Files Independently: If possible, consider having basic pipelines in the projects hosting your included files to ensure they are syntactically correct and functional in isolation.
  • Understand Merge Behavior: Be aware that included configurations are merged. Conflicts are resolved by the last definition overriding previous ones. Use extends for explicit inheritance.

FAQs – GitLab CI/CD include

What is the include keyword in GitLab CI/CD?
The include keyword in GitLab CI/CD allows you to import external YAML configuration files into your main .gitlab-ci.yml file. This helps modularize and reuse pipeline definitions across projects, making your CI/CD setup more maintainable and DRY (Do not Repeat Yourself).

How to use the include keyword to import another .yml file?
You can include other YAML files using several methods. Here is a basic example of including a local file:

include:
  - local: 'ci-templates/test.yml'

This imports the contents of test.yml from within your repository and merges it into the main pipeline.

What are the different types of includes supported in GitLab?
GitLab supports multiple include types:

  • local: Include a file from the same repository.
  • file: Include a file from another project in GitLab.
  • remote: Include a file from an external URL.
  • template: Include a predefined template provided by GitLab.

Here’s an example with all types:

include:
  - local: 'ci/rails.yml'
  - file:
      project: 'mygroup/shared-configs'
      file: '/templates/docker.yml'
      ref: 'main'
  - remote: 'https://example.com/ci/config.yml'
  - template: 'Jobs/SAST.gitlab-ci.yml'

Can I include multiple configuration files in one pipeline?
Yes, you can include multiple files using a list. GitLab will merge them in order:

include:
  - local: 'ci/lint.yml'
  - local: 'ci/deploy.yml'

Each included file should be a valid GitLab CI YAML file with jobs, variables, or rules.

What happens if the included file is invalid or missing?
If the included file is not found or contains syntax errors, the pipeline will fail to compile and show an error before execution starts. This prevents the CI/CD process from running with broken configuration.

How to include a file from a different branch or ref in another project?
Use the ref attribute along with the file type to include a file from a specific branch or commit SHA:

include:
  - file:
      project: 'mygroup/shared-ci'
      file: '/templates/test.yml'
      ref: 'develop'

This pulls the file from the develop branch of the specified project.

Can I conditionally include a file based on the branch or variable?
Yes, you can use rules to conditionally include files:

include:
  - local: 'ci/feature.yml'
    rules:
      - if: '$CI_COMMIT_BRANCH == "feature-branch"'

This will only include the file if the current branch is feature-branch.

How does GitLab merge multiple include files into the main pipeline?
GitLab merges included configurations into the main pipeline config as if they were part of the .gitlab-ci.yml. If there are conflicts (e.g., same job name), the main file’s definitions will override those from included files.

Can I override jobs from included YAML files?
Yes, any job defined in the main .gitlab-ci.yml can override or extend jobs from included files. You can redefine a job by using the same name:

# Included file defines 'build' job

build:
  script:
    - echo "Overridden build job"

This overrides the build job that might have been defined in an included file.

What are GitLab predefined templates and how to use them with include?
GitLab provides built-in templates (e.g., SAST, DAST, Code Quality) that can be included directly:

include:
  - template: 'Security/SAST.gitlab-ci.yml'

These templates contain preconfigured jobs maintained by GitLab and are useful for quickly enabling security scans and compliance checks.

Is there a way to debug which files have been included in the pipeline?
Yes, you can inspect the final composed pipeline by going to CI/CD → Pipelines → View pipeline → View configuration. This shows the combined YAML after all includes and merges have been applied.

Does using include affect pipeline performance?
Using include has minimal impact on performance, but too many remote or cross-project includes may cause slight delays in pipeline compilation. Always ensure included files are optimized and lean.

Author

Debjeet Bhowmik

Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins. In my free time, I write blogs on ckdbtech.com

Leave a Comment