GitLab CI/CD – Secrets

GitLab CI/CD – Secrets

by

This article covers the secrets keyword in GitLab CI/CD, explaining its utility in securely passing external secrets (like those from HashiCorp Vault) into your pipeline jobs. We will explore how to define secret variables, specify their providers, and manage their access within your CI/CD configuration, ensuring sensitive information remains protected while enabling secure interactions with external services.

Understanding the secrets Keyword in GitLab CI/CD

The secrets keyword in GitLab CI/CD is used to define external secrets that a job requires. Unlike regular CI/CD variables (even protected ones), the secrets keyword is specifically designed for integration with external secret management systems, primarily HashiCorp Vault. It allows you to inject sensitive data from these external sources into your pipeline jobs in a controlled and secure manner, without exposing them directly in your .gitlab-ci.yml file or as regular environment variables.

Why Use the secrets Keyword?

Using the secrets keyword offers significant advantages for security and secret management:

  • Enhanced Security: Prevents sensitive data from being hardcoded or directly exposed in your .gitlab-ci.yml file or in job logs.
  • Centralized Secret Management: Integrates with dedicated secret management systems (like Vault), which are designed for secure storage, access control, and auditing of secrets.
  • Dynamic Secret Generation: Allows for dynamic generation of credentials (e.g., short-lived database credentials), improving security by reducing the lifetime of secrets.
  • Reduced Secret Sprawl: Avoids scattering secrets across various CI/CD variables or environments.
  • Compliance: Helps meet compliance requirements for handling sensitive information.

Configuring secrets in .gitlab-ci.yml

The secrets keyword is defined at the job level. When a job with secrets runs, GitLab contacts the configured secret management provider (e.g., Vault), retrieves the specified secrets, and injects them as environment variables into the job’s execution environment.

Basic secrets Definition (HashiCorp Vault Example)

To use the secrets keyword, you first need to have GitLab integrated with HashiCorp Vault. This involves configuring the Vault server and the GitLab Runner to communicate with it. Once configured, you can define secrets like this:

# .gitlab-ci.yml

production_deploy:
  stage: deploy
  script:
    - echo "Deploying to production using secrets..."
    - >
      # Access secrets as environment variables
      # VAULT_DB_USERNAME and VAULT_DB_PASSWORD are automatically set
      # by GitLab from the Vault path
      ./deploy_app.sh "$VAULT_DB_USERNAME" "$VAULT_DB_PASSWORD"
  secrets:
    VAULT_DB_USERNAME:
      vault: secret/data/production/webapp/db:username # Path to secret in Vault
    VAULT_DB_PASSWORD:
      vault: secret/data/production/webapp/db:password # Path to secret in Vault
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual # Only deploy to prod manually from main branch

In this example:

  • VAULT_DB_USERNAME and VAULT_DB_PASSWORD: These are the local environment variable names that will be created in the job’s environment. You can choose any valid variable name.
  • vault: This indicates that the secret is coming from a HashiCorp Vault integration.
  • secret/data/production/webapp/db:username: This is the path to the secret in Vault, followed by a colon : and the key within that secret whose value you want to retrieve.
    • secret/data/production/webapp/db: The full path to the secret in your Vault.
    • username: The specific key within that secret.

When production_deploy job runs, GitLab will:

  1. Authenticate with the configured HashiCorp Vault server.
  2. Read the secret at secret/data/production/webapp/db.
  3. Extract the values for username and password.
  4. Inject these values as environment variables VAULT_DB_USERNAME and VAULT_DB_PASSWORD into the job’s shell environment.

Defining Multiple Secrets and Providers

You can define multiple secrets under the secrets keyword. While currently primarily used with HashiCorp Vault, the structure is designed to support other secret providers in the future.

deploy_with_multiple_secrets:
  stage: deploy
  script:
    - echo "Using multiple secrets from Vault..."
    - ./configure_service.sh "$VAULT_API_KEY" "$VAULT_QUEUE_ACCESS_TOKEN"
  secrets:
    VAULT_API_KEY:
      vault: production/api/service:api_key
    VAULT_QUEUE_ACCESS_TOKEN:
      vault: common/messaging/queue:access_token

Secret Defaults (default)

(GitLab Premium/Ultimate) You can define default settings for secrets at the top level of your .gitlab-ci.yml, which jobs can then override or extend.

# .gitlab-ci.yml

secrets:
  default:
    vault:
      engine: 'kv-v2' # Specify the Vault KV engine version
      path: 'secret/data' # Default base path in Vault

stages:
  - deploy

deploy_backend:
  stage: deploy
  script:
    - echo "Deploying backend with default secret path..."
    - ./deploy_backend.sh "$DB_CONN_STR"
  secrets:
    DB_CONN_STR:
      vault: 'backend/config:connection_string' # This path will be relative to default.path if engine is 'kv-v2' and path isn't absolute.
      # Full path would be secret/data/backend/config:connection_string

The secrets:default block, especially for vault, allows you to configure global parameters for Vault integration, such as the engine (e.g., kv-v2 for KV Secret Engine – Version 2) and path (a default base path for secrets). This reduces repetition in individual job definitions.

How Secrets are Accessed in Jobs

Once defined with secrets, the values are injected as environment variables into the job’s shell. This means you can access them using standard shell syntax (e.g., $VAULT_DB_USERNAME in Bash, %VAULT_DB_USERNAME% in Windows Batch).

GitLab runners ensure that these variables are:

  • Not exposed in job logs: Their values are masked.
  • Available only during the job’s execution: They are not persisted after the job completes.
  • Accessible within the job’s script context: Any command or script executed by the job can use them.

Pre-requisites and Setup

To use the secrets keyword with HashiCorp Vault:

  1. HashiCorp Vault Server: You need a running and accessible HashiCorp Vault server.
  2. GitLab Instance/Project Integration: Your GitLab instance (or specific project) must be configured to integrate with HashiCorp Vault. This typically involves:
    • Enabling the Vault integration in GitLab settings.
    • Configuring the Vault address and an authentication method (e.g., JWT, AppRole).
    • Ensuring the GitLab Runner has network access to the Vault server.
    • Configuring Vault policies to grant GitLab read access to the necessary secret paths.
  3. Secrets Stored in Vault: The secrets you reference in your .gitlab-ci.yml must exist at the specified paths and keys within your Vault.

Best Practices for Using secrets

  • Integrate with a Dedicated Secret Manager: Always use a dedicated secret management solution like HashiCorp Vault for sensitive data rather than relying solely on CI/CD variables for highly critical information.
  • Least Privilege: Configure Vault policies to grant GitLab (and specifically, the pipeline’s identity) only the minimum necessary read permissions to the secret paths it requires.
  • Use Specific Paths/Keys: Reference specific secret paths and keys within Vault to minimize the scope of access. Avoid overly broad permissions.
  • Masking in Logs: While secrets are automatically masked, be careful not to echo their values explicitly in your script unless absolutely necessary for debugging and you understand the risks.
  • Short-Lived Credentials (Dynamic Secrets): If your Vault configuration supports it, use dynamic secrets (e.g., dynamically generated database credentials) to further reduce the risk associated with long-lived credentials.
  • Regular Auditing: Regularly audit access to your secrets in Vault and monitor GitLab CI/CD job logs for any suspicious activity.
  • Consider Environments: Scope your secrets in Vault by environment (e.g., production/db-creds, staging/db-creds) to align with your deployment strategy and prevent accidental use of production secrets in lower environments.

FAQs – GitLab CI/CD Secrets

What is the secrets keyword in GitLab CI/CD?
The secrets keyword in GitLab CI/CD is used to securely inject secrets (like passwords, credentials, API keys, tokens, etc.) into your job at runtime. These secrets are managed using GitLab’s built-in Vault integration, allowing access to ephemeral, scoped, and revocable secret values.

How is secrets different from variables in GitLab CI/CD?

  • variables: Stored in GitLab (project/group-level), encrypted at rest, but exposed as environment variables during job execution.
  • secrets: Pulled from an external secrets manager (Vault) and are not stored in GitLab. Secrets are dynamically retrieved and safer for sensitive data.

How do I configure secrets in a GitLab CI/CD job?
To use secrets, you must first connect GitLab to HashiCorp Vault (or use GitLab’s internal Vault). Then define secrets in your job like this:

deploy:
  stage: deploy
  script:
    - echo "$MY_SECRET_VAR"
  secrets:
    MY_SECRET_VAR:
      vault: production/db/password

Here, MY_SECRET_VAR will hold the secret fetched from the Vault path production/db/password.

Do I need to install Vault separately to use secrets in GitLab?
No. GitLab provides a built-in Vault integration (available in GitLab Premium/Ultimate and GitLab.com).
You need to enable it in project settings under CI/CD → Vault and configure access policies.

How are secrets injected into a job using the secrets keyword?
Secrets defined under the secrets: block are injected as environment variables just like CI/CD variables. However, they are retrieved at job runtime, reducing the risk of exposure.

Can I use secrets from an external Vault server (self-managed)?
Yes. GitLab supports integration with external Vault servers. You must:

  1. Enable JWT authentication in Vault.
  2. Configure policies to allow secret access.
  3. Register your Vault details in GitLab’s project settings.

Can I use secrets in all GitLab plans?
No. The secrets keyword requires:

  • GitLab Premium (self-managed)
  • GitLab.com Premium or Ultimate
  • GitLab 14.4+ for basic support, full features in later versions

It is not available on the Free tier.

What happens if a secret is not accessible or Vault is misconfigured?
If the Vault integration fails or the secret path is inaccessible:

  • The job will fail with an error
  • GitLab Runner logs will indicate the problem
  • You should check Vault policies, paths, and integration settings

Can I specify multiple secrets in a job?
Yes. You can list multiple secret variables under the secrets block:

job:
  script:
    - echo "Using secrets"
  secrets:
    AWS_ACCESS_KEY:
      vault: aws/creds/key
    AWS_SECRET_KEY:
      vault: aws/creds/secret

Each secret is injected as its own environment variable.

Can I limit which jobs can access specific secrets?
Yes. Secrets are job-scoped, and access is defined by Vault policies. You can:

  • Restrict which paths a job token can access in Vault.
  • Use dynamic path templating (e.g., use $CI_JOB_ID, $CI_COMMIT_BRANCH).

This enhances security by minimizing access exposure.

Are secrets masked in GitLab job logs?
Yes. Secrets injected using the secrets keyword are automatically masked in logs, just like masked CI/CD variables. However, you should still avoid echoing them.

Can I use secrets in child pipelines or included YAML files?
Yes. The secrets keyword is fully supported in all job contexts, including:

  • Child pipelines (trigger)
  • External includes
  • Dynamic .gitlab-ci.yml generation

Just ensure that Vault access is properly configured across pipeline scopes.

What is an example of a complete .gitlab-ci.yml using secrets with Vault?

stages:
  - deploy

deploy_prod:
  stage: deploy
  script:
    - echo "Deploying using secret: $DB_PASSWORD"
  secrets:
    DB_PASSWORD:
      vault: production/database/password

Assuming Vault contains a secret at production/database/password, the value is injected as DB_PASSWORD.

Can I use secrets with Kubernetes or Docker runners?
Yes. The secrets keyword works with all runner types, as long as Vault integration is configured. For Kubernetes runners, this is often paired with service accounts or injected tokens.

Author

Debjeet Bhowmik

Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins. In my free time, I write blogs on ckdbtech.com

Leave a Comment