GitLab CI/CD - Needs

GitLab CI/CD – Needs

by

In GitLab CI/CD, the traditional way pipelines execute is in stages: all jobs within a stage run in parallel, and all jobs in a stage must complete successfully before the next stage begins. While this sequential stage execution works well for many common workflows (build -> test -> deploy), it can become a bottleneck for more complex scenarios or highly optimized pipelines.

This is where the needs keyword is useful. The needs keyword allows you to break free from the strict stage-by-stage execution flow. It enables you to define explicit dependencies between jobs, creating a Directed Acyclic Graph (DAG) of your pipeline. This means a job can start as soon as its specific dependencies are met, regardless of what stage those dependencies belong to.

The Limitation of Stages (and Why needs Helps)

Consider a common pipeline:

stages:
  - build
  - test
  - deploy

build_app:
  stage: build
  script: echo "Building app..."

unit_tests:
  stage: test
  script: echo "Running unit tests..."

integration_tests:
  stage: test
  script: echo "Running integration tests..."

deploy_staging:
  stage: deploy
  script: echo "Deploying to staging..."

In this setup:

  1. build_app runs.
  2. Once build_app is finished, unit_tests and integration_tests run in parallel.
  3. Only when both unit_tests and integration_tests are completed (even if unit_tests finishes much earlier), does deploy_staging begin.

What if deploy_staging only requires the output of build_app and unit_tests, but not integration_tests? With stages alone, deploy_staging still waits for integration_tests to finish. This creates an unnecessary delay.

This is where needs comes into picture.

How needs Works

The needs keyword is defined within a job and takes a list of other job names that must complete successfully before the current job can start.

Syntax:

job_name:
  stage: <any_stage_name> # Stage still required, but primarily for visualization/grouping
  needs:
    - job_dependency_1
    - job_dependency_2
  script:
    - # ... commands

When a job has needs, it will:

  1. Start as soon as its needs are met: It does not wait for its entire stage to start, nor for previous stages to finish (unless those are implicitly part of its needs graph).
  2. Pass artifacts automatically: If the jobs listed in needs produce artifacts, those artifacts are automatically downloaded and made available to the needing job. This is a crucial benefit over manual artifact passing.
  3. Visualized as a DAG: In the GitLab UI, pipelines using needs are beautifully visualized as a directed acyclic graph, showing the precise flow of dependencies.

Practical Examples of needs

Let us revisit our example with needs:

stages:
  - build
  - test
  - deploy
  - qa # New stage for QA tasks

build_app:
  stage: build
  script: echo "Building app and creating artifact..."
  artifacts:
    paths:
      - build/app.jar # Output artifact

unit_tests:
  stage: test
  script: echo "Running unit tests..."
  needs: ["build_app"] # Requires build_app to finish
  artifacts:
    paths:
      - test_results/unit.xml # Unit test results

integration_tests:
  stage: test
  script: echo "Running integration tests (takes longer)..."
  needs: ["build_app"] # Also requires build_app
  artifacts:
    paths:
      - test_results/integration.xml # Integration test results

deploy_staging:
  stage: deploy
  script: echo "Deploying app and unit test results to staging..."
  needs: ["build_app", "unit_tests"] # Only needs build_app and unit_tests
  # No need to explicitly download artifacts, they are available

qa_smoke_tests:
  stage: qa
  script: echo "Running QA smoke tests on deployed staging environment..."
  needs: ["deploy_staging"] # Needs staging deployment to be done

full_regression_tests:
  stage: qa
  script: echo "Running full regression tests (requires all tests to pass first)..."
  needs: ["integration_tests", "unit_tests"] # Needs all tests to complete

What happens here with needs:

  • build_app starts immediately.
  • As soon as build_app finishes:
    • unit_tests starts.
    • integration_tests starts.
  • As soon as unit_tests finishes (it might finish before integration_tests):
    • deploy_staging can start, because build_app is also done. It does not wait for integration_tests.
  • qa_smoke_tests waits for deploy_staging.
  • full_regression_tests waits for both unit_tests and integration_tests to complete.

This significantly speeds up the pipeline by allowing deploy_staging to proceed concurrently with the (potentially longer) integration_tests.

Advanced needs Features: Fine-Grained Control

The needs keyword offers several additional attributes to provide even more specific control over job dependencies and artifact handling.

needs:artifacts (or artifacts: false)

Purpose: By default, when a job needs another job, all artifacts from the needed job are downloaded. Use artifacts: false within a needs entry if the needing job does not require the artifacts from a specific dependency, saving time and disk space.

Syntax:

job_name:
  needs:
    - job: <needed_job_name>
      artifacts: false

Example:

lint_code:
  stage: lint
  script: echo "Linting code..."
  artifacts:
    paths:
      - lint_report.txt # Produces a report, but not needed by build

build_app:
  stage: build
  script: echo "Building application..."
  needs:
    - job: lint_code
      artifacts: false # build_app only needs lint_code to pass, not its report

Explanation: build_app will wait for lint_code to complete successfully, but it would not download lint_report.txt, making the build_app job start faster if that report is large.

needs:optional

Purpose: If a job listed in needs is set with allow_failure: true, you can mark it as optional: true within the needs definition. The dependent job will then run even if the optional needed job fails. If the optional job passes, the dependent job also runs.

Syntax:

job_name:
  needs:
    - job: <needed_job_name>
      optional: true

Example:

security_scan:
  stage: security
  script: echo "Running optional security scan..."
  allow_failure: true # This job would not fail the pipeline if it finds issues
  artifacts:
    paths: [security_report.json]

deploy_review_app:
  stage: deploy
  script: echo "Deploying review app, even if security scan has warnings..."
  needs:
    - job: security_scan
      optional: true # This job runs even if security_scan fails due to allow_failure: true
      artifacts: true # Still get the security report if it exists

Explanation: deploy_review_app will start once security_scan completes, regardless of whether security_scan passed or failed (as long as security_scan had allow_failure: true).

needs:project and needs:pipeline:job (Cross-Project Pipeline Dependencies)

Purpose: These advanced features allow a job in one pipeline to depend on a job in a different project’s pipeline. This is crucial for microservice architectures or monorepos where components are built and tested in separate projects.

needs:project: Specifies the project path where the dependent job resides.

needs:pipeline:job: Specifies the exact job name in the dependent project’s pipeline.

Syntax:

job_name:
  needs:
    - project: <group>/<subgroup>/<project_path>
      ref: <ref_name> # Optional: specific branch/tag/commit in the other project
      job: <job_name_in_other_project>
      artifacts: true/false # Optional: whether to download artifacts from the other job

Example:

deploy_frontend:
  stage: deploy
  script: echo "Deploying frontend using artifact from backend build..."
  needs:
    - project: my-group/backend-service # Dependent on a job in 'backend-service' project
      ref: main # Depend on the 'main' branch of the backend
      job: build_backend_api # Specifically on the 'build_backend_api' job
      artifacts: true # Download its artifacts

Explanation: deploy_frontend will only start after the build_backend_api job in the my-group/backend-service project’s pipeline (specifically from its main branch) has successfully completed. This automatically fetches the build_backend_api‘s artifacts.

needs:parallel:matrix

Purpose: When using parallel:matrix to create multiple jobs from a single job definition, needs can specifically target these generated jobs. This ensures dependencies are correctly mapped to each matrix combination.

Syntax:

needs:
  - job: <job_name>
    artifacts: true/false
    optional: true/false
    # For parallel:matrix jobs, specify the matrix variables
    variables:
      - VAR_1: value_A
      - VAR_2: value_B

Example:

build_matrix_job:
  stage: build
  parallel:
    matrix:
      - OS: [linux, windows]
        ARCH: [amd64, arm64]
  script: echo "Building on $OS with $ARCH"
  artifacts:
    paths:
      - build_$OS_$ARCH/

test_specific_build:
  stage: test
  script: echo "Testing Linux AMD64 build..."
  needs:
    - job: build_matrix_job
      artifacts: true
      variables:
        OS: linux
        ARCH: amd64 # This job specifically needs the 'linux-amd64' build from the matrix

Explanation: test_specific_build will wait only for the build_matrix_job that ran with OS: linux and ARCH: amd64 to complete successfully, and it will automatically get its specific artifacts.

When to Use needs

  • Speed Optimization: When you have long-running jobs that are not strict prerequisites for subsequent critical path jobs. needs helps reduce overall pipeline duration by running independent jobs concurrently.
  • Complex Dependencies: For pipelines where the flow is not a simple linear progression through stages, but rather a branching or converging set of tasks.
  • Resource Management: If certain jobs require specialized runners or resources, needs can help ensure those resources are utilized efficiently by triggering jobs only when their exact prerequisites are met.
  • Monorepos & Microservices: Cross-project dependencies (needs:project) are invaluable for managing complex build and deployment flows across multiple repositories.

Best Practices for needs

  1. Keep Stages for Visualization: While needs breaks sequential stage execution, jobs still require a stage keyword. Use stages to logically group your jobs in the pipeline graph for better readability and organization in the GitLab UI.
  2. Clear Dependencies: Only define needs for direct, actual dependencies. Avoid adding unnecessary needs as it can complicate the graph and reduce parallelism.
  3. Monitor the DAG View: Regularly check the pipeline’s DAG (available in the pipeline view within GitLab) to ensure your needs setup is working as intended and identify any bottlenecks or unexpected flows.
  4. Artifact Management: Be mindful of artifact sizes. needs will automatically download artifacts by default. If a job is needs-ing many jobs with large artifacts but only requires a few, consider using artifacts: false for unnecessary ones.
  5. Error Handling: Remember that if a required job in needs fails, the current job will not run. This is the default fail-fast behavior. Use allow_failure: true on the needed job if its failure should not block the needing job, and then use optional: true in the needs definition for clarity.

FAQs – Needs

What is the needs keyword in GitLab CI/CD?
The needs keyword allows a job to explicitly declare its dependencies on other jobs, enabling jobs from later stages to run earlier, in parallel with earlier jobs. It overrides the default sequential stage execution and can speed up pipelines significantly.

How is needs different from stages in GitLab?
By default, GitLab CI/CD executes jobs sequentially stage by stage. With needs, jobs can run as soon as their dependencies finish, even if the dependent job is in a later stage.

Example comparison:

# Without needs: 'test' waits for all 'build' jobs to complete
stages:
  - build
  - test

# With needs: 'test' runs as soon as 'build-job' completes
test:
  stage: test
  needs: [build-job]

How do I use the needs keyword in a job?
You define needs as a list of job names the current job depends on:

build-job:
  stage: build
  script: make

test-job:
  stage: test
  needs: [build-job]
  script: ./run-tests.sh

This allows test-job to start immediately after build-job finishes, instead of waiting for all build stage jobs.

Can needs improve pipeline performance?
Yes. By allowing jobs to run earlier instead of waiting for an entire stage to complete, needs helps reduce total pipeline time through parallelism.

Can I specify artifacts with needs?
Yes. To download artifacts from the needed job, define it like this:

test:
  stage: test
  needs:
    - job: build
      artifacts: true
  script:
    - ./test-from-artifacts.sh

This ensures artifacts from build are downloaded before test runs.

What happens if I use needs without specifying artifacts?
If you define a job using needs without the artifacts: true flag, artifacts are not downloaded automatically. If your job depends on output files from another, include artifacts: true.

Can I use needs across stages?
Yes. needs is designed to cross stage boundaries. It enables jobs in later stages to start earlier based on actual job dependencies.

stages:
  - build
  - test
  - deploy

deploy:
  stage: deploy
  needs: [test]
  script: ./deploy.sh

In this case, deploy runs as soon as test completes.

Can I use needs with jobs in the same stage?
Yes. You can define dependencies within the same stage:

job1:
  stage: build
  script: echo "Job 1"

job2:
  stage: build
  needs: [job1]
  script: echo "Job 2"

This causes job2 to wait for job1, even though both are in the same stage.

What is the maximum number of jobs I can reference in needs?
In GitLab.com:

  • Maximum: 50 needs per job.
  • For self-managed GitLab, this can be configured by the admin.

Defining too many needs can affect pipeline performance and readability.

What happens if a job listed in needs fails?
If any job listed in needs fails, the dependent job will not run. This ensures that jobs relying on earlier steps do not proceed with broken inputs.

Can I use needs with trigger jobs (multi-project pipelines)?
Yes, but only for downstream jobs, not upstream. Use needs in combination with trigger to define fine-grained control in child pipelines:

trigger-build:
  trigger:
    include: child-pipeline.yml
  needs: []

This ensures the job does not wait for anything before triggering the child pipeline.

How do I visualize jobs with needs in the pipeline graph?
GitLab displays a Directed Acyclic Graph (DAG) when using needs, showing job dependencies clearly.
To view it:

CI/CD → Pipelines → View Pipeline Graph

Jobs connected with needs will be linked with arrows, showing their execution order.

Can I use needs and dependencies together?
Yes, but they serve different purposes:

  • needs: Controls execution order and concurrency.
  • dependencies: Controls which artifacts are downloaded from previous jobs.

Example:

test:
  stage: test
  needs:
    - job: build
      artifacts: true
  dependencies:
    - build

This ensures both early execution and artifact sharing.

Can I define an empty needs list to skip automatic stage ordering?
Yes. You can use needs: [] to tell GitLab not to wait for earlier jobs or stages:

independent-job:
  stage: deploy
  needs: []
  script: echo "Runs immediately"

This job runs as soon as a runner is available, regardless of the stage order.

Author

Debjeet Bhowmik

Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins. In my free time, I write blogs on ckdbtech.com

Leave a Comment