Skip to content

Backend Monorepo CI/CD Pipeline

This document outlines the GitLab CI/CD pipeline for the Coldtivate backend monorepo - a Docker-based service collection. The pipeline's structure, technologies, and design rationale are covered to provide both technical details and clarity.

The pipeline aims to deliver:

  • Robust Docker Image Standardization: Each service in the monorepo follows uniform build patterns, producing consistent and reliable Docker images for improved deployment predictability.
  • Automated Regression Gates: Merge requests run the Base API and reporting-service test suites before code can progress to build and deployment stages.
  • Fast, Safe Deployments: Services get updated quickly with almost no downtime.
  • Environment Isolation: Separate staging and production environments for stability.
  • Security First: Safely handling SSH keys and secrets during deployments.

A Step-by-Step Breakdown

graph LR
    A[Start] --> B{GitLab CI Pipeline};
    B --> T[Test Stage];
    T --> C[Build Stage];

    subgraph Build Stage
        direction LR;
        C --> |Multiple Services| D[Service Images];
    end

    D --> M[Docker Registry];
    M --> N[Deploy Stage];

    subgraph Environments
        direction LR;
        N -- main --> P[Deploy to Staging];
        N -- tags --> Q[Deploy to Production];
    end

    %% Note: Services include gateway, base-api, ml4-india, ml4-nigeria,
    %% Impact-Dashboard-Backend, Impact-Dashboard-Backend scheduler,
    %% App-Impact-Reporting, Farmers-Dashboard-Backend,
    %% and Farmers-Dashboard-Backend scheduler

The pipeline is structured into three main stages:

  1. Test: Runs the Base API, App Impact Reporting, Impact Dashboard, and Farmers Dashboard suites against disposable service containers.
  2. Build: Creates Docker images for each monorepo service.
  3. Deploy: Pushes the Docker images to staging or production.

Underlying Technologies and Tools

  • GitLab CI/CD: Handles automated building and deploying.
  • Docker: Containers for all backend services.
  • Docker BuildKit: Speeds up Docker build processes with better efficiency.
  • SSH: Securely deploys code to remote servers.
  • Git Submodules: Include external repos directly in the monorepo.

Key Pipeline Components

  • Automated Test Jobs:

    Job Service Command
    base-api unit tests Base API pipenv run python -Wa manage.py test base -v 2
    app-impact-reporting tests App Impact Reporting python -m unittest discover -s tests -v through the JUnit runner
    impact-dashboard tests Impact Dashboard python -m pytest tests/ -v
    farmers-dashboard tests Farmers Dashboard python -m unittest discover -s tests -v through the JUnit runner

    All four jobs are configured with allow_failure: false. The Base API job uses Python 3.12 and starts PostGIS 16 and Valkey 7; Django creates and destroys a fresh test database without --keepdb. The reporting services run their test suites on Python 3.9, and each receives its own empty PostGIS 16 service.

  • Reporting Database Bootstrap:

    • scripts/ci/wait_for_postgres.sh fails with a clear timeout if the PostGIS service is unavailable.
    • scripts/ci/configure_disposable_db.sh overwrites inherited database targets at shell runtime, creates ci_test_<CI_JOB_ID>, and records the job ID inside it.
    • scripts/ci/assert_disposable_db.sh refuses migrations or SQL unless the stage is test, the host is the postgis service alias, the database name matches the current job, and the stored marker is valid.
    • scripts/ci/bootstrap_reporting_db.sh installs a job-local Python 3.12 runtime, applies Base API migrations, then creates the reporting tables/views required by that job. This keeps the reporting test runtime on Python 3.9 while satisfying the Base API runtime requirement.
    • Reporting tests resolve connection settings from the forced CI-only DB_HOST, DB_PORT, DB_NAME, DB_USERNAME, and DB_PASSWORD values. DATABASE_URL is unset.
    • CI forces REQUIRE_TEST_DB=true, so a database connection failure is an error rather than a silently skipped integration suite.
    • The Base API and bootstrap install the committed lock with pipenv install --dev --deploy; a stale lock therefore fails the job.

  • Test Results:

    • The AIR, Impact Dashboard, and Farmers Dashboard jobs publish JUnit reports to the merge request's Tests tab.
    • Base API results appear in the base-api unit tests job log.
    • Every job retains its full stdout in GitLab, including database bootstrap progress and test-runner output.
  • .build-docker-image Template:
    • Central template for all Docker image builds.
    • Uses BuildKit for speed and GitLab registry login.
    • Sets up ssh-agent for submodule access.
    • Speeds up builds with CACHE_IMAGE for layer caching.
    • Tags images with branch slug and commit SHA for tracking.
    • Handles git submodules.
  • Build Jobs:
    • Each service in the monorepo has a dedicated build job extending the .build-docker-image template.
    • Jobs like build gateway image and build ml4-india image exist for different services.
    • Jobs set custom variables (CONTEXT_PATH, IMAGE_AND_TAGS, etc.) to control their build behavior.
  • .deploy-to-environment Template:
    • Template for standardized deployment steps.
    • Connects to servers via SSH.
    • Handles remote Docker login.
    • Runs deployment script on target server.
    • Works around GitLab CI limitations for env variables.
  • Deployment Jobs:
    • Staging and production deployment jobs use the .deploy-to-environment template.
    • Each job sets variables like ENVIRONMENT, HOST, USER, PRIVATE_KEY, and DOT_ENV.

Build Process Details

  • Docker BuildKit: Uses Docker BuildKit for faster builds through better caching and parallel processing.
  • Caching: Using CACHE_IMAGE for Docker layer caching - speeds up builds by reusing previous work.
  • Tagging: Images get tagged with branch slug (${CI_COMMIT_REF_SLUG}) and short commit SHA (${CI_COMMIT_SHORT_SHA}) for clear version tracking.
  • Git Submodules: Pipeline automatically handles fetching and updating any dependent repositories during builds.

Deployment Process Details

  • SSH Deployment: Uses SSH for secure server communication.
  • Environment Variables: The DOT_ENV variable passes environment settings to the deployment script.
  • Deployment Script: A bash script manages the actual deployment on remote servers.
  • Manual Deployments: Deployments require explicit approval, giving teams control over release timing.

Branching Strategy and Deployment Triggers

  • Merge requests: Trigger all four test jobs automatically.
  • Default branch: Triggers all four test jobs. Plain feature-branch pushes without a merge request are excluded to avoid duplicate branch/MR pipelines.
  • Main branch: Triggers all four test jobs before builds and enables manual staging deployment.
  • Tags: Trigger all four test jobs before builds and enable manual production deployment.

Because test precedes build and deploy, a failed test job prevents later stages from starting. The GitLab project must also enable Pipelines must succeed under its merge checks for a failed or pending pipeline to block merging.

Test Job Results

Open the merge request's pipeline view for the four job statuses and full logs. The merge request's Tests tab contains the JUnit results published by the three reporting jobs.

Environment Configuration

  • Staging: Runs on dedicated staging server (20.199.9.192).
  • Production: Runs on a dedicated production server (20.74.81.183).

Key Considerations

  • SSH Key Management: Keep SSH keys secure.
  • Environment Variable Security: Keep secrets safe.
  • Script Maintenance: Regularly update and test the environment-deployment-management.sh script.
  • Docker Registry Access: Ensure that the GitLab CI/CD pipeline has proper access to the Docker registry.
  • Git Submodules: Keep submodules updated and properly configured.