Pypyr: YAML Pipeline Automation for Complex CI/CD Workflows

Cloud CI platforms (GitHub Actions, GitLab CI, BitBucket Pipelines) give you steps and runners. But the step definitions are often rigid — error handling is limited, context passing between steps is awkward, and conditional execution requires awkward shell workarounds. Pypyr fills that gap as a YAML-defined pipeline task runner that handles the complexity cloud CI platforms don’t cover well.

How Pypyr Fits in the CI/CD Stack

%%{ init: { 'look': 'handDrawn' } }%%
graph TD
    A[CI Trigger<br/>push / schedule / webhook] --> B[Cloud CI Platform<br/>GitHub Actions / GitLab CI]
    B -->|run pipeline job| C[Pypyr<br/>YAML pipeline]
    C --> D[Step 1: Setup<br/>terraform init]
    C --> E[Step 2: Plan<br/>terraform plan]
    C --> F[Step 3: Apply<br/>terraform apply]
    D -->|on failure| G[Notify Slack]
    E -->|on failure| G
    F -->|on success| H[Update PR comment]
    G --> I[Exit with error code]
    H --> J[Exit with success]

Pypyr runs as a single CI step but orchestrates complex multi-step workflows with proper error handling, conditions, and context passing.

The Core Concepts

%%{ init: { 'look': 'handDrawn' } }%%
graph LR
    A[Pipeline YAML] --> B[Context<br/>key-value state]
    B --> C[Steps<br/>executed in order]
    C -->|on_success| D[Next step]
    C -->|on_failure| E[Error handler]
    C -->|conditional| F[Skip step]
    B -->|update| A

• Context: A dictionary that persists across all steps. Steps can read from it, write to it, and modify it.
• Steps: Individual units of work — shell commands, Python code, API calls.
• on_success / on_failure: Handlers that run after each step, enabling recovery workflows.
• run / skip / swallow: Step execution modes — run normally, skip entirely, or run but ignore failures.

A Complete Pipeline Example

# .pypyr/ pipelines/deploy.yaml
context_parser: pypyr.parser.yamlfile

steps:
  - name: pypyr.steps.cmd
    description: Run terraform format check
    in:
      cmd: terraform fmt -check -recursive
    on_failure:
      - name: pypyr.steps.cmd
        description: If formatting fails, fix it automatically
        in:
          cmd: terraform fmt -recursive
        run: always  # Run even if previous step failed

  - name: pypyr.steps.cmd
    description: Initialize Terraform
    in:
      cmd: terraform init -backend=true
    on_success:
      - name: pypyr.steps.cmd
        description: Run plan only on success
        in:
          cmd: terraform plan -out=tfplan
        swallo: true  # Don't fail pipeline if this errors

  - name: pypyr.steps.python
    description: Evaluate plan output size
    in:
      python_code: |
        import os
        plan_size = os.path.getsize('tfplan')
        context['plan_size_mb'] = plan_size / (1024 * 1024)
        print(f"Plan size: {context['plan_size_mb']:.2f} MB")

  - name: pypyr.steps.cmd
    description: Apply only if plan is under 50MB
    in:
      cmd: terraform apply -input=false tfplan
    condition: context['plan_size_mb'] < 50
    on_success:
      - name: pypyr.steps.cmd
        description: Update deployment tracking
        in:
          cmd: ./scripts/update-deployment.sh

  - name: pypyr.steps.notify
    description: Notify on failure
    run: on_failure
    in:
      message: "Terraform deployment failed. Check CI logs."

Why Not Just Use Makefile?

Makefiles work for simple linear builds. Pypyr handles anything that needs recovery logic, conditional branches, or state passing between steps.

Real-World CI/CD Use Case

%%{ init: { 'look': 'handDrawn' } }%%
graph TD
    A[PR opened] --> B[CI runs pypyr pipeline]
    B --> C[terraform fmt]
    C -->|fail| D[Format fix + commit]
    C -->|pass| E[terraform init]
    E --> F[terraform plan]
    F -->|large plan| G[Flag for manual review]
    F -->|small plan| H[auto-apply in CI]
    G --> I[Post PR comment<br/>"Plan too large, manual approval needed"]
    H --> J[terraform apply]
    J --> K[Post PR comment<br/>"Deployed successfully"]
    D --> L[Push formatted code]
    L --> A

For infrastructure repos, Pypyr can evaluate the size of terraform plan and decide: small changes get auto-applied, large changes require manual review. That’s conditional logic that plain bash can’t express cleanly.

Getting Started

pip install pypyr

# Run a pipeline
pypyr pipeline deploy

# With arguments passed to context
pypyr pipeline deploy arg1 arg2

Pypyr pipelines live in .pypyr/pipelines/ by default. Each pipeline is a YAML file that defines steps, conditions, and error handlers.

For teams already using cloud CI, Pypyr isn’t a replacement — it’s the tool that handles the complex orchestration within a single CI step, while the cloud platform handles triggers, secrets, and parallelism across jobs.