Skip to content

CICD-Cron-Backup

This workflow automatically backs up n8n workflows from multiple environments (Production, Development, and Staging) to a GitHub repository. It runs daily at midnight and can also be triggered manually, ensuring that workflow definitions are version-controlled and safely stored in the experienceeducate/educate-sifa repository across different branches.

Purpose

No business context provided yet — add a context.md to enrich this documentation.

This workflow serves as a critical backup and version control system for n8n workflows across different deployment environments. It ensures that workflow definitions are automatically synchronized to GitHub, providing disaster recovery capabilities and maintaining a historical record of workflow changes. The system supports three environments: Production (main branch), Development (develop branch), and Staging (staging branch), allowing teams to track workflow evolution across the entire development lifecycle.

How It Works

The workflow operates with three parallel processing lanes, one for each environment:

  1. Environment Setup: Each lane begins by setting environment-specific configuration including API endpoints, authentication keys, and target GitHub branches
  2. Workflow Discovery: The system connects to each n8n instance and retrieves a list of all workflows, filtering out CI/CD-related workflows to avoid recursive backups
  3. Individual Processing: Each workflow is processed one at a time to prevent API rate limiting and ensure reliable operation
  4. Content Preparation: Workflow definitions are cleaned by removing volatile fields (timestamps, execution data) and normalized for consistent comparison
  5. Change Detection: The system compares the current workflow definition with the existing version in GitHub to determine if changes have occurred
  6. Selective Updates: Based on the comparison, workflows are either skipped (if unchanged), updated (if modified), or created (if new) in the appropriate GitHub branch
  7. Aggregation: At the end of each lane, statistics are collected showing how many workflows were processed in each category

The staging lane is currently configured but inactive, as it will fail gracefully until a staging environment is provisioned.

Workflow Diagram

graph TD
    A[Manual Trigger] --> B[Globals PROD]
    A --> C[Globals DEV]
    D[Midnight Everyday] --> B
    D --> C

    B --> E[List All PROD Workflows]
    E --> F[Split Out PROD]
    F --> G[Skip CICD-* PROD]
    G --> H[OneAtATime PROD]

    H --> I[Aggregate PROD]
    H --> J[Fetch Full PROD]
    J --> K[Prepare Commit PROD]
    K --> L[GitHub File:Get PROD]
    K --> M[Merge PROD]
    L --> M
    M --> N[isDiffOrNew PROD]
    N --> O[Switch PROD]

    O --> P[NoOp same PROD]
    O --> Q[GitHub File:Edit PROD]
    O --> R[GitHub File:Create PROD]

    P --> H
    Q --> H
    R --> H

    C --> S[List All DEV Workflows]
    S --> T[Split Out DEV]
    T --> U[Skip CICD-* DEV]
    U --> V[OneAtATime DEV]

    V --> W[Aggregate DEV]
    V --> X[Fetch Full DEV]
    X --> Y[Prepare Commit DEV]
    Y --> Z[GitHub File:Get DEV]
    Y --> AA[Merge DEV]
    Z --> AA
    AA --> BB[isDiffOrNew DEV]
    BB --> CC[Switch DEV]

    CC --> DD[NoOp same DEV]
    CC --> EE[GitHub File:Edit DEV]
    CC --> FF[GitHub File:Create DEV]

    DD --> V
    EE --> V
    FF --> V

Trigger

  • Schedule Trigger: Runs automatically every day at midnight (Africa/Johannesburg timezone)
  • Manual Trigger: Can be executed on-demand for immediate backup operations

Nodes Used

Node Type Purpose
Manual Trigger Allows on-demand execution of the backup process
Schedule Trigger Automatically runs the workflow daily at midnight
Set (Globals) Configures environment-specific settings (URLs, API keys, repository details)
HTTP Request Fetches workflow lists and individual workflow definitions from n8n APIs
Split Out Converts workflow arrays into individual items for processing
Filter Excludes CI/CD workflows from backup to prevent recursive operations
Split in Batches Processes workflows one at a time to manage API rate limits
Code Handles data transformation, comparison logic, and aggregation
GitHub Manages file operations (get, create, edit) in the target repository
Merge Combines data from GitHub file retrieval with prepared commit data
Switch Routes workflows based on their change status (same, different, new)
No Operation Handles unchanged workflows without taking action

External Services & Credentials Required

n8n API Access

  • Production: https://n8n.dev.educate-agent.com
  • Development: https://experienceeducate.app.n8n.cloud
  • Staging: https://n8n.educate-agent.com (currently inactive)

GitHub Integration

  • Repository: experienceeducate/educate-sifa
  • Required Permissions: Contents read/write access
  • Credentials Needed:
    • Francis Educate Fine Grained Tokens (for PROD and DEV)
    • GitHub account 5 (for STAGING)

Environment Variables

Environment-specific configuration is managed through Set nodes:

Production (PROD)

  • env_name: "prod"
  • env_url: Production n8n instance URL
  • repo_branch: "main"
  • path_prefix: "workflows/"

Development (DEV)

  • env_name: "dev"
  • env_url: Development n8n instance URL
  • repo_branch: "develop"
  • path_prefix: "workflows/"

Staging (STAGING)

  • env_name: "staging"
  • env_url: Staging n8n instance URL (inactive)
  • repo_branch: "staging"
  • path_prefix: "workflows/"

Data Flow

Input

  • Workflow lists from n8n API endpoints
  • Individual workflow definitions with full configuration
  • Existing workflow files from GitHub repository

Processing

  • Workflow names are converted to URL-safe slugs for file naming
  • Volatile fields are stripped from workflow definitions
  • JSON objects are sorted deterministically for consistent comparison
  • Change detection compares normalized workflow content

Output

  • JSON files in GitHub repository under workflows/ directory
  • Commit messages indicating the backup source and change type
  • Aggregated statistics showing backup operation results

Error Handling

The workflow implements several error handling strategies:

  • Continue on Error: All HTTP requests and GitHub operations use continueRegularOutput to prevent single failures from stopping the entire process
  • Retry Logic: API calls include automatic retry with exponential backoff (3 attempts, 2-second intervals)
  • Graceful Degradation: The staging lane is designed to fail silently until the environment is provisioned
  • Timeout Protection: HTTP requests have 30-second timeouts to prevent hanging operations

Known Limitations

  • The staging environment is currently inactive and will fail at the workflow listing stage
  • Workflows starting with "CICD-" are excluded from backup to prevent recursive operations
  • API rate limiting is managed through sequential processing, which may slow down large backup operations
  • The workflow requires manual credential configuration for GitHub access

This workflow is part of a CI/CD system and excludes other CI/CD workflows from backup to prevent conflicts and recursive operations.

Setup Instructions

  1. Import the Workflow

    • Copy the workflow JSON and import it into your n8n instance
    • Ensure the workflow is initially inactive

  2. Configure GitHub Credentials

    • Create GitHub Personal Access Tokens with Contents read/write permissions for the target repository
    • Set up n8n credentials named:
      • Francis Educate Fine Grained Tokens (for PROD/DEV lanes)
      • GitHub account 5 (for STAGING lane)

  3. Update Environment Configuration

    • Modify the Globals nodes for each environment with correct:
      • API endpoints (env_url)
      • Authentication tokens (api_key)
      • Repository details if different from defaults

  4. Configure n8n API Access

    • Generate API keys for each n8n environment
    • Update the api_key values in the respective Globals nodes

  5. Test the Setup

    • Run the workflow manually to verify connectivity
    • Check that files are created in the correct GitHub repository branches
    • Verify that the aggregation nodes show expected statistics

  6. Activate the Workflow

    • Enable the workflow to start automatic daily backups
    • Monitor the first few scheduled runs to ensure proper operation

  7. Enable Staging (Optional)

    • When staging environment is ready, update the Globals (STAGING) node with correct URL and API key
    • The staging lane will automatically begin working once configured