Skip to content

CICD-Cron-Backup

This workflow automatically backs up n8n workflows from multiple environments (production, development, and staging) to a GitHub repository. It runs daily at 2:00 AM SAST and can also be triggered manually, ensuring all workflow definitions are version-controlled and synchronized across different branches based on their environment.

Purpose

No business context provided yet — add a context.md to enrich this documentation.

This workflow serves as a CI/CD backup system for n8n workflows across multiple environments. It automatically synchronizes workflow definitions from n8n instances to corresponding GitHub branches, providing version control and backup capabilities for workflow automation infrastructure.

How It Works

The workflow operates with three parallel processing lanes, one for each environment:

  1. Initialization: Both manual and scheduled triggers fan out to initialize three environment-specific processing lanes (PROD → main branch, DEV → develop branch, STAGING → staging branch)

  2. Environment Setup: Each lane begins by setting up environment-specific configuration including API endpoints, authentication keys, and target repository details

  3. Workflow Discovery: The system fetches a list of all workflows from each n8n environment, filtering out any workflows that start with "CICD-" to avoid backing up CI/CD workflows themselves

  4. Individual Processing: Each workflow is processed one at a time through a batch system to prevent overwhelming the APIs

  5. Data Preparation: For each workflow, the system fetches the complete workflow definition, strips out volatile fields (timestamps, execution data), and creates a deterministic JSON representation

  6. GitHub Comparison: The prepared workflow data is compared against the existing file in the GitHub repository to determine if it's new, different, or unchanged

  7. Conditional Updates: Based on the comparison:

    • Same: No action taken (NoOp)
    • Different: Existing file is updated with new content
    • New: New file is created in the repository

  8. Aggregation: After processing all workflows in an environment, summary statistics are generated showing counts of same, different, and new workflows

Mermaid Diagram

graph TD
    A[Manual Trigger] --> D[Globals PROD]
    B[Daily 02:00 SAST] --> D
    A --> E[Globals DEV]
    B --> E

    D --> F[List All PROD Workflows]
    F --> G[Split Out PROD]
    G --> H[Skip CICD-* PROD]
    H --> I[OneAtATime PROD]
    I --> J[Fetch Full PROD]
    I --> K[Aggregate PROD]
    J --> L[Prepare Commit PROD]
    L --> M[GitHub File:Get PROD]
    L --> N[Merge PROD]
    M --> N
    N --> O[isDiffOrNew PROD]
    O --> P[Switch PROD]
    P --> Q[NoOp same PROD]
    P --> R[GitHub File:Edit PROD]
    P --> S[GitHub File:Create PROD]
    Q --> I
    R --> I
    S --> I

    E --> T[List All DEV Workflows]
    T --> U[Split Out DEV]
    U --> V[Skip CICD-* DEV]
    V --> W[OneAtATime DEV]
    W --> X[Fetch Full DEV]
    W --> Y[Aggregate DEV]
    X --> Z[Prepare Commit DEV]
    Z --> AA[GitHub File:Get DEV]
    Z --> BB[Merge DEV]
    AA --> BB
    BB --> CC[isDiffOrNew DEV]
    CC --> DD[Switch DEV]
    DD --> EE[NoOp same DEV]
    DD --> FF[GitHub File:Edit DEV]
    DD --> GG[GitHub File:Create DEV]
    EE --> W
    FF --> W
    GG --> W

Trigger

  • Manual Trigger: Can be executed on-demand for immediate backup operations
  • Schedule Trigger: Automatically runs daily at 2:00 AM South Africa Standard Time (SAST)

Nodes Used

Node Type Purpose
Manual Trigger Allows on-demand execution of the backup process
Schedule Trigger Runs the backup automatically at 2:00 AM SAST daily
Set Configures environment-specific variables (URLs, API keys, repository settings)
HTTP Request Fetches workflow lists and individual workflow details from n8n APIs
Split Out Separates individual workflows from the API response array
Filter Excludes CI/CD workflows from backup to prevent recursive operations
Split in Batches Processes workflows one at a time to manage API load
Code Handles data transformation, comparison logic, and aggregation
GitHub Manages file operations (get, create, edit) in the target repository
Merge Combines GitHub file content with prepared workflow data for comparison
Switch Routes workflows based on their status (same, different, new)
NoOp Placeholder for workflows that don't need updates

External Services & Credentials Required

GitHub Integration

  • Credential Name: Francis Educate Fine Grained Tokens
  • Type: GitHub API Personal Access Token
  • Required Permissions: Contents read/write access to the GaturaN/educate-sifa repository
  • Repository: GaturaN/educate-sifa

n8n API Access

  • Production Environment: https://n8n.dev.educate-agent.com
  • Development Environment: https://n8n.educate-agent.com
  • Staging Environment: TBD (placeholder configuration exists)

Environment Variables

Each environment lane uses the following configuration variables:

Production (PROD)

Development (DEV)

Staging (STAGING)

  • env_name: "staging"
  • env_url: "TBD-staging-url"
  • api_key: "TBD_STAGING_API_KEY"
  • repo_branch: "staging"

Common Variables

  • repo_owner: "GaturaN"
  • repo_name: "educate-sifa"
  • path_prefix: "workflows/"

Data Flow

Input

  • Workflow lists from n8n API endpoints (up to 250 workflows per environment)
  • Individual workflow definitions with complete configuration

Processing

  • Strips volatile fields: updatedAt, versionId, triggerCount, lastUpdatedBy, pinData, staticData, isArchived, versionCounter, activeVersionId, createdAt, meta, shared, activeVersion
  • Removes execution data from nodes
  • Sorts JSON keys deterministically for consistent comparison
  • Generates workflow slugs for filename creation

Output

  • JSON files in GitHub repository under workflows/ directory
  • Commit messages indicating the operation type and environment
  • Aggregated statistics showing backup operation results

Error Handling

The workflow includes several error handling mechanisms:

  • HTTP Request Retries: All API calls have retry logic (3 attempts with 2-second delays)
  • Continue on Error: GitHub operations and HTTP requests are configured to continue processing even if individual operations fail
  • Timeout Protection: 30-second timeouts on HTTP requests to prevent hanging
  • Always Output Data: GitHub file operations always produce output, even on failure

Known Limitations

  • Staging environment configuration is incomplete (placeholder values for URL and API key)
  • Workflow limit of 250 per environment may not capture all workflows in large installations
  • API keys are stored directly in workflow configuration rather than using n8n credentials
  • No notification system for backup failures or success

This workflow is part of a CI/CD system and excludes other workflows that start with "CICD-" from backup operations to prevent recursive processing.

Setup Instructions

  1. Import the Workflow

    • Import the JSON definition into your n8n instance
    • Ensure the workflow is initially inactive

  2. Configure GitHub Credentials

    • Create a GitHub Personal Access Token with Contents read/write permissions
    • Add the credential in n8n as "Francis Educate Fine Grained Tokens"
    • Verify access to the GaturaN/educate-sifa repository

  3. Update Environment Configuration

    • Modify the "Globals" nodes for each environment with correct:
      • API endpoints (env_url)
      • API keys (api_key)
      • Repository settings if different from defaults

  4. Configure Staging Environment

    • Update the "Globals (STAGING)" node with actual staging environment details
    • Replace "TBD-staging-url" and "TBD_STAGING_API_KEY" with real values

  5. Test the Workflow

    • Run a manual execution to verify connectivity to all environments
    • Check that files are created/updated in the correct GitHub branches
    • Verify the aggregation output shows expected workflow counts

  6. Activate the Workflow

    • Enable the workflow to start automatic daily backups
    • Monitor initial executions to ensure proper operation

  7. Repository Setup

    • Ensure the target GitHub repository has the appropriate branches (main, develop, staging)
    • Verify the workflows/ directory exists or will be created automatically