V4 - Model Benchmark Harness - Edith¶

A standalone testing tool that evaluates multiple AI models side-by-side using real Sifa V4 coaching prompts. This workflow helps assess model performance, response quality, latency, and cost across different OpenRouter models without affecting live coaching sessions.

Purpose¶

No business context provided yet — add a context.md to enrich this documentation.

This workflow serves as a quality assurance and model evaluation tool for the Sifa V4 coaching system. It allows developers and coaches to:

Compare response quality across different AI models using authentic coaching scenarios
Measure performance metrics (latency, token usage, cost) for budget planning
Test prompt effectiveness with real user contexts before deploying to production
Validate new models or prompt variations in a safe, isolated environment

How It Works¶

Test Setup: Configure test parameters including user message, prompt type, temperature, and list of models to test
Context Retrieval: Optionally fetch real user context from the database or use sample data for testing
Prompt Building: Select and populate one of six Sifa V4 prompt templates with user context
Model Fan-out: Create separate requests for each specified OpenRouter model
Parallel Testing: Execute chat completions for all models, measuring latency and capturing responses
Results Aggregation: Compile all results into a comparison table showing metrics and full responses
Analysis Output: Generate a comprehensive report for side-by-side quality assessment

Workflow Diagram¶

graph TD
    A[When clicking Test] --> B[Test Inputs]
    B --> C[Get Session Context]
    C --> D[Build Prompt]
    D --> E[Fan Out Models]
    E --> F[Loop Over Models]
    F --> G[Comparison Table]
    F --> H[Stamp Start]
    H --> I[Call OpenRouter]
    I --> J[Parse Result]
    J --> F

Trigger¶

Manual Trigger: Click "Test workflow" button to start the benchmark process. This is a development tool designed for on-demand testing rather than automated execution.

Nodes Used¶

Node Type	Purpose
Manual Trigger	Initiates the benchmark test when clicked
Set	Configures test parameters (user message, models, temperature, etc.)
Postgres	Retrieves real user context from database (optional)
Code	Builds prompts, fans out models, timestamps requests, parses results
Split in Batches	Processes one model at a time in the testing loop
HTTP Request	Makes chat completion calls to OpenRouter API

External Services & Credentials Required¶

OpenRouter API¶

Credential Type: OpenRouter API Key
Purpose: Access to multiple AI models for chat completions
Required Permissions: Chat completions access
Usage Tracking: Enabled to capture token counts and costs

PostgreSQL Database¶

Credential Type: PostgreSQL connection
Purpose: Optional retrieval of real user context via get_session_context_v4() function
Access Level: Read-only queries only
Database: sifaV4Dev environment

Environment Variables¶

No environment variables are required. All configuration is handled through: - Node credentials (OpenRouter API, PostgreSQL) - Test input parameters within the workflow - Model list specified in the modelsText field

Data Flow¶

Input Data¶

User Message: The coaching message to test (required)
Prompt Choice: One of six Sifa V4 prompt types (coaching_general, coaching_evening, coaching_midday, onboarding, emotional, help, or custom)
Models List: OpenRouter model IDs, one per line
User Phone: Optional - fetches real context if provided, uses sample data if empty
Temperature: Model creativity setting (default 0.7)
Sample Context: Fallback user data when no phone number provided

Output Data¶

Summary Report: Markdown-formatted comparison table and full responses
Performance Metrics: Latency, token counts, and USD costs per model
Quality Assessment: Complete model responses for side-by-side evaluation
Error Tracking: Failed requests with error details

Error Handling¶

The workflow includes robust error handling:

HTTP Request Failures: Continues execution and marks failed models with error status
Invalid Model IDs: Shows "ERROR" status in results table with error details
Database Connection Issues: Falls back to sample context data
Missing Parameters: Validates required inputs and shows descriptive error messages
Malformed Responses: Captures and reports API response issues

All errors are surfaced in the final comparison table rather than stopping execution.

Known Limitations¶

No Live Integration: This is a testing harness only - no data is written to production systems
Sequential Processing: Models are tested one at a time to avoid rate limiting
Cost Accumulation: Each test run incurs API costs across all specified models
Context Staleness: Database context is read-only and may not reflect the most current user state
Tool Simulation: Advanced Sifa features (points awarding, micro-action saving) are not executed

This workflow references the live Sifa V4 coaching system but operates independently. It uses identical prompt templates to ensure testing accuracy but does not interact with other workflows.

Setup Instructions¶

1. Import Workflow¶

Copy the workflow JSON and import into your n8n instance
Ensure you have the required node types installed

2. Configure Credentials¶

OpenRouter API: Add your OpenRouter API key in n8n credentials
PostgreSQL: Configure connection to sifaV4Dev database (optional)

3. Customize Test Parameters¶

Edit the "Test Inputs" node to configure: - userMessage: Enter a realistic coaching message in Swahili/Sheng - promptChoice: Select from available prompt types - modelsText: List OpenRouter model IDs (one per line, # for comments) - userPhone: Leave blank for sample data or enter real number for live context - temperature: Adjust model creativity (0.0-1.0)

4. Run Test¶

Click "Test workflow" button
Monitor execution progress
Review results in "Comparison Table" output

5. Analyze Results¶

The summary field contains: - Performance comparison table (latency, tokens, cost) - Full model responses for quality assessment - Error details for failed requests