V4 Model Eval Runner (Webhook) - Edith¶

A webhook-driven evaluation runner that executes single AI model tests for the SIFA coaching system. This workflow receives test parameters via HTTP POST, builds appropriate prompts for different coaching scenarios, calls AI models through OpenRouter, and returns detailed performance metrics including response quality, latency, token usage, and costs.

Purpose¶

No business context provided yet — add a context.md to enrich this documentation.

This workflow appears to be part of a model evaluation system for SIFA, an AI-powered business coaching platform for young Kenyan entrepreneurs. It enables systematic testing of different AI models across various coaching scenarios (general coaching, evening check-ins, midday interactions, onboarding, emotional support, and help requests) to ensure optimal performance before deployment.

How It Works¶

Receive Test Request: A webhook receives a POST request containing test parameters including the test ID, scenario type, model to test, user message, and contextual information about the entrepreneur
Build Prompt: The system constructs the appropriate system prompt based on the scenario type (coaching, onboarding, emotional support, etc.) and injects relevant user context like name, business type, and current micro-actions
Call AI Model: The formatted prompt and user message are sent to the specified AI model via OpenRouter's API
Parse Response: The AI model's response is analyzed to extract the generated content, token usage, costs, latency, and any errors
Return Results: All metrics and the response are returned as JSON to the calling system for evaluation analysis

Workflow Diagram¶

graph TD
    A[Webhook] --> B[Eval Build Prompt]
    B --> C[Call OpenRouter]
    C --> D[Eval Parse]
    D --> E[Respond]

Trigger¶

Webhook: POST requests to /v4-model-eval endpoint

Expected payload structure:

{
  "testId": "string",
  "scenario": "string", 
  "promptChoice": "coaching_general|coaching_evening|coaching_midday|onboarding|emotional|help|custom",
  "customSystemPrompt": "string (if promptChoice is 'custom')",
  "userMessage": "string",
  "model": "string",
  "temperature": "number",
  "runNo": "number",
  "context": {
    "firstName": "string",
    "businessOwned": "string",
    "currentMicroAction": "string",
    "currentMicroActionType": "string",
    "totalPoints": "number",
    "phone": "string"
  }
}

Nodes Used¶

Node Type	Node Name	Purpose
Webhook	Webhook	Receives POST requests with test parameters
Code	Eval Build Prompt	Constructs system prompts for different coaching scenarios and injects user context
HTTP Request	Call OpenRouter	Makes API calls to OpenRouter for AI model inference
Code	Eval Parse	Extracts response content, usage metrics, costs, and latency from API response
Respond to Webhook	Respond	Returns evaluation results as JSON response
Sticky Note	Sticky Note	Documentation and workflow description

External Services & Credentials Required¶

OpenRouter API - Service: OpenRouter (https://openrouter.ai) - Credential Type: openRouterApi - Credential Name: openrouter_eval_temp - Required: API key for accessing various AI models - Note: Uses temporary credential that should be deleted after evaluation runs

Environment Variables¶

No environment variables are directly referenced in this workflow. All configuration is passed through the webhook payload.

Data Flow¶

Input (via webhook POST body): - Test metadata (testId, scenario, runNo) - Model configuration (model name, temperature) - Prompt selection (coaching type or custom prompt) - User message to evaluate - User context (name, business details, current state)

Output (JSON response):

{
  "testId": "string",
  "scenario": "string", 
  "promptChoice": "string",
  "model": "string",
  "runNo": "number",
  "latencyMs": "number",
  "response": "string",
  "finishReason": "string",
  "reasoningTokens": "number",
  "promptTokens": "number", 
  "completionTokens": "number",
  "totalTokens": "number",
  "costUsd": "number",
  "providerModel": "string",
  "error": "string|null"
}

Error Handling¶

The workflow includes comprehensive error handling:

HTTP Request Errors: The OpenRouter call is configured with onError: continueRegularOutput to capture API failures
Response Parsing: Detects and reports various failure modes including empty responses, API errors, and malformed responses
Prompt Building: Validates required parameters and throws descriptive errors for missing or invalid inputs
Timeout Protection: HTTP requests have a 120-second timeout to prevent hanging
Error Categorization: Distinguishes between different types of failures (API errors, empty content, reasoning token exhaustion)

Known Limitations¶

Uses temporary OpenRouter credentials that must be manually managed
Execution data saving is disabled for security (responses not persisted in n8n)
Single model call per request (no batch processing)
120-second timeout may not be sufficient for very large models
No retry logic for transient API failures

No related workflows specified in the provided context.

Setup Instructions¶

Import Workflow: Import the JSON workflow definition into your n8n instance
Configure OpenRouter Credentials:
- Create a new OpenRouter API credential in n8n
- Add your OpenRouter API key
- Update the credential reference in the "Call OpenRouter" node
Activate Webhook:
- The webhook will be available at: https://your-n8n-instance.com/webhook/v4-model-eval
- Ensure the webhook path matches your evaluation orchestration system
Test Configuration:
- Send a test POST request with sample payload
- Verify the response includes all expected metrics
- Check that different prompt choices work correctly
Security Considerations:
- Execution data saving is disabled by default for security
- Consider implementing authentication for the webhook endpoint
- Monitor API usage and costs through OpenRouter dashboard
Integration:
- Configure your evaluation orchestration system to call this webhook
- Ensure proper error handling in the calling system
- Set up monitoring for failed evaluations