AI Assistant Agent: RAG Input Flow

This workflow processes documents from Google Drive folders and loads them into a Supabase vector database for AI-powered search and retrieval. It handles multiple file formats (PDF, DOCX, Markdown, PowerPoint, JSON), extracts metadata, chunks content appropriately, and includes human approval steps for data management operations.

Purpose

No business context provided yet — add a context.md to enrich this documentation.

This workflow serves as a document ingestion pipeline for building a knowledge base that can be queried by AI assistants. It allows administrators to: - Automatically process documents from organized Google Drive folders - Extract searchable metadata and keywords from document content - Store document chunks in a vector database for semantic search - Manage the knowledge base with approval workflows for updates and deletions - Handle special document types like financial budget codes and intent definitions

How It Works

1. Folder Selection: The workflow scans configured Google Drive folders and presents them to an administrator for selection via Telegram
2. File Processing: For each approved folder, it downloads and processes all supported files (PDF, DOCX, Markdown, PowerPoint, JSON)
3. Content Extraction: Documents are converted to text using appropriate extractors (Google Docs API for Google formats, file extraction for PDFs, etc.)
4. Metadata Generation: AI extracts themes and keywords from document content for enhanced searchability
5. Text Chunking: Large documents are split into manageable chunks with overlap to preserve context
6. Vector Storage: Document chunks are embedded using OpenAI embeddings and stored in Supabase vector database
7. Database Updates: Special handling for JSON files containing budget codes and intents, which update structured database tables
8. Approval Workflows: Human-in-the-loop approval for destructive operations like deleting existing vectors
9. Notifications: Telegram notifications keep administrators informed of processing status

Mermaid Diagram

graph TD
    A[Manual Trigger/Webhook] --> B[Configuration]
    B --> C[List Subfolders]
    C --> D[Choose Folder]
    D --> E{Folder Approved?}
    E -->|Yes| F[List Files]
    E -->|No| G[Loop Over Items1]
    F --> H[Mapping]
    H --> I[File Id List]
    H --> J[Collection Name]
    I --> K[Confirm Delete Vectors]
    J --> K
    K --> L{Delete Approved?}
    L -->|Yes| M[Delete Old Documents]
    L -->|No| N[Send Declined Message]
    M --> O[Start Upsert]
    N --> O
    O --> P[Loop Over Items]
    P --> Q[Download File From Google Drive]
    Q --> R[Switch - File Type]
    R -->|DOCX/MD| S[Google Docs]
    R -->|PDF| T[Extract from File]
    R -->|JSON_FIN| U[Extract from JSON]
    R -->|Invalid| V[Send Invalid Message]
    S --> W[Split Out1]
    T --> X[Edit Fields]
    X --> W
    U --> Y[Split Out Codes]
    U --> Z[Split Out Categories]
    U --> AA[Intents]
    Y --> BB[Upsert Codes]
    Z --> CC[Upsert Categories]
    AA --> DD[Split out intents]
    DD --> EE[Upsert Intents]
    W --> FF[Chunking]
    FF --> GG[Extract Meta Data]
    GG --> HH[Merge]
    HH --> II[Data Loader]
    II --> JJ[Supabase Vector Store]
    JJ --> KK[Wait]
    KK --> P
    P -->|Complete| LL[Send Completed Message]

Trigger

• Manual Trigger: For testing and manual execution
• Webhook: Production endpoint at /upsert for external system integration
• Schedule Trigger: Disabled daily trigger at 12:00 PM (can be enabled for automated runs)

Nodes Used

Node Type	Purpose
Google Drive	Download files and list folder contents
Google Docs	Extract text content from Google Docs format
Extract from File	Process PDF and other binary file formats
Switch	Route files based on type (PDF, DOCX, JSON, etc.)
Telegram	Send notifications and approval requests to administrators
Supabase Vector Store	Store document embeddings for semantic search
PostgreSQL	Store structured data (budget codes, categories, intents)
OpenAI Chat Model	Generate metadata and convert content to markdown
OpenAI Embeddings	Create vector embeddings for documents
Text Splitter	Chunk large documents with overlap
Information Extractor	Extract themes and keywords using AI
Code	Custom JavaScript for advanced text chunking
Split In Batches	Process files in manageable batches
Merge	Combine data streams
Set	Transform and map data fields

External Services & Credentials Required

• Google Drive OAuth2: Access to read files and folders
• Google Docs OAuth2: Extract content from Google Docs
• Telegram API: Send notifications and approval requests
• OpenAI API: Generate embeddings and extract metadata
• Supabase: Vector database storage and structured data
• PostgreSQL: Database for budget codes and intent data

Environment Variables

The workflow uses hardcoded configuration in the Configuration node: - folder_id: Google Drive folder ID to scan - admin_chat_id: Telegram chat ID for notifications

Data Flow

Input: Google Drive folder containing documents (PDF, DOCX, Markdown, PowerPoint, JSON)

Processing: - Document content extraction and text conversion - AI-generated metadata (themes, keywords) - Text chunking with pagination information - Vector embedding generation

Output: - Document chunks stored in Supabase vector database with metadata - Structured data (budget codes, categories, intents) in PostgreSQL tables - Telegram notifications to administrators

Error Handling

• File Download Errors: Continue processing other files if individual downloads fail
• Invalid File Types: Send notification and skip unsupported files
• Processing Errors: Error outputs are handled gracefully with continuation
• Human Approval: Timeout handling for approval requests (15-minute limit)
• Batch Processing: Failed items don't stop the entire batch

Known Limitations

• Hardcoded configuration values need manual updates
• Limited to specific file types (PDF, DOCX, Markdown, PowerPoint, JSON)
• Requires manual approval for each folder and deletion operation
• Text chunking uses simple character-based approximation for tokens
• No automatic retry mechanism for failed operations

Related Workflows

This workflow appears to be part of a larger AI assistant system that would include: - Query/search workflows that use the populated vector database - Document update workflows for maintaining the knowledge base - User-facing chat interfaces that leverage the stored knowledge

Setup Instructions

1. Import Workflow: Import the JSON into your n8n instance

2. Configure Credentials:

   • Set up Google Drive OAuth2 credentials
   • Set up Google Docs OAuth2 credentials
   • Configure Telegram Bot API credentials
   • Set up OpenAI API credentials
   • Configure Supabase API credentials
   • Set up PostgreSQL database credentials

3. Update Configuration:

   • Modify the Configuration node with your Google Drive folder ID
   • Update the Telegram admin chat ID
   • Adjust the Supabase collection/table names as needed

4. Database Setup:

   • Ensure Supabase has a documents table configured for vector storage
   • Create PostgreSQL tables: budget_codes, budget_categories, budget_intents

5. Test Setup:

   • Use the manual trigger to test the workflow
   • Verify all credentials are working
   • Test with a small folder first

6. Production Deployment:

   • Enable the webhook trigger for external integration
   • Optionally enable the schedule trigger for automated runs
   • Monitor Telegram notifications for processing status