Automated Document Intelligence and Evaluation System

We designed a Databricks-based Retrieval-Augmented Generation workflow purpose-built for document interpretation and validation in regulated operational environments. The planning phase established a LangGraph-powered multi-agent architecture capable of parsing structured and unstructured data across Excel spreadsheets, scanned PDFs, operational reports, and email-embedded attachments. Each document type was handled by a specialised parsing agent that understood its expected schema, key fields, and typical anomaly patterns. A Delta Lake-backed ingestion layer standardised every extracted record into a governed canonical format, so downstream consumers (dashboards, alerts, regulatory filings) saw a consistent shape regardless of the source format. The core differentiator was the Agent Evaluation Framework. Every model-generated output, whether a summary, an extracted field, or a flagged anomaly, was scored against factual accuracy and operational compliance benchmarks before being trusted by downstream processes. The evaluation layer used DeepEval to continuously assess agent reliability and MLflow Evaluate for precision tracking and performance benchmarking. Outputs that failed evaluation thresholds were automatically routed to the human review queue with full context about why they failed. A modular orchestration pattern allowed the RAG system to dynamically scale across multiple use cases without rebuilding the foundation. The same core platform was used for daily report summarisation, anomaly flagging in operational logs, and synthesis of multi-document briefings for shift handovers. Audit traceability and explainability were built in from day one through Unity Catalog lineage tracking, so every AI-generated artefact could be traced back to its source documents, the model version that produced it, and the evaluation scores that justified its release.