{"id":155,"date":"2026-03-24T15:38:14","date_gmt":"2026-03-24T15:38:14","guid":{"rendered":"https:\/\/thelab.wppresolve.com\/?post_type=research_pod&p=155"},"modified":"2026-04-28T10:00:02","modified_gmt":"2026-04-28T10:00:02","slug":"data-quality-agent-pod","status":"publish","type":"research_pod","link":"https:\/\/cms.research.wpp.com\/?research_pod=data-quality-agent-pod","title":{"rendered":"Data Quality Agent Pod"},"content":{"rendered":"\n

Silent data corruption is a well-documented challenge across modern ML pipelines. Broken ingestion jobs, schema drift, logical inconsistencies: these issues rarely trigger alerts, and by the time they’re caught, downstream models may have already been learning from compromised data. We built an autonomous agent that audits data directly in BigQuery, runs forensic structural and logical checks with zero manual input, and , crucially, remembers. Its persistent memory architecture means every audit sharpens the next, elevating data quality from a routine operational task into a compounding strategic advantage. The results: F1 of 0.88, perfect detection on 73% of test scenarios, and 100% consistency across runs.<\/em><\/p>\n\n\n\n

For a high-level overview of this technical report please visit our corresponding non-technical blog post here<\/a>.<\/strong><\/p>\n\n\n\n

Data quality assurance agent technical walkthrough<\/strong><\/h1>\n\n\n\n

Introduction<\/strong><\/h2>\n\n\n\n

Data quality assurance (QA)<\/strong> is a critical bottleneck in modern data engineering pipelines. Engineers frequently dedicate a disproportionate amount of time to manually profiling, verifying, and debugging datasets before they are cleared for downstream consumption by data scientists to build machine learning models. This manual intervention is unscalable, computationally inefficient, and prone to human error, particularly when validating complex, cross-column business logic within wide tables.<\/p>\n\n\n\n

To address this infrastructure gap, we architected and deployed the Data Quality Assurance Agent.<\/strong> Operating directly against our Google BigQuery<\/strong><\/a> data warehouse, the agent can autonomously interpret schemas, execute targeted NL2SQL<\/strong> anomaly detection queries, and generate comprehensive diagnostic reports.<\/p>\n\n\n\n

An important feature of this agent is its long-term memory architecture<\/strong>, hosted via Vertex AI Agent Engine<\/a><\/strong>. By indexing and retrieving historical context across sessions, the agent dynamically suppresses established baseline anomalies and adapts its detection heuristics based on previous human-in-the-loop corrections.<\/p>\n\n\n\n

To validate the agent’s detection capabilities under controlled and reproducible conditions, we developed a synthetic data generation pipeline that injects known structural and logical anomalies at configurable rates into anonymised, marketing data. Evaluation was conducted across 3 experimental configurations<\/strong>, spanning three prompt complexity levels and two memory modes, with scoring fully automated via an LLM-as-a-Judge pipeline using Gemini as the evaluator. The agent achieved a peak detection rate of 0.883%<\/strong> across all injected error categories under forensic-level prompting.<\/p>\n\n\n\n

\n\n\n\n

Agent architecture<\/strong><\/h2>\n\n\n\n

The solution is structured around a hierarchical multi-agent orchestration pattern<\/strong>, with a central Root Agent coordinating seven specialized sub-agents, as illustrated in the diagram below. The Root Agent functions as an LLM-powered intent classifier: it parses each incoming user request, decomposes compound instructions into an ordered execution plan, and dynamically routes sub-tasks to the appropriate specialist agent, without relying on hard-coded conditional routing logic. This design enables the system to handle chained, multi-step requests (e.g., “query the database and then plot the results”<\/em>) by composing multiple sub-agents in sequence within a single session. The architecture was inspired by ADK\u2019s official examples repo<\/a>.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Sub-agent inventory<\/strong><\/h2>\n\n\n\n

The system employs a multi-agent orchestration architecture where a primary Root Agent<\/strong> delegates tasks to specialized sub-agents based on user intent. All agents are powered by Gemini 2.5 Flash<\/strong>, optimised for complex multi-step reasoning, low inference latency, and cost efficiency under high request volume.<\/p>\n\n\n\n

Sub-Agent<\/strong><\/th>Core Responsibility<\/strong><\/th><\/tr><\/thead>
Auditor Agent<\/strong><\/td>Drives autonomous data quality auditing by executing structural and logical checks against BigQuery, maintaining historical context via the Memory Bank.<\/td><\/tr>
BigQuery Agent<\/strong><\/td>Facilitates Text-to-SQL (NL2SQL) translation, generating optimized queries and executing them directly against the data warehouse.<\/td><\/tr>
Analytics Agent<\/strong><\/td>Performs Advanced Data Analysis (NL2Py) by dynamically generating and executing Python code within a secure Vertex AI sandbox for statistical profiling and visualization.<\/td><\/tr>
BQML Agent<\/strong><\/td>Orchestrates BigQuery ML workflows, including model training, batch inference, and model lifecycle management.<\/td><\/tr>
Artifact Agent<\/strong><\/td>Handles session-scoped file management to save, retrieve, and list generated execution artifacts (images, CSVs, PDFs).<\/td><\/tr>
Report Agent<\/strong>*<\/td>Synthesizes audit findings into multi-format reports (Markdown, HTML, PDF, JSON) and manages artifact uploads to Google Cloud Storage.<\/td><\/tr>
Comparison Agent<\/strong>*<\/td>Executes schema-level and volume-level structural comparisons across discrete BigQuery tables.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n
  • Note: The Report and Comparison agents are fully implemented in the repository but are intentionally disconnected from the Root Agent’s execution chain in this evaluation instance.<\/em><\/li>\n<\/ul>\n\n\n\n

    Key system capabilities<\/strong><\/h2>\n\n\n\n
      \n
    1. Dynamic Intent Classification:<\/strong> The Root Agent accurately decomposes complex natural language requests, determines the optimal execution path, and dynamically invokes the correct sub-agent chain.<\/li>\n\n\n\n
    2. NL2SQL Querying:<\/strong> The BigQuery Agent translates natural language into optimised SQL, executing it directly against the data warehouse to extract and analyze data without friction.<\/li>\n\n\n\n
    3. NL2Py Analysis:<\/strong> The Analytics Agent dynamically generates and executes Python code within a secure Vertex AI Code Interpreter sandbox, enabling advanced statistical profiling, custom visualisations, and complex cross-dataset joins.<\/li>\n\n\n\n
    4. Autonomous Data Auditing:<\/strong> The Auditor Agent runs a comprehensive suite of structural and logical validation checks against BigQuery datasets, producing structured, reproducible diagnostic reports.<\/li>\n\n\n\n
    5. Stateful Memory Persistence:<\/strong> By querying a persistent Memory Bank<\/strong>, the Auditor contextualizes newly detected anomalies against historically resolved or suppressed issues, ensuring the agent learns and adapts from past executions.<\/li>\n\n\n\n
    6. Multi-Format Report Compilation:<\/strong> The Report Agent synthesizes raw audit findings into polished, user-preferred output formats and automatically pushes the final artifacts to Google Cloud Storage for human review.<\/li>\n<\/ol>\n\n\n\n

      Long-term memory bank<\/strong><\/h2>\n\n\n\n

      The system’s persistent Memory Bank<\/a><\/strong>, hosted on Vertex AI Agent Engine<\/a>, gives the auditor institutional knowledge across sessions, eliminating cold-start noise and adapting its behaviour to individual user preferences over time. The Memory Bank tracks two custom semantic categories:<\/p>\n\n\n\n

      Topic<\/strong><\/th>Examples<\/strong><\/th><\/tr><\/thead>
      Data Quality Issues<\/strong><\/td>Missing columns, inflated metric values, recurring table-level anomalies<\/td><\/tr>
      User Preferences<\/strong><\/td>“Always include an executive summary”<\/em>; “Flag outliers beyond 3\u03c3”<\/em><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

      How memory is saved<\/strong><\/h3>\n\n\n\n

      Memory persistence is user-directed. The Auditor invokes the save_memory<\/code> tool only when explicitly asked (e.g., “\u2026and save these findings to memory”<\/em>). The Vertex AI Agent Engine<\/strong> then asynchronously extracts clean semantic facts from the session, stripping noise and verbose phrasing, and indexes them against the user’s user_id<\/code> scope. When a new lesson or correction occurs, the agent doesn’t just blindly append a new memory; instead, it actively scans for similar existing entries. If a related memory is found, the system updates and refines the existing rule rather than creating a duplicate. This deduplication process ensures the knowledge base remains clean, concise, and highly effective, preventing the auditor from getting overwhelmed by redundant information over time.<\/p>\n\n\n\n

      Agent Engine also persists the full conversation session<\/strong> alongside the extracted facts, meaning the complete interaction history (including tool calls, SQL queries, and agent reasoning) is retained across runs. This gives the system two complementary layers of recall: structured fact memory<\/strong> (distilled semantic facts) and full session continuity<\/strong> (complete conversation history), both managed within a single Vertex AI service.<\/p>\n\n\n\n

      How memory is loaded<\/strong><\/h3>\n\n\n\n

      When a query includes a load instruction (e.g., “load memories then check table X”<\/em>), the Auditor calls the ADK LoadMemoryTool<\/code>, which runs a similarity search<\/strong> against the Memory Bank scoped to the current user_id<\/code>. Retrieved facts are injected into the agent’s working context before analysis begins, enabling it to:<\/p>\n\n\n\n

        \n
      • Suppress re-flagging of known, already-resolved issues<\/li>\n\n\n\n
      • Apply user formatting preferences from the first response<\/li>\n\n\n\n
      • Re-verify previously detected anomalies to check if they persist<\/li>\n<\/ul>\n\n\n\n
        \n\n\n\n

        Technology stack<\/strong><\/h1>\n\n\n\n
        Component<\/strong><\/th>Technology<\/strong><\/th>Role<\/strong><\/th><\/tr><\/thead>
        Agent Framework<\/strong><\/td>Google Agent Development Kit (ADK)<\/a><\/td>Agent orchestration, tool binding, session management, and A2A protocol support<\/td><\/tr>
        LLM (Agents)<\/strong><\/td>Gemini 2.5 Flash<\/a><\/td>Powers all sub-agents; chosen for low latency and strong instruction-following<\/td><\/tr>
        LLM (Judge)<\/strong><\/td>Gemini 2.5 Flash<\/a><\/td>Powers the LLM-as-a-Judge evaluation pipeline; stronger reasoning for unbiased scoring<\/td><\/tr>
        Data Warehouse<\/strong><\/td>Google BigQuery<\/a><\/td>Primary data store; queried via NL2SQL by both the BigQuery and Auditor agents<\/td><\/tr>
        Code Execution<\/strong><\/td>Vertex AI Code Interpreter<\/a><\/td>Sandboxed Python runtime for the Analytics Agent<\/td><\/tr>
        Long-Term Memory<\/strong><\/td>Vertex AI Agent Engine<\/a><\/td>Hosts the persistent Memory Bank; enables cross-session learning and recall<\/td><\/tr>
        Artifact Storage<\/strong><\/td>Google Cloud Storage<\/a><\/td>Persistent store for generated reports, JSON profiles, and session artifacts<\/td><\/tr>
        Deployment<\/strong><\/td>Google Cloud Run<\/a><\/td>Hosts both the A2A API backend and the interactive web UI as containerized services<\/td><\/tr>
        CI\/CD<\/strong><\/td>Bitbucket Pipelines<\/a><\/td>Automated build, Docker image push to Artifact Registry, and Cloud Run deployment on merge<\/td><\/tr>
        Interoperability<\/strong><\/td>A2A Protocol<\/a><\/td>Open HTTP-based standard enabling external agents and services to discover and invoke the system programmatically<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
        \n\n\n\n

        Dataset synthesis<\/strong><\/h2>\n\n\n\n

        Evaluating an autonomous auditing agent requires a controlled, reproducible ground truth, something that real-world production data cannot provide, since errors are unverified by definition. To solve this, we engineered a modular synthetic corruption pipeline<\/strong> that operates on proprietary synthetic datasets<\/a> designed to mirror real-world marketing dynamics, and produces deterministically corrupted BigQuery tables accompanied by a complete ground truth registry for automated scoring.<\/p>\n\n\n\n

        Source dataset<\/strong><\/h3>\n\n\n\n

        To ensure robust and repeatable results, we start with proprietary synthetic datasets<\/a> giving us complete control and clear visibility into the drivers of campaign performance. The source dataset is a digital marketing performance table comprising 7,618 rows and 87 feature columns<\/strong>. Each record represents a unique daily measurement at the intersection of a campaign, audience segment, delivery platform, ad placement, and creative asset. Columns are organised into six functional groups:<\/p>\n\n\n\n

        Column Group<\/strong><\/th>Description<\/strong><\/th><\/tr><\/thead>
        Brand & Advertiser<\/strong><\/td>Identity of the brand and advertiser running the campaign<\/td><\/tr>
        Campaign & Media Buy<\/strong><\/td>Campaign IDs, names, and media buying hierarchy<\/td><\/tr>
        Geo Targeting<\/strong><\/td>Geographic targeting and exclusion rules (countries, regions, cities)<\/td><\/tr>
        Audience Targeting<\/strong><\/td>Demographic segments: gender, age group, generation, interests, and behaviors<\/td><\/tr>
        Delivery & Platform<\/strong><\/td>Campaign objective, platform (Meta\/Instagram), device type, and ad placement<\/td><\/tr>
        Performance Metrics<\/strong><\/td>Funnel KPIs: impressions, clicks, spend, conversions, video plays, video completions, landing page views, add-to-cart events, and purchases<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

        The Performance Metrics group is the most analytically significant: the columns encode a strict, real-world causal funnel (impressions \u2192 clicks \u2192 landing page views \u2192 add-to-cart \u2192 conversions\/purchases<\/code>) where each downstream metric is physically bounded by the upstream one. Violations of these relationships (for example, clicks > impressions<\/code>) are logically impossible under normal operating conditions. This funnel structure forms the basis for all logical error injection. Additionally, the dual attribution windows (immediate vs. 7-day) introduce latent complexity: the complex prompt level successfully identified cross-window contradictions as an un-injected source of potential logical ambiguity.<\/p>\n\n\n\n

        Corruption pipeline<\/strong><\/h3>\n\n\n\n

        The pipeline is structured as a three-stage process. Anonymisation is performed first, followed by structural error injection, and concluding with logical error injection. These stages consist of composable modules that can be selectively enabled or combined to produce datasets with precisely controlled corruption profiles. The error rate is fully configurable per stage and can be held constant<\/strong> (for fixed-recall benchmarks) or varied progressively<\/strong> from 5 to 40% (to model the agent’s sensitivity as a function of corruption severity).<\/p>\n\n\n\n

        Stage 1 \u2014 Anonymisation<\/strong><\/h4>\n\n\n\n

        As a preprocessing step, the pipeline replaces PII and commercially sensitive fields (brand, campaign, creative) with generic identifiers (e.g., brand_1<\/code>, campaign_1<\/code>), while cleanly preserving all structural relationships.<\/p>\n\n\n\n

        Stage 2 \u2014 Structural errors<\/strong><\/h4>\n\n\n\n

        Structural anomalies target individual cells, columns, or rows, and are generally detectable through standard data profiling techniques. This stage consists of five independent injection modules:<\/p>\n\n\n\n

        Error Type<\/strong><\/th>Simulation \/ Injection Method<\/strong><\/th><\/tr><\/thead>
        Missing Values<\/strong> (Nulls)<\/td>Injects NaN<\/code> values across a configurable subset of columns to simulate missing or dropped data.<\/td><\/tr>
        Outliers<\/strong><\/td>Replaces numeric values with statistical extremes (mean \u00b1 k \u00d7 std<\/code>) to simulate sensor noise or ETL overflow.<\/td><\/tr>
        Duplicate Rows<\/strong><\/td>Duplicates randomly selected rows and re-inserts them at random positions to simulate pipeline idempotency failures.<\/td><\/tr>
        Categorical Errors<\/strong><\/td>Replaces valid categories with unique random alphanumeric strings (e.g., a3x7h9<\/code>) guaranteed not to be in any valid vocabulary.<\/td><\/tr>
        Schema Drift<\/strong> (Col Drops)<\/td>Randomly removes entire columns to simulate upstream data source failures.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

        Stage 3 \u2014 Logical errors<\/strong><\/h3>\n\n\n\n

        Logical errors are the hardest class of anomalies to detect. Every individual cell value is numerically valid; the violation only becomes apparent when two or more columns are evaluated relationally. This stage injects records that violate any of the following seven business rules:<\/p>\n\n\n\n

        #<\/strong><\/th>Rule Violated<\/strong><\/th>Condition Injected<\/strong><\/th><\/tr><\/thead>
        1<\/td>Clicks \u2264 Impressions<\/td>clicks > impressions<\/code><\/td><\/tr>
        2<\/td>Conversions \u2264 Clicks<\/td>conversions > clicks<\/code><\/td><\/tr>
        3<\/td>Spend requires Impressions<\/td>spend > 0<\/code> AND impressions = 0<\/code><\/td><\/tr>
        4<\/td>Video Completions \u2264 Plays<\/td>video_completions > video_plays<\/code><\/td><\/tr>
        5<\/td>Purchases require Add-to-Cart<\/td>purchases > 0<\/code> AND add_to_cart = 0<\/code><\/td><\/tr>
        6<\/td>Landing Page Views \u2264 Clicks<\/td>landing_page_views > clicks<\/code><\/td><\/tr>
        7<\/td>Non-negative Metric Values<\/td>Negative values injected into impressions<\/code>, clicks<\/code>, spend<\/code>, or conversions<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

        Ground truth registry<\/strong><\/h2>\n\n\n\n

        The evaluation framework is anchored by our ground truth dataset,<\/strong> a structured registry of all 59 BigQuery test tables<\/strong> used in the experiment suite. Each row maps a table’s BigQuery name to its complete injection specification:<\/p>\n\n\n\n

          \n
        • the number of logical errors injected (out of a maximum of 7 possible rule types)<\/li>\n\n\n\n
        • the exact error type labels (e.g., clicks_exceed_impressions, purchases_without_add_to_cart)<\/li>\n\n\n\n
        • the number of structural errors injected (out of 4 possible types), and their corresponding labels (e.g., null values, outliers, duplicates, categorical errors)<\/li>\n<\/ul>\n\n\n\n

          The registry covers two tiers of test tables: 48 single-error tables<\/strong> (examples 1\u201348), each containing one isolated error type at varying injection rates of 5%, 10%, 20%, and 40%, and 11 compound synthetic tables<\/strong> (examples 49\u201359) with progressively stacked errors, starting from a single logical violation and escalating to the maximum combination of all 7 logical and all 4 structural error types simultaneously.<\/p>\n\n\n\n

          Table Name<\/strong><\/th>Logical Errors<\/strong><\/th>Logical Error Types<\/strong><\/th>Structural Errors<\/strong><\/th>Structural Error Types<\/strong><\/th><\/tr><\/thead>
          ..._categorical_error_5_percent<\/code><\/td>0<\/td>\u2014<\/td>1 \/ 4<\/td>categorical errors<\/code><\/td><\/tr>
          ..._logical_error_1_5_percent<\/code><\/td>1 \/ 7<\/td>clicks_exceed_impressions<\/code><\/td>0<\/td>\u2014<\/td><\/tr>
          log_1_5_pt..._dup_5_pt_cat_5_pt<\/code><\/td>7 \/ 7<\/td>clicks_exceed_impressions, conversions_exceed_clicks, landing_page_views_exceed_clicks, negative_metric_values, purchases_without_add_to_cart, spend_with_zero_impressions, video_completions_exceed_plays<\/code><\/td>4 \/ 4<\/td>null values, outliers, duplicates, categorical errors<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          \n\n\n\n

          Evaluation pipeline<\/h1>\n\n\n\n

          Rigorous evaluation of the auditor agent is essential to ensure it consistently and accurately identifies true data corruption without generating false positives. To accomplish this, the evaluation pipeline uses an automated, four-step process to continuously assess the agent’s performance. First, the pipeline utilises synthetic ground truth data stored in BigQuery tables, seeded with deliberate structural and logical errors (such as NULLs, duplicates, and business-rule violations). Second, the auditor agent is executed against these tables through multiple experimental setups, including prompt comparisons (simple vs. complex queries), table anomaly sweeps, and memory ablation studies (cold starts vs. loading past audits). During these runs, the agent uses its SQL tools to investigate the data and generates a comprehensive final audit report.<\/p>\n\n\n\n

          Third, rather than relying on slow manual review, we automate the evaluation using an LLM-as-a-Judge approach. A separate Gemini Flash instance receives the agent’s full audit report alongside the complete ground truth registry. Acting as an expert evaluator, the judge compares the outputs and produces a structured scorecard with \u2705\/\u274c verdicts and brief explanations for every error category. This eliminates subjective scoring bias and allows new prompt designs or memory configurations to be evaluated end-to-end in minutes. Finally, these scorecards are parsed to compute precision, recall, and F1 scores per error type, which are then exported to CSV for detailed analysis.<\/p>\n\n\n\n

          This is also illustrated in the diagram below:<\/p>\n\n\n\n

          \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502                       EVALUATION PIPELINE                      \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 1. Synthetic Data   \u2502 Tables in BigQuery with injected errors: \u2502\n\u2502    (ground truth)   \u2502 NULLs, duplicates, outliers, categorical \u2502\n\u2502                     \u2502 errors, logical violations.              \u2502\n\u2502                     \u2502                                          \u2502\n\u2502 2. Run Auditor      \u2502 4 experiments test different factors:    \u2502\n\u2502    Agent            \u2502  \u2192 Exp 1: Prompt Comparison              \u2502\n\u2502                     \u2502  \u2192 Exp 2: Table Sweep                    \u2502\n\u2502                     \u2502  \u2192 Exp 3: Memory Ablation                \u2502\n\u2502                     \u2502 Agent uses tools to run SQL & produce an \u2502\n\u2502                     \u2502 audit report per run.                    \u2502\n\u2502                     \u2502                                          \u2502\n\u2502 3. LLM-as-Judge     \u2502 Gemini Flash compares agent report to    \u2502\n\u2502    (Gemini Flash)   \u2502 ground truth.                            \u2502\n\u2502                     \u2502  \u2192 Scores each error: \u2705 detected \/ \u274c   \u2502\n\u2502                     \u2502                                          \u2502\n\u2502 4. Metrics          \u2502 Parse scorecards \u2192 compute precision,    \u2502\n\u2502    Generation       \u2502 recall, F1 per error type.               \u2502\n\u2502                     \u2502  \u2192 Save to CSV                           \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n<\/code><\/pre>\n\n\n\n
          Experimental setup<\/h2>\n\n\n\n
          To rigorously validate the Auditor agent’s detection capabilities, we designed a suite of three complementary experiments, each isolating a different factor that influences audit performance:<\/p>\n\n\n\n
            \n
          1. Experiment 1 \u2014 Prompt Comparison:<\/strong> Measures how the complexity and specificity<\/em> of the user prompt affects the agent’s ability to detect both structural and logical errors, comparing a simple exploratory prompt against a medium-structured prompt and a forensic-level complex prompt.<\/li>\n\n\n\n
          2. Experiment 2 \u2014 Table Sweep:<\/strong>\u00a0Stress-tests the agent’s scalability and robustness<\/em> by sweeping across 11 synthetic tables with progressively stacked error combinations, ranging from a single isolated violation to the maximum of 11 simultaneous error types. This maps the detection ceiling under the best-performing prompt.<\/li>\n\n\n\n
          3. Experiment 3 \u2014 Memory Ablation:<\/strong> Isolates the contribution of the long-term Memory Bank<\/em> by comparing a cold-start baseline (no prior context) against a memory-augmented run, quantifying how historical context from past audit sessions improves detection accuracy.<\/li>\n<\/ol>\n\n\n\n
            Together, these experiments span the key dimensions of agent performance: prompt engineering, error complexity, and contextual memory, providing a comprehensive view of the system’s strengths and current limitations. All experiments use the same synthetic corruption pipeline and LLM-as-a-Judge scoring framework described above.<\/p>\n\n\n\n
            Experiment 1: Prompt Comparison<\/h3>\n\n\n\n
            Our first research question was whether prompt specification<\/strong> (instructional structure, domain constraints, and required check set) is a first-order driver of audit performance, independent of the underlying dataset and injected corruption profile. In other words, does increasing prompt information content<\/strong> and enforcing explicit cross-column invariants<\/strong> improve the agent\u2019s ability to surface structural anomalies and relational business-rule violations, and what is the marginal lift as we move from a zero-shot \u201chealth check\u201d prompt to a forensic, hypothesis-driven audit prompt?<\/p>\n\n\n\n
            To isolate this variable, we held the dataset and error profile constant, injecting known errors at a flat 5% rate per type into a table of anonymised marketing data, and varied only the prompt complexity across three levels:<\/p>\n\n\n\n
            Prompt Level<\/strong><\/th>Description<\/strong><\/th><\/tr><\/thead>
            Simple<\/td>Basic health check: explore, verify, report<\/td><\/tr>
            Medium<\/td>Structured assessment organized by data quality pillars<\/td><\/tr>
            Complex<\/td>Forensic audit with cross-column hypothesis testing and business context<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Results<\/h4>\n\n\n\n
            To quantify the impact of prompt engineering, we measured the detection accuracy for each of the three prompt levels against our ground truth dataset. The table below summarizes the results:<\/p>\n\n\n\n
            Metric<\/strong><\/th>Simple Prompt<\/strong><\/th>Medium Prompt<\/strong><\/th>Complex Prompt<\/strong><\/th><\/tr><\/thead>
            Structural errors detected<\/td>3\/4<\/td>3\/4<\/td>4\/4<\/td><\/tr>
            Logical errors detected<\/td>1\/7<\/td>3\/7<\/td>4\/7<\/td><\/tr>
            Total score<\/td>4\/11 (36%)<\/td>6\/11 (55%)<\/td>8\/11 (73%)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            The Simple Prompt (scoring 4 out of 11) successfully detected missing values, outliers, categorical errors, and negative metric values, but failed to detect duplicate rows and missed most cross-column logical violations. The Medium Prompt (scoring 6 out of 11) was a significant step up; it detected missing values, identified duplicate rows, and found categorical errors, while additionally detecting key funnel violations like clicks being greater than impressions and conversions being greater than clicks. The Complex Prompt (scoring 8 out of 11) was the strongest performer, achieving 100% on structural errors with forensic-level explanations. On logical errors, it detected negative metrics, two funnel violations, and video completion inconsistencies, and the Auditor also discovered un-injected errors beyond the seeded corruption, including data mapping flaws. Our key observations are as following:<\/p>\n\n\n\n
              \n
            1. Prompt complexity directly impacts detection quality.<\/strong> Moving from simple to complex prompts increased total detection from 36% to 73%.<\/li>\n\n\n\n
            2. Structural errors are easier to detect than logical errors.<\/strong> Even the simplest prompt found 75% of structural errors, while logical error detection ranged from 14% to 57%.<\/li>\n\n\n\n
            3. The complex prompt exhibited emergent behaviour<\/strong>, discovering data quality issues beyond the injected errors, which validates the agent’s analytical depth. Specifically, it identified a many-to-one mapping flaw where a single campaign_id mapped to multiple campaign_names, and logical contradictions between 7-day and immediate conversion windows.<\/li>\n\n\n\n
            4. Error analysis reveals specific failure modes.<\/strong> For the “Spend > 0 while Impressions = 0” error, the agent checked the inverse condition (“Impressions > 0 AND Spend = 0”), demonstrating that the agent’s logical reasoning was sound but directionally inverted. This suggests that targeted few-shot examples or tool-level guardrails could address remaining gaps.<\/li>\n\n\n\n
            5. Certain error types remain challenging<\/strong> regardless of prompt level, particularly those requiring knowledge of the full marketing funnel (e.g., purchases without add-to-cart, landing page views vs. clicks). These represent areas for future improvement because evaluating complex logical anomalies requires a deep contextual understanding of domain-specific business rules. Providing this context, whether through a persistent memory system that stores historical performance baselines and funnel definitions, or via highly explicit user prompts that clearly map expected relationships, is essential for the agent to accurately validate these scenarios rather than relying on generic data logic.<\/li>\n<\/ol>\n\n\n\n
              Experiment 2: Table sweep<\/h3>\n\n\n\n
              Having identified the complex prompt as the strongest performer, we next evaluated its scaling behaviour under increasing anomaly superposition<\/strong>: specifically, how detection performance (precision\/recall trade-offs) degrades or saturates as the number of simultaneously injected error modes per table increases. While a single-error table primarily probes per-check sensitivity<\/em>, production-like settings exhibit error co-occurrence and interaction effects<\/strong> (masking, confounding, and correlated rule violations) that can materially alter the agent\u2019s search strategy, query budget, and false-positive propensity.<\/p>\n\n\n\n
              To probe this, we ran the Auditor against 11 synthetic BigQuery tables<\/strong> with progressively stacked error combinations \u2014 from a single isolated logical violation up to the maximum of all 7 logical and all 4 structural error types simultaneously (11 errors total per table). All runs used the complex<\/code> prompt level, allowing us to map the agent’s detection ceiling as the error landscape grows increasingly complex.<\/p>\n\n\n\n
              Results: Per-table and Aggregate Metrics<\/strong><\/p>\n\n\n\n
              *(Legend: L = Logical errors, S = Structural errors)<\/p>\n\n\n\n
              Table<\/strong><\/th>Error Profile<\/strong><\/th>Expected<\/strong><\/th>TP<\/strong><\/th>FP<\/strong><\/th>FN<\/strong><\/th>F1 Score<\/strong><\/th><\/tr><\/thead>
              synthetic_1_log_error<\/code><\/td>1L<\/td>1<\/td>1<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_2_log_errors<\/code><\/td>2L<\/td>2<\/td>2<\/td>6<\/td>0<\/td>0.400<\/strong> \u26a0\ufe0f<\/td><\/tr>
              synthetic_3_log_errors<\/code><\/td>3L<\/td>3<\/td>0<\/td>0<\/td>3<\/td>0.000<\/strong> \u274c<\/td><\/tr>
              synthetic_4_log_errors<\/code><\/td>4L<\/td>4<\/td>4<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_5_log_errors<\/code><\/td>5L<\/td>5<\/td>5<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_6_log_errors<\/code><\/td>6L<\/td>6<\/td>6<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_7_log_errors<\/code><\/td>7L<\/td>7<\/td>7<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_7_log_1_struct<\/code><\/td>7L+1S<\/td>8<\/td>2<\/td>0<\/td>6<\/td>0.400<\/strong> \u26a0\ufe0f<\/td><\/tr>
              synthetic_7_log_2_struct<\/code><\/td>7L+2S<\/td>9<\/td>9<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_7_log_3_struct<\/code><\/td>7L+3S<\/td>10<\/td>10<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr>
              synthetic_7_log_4_struct<\/code><\/td>7L+4S<\/td>11<\/td>11<\/td>0<\/td>0<\/td>1.000<\/strong> \u2705<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
              Metric<\/strong><\/th>Value<\/strong><\/th><\/tr><\/thead>
              Perfect Detection (F1 = 1.0)<\/strong><\/td>8 \/ 11 tables (72.7%)<\/strong><\/td><\/tr>
              Total True Positives (TP)<\/strong><\/td>57<\/td><\/tr>
              Total False Positives (FP)<\/strong><\/td>6<\/td><\/tr>
              Total False Negatives (FN)<\/strong><\/td>9<\/td><\/tr>
              Overall Precision<\/strong><\/td>57 \/ 63 = 0.905<\/strong><\/td><\/tr>
              Overall Recall<\/strong><\/td>57 \/ 66 = 0.864<\/strong><\/td><\/tr>
              Overall F1 Score<\/strong><\/td>0.883<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
              We also tested the auditor agent’s baseline ability to detect the same logical error at different prevalence levels (5%, 10%, 20%, and 40%). The agent successfully detected and accurately quantified the discrepancy at the 5%, 10%, 20% and 40% rates, demonstrating robust, range-agnostic capability that catches both rare edge cases and widespread corruption equally well. Ultimately, the results indicate that error rate prevalence does not significantly impact the agent’s detection performance when the audit completes successfully.<\/p>\n\n\n\n
              Finally, we ran the identical configuration three times for one table as a consistency check<\/strong>, and observed perfect reproducibility: the auditor consistently detected both injected errors with the same metrics and explanations across all three runs. This deterministic behaviour indicates that the complex prompt configuration is stable, reducing the need for redundant audits.<\/p>\n\n\n\n
              Experiment 3: Memory ablation<\/h3>\n\n\n\n
              The previous experiments characterized the agent\u2019s single-session capability envelope<\/strong> under a fixed prompt specification. In a production setting, however, auditing is inherently iterative and longitudinal<\/strong>: the agent re-encounters the same schemas, recurring anomaly modes, and known \u201cbenign\u201d deviations across repeated runs. This motivates a key question: does persistent, user-scoped memory<\/strong> (i.e., accumulated priors from prior audits) measurably improve detection performance and efficiency over time by biasing the agent toward higher-yield checks, reinstating domain-specific invariants without re-deriving them from scratch?<\/p>\n\n\n\n
              To isolate the contribution of the long-term Memory Bank, we ran the agent twice on the same table under identical conditions, first with no prior context (cold start) and then with memories loaded from previous audit sessions. We evaluated the agent on a synthetic table (synthetic_7_log_4_struct<\/code>) containing 7,999 rows, deliberately corrupted with 11 distinct error types (4 structural, 7 logical) at a ~5% error rate. The two conditions differed only in whether the agent had access to its Memory Bank before beginning the audit.<\/p>\n\n\n\n
              Results<\/h4>\n\n\n\n
              Without memory, the agent received a minimalist zero-shot prompt (“Check if there are any errors for table X?”<\/em>) and relied solely on exploratory analysis. Under these cold-start conditions, it achieved an overall detection rate of 45% (5\/11)<\/strong>, identifying 2 of 4 structural errors and 3 of 7 logical errors.<\/p>\n\n\n\n
              When the same agent was instructed to load past context (“load memories about auditing tables…”<\/em>), the results improved dramatically. By retrieving specific logical checks and known error patterns from prior sessions, the memory-augmented agent achieved a 91% detection rate (10\/11)<\/strong>, a 102% relative improvement over the baseline.<\/p>\n\n\n\n
              Structural error detection reached a perfect 100% (4\/4), while logical error detection rose from 43% to 86% (6\/7), successfully uncovering complex violations such as negative metric values and spend recorded against zero impressions.<\/p>\n\n\n\n
              \"\"<\/figure>\n\n\n\n
              The figure shows a clear performance gap between the memory-augmented agent (blue) and the baseline agent without memory (red). For structural errors, memory enabled perfect detection (100%) compared to 50% without memory. For logical errors, memory improved detection from 43% to 86%, demonstrating that access to prior audit patterns and domain knowledge substantially enhances the agent’s ability to identify complex data quality issues beyond basic exploratory analysis.<\/p>\n\n\n\n
              The sole undetected error was a funnel sequence violation (purchases without add-to-cart). The agent did not miss this check due to a detection failure. It correctly reasoned that the validation was impossible given the aggregated schema, which lacked the transaction-level granularity required to verify a purchase-to-cart relationship. This suggests the miss was an analytically sound decision rather than a detection failure.<\/p>\n\n\n\n
              Memory vs. prompt complexity<\/strong><\/h3>\n\n\n\n
              These results raise an important nuance: if a prompt is already sufficiently detailed and structurally prescriptive (as in our complex prompt from Experiment 1), the memory module provides only marginal uplift. However, memory becomes highly valuable in continuous operational scenarios, where its benefits compound over time:<\/p>\n\n\n\n
                \n
              • Adaptability:<\/strong> The agent iteratively learns from past edge cases, refining its checks with each audit cycle.<\/li>\n\n\n\n
              • Contextual Awareness:<\/strong> It builds a deep, automated understanding of project-specific business rules and historically common data quality issues.<\/li>\n\n\n\n
              • Consistency & Efficiency:<\/strong> Audit coverage remains stable across sessions, with fewer redundant exploratory queries needed to reach comprehensive detection.<\/li>\n<\/ul>\n\n\n\n
                \n\n\n\n
                Cloud deployment<\/strong><\/h1>\n\n\n\n
                The system is deployed as a production-grade, cloud-native service on Google Cloud<\/strong>, following a containerised, infrastructure-as-code workflow from local development through to automated CI\/CD and managed compute.<\/p>\n\n\n\n
                CI\/CD pipeline<\/strong><\/h2>\n\n\n\n
                The project uses a fully automated Bitbucket Pipelines<\/strong> CI\/CD pipeline with two distinct execution stages:<\/p>\n\n\n\n
                  \n
                • On Pull Request:<\/strong> Automated linting and static analysis run immediately to enforce code quality standards before any merge is permitted.<\/li>\n\n\n\n
                • On Merge to main<\/code>:<\/strong> The pipeline builds two independent Docker images<\/strong> (one for the headless A2A API backend, one for the interactive web UI), pushes both to Google Artifact<\/strong> Registry<\/strong>, and triggers rolling deployments to their respective Cloud Run<\/strong> services. All runtime configuration (model identifiers, dataset IDs, memory service URIs, Cloud Storage bucket names) is injected exclusively via environment variables, ensuring no secrets or environment-specific values are hardcoded into the images.<\/li>\n<\/ul>\n\n\n\n
                  Dual-service deployment architecture<\/strong><\/h2>\n\n\n\n
                  The agent is deployed as two independent, containerised Cloud Run services<\/strong>, each built from its own Dockerfile and serving a distinct class of consumer:<\/p>\n\n\n\n
                  Service 1 \u2014 A2A API backend<\/strong><\/h3>\n\n\n\n
                  The backend service exposes a headless Agent-to-Agent (A2A)<\/strong> interface, an open, HTTP-based protocol designed for agent interoperability across frameworks. It publishes an Agent Card <\/strong>(a structured capability manifest) that allows any external service or AI agent to programmatically discover what the Data Quality Agent can do without requiring any knowledge of the underlying ADK implementation.<\/p>\n\n\n\n
                  Clients interact with the backend by sending structured JSON-RPC messages<\/strong> over standard HTTP. This means the auditor can be:<\/p>\n\n\n\n
                    \n
                  • Integrated into classical data pipelines<\/strong> (like Airflow or dbt) to trigger automatic quality checks.<\/li>\n\n\n\n
                  • Orchestrated by other AI agents<\/strong> as part of a larger, automated workflow.<\/li>\n\n\n\n
                  • Invoked from any programming language<\/strong>, completely independent of the underlying Python stack.<\/li>\n\n\n\n
                  • Embedded in CI\/CD or alerting systems<\/strong> using simple HTTP requests.<\/li>\n<\/ul>\n\n\n\n
                    Service 2 \u2014 Interactive Web UI<\/strong><\/h3>\n\n\n\n
                    The web UI service hosts an interactive conversational frontend, allowing data engineers and data scientists to interact directly with the full agent system through a browser. It communicates with the agent backend and provides a session-aware interface where users can issue audit requests, review structured findings, retrieve generated reports, and provide manual corrections that are subsequently persisted to the Memory Bank.<\/p>\n\n\n\n
                    Google Cloud Agent Engine<\/strong> provides shared, persistent session storage for both services, ensuring that conversation context and session state survive container restarts and instance scale-out events.<\/p>\n\n\n\n
                    \n\n\n\n
                    Conclusion<\/h1>\n\n\n\n
                    This report demonstrates a highly effective and intelligent agent for automating data quality assurance, utilizing a long-term memory architecture that not only frees up valuable engineering resources but also gets smarter with every interaction. By reclaiming data engineering bandwidth, it liberates engineers to focus on building infrastructure rather than performing manual data QA. It also catches errors in BigQuery tables before data scientists spend hours training models on corrupted data, shifting quality checks earlier in the pipeline. Ultimately, this compound intelligence ensures the system never resets; instead, every manual correction and interaction makes the auditor permanently better and more adapted to our data ecosystem.<\/p>\n\n\n\n
                    Lessons learned<\/h3>\n\n\n\n
                      \n
                    • Test with Synthetic Data First<\/strong>: Without a meticulously crafted synthetic dataset, we would have had no objective way to measure if our prompt strategies were improving the agent’s performance.<\/li>\n\n\n\n
                    • Memory is Context, Context is King<\/strong>: The ability to retrieve facts from past runs, including past errors, user feedback, and specific constraints, is what makes the difference between a stateless tool and an adaptive auditor.<\/li>\n\n\n\n
                    • Start Specific, Then Generalize<\/strong>: We focused on nailing the Auditor Agent’s specific use case with BigQuery first. This created a robust foundation before we expanded to other functions like report generation.<\/li>\n\n\n\n
                    • Leverage a Unified Cloud Ecosystem<\/strong>: Building entirely on Google Cloud services<\/strong> (ADK, Vertex AI, BigQuery, Cloud Run, Cloud Storage) eliminated integration friction between components and allowed us to move from prototype to production deployment without stitching together tools from multiple vendors.<\/li>\n<\/ul>\n","protected":false},"author":20,"featured_media":741,"template":"","meta":{"_acf_changed":false},"pod_status":[{"id":4,"name":"Active","slug":"active"}],"ppma_author":[{"id":20,"display_name":"Anastasios Stamoulakatos","first_name":"Anastasios","last_name":"Stamoulakatos","nickname":"anastasios.stamoulakatos","user_nicename":"anastasios-stamoulakatos","user_email":"anastasios.stamoulakatos@satalia.com","biographical_info":"Anastasios (Tasos) Stamoulakatos is a Data Scientist at Satalia (WPP), focusing on agentic AI solutions for marketing. His work spans multi-agent systems, RAG and GraphRAG, and image retrieval, developing scalable AI solutions from early-stage POCs to production. He holds a PhD in Applied AI and Computer Vision from the University of Strathclyde and has over four years of commercial experience across industries including marketing, agriculture, pharmaceuticals, oil and gas, and manufacturing, with a strong focus on applied research and turning complex AI into practical business value.","avatar_url":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/03\/headshot_small.jpg","job_title":"Data Scientist","is_lead":null,"display_as_researcher":null,"order_priority":null},{"id":21,"display_name":"Thanos Lyras","first_name":"Thanos","last_name":"Lyras","nickname":"thanos.lyras","user_nicename":"thanos-lyras","user_email":"thanos.lyras@satalia.com","biographical_info":"Thanos Lyras is a data scientist at Satalia specializing in building end-to-end AI pipelines and deploying real-world AI applications. A graduate in Computer Engineering with an MSc in Data Science, his work has led to research contributions in the fields of Big Data, AI, and database performance. Currently, he is focused on pioneering research in agentic projects, exploring the next wave of artificial intelligence.","avatar_url":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/04\/profile_photo.png","job_title":"Data Scientist","is_lead":false,"display_as_researcher":true,"order_priority":null}],"class_list":["post-155","research_pod","type-research_pod","status-publish","has-post-thumbnail","hentry","pod_status-active"],"acf":{"subtitle":"From manual checks to machine intuition: an AI agent that guards your data quality and never forgets a lesson.","quarter":"Q1 2026","focus_focus":"","is_featured":false},"related_publications":[{"type":"research_feed","title":"New Blog Post","slug":"new-blog-post-2","date":"2026-03-30T14:58:56+00:00"},{"type":"blog","title":"Meet Your New Agentic Data Guardian","slug":"data-quality-assurance-agent-blog-post","date":"2026-03-24T15:40:43+00:00"}],"featured_image_url":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/03\/data_guardian_agent-resized-to-1024x600-1.jpeg","featured_image_sizes":{"thumbnail":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/03\/data_guardian_agent-resized-to-1024x600-1-150x150.jpeg","medium":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/03\/data_guardian_agent-resized-to-1024x600-1-300x176.jpeg","large":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/03\/data_guardian_agent-resized-to-1024x600-1.jpeg","full":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/03\/data_guardian_agent-resized-to-1024x600-1.jpeg"},"_links":{"self":[{"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=\/wp\/v2\/research_pods\/155","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=\/wp\/v2\/research_pods"}],"about":[{"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=\/wp\/v2\/types\/research_pod"}],"author":[{"embeddable":true,"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=\/wp\/v2\/users\/20"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=\/wp\/v2\/media\/741"}],"wp:attachment":[{"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=155"}],"wp:term":[{"taxonomy":"pod_status","embeddable":true,"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=%2Fwp%2Fv2%2Fpod_status&post=155"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/cms.research.wpp.com\/index.php?rest_route=%2Fwp%2Fv2%2Fppma_author&post=155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}