<\/p>\n
<\/p>\n
The core objective of this project is to move beyond traditional multimodal model training by leveraging the reasoning capabilities of Large Language Models (LLMs). We aim to convert complex multimodal data types (numerical data, images, text, and video) into a single unified modality: semantic text. This text is then used to predict media post performance.<\/p>\n
<\/p>\n
<\/p>\n
An autonomous agent performs iterative training of a downstream model by optimizing the prompt used for the multimodal-to-text conversion. This approach addresses two challenges in modern advertising analytics:<\/p>\n
<\/p>\n
<\/p>\n
A. Modality Conversion to Text<\/strong> Traditional feature extraction requires manually defining what matters (e.g., detecting faces, measuring brightness, OCR). This is rigid and often misses high-level semantic nuances like “humor,” “urgency,” or “brand alignment.”<\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n B. Prompt Optimization<\/strong> Human engineers often struggle to write the “perfect” prompt to extract the right predictive features. We might ask an LLM to “describe the image,” but we don’t know if mentioning the color palette is more predictive of ad success than mentioning the audio pace.<\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n The proposed architecture functions as a feedback loop consisting of three distinct engines: The Transmuter<\/strong>, The Predictor<\/strong>, and The Optimizer<\/strong>.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n This module is responsible for unifying the data modalities.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n This module evaluates performance based purely on the text output from Phase I.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n This is the meta-learning layer that improves the system over time without human intervention.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n Summary of Workflow:<\/strong><\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ol>\n <\/p>\n <\/p>\n Repeat:<\/strong><\/p>\n <\/p>\n <\/p>\n Loop until performance plateaus.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n We utilized the Instagram Influencer Dataset<\/strong><\/a> to extract text descriptions of posts and predict engagement metrics (such as the number of likes).<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n To better understand the target variables for our Predictor engine, we conducted a rigorous EDA on the dataset, revealing several key structural behaviours:<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n To ensure the integrity of our predictive modelling, we first applied filters based on our EDA. Roughly 13.8% of the dataset contained sponsor labels. We removed these sponsored posts entirely, as financial backing artificially skews organic engagement rates. We then narrowed our focus to create two high-density subsets: one featuring posts from the top 20 influencers, and a larger subset featuring the top 100 influencers (minimum 100 posts each). We opted to use the top 20 influences dataset in most of our experiments.<\/p>\n <\/p>\n <\/p>\n To handle the computational load of the iterative prompt optimization, we built a scalable cloud pipeline. Raw images and post metadata were staged in GCP Buckets<\/strong>. Gemini 2.5 Flash was deployed as the Transmuter<\/em> to generate the text profiles, capturing both general post context and specific image content. Because the agentic loop required regenerating descriptions for thousands of posts across multiple prompt iterations, we leveraged Google Batch Predictions<\/strong>. This allowed us to asynchronously and cost-effectively generate the text profiles for each optimization round. Finally, the poster\u2019s profile description, bio, and category were appended to the end of each generated description to provide complete semantic context for the downstream classifier.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n We initially framed the problem as a 3-class classification task (predicting low, average, and high likes) using a custom Deep Neural Network (three Linear layers with ReLU activation, and Cross-Entropy Loss). However, results showed that treating the problem as a regression task<\/strong> on the log(likes + 1) target yielded significantly better, more granular predictive performance.<\/p>\n <\/p>\n <\/p>\n For the regression task, we benchmarked three models:<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/ul>\n <\/p>\n <\/p>\n<\/p>\n
<\/p>\n
High-Level Approach<\/strong><\/h1>\n
Phase I: Semantic Translation (The Transmuter)<\/strong><\/h2>\n
<\/p>\n
Phase II: Performance Prediction (The Predictor)<\/h2>\n
<\/p>\n
Phase III: Iterative Optimization (The Optimizer Agent)<\/h2>\n
<\/p>\n
<\/p>\n
<\/figure>\n
Illustration of the proposed approach\n<\/code><\/pre>\nDataset<\/h1>\n
Dataset overview<\/h2>\n
<\/p>\n
Exploratory Data Analysis (EDA)<\/strong><\/h2>\n
<\/p>\n
![\"\"](\"https:\/\/research.wpp.com\/wp-content\/uploads\/2026\/03\/image-6.png\")
<\/figure>\n Histogram of likes and log(likes +1)\n<\/code><\/pre>\n![\"\"](\"https:\/\/research.wpp.com\/wp-content\/uploads\/2026\/03\/image-7.png\")
<\/figure>\n Scatter plot of likes vs followers\n<\/code><\/pre>\nResults<\/h1>\n
Data Preparation<\/h2>\n
Baseline Modelling: Classification vs. Regression<\/h2>\n
<\/p>\n
![\"\"](\"https:\/\/research.wpp.com\/wp-content\/uploads\/2026\/03\/image-8.png\")
<\/figure>\n