{"id":1823,"date":"2026-07-03T12:29:29","date_gmt":"2026-07-03T12:29:29","guid":{"rendered":"https:\/\/cms.research.wpp.com\/?post_type=research_feed&p=1823"},"modified":"2026-07-03T12:34:21","modified_gmt":"2026-07-03T12:34:21","slug":"are-open-source-embedding-models-good-enough-a-comparative-study","status":"publish","type":"research_feed","link":"https:\/\/cms.research.wpp.com\/?research_feed=are-open-source-embedding-models-good-enough-a-comparative-study","title":{"rendered":"Are open-source embedding models good enough? A comparative study"},"content":{"rendered":"\n

Throughout the years, a very large number of embedding models have emerged, each one having different strengths and weaknesses. We set out to answer a practical question many data professionals face often:<\/strong><\/p>\n\n\n\n

\n

Do premium embedding models significantly outperform free, open-source alternatives when predicting social media success?<\/strong><\/p>\n<\/blockquote>\n\n\n\n

\"\"
Figure 1<\/figcaption><\/figure>\n\n\n\n

In our latest research, we examined the performance of various embedding models across two distinct tasks. Our findings suggest that whilst premium models from industry leaders like OpenAI and Google technically produce the optimal results for most cases, the margin of victory is surprisingly narrow. Our main takeaway? For many predictive marketing tasks and Retrieval-Augmented Generation (RAG) tasks, free open alternatives are often good enough.<\/p>\n\n\n\n

The methodology and model selection<\/h2>\n\n\n\n

To test the capabilities of different embeddings, we designed two evaluation tasks:<\/p>\n\n\n\n

    \n
  1. Predictive downstream modelling<\/strong>: We generated embeddings from our data sources and fed them into a downstream LightGBM<\/strong> model to predict the final target (such as post popularity).<\/li>\n\n\n\n
  2. Information retrieval:<\/strong> We compared the embeddings inside a RAG framework using a vector index to measure direct context retrieval performance.<\/li>\n<\/ol>\n\n\n\n

    We tested seven different embedding models to get a comprehensive view of the landscape:<\/p>\n\n\n\n

    Model<\/strong><\/th>Provider<\/strong><\/th>Base Architecture<\/strong><\/th>Category<\/strong><\/th><\/tr><\/thead>
    text-embedding-005<\/code><\/td>Google<\/td>Proprietary<\/td>Premium (Paid API)<\/mark><\/strong><\/td><\/tr>
    text-embedding-3-large<\/code><\/td>OpenAI<\/td>Proprietary<\/td>Premium (Paid API)<\/mark><\/strong><\/td><\/tr>
    thenlper\/gte-base<\/code><\/td>Alibaba<\/td>BERT<\/td>Open-Source<\/mark><\/strong><\/td><\/tr>
    BAAI\/bge-base-en-v1.5<\/code><\/td>BAAI<\/td>BERT<\/td>Open-Source<\/mark><\/strong><\/td><\/tr>
    all-mpnet-base-v2<\/code><\/td>SBERT<\/td>MPNet (Microsoft)<\/td>Open-Source<\/mark><\/strong><\/td><\/tr>
    all-roberta-large-v1<\/code><\/td>SBERT<\/td>RoBERTa (Meta AI)<\/td>Open-Source<\/mark><\/strong><\/td><\/tr>
    all-miniLM-L6-v2<\/code><\/td>SBERT<\/td>MiniLM (Microsoft)<\/td>Open-Source<\/mark><\/strong><\/td><\/tr><\/tbody><\/table>
    Embedding models<\/figcaption><\/figure>\n\n\n\n

    These open-source alternatives were primarily chosen to correlate with the embedding dimensions of the premium baselines, while specifically including a smaller (all-miniLM-L6-v2<\/code>) and a larger (all-roberta-large-v1<\/code>) alternative to see how model size impacted performance.<\/p>\n\n\n\n

    \n\n\n\n

    Experiment 1: Predicting Instagram post popularity<\/h3>\n\n\n\n

    For our first experiment, we used the public Instagram Influencer Dataset<\/a>, which consists of various posts from different online influencers, to predict post popularity.<\/p>\n\n\n\n

    Since the raw data is highly visual, we first used Gemini to generate detailed textual descriptions of each post. We then converted these descriptions into embeddings using our seven chosen models, which were then used to train our downstream LightGBM model.<\/p>\n\n\n\n

    The results:<\/strong> OpenAI\u2019s text-embedding-3-large<\/code> produced the best overall results with an average R\u00b2 score of 0.475<\/strong>, closely followed by Google\u2019s text-embedding-005<\/code> at 0.470. However, the smallest model we tested (all-miniLM-L6-v2<\/code>) still achieved a score of 0.440<\/strong>. This competitive showing from the open-source models is particularly impressive when you consider the potential “family alignment” advantage in the workflow, where descriptions generated by Gemini might naturally favour Google’s own embedding model. Despite premium models taking the lead, the R\u00b2 performance gap between the best and worst models was a mere 0.035.<\/p>\n\n\n\n

    \n\n\n\n

    Experiment 2: The SMP challenge image dataset<\/h3>\n\n\n\n

    To validate our initial findings, we applied the exact same framework to the image dataset from the Social Media Prediction (SMP) Challenge<\/a>, which was instead used to predict popularity of Flickr posts.<\/p>\n\n\n\n

    The results:<\/strong> Once again, the premium models topped the charts, with both OpenAI and Google producing an identical R\u00b2 score of 0.234<\/strong>. Just like our Instagram experiment, the weakest free alternative trailed by that same narrow margin, coming in at 0.196<\/strong>.<\/p>\n\n\n\n

    \n\n\n\n

    Experiment 3: Retrieval-augmented generation (RAG) performance<\/h3>\n\n\n\n

    To see how these findings hold up beyond downstream regression tasks, we introduced a third experiment: a traditional Retrieval-Augmented Generation (RAG) evaluation.<\/p>\n\n\n\n

    At its core, a RAG framework acts as an open-book exam for an LLM. Instead of relying solely on its pre-trained internal knowledge, the system first searches an external database to retrieve the most relevant documents matching a user’s prompt. It then passes these documents alongside the question to the LLM, ensuring the final generated response is accurate, contextually grounded, and factual.<\/p>\n\n\n\n

    Using the industry-standard BEIR (SciFact)<\/a> dataset, we indexed 5,180+ scientific documents and evaluated how effectively each embedding model could retrieve the exact context needed to answer 200 distinct queries.<\/p>\n\n\n\n

    We measured two key retrieval metrics:<\/p>\n\n\n\n