{"id":1584,"date":"2026-06-10T16:07:38","date_gmt":"2026-06-10T16:07:38","guid":{"rendered":"https:\/\/cms.research.wpp.com\/?p=1584"},"modified":"2026-06-11T09:43:55","modified_gmt":"2026-06-11T09:43:55","slug":"learning-through-experience-teaching-the-viral-hermes-agent-to-automate-our-work","status":"publish","type":"post","link":"https:\/\/cms.research.wpp.com\/?p=1584","title":{"rendered":"Learning through experience: teaching the viral Hermes agent to automate our work"},"content":{"rendered":"\n
\"\"
Hermes<\/figcaption><\/figure>\n\n\n\n

What is Hermes ?<\/strong><\/h2>\n\n\n\n

Hermes Agent<\/strong><\/a> is an open-source, self-hosted AI agent released by Nous Research<\/a>, the lab behind the Hermes model family under an MIT license. Its main pitch is a built-in learning loop<\/em>. Instead of resetting to zero every session, Hermes runs a post-execution review after each successful task, distills the steps that worked into a reusable, Markdown-defined “skill” and refines those skills the next time it hits a similar problem. It also keeps persistent memory across sessions, so it gradually builds a model of your projects and how you like things done, effectively learning through experience.<\/strong><\/p>\n\n\n\n

Unlike a copilot tethered to an IDE, Hermes is meant to live on a server and run unattended – a $5 VPS, a GPU box or serverless infra that costs almost nothing when idle. It talks to hundreds of LLMs through the OpenAI-compatible interface, can communicate via Telegram, Discord, Slack, WhatsApp, Signal, email and a CLI, supports natural-language cron for scheduled jobs, and can spin up subagents to parallelize work. It also ships with 40+ built-in skills out of the box.<\/p>\n\n\n\n

What made it especially attractive to us is its ability to write and improve its own playbook. This ability made us think: could it learn how to do our job and fully automate the work we do at the Quick Reactions pod?<\/em><\/p>\n\n\n\n

Our pod\u2019s mission is to pick up state-of-the-art AI tools, experiment with them, and write an honest, evidence-backed assessment.<\/p>\n\n\n\n

What makes our work tricky is the fact that every new tool we evaluate is different. We need to study it, figure out how to set it up, run it, and evaluate the results.<\/p>\n\n\n\n

Our 6-step evaluation protocol<\/strong><\/h2>\n\n\n\n

We begin by encoding our workflow into a strict, 6-step protocol that the agent can follow for every evaluation:<\/p>\n\n\n\n

    \n
  1. Workspace initialization<\/strong> – spin up a clean, isolated project environment so each evaluation starts from a known state.<\/li>\n\n\n\n
  2. Baseline replication<\/strong> – run the tool’s own “getting started” examples first, to confirm the headline claims reproduce before we push further.<\/li>\n\n\n\n
  3. Rigorous verification<\/strong> – design and run additional autonomous tests that probe the tool under conditions its authors didn’t pick, rather than extrapolating from the happy path.<\/li>\n\n\n\n
  4. Data synthesis & metrics control<\/strong> – measure the things that matter (recall, latency, accuracy) against ground truth data.<\/li>\n\n\n\n
  5. Adversarial peer review<\/strong> – hand the findings to a separate agent powered by a powerful LLM (Claude Opus 4.8), to receive feedback and iterate.<\/li>\n\n\n\n
  6. Finalization & delivery<\/strong> – format the assessment doc and ship it to the right channel (to our internal Notion<\/a> knowledge base in our case).<\/li>\n<\/ol>\n\n\n\n

    The idea was simple: if these are the steps a human pod member walks through, can an agent walk through them unattended – and would the writeup at the end be any good?<\/p>\n\n\n\n

    The Skill system: how Hermes improves itself<\/strong><\/h2>\n\n\n\n

    The first time we ran Hermes on a real task, we walked it through the 6-step protocol by hand. Instead of just following along, Hermes wrote each step down as its own skill – a short Markdown file it can pull up at the start of any future run. So the protocol stopped being something we had to repeatedly provide as input; it became something the agent already knows.<\/p>\n\n\n\n

    Hermes used this knowledge to build a small set of skills covering everything we do: how to set up a clean workspace, how to test a new tool properly, and how to draft the write-up and run it past the reviewers. On top of those sits one master skill that holds the whole run together – it treats the 6 steps as a checklist and won’t let Hermes jump ahead before the previous step is actually done.<\/p>\n\n\n\n

    Impressively, when the LLM peer reviewer flagged something during an experiment (e.g. an unfair baseline, a missing caveat, a poorly designed experiment), Hermes would learn from the feedback and address the issue. It would also record its new learnings in the related skill files, so the next experiment started from a slightly stronger playbook.<\/p>\n\n\n\n

    That’s the self-improvement loop the Hermes pitch promises, and we actually watched it happen. The more experiments we run, the better the skills get, and the less we need to babysit Hermes for the next one. The underlying LLM that powers Hermes (Gemini 3.1 Pro) isn’t getting smarter – its playbook is, and Hermes is the one rewriting it.<\/p>\n\n\n\n

    Putting Hermes to the test<\/strong><\/h2>\n\n\n\n

    We gave Hermes a single, real assignment: take a brand-new open-source tool called Turbovec<\/a> – which claims to store huge amounts of data in a tiny amount of memory and search it faster than the popular alternative – and find out whether those claims actually hold up.<\/p>\n\n\n\n

    We handed the agent the tool, its documentation, a bare cloud machine to work, and nothing else: no starter code, no template, no outline. Hermes had to decide what and how to test, run the experiments and, write the whole thing up on its own.<\/p>\n\n\n\n

    We reviewed Hermes\u2019 output in the exact same way we would review a human colleague’s work, via three simple questions:<\/p>\n\n\n\n