👋 Hey, Jai here! I’m a software engineer and tech investor turned solo founder, now bootstrapping a software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me:

Subscribe now

Or if you don’t like my work, send me hate mail.

In my last post I described how we overhauled our enterprise “AI brain” product. During the development process, there was one critical constraint: we only shipped code that had a measurable positive impact on our AI’s performance.

Today I’ll dive into the eval system we built to enforce this.

Evals (short for evaluations) measure whether an AI system performs well by grading its responses to a fixed set of questions or tasks. They help steer product development by telling you whether new features improve or regress results, and by pointing to the gaps that you should prioritise.

Evals have a reputation for being a complex and unwieldy discipline reserved for companies with big budgets; something that OpenAI and Anthropic do with dedicated teams, annotation pipelines, and fancy in-house platforms.

But in reality the inverse is true. Evals are incredibly useful for small teams. They’re cheap and easy to build while your product is still small.

My company QX Labs is fully bootstrapped: I founded it solo and built our knowledge management system with one other engineer. Our eval harness took a few days to build and tune, and it has since dictated critical product decisions.

Here’s my underlying thesis. In business you need to solve for two things: velocity and trajectory. The most valuable thing you can build in the AI age isn’t the system itself, but the feedback loop that tells you whether your system works and how to improve it. Build the loop and you can sustain velocity without hitting a wall, because you’ll always know your next step. Evals are one of those loops.

By the end of this post you’ll have:

• A working mental model of what evals are and how the biggest AI companies use them

• A copyable approach to evals for small teams

• Real before/after numbers from our RAG evals

What an eval actually is

An eval is made up of three things:

1. A dataset that represents your real workload (documents, tickets, conversations, or whatever your system operates on)

2. A set of graded questions or tasks with pre-defined correct answers (“golden” questions)

3. A scorer that runs every question through your system and outputs numbers

You run it before and after a change, and the delta tells you the result of that change.

The closest analogy is an engineering test suite, but with one key difference: unit tests are binary (they pass or fail), while evals are statistical and directional (AI can get better for one set of questions and worse for another).

That’s the trap with LLM systems: they don’t fail by erroring, but instead by confidently delivering mediocre results. A retrieval system that misses half the relevant documents will write an authoritative response with gaping holes. Without evals, you’d never even know.

What the experts recommend

Before building our eval harness, I looked at how the top labs run theirs. Surprisingly, most of their advice is highly suited to small teams. Greg Brockman (President of OpenAI) is a strong advocate:

Here’s what they say:

Gate your evals like a continuous integration (CI) pipeline. OpenAI open-sourced their evals framework back in 2023, calling evals “one of the most impactful things you can do” when building with LLMs. They name the practice eval-driven development. “Evaluate early and often. Write scoped tests at every stage.”

Start small and cover real failures. Anthropic’s engineering guidance says: “We see teams delay building evals because they think they need hundreds of tasks. In reality, 20–50 simple tasks drawn from real failures is a great start.” That’s totally feasible for a small team or a solopreneur.

Eyeball your system’s outputs and errors before coming up with metrics. Hamel Husain’s “Your AI Product Needs Evals” is worth a read. He insists that the first step is to read your system’s actual outputs one-by-one and categorise the failures. Only then should you come up with the appropriate metrics to measure against those failures. Similar to Anthropic he recommends starting with ~30 hand-graded examples. In short: use your own product like your customers would, manually log and categorise the failures, then feed that data into your eval system.

Avoid using LLM “judges” for grading. The big shops need AI judges to grade open-ended outputs at scale, but they calibrate them against human graders because research shows that LLM-graded evals can otherwise be very flimsy. The MT-Bench paper (Zheng et al.) outlines their biases: position bias (judges favour the first response rather than the best one), verbosity bias (judges favour longer responses >90% of the time regardless of quality), and self-enhancement (models score their own outputs higher). The lesson for a small team is not “don’t use judges,” it’s “don’t start with judges.” Stick to mechanical, deterministic scoring first.

In the end big-tech evaluation is expensive due to scale and not the underlying to process. A small team doesn’t need thousands of evals in CI, expensive annotation or judge calibration pipelines. They just need to run a process of getting fixed data, creating graded questions and scoring every change. That costs almost nothing.

The small-team version: our recipe for evals

First, some context: one of our products at QX ingests a company’s documents (Drive, SharePoint, Granola, Notion, etc.) and lets AI agents answer questions across them. When we originally built the MVP it used a simple RAG pipeline for indexing and retrieval, but we wanted to turn it into a full knowledge graph. However, we needed a way to measure whether the changes we implemented were actually improving the agent’s ability to query data.

So before doing anything, we built the eval harness. Here’s how we approached it.

1. Build the eval around your customer archetype

Our dataset uses ~1,000 real documents that represent one hypothetical customer’s universe. In our case we mimicked an investment firm’s internal data: PDFs, spreadsheets, decks, memos, meeting notes, scanned pages. Specifically, we wanted things to look multimodal, messy, and organised into a folder hierarchy an actual firm might accumulate over time. Having raised my owned fund and worked as an investor for 10 years, I had a very good mental model of what this should look like.

That right there is the secret sauce to good evals: model them closely on your end-customers’ files and queries. Synthetic data is appealing because it’s easier to create, but it’s too clean. Real documents have tables that span pages, scans, duplicates, vague project codenames, folders whose names carry information the documents don’t, and so on. Modelling all of those nuances in a synthetic dataset is practically impossible.

And the gap between real and synthetic data is meaningful: Jason Liu found that RAG systems had 97% recall on synthetic questions, then dropped to 55-65% on real user queries. If you truly have no real data, dig out some real public documents and then create synthetic documents woven around the same entities so that cross-document structure exists.

One decision that made it much cheaper for us to run evals is we froze the document extraction step. Extracting text from 1,000 multimodal files (with vision models, table parsing, etc.) is slow and expensive but broadly deterministic. So we did it up-front and committed the extracted text to its own git repo. That way, we could run every re-ingest from already-extracted text and only needed to pay for embedding + indexing the text. The eval loop became ~100x cheaper and similarly faster. It also enabled us to isolate document extraction from indexing + retrieval (two distinct processes that should be eval’d separately).

2. Segment the questions by your product’s logic

Instead of writing 100 random questions, we wrote ~20 questions in each of five classes that (1) matched customer behaviour (2) captured the spectrum of capabilities a “company brain” should have. The question categories broke down as follows:

• Needle questions (that pull out one fact). Example: “What discount rate are we assuming in our DCF analysis for Acme?”

• Entity questions (that require the complete document set for one thing). Example: “What do we know about Acme Inc?”

• Multi-part questions (that require documents for different entities to co-appear). Example: “Compare Corp A and Corp B’s valuation metrics.”

• Aggregation questions (that need exact lists or counts). Example: “Do we have any expert calls discussing AI regulation in Europe?”

• Thematic questions (that broadly coverage a topic). Example: “What are the recurring risks across our food-delivery investments?”

NB: each of our question categories had their own unique set of metrics that we evaluated (such as recall@20, mean reciprocal rank and F1 score). Those metrics are very specific to retrieval and recommendation systems, which you can read about here. Naturally your metrics should be well thought out and reflect what it is that you’re actually evaluating.

Why segment like this? Because a single overall score hides the nuance. Our baseline system scored a decent 0.83 overall, but scored only 0.59 on thematic questions and 0.69 on aggregation. The average score would have you believe that things were pretty good. But the category breakdown revealed significant failures for important use-cases.

Different question classes also improve for different reasons, which helps you connect metric movements back to specific engineering decisions. If your product has distinct usage patterns then each mode deserves its own question class.

3. Make scoring mechanical (no judges)

All of our questions were scored using deterministic measures, for example:

• Document-recall against expected document lists (“these specific documents should appear”)

• Substring checks against expected evidence (“the retrieved chunk must contain 43%”)

• Set precision/recall for list questions (“this many results should appear”)

• Coverage of expected string groups for thematic questions (“these words or sentences should appear”)

We didn’t use an LLM judge at all. This kept it free and fast, whereas a judge would have added a second AI system to tune and eval (Shankar et al. address this in “Who Validates the Validators”). Add a judge only when a metric you need is actually unmeasurable without one. And when you do, calibrate it against your own judgment.

One other pointer: make sure your eval system gracefully handles version changes (new questions, new files etc). We version both the questions and the final eval reports in git. Our harness auto-generates reports with deltas against the previous run and can gracefully handle drift in the underlying questions and documents.

4. Curate the questions yourself

We semi-automated the question generation: an LLM read the ingested corpus and proposed ~250 candidate questions along with ground truth. I then spent a couple hours removing bad questions, sharpening good ones, and drafting my own where I felt there were gaps. I ended up with 100 questions.

That couple of hours is crucial: the golden set is where you encode what “good” means for your customers. That’s a judgment only someone who properly understands the customer can make. Big labs pay annotation teams to do this, but frankly you have a better understanding of your customer than any outsourced annotator ever will, so make the most of it. It’s one of the few genuine advantages a small team has over a big one.

How it played out for us: real numbers

We ran the eval at three checkpoints when improving our knowledge management system:

• Phase 0 (baseline; our old RAG system implemented in 2024)

• Phase 1 (improved document ingestion by adding contextual chunk headers, upgraded metadata extraction + filtering)

• Phase 2 (built the entity layer that turned our system into a knowledge graph)

A few things stood out in the results:

The biggest win was also the easiest to implement. Thematic recall jumped from 0.59 to 0.83 in Phase 1, the largest single movement we saw. Most of it came from some really easy changes, most notably prepending a two-sentence AI-generated context header to every chunk before embedding it (saying something like “This text is from document X and talks about Y”). Anthropic discuss that approach here. The fancy entity layer we were so excited about in Phase 2 had a much smaller impact. Without the eval we would have credited the wrong work.

The eval caught a regression that we would have otherwise missed. Multi-part recall actually dropped in Phase 2 from 0.88 to 0.82. It turned out that adding new retrieval tools to our agent impacted how results merged for comparison questions. This kind of degradation would have been invisible without evals, creating a hidden pile of technical debt.

The eval also told us what not to build. Our original research mapped out a Phase 3: community detection and cluster summaries over the knowledge graph. But by Phase 2 our thematic scores had already hit 0.91 via much cheaper means so we parked it. Sometimes less work is good work; that’s the ethos of The Minimum Viable Founder.

It’s just a loop

If you look at what we’re doing with evals, it’s just a feedback loop:

1. Decide what “good” looks like (i.e. set the golden questions and choose what metrics to measure)

2. Make measurement cheap so that the loop can run frequently

3. Change one thing then measure the result

4. Decide what to ship (or not) based on the result

These loops show up everywhere in business: pricing experiments, ad management, onboarding funnels, sales scripts. It’s nothing new.

The startup checklist for building an eval harness

• Use a real corpus that reflects your customers. Only use synthetic files to fill gaps.

• Segment questions based on usage patterns.

• Write the golden set yourself (AI-assisted, not AI-generated).

• Freeze any steps in your eval that are expensive and invariant. In our case, document extraction.

• Do mechanical scoring. Add an LLM judge if it’s the only way to measure your chosen metrics.

• Make running the eval super easy (one command). Otherwise you’ll never run it.

• Commit eval reports next to code. E.g. an /evals folder in your project root so that it’s easy to track history.

• Introduce new metrics as the product evolves. Don’t let the eval become dated and obsolete.

If you’re building your own eval system or want to better understand how we approach things, I’d love to exchange notes.

👋 Hey, Jai here! I’m a software engineer and tech investor turned solo founder building a bootstrapped software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me:

Subscribe now

Or if you don’t like my work, send me hate mail.

Almost every business I’ve spoken to at QX has asked for some version of an AI “company brain” connected to their internal documents and systems. They echo the same complaint that their data is scattered across silos, and believe that the key to their competitive edge lies in unlocking this pent-up institutional knowledge.

In a world where AI is commoditising access to public knowledge, internal knowledge accumulated through domain experience and relationships become the primary moat. Two years ago, the AI brain discussion warranted an exploratory pilot, but today it’s table stakes.

The common way to build this is called RAG, or retrieval-augmented generation. This process involves breaking your company’s documents into passages, converting them into numerical representations that encode their meaning, and letting an AI answer questions by retrieving the passages that best match a query. It’s simple, cheap and demos well.

RAG was originally proposed by Meta AI researchers in a May 2020 paper. In AI-years that’s almost pre-historic. And whilst it performed well on a limited set of questions, today it falls short for use-cases where real value is added. Microsoft then entered the frame in 2024 and published a technique called GraphRAG. Their idea was to create a knowledge graph by using LLMs to extract the entities and relationships from documents, not just the text content. This would enable us to traverse a structured graph during retrieval (e.g. “find me documents related to this person, company, event etc”) instead of a simple text search. It gained a lot of hype.

I’ll get into the detail of GraphRAG shortly, but my view is that in practice it’s complete overkill. At QX, with the benefit of experience and testing, we have chosen a more pragmatic approach: take the best parts of GraphRAG and knowledge graphs, fit them into a much simpler architecture, and scrap the rest. Our approach has been shaped by what actually kills these AI brain projects in practice:

• Information is unstructured and spread across disconnected systems.

• Retrieval quality has to sustain with scale, i.e. beyond a pilot.

• Initial implementation needs to be fast and cheap, so that companies can see results and drive uptake before committing outright. Behavioural change is always the primary goal, everything else follows.

Crucially, we wanted a system that our mid-market customers could switch on themselves, without consultants or setup workshops.

Three kinds of questions

While analysing the queries customers sent to their internal knowledge base, we identified three distinct categories.

1. Needle questions. “What was the take rate in the 2017 valuation memo?” These typically ask for one fact in one document. Classic RAG is excellent here because searching by meaning finds that one passage reliably.

2. Portfolio questions. “Tell me everything we know about Acme Inc.” This is a broad, often multi-step question that needs every relevant document, not the top ten passages. RAG gets you a decent sample but silently drops the rest which is dangerous for diligence or research work (creates blind spots).

3. Counting questions. “Which fintech companies have we evaluated?” or “Which contracts mention this supplier?” These need an exact list or tally across the entire corpus. Top-k search results cannot answer them. The system will confidently name the companies it happened to retrieve and skip the rest, and you’ll never know.

Most “AI brain” demos show needle questions, but real business value is often trapped behind the other two.

Enter GraphRAG

GraphRAG took direct aim at solving for the limitations of classic RAG. Instead of just chopping documents into searchable passages, Microsoft proposed that an AI should parse the whole corpus and build a knowledge graph out of it. Every company, person, and product becomes a node. Relationships between them become edges. Clusters of related entities get pre-written AI summaries, so the system can answer big-picture questions from the summaries instead of hunting through passages.

It’s a clever idea and it works well for those latter question categories. The research is consistent: graph-based approaches shine on questions that require connecting information across many documents, what the original paper called “global sensemaking”.

GraphRAG became shorthand for “the serious way to do enterprise RAG”. But it came at a real cost.

What the research says

There were three major findings when evaluating GraphRAG, and most of them came directly from Microsoft.

The index is very expensive. Building the graph means running AI extraction and summarisation over everything before anyone asks a single question. Microsoft’s own follow-up work, LazyGraphRAG, states the cost: a plain vector index costs about 0.1% of a full GraphRAG index. Three orders of magnitude, paid up front, on content nobody may ever query. The same work showed a lazy approach matching full GraphRAG’s answer quality on global questions at a small fraction of the query cost. In effect, Microsoft showed that the expensive part of GraphRAG is mostly unnecessary. Their productised solution accelerator for GraphRAG was even archived in 2025.

The wins are narrow. A systematic evaluation published in 2025 (RAG vs. GraphRAG, arXiv:2502.11371) found what practitioners kept rediscovering: vanilla RAG wins on direct factual questions, graphs win on multi-hop and summary questions, and the best results come from routing between approaches rather than betting on either.

Entity resolution is the hardest part. For a graph to be useful, “Acme”, “ACME Holdings Inc.” and “Acme Corp” must become one node. Get that wrong in one direction and your graph fragments into disconnected shards. Get it wrong in the other direction and two different companies inconspicuously merge into a single fictional one, and every answer built on top inherits the error. This problem is old and largely unsolved in the popular GraphRAG tooling. Meanwhile a graph database brings its own operational tax: another system to run, back up, secure, and keep consistent with your documents, multiplied by every tenant if you’re a SaaS product.

The philosophy behind GraphRAG is right: documents alone aren’t enough, and tracking the entities that your corpus talks about unlocks question types that ordinary search can’t address. The architecture most people associate with it, an eagerly built knowledge graph in a graph database, is overkill for the majority of real workloads.

What we built instead: graph-like RAG

We kept the parts with evidence behind them and dropped the rest. The result looks like a graph but runs on a regular database: it has entities, relationships, and rich metadata that evolve over time—but no Neo4j, no up-front graph build, and no ontology to custom-design.

The breakdown is as follows:

Everything stays in ordinary infrastructure. Passages live in a search index. Entities live as plain database records: one record per company, person, product, project, event, or location, per client workspace. An entity record knows its name, its alternate spellings, which documents mention it, and which other entities it tends to appear alongside. Those “appears alongside” links are what effectively form the graph. They’re just a list stored on each record, computed by counting co-occurrences. There’s no graph query language and no new moving parts.

The ontology is fixed, small, and universal. Six types (company, person, event, etc.) plus a handful of label fields (industry, category, topic) that adapt to whatever the documents contain. The fixed ontology is what makes the system self-serve. Custom ontologies are otherwise why traditional knowledge-graph projects begin with months of workshops. A pharma client and a private equity client get the same types, and the content of the graph adapts to their world automatically, because it’s extracted from their documents and evolves naturally as the corpus grows.

Entity resolution follows a clear waterfall. When a document mentions a name, we check a “phone book” containing every spelling we’ve seen before for that entity. If there’s an exact hit, the entity is matched for free. If we miss, a similarity search proposes look-alike candidates for fractions of a cent. Only when there’s a genuinely ambiguous candidate does a small AI model get asked “are these the same real-world thing?”, and it’s instructed to say no when unsure. A wrong merge poisons everything downstream; a missed merge just leaves two entries that a recurring cleanup job can unite later. Every merge is reversible by design.

Entity summaries are built “lazily”. Nothing gets an AI-written profile until someone actually asks about it. This is the lesson from Microsoft’s LazyGraphRAG work. We want costs that scale with usage instead of costs that scale with the size of the corpus.

The AI decides what retrieval strategy (graph vs text) to use for a given query. The agent answering your question has plain search for needle questions, a “resolve” tool that pulls everything about one entity, “expand” that walks to related entities (similar to “hopping” a graph database), and exact counting over the whole corpus for list questions. You can watch it chain these together. See the example below: we ask a broad portfolio-style question. It first uses facet to count entities (in this case companies) that get mentioned frequently in the context of food delivery, and then resolves each entity to dig deeper.

Does it work?

I’m obsessed with measurement, so before building any of this we developed an evaluation harness: a corpus of thousands of real documents and a graded set of questions across all three question types, scored automatically. Every architectural decision above had to move a number or it didn’t ship. In the end, the entity layer produced a clear uplift on exactly the question classes where plain RAG fails (portfolio and counting questions) while leaving needle questions at ceiling. The eval system deserves its own write-up so maybe I’ll get to that in a future post.

The trade-offs

It doesn’t do deep multi-hop reasoning. If your use case is “find the hidden path between these two entities four steps apart”, you want a real graph that can handle complex traversal.

Our links are co-occurrence, not typed relationships: the system knows that the company Acme and the person Jane Smith appear together constantly, but not that she’s the CEO. Our thesis is that, once the AI is supplied with the surrounding context for these two entities, it can gracefully deduce that Jane is the CEO without having to encode this in the graph itself.

Our conservative merging also means an occasional entity exists twice for a day until the cleanup job unifies it.

If your corpus is small and static, and your questions require you to connect tenuous dots across the graph, full GraphRAG is a good choice and the costs are probably worth it. Our bet is different: for working companies with living document sets, the graph-like version captures most of the value at a fraction of the cost and none of the operational weight.

The self-serve factor

Everything above runs self-serve. Connect Google Drive, SharePoint, Notion, Granola etc, and decide how frequently you want them to sync. The system extracts, resolves, links, and trues itself up on a schedule, and the costs of that upkeep scale with what changed, not with how much you’ve stored. Nobody designs a schema or labels anything. That was the constraint we started with, and I believe it was the best design decision we made.

This is how we approached it at QX Labs. If you’re wrestling with the same problems, or you think we’re wrong somewhere, I’d love to exchange notes.

Sources and further reading:

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401), the first paper to propose RAG

• Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv:2404.16130), the original Microsoft GraphRAG paper

• Microsoft Research, LazyGraphRAG: Setting a new standard for quality and cost (Nov 2024), source of the 0.1% indexing cost and 700x query cost comparisons

• Han et al., RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv:2502.11371)

• Anthropic, Introducing Contextual Retrieval (2024), the highest-evidence cheap retrieval upgrade we adopted, measured at a 49% reduction in retrieval failures

👋 Hey, Jai here! I’m a software engineer and tech investor turned solo founder building a bootstrapped software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me:

Subscribe now

Or if you don’t like my work, send me hate mail.

Before we dive in...

If you’re keen to understand how deep research works, you can play with my Python implementation here (~800 stars on GitHub) or install it with `pip install deep-researcher`. It produces detailed reports with one CLI command:

deep-researcher --query "Write about Plato's life and works" 

A New Hope

Back in 2023, AutoGPT dropped — and it felt like magic. It was the first major example we had of an AI agent that could plan and execute complex tasks autonomously. Given a high-level goal, it would break problems down into sub-tasks and use tools in a continuous loop to solve them. It could google stuff, open websites, write files, and even reflect on its own performance.

But... it also kind of sucked.

• It got stuck in loops

• It forgot what it was doing mid-way

• It hallucinated confidently

• It burned through tokens (and your wallet)

The LLM As The “Central Brain”

Despite these limitations, AutoGPT and similar prototypes proved the concept of autonomous agentic AI. They established a basic architecture that many later systems would build on: an LLM as the central “brain”.

Research from this period introduced techniques like ReAct (Reason + Act), which had the model interweave reasoning statements with tool calls (e.g. “Thought: ...; Action: ...”), rather than issuing commands without explanation​. This helped create an internal chain-of-thought and reduced obvious failures. Other experiments introduced forms of self-reflection (having the model critique its own past actions) to prevent it from getting stuck in loops.

The early agent systems laid the groundwork by identifying what was needed: the ability to decompose tasks, remember intermediate results, leverage external knowledge sources, and safely remain goal-directed.

What Was Missing

These early recursive agents needed 4 critical upgrades to work well:

1. Larger context windows: To remember the original task as well as historical actions and findings. GPT-3.5 originally had a context window of only 4k tokens vs 128k for GPT-4o and 200k for o3-mini.

2. Multi-modal capability: To enable agents to interpret and interact with more resources - especially images.

3. A stronger ecosystem of tools: The ability to use browsers, search the web, execute code in a sandboxed environment, access local files and APIs etc. The open source community has seen an explosion of tooling in the last year, for example:

   • Browser Use to control browsers

   • Tavily, Exa, Firecrawl and Crawl4AI for web search and retrieval

   • E2B for executing code in a sandbox

   • The MCP protocol from Anthropic

4. Better tool selection capability: To actually pick the right tool for each task without getting stuck in loops, hallucinating or veering off-topic. GPT-4o, 4o-mini and o3-mini introduce massive improvements in tool-selection. This leaderboard is my favourite reference on this - what’s incredible is how performant 4o-mini is despite being 10-50x cheaper (and substantially faster) than similarly ranked models.

Fast Forward to Now...

Huge improvements across all three fronts have led to the rise of:

• Deep research agents: OpenAI, Google, Perplexity all offer their own versions of deep research.

• Open-source versions: Hugging Face, LangChain and even my own version

• Other agentic apps: Like Manus AI for general tasks, Convergence for productivity, Roo Code for coding. These combine deep research logic with broader, multi-tool capabilities - think coding, file analysis, PDFs, browsing, image understanding, etc.

How Deep Research Works

You can think of two paradigms for building AI apps that sit on opposite ends of a spectrum.

• Workflow tools follow a fixed flow often in a loop
  E.g. Think → Search → Read → Repeat → Summarize.

• Purely agentic tools go wild with autonomy
  E.g. one or many agents with defined roles autonomously deciding what tools to use, what to do next, who to pass tasks to, and when they’ve finished their task. This could mimic human teams - a supervisor, line-manager, teams of analysts working on sub-tasks.

Most real-world implementations sit somewhere in between. Bear in mind that Deep Research is just one use-case (i.e. producing long reports), but similar frameworks with a wider set of tooling can cater to broader use-cases. For example, Manus AI demonstrates what you can achieve with a set of agents wrapped around Claude and Qwen with access to 29 tools. Under the hood, all of these apps are using fairly fixed workflows to minimise token usage and improve reliability.

OpenAI’s Approach (Deep Research)

• Based on a smart model (o3, not yet available via API) fine-tuned for multi-step reasoning and tool use

• Uses an internal framework to plan → search → read → think → write

• Utilizes their in-house browser tool (Operator) to interact with the web

• Has the ability to backtrack to avoid getting stuck in loops

• Outputs full reports with citations and source traceability

• Achieves state-of-the-art results on real-world benchmarks (GAIA, etc.)

• But: it’s proprietary, compute-intensive, and gated to higher-tier users

Open Source Agents

Most open-source approaches:

• Stick with workflow-first logic (to keep things predictable and cheap)

• Can achieve surprisingly good performance even with open models like DeepSeek

• Are often faster and cheaper to run and highly customizable.

But they don’t yet have:

• Vision support (unless you bolt it on)

• Sophisticated browsers like OpenAI’s Operator

• As much safety and hallucination mitigation

Hugging Face’s implementation uses code-based agents: instead of saying “search for X using the search tool” (in a JSON output), the model writes Python code to do it. This might sound odd at first, but it has some powerful advantages:

• Code can express more complex logic than a single / sequential set of tool calls

• Code is more concise - e.g. it can write code to run a bunch of tools in parallel, which lowers token usage

• LLMs are good at writing code and can self-debug if there are errors

A recent paper showed that letting the agent express complex action sequences in code can make the whole process more efficient and reliable​. The Hugging Face open deep research project implemented this idea and saw a big boost in performance: their accuracy on one benchmark jumped from 33% to 55% simply by switching from JSON-style actions to code-style actions (with the same underlying model and tasks).

The QX Labs Implementation

For illustration, let me break down how my implementation works.

TL;DR: It leans workflow-heavy but with autonomy on tool selection and async execution for scale.

1. Carries out initial research on the query to understand the topic

2. Splits the research topic into sub-topics and sub-sections

3. Iteratively runs research on each sub-topic - this is done in async/parallel to maximise speed

4. Consolidates all findings into a single report with references

5. (Optional) Includes a full trace of the workflow and agent calls in OpenAI’s trace system

It has 2 modes:

• Simple Mode
  Runs the iterative researcher (Step 3 above) on the full query in one go.
  Great for fast answers on narrower topics.

• Deep Mode
  Runs the planning phase (Step 1+2 above) then spawns concurrent researchers for each sub-topic.
  Ideal for multi-dimensional or expansive research questions.

💡 Interesting Findings

• gpt-4o-mini works surprisingly well for this use case

  • Matches o3-mini on tool-use benchmarks

  • Faster + cheaper

  • You don’t need the full GPT-4o brain since deep research relies more on retrieved content than parametric knowledge

• LLMs suck at following word count instructions

  • Better to use heuristics they’ve seen before like:

    • “Length of a tweet”

    • “A few paragraphs”

    • “2 pages of text”

• Most LLMs cap at ~2,000 words of output even though they can in theory output thousands more tokens

  • Why? Because they weren’t trained to write 50-page documents

  • Want longer reports?

    • Use a streaming writer pattern that chains LLM outputs (like this approach)

    • Or chunk by section and concatenate intelligently (watch out for repetition)

Being born in the ‘90s - and not named John Smith - made it easy to secure myfullname.com. My 14-year quest to buy j.ai, however, is a different story. The domain has remained in the same hands all these years, leased to various users including jasper.ai most recently. The owner asked for $50k back in 2012. It’s now listed for sale for $5m so I’d say the budget constraint still holds.

I recently had to update and migrate the old website just to keep it running. Expired SSL certificates, deprecated PHP code... total mess. But it was worth it just to salvage that sordid blog from the archives.

It also gave me renewed inspiration to maintain a subtle web presence - less so on social media, more so tucked away behind an obscure domain: j2.gg.

Why J2?

• This is version 2 of my website: J1 -> J2

• It reflects my initials: JJ -> J2

• It’s a numeronym for my name: Jai -> J2

Pick your interpretation.

Why now?

Two years ago I stepped out of the cushy world of investing and decided to bootstrap a software/SaaS company as a solo-founder. It was a curious choice given that the biggest edge I gained as an investor was the ability to fundraise, but a deliberate choice nonetheless. The framework behind bootstraping vs. fundraising is perhaps a discussion for another post.

Caution has been thrown to the wind and I’m learning a tonne along the way. I hope to share my thoughts and experiences here.

Stay tuned. ✌️