evaluation

These results show that a model’s capabilities are compatible, generalizable and can carry over to other harnesses. Providers do report to optimize some models for their own coding environments: OpenAI, for example, describes GPT-5-Codex as optimized for agentic software engineering in Codex.12 Yet we observe that a shared provider does not guarantee the best pairing. The practical question remains which harness delivers the best balance of cost and task success for a given model and workload.

Modern retrieval agents expose many configuration choices—LLM, retriever, number of documents, number of hops, and synthesis strategy—each shaping both
answer quality and serving cost. Today, these pipelines are typically hand-tuned
once per workload, leaving substantial per-query optimization untapped. We formulate Query2Conf : given a natural-language query and either an accuracy or
a budget target, select from a predefined pipeline catalog the configuration that
minimizes cost (or maximizes accuracy) at inference time. We propose BRANE
which uses an LLM to convert each query into workload-specific characteristics,
then trains a lightweight per-configuration predictor that estimates whether the
pipeline will answer the query correctly. At inference time, BRANE selects the
configuration that maximizes predicted correctness penalized by cost, exposing a
tunable cost–quality tradeoff without retraining. Across MuSiQue, BrowseCompPlus, and FinanceBench, BRANE consistently pushes the cost–quality Pareto
frontier, matches the best fixed configuration’s accuracy at up to 89% lower cost,
and outperforms LLM-routing, rule-based, and fine-tuned Qwen3-4B baselines.
These results show that per-query configuration of the full retrieval pipeline is a
practical alternative to static workload-level tuning.

We leverage probabilistic programming principles to tackle complex tasks, and utilize differentiable and classical programming paradigms with their respective strengths. The framework introduces a set of polymorphic,
compositional, and self-referential operations for multi-modal data that connects multi-step generative processes and aligns their outputs with user objectives in complex workflows. As a result, we can
transition between the capabilities of various foundation models with in-context learning capabilities
and specialized, fine-tuned models or solvers proficient in addressing specific problems. Through
these operations based on in-context learning our framework enables the creation and evaluation of
explainable computational graphs. Finally, we introduce a quality measure and its empirical score
for evaluating these computational graphs, and propose a benchmark that compares various stateof-the-art LLMs across a set of complex workflows.

I trained a small transformer from scratch in 1.5hrs on a 5090
Beats many LLMs, and scores the same as TRM/HRM

The core insight is this: AI agents don’t save you time by finishing your work. They expose how much work was always possible but never attempted. Five constraints now govern how much of that backlog any organization can actually capture: judgment, planning, coordination, evaluation, and absorption. Understanding those five constraints tells you exactly which new roles are being created and why.

If you cannot articulate what success looks like in a way that can be evaluated, no amount of autonomous execution will produce it. Agents cannot clarify requirements they are never given. They will fill the gaps with assumptions, and those assumptions compound.