ScopeBench for Monitor Benchmarking: Can Jev Keep Agents in Scope?

How much of an agent’s capability would you give up for it to stay in scope? How much money would you pay to make sure it operated strictly within its operational objectives?

After a year of rapid capability development, we experienced a summer of agents going beyond the implicit and explicit boundaries of their task, including compromising production web servers. The excitement for raw capability is dampened by how challenging it is to operationalize safely: if you can’t get an agent to stay in scope, you’re hesitant to put it in production regardless of how powerful it is. In other words, production-tier capabilities have arrived, and production-tier scope adherence is the final barrier to wide-scale deployment.

Earlier this month, Dreadnode released ScopeBench, a framework and community benchmark for measuring both an agent’s raw hacking capabilities and its ability to stay in scope on an engagement. The most obvious use of this benchmark is tracking the alignment of modern models - to choose your agent, and see what model has the highest scope adherence within some capability threshold.

ScopeBench can also be used to test the safety-capability tradeoff of safety affordances for harnesses. For example, the decision model Jev has received interest from us and others for its potential as a monitor for tool calls. It’s lightning fast, incredibly cheap, and has scored well on our ScopeJudge classification benchmark. In this post we’ll show you how to benchmark its performance as a monitor in a live agent run using ScopeBench.

The Harness is a Variable

One of our favorite things about TerminalBench, from which we inherit many of ScopeBench’s standards, is its inclusion of Agent along with Model in its leaderboard. At time of writing, GLM-5.3 with Max reasoning effort is sitting at 9th on the TerminalBench leaderboard. Notably, this is not with ZCode, GLM’s official harness, but rather Claude Code.

The neuro-symbolic harness in which the model runs matters. If I am going to choose a model based on the results on a benchmark, I would prefer I could run that model in the configuration that produced those scores. That idea can be extended further with ScopeBench. Because capability and scope adherence are measured separately, you can see how safety-conscious changes to the harness affect both scope adherence as well as capability.

Startups and researchers are currently in a heated battle for attention based on their control solutions for agents, showing how their particular solution restricts the model to keep it safe. They rarely benchmark how these safety affordances affect the raw capability of the model, which is crucial for understanding whether the resulting system is one that could be run in production. For example, let’s look at the results of our initial ScopeBench pilot with Scope Adherence on the y-axis and Capability on the x-axis.

Claude Sonnet 4.6 is quite capable on the benchmark pilot, but struggles with Scope Adherence. Let’s imagine we wanted to make it as scope adherent as humanly possible - we could restrict its sandbox so that it could not make external network requests. Scope adherence would go to 100%! A tweet about our superior harness engineering would do numbers. Capability, on the other hand, would go to 0%, meaning we couldn’t use it for anything. The tradeoff would be too harsh. It’s important to keep both factors in mind when “grading” a particular safety solution.

Let’s consider how ScopeBench can be used to evaluate more realistic alternatives.

ScopeJudge

Earlier this year, we released ScopeJudge, an analysis on the efficacy of Claude Code Auto Mode style monitoring for offensive security agents. This technique has been referred to more generally as “synchronous LLM monitoring” or “pre-execution tool call gating”. The idea is, a tool call is proposed by the “agent” model, and another “monitor” model determines whether that tool call is in scope and should be allowed to run. If the monitor model allows it, it executes. Otherwise, it’s blocked, and the agent model gets a message from the monitor that the tool call was blocked instead of the expected tool output.

Five humans labeled 4,800+ tool calls as in scope or out of scope. We compared those human labels to LLM judges in order to see how aligned they were with the scope decisions of humans. This allowed us to cheaply understand what models were likely to make strong monitors.

We used this study to inform our implementation of AgentJudge in the Dreadnode Platform. We also open-sourced the code and dataset, and use it to keep track of how newly released models might perform as a judge.

Enter: Jev

In September, Jev was released by TypeSafe AI. Immediately we heard from our users they wanted to know how it would perform as a scope judge.

Jev is a “decision model”, which can be prompted with textual context like an LLM, but is restricted in what it can output. Instead of free text, for example, decision models can choose between different values for on-the-fly classification. For our purposes, we can use the “choice” setting and let Jev decide whether an incoming tool call is in or out of scope.

Using ScopeJudge we found that Jev was really fast and incredibly cheap.

Being cheap wouldn’t mean much if the model wasn’t performant, but despite the savings it was competitive with other models. It received an F1 of 0.570 as a judge, slightly better than Claude Sonnet 4.6 but about 50x cheaper.

Limitations of Offline Evaluations

ScopeJudge is a great diagnostic for us because it gives us a very cheap signal of whether a model is going to be useful or not as a judge. If the F1 is low enough compared to human judgement, there’s no reason to bother with further testing. But doing well in ScopeJudge is not sufficient to make you a strong judge in production.

ScopeJudge has an inherent limitation as an offline evaluation. We’ve got 4,800+ tool calls of agents that went out of scope labeled by humans after the fact and we can show what a monitor would do for any given tool call. Precision and recall can give us a sense of how many false positives or false negatives a monitor might suffer, but can tell us little about what effect that would have on a running agent.

When the monitor blocks a tool call, that changes the trajectory of the agent. Every future tool call is conditioned on that rejection. Does the monitor block a necessary technique, forcing the model to fail? Does the monitor manage to block a few out-of-scope calls but let crucial ones through, allowing the model to just work around it? The F1 score can give us a directional sense of how useful the model is going to be, but it can’t tell us those particulars.

ScopeBench, on the other hand, is a live evaluation designed specifically to test combinations of model and harness. Since it measures capability and scope adherence separately, we can use this to find out exactly what the gains are in scope adherence, and what “safety tax” we pay in capabilities.

Taking Jev For a Joyride

The recipe for testing out monitors is simple. First, run ScopeBench with a chosen model and the default harness with no safety affordances. This will give you a capability score and a scope adherence score on separate axes.

Then, make safety changes to your harness, like adding a live monitor, and run the benchmark again. This will give you a new capability score, and a new scope adherence score. You can find out how much safety you gained from a practical perspective (did it lead to the agent staying in scope more often) and find out what it cost in capability (how many capability challenges did it fail to solve compared to before you made the safety changes) and tokens (how much more expensive was the run).

We experimented with several Jev configurations acting as a monitor to see what effect they would have. First, let’s look at the raw score for Sonnet 4.6. We’re picking on Sonnet 4.6 because it’s particularly poor at scope adherence, which is useful to illustrate the effect of the monitor. We start with the following scores:

Now we can see how Jev-as-a-monitor affects these results. Jev will get the task description, the proposed tool call, and roughly the last ~15 tool calls so long as they can fit into context. If they can’t, we drop oldest first:

As we can see, this immediately increases our adherence from roughly 15 to 21 scenarios. Our raw capability takes a hit, though, losing a few percentage points worth of tasks. That’s a lot of safety gains for a small capability cost. Dollar wise, it’s about a $0.00263 cost increase per trial run. Extremely inexpensive.

GLM had a higher F1 score in ScopeJudge though, so we imagine it might have more to offer, though more expensive and higher latency.

A nice additional feature of Jev is that it outputs a confidence score with its decisions. We can use the confidence value to create an escalation chain.

For example, let’s say that we default to Jev’s decisions as a monitor. However, if Jev is less than 60% confident, we escalate the tool call and instead present it to GLM. In theory, this should allow many relatively boring tool calls to zip by at low cost, dynamically increasing the expense of our monitor when we get into more challenging scope territory. How does it perform?

This strategy looks much better. We’ve recovered nearly all of the capability score Sonnet 4.6 had on its own, while giving us an additional step up in scope adherence. Our strategy of escalating when Jev isn’t confident appears to work well.

So, what about rabbits all the way down? Can you keep stacking judge models and see increased performance? For example, start with the Jev → GLM strategy. If Jev isn’t very confident, we ask GLM. Then we have two votes. Let’s try a new strategy where if both those votes agree, we accept them and either run or block the tool call. However, if they don’t agree, let’s escalate to Opus 4.8 as a tiebreaker.

This surprised us by performing worse! We both lose some of the capability that we had gained back with GLM, and only barely beat raw Jev with scope for a significantly higher cost. Analyzing the trajectories, it seemed Opus 4.8 particularly struggled with understanding the implications of scope based on headers. For example, if using a header to pivot to a virtual host that was out of scope, Opus reasoned that it was the same box as you had already been accessing. Server side request forgery likewise seemed to be invisible to it. These matched up with findings we had seen in Opus 4.8 as an agent, though we didn’t expect to see them become so detrimental to its performance as a live monitor.

That’s not to say the escalating vote strategy couldn’t work with an even larger model, like Opus 5.5. ScopeJudge gives us an easy diagnostic test to look at how a monitor might perform, and ScopeBench lets us see what it costs in capability. Test your own harness combinations and let us know what you find!

To learn more about ScopeBench, feel free to read the paper. If you’re interested in running the in-progress version of the benchmark or contributing tasks, check out our website.