OWASP’s Securing Agentic Applications V2, published on 10 November, has one sentence I keep quoting: “A guardrail that cannot observe a tool call before it executes is documentation, not a defense.” The same section, 3.3.7, says what a hook has to do when it can’t reach a decision: “an error, a timeout, or an unreachable enforcement service - the action must fail closed (denied or escalated, never allowed through by default). A hook that fails open can be bypassed by anyone able to make it fail.”
The guide lists 18 hook points. It doesn’t say which agent frameworks honour that rule. So I tested the one that matters most, the hook in front of a tool call, in five of them. I put a policy check in front of a tool that sends an email to [email protected], then made the check crash, and then made it hang. Then I looked at whether the email went out.
TL;DR
- CrewAI 1.15.41 fails open by design, with no setting to change it. If a
before_tool_call hook raises anything other than HookAborted, the exception is swallowed and the tool runs. The source calls this deliberate. A redaction hook that raises in after_tool_call hands the unredacted output to the model. - The Claude Agent SDK 0.2.182 fails open on errors by default. A
PreToolUse callback that raises counts as “no decision”, and in my test the Bash command ran. Setting on_failure="block" on the HookMatcher makes it fail closed. A hook that times out was denied in both modes. - LangChain 1.5 depends on which hook you use. Middleware (
wrap_tool_call) fails closed. A callback handler that raises in on_tool_start fails open unless you set raise_error = True. - The OpenAI Agents SDK 0.24.0 and the MCP Python SDK 2.3.0 fail closed on errors in their tool interceptors.
- Only the Claude Agent SDK puts a timeout on hooks. In the other four, a hook that hangs blocks the run indefinitely. That’s fail closed, and it’s also a denial of service waiting for a slow policy service.
- None of the five pins tool definitions by hash, which section 6.4.3 of the guide asks for on every session start.
What the guide asks for
Section 3.3.7 of the guide defines a hook as “a point in the agent’s code-level execution loop where a builder can wire logic that observes an action and returns a disposition before the action takes effect.” It lists five dispositions: allow, deny, modify, escalate, defer. And it separates two things that frameworks often blur: “the hook is the interception point, while the guardrail/policy is the separate logic that runs there.”
That separation is why failure handling matters. The policy logic is usually the fragile part. It calls a classifier, a policy engine like OPA, a remote service, a regex someone wrote under deadline. The hook is the framework’s code. If the framework treats an exception from your policy as “no opinion”, every way an attacker can make your policy throw becomes a way to run the tool. A malformed argument, an oversized input, or a slow downstream service would each do it.
The guide also says where to start: “If you wire only two hooks, wire toolCallRequest and agentResponse.” These tests cover toolCallRequest, and toolCallResult where a framework puts both in one place.
The test
Every test has the same shape. There’s one tool that records whether it ran. A pre-tool hook sits in front of it in four configurations: no hook (baseline), a hook that denies, a hook that raises RuntimeError("policy service unreachable"), and a hook that sleeps for 30 seconds or more. For each case, did the tool run?
| Framework | Version | Hook tested | Model |
|---|
| OpenAI Agents SDK | openai-agents 0.24.0 | tool_input_guardrail, RunHooks.on_tool_start | ScriptedModel from agents.testing, no network |
| LangChain / LangGraph | langchain 1.5.0, langgraph 1.2.21, langchain-core 1.8.0 | AgentMiddleware.wrap_tool_call, callback on_tool_start | fake chat model, no network |
| CrewAI | crewai 1.15.41 | before_tool_call, after_tool_call | none: drives the executor’s tool path directly |
| MCP Python SDK (server) | mcp 2.3.0 | Extension.intercept_tool_call, ServerMiddleware | none: in-memory client |
| Claude Agent SDK | claude-agent-sdk 0.2.182 | PreToolUse callback | live claude-haiku-4-5, Bash pre-allowed, bypassPermissions |
Four of the five need no model at all. The Claude Agent SDK runs hooks through its bundled CLI, so that test makes real model calls. I ran each suite twice on 14 November and got the same results both times. The harness and raw output are in agent-hook-failure-tests.
The results
“Ran” means the tool executed. Bold marks a result that breaks the guide’s fail-closed rule.
| Framework and hook | Hook denies | Hook raises | Hook hangs |
|---|
OpenAI Agents SDK, tool_input_guardrail | blocked | blocked, run aborts | blocked, no timeout |
OpenAI Agents SDK, RunHooks.on_tool_start | n/a (observe) | blocked, run aborts | not tested |
LangChain, middleware wrap_tool_call | blocked | blocked, run aborts | blocked, no timeout |
LangChain, callback on_tool_start | n/a (observe) | ran | not tested |
LangChain, callback with raise_error = True | n/a | blocked, run aborts | not tested |
CrewAI, before_tool_call | blocked | ran | not tested |
CrewAI, before_tool_call raising HookAborted | blocked | n/a | n/a |
| CrewAI, hook 1 raises, hook 2 denies | n/a | blocked (1.6.1: ran) | n/a |
CrewAI, after_tool_call redaction raises | n/a | unredacted output to model | n/a |
MCP SDK server, intercept_tool_call | n/a | blocked, error to client | blocked, no timeout |
MCP SDK server, ServerMiddleware | n/a | blocked, error to client | not tested |
Claude Agent SDK, PreToolUse default | blocked | ran | blocked at timeout |
Claude Agent SDK, PreToolUse with on_failure="block" | blocked | blocked | blocked at timeout |
CrewAI: fail-open is the stated design
In CrewAI 1.15.41 every hook family (before and after LLM calls, before and after tool calls) runs through one dispatcher, crewai/hooks/dispatch.py. Its module docstring states the policy:
HookAborted propagates by design. Any other exception raised by a hook is swallowed (fail-open) to preserve the framework’s protection against a buggy user hook.
The function that runs each hook, _invoke_hook, does what that says. It catches HookAborted and re-raises it. It catches every other Exception, prints “Error in {point} hook” if verbose is on, and returns as though the hook had made no change. The tests confirmed it:
- A hook that raises allows the call. The test hook raised
RuntimeError, and the email tool ran with the attacker’s address. - Only two things block: returning exactly
False (the older style) or raising HookAborted. A hook that returns None, 0 or "deny" allows the call. - A failed redaction is silent. An
after_tool_call hook meant to strip a secret raised, and the model saw SECRET=abc123. In OWASP terms, a toolCallResult Modify became an Allow.
The reasoning in the docstring is fair for a hook that logs or rewrites a prompt: a bug in your hook shouldn’t take down the crew. It’s the wrong default for a hook whose job is to say no. There is no setting to choose between the two, so the choice has to be made in your hook code.
Version matters here. My first run used 1.6.1, which pip had resolved alongside the other packages. There, all before-hooks shared one try block, so a hook that raised also stopped every hook after it. A raising hook followed by one that always blocks let the tool run. In 1.15.41 each hook runs in its own try, and that case is blocked. The fail-open default is the same in both.
Until there’s an option, convert every failure into HookAborted yourself:
1
2
3
4
5
6
7
8
9
10
| from crewai.hooks import before_tool_call, HookAborted
@before_tool_call
def policy(ctx):
try:
allowed = check(ctx) # your logic
except Exception as e:
raise HookAborted(f"policy check failed: {e}") # fail closed
if not allowed:
raise HookAborted("denied by policy")
|
Do the same in after_tool_call hooks that redact. If redaction fails, abort or replace the whole result instead of returning the original.
I’ve asked CrewAI for an opt-in fail-closed mode in issue #9434.
Claude Agent SDK: fail-closed is one argument away
The Claude Agent SDK says what it does. HookMatcher in claude_agent_sdk/types.py has two relevant fields. One is timeout, which defaults to 60 seconds. The other is on_failure:
With “block”, a callback that raises, returns None (a path with no return), or returns a value the CLI rejects, such as permissionDecision “Deny”, blocks the action, so a callback with no opinion must return {}. […] Unset or “continue” counts a failure as no decision.
“No decision” means the call goes on to the normal permission flow. In my test, Bash was pre-allowed and the session ran with bypassPermissions, so a raising PreToolUse callback let touch marker.txt run. With on_failure="block", the same callback blocked it. Under a stricter permission mode, “no decision” may end at a permission prompt instead. The trouble is that you can’t rely on that in the unattended deployments where hooks matter most.
Timeouts behave differently. A callback that slept past a 5-second timeout was denied with or without on_failure. The model got this tool result back: “PreToolUse hook did not respond before its timeout (host client may be unreachable). The tool call was not executed”. So timeouts fail closed by default. Exceptions fail open by default.
The fix is one argument on every matcher that enforces anything:
1
| HookMatcher(matcher="Bash|Write|Edit|mcp__.*", hooks=[policy], timeout=10, on_failure="block")
|
Note the SDK’s own warning in the same docstring: with “block”, a callback with no opinion “must return {}”. A callback that falls off the end returns None, which then counts as a failure and blocks.
LangChain: two hook systems, two answers
LangChain 1.x has two places to put a check, and they behave in opposite ways.
Middleware is the enforcement layer. A wrap_tool_call that raises aborts the run before the tool executes. One that returns a ToolMessage without calling the handler blocks the call and tells the model why.
Callbacks are the observation layer. BaseCallbackHandler.raise_error defaults to False (langchain_core/callbacks/base.py, line 506). A handler that raises in on_tool_start produces a warning, “Error in CbRaises.on_tool_start callback”, and the tool runs. Setting raise_error = True on the handler makes it abort the run. This is the case the OWASP guide names: where hooks are observe-only, “you cannot enforce a true fail-closed posture through the framework’s callback event bus.” If a guardrail integration you use attaches as a callback handler, check whether it sets raise_error.
OpenAI Agents SDK and the MCP SDK: closed, but check what the hook covers
Both failed closed on exceptions. In the OpenAI Agents SDK, an exception from a tool_input_guardrail or from RunHooks.on_tool_start aborted the run before the tool executed. In the MCP Python SDK, an exception from an Extension.intercept_tool_call interceptor or a ServerMiddleware came back to the client as an error, and the tool handler never ran.
For the OpenAI SDK the question is coverage, which the guide calls “the condition that fails quietly.” Tool guardrail fields exist on FunctionTool and on MCP servers (tool_input_guardrails on the server applies to every tool it exposes). Shell, apply-patch and custom tools have no guardrail field and use a separate needs_approval mechanism instead. RunHooks.on_tool_start does fire for them, and for local computer-use calls (agents/run_internal/tool_actions.py), so that hook is the broader interception point. I only tested it with a function tool. Hosted tools such as web search, file search and code interpreter run on OpenAI’s side. I found no local hook call site for them. That’s the guide’s “paths that run outside the harness entirely”, and the controls it prescribes for them are credential scoping, egress controls and server-side authorization.
The MCP SDK result is server-side. The MCP client SDK has no tool-call interception point, because the host application is the harness. Your enforcement is whatever the host does before it calls call_tool.
The other failure: hooks that never return
Only the Claude Agent SDK enforces a hook timeout. In the OpenAI Agents SDK, LangChain and the MCP SDK, a hook that awaited for 30 seconds held the run for as long as I let it. I cancelled from outside after three to five seconds. The tool never ran, so this is fail closed. But a policy check that calls a remote service with no timeout of its own turns that service’s latency into the agent’s latency. Its outage becomes the agent’s outage.
Put a timeout inside the hook, and make the timeout path deny:
1
2
3
4
5
| async def guarded(data):
try:
return await asyncio.wait_for(policy(data), timeout=2)
except Exception: # includes TimeoutError
return deny()
|
Section 6.4.3 of the guide asks clients to “pin tool descriptions and schemas by cryptographic hash at first registration,” to “re-hash on every session start and on every catalog refresh,” and to never “carry ‘approved’ status across session boundaries.” I found no such mechanism in any of the five packages. I searched each client package for hashing of tool definitions and found only name shortening and identity hashing. That’s a grep result, not a test. It means the “approve once, trust forever” behaviour the guide warns about is the default everywhere, and pinning is something you build at the toolCallRequest hook or in a gateway in front of your MCP servers.
Limits of this test
- It covers one hook point of the guide’s 18, plus
toolCallResult for CrewAI. Session, memory, sub-agent, compaction and skill hooks are not tested here. - It tests Python packages only. The TypeScript SDKs and the Claude Code CLI’s own command hooks (configured in
settings.json) may behave differently. - The CrewAI test calls the executor’s tool path,
execute_tool_and_check_finality, directly instead of running a full crew with a model. Both the crew agent executor and LiteAgent call that function to run tools. I found one other path, an LLM-level function-calling helper, that runs functions without hooks. In 1.15.41 it is used by the chat CLI and the reasoning planner. The native tool-calling and step executors go through hooks. - The Claude Agent SDK result depends on permission mode. The raising-hook case ran under
bypassPermissions with Bash pre-allowed. - Versions move. These results are for the versions in the table, tested on 14 November 2026.
The takeaway
Every framework I tested has a hook in front of the tool call. That part of the OWASP guide is easy to meet. Whether a broken hook stops the tool is decided by the framework’s exception handling, and in two of the five the default is to let the call through. A policy check that can be made to throw is a policy check that can be skipped.
Check three things in your own agent this week. Does an exception in your pre-tool hook stop the call? Does a slow policy service stop the call, or stall the agent forever? Does one failing check skip the checks after it? Each takes a few minutes to test with a hook that raises, and the answers above tell you where to look first.
If you want a structured read on your agents against the OWASP Agentic Top 10, the Agent Security Scorecard takes about twelve minutes. If you want help testing your own enforcement points, get in touch.
References
Standard
Framework source (versions tested on 14 November 2026)
- CrewAI 1.15.41,
crewai/hooks/dispatch.py: module docstring (lines 21–23) and _invoke_hook (lines 265–292); crewai/hooks/tool_hooks.py, run_before_tool_call_hooks (lines 173–190). PyPI - CrewAI 1.6.1,
crewai/utilities/tool_utils.py, lines 117–130, for the earlier shared-try behaviour. PyPI - Claude Agent SDK 0.2.182,
claude_agent_sdk/types.py, HookMatcher.timeout and HookMatcher.on_failure, lines 854–866. PyPI - langchain-core 1.8.0,
langchain_core/callbacks/base.py, line 506 (raise_error: bool = False). PyPI - openai-agents 0.24.0,
agents/run_internal/tool_execution.py, _execute_tool_input_guardrails, lines 2781–2810; agents/mcp/server.py, line 573. PyPI - mcp 2.3.0,
mcp/server/extension.py, Extension.intercept_tool_call; mcp/server/context.py, ServerMiddleware. PyPI
Test harness
