Agent guardrails mapped to the NIST AI RMF

Why I built AgentGuard

I started AgentGuard after watching a team ship an LLM‑powered agent that could read a database, call a payment API and send email, all with a single prompt. The agent worked in the demo, but a simple typo in the user query caused it to transfer money to the wrong account. No one had checked what the agent was allowed to do before the call, and there was no way to stop it once the mistake happened.

I realized that the problem was not the model itself but the missing boundaries around its actions. I wanted a repeatable way to list every tool an agent can reach, decide who can enable each tool, test the agent against the permissions we think it has, and react when something goes wrong. The NIST AI Risk Management Framework gave me the four functions that line up exactly with those steps.

In the sections below I show how each function maps to concrete guardrail practices, how we rate the risk of a tool call, and what you can actually do today with the AgentGuard product.

Core concepts: input filters, tool permissions, output checks

Input filters

These are checks that run on the user prompt or the agent’s internal reasoning before a tool is invoked. Examples include:

- Detecting prompt injection patterns that try to ignore prior instructions. - Checking for disallowed keywords or phrases that signal a request to access a restricted system. - Validating that the prompt contains a required approval token when a high‑risk tool is referenced.

We implement input filters as lightweight regex engines combined with a small allow‑list of safe patterns. They add less than two milliseconds of latency per call.

Tool permissions

Tool permissions define what data a tool can read, write or delete, and which external services it may contact. We store each permission as a tuple:

- Tool identifier - Allowed data sources (e.g., read‑only access to customers table, write access to audit_log) - Network destinations (e.g., api.paymentprovider.com on port 443) - Required approval level (none, peer review, manager sign‑off)

When the agent asks to use a tool, the permission service checks the tuple against the current context and either allows the call, modifies the parameters, or blocks it.

Output checks

After a tool returns a result, output checks verify that the response stays within the expected bounds. Examples:

- Ensuring a SQL query returns no more than 100 rows. - Confirming that an email payload does not contain external URLs. - Scanning for leaked secrets such as API keys or passwords in the response.

If a check fails, the system can truncate the result, redact sensitive bits, or trigger an alert.

Mapping guardrails to NIST AI RMF functions

The framework’s four functions give a clear structure for organizing guardrail work. Below is a table that shows which guardrail belongs to each function, the concrete artifact we produce, and a short example.

NIST function | Guardrail type | Artifact | Example Govern | Approval workflow | Tool enablement record | A signed document that lists who approved the send_email tool, the date, and the required MFA for the approver. Map | Tool inventory | Permission matrix | A spreadsheet that enumerates every tool, the tables it can read/write, and the external endpoints it may call. Measure | Test suite | Adversarial prompt set | A collection of prompts designed to trigger privilege escalation, data exfiltration, or unauthorized external calls. Manage | Runtime response | Kill‑switch log | A timestamped entry that records when the agent tried to delete a production record, the action that was blocked, and the operator who was notified.

Each function builds on the previous one. Govern decides who may add a new tool to the inventory. Map creates the inventory that Measure will test against. Measure produces evidence that Manage can act on. The loop repeats whenever the tool list or the agent’s prompt changes.

How we rate a tool call

Step 1: Identify the tool and its permission class

We start by looking up the tool in the permission matrix. The matrix assigns a base risk level:

- Read‑only lookup (e.g., fetching a product catalog) → low - Write to an internal log → medium - External action with side effects (e.g., sending money, deleting a record) → high

If the tool requires an approval token, we add one level to the base risk.

Step 2: Analyze the prompt for injection risk

We run the user prompt through the input filter engine. If the engine detects a pattern from the OWASP prompt injection cheat sheet (https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) we increase the risk by one level.

Step 3: Check data sensitivity

We examine the data sources the tool can touch. If any source contains personally identifiable information (PII) or financial data, we add another level.

Step 4: Produce the final rating and suggested filters

The final rating is the sum of the base level, prompt‑level, and data‑level adjustments. Based on the rating we recommend:

- Low → no additional filter, just log the call. - Medium → add an input filter that strips disallowed keywords and an output check that limits row count. - High → require a human‑in‑the‑loop approval, enforce a strict allow‑list of parameters, and enable full output redaction.

This rating is recomputed each time the tool list changes, ensuring the guardrails stay aligned with the agent’s capabilities.

What this is NOT

I want to be clear about the limits of guardrails so you do not mistake them for a silver bullet.

- Guardrails do not guarantee that an agent will never cause harm. They reduce the blast radius of a mistake, but a determined attacker or a badly scoped tool can still produce unwanted effects inside the allowed boundary. - They are not a replacement for secure coding practices. If the underlying tool has a vulnerability (for example, a SQL injection flaw described in the SQL injection cheat sheet https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html), the guardrail cannot fix that flaw. - They do not provide a 100 % safety certification. Think of them as decision‑support aids that help you make informed choices about tool usage. - They are not a set‑and‑forget configuration. Whenever the agent’s prompt, the tool list, or the data a tool can reach changes, you must re‑run the review process.

Putting it into practice: a step‑by‑step review workflow

1. Gather the agent’s system prompt and tool list

Export the exact text that the agent receives at startup and the JSON or YAML file that declares every tool it can call.

2. Populate the permission matrix

For each tool, record:

- Allowed read targets - Allowed write targets - External endpoints and required authentication - Any approval token or role needed

3. Run the baseline risk rating

Apply the four‑step rating process described earlier. Record the initial risk level for each tool.

4. Generate input and output filter suggestions

Based on the risk level, produce concrete filter rules (regex patterns, row‑count limits, redaction rules). Store them alongside the tool definition.

5. Execute the adversarial test suite

Run the agent against the prompt set from the Measure column of the table. Verify that the filters block or mitigate the expected failure modes.

6. Document the outcome

Create a short report that lists:

- Tool name - Initial risk rating - Applied filters - Test results (pass/fail) - Any required manual approval steps

7. Set up monitoring and alerts

Configure the agent’s runtime environment to log every tool call, the applied filters, and the output check results. Route anomalies to your incident‑response channel.

8. Schedule a review trigger

Mark the workflow to repeat when any of the following occurs:

- A new tool is added to the list - The system prompt is edited - A tool’s permission changes (for example, a database gains a new column with PII)

Common pitfalls and how we avoid them

Pitfall 1: Over‑reliance on a single filter type

Teams sometimes think that adding a keyword blocker is enough. We counter this by requiring both input and output checks for any medium or high‑risk tool, and we log when either check fires.

Pitfall 2: Forgetting to update the permission matrix

When a tool gains a new capability (e.g., a payment API now supports refunds), the matrix must be edited before the agent can use it. We enforce this by making the matrix a version‑controlled file that the CI pipeline validates on every push.

Pitfall 3: Ignoring low‑risk tools that become high‑risk in context

A read‑only lookup that returns a user’s email address can become dangerous if the agent later uses that address in a password‑reset flow. We treat data sensitivity as a separate rating factor, so a low‑level tool that touches PII is automatically upgraded.

Pitfall 4: Skipping the human‑in‑the‑loop step for high‑risk tools

For tools that can move money or delete records, we mandate an explicit approval step that cannot be bypassed by automation. The approval is recorded in an immutable log.

Pitfall 5: Treating the guardrail setup as a one‑time task

We embed the review workflow into the team’s definition of done for any feature that changes the agent’s behavior. This makes the process repeatable rather than a forgotten checklist item.

References

- https://genai.owasp.org/llmrisk/llm01-prompt-injection - https://owasp.org/www-project-top-10-ci-cd-security-risks/CICD-SEC-06-Insufficient-Credential-Hygiene - https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html - https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html

Frequently asked questions

Do guardrails make an agent safe?

No. They constrain the blast radius of a mistake. A determined attacker, a mis-scoped tool or a bad prompt can still cause harm inside the boundary you allowed.

What do we need to run a review?

The agent's system prompt and its tool list. That is enough to rate calls and suggest filters.

Is the EU AI Act mapping legal advice?

No. It is a mapping to help you find the right obligations. Classifying your system and meeting the obligations still needs a qualified reviewer.

How often should we re-run it?

Whenever the tool list, the system prompt or the data a tool can reach changes.

Does it test the agent at runtime?

It reviews the prompt and tool configuration. Runtime monitoring is a separate layer you still need.

What if a tool is genuinely high risk?

Require an approval step, narrow its parameters, and log every invocation. High-risk tools usually need a human in the loop rather than a filter.

References

- OWASP — Top 10 for Large Language Model Applications - NIST — AI Risk Management Framework

Try the product: AgentGuard

2026-09-20 · primary sources only · no fabricated traffic metrics · decision-support only

Related articles

- AI agent tool allowlists: why deny-by-default wins - Explore the full AI micro-SaaS topic matrix →