Back to Blog
Engineeringv0.0.2a1

Webhook & Validation Pipeline

Engineering Brief — v0.0.2a1

Every agent call now runs through four real engines before a webhook ever fires. v0.0.2a1 replaces the two-line approval stub with an actual scoring pipeline — schema, tool-trust, policy, consistency, and grounding checks feed a confidence score, a four-factor risk formula, and a priority-ordered decision table.

June 23, 202612 min read
WebhooksValidationSDKRuntimeSecurity
Internal Engineering Brief

Every agent call now runs through four real engines before a webhook ever fires.

v0.0.2a1 replaces the two-line approval stub with an actual scoring pipeline — schema, tool-trust, policy, consistency, and grounding checks feed a confidence score, a four-factor risk formula, and a priority-ordered decision table.

validate() → 4 engines → decisionpayment-agent → BLOCKdatabase-agent → APPROVEemail-agent → BLOCK (adversarial)
00Context

What actually changed

The embedded gateway stopped being a placeholder. The decision your webhook reports is now backed by a real rule engine, not a coin flip dressed as a policy check.

decision = "approve" if not failures else ("block" if schema_score == 0 else "retry")
+decision = DecisionEngine.decide(confidence, policy_score, risk_tier)  # 4-engine pipeline, _real_validator.py
Before: any non-empty dict from the agent was approved. After: 6 deterministic checks → weighted confidence → 4-factor risk → priority-ordered decision rules.
01System Map

Architecture, end to end

One synchronous call out, one decision back, then a fire-and-forget fan-out that never touches the agent's return path.

Your code

Agent function → AgentTrustClient.validate()

POST /v1/runtime/validate · Bearer token · JSON envelope (agent_id, request, execution, output)

Embedded gateway (FastAPI)

_real_validator.validate_payload()

Runs the full pipeline below, synchronously, on the request path.

4The Four Engines
Stage 1
ValidationEngine
6 deterministic checks · <20ms · zero LLM calls
Stage 2
ConfidenceEngine
Weighted average of 5–7 signals → final_confidence
Stage 3
RiskEngine
4-factor multiplicative formula → risk_tier, risk_score
Stage 4
DecisionEngine
Priority-ordered thresholds → one of 5 outcomes
Response

{ decision, confidence, risk, failures, rationale }

Returned to your code unconditionally — webhook delivery never affects this.

Notification layer (client-side, async)

WebhookDispatcher.dispatch()

Loads enabled webhooks from SQLite → filters by event type → builds payload → POSTs, 5s timeout, fail-open on error.

Discord — rich embed
Slack — JSON
PagerDuty / custom — JSON
02Inside the gateway

The validation pipeline, stage by stage

This is the part worth walking the team through slowly — it's also the part most likely to surprise people the first time an agent they expected to pass gets blocked.

Stage 1

ValidationEngine — 6 checks

CheckWhat fails it
Schemamissing envelope field, −20 pts each
Tool trusttool called with no matching result
Policybase rules + financial pack + adversarial scan
Consistencyempty output, no model, latency ≤ 0
Groundingnumeric claim with no matching tool result (±50%)

A critical policy violation hard-caps the policy score — nothing else in this stage can recover it.

Stage 1c — the one that surprises people

Adversarial scan

Scans request input and serialized output for injection patterns. Any match caps policy_score at 20, regardless of every other rule.

PatternExample trigger
Instruction override"ignore previous instructions"
Jailbreak keywords"jailbreak", "DAN"
Identity override"pretend you are a different AI"
Data exfiltration"exfiltrate user data"
Stage 2

ConfidenceEngine

A weighted average over whichever signals are actually present — absent signals are excluded from both numerator and denominator, so the score stays normalised to 0–100.

final_confidence = Σ(score_i × weight_i) ───────────────────────── Σ(weight_i) [present signals only]
SignalWeight
Schema / Tool trust / Policy20% each
Consistency15%
Grounding / Judge*10% each
Historical reliability*5%

*Not available in embedded mode — excluded, weights renormalise.

Stage 3

RiskEngine

risk_score = ActionSeverity × BusinessImpact × ConfidenceGap × PolicySensitivity ─────────────────────────────────── × 100 10,000
FactorDriven by
Action severitytool name (delete/transfer = high, search/list = low)
Business impactagent_id pattern (medical-* = 9, faq-* = 2)
Confidence gap(100 − confidence) / 10
Policy sensitivity(100 − policy_score) / 10
Stage 4 — evaluated top to bottom, first match wins

DecisionEngine priority order

  1. 1confidence < 50 or policy_score < 60BLOCK
  2. 2risk_tier == criticalESCALATE
  3. 3risk_tier == high and confidence < 80ESCALATE
  4. 4confidence ≥ 90 and risk_tier in (low, medium)APPROVE
  5. 5confidence ≥ 70 and risk_tier == lowAPPROVE
  6. 650 ≤ confidence < 70RETRY
  7. 7fallbackREQUEST_EVIDENCE
03Worked Examples

Live trace — the four demo agents

The exact scenario in examples/06_webhook_integration.py. Click an agent to see its scores move through the pipeline and land on a decision.

payment-agent · user alice

input: “Transfer $500 to external account”
output: {"status": "ok"}
⛔ BLOCK
Schema
100
Tool trust
100
Policy
0
Consistency
66.7
Grounding
100
Final Confidence
70.6%
Why: policy_score = 0 (critical: payment_amount_present missing) — below minimum threshold of 60
04After the Decision

Webhook dispatch & delivery

Independent of the validation pipeline above — this runs once per registered webhook, every time, and never blocks the response your agent already received.

Decision → colour

DecisionColourMeaning
BLOCK#FF4757policy or confidence floor breached
ESCALATE#FFA502critical risk, human review
APPROVE#2ED573all checks passed
RETRY#6B6B80not yet classified

Filter → destination

events filterFires on
"all"every decision — audit trail
"block"block only — on-call paging
["block","escalate"]combined high-priority channel
"approve"confirmation log

All-approve traffic against a block/escalate-only filter producing zero notifications is correct behaviour, not a bug.

Per-webhook loop (runs for every registered hook, every call)

  1. 1Load enabled webhooks from ~/.agentrust/webhooks.db
  2. 2Does this decision match this webhook's events filter? — no → skip silently, no request sent
  3. 3Build payload — Discord URL detected → rich colour embed · anything else → flat JSON
  4. 4POST with a 5 second timeout
  5. 5Failure → logged at WARNING, loop continues — never raised back to your agent
05Getting It Running

Setup runbook

Pick one destination, prove the wiring locally, then promote to programmatic registration once you need more than one sink.

01

Create the webhook URL at the destination

Discord

Channel settings → Integrations → Webhooks → New Webhook → copy URL. Treat this like a password the moment it's copied.

02

Drop it into .env — never into source, never into a message

Confirm .env is in .gitignore before you save — every time, don't assume the template already did it.

AGENTRUST_WEBHOOK_URL=https://discord.com/api/webhooks/YOUR_ID/YOUR_TOKEN
AGENTTRUST_WEBHOOK_EVENTS=block,escalate
03

Smoke-test with two terminals before real traffic

This separates "is my filter/URL correct" from "is my agent producing the decision I expect" — two different failure classes.

# Terminal 1 — disposable local sink
python3.11 demo/webhook_receiver.py

# Terminal 2 — run the demo agents
export $(grep -v '^#' .env | grep -v '^$' | xargs)
python3.11 examples/06_webhook_integration.py
04

Or skip code entirely

env vars only

Works with any @harness-wrapped function, no other code changes required.

export AGENTRUST_WEBHOOK_URL="https://discord.com/api/webhooks/YOUR_ID/YOUR_TOKEN"
export AGENTRUST_WEBHOOK_EVENTS="block,escalate"
05

Promote to programmatic registration for multiple sinks

Registrations land in SQLite and survive process restart — do this once per environment, not once per boot.

dispatcher.register(url=DISCORD_URL, events=["block","escalate"], name="discord-oncall")
dispatcher.register(url=SLACK_URL,   events=["all"],              name="slack-audit-log")
06

Verify before trusting it in production

dispatcher.list_webhooks() — check url_masked, events, and enabled for each entry. You're confirming shape and filter, not the literal secret — that's the point of the mask.

06Non-negotiables

Security guidelines

Webhook URLs are bearer credentials. Anyone holding the URL can post into your Discord channel or Slack workspace. Store only in .env or a secrets manager — never in chat, email, or a PR comment.
Tokens are masked everywhere — logs and list_webhooks() show only the last 8 characters. If a URL is ever exposed, delete and recreate it at the destination immediately; rotating in .env after the fact is not sufficient since the old URL stays live until deleted.
HTTPS is enforced for every destination except http://localhost:* for local development.
07Before You Ship This

Engineering review notes

What I'd actually raise in a design review before treating this as production-ready.

VERIFY
The JWT tier claim needs to be signed and checked, not just read

embed_gateway() issues a JWT-shaped token with tier: team in the payload to stop _apply_tier_mask from stripping scores. Confirm auth.py actually verifies a signature — a token whose claims are trusted on read alone is a local convenience, not an access control, the moment this gateway is reachable from anywhere but localhost.

WATCH
Deterministic rules will both over- and under-trigger at the edges

The financial pack matches on agent_id glob (payment-*, loan-*...). An agent named outside that pattern handling money gets none of those critical checks; one named inside it but doing something unrelated inherits rules it doesn't need. Treat the glob list as something the team reviews on every new agent, not a one-time setup.

WATCH
Confidence and policy weights silently renormalise around missing signals

With judge_score and historical_reliability absent in embedded mode, their 15% combined weight redistributes across the remaining five signals. That's correct math, but it also means embedded-mode confidence numbers are not directly comparable to a future deployment where the LLM judge is wired in.

RISK
Client-side fan-out still doesn't scale past one process

Multiple worker processes each read the same SQLite registry independently — no shared rate limit, no dedup. A single block replicated across N workers means N near-simultaneous POSTs to the same channel. Fine for one process; revisit before running this across a fleet.

RISK
Fail-open with no retry means a flaky network during an incident drops the alert silently

A timeout becomes a WARNING log line and nothing else. For anything routed to on-call, I'd want at least one retry with backoff, or a local fallback log you can replay — as shipped, a missed page during an actual outage looks identical to a quiet day.

WATCH
The audit table grading itself 100% green is a smell, not a guarantee

A self-reported parity table against Adrian with zero caveats reads like changelog copy. Before presenting "real validation engine" to stakeholders as fact, I'd want someone other than the author to have run tests/test_webhooks.py and read _real_validator.py directly — v0.0.2a1 is still an alpha version string.

READY TO GOVERN YOUR AGENTS?

No AI Agent enters production without AgentTrust

Confidence in every decision — pre-production certification to real-time runtime governance. Start free, no credit card required.

See Pricing & Start Free →
AgentTrust OS — internal engineering brief — built for team walkthroughsource: WEBHOOK_GUIDE.md v0.0.2a1

More from the blog

AI ComplianceJuly 22, 2026AI ComplianceJuly 22, 2026AI GovernanceJuly 22, 2026AI GovernanceJuly 22, 2026AI ArchitectureJuly 22, 2026AI SecurityJuly 16, 2026MLOpsJuly 10, 2026AI ImplementationJuly 8, 2026AI Agent ArchitectureJuly 5, 2026AI Agent ArchitectureJune 30, 2026AI Agent ArchitectureJuly 2, 2026AI Agent ArchitectureJuly 1, 2026IntegrationsJuly 1, 2026IntegrationsJuly 1, 2026IntegrationsJuly 2, 2026AI SecurityJuly 3, 2026AI SecurityJuly 1, 2026AI ComplianceJuly 2, 2026AI ComplianceJuly 3, 2026AI StrategyJuly 2, 2026AI StrategyJuly 3, 2026AI StrategyJuly 3, 2026AI StrategyJuly 3, 2026AI GovernanceJuly 28, 2026Healthcare AIJuly 28, 2026ArchitectureJuly 29, 2026ArchitectureJuly 29, 2026AI StrategyJuly 29, 2026AI StrategyJuly 30, 2026AI SecurityJuly 30, 2026AI ComplianceJuly 30, 2026EngineeringJuly 30, 2026AI GovernanceAugust 4, 2026EngineeringAugust 4, 2026EngineeringAugust 4, 2026