Vendors talk about “autonomous SOCs” as if autonomy were a switch. In practice an agent can be excellent at closing VPN-related sign-in alerts and unreliable on suspicious PowerShell in the same week. Trust has to be granted per alert class (one analytics rule or detection type), per tenant, based on evidence.

Step 1: run in shadow mode

The agent investigates every incident and records its verdict, but analysts still make every decision. Each human closure becomes a labelled example to compare against. Run shadow mode long enough to see each alert class many times, including edge cases.

Step 2: measure accuracy per class, and the right kind

What matters is how often the agent says “benign” when the analyst says “true positive.” That's the error that hurts. An agent that escalates too often only costs time. An agent that closes a real incident costs a client.

• Track false benign verdicts separately from overall agreement.
• Require a meaningful sample per class before it's eligible. A handful of agreements proves nothing.
• Set the threshold by severity: stricter for identity and endpoint classes than for informational ones.

Step 3: promote a class, with guardrails

When a class meets the bar, enable auto-close for it in that tenant only. Every auto-closure should:

• Leave a comment on the Sentinel incident with the evidence and queries used
• Use a distinct classification reason so auto-closures can be reported separately
• Stay within limits, for example never auto-closing when the entity is a privileged account

Step 4: audit samples continuously

Have an analyst review a random sample of auto-closures every week. If the sample turns up a false benign, the class goes back to approval mode until the cause is understood. That isn't a failure. It's the system working as designed.

Step 5: know what resets trust

Accuracy measured last quarter doesn't hold if the inputs change. Drop a class back a level when:

• The analytics rule's logic or thresholds are edited
• A new data source or connector changes what the agent can see
• The client's environment changes significantly (a merger, a new VPN, a new identity provider)
• The underlying model or agent prompt is updated

Response actions come last

Closing a benign alert is reversible: reopen it. Isolating a production server is not. Keep containment actions approval-gated well after auto-close is trusted, and pre-approve only narrow, low-blast-radius actions, such as revoking one user's sessions, with a clear list of exceptions.