AI SOC evaluation criteria are the tests a security team uses to judge whether an AI SOC platform reaches correct, explainable decisions on its own alerts, acts only within policy and leaves a record analysts can audit. The most reliable way to apply them is a time-boxed proof of concept (POC) on real alerts whose outcomes you already know.
- Test an AI SOC on your own alerts with known outcomes, not on a demo dataset the vendor prepared.
- Score each verdict twice: once for whether it is correct, and once for whether an analyst can check the evidence behind it.
- Start response actions as recommendations or approval requests, and widen automation only after you have reviewed the outcomes.
This checklist is for SOC managers and CISOs who already have a shortlist and need a fair way to test it. If you are still working out what to shortlist, start with our Autonomous SOC platform buyer's guide. This article picks up where that guide ends, with the hands-on test.
What are AI SOC evaluation criteria?
AI SOC evaluation criteria are the questions you score during an AI SOC POC: what the platform can see, what it decides, how it shows its reasoning, what it is allowed to do and what it leaves on record. We group them into seven areas, so that every platform on the shortlist faces the same questions.
- Data and integrations: can the platform read the data your investigations need and reach the tools you already run?
- Triage quality: do its verdicts match what your analysts would decide?
- Evidence: can an analyst see why it reached each verdict, and correct it?
- Investigation depth: does an escalation produce findings a second analyst can continue from?
- Response control: which actions run automatically, which wait for approval, and who decides?
- Governance: where does your data go, and who can change the policies?
- Measurement: which outcomes are measured, and which are estimates?
How do you run an AI SOC POC?
An AI SOC POC works best when it is narrow and scored in advance: one or two workflows, a labeled sample of alerts and success criteria everyone agrees on before the first run. Here are the steps we recommend, in order.
- Pick one or two workflows your team knows well, such as a recurring endpoint alert type or reported phishing emails.
- Build a labeled sample from alerts your analysts have already closed, with agreed dispositions. Include confirmed true positives and alerts the team found hard to call, not only easy noise.
- Connect the data sources and tools those investigations need. Use a separate tenant or test environment, and read-only credentials wherever you can.
- Write down the scoring and the pass criteria before the first run, and share them with every vendor on the shortlist.
- Run the platform beside your analysts with response actions limited to recommendations, so it cannot change anything in production.
- Review every disagreement between the platform and your analysts in a regular session, and record who was right and why.
- Decide whether to expand, adjust the policies or stop, based on the reviewed results rather than on the demo.
The AI SOC evaluation criteria checklist
Use the checklist below as the scoring sheet for your AI SOC POC. Each item should be answered with evidence from the POC run, such as a record, a log entry or a screenshot, and not with a statement from a sales deck.
Data and integrations
- The platform ingests alerts from the SIEM, EDR, identity and email tools you run today, without custom work for each source.
- The same connectors expose the actions your playbooks need, not only data ingestion.
- A connection test confirms that authentication works, not only that the endpoint responds.
- In a multi-tenant setup, each tenant's endpoints and credentials stay separate.
- There is a supported way to add an internal API or a tool that is not in the catalog.
Triage quality
- Agreement with analyst labels, measured separately for benign and malicious alerts.
- Every missed true positive is counted and reviewed one by one. A single missed intrusion outweighs many correct closures.
- Alerts the platform cannot decide go to an analyst queue instead of being forced into a verdict.
- The same alert, run twice with the same data, gets the same verdict.
Evidence and explainability
- Each verdict shows the entities it extracted (host, user, indicators), the enrichment results and the reasons for the decision.
- An analyst can correct a verdict, and the correction is recorded.
- Answers and findings point back to the source records, so an analyst can check them.
Investigation depth
- An escalated alert produces an investigation plan, the tools it queried, their results, the findings and the recommendations.
- A second analyst can pick up the case without redoing the evidence gathering.
Response control
- You can choose between recommendation only, approval for each action, and automatic execution within a risk threshold.
- High-impact actions, such as isolating a host or blocking an account, wait for approval by default.
- Approval steps have a defined timeout and a defined path when an action is rejected.
- Every executed action is logged with what ran, what it returned and who or what approved it.
Governance and data handling
- You know where the language model runs and which alert data leaves your environment.
- Personal data in alerts is handled according to your data protection policy.
- Decisions and actions can be exported as evidence for audits.
- Permissions control who can change routing, thresholds and approval lists.
Measurement
- Measured outcomes (verdict counts, analyst overrides, open and closed cases) are reported apart from modeled estimates such as time saved.
- The baseline behind any time-saved estimate is visible and configurable.
- You can measure the time your analysts spend reviewing the platform's output, not only the time it saves.
To keep the scoring identical across platforms, we keep the checklist in a small file next to the POC notes. The YAML below is an illustrative example of that scorecard: the criteria, how each one is checked and where the evidence is kept. Adapt the weights and pass criteria to your own risk tolerance.
# Illustrative example: AI SOC POC scorecard
poc:
workflows: [endpoint-alerts, reported-phishing]
sample: labeled alerts closed by analysts, incl. true positives
response_mode: recommend-only
criteria:
- id: triage-agreement
check: verdicts vs analyst labels, benign and malicious separately
evidence: verdict export
- id: missed-true-positives
check: every miss reviewed individually
pass: agreed with SOC lead before the run
- id: verdict-evidence
check: entities, enrichment and reasons visible per verdict
evidence: screenshots of 10 random verdicts
- id: approval-control
check: high-impact actions wait for approval by default
evidence: action log
- id: measured-vs-modeled
check: estimates labeled, baseline visible
evidence: report exportHow do you compare AI SOC vendors fairly?
You compare AI SOC vendors fairly by giving each one the same alert sample, the same scorecard and the same review process, and by scoring the raw records instead of the summary dashboards. A platform that looks strong on its own demo data can behave very differently on your alert mix.
Treat catalog figures with care. A connector count describes what a platform can connect to, not what is connected and working in your POC. Ask each vendor to show the specific connectors and actions your workflows need, configured and tested in your environment. Ask for the run records behind every claimed outcome, and keep a list of what you could not test, so the gaps are visible when you decide.
For the AI-specific risks, such as prompt injection through alert content or excessive agency in response actions, the OWASP Top 10 for LLM Applications gives you a shared vocabulary to question each vendor. The NIST AI Risk Management Framework helps you frame the governance questions for your risk owners.
How Imperum Autonomous SOC maps to this checklist
Imperum Autonomous SOC is an AI SOC that brings alert intake, AI analysis, case investigation and controlled response into one operating flow. We built it around the same questions this checklist asks: what was decided, why, and what still needs a person. Here is how the parts line up, as described on the Autonomous SOC page.
- Triage quality and evidence: Virtus Triage extracts the host, user and indicators, enriches the available evidence, applies policy gates and retrieves relevant knowledge. Where needed, an AI model assesses that context, and validation and audit stages retain the decision and its reasons. Configured actions can close eligible false positives, escalate into a case or send uncertain alerts to the Virtus Triage Queue, and analysts can inspect the reasoning and correct the verdict.
- Investigation depth: Virtus Pilot enriches the alert, builds an investigation plan, uses configured connector tools to test it and produces a report with findings, recommendations and action status.
- Response control: Pilot has three execution modes. Audit skips execution, Guided requires approval for every planned action, and Auto runs actions allowed by your risk threshold while higher-risk actions and actions on the approval list wait for review. The built-in mode is Auto with a low-risk threshold, isolation is on the default approval list, and tenant configuration can override the defaults.
- Measurement: the SOC overview shows verdict counts, analyst overrides and open or closed cases for the selected period, and labels time savings as estimates based on a configurable manual-triage baseline.
For a POC, we follow the getting-started path on that page: choose one input, define the handoff between Triage, Pilot and the analyst, review Pilot's plans in Audit or Guided mode, and extend the boundaries only from reviewed outcomes. Teams that also want detections, playbooks and forensics in the same test can add the SecOps Platform, and the Integrations catalog shows which connectors and actions your workflows can use.
What an AI SOC POC will not tell you
An AI SOC POC tests a narrow slice of your operations over a short period, so some questions stay open when it ends. Plan for them instead of assuming the POC answered them.
- Rare attacks: a labeled sample rarely contains the intrusions you most fear. Add a few replayed or simulated cases mapped to MITRE ATT&CK techniques you care about.
- Change over time: new log sources, new tools and new attacker behavior change the alert mix. Agree how you will re-test after go-live.
- Prerequisites: the checks and actions available depend on your licensed modules, connected data sources, AI services and user permissions. Confirm them before the POC starts, not halfway through.
- Human judgment: policy decisions, approval of high-impact actions and the call on uncertain alerts stay with your analysts. A POC should confirm that the platform leaves room for those decisions, not remove them.
Frequently asked questions
What should I look for in an AI SOC platform?
Look for an AI SOC platform whose verdicts match your analysts' decisions on your own alerts, whose reasoning an analyst can inspect and correct, and whose response actions follow an approval policy you control. It should also keep a record of every decision and action, and separate measured outcomes from modeled estimates. Connector counts and demo results matter less than how the platform performs in a POC on your data.
How long should an AI SOC proof of concept run?
An AI SOC proof of concept should run long enough to cover your normal alert mix and at least one full review cycle in which analysts examine every disagreement with the platform. Agree the duration and the pass criteria before the first run, and keep the scope narrow: one or two workflows tested well tell you more than ten workflows tested briefly.
How do you use AI to cut SOC false positives safely?
To cut SOC false positives safely with AI, let the platform close only alerts that meet policy conditions you have defined, and send everything uncertain to an analyst queue. Record the reasons for every automatic closure, sample closed alerts for review, and track analyst overrides. Missed true positives should be reviewed one by one before you widen the conditions.
Will AI replace the SOC?
AI will not replace the SOC; it changes where analysts spend their time. An AI SOC can take on repeated first-pass checks, evidence gathering and recommendations, while analysts keep the decisions that need judgment: uncertain verdicts, approval of high-impact actions and the policies that govern automation. A good AI SOC evaluation tests that this division of work holds up on your alerts.
Next steps
Take the checklist and the scorecard above, pick one workflow your team knows well and build a labeled sample from alerts you have already closed. Share the scoring with every vendor on your shortlist before the POC begins. If you are still building that shortlist, read our Autonomous SOC platform buyer's guide. To see how one of your own alerts would move through triage, investigation and approval, start from the Autonomous SOC page.
Start with onboarding





