AI SRE agents promise to reduce the time engineers spend investigating production incidents.
An alert fires. The agent retrieves logs, traces, metrics and deployment history, analyzes the available evidence, and suggests a likely root cause.
It sounds straightforward.
But there's an important question engineering and security teams should ask before giving an AI agent access to production systems:
Where does your production data actually go during that investigation?
This matters because production telemetry is not just technical information. It can contain customer identifiers, internal service details, API paths, error payloads, infrastructure metadata and occasionally sensitive information that was never intended to reach an external system.
Let's look at the data flow and the architectural decisions that matter.
1. What data does an AI SRE agent need?
An AI SRE agent needs context to investigate incidents effectively.
Depending on the system and integrations, that context can include:
- Logs: Application events, exceptions, stack traces and error messages.
- Metrics: CPU, memory, latency, throughput and resource utilization.
- Distributed traces: Request paths and dependencies across services.
- Error data: Exception groups, affected endpoints and failure patterns.
- Session context: User interactions leading up to a frontend error.
- Git and deployment metadata: Recent commits, code changes and deployments.
The more useful context an agent has, the better it can investigate.
For example, a latency spike might initially look like a database problem. But when the agent correlates traces, application logs and a recent deployment, it may discover that a new downstream API call is responsible.
The important point is that the agent may need access to considerably more than the original alert.
2. Follow the data through an AI investigation
A typical investigation has several stages.
Stage 1: Data collection
Telemetry is collected from applications and infrastructure through instrumentation and integrations.
Stage 2: Context retrieval
The agent queries relevant data sources. It may retrieve logs, traces, deployment events, repository information and infrastructure metadata.
Stage 3: AI inference
Relevant context is assembled into prompts or model requests. The AI model reasons over the evidence and generates an investigation result.
Stage 4: Results and remediation
The agent returns a root-cause hypothesis, supporting evidence and potentially a suggested fix or pull request.
Each stage introduces a different data-handling question.
Where is telemetry stored? Where is the context assembled? Which model processes it? Where are prompts and responses retained?
Looking only at the location of the observability database does not answer all these questions.
3. Three deployment models to understand
Vendor-hosted SaaS
In a traditional SaaS architecture, telemetry is sent to infrastructure operated by the vendor.
The vendor may also process selected context through an AI model provider.
This can be convenient because the infrastructure and integrations are managed for you. However, engineering and security teams should understand which data leaves their environment, where it is processed and which subprocessors are involved.
Bring Your Own Cloud (BYOC)
With BYOC, the platform runs inside infrastructure controlled by the customer.
The customer retains control over the cloud account and the environment where application telemetry is stored and processed.
This changes the data boundary compared with a conventional shared SaaS deployment.
However, BYOC alone does not answer every AI-related security question. You still need to understand where model inference occurs and whether prompts or retrieved context are sent elsewhere.
Bring Your Own Key (BYOK)
BYOK allows a customer to connect their own AI model API key or deployment.
For example, a company may use its own Azure OpenAI or Amazon Bedrock deployment rather than a model account managed by the software vendor.
This gives the customer greater control over model selection, account ownership and associated AI usage.
One distinction is worth remembering:
BYOC controls where the platform runs. BYOK controls whose model credentials and inference deployment are used.
They address related but different parts of the data architecture.
4. Questions to ask before adopting an AI SRE agent
Before connecting an agent to production, ask the vendor:
- Where is telemetry stored and processed?
- Does raw application data leave our cloud environment?
- Which AI model receives investigation context?
- Are prompts and responses retained by the model provider?
- Is customer data used for model training?
- Can we use our own model key and deployment?
- What permissions does the agent require?
- Can we audit the data it retrieves and actions it performs?
- Does it require write access to production, or can it begin in read-only mode?
- Can we verify the architecture through our own cloud and network logs?
Don't settle for an answer that simply says the platform is secure or enterprise-ready.
Ask for the actual data flow.
5. Security is also about what the agent can do
Data access is only half the problem.
Consider two AI agents:
- Agent A can read telemetry, investigate errors and suggest a fix.
- Agent B can read telemetry, modify infrastructure, execute commands and deploy changes.
These are very different permission models.
A sensible adoption approach is to begin with read-only investigation and gradually introduce approval-based remediation.
For example, an agent could generate a pull request containing a suggested code fix. An engineer reviews the evidence and changes before merging it.
That is very different from allowing an AI system to modify production automatically.
The objective should be to reduce investigation effort without removing engineering control.
6. How Modulis approaches data ownership
Disclosure: I work with Modulis.
Modulis follows a Bring Your Own Cloud architecture. Its software is deployed inside the customer's own cloud environment, and application telemetry such as logs, traces, metrics, errors and session data is processed and stored there.
For AI-assisted resolution, Modulis supports Bring Your Own Key. Customers connect their own model key, such as an Azure OpenAI or Bedrock deployment, so inference runs within their cloud boundary rather than through a third-party model controlled by Modulis.
The intention is to give engineering teams a managed production intelligence platform while retaining control over their telemetry and AI model usage.
You can read more about the architecture and data-handling approach here:
https://modulis.ai/blog/ai-sre-agent-where-your-data-goes/
Final thoughts
AI SRE agents can make incident investigation significantly more efficient, but adopting one should involve more than evaluating the quality of its root-cause analysis.
Understand the complete data lifecycle:
- What data is collected?
- Where is it stored?
- Where does inference happen?
- Who can access it?
- What can the agent change?
An AI agent should make production operations easier without creating uncertainty about where production data goes.
The right question isn't just how intelligent the agent is. It's whether you understand and control the environment in which it operates.
