DEV Community: ObservabilityGuy

The Player Says 'My Top-Up Never Arrived.' The Log Only Says BAG_FULL.

ObservabilityGuy — Thu, 29 Oct 2026 05:24:09 +0000

This article introduces how the SLS Semantic Layer and DataAgent automate multi-log troubleshooting to deliver verifiable business conclusions.

1. Why Does a Single "Top-Up Not Credited" Complaint Require Checking Multiple Logs?

At 9:47 p.m., support agent Lin picks up an urgent ticket. The player says they just paid 648 CNY, the money is gone, but the top-up package never showed up.

Lin checks the order first: payment succeeded. She checks the payment callback: signature verification passed too. Neither the money nor the payment channel is at fault, so the problem has to be in delivery. She switches to the delivery log and finds a single line: deliver_status = failed. Following the trace_id into the system log, she picks up one more clue: the error code BAG_FULL.

A veteran agent reads this instantly: the player's inventory was full, the system retried delivery, and it kept failing. To someone new to the business, though, this is just a handful of fields and values scattered across different LogStores. To give the player one accurate answer, Lin has to jump back and forth between orders, payments, delivery, and the inventory transaction log, translating machine-recorded facts into a conclusion the player can understand.

This ticket comes from a demo we built, and the logs are synthetic. But cases like this are daily work for game support: top-ups and payments, item changes, matchmaking and ranked play, anti-cheat appeals, social disputes. Answering one of them usually spans several LogStores and takes three to six queries in sequence. The most obvious cost is time: the agent switches repeatedly among orders, payment callbacks, delivery, and the inventory transaction log, several rounds of queries for a single question. And finding the data does not mean understanding it: a newcomer and a veteran can read the same field and reach different conclusions. Add phrases like "revenue," "credited," or "I won but lost points," which different business teams compute differently, and you first have to ask which metric definition the question means.

2. Logs Record Facts, Not Conclusions

The logs are all there. The difficulty is that logs never state the business conclusion outright. Look at deliver_status = delivered and item_id = WPN_LEG_001: neither a person nor an AI can know from the values alone that they mean "delivered" and "Skysplitter Blade." The machine records facts; what those facts mean for the business is not written anywhere in the log.

The SLS semantic layer and DataAgent supply that missing translation. The semantic layer organizes the fields, enum values, business metrics, glossary terms, and samples in your LogStores into a business dictionary. Once the dictionary exists, everyone who asks gets the same metric definition. The DataAgent is the investigator who carries that dictionary. It is attached to at least one Context Model; it maps the player's natural-language question to an explicit metric definition and data source, then builds the queries, analyzes the evidence, and finally produces a conclusion and a draft reply.

3. Three-Layer Collaboration: From Raw Logs to Support Conclusions

All synthetic logs for the sample game are written to the SLS project semantic-game-demo. When troubleshooting, the DataAgent uses three business-evidence LogStores.

LogStore	Main data
game-biz-logstore	Business transaction records for orders, payment callbacks, and delivery; inventory and item changes; gift mail, guild rewards, and season rewards
game-audit-logstore	Audit records for payment risk control, player trades, matchmaking and match settlement, anti-cheat and appeals, reports, and season participation
game-system-logstore	System runtime records for service faults, dependency failures and recovery, performance metrics, and season rule configuration

Most player events carry the common fields user_id, server_id, timestamp, and trace_id. Cross-LogStore troubleshooting then relies on business identifiers such as order_id, battle_id, and item_uuid to string the evidence together. On top of that, each of the three layers has its own job.

(1) Source Model: Describing the Data as It Is

A Source Model maps one-to-one to a project, so you do not name it separately. Once you start automatic generation, the system uses an LLM to analyze dashboards, alerts, Scheduled SQL, and data sampling together, and extracts field semantics, metrics, and glossary terms. Generation time depends on how many LogStores the project holds and how much data they contain; it usually finishes within a few minutes. Extraction results fall into three states by how strong the evidence is. Auto-adopted means the evidence is sufficient and the entry is ready to use. Not adopted means the evidence is thin or ambiguous and someone needs to confirm it. Manually adopted means a person has already checked it. The DataAgent uses only adopted content, so a field meaning still in doubt is never treated as a business fact. A few examples make clear what it extracts.

Semantic element	Game scenario example	Problem it solves
Field	In the order log, deliver_status = delivered means delivered, and amount is denominated in cents	How to interpret fields and enum values
Metric	"Recognized recharge revenue" = SUM(CASE WHEN deliver_status = 'delivered' THEN amount ELSE 0 END)	How a business metric is computed
Glossary term	"Diamonds" = currency_diamond; "Skysplitter Blade" = WPN_LEG_001	How player wording maps to the logs

(2) Context Model: Scoped by Business, Aggregated Centrally

The Context Model is the account-level semantic layer. Each project has one Source Model. When a business spans several projects, you first pick the Source Models of the relevant projects, then select from them the LogStores this business actually needs, and the scattered data semantics come together into one unified model. The game-support Context Model in this article, for instance, selects only the three business-evidence LogStores; unrelated data sources and evaluation-only data sources stay out. Content in a Context Model comes from two places. One is automatic synchronization: whatever the selected LogStores hold in the Source Model flows in automatically, with a maximum delay of 60 seconds, and entries are only added, never removed. The other is manual additions: you can append metrics, glossary terms, and samples, and automatic synchronization never touches these entries. A Context Model aggregated this way serves the DataAgent, and agents such as Codex, OpenClaw, and Claude can connect to it and reuse it as well.

(3) DataAgent: Troubleshooting with a Unified Definition

A DataAgent is a conversational assistant instance, and you attach at least one Context Model when you create it. Given a question, it first identifies the business terms and the metric definition, then chooses data sources, builds queries, strings multi-step log evidence together, and finally outputs a conclusion, the supporting evidence, and a draft reply. You can check every step.

4. Can We Skip the Semantic Layer and Let the LLM Query Logs Directly?

By now you are surely asking: querying SLS logs through Model Context Protocol (MCP) or NL2SQL already works today. If the LLM can write the query itself, why build a semantic layer first?

Take "revenue" from earlier. It could mean order face value, the amount actually paid, or the delivered amount. Ask the LLM directly and it has to guess which field to use and whether to filter for successful delivery; ask the same question twice and it may guess differently. The semantic layer pins down the formula for recognized recharge revenue, so the same metric definition is computed no matter who asks. Now take "Big Sword." That is the name players use; the log stores WPN_LEG_001. Without the glossary mapping, the LLM does not know which log table to open or which value to match, and will likely run a full-text search on the player's wording, find nothing, and tell the player the item was never lost. Then there is that top-up ticket. The useful conclusion is that a full inventory caused delivery to fail, and it takes the payment, delivery, and system logs together to support it. A one-shot query tends to stop at the first plausible answer: it finds deliver_status = failed, blames delivery, and never asks why delivery failed. The DataAgent gets as far as inventory capacity because the Context Model tells it in what order to look and which pieces of evidence to connect.

The query capability is already there. To make queries trustworthy, every conclusion has to map to fixed fields and formulas, and the path of verification has to be deterministic. That is exactly what the semantic layer does. Grant an LLM query permissions without this layer and its answers will read smoothly, so smoothly that you will have trouble noticing whether the metric definition is right and whether the evidence is complete.

5. Best Practice: Build a Game-Support DataAgent in Four Steps

There are three prerequisites. Business logs are already in the project semantic-game-demo, the relevant LogStores have the field indexes required for querying, and the operator has permissions to create and configure a DataAgent.

Step 1: Generate the Source Model for Semantic-game-demo

Open DataAgent from the SLS console home page.

On the source model tab, select the business project semantic-game-demo.

After you start the source model generation task, you can check its status on the details page. Generation time depends on how many LogStores the project holds and how much data they contain; it usually finishes within a few minutes.

When generation finishes, the system lists the field semantics, metrics, and business glossary terms extracted from the LogStores.

For entries that are not adopted, verify the business meaning first, then adopt them in bulk once they are correct, or edit and adopt them one by one.

Step 2: Create the Game Context Model

Create a Context Model and associate the source model of semantic-game-demo. In the description, we recommend stating the business scope the model covers and its main uses, for example support scenarios such as top-ups, items, ranked play, and anti-cheat.

Once associated, the adopted field semantics, metrics, and glossary terms from the source model are synchronized in. You can then add domain-specific terms, metric definitions, and common samples, where a sample can include the question description, the query statement, and the related fields.

Step 3: Create the Game-support DataAgent

Create the DataAgent and attach the context model you built in the previous step.

A DataAgent requires a dedicated RAM role at runtime. By default you can use the system role aliyunslsdataagentrole; if authorization is not yet complete, grant it through the authorization link first. To restrict which resources the agent can access, switch to a custom role, in which case the operator needs the corresponding ram:PassRole permission.

Step 4: Run Conversational Tests with Typical Complaints

Create a conversation, select the DataAgent you just created, and paste in the player's complaint verbatim. Using the context model, the DataAgent identifies the glossary terms and metric definitions, queries the relevant logs, and outputs the key evidence, its analysis, and a draft support reply.

6. How Did the Blind Test Perform?

The test runs on reproducible synthetic logs and covers 13 categories of customer-complaint scenarios.

First, a word on evaluation isolation. game-qa-logstore stores only the questions, the ground truth, and the evaluation evidence; it is not a business query data source. During a blind test, do not grant the DataAgent query permissions on this LogStore, or it will see the ground truth. Take "top-up not credited" as an example. The DataAgent first checks payment: the callback succeeded, signature verification passed, the money really was paid. Next it checks delivery: delivery failed after three retries. It then looks into the cause, and the system error and the inventory capacity point to the same one: a full inventory. The final conclusion rests on the payment, delivery, and system logs together; a single error code cannot get you there.

Now three representative cases.

Player question	Key evidence the DataAgent connects	Conclusion produced
Item missing "Big Sword" WPN_LEG_001	Joins the inventory transaction log with trade records via item_uuid; logon devices normal, no unusual logon; change_reason = trade_sell, sale price 64,663 gold	Based on the available logs, the item was sold on the auction house from the current account, and no evidence of unusual logon or abnormal device activity has been found so far. We recommend providing the trade record and prompting the player to check their account security
Won but lost rating points Match BT_FC3E058B	battle_result = win; rating dropped from 1259 to 1237; rank_change_reason = disconnect_penalty; disconnected for 130–200 seconds, over the 120-second threshold	The rating change comes from the "disconnect counts as forfeit" penalty rule, not from a fault in the normal win-rating logic
Wrongful cheating-ban appeal Event CHT_922FE15A	Detection value 8.85, about 4.1% above the 8.5 threshold; ml_score = 0.62, below the high-confidence bar of 0.7; memory scan result clean; ping 180–250 ms	The available evidence better supports a movement-speed calculation deviation caused by high latency; we recommend handing the case to the anti-cheat team for manual review

The restraint in these conclusions is worth noting. The missing-item conclusion does not say the item "was not stolen"; it says the available logs show no anomalous evidence, and it recommends prompting the player to check their account security. The wrongful-ban appeal does not overturn the ban either; it says the evidence better supports a high-latency deviation and recommends manual review. The conclusion stops where the evidence stops, and the action is left to a person. When a question could map to more than one metric definition, for instance "revenue" meaning order face value, the amount actually paid, or the delivered amount, the DataAgent first asks which definition you want, then continues the query. To reduce these clarifications, add more precise glossary terms, metric descriptions, and samples to the context model.

Making It Work Reliably in Your Own Business

The data is already in SLS, but the knowledge needed to understand it is scattered. Field meanings live in the code, status explanations live in stray documents; dig around and you can still find both. The other two are harder to get. Metric definitions live in business people's heads, and knowing which logs to check for which problem rests on the experience of a few experts. Newcomers keep asking, business people keep depending on engineering, and an AI that does not know the real metric definition can reach a wrong conclusion.

The semantic layer and DataAgent move all this knowledge into one place. Field meanings, metric definitions, and query paths are pinned down; business people can ask in natural language, and the DataAgent understands the question, picks data sources, runs queries, and organizes the evidence, with results you can check. The time newcomers spend asking and engineers spend explaining goes away. You do not have to do it all at once, either: start by getting one question to work end to end. Find the high-frequency question your team answers every day, pick the relevant LogStores, confirm the fields, glossary terms, and metric definitions, add typical phrasings and query samples, and let the DataAgent go from question to conclusion on its own. Once that proves out, extend to more questions, more projects, and more business roles. So we recommend the following:

1. Start with high-frequency, high-value scenarios. Pin down the metric definitions for questions like top-up not credited and duplicate charge first; they make a verifiable baseline easy to establish.

2. Get definitions right before pursuing coverage. Confirm these one by one first: amount units, status enums, time fields, unique identifiers, and metric formulas. If the definitions are wrong, the more questions you cover, the further off you get.

3. Build human review into the process. Set clear boundaries for actions such as refunds, redelivery, bans, and unbans: the DataAgent produces evidence and recommendations, and a person reviews them before execution.

A few limits are worth stating plainly as well:

Limit	Details
Centralized services (context model, DataAgent)	Available only in the cn-beijing central region
Source model	Deployed per region; supported in cn-beijing, cn-shanghai, cn-hangzhou, cn-chengdu, and cn-heyuan
Number of context models/DataAgents	50 each per account and per user by default; you can request an increase

Conclusion: Start with One High-Frequency Question

Back to that opening ticket. With the metric definitions configured, the next time a player asks about a top-up that never arrived, Lin no longer switches back and forth among orders, payments, delivery, and inventory. She hands the question to the DataAgent, gets evidence from three logs plus a draft reply, checks it once, and sends it to the player.

The logs in the demo are synthetic so the case is reproducible. The 13 scenarios are modeled on common customer complaints, not on any one studio's real data. Outside gaming, after-sales, risk control, and O&M have the same kind of question, the kind that takes several logs to answer, and the approach is the same. With your own logs, start with the question your team answers over and over: get the fields, glossary terms, and metric definitions right, make that one work, then decide whether to add more. Questions like this used to be answerable only by the people who understood the logs. Once the first one gets answered, the people who can answer are no longer just those few.

Try the new semantic capabilities in SLS today.

The AI Agent Era Doesn't Need More Data — It Needs a Semantic Layer

ObservabilityGuy — Thu, 29 Oct 2026 02:04:57 +0000

This article introduces UnifiedModel, an open-source semantic layer that organizes enterprise data into a queryable object graph for AI agents.

TL;DR

In the AI Agent era, what's really missing isn't more data — it's an open map of the system.

UModel is an object-graph semantic runtime for enterprise AI. It describes the enterprise world as objects and relationships, and makes those descriptions queryable, verifiable, and callable from agent code. It is not another observability tool, Configuration Management Database (CMDB), or knowledge graph. It sits above them and organizes the facts they already hold into a unified object graph. Taken together, these capabilities mean enterprise data is no longer whatever each system happens to record on its own: it is organized, queried, verified, and invoked around objects.

UnifiedModel is an open source digital twin semantic layer built for Agents. It organizes enterprise assets, runtime data, and system relationships into a queryable object graph. In controlled DataAgentBench experiments, all four flagship models gained 10-20 percentage points once the semantic layer was in place, with GLM-5.2 reaching 50.2% pass@1. Instead of facing isolated data fragments, the Agent reads metrics per object, walks relationships to localize root causes, and actually understands the system it is looking at.

The Problem Isn't a Weak Model — It's the Missing System Structure

Agents have improved fast over the past year. They write code, call tools, plan across multiple steps, and search back and forth across logs, metrics, traces, and change tickets. But drop one into a real production system and ask, "What is actually wrong with payment-gateway right now?" and it will still happily produce an answer that sounds authoritative and is quietly dangerous.

The cause isn't wrong information from any single tool. CPU at 88% is correct. P99 at 2150 ms is correct. That one slow trace is correct too. The problem is that all of these are only symptoms. The Agent has plenty of fragments, but it doesn't know which object each fragment belongs to, how the objects connect, who depends on whom, or which upstream configuration just changed.

This is the engineering version of the blind men and the elephant. Metrics, logs, traces, tickets, and code repositories each touch a real piece of the animal. No single structure tells the Agent what the whole elephant looks like.

The usual first reaction is to give the Agent more capability:

Swap in a stronger model so it can reason longer.
Give it a longer context and push in more logs and documents.
Add retrieval-augmented generation (RAG) so it can retrieve more fragments.
Add memory so it keeps past conversations and lessons.
Wire up more Model Context Protocol (MCP) tools, or even go multi-agent.

Every one of these makes sense, but they add capability and data. They don't address system structure. Connecting to every system is not the same as understanding it. Having every fragment in context is not the same as knowing how they relate.

What's missing is a semantic layer — a structure that captures the objects, relationships, fields, evidence, and action context of the real system.

What a Semantic Layer Really Is: An Externalized World Model

To an Agent, a complex system should not be a pile of tables, APIs, and log files. It should be a queryable world model:

Which objects exist: services, hosts, databases, configurations, deployments, teams, alerts.
How objects connect: services call services, services run on hosts, deployments affect services, configurations act on traces.
Which fields each object class carries: status, owner, service-level objective (SLO), lifecycle, primary key, labels.
Which observability evidence hangs off which object: metrics, logs, traces, events, Runbooks.
Which actions are allowed under which conditions: rollback, rate limiting, scale-out, retry tuning.

In information science, this idea is rooted in ontology — the study of what kinds of things exist in a domain and how they relate. UnifiedModel brings that thinking into engineering systems, describing a digital twin object graph with a very small set of unified primitives.

A semantic layer does not copy all your data into a new database, nor chop documents into segments for a vector store. Think of it as a system map for the Agent. The Agent can still call Prometheus, Simple Log Service (SLS), Elasticsearch (ES), MySQL, Kubernetes, and the CMDB — but it starts from objects rather than from a database engine, follows relationships to the evidence, and then generates an executable query or action plan.

UnifiedModel: Design and Capabilities

The Minimal Primitives of UnifiedModel: Set + Link + Field

UnifiedModel reduces the object graph to three core primitives:

Primitive	Meaning	Typical types
Set	A node — a class of objects or a class of data collections	EntitySet, MetricSet, LogSet, TraceSet, EventSet, RunbookSet
Link	An edge — a relationship among objects, data, and storage	EntitySetLink, DataLink, StorageLink
Field	Field semantics and constraints	Type, unit, semantic role, primary key, mapping, display form

Combine the three and you can express the full entity-data-storage chain:

EntitySet ──DataLink──> DataSet ──StorageLink──> Storage
│
└──EntitySetLink──> EntitySet

Here's the smallest example from an observability scenario:

platform.service is an EntitySet representing the service entity class.
platform.host is another EntitySet, representing hosts.
Services and hosts are joined by a runs_on relationship — that's an EntitySetLink.
A service's latency, error rate, and QPS hang off a MetricSet — that's a DataLink.
That MetricSet physically lives in Prometheus or SLS — that's a StorageLink.
latency_p99_ms is a Field. It carries more than a column name: type, unit, semantics, and query mapping.

What matters here isn't that you drew a graph. It's that the graph can be queried and validated — and discovered and called by an Agent.

Classes and Entities: Define the Model Once, Write to the Runtime Continuously

Two levels in the object graph are easy to conflate: classes and individuals.

Classes are the definition layer — what fields a "service" object has, what its primary key is, which datasets it can associate with, and which entities it can relate to. In UnifiedModel, EntitySet, DataSet, Link, and Field describe this layer. It corresponds to the TBox in ontology.

Individuals are runtime data.checkout-service, payment-gateway, and catalog-api are concrete service entities; payment-gateway calls risk-control is a concrete relationship. They are written, updated, and expired continuously as the system runs. This layer corresponds to the ABox.

A simplified definition layer might look like this:

kind: entity_set
domain: platform
name: platform.service
pk:
  - id
fields:
  - name: id
    type: string
    semantic_role: entity_id
  - name: status
    type: string
  - name: latency_p99_ms
    type: double
    unit: ms

The runtime then writes entities and relationships continuously:

{
  "entity_set": "platform@entity_set@platform.service",
  "entity_id": "payment-gateway",
  "fields": {
    "status": "degraded",
    "latency_p99_ms": 2150
  }
}

This split matters. Classes tell the Agent which types and methods exist in the world; individuals show it what is happening in that world right now. The first is stable, the second dynamic, and only together do they make a live object graph.

Runtime: Turning the Object Graph into One Query Surface for Humans and Agents

UnifiedModel Runtime isn't another isolated platform. It adds a semantic runtime on top of the systems you already run.

Top to bottom, it has four layers:

Access layer: Web UI, CLI, Skills, MCP Gateway. Humans query it; agents query it too.
Runtime services: Workspace, definition validation, entity writes, relationship writes, Query Service, Agent Gateway.
Graph abstraction layer: a single encapsulation of objects, relationships, methods, datasets, and storage mappings.
Storage layer: memory, files, graph databases, plus external data sources such as Prometheus, SLS, ES, and MySQL.

One Query Surface: SPL Covers Definitions, Entities, Topology, and Data Plans

An object graph has to be queryable in a stable way, or it's just documentation. UnifiedModel covers several viewpoints with a single SPL query surface:

Query surface	Purpose
`.umodel`	Query model definitions and metadata
`.entity`	Query specific entities
`.entity_set`	Call methods on an entity class — list datasets, generate a metric plan, generate a log plan
`.topo`	Query topology relationships and neighbors
`.runbook_set`	Query the O&M knowledge and recommended actions attached to an object

An entity query can be very direct:

.entity with(domain='platform', name='platform.service')
| project id, display_name, status, owner
| limit 20

A topology query starts from an object and looks at its neighbors:

.topo
| graph-call getNeighborNodes(
    'platform@entity_set@platform.service',
    'payment-gateway',
    2
  )

Method calls matter even more. An Agent can first ask an EntitySet which methods it has, then call one by signature. A service object, for example, can expose get_metrics, get_logs, and list_data_set. The call doesn't dump full data into the model — it returns an executable plan:

.entity_set with(
  domain='platform',
  name='platform.service',
  ids=['payment-gateway']
)
| entity-call get_metrics('latency_p99_ms', '5m')

The result can be PromQL, an SLS query, ES DSL, or a structured query plan. The Agent executes the plan afterward, which cuts hallucination and reduces the risk of querying the underlying data sources incorrectly.

AI-Friendly: Self-Describing, Progressive Disclosure, MCP, and Skills

"Agent-friendly" doesn't mean stuffing docs into the model. It means making the runtime discoverable.

UnifiedModel does three things for Agents.

First, self-description and progressive disclosure. The Agent doesn't need the full schema up front. It can call __list_method__ to learn which methods the current EntitySet offers, what the parameters are, and what comes back, then decide the next step. Context doesn't fill up all at once, and the Agent never has to guess.

Second, the MCP Gateway. A standard protocol exposes query, explanation, example, and validation capabilities to any Agent. Read tools are available by default; write tools are off by default or require explicit authorization; resources expose metadata only; and every access goes through the Query Service.

Third, Skills. Common capabilities are packaged as loadable skills — object graph queries, root-cause analysis (RCA), impact analysis. Claude Code, Cursor, and Codex then share the same semantic capabilities instead of each Agent writing its own adapter.

The design principle behind all three: discover before calling, get a plan before executing, go through the semantic layer before touching the underlying tools.

UnifiedModel in Practice

Onboarding: From a Real System to a Queryable Object Graph

Onboarding a real system into UnifiedModel takes three steps.

Step 1: model it. Write the model package YAML and define entities, fields, relationships, datasets, and storage mappings. Then run the validation tool to confirm the model is legal.

umctl umodel validate <workspace> --file model-pack.yaml
umctl umodel import <workspace> --file model-pack.yaml

Step 2: write runtime individuals. Entities, relationships, lifecycle, status, and observability evidence stream in continuously. Entities can come from the CMDB, Kubernetes, OpenTelemetry Resource, the Service Catalog, or your deployment system; relationships can come from traces, configuration systems, code dependencies, or manual registration.

umctl entity write <workspace> --file entities.json
umctl topo write <workspace> --file relations.json

Step 3: query the graph. Humans query with .entity and .topo, and Explorer visualizes the result automatically; Agents discover methods through MCP and Skills, generate query plans, and run troubleshooting flows.

Today you write models and data by hand. Automatic registration, OTel mapping, service discovery, and code scanning will keep lowering that cost. The key point: modeling isn't a big-bang project. It grows incrementally — one scenario, one domain, one service path at a time.

Case Study 1: Read Metrics per Object Instead of Hand-Writing PromQL

Back to the question from the talk: someone asks the Agent, "How is payment-gateway doing?"

Without a semantic layer, the Agent has to work out for itself:

Which service object does payment-gateway map to?
Which system holds the metrics?
What is the metric name for P99?
Should the label be service, service_id, or app?
Is the unit seconds or milliseconds?

Get any of these wrong and the result looks reasonable but is useless.

With an object graph, the flow becomes:

Use .entity to locate platform.service/payment-gateway.
Follow DataLinkto the MetricSet and LogSet it's associated with.
Call get_metrics to generate a query plan, with the entity id, window, unit conversion, and metric semantics filled in automatically.

The Agent can then answer:

Status: degraded
QPS: 4200
Error rate: 14.8%
P99: 2150 ms
Replicas: 5/5

The numbers aren't the point. The point is that they were retrieved per object. The Agent didn't need to master a PromQL dialect or guess a label. The object graph turned "that service" into a precise query.

Case Study 2: RCA Isn't Guessing the Latest Release — It's Walking the Relationship Chain

The second scenario is root cause localization: why did payment-gateway blow its P99 SLO?

The object graph gives you a traversable chain of relationships:

Flash Sale 大促
  └─triggers─> cfg-checkout-retry
      └─affects─> checkout-service
          └─calls─> payment-gateway

This chain connects a business event, a configuration change, an upstream service, and the affected service. The Agent no longer just sees that payment-gateway got slow. It walks up the relationships and pins down the cause: the sales promotion triggered the upstream checkout retry configuration, which amplified traffic.

It can also rule out red herrings. A payment-gateway v3.2.1 deployment 12 hours earlier looks suspicious, but the relationships and evidence show it only changed the log format, which doesn't explain a P99 SLO breach. So the Agent doesn't mechanically blame the latest release.

4000 QPS × 3.5x sales promotion traffic × (5 / 2) retry amplification = 35000 QPS 
35000 / 4000 = 8.75x

The conclusion becomes:

The root cause isn't that deployment. It's the upstream retry count going from 2 to 5 and then meeting promotion traffic.
The overload is roughly 8.75× the original capacity.
The recommended action is to roll retries back to 2 and add rate limiting.

Closing: Model the Real World First, Then Organize the Data

In the agent era, what we need to add isn't more fragments. It's a structure that organizes them. Data is just the symptom; objects and relationships are the system structure.

UnifiedModel's core bet is this: use Set + Link + Field to build a real system into an object graph that can be queried, traversed, and executed against; hand that graph to humans and Agents alike through SPL, the Runtime, MCP, and Skills; then close the loop across data, knowledge, and action so the Agent doesn't just see metrics — it understands the system along its real relationships.

One line: model the real world first, then organize the data. The semantic layer is the infrastructure of the Agent era.

Inside STAROps: An Intelligent Log Inspection Architecture for Surfacing Unknown Anomalies

ObservabilityGuy — Fri, 23 Oct 2026 10:21:42 +0000

This article introduces STAROps' AI-driven log inspection architecture for proactively surfacing unknown anomalies.

For most people, log analysis starts with an alert. An API returns errors, so you search for the request ID. A pod restarts, so you look for the exception stack it printed before exiting. A user reports a failed payment, so you take the account and order number and trace the call path. By that point the problem has already happened, and logs help you reconstruct the scene.

But logs hold far more than the scene of a failure. Application logs record how business logic executes, access logs reflect traffic and service quality, audit logs record who did what and when, and Kubernetes Events record changes in resource state. Developers use them to troubleshoot, security teams use them to audit, and operations teams can use them to spot anomalies in payments, inventory, devices, or user behavior.

What gets overlooked are the changes that have not yet triggered an alert and that nobody has thought to search for: a new version starts sporadically surfacing exception stacks no one has seen before; the overall error rate looks fine, yet one region keeps degrading; a class of high-risk operation appears for the first time outside its authorized window.

On their own, these signals look unremarkable. Left alone, they can also become the starting point of the next large-scale outage.

What Intelligent Log Inspection does is take a look when nobody is asking, and tell us: something here has changed in a way it never did before, and it is worth investigating now.

Rules Catch Known Problems; Inspections Look for Unknown Changes

Rules and alerts are useful. We know which error code means failure and which threshold needs attention, so a single SQL query or alerting rule can monitor it indefinitely. These problems are known knowns: we know the problem, and we know how to find it.

Not every problem in a production environment is that clear.

Sometimes we know something is wrong but not where it is concentrated. The error rate climbs from 1% to 10%, and for the moment we cannot say which service, API, version, or tenant is responsible. That is a known unknown.

Other knowledge lives only in engineers' heads. BizException(code=USER_CANCEL) is a normal business cancellation. A 404 on /healthz can be ignored. ci-bot rotating keys in the small hours is an authorized operation. Without that background, the system keeps reporting all of it as problems. Think of this as the unknown known: the team already knows, the system does not.

The hardest case is the unknown unknown: nobody has defined the problem in advance, and nobody knows what to search for. It might be a new log pattern, a combination of fields that has never occurred before, or a long-tail dimension that has just drifted away from normal.

Intelligent Log Inspection cannot promise to find every unknown problem. What it can do is widen the search: alongside the checks already defined, it keeps looking at new patterns, shifts in distribution, numeric tails, and contradictory signals, and surfaces candidates worth investigating as early as possible.

STAROps' Intelligent Log Inspection

When you hand log inspection to an Agent, the hard part is finding anomalies reliably in a huge volume of data while keeping enough evidence for a human to review. Reading every line is impossible, and sampling a handful of lines is unreliable: low-frequency anomalies slip through, and a few dramatic samples can skew the judgment.

STAROps splits the work into several connected stages:

Simple Log Service (SLS) performs broad analysis on the data side first
The LogReduce and dimension drill-down AI operators compress the search space
The Agent plans its investigation path dynamically from the result of each step
Confirmed problems enter a watch list for ongoing tracking
Correlating code, metrics, traces, Skill (a domain skill), and UModel (UnifiedModel, a unified observability modeling framework) extends the investigation across entities and data sources

How SLS Operators and the Agent Divide the Work

In real production, one physical limit cannot be avoided: model context is finite, so you cannot load every log line from an inspection window into it. Naive sampling has its own problems — a low-frequency anomaly may never be sampled, and a few unusual lines can lead the Agent to misjudge the whole picture. Pre-filtering with keywords and fixed SQL only covers the errors you already know about.

SLS provides data-side capabilities such as query and analysis, LogReduce, and dimension drill-down, compressing large volumes of raw logs into structured results a human can review. The STAROps Agent does not replace these capabilities. It generates a plan from the inspection goal, picks time windows, fields, and operators, reviews the results, and then decides whether to drill down further, switch data sources, or stop.

LogReduce extracts stable constants and dynamic variables from indexed text, groups similar logs into patterns (log templates), and then compares pattern counts and variable distributions across windows to find patterns that are new, missing, or spiking. Clustering can also group by fields such as service or module, so the current window can be compared directly against the same period yesterday, the same period last week, or the window before a release.

For example, a mixed set of logs may contain both connection pool exhaustion and downstream call timeouts:

1WARN  [worker-17] Connection pool exhausted, tenant=shop-102, wait=812ms2ERROR [gateway] Downstream timeout, provider=bank-a, api=/pay/confirm, elapsed=3201ms3WARN  [worker-08] Connection pool exhausted, tenant=shop-518, wait=943ms45Pattern A: <*> WARN [<*>] Connection pool exhausted, tenant=<*>, wait=<*>ms6Pattern B: <*> ERROR [gateway] Downstream timeout, provider=<*>, api=<*>, elapsed=<*>ms7

In a real inspection, what the Agent sees is the overall pattern distribution for the analysis window: how often each pattern occurs now and historically, which ones are new or spiking, and how variable values are distributed. "Connection pool exhausted" does not need to be predefined as a keyword — as long as it forms a new log pattern, the inspection still has a chance to surface it.

Dimension drill-down builds a current window and a historical baseline, then searches dimensions such as service, region，version, and operation — and their combinations — for the ranges with the most significant deviation and the largest contribution, reporting the actual value, the baseline value, and the difference.

This works better than running group by on one field after another. The dimension with the most traffic is not necessarily the one causing the anomaly, and a single field may show nothing at all; the real anomaly may appear only in a combination such as project-a + logstore-x. These two operators detect change and narrow scope; neither is a root cause verdict on its own.

Dynamic Planning and Precise Drill-Down

A log inspection does not hard-code one investigation flow for every system. The first time a task runs, STAROps generates an initial plan from the log structure, the available history, and the inspection goal. Stable checks then repeat on that plan, while the live investigation is decided by the Agent from the evidence in the current round.

One investigation might start with a rising failure ratio, confirm the denominator and the historical baseline, and then call the dimension drill-down AI operator to find the range contributing most. Another might start with a new pattern and then check whether it is concentrated in one version, region, or upstream dependency. When fields cannot explain the change, the Agent turns to clustering the message body. When it suspects a release, it checks versions and changes. When it needs to judge user impact, it goes to requests, traces, or business entities.

The first layer of candidates an operator returns is only an entry point. The Agent verifies that the current window and the historical baseline are comparable, re-queries the actual difference for the leading candidates, and then uses the high-contribution ranges as conditions for the next round of queries. It might first pin down project-a, then compare LogStores and versions only within that project. Each step's result determines what the next step queries, the scope narrows layer by layer, and the query chain is preserved in the Finding (a continuously tracked issue record).

Each inspection round also sets aside room for one bounded piece of autonomous exploration. Even when the existing checks report nothing, the Agent picks one direction to keep looking — unexplained logs, new fields, pattern changes, numeric tails, or contradictory signals. It stops when the evidence already explains the problem, when further queries offer no clear gain, or when the data really is insufficient, and it states the conclusion or the data gap explicitly.

Running Findings Continuously and Improving over Time

A problem found in one inspection should not end up as an isolated notification. STAROps creates a Finding that can be tracked over time, holding a stable identity, its current evidence, the scope of impact, and status changes. When the same class of problem appears again, the system links it to the existing Finding and distinguishes whether it is ongoing, getting worse, recurring, or shrinking in impact.

Notification policy also follows how a Finding changes. A new anomaly, a worsening problem, a recurrence, or a clear expansion in scope can trigger a DingTalk or Lark message; unchanged problems stay in the report instead of interrupting the on-call engineer every cycle.

To start an inspection task, you only need to state the business scope, the run cycle, the priorities, and the notification method. Everything after that improves from real feedback. If BizException(code=USER_CANCEL) is a normal business cancellation, tell the Agent. If health checks, stress test traffic, or authorized operations do not need attention, add the applicable scope. STAROps captures confirmed business semantics and the reasoning behind each judgment as Learning, and uses them to adjust later inspections.

Once a human confirms an unknown problem, it also becomes reusable experience. If a given exception stack has been confirmed to relate to connection pool configuration, a later inspection can reuse the previous drill-down path and review method, first checking whether it is the same problem and then whether it is getting worse. When log fields, business scope, or priorities change, the inspection plan is re-evaluated as well; only validated adjustments become stable checks.

Custom Probes and Extensions

Beyond the checks the system generates, you can add your own probe requirements — for example, watching only fund discrepancies inside the settlement window, checking high-risk operations outside authorized hours, or adding dedicated logic for a particular class of business error code. Custom probes join the inspection plan and run alongside LogReduce, dimension drill-down, and the Agent's own exploration.

Skill and Runbook (a standardized troubleshooting manual) add domain knowledge and investigation methods. After a new Java exception stack shows up, the code MCP (Model Context Protocol) server can retrieve the corresponding file, commits, and published version. When a database connection timeout appears, a dedicated troubleshooting Skill can go on to check the connection pool, slow SQL, and database metrics. Enterprises can also write their own business error codes, authorization windows, handling standards, and system knowledge into their own Skills.

When a single LogStore cannot explain a problem, the Agent uses UModel to locate the related entities and where their data lives, then uses Skills to query metrics, traces, changes, or code. UModel organizes entities such as services, pods, nodes, Agent Threads, and Skills together with their LogSets, MetricSets, TraceSets, and actual storage in one relationship graph. The raw data does not have to be moved into a single store; UModel supplies the object relationships, the data locations, and the field semantics.

That keeps a log inspection from stopping at a single LogStore. The general inspection finds the change first, UModel decides which entities and data the investigation can follow, domain Skills decide how each class of object should be queried, and the code MCP server and other tools gather the evidence — forming a chain of evidence: operators surface candidates, the Agent reviews them step by step, and UModel gathers evidence across sources.

How STAROps Uses Intelligent Log Inspection to Improve Business Reliability

For STAROps, healthy service metrics do not mean the business is reliable. A single task passes through the ingress gateway, authentication, AgentRuntime, Sandbox, tool services, and more; a problem at any one of them can make the task fail, slow it down, or return an incomplete result. We also need to keep watching whether tasks complete, whether performance has shifted, and which Agents, users, and dependency paths the errors cluster around.

So we put STAROps' own Agent Thread logs under routine intelligent inspection. The inspection runs hourly and continuously analyzes request volume, completion status, latency, Agent distribution, and user distribution. When it finds a new error pattern or degradation in a key metric, it follows the Agent and UModel to LLM, tool, and gateway data to confirm what the problem affected and where it occurred.

Starting Inspection from a Critical Business Log

Open the target LogStore in the SLS console and go to the intelligent inspection page to create an inspection task. The inspection requirements can stay generic at first, letting the Agent build an initial plan from the characteristics of the logs; if you already have specific business concerns, you can add them at creation time.

After the first run, the inspection found that log record volume had grown 147% over the previous window, with the increase concentrated in one batch task execution period. The report also included the current window, the comparison window, the actual queries, and a note on uncertainty. Based on the available evidence, the growth looked more like a change in system scheduling behavior than anything that directly affected service health.

The result is well-grounded, but still not closely enough aligned with the STAROps business. Record counts and generic fields alone make it hard to tell which entry point and which users the change came from, or whether it affected real tasks.

Adding Business Semantics to the Inspection Plan

We told STAROps directly: skill indicates which entry-point Skill handled the request, employeeId indicates which user the request came from, and both fields deserve close attention. The Agent updated the inspection plan accordingly, adding Skill request distribution, Skill completion status, Skill performance, user request distribution, and a cross analysis of users against Skills — taking the checklist from 6 items to 11. The next inspection round runs these new checks automatically.

The updated inspection showed its value quickly. In one window, overall P90 latency rose from 94 seconds to 171 seconds, which looked like clear performance degradation. When the Agent broke the numbers down by Skill, however, no individual Skill's P90 had degraded; what actually changed was traffic composition, as Skills with a higher share of slow requests accounted for more of the current window. The final report classified it as a change in traffic composition rather than a drop in service performance.

As you can see, user feedback changed how the Agent read the logs, and with it the subsequent checks and drill-down paths. Confirmed business semantics carry into later inspection plans, so nothing has to be explained again each round.

Running Continuously, Notifying on Problems

The inspection task needs to run hourly, but notifications do not need to go out every hour. We added problem trigger conditions to the task: an error rate above 5%, or a new significant finding. STAROps then breaks that down itself: it pushes to the "STAROps Alerts" DingTalk bot only when a new error type appears, when a Skill's non-completion rate is markedly high, when P90 clearly degrades, or when errors concentrate in a small number of users.

In a later inspection, the non-completion rate rose from 1.01% in the previous window to 6.2%, crossing the notification threshold. The notification the bot received carried the error distribution by Skill and by user, performance metrics, and an initial evaluation; the problem had already narrowed to context canceled clustered on one Skill path within a specific time period. Normal inspections that trigger nothing still produce a report, but they do not repeatedly interrupt anyone.

Widening the Investigation with Skills and UModel

A single LogStore can tell us what happened in an Agent Thread, but not necessarily why. We went further and linked this inspection to STAROps' own Skills and UModel: an Agent Thread can be correlated to its entry-point Skill, user, LLM calls, tools, and gateway, and each class of entity has its own LogSet, MetricSet, or other data sources.

When Thread logs show an abnormal non-completion rate, the Agent can follow UModel to check the related entities. In one cross-source inspection, the failure ratio on Threads was still 0%, while the correlated LLM call data showed clustered 429 rate limiting; some gateway SSE statistics and /metrics requests had already been identified as known noise. The final report split the evidence into P1 normal, P2 abnormal, and P3 known noise, rather than lumping every red metric together.

The STAROps reliability inspection began with a single LogStore. The first round was not closely enough aligned with the business, so we added business semantics; the check plan updated accordingly, and the notification policy moved from "send every round" to "notify on problems." Skills and UModel then extended the investigation to more entities and data sources. As the system and the business keep changing, this inspection will keep being calibrated, becoming a continuously running reliability safeguard.

Conclusion

We used to open the logs only after something had gone wrong. Intelligent Log Inspection moves that one step earlier: the Agent continuously watches for pattern and distribution changes in logs, and looks for signals worth investigating even when nobody is asking and no rule exists yet.

A useful inspection task does not need every rule written on day one. It can start from one LogStore and a generic plan, then use its own results to fill in business semantics, adjust the check plan and the notification policy, and finally reach more entities and data sources through Skills and UModel.

Round after round of inspections and feedback turn unknown problems into reusable experience, retire known noise from notifications, and keep tracking the problems that genuinely changed. Problems surface earlier, investigations narrow faster, and every notification that does go out is worth a closer look.

Handling Checkout Peaks with Confidence: How Lemeng Built AIOps for Retail Chains

ObservabilityGuy — Wed, 21 Oct 2026 03:26:15 +0000

This article introduces how Lemeng uses Alibaba Cloud's STAROps to build a proactive AIOps system for retail operations.

Background

When hundreds of thousands of store terminals fire checkout and payment requests at the same time during business peaks, a single moment of jitter anywhere on the path turns into queues at the counter and failed transactions. For Lemeng, a digital services provider for the food retail industry, O&M is not a back-office support function but a front-line business activity that directly shapes the in-store experience and the brand's reputation. Absorbing peak traffic from a vast store network while keeping its multi-tenant SaaS stable through frequent release cycles is the challenge the company has to meet.

Zhejiang Lemeng Information Technology Co., Ltd. ("Lemeng") is a software and information technology service provider focused on digitalization for the food retail industry. Under the Lemeng brand, the company delivers integrated retail digital-intelligence solutions for supermarkets, chain stores and trade distributors, with a product matrix that covers the full range of store operations: a retail POS suite in single-store, chain, supermarket and overseas editions, chain-store ERP, trade distribution and inventory management, a warehouse management system (WMS), a fund and accounting system (aggregated payment and split settlement), omni-channel membership CRM, and a WeChat mini-program new-retail mall, plus extended capabilities such as fresh-produce delivery, visual merchandising and AI smart scales. Its industry solutions cover twelve verticals — supermarkets, snacks, fruit, fresh produce, cooked food, roasted nuts, bakery, tea drinks, catering, meat, alcoholic beverages and trade distribution. To date the company serves more than 200,000 stores in more than 200 cities nationwide, has been chosen by more than 15,000 retail brands and more than 100 leading regional enterprises, and provides 24/7 human support.

Business Challenges

Lemeng's business model makes its O&M far more complex than that of a typical SaaS provider. Huge numbers of store terminals generate checkout and payment requests all at once during business peaks; the product line spans checkout, ERP, WMS, payment and split settlement, mini-programs and more; and observability data sits in different technology stacks. As business lines and store numbers keep growing, that complexity has climbed to a new order of magnitude.

Fragmented alert governance, with real anomalies buried in storms

Checkout SaaS, chain ERP, WMS and payment accounting each relied on a different observability system — Simple Log Service (SLS), Prometheus metrics, application performance management (APM) and cloud service monitoring — so alert rules had to be configured and maintained in separate consoles. As the business expanded, rule counts ballooned. Misconfigurations and omissions were easy to introduce and hard to detect, and traffic peaks during sales promotions and holidays set off alert storms in which engineers on duty struggled to pick out the issues that genuinely needed action.

Observability data hard to produce, hard for the business to consume

Reporting needs across internal R&D, customer success and regional operations teams vary widely, yet all of them matter: API success rates, latency distribution and resource load on the system side; store online rate, checkout transaction volume, payment success rate and split-settlement timeliness on the business side. Building such dashboards usually meant writing complex queries across several kinds of observability data — slow and error-prone. For business users, query syntax was another barrier. Observability stayed in the hands of a few experts, and response times could not keep pace with the business.

Manual monitoring before failures, manual digging after them

On the prevention side, engineers logged in and checked the container clusters and cloud service resources behind the multi-tenant business item by item. There was no reusable inspection SOP and no systematic way to spot resource bottlenecks and latent risks before a failure. On the diagnosis side, a single failed checkout in a store could originate in the store network, the POS terminal, the SaaS application, the database or the payment channel. Cross-system troubleshooting meant switching between platforms again and again and correlating data by hand, and the process leaned heavily on the experience of senior engineers. Every additional minute spent diagnosing the payment path directly affects store operations and financial compliance.

What Lemeng needed was not another monitoring tool, but an artificial intelligence for IT operations (AIOps) system that would unify the alert entry point, lower the barrier to consuming data, move the line of defense forward, and let AI handle cross-system diagnosis.

Solution: Four Layers of AIOps on a Unified Data Foundation

After evaluation, Lemeng chose Alibaba Cloud's Cloud Monitor 2.0 (CMS) together with STAROps, Alibaba Cloud's full-domain AIOps platform, as the core of this upgrade. Lemeng had already accumulated logs, metrics, traces and other observability data on the Alibaba Cloud observability stack, and STAROps uses the UModel observability model to unify them into an end-to-end digital twin that alert governance, automated inspection, root cause analysis and other capabilities then consume. The platform needs no separate activation — it is enabled by default through the STAROps entry point in the CMS 2.0 or SLS console — so it slots smoothly into Lemeng's existing observability setup. The upgrade delivered four layers of capability to address the three challenges.

(1) Unified Alert Governance: Consolidating Alerts from Multiple Systems into a Single Entry Point

With the Alert Management Center in CMS 2.0, Lemeng brought alert rules that had been scattered across SLS, Prometheus metrics, application monitoring and cloud service monitoring into one platform, so engineers no longer jump between systems to configure them separately. Lemeng also brought in the alert rule governance Skill built into STAROps: it inspects existing alert rules automatically and issues governance recommendations, flagging duplicate rules, rules with unsound thresholds and stale rules that have not fired in a long time. Combined with alert and event analysis, it consolidates alerts during peak periods, significantly reducing the drain of alert storms on on-duty efficiency.

(2) Natural-Language Observability: Generating Dashboards with a Single Sentence

Using the natural-language-to-SQL (NL2SQL) capability in STAROps, Lemeng's developers and operations staff describe what they want to observe in plain language, and the platform generates the matching queries and dashboards. Typical uses include checking the trend in checkout API success rates for stores in a given region; comparing latency distribution across payment channels during sales promotions; tallying delay distribution and failure details for split-settlement jobs; and summarizing alert handling and acknowledgment status by line of business. Producing a dashboard has collapsed from a multi-step process — file a request, wait for scheduling, write the query — into a single conversation, so observability data arrives quickly and business roles can use it directly.

(3) Scheduled Intelligent Inspection: Delegating Repetitive Checks to Digital Employees

Through the STAROps long-running task (Mission) mechanism, Lemeng lets digital employees (Agents) run inspections, analyses and change operations on their own, either on a schedule or event-driven, turning repetitive checks that once depended on people into reliable automated flows. The inspection scenarios in production include container cluster health inspection, application health inspection, cloud service resource health inspection, and quality checks on the payment and split-settlement data pipelines. Inspection results are captured as structured reports and can be compared against historical results, so resource bottlenecks and configuration drift surface before they turn into failures. For high-risk actions such as scaling and configuration changes, the human-in-the-loop (HIL) mechanism built into Mission prompts for manual confirmation, raising the level of automation while holding the safety line on changes. This layer upgrades the line of defense from manual item-by-item checking to automated protection.

(4) Dedicated O&M Digital Employee: Rapidly Narrowing Root Causes Across Systems

By combining default rules, Skills and Model Context Protocol (MCP) tools, Lemeng customized the general-purpose digital employee into a dedicated SRE agent fitted to its own business, configuring that agent's responsibilities, permissions and callable tools along its own system boundaries. When an alert fires on the checkout or payment path, the digital employee can trigger rapid root cause analysis from the alert context or from an engineer's natural-language question: using the O&M digital twin built with UModel, it correlates applications, services, resources, topology, alerts and changes in one view, and with AI analysis operators such as metric anomaly detection, log clustering, trace analysis and change retracing, it performs causal inference across systems. Troubleshooting that once relied on senior engineers' experience becomes reusable automated analysis, which directly addresses the problem of long fault-localization chains.

Value Delivered: Three Upgrades from "Reactive Firefighting" to "Proactive Autonomy"

(1) Alert Governance: From Fragmented Manual Work to Unified Standard Actions

Before, each of the four observability systems had its own alert entry point. Rule configuration depended on individual experience, misconfigurations and omissions went unnoticed, alerts flooded the screen at peak times, and engineers on duty had to sift real anomalies out of the noise.

Now, a unified Alert Management Center plus the rule governance Skill turns adding, reviewing and optimizing rules from manual work driven by individual experience into a standard, tool-supported process. Peak-time alerts are consolidated as needed, and alert storms no longer disrupt on-duty judgment. Alerts have been reduced by 70%.

(2) Data Access: From Request-and-Schedule to a Dashboard in One Conversation

Before, a business observability dashboard went through filing a request, waiting for scheduling and writing complex queries — slow to build and error-prone — and business users who did not know the query syntax could only wait for an expert to get back to them.

Now, with NL2SQL and conversational data retrieval, developers, customer success staff and operations teams get core application observability metrics and store-operations data in natural language, without knowing any query syntax. Observability has spread from a handful of experts to a much wider set of business roles, and response times are now measured in minutes.

(3) O&M Posture: From Reactive Response to Proactive Protection, with AI-Assisted Troubleshooting

Before, resource inspection meant checking items by hand, with no guarantee of coverage or consistency, and fault localization meant switching between platforms and correlating data manually, so mean time to repair (MTTR) was hard to bring down consistently.

Now, with a release every two weeks, scheduled Mission inspections keep producing structured reports and comparisons against history, configuration drift is caught early, and automated inspection provides a steady safety net for release quality. When a failure occurs, the dedicated digital employee triggers rapid root cause analysis straight from an alert or a question, and uses the digital twin topology together with AI analysis operators to narrow the search across systems — cutting the time engineers spend switching between consoles and correlating data by hand. MTTR on the checkout and payment paths has been steadily reduced to a matter of minutes.

Future Outlook: Pushing the Boundary of "Proactive Autonomy" Further

Unified alert governance, natural-language observability, automated inspection and intelligent root cause analysis are now running on the core paths. Next, Lemeng and Alibaba Cloud will deepen the work in three directions.

(1) From "Core-Path Pilot" to "Full Business-Line Coverage"

Lemeng will widen the digital employee's remit from the checkout and payment paths to more systems — chain ERP, WMS, mini-program new retail and others — and reuse the proven alert governance, inspection and root cause analysis capabilities as the standard configuration for every product line, so that AIOps becomes standard practice across all business lines.

(2) From "Technical Metric Observability" to "Business Continuity Measurement"

Building on NL2SQL cross-data-source querying, Lemeng will connect technical metrics such as API success rate and latency with business metrics such as store online rate, checkout transaction volume and split-settlement timeliness, building a measurement framework centered on business continuity so that the value of O&M can be expressed directly in business language.

(3) From "Human-Machine Collaboration" to "Higher-Level Autonomous O&M"

Within the safety frame of the human-in-the-loop mechanism, Lemeng will gradually widen the range of scenarios the digital employee closes out on its own, and keep converting the experience gathered in inspection, diagnosis and change into reusable skill assets — so that O&M know-how evolves from individual capability into an organizational asset.

From Reading Bills to Managing Cloud Spend: How STAROps Closes the FinOps Loop

ObservabilityGuy — Mon, 12 Oct 2026 08:23:26 +0000

This article introduces how STAROps enables continuous, AI-driven cloud cost governance and optimization across the FinOps lifecycle.

Once a company is on the cloud, the bill is usually visible enough. What's missing is being able to read it, chase it down, and act on it in time.

Most teams run into the same set of problems:

Finance spots a cost increase only at month-end, and platform and business teams then work backward to find the cause.
Billing fields are complex, and time windows, cost definitions, and usage units are easy to misalign.
Cost shows up only as a total per product, with no way to drill down to billing items, resources, and usage changes.
High or rising spend does not automatically mean waste, and the bill alone cannot tell you whether resource configuration matches actual load.
Inspections rely on manual triggers, so anomaly detection, root-cause analysis, and follow-up lack any continuous mechanism.

FinOps is not about making the bill look better. It is about making cost changes visible in time, explaining them clearly, and routing them back to the right team.

That also means cost governance cannot stop at bill analysis. Even after you pinpoint the product, billing item, and resource behind a cost increase, you have not yet found the optimization opportunity. You still need to look at CPU, memory, disk, connection, bandwidth, and traffic utilization to judge whether resource configuration matches actual load — and then turn that into actionable recommendations under constraints such as business purpose, capacity requirements, and service level agreement (SLA).

STAROps makes this process easy to start and sustain. Users simply ask a question, and the system handles data checks, definition validation, controlled queries, cost drill-down, resource utilization analysis, and optimization recommendations. If the data is not ready yet, it guides users through syncing the bill first, then picks up the analysis where it left off.

From "Reading the Bill" to Managing Cloud Spend

Put these problems into daily work and the picture is familiar.

At month-end the bill gets exported, finance spots a cost spike, the platform team helps break out the details, and engineering confirms whether a business change is behind it. Along the way, everyone keeps re-aligning time windows, product definitions, tag conventions, resource ownership, and cost baselines.

What STAROps adds is the full chain after "seeing the number": align cost and usage definitions, identify abnormal changes, and drill down to specific billing items and resources. Then bring in resource utilization to judge whether configuration matches load, and finally name what to check first, which direction to optimize, and what still needs confirmation. The results can also feed a recurring inspection that keeps tracking cost, usage, utilization, and optimization progress. In STAROps, a cost analysis looks more like this:

From cost changes to resource utilization and optimization recommendations, STAROps extends a single analysis into a cost-governance action you can keep tracking.

Users still ask in natural language, for example:

"Which business line's cloud cost increased most noticeably this month?"
"Simple Log Service (SLS) cost increased in the last 7 days — is it a usage problem or a change in billing item structure?"
"Which resources have no cost-allocation tag, and how does that affect cost attribution?"
"Which billing items had the usage changes that drove the cost increase?"
"Which high-cost resources have stayed at low utilization for a long time, and what optimization actions should be evaluated first?"
"Inspect cost anomalies, cost allocation dimensions, and usage changes for me every day."

What the system really has to answer goes beyond "how much." The bigger questions are: who is using these resources, why cost changed, whether resource configuration matches actual load, which items deserve a closer look, and what optimization direction to take next.

This chain runs the whole way from cost visibility to continuous governance, and it maps onto the three core FinOps phases.

Aligning with the Three FinOps Phases: Inform, Optimize, Operate

Mapped to the FinOps Foundation's three phases — inform, optimize, operate —, STAROps covers the full journey from data reliability and cost visibility to anomaly detection, optimization recommendations, and recurring governance.

FinOps phase	What the enterprise needs to solve	STAROps cost governance capabilities
Inform: see clearly	Are cost, usage, ownership, and billing cycle definitions timely and accurate?	Data ingestion checks, natural language queries, cost attribution, cost-allocation inspection, multi-dimensional reports
Optimize: find opportunities	Which anomalies, waste, and structural changes are worth acting on?	Anomaly filtering, Top N ranking, usage/cost breakdown, resource drill-down, optimization recommendations
Operate: keep it running	How to turn cost governance into a continuous mechanism	Scheduled inspections, parallel tasks, billing data preparation, task recovery, report archiving

These three phases are not a one-time exercise. The cause of today's cost increase can become tomorrow's inspection focus; a cost-allocation gap found this time feeds into the next governance cycle. Each phase reinforces the others in a loop — which is exactly what the FinOps Foundation means by a continuous process.

Inform: Make Cost Data Reliable and Ownership Clear

Reliable Data, Trustworthy Definitions

The first step in FinOps is cost visibility. But "visible" does not mean staring at a single summary table. What actually matters is knowing what data that table covers, how the numbers are calculated, and whether you can break them out by business dimension.

For an enterprise, the more important question is whether these points can be verified:

Can cost be attributed by product, project, department, account, region, and tag?
Can usage and cost be traced to specific resources and billing items?
Can trends, anomalies, and ownership be compared under one consistent set of definitions?
Can business teams understand the cost they own without having to understand complex billing fields?

In the STAROps cost-analysis pipeline, a natural-language question does not turn straight into an ad-hoc SQL query. The system first generates a structured analysis plan — specifying the time window, comparison baseline, metric definitions, aggregation dimensions, filter conditions, sorting rules, and Top N scope — and only then produces a controlled SQL query or request.

To keep results reliable, the system runs two kinds of validation before execution:

Validation type	Purpose
Semantic validation	Checks whether the time window, cost definition, usage units, currency, and dimension combination fit cost analysis semantics
Structural validation	Checks whether query fields, filter conditions, aggregation methods, sorting, and Top N stay within the controlled scope

The upside: users never have to understand the underlying billing fields or write SQL themselves, yet the system still enforces guardrails on query fields, aggregation methods, and cost definitions.

Cost Attribution

User question: Analyze this month's cloud cost by business project and compare cost distribution and changes across projects.

Agent answer (sample data): The e-commerce platform spent ¥100,000 this month, 50% of the total, up ¥20,000 from last month, with the increase coming mainly from Elastic Compute Service (ECS) and ApsaraDB RDS. The data platform spent ¥60,000, or 30%, roughly flat month over month. Internal R&D spent ¥40,000, or 20%, down ¥10,000. Recommendation: have the e-commerce platform team check the newly added compute and database resources first.

FinOps puts a lot of weight on "whoever uses it, owns it." But to make that principle real, cost first has to land on the right dimension.

For cost attribution, STAROps aggregates and filters mainly on fields already in the bill — Tag, ResourceGroup, CostUnit, OwnerID — and lays the cost distribution out for the user:

Tag-based cost allocation: aggregate by the tag field in the bill to view Top N tags, tags with abnormal increases, and the share of cost carried by values like (no tag) or none.
Resource-group cost allocation: aggregate by ResourceGroup to see which resource groups most cost falls into, and how much of it sits in the default resource group.
Cost units: queries and aggregation by CostUnit are supported at the underlying layer, so you can see how cost is distributed across cost units and how much remains unallocated.
Multi-account ownership: split cost by OwnerID across the primary account and financial sub-accounts to establish cost ownership in multi-account setups.

One clarification: without a corporate tagging standard or mapping table, the system will not assert outright that "the cost center is wrong". More precisely, STAROps first aggregates the ownership fields already present in the bill so users can see cost distribution across tag, resource group, cost unit, and account. Whether a given case is a cost allocation gap, a naming inconsistency, or an organizational mapping issue has to be confirmed against the company's own rules.

Looking at this only at month-end leaves you reacting. Put it into a scheduled inspection, and finance spends less time re-confirming numbers at close, while engineering sees the cost impact of new resources and architecture changes much earlier.

Optimize: Find the Opportunities Worth Acting On

Anomaly Detection and Multi-Dimensional Drill-Down

User question: Which ECS cost increases in the last 7 days deserve the most attention? Please drill down to specific instances and billing items.

Agent answer (sample data): ECS is up ¥10,000 (+10%) versus the previous 7 days. On instance i-example03, the "system disk" billing item rose from ¥1,000 to ¥5,000, an increase of ¥4,000 and 40% of the abnormal ECS increase. On instance i-example04, the "cloud server configuration" billing item rose ¥3,000, or 30% of the abnormal increase. Recommendation: check resource changes on these two instances and their billing items first.

A cost increase is not necessarily waste. Business activity, traffic growth, and architecture migration can all drive legitimate growth. What FinOps really needs to watch are abnormal changes that lack an explanation, follow-up, or owner.

STAROps compares cost and usage between the current period and a baseline period, analyzing change amount, change ratio, usage trends, deductions and discounts, resource concentration, and the breakdown across tags, regions, products, and billing items to flag the anomalies worth attention. It also uses the baseline amount to filter out "small-amount, big-percentage" noise — a resource jumping from ¥1 to ¥10 is a steep rise, but usually matters less than a core product line going from ¥100,000 to ¥120,000.

Once anomalies are identified, the system selects the objects that matter by amount and contribution, then drills down level by level:

Drill-down level	Question to answer
Total bill level	Has overall cost deviated from the historical baseline?
Product level	Which cloud products drove most of the change?
Billing item level	Does the change come from storage, traffic, requests, compute, processing, or deductions?
Resource level	Which resources or instances contributed most of the increase?
Cause level	Is the cause usage growth, a unit price change, reduced deductions, or new resources?

STAROps retains the cost change, anomaly contribution, and relative share at every level, so dozens of pages of line items converge into a handful of actionable checks. When SLS cost rises, for example, the analysis breaks it down further into billing items — storage, index traffic, read/write traffic, and data transformation — and cross-references them with specific projects, LogStores, resources, and tags to determine whether the main driver is usage growth or a shift in cost structure. Cost data becomes truly actionable only once you can pin it down to a specific billing item, resource, and cause.

Resource Utilization Analysis and Optimization Recommendations

User question: Analyze ECS cost for the last 7 days and find the low-utilization instances worth checking first.

Agent answer (sample data): Instance i-example01 cost ¥1,000 over 7 days, with CPU averaging 10% and peaking at 20% and memory averaging 10% and peaking at 20% — a low-utilization instance to check first. Instance i-example02 averages 40% CPU with an 80% peak and 60% memory on average, so it is not a low-utilization candidate for now. Optimization advice: first confirm i-example01's business purpose, peak cycle, SLA, and capacity requirements. If the low load is confirmed to be long-term, evaluate downsizing the instance. If it belongs to a test or staging environment that does not run continuously, evaluate scheduled start/stop as well. Before implementing, confirm data protection, dependencies, and the change window.

STAROps analyzes cost change, resource utilization, and business constraints together, taking users from "a cost anomaly was found" to "here is what to check and how to optimize".

The analysis covers:

Cost insight: pinpoint the main cost changes and the products, billing items, and resources behind them.
Utilization analysis: review trends for CPU, memory, disk, connections, bandwidth, and traffic utilization.

Users can read the results from several angles:

Identify the main cost changes and the resources that need attention.
Review resource utilization trends and load behavior.
Get actionable checks and optimization directions that fit the business scenario.
See what still needs business confirmation, so follow-up can be coordinated.

The results explain cost changes, resource utilization, which items to watch, and what still needs business confirmation.

If no Intelligent Utilization Analysis results are available, the report states the information gap explicitly and explains how to enable Intelligent Utilization Analysis.

If additional authorization is required, the report gives clear instructions so the user can complete the analysis.

Operate: Turn FinOps into a Mechanism That Keeps Running

Recurring Inspections and Long-Running Missions

User question: Every day at 9:00 AM, check the previous day's cost anomalies, cost allocation risks, and usage changes, and generate a report.

Agent answer (sample data): Today's inspection is complete: yesterday's total cost was ¥30,000, +10% versus the previous day; 2 key items were found — system disk cost +20%, and cost without an environment tag +10%. The report lists the main contributing resources and the owners to be confirmed. The next inspection will run tomorrow at 9:00 AM.

FinOps cannot rely on a month-end review. Effective cost governance has to keep running day to day.

For recurring cost inspections, STAROps uses its long-running task capability, Mission, to run multiple inspection tracks in parallel:

Inspection area	What it finds
Cost anomalies	Whether total, product, and resource cost fluctuate abnormally
Cost allocation risks	Whether tag, project, department, and account ownership is missing or abnormal
Deduction changes	Compares changes in deducted usage under resource plans, billed usage, and total usage
Structural changes	Identifies new and disappearing products, plus shifts in cost distribution across dimensions such as region and resource group

These tasks run in parallel and then roll up into a single inspection report. The report retains the analysis billing period, baseline period, current amount, baseline amount, change ratio, thresholds triggered, key dimensions, and suggested drill-down directions.

Mission suits exactly this kind of long-running, staged, resumable work: start with the total bill check, then branch out in parallel across product, resource, cost allocation, and usage; once something significant surfaces, keep drilling down.

If an inspection hits temporarily unavailable data, a Cost Manager sync refresh, or an interrupted drill-down, STAROps records the task state and the branches already completed, then resumes once conditions allow. For a production environment, that is far closer to a genuinely long-running system than "fail and rerun by hand."

Preparing Billing Data

Before any cost analysis can run, the billing data has to be queryable.

When a user starts a cost analysis or inspection, STAROps first confirms that Cost Manager has finished syncing the bill. Cost Manager syncs cloud bills into SLS, and every natural language query, structured SQL query, and recurring inspection afterward runs on that billing data.

If the billing data is not ready, the system prompts the user in the STAROps chat to enable Cost Manager, sync billing data, or update the required fields. Once the data is ready, the original analysis continues and the recurring inspection can be triggered again. What the user gets is not "this isn't available right now" but a path forward.

What STAROps Solves for Users

STAROps is not just another billing report. Think of it as a cost-analysis entry point you can question, drill into, and run continuous inspections from.

Common user problem	What STAROps changes
No idea why cost went up	Drills from the total bill down to products, billing items, resources, and usage changes
No appetite for writing SQL or learning complex billing fields	Starts the analysis in natural language; the system generates controlled queries
Worried that analysis definitions are inconsistent	Validates the time window, metric definitions, dimensions, and SQL structure
Too much detail, key items buried	Surfaces the key items using thresholds, Top N ranking, and drill-down results
Cost governance depends on someone remembering to check	Uses Mission to carry recurring inspections and task recovery
Billing data not ready yet	Guides you through enabling Cost Manager and syncing the bill in the STAROps chat, then returns to the analysis

The bottom line: cost questions no longer wait for month-end. You can ask any time, let the system query automatically, and keep inspections running. Cost spikes surface earlier, root causes get explained faster, and governance actions are captured so nothing falls through the cracks.

As enterprise cloud architectures grow increasingly complex, cost governance will shift from after-the-fact analysis to real-time awareness. That is exactly the path STAROps is building — from visible data to explainable anomalies to governance that never stops running.

From Business Alerts to Database Root Causes: How STAROps and ApsaraDB Agent Sharpen Cross-Layer Diagnostics

ObservabilityGuy — Sat, 10 Oct 2026 07:19:30 +0000

This article introduces how STAROps and ApsaraDB Agent enable precise, cross-layer root cause analysis from app alerts to database commands.

The alert fires. The frontend P95 latency spikes to 1,800 ms. You trace the call chain downstream—Redis connections are fluctuating, ApsaraDB RDS (RDS) active sessions are climbing—and the clues gradually converge on the database layer. At this point, opening more monitoring dashboards won't help; what you need is something that can run deep diagnostics against the database itself.

The problem isn't a lack of monitoring data—Simple Log Service (SLS), Application Real-Time Monitoring Service (ARMS), Grafana dashboards—it's all there. What's truly missing is something that can stitch cross-layer data into a causal chain. STAROps excels at full-stack correlation analysis, narrowing the troubleshooting scope from many components down to a few suspects. But once the evidence points to the database layer, you need a database diagnostic partner that can drill down on real metrics. Alibaba Cloud ApsaraDB Agent (product ID: alibabacloud-yaochi-agent) fills exactly that role. In this post, we first explain why cross-layer root cause analysis is hard, then walk through three real-world cases—one RDS and two Redis—escalating from single-instance diagnostics (L3) to cross-layer root cause analysis (L4).

Where Do the Difficulties of Cross-layer Root Cause Analysis Lie?

STAROps integrates metrics, logs, and traces across applications, middleware, and infrastructure for full-stack correlation analysis. That's its strong suit. But once the scope narrows to the database layer, the nature of the problem changes. "Redis anomaly" isn't enough—you need to know whether a Lua script blocked the single thread or a hot key maxed out the CPU. "RDS session spike" isn't enough either—you need to know which table is missing which index and which SQL query is doing a full table scan. These fall squarely in the specialized domain of database diagnostics, beyond what a general observability platform covers. ApsaraDB Agent supports RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB, enabling instance-level diagnostics and root cause analysis based on real metrics. Mount it as a skill on a STAROps digital worker, and a single evidence chain can reach from the application layer all the way down to a specific database command.

A digital worker is an AI operations assistant in STAROps that can be equipped with skills. Attach ApsaraDB Agent to a digital worker and it gains professional diagnostic capabilities at the database layer. ApsaraDB Agent can do far more than fault diagnosis—beyond instance diagnostics, it also covers everyday scenarios such as instance specification selection, kernel parameter change evaluation, cost analysis, schema modeling analysis, kernel parameter explanation, and backup status queries.

Frontend Service Chain and Alert Propagation Path

The cross-layer scenarios that follow all involve the same e-commerce application. Below is the frontend service call chain extracted from Cloud Monitor 2.0, which also represents the actual alert propagation path:

The frontend acts as the entry point for user requests, distributing them via frontend-proxy to business services such as wishlist and promotion. These services in turn call data stores like Redis (Tair, collectively referred to as Redis below) and RDS. When Redis experiences single-thread blocking, the downstream business services are the first to time out. Errors propagate upward layer by layer along the wishlist/promotion → frontend-proxy → frontend path, ultimately surfacing as a slow-response alert on the frontend.

The agent replies shown in the following scenarios are the final diagnostic conclusions. During actual use, the agent first displays analysis progress (querying metrics, analyzing slow commands…). Single-instance diagnostics typically complete within a few minutes, while cross-layer root cause analysis takes roughly 5–10 minutes. If the problem description is incomplete, the agent may ask follow-up questions.

Scenario 1: Abnormal Spike in Active Sessions Due to a Missing Index

Question

Analyze the cause of the abnormal spike in active sessions for RDS instance rm-j6c3l32c730ti37ur during the period 2026-07-18T12:21:30+08:00 → 2026-07-18T12:38:30+08:00

Agent Reply

The digital worker identifies that the issue clearly points to the RDS database layer and invokes ApsaraDB Agent for instance diagnostics.

Within the specified time window, active sessions on the RDS instance rose sharply from a normal level to a peak, lasting about 17 minutes before dropping back. Analysis reveals that during the alert period, a large volume of queries concentrated on the inventory table, all containing correlated subqueries on the product_id column. The client source was IP 10.0.20.210, with 540 concurrent connections executing these queries simultaneously. Because the inventory table lacked an index on product_id, every query performed a full table scan, leading to a backlog of active sessions under high concurrency. Root cause: the inventory table is missing the product_id index, and 540 concurrent high-cost correlated subqueries overwhelmed the RDS instance. Recommendation: add an index on product_id to the inventory table.

Summary

This is an L3 diagnosis—the alert directly points to a single RDS instance with no cross-layer propagation involved. ApsaraDB Agent's diagnostic path: confirm metric anomaly → analyze SQL distribution → trace the client source (IP 10.0.20.210, 540 concurrent connections) → locate the missing index (inventory.product_id). From "elevated active sessions" to "which table is missing which index," the diagnostic precision is enough to directly guide remediation. You can apply the index change right from the RDS console.

Scenario 2: Alert in Frontend, Root Cause in Redis Lua Script

Question

A slow frontend interface response alert occurred from 2026-08-10 10:31 to 10:50. Help me diagnose the root cause of the issue and provide a solution

Agent Reply

The digital worker first kicks off STAROps full-stack correlation analysis, gradually narrowing the troubleshooting scope from the frontend layer to the Redis layer through timeline comparison, topology tracing, and change-record cross-referencing. It then invokes ApsaraDB Agent for in-depth diagnostics at the Redis layer.

The in-depth Redis diagnostics reveal the following key metrics: CPU maxed out at 98 % yet throughput dropped (QPS fell from 1,200 to 180), indicating that an operation was blocking Redis's processing thread. Slow-command log analysis pinpointed the source: a Lua script with heavy computational logic monopolized the Redis single thread, and it fired twice during the alert window. Cascading propagation path: Redis single thread frozen → promotion and wishlist services hit SocketTimeout → frontend-proxy returns timeout errors → frontend response slows down. Root cause: the single thread of the managed Redis instance was monopolized by a Lua slow command with heavy computational logic, firing twice. Recommendation: refactor the Lua script to avoid heavy computation in the script body; consider splitting into pipeline batches.

Summary

This is an L4 diagnosis—the alert fires at the frontend layer, but the root cause lies in the Redis layer, involving cross-layer propagation. The diagnostic path: frontend alert → STAROps full-stack correlation analysis → scope narrowed to Redis → ApsaraDB Agent Redis metric analysis → slow-command log pinpoints Lua script blockage → cascading propagation confirmed (Redis → microservices → frontend). The diagnostic precision goes down to the command type and behavioral characteristics, rather than merely answering "Redis has a performance issue."

Scenario 3: Frontend Response Time Spike, Root Cause in Redis EVAL Command

Question

Analyze the cause of the frontend response time spike alert during the period from 2026-08-10 06:19 to 06:25

Agent Reply

Following the same troubleshooting path as Scenario 2: STAROps full-stack correlation analysis narrows the scope to the Redis layer, and ApsaraDB Agent takes over for in-depth Redis diagnostics.

Redis instance metrics show near-CPU saturation and a sudden QPS drop, a pattern similar to Scenario 2. Slow-command log analysis pinpointed the blockage source: EVAL commands. These commands contain extensive inline logic, executing complex calculations directly within Redis. The single thread is fully occupied during execution, forcing all other commands to queue. Root cause: the Redis single thread is monopolized by slow EVAL commands, causing all commands to queue. Recommendation: locate and block the client source issuing the abnormal EVAL calls, and optimize the script logic to avoid large-scale loops.

Summary

This is also an L4 cross-layer root cause analysis (RCA), with the same "alert in frontend, root cause in Redis" propagation chain and nearly identical metric behavior. Yet the underlying details differ: Scenario 2 involves heavy computation in the business's resident Lua script—fix it by refactoring the script. Scenario 3 involves a temporary inline EVAL script from an anomalous source (EVAL is Redis's command for executing inline Lua scripts)—fix it by blocking the anomalous source and moving the computation out of Redis. Same symptoms, different remediation paths.

Scenario Comparison: Same Symptoms, Different Causes

Looking at Scenario 2 and Scenario 3 together:

	Scenario 2 (10:31 Alert)	Scenario 3 (06:19 Alert)
Symptom	Slow frontend response	Frontend response time spike
Propagation chain	Redis → wishlist/promotion → frontend-proxy → frontend	Same as left
Metric behavior	CPU 98%, QPS 1200→180	CPU saturated, sudden QPS drop
Root cause	Heavy computation in resident business Lua script, monopolizing single thread (occurred twice)	Temporary inline EVAL script monopolizing single thread
Remediation direction	Refactor Lua script	Block anomalous source, move computational logic out of Redis

Same symptom, same middleware, and the root cause mechanism in both cases is "slow commands monopolizing the single thread"—yet the specific causes and remediation actions differ. This is precisely the value of command-level diagnostic precision: it pins the conclusion down to "which command, where it came from, and how to handle it," rather than merely answering "Redis has a performance issue."

Scope

ApsaraDB Agent currently supports the following Alibaba Cloud database products: RDS, PolarDB, Tair (Redis-compatible), MongoDB, Lindorm, AnalyticDB, ClickHouse, and SelectDB. Self-managed databases and non-Alibaba Cloud databases are not supported.

When the evidence is insufficient for a definitive conclusion, the agent marks the confidence level and lists potential troubleshooting directions rather than forcing a root cause. Before executing any fix, it is best to have a DBA or engineering lead review the diagnostic conclusions.

Quick Start

Log on to the STAROps console and create a digital employee.
Search for alibabacloud-yaochi-agent under "Skills → Features" and associate it with the employee.
Ask your question in the dialog box. Single-instance diagnostics typically return results within a few minutes; cross-layer root cause analysis takes a bit longer (roughly 5–10 minutes for the scenarios in this post).

You can use the following template to construct your prompt, replacing the content in the brackets:

Analyze the cause of [Anomaly Description] for [Instance Type: RDS/Redis/Tair/MongoDB/PolarDB] instance [Instance ID] during the period from [Start Time] to [End Time]

For example: "Analyze the cause of the abnormal spike in CPU utilization for the Redis instance r-xxxxx during the period from 2026-08-10 10:00 to 11:00." Describe the problem you encountered and the time it occurred using natural language. If you are unsure of the specific instance ID, you can describe the application name and the anomalous symptoms, and the Agent will attempt to correlate it with the underlying database instance.

ApsaraDB Agent raises database-layer diagnostic precision from "some middleware has a problem" to "which specific command or index caused the issue." STAROps, in turn, stitches cross-layer evidence into a complete causal chain. Together, SREs and DBAs can use the same toolset for end-to-end troubleshooting—from alert to fix. For complex faults involving cross-layer propagation, this combination is worth a try.

STAROps currently offers new users 10,000 credits for the first month plus a recurring free tier of 2,500 credits every month. Give STAROps a spin—feed it your most recent production incident and see whether it can actually help: pinpointing the exact command, tracing where it came from, and telling you how to fix it.

Try it now: https://starops.console.aliyun.com/

Zero-Code Transformation: Making the AI Agent Sandbox No Longer a Black Box

ObservabilityGuy — Fri, 18 Sep 2026 06:06:51 +0000

This article introduces Alibaba Cloud's OBI for zero-code observability of AI Agents in ACS Agent Sandboxes.

In 2026, AI Agents are making the critical leap from "chatting" to "doing": writing and executing code themselves, operating browsers to search for information, and invoking a series of tools to complete complex tasks. The prerequisite for allowing an Agent to safely "take action" is providing a secure and efficient execution environment. Consequently, sandboxes have become the de facto standard for Agent infrastructure.

Agent Sandbox, launched by Alibaba Cloud Container Compute Service (ACS), serves as the sandbox compute foundation for production-grade AI Agents, featuring MicroVM-level isolation, elasticity of 15,000 instances per minute, and memory-level hibernation and wake-up. However, a new problem arises: the better the isolation, the more the sandbox interior resembles a black box. Operations and security teams find it difficult to see which models the Agent invoked, what code it ran, and which external services it accessed.

Now, this puzzle piece is in place: OpenTelemetry eBPF Instrumentation (OBI) from Cloud Monitor 2.0 (CMS) officially supports the ACS Sandbox scenario. Without changing a single line of code or rebuilding images, every model invocation, tool call, and external access by an AI Agent within the sandbox automatically generates standard traces and metrics.

Sandbox Foundation: Isolation, Elasticity, and State

Code executed by AI Agents has a distinct characteristic: much of it is dynamically generated by large models on the spot. Such code is untrusted and unpredictable, and must never run in the same container as production workloads. Agent Sandbox addresses this with a MicroVM-level isolated execution environment: each sandbox has a dedicated kernel with end-to-end isolation across compute, network, and storage. Even if the code inside exhibits abnormal behavior, it remains securely locked in the "box."

Beyond security, there are three sets of capabilities tailored for Agent scenarios.

Large-scale, low-latency elasticity. Agent tasks are often bursty: a reinforcement learning session (AgentRL) might instantly require thousands of environments, while peak requests for an online service (Agent Serving) can surge. Agent Sandbox supports creating up to 15,000 sandboxes per minute. Combined with a warm pool, it achieves hundred-millisecond startup times, and image caching reduces pull times by over 90%. In a public case study, MiniMax spun up 5,000 sandboxes within 10 seconds.

State retention. Running sandboxes support on-demand hibernation with memory state fully preserved, enabling rapid wake-up within 1 to 10 seconds. Combined with checkpoint/restore, an Agent's execution state can be saved, migrated, or cloned. When multi-turn conversations are interrupted, there is no need to destroy and rebuild the environment. No CPU or memory fees are charged during hibernation, and per-second billing makes "use it and pause it" truly cost-effective.

Open ecosystem. Agent Sandbox provides two integration methods: an E2B-compatible SDK and the Kubernetes-native Sandbox CR. It is compatible with mainstream Agent frameworks such as AgentScope and LangGraph, covering typical sandbox scenarios like Code Interpreter (code execution) and Browser Use (browser operations).

Monitoring Blind Spots: The More Secure, the More Like a Black Box

The sandbox solves the problem of "whether it is safe to run," but "how well it runs" remains shrouded in mystery. Attempting to apply traditional Application Performance Management (APM) solutions one by one, you find that all three paths lead to dead ends.

SDK instrumentation cannot get in. Traditional application monitoring relies on integrating SDKs or attaching language-specific instrumentation agents in the code. However, the code running inside the sandbox is generated by large models on the fly — it is diverse and unpredictable. Agent frameworks themselves also span multiple languages such as Python and Node.js. You cannot expect the model to "add monitoring instrumentation while generating code."

Host-level collection has nowhere to go. ACS is a serverless container compute service with no user-visible nodes, leaving traditional DaemonSet-based host-level monitoring components with no foothold. The strong isolation of MicroVM completely blocks the path of "looking in from the host."

Sandbox lifecycles are too short. Created in hundreds of milliseconds, destroyed after use, hibernated at any time — by the time you log in to troubleshoot, the environment has long been gone. A host-centric monitoring model simply cannot keep up with this ephemeral rhythm.

The result: when an Agent task fails, you do not know whether it was a slow model response, a tool timeout, or a code error. When token costs rise, you cannot tell which stage consumed them. When the security team wants to audit which external addresses the sandbox code accessed, they can only sift through network logs manually.

The OBI Solution: Achieving AI Agent Observability with Zero Modification

OBI takes a different approach: instead of entering the application, it stands between the application and the kernel.

OBI intercepts application communications at the kernel and library function levels, parses protocol semantics, and directly outputs standard OpenTelemetry telemetry data. This means that regardless of whether the sandbox runs Go, Java, Python, Node.js, or .NET, and regardless of the HTTP framework, database, or large model provider involved, monitoring data is automatically collected without modifying any code. For sandbox scenarios where "code is dynamically generated by models on the spot," this is virtually the only viable solution.

In the ACS Sandbox environment, OBI runs alongside the sandbox workload as a sidecar. After installing the ARMS instrumentation onboarding assistant ack-onepilot (v5.2.2 or later), you only need to add a few labels to the workload YAML. The OBI sidecar is then automatically injected, and the collected data is centrally aggregated in the CMS console. (Running eBPF inside a sandbox requires specific Linux kernel permissions, which can be enabled for your account by submitting a ticket.)

Capability List: Traces, Metrics, and GenAI Semantics

What can you see after integration?

The first layer is the full suite of classic application monitoring: application topologies, traces, abnormal transactions, slow transactions, and SQL analysis. Every external call by the Agent inside the sandbox generates a trace — whether the bottleneck is model inference or tool execution, a single trace reveals it all.

The second layer is the true highlight of this upgrade: AI Agent observability. OBI features built-in protocol-level tracing for four major GenAI providers — OpenAI, Anthropic, Google Gemini, and Qwen — automatically identifying LLM calls and generating telemetry data according to GenAI semantic conventions:

LLM invocations: Automatic extraction of model names, token consumption, latency, and error states;
Tool calling: Automatic parsing of tool calling information from model responses, making the Agent's "decision on which tool to use" clear at a glance;
Model Context Protocol (MCP), embedding, reranking, and vector retrieval: full coverage of the core stages in the RAG pipeline.

If your model traffic flows through self-hosted inference services or LLM gateways such as vLLM or LiteLLM, you only need to declare the corresponding host in the configuration. OBI parses them using OpenAI-compatible protocols and includes them in the AI Agent observability view.

For sandbox scenarios, this data simultaneously answers three questions: Performance—which stage is slowing down the task; Cost—which types of invocations are consuming tokens; Security—which external addresses the untrusted code in the sandbox accessed. All of these are recorded out-of-band, naturally serving as an audit ledger.

Three-Step Integration: Install, Label, and View Data

The entire integration process requires no changes to the application code.

Step 1: Install the ack-onepilot component (v5.2.2 or later) for your ACS cluster in the ACS console.

Step 2: Add OBI labels to the YAML file of the target workload:

labels:
  apsara.apm/application-type: obi    # Indicates that this application is integrated via OBI 
  armsPilotAutoEnable: 'on' 
  armsPilotCreateAppName: "your-application-name" 
  armsPilotAppWorkspace: "your-workspace-name"

Step 3: Wait about 2 minutes, then open the CMS console: view topologies and traces under "Application Monitoring," and view model and tool call details under "AI Agent Observability."

Step 4: Go to the AI Agent application in CMS to view monitoring data:

Final Thoughts

Competition in Agent infrastructure is moving from the "make it run" phase to the "make it transparent" phase. ACS Agent Sandbox ensures that AI Agents run securely and elastically, while OBI makes every execution transparent and visible. Together, "security" and "observability" for production-grade Agents can coexist for the first time — all at the cost of zero lines of code.

If you are building Agent applications, start with the official documentation "Access ACS applications through OBI": https://www.alibabacloud.com/help/en/cms/cloudmonitor-2-0/access-acs-applications-through-opentelemetry-non-intrusive-monitoring-obi

Toward Production-Grade AgenticOps: How STAROps Builds Generalizable Root Cause Analysis

ObservabilityGuy — Wed, 16 Sep 2026 07:23:26 +0000

This article introduces STAROps, an AgenticOps framework for reliable, generalizable root cause analysis in production.

When people evaluate AgenticOps, they often start with the obvious questions: How many tools can the agent use? Can it run automated inspections, propose a fix, or even carry out a change?

But as we built STAROps, we kept coming back to a more fundamental question: Does it identify the right root cause?

We see root cause analysis (RCA) as the core of AgenticOps because it determines where every subsequent step is directed and what that step does. If the root cause is wrong, impact evaluation, remediation, change execution, and outcome validation all focus on the wrong target. When an agent can only advise, a wrong RCA wastes engineers' time. When an agent has permission to act, a wrong RCA becomes a production risk.

RCA sets the direction. Every other capability determines only how quickly and how far the agent can follow it.

When the Root Cause Is Wrong, Every Action That Follows Is Wrong

An alert usually tells us which metric crossed a threshold, not where the failure began.

A frontend endpoint returning 5xx errors may point to a problem in the frontend code, but it could also be caused by a slow downstream database, an unavailable cache, exhausted node resources, or a recent configuration change. High CPU is not necessarily the root cause either: it may be the result of a traffic spike, or it may coincide with a full garbage collection cycle, thread pool exhaustion, or a runaway loop.

Those distinctions change the response. Mistake a node resource problem for insufficient application capacity, and you may keep scaling out pods that run on the faulty node. Mistake a slow SQL query for a frontend code defect, and you may trigger a rollback that changes nothing. Mistake a traffic drop caused by an accidentally scaled-down Deployment for a network problem, and the investigation starts in the wrong direction.

General-purpose agents can already query a wide range of data and produce well-structured reports that sound plausible. But a complete report does not prove that the agent found the object that actually failed. If it stops at the entity where the alert fired, or mistakes a pronounced anomaly along the propagation path for the root cause, more elaborate reasoning can make the wrong conclusion even more convincing.

Why a Proof of Concept Cannot Prove RCA Capability

When teams validate RCA, they usually choose a handful of known faults: a Pod stuck in a crash loop, a slow SQL query, or a Redis outage. They then check whether the system returns the expected answer.

This is a reasonable way to validate a proof of concept. The scenarios are well defined and the results are easy to verify, so teams can quickly confirm that the data, tools, and product workflow operate end to end. But it proves only that the system handles scenarios A, B, and C. It does not show that the system will know where to look when fault type D appears.

If every alert type is mapped to a Skill or workflow, a proof of concept will usually perform well. The alert type, investigation entry point, query sequence, and expected answer are all specified in advance, so the agent only has to follow a path that someone has already designed. Change the alert, or let the same fault surface at a different point in the business, and that path may no longer work.

That does not mean Skills lack value. Experienced site reliability engineers (SREs) know which metrics to check first when latency rises, when a change must be examined, and which symptoms tend to send an investigation down the wrong path. That experience should be captured. Skills work best as investigation leads and checklists that help the agent avoid dead ends; they should not define the limits of RCA.

Real Production Faults Follow No Fixed Playbook

In production, however, far more things can go wrong. A problem may originate in an application, container, node, database, cache, or network. It may also follow a release, a scaling operation, or a configuration change. Topologies keep changing, data may be incomplete, and several anomalies can appear in the same time window.

The same alert can have entirely different causes, and the same root cause can trigger different alerts in different systems. The root cause may be in a database while the alert fires on the frontend. It may be on a node while the first user-visible symptom is a traffic drop on a business endpoint. Real faults routinely cross data domains and propagate through call, deployment, and host relationships.

Generalization does not mean building a model that knows the answer to every fault. A more realistic standard is that, when no workflow matches exactly, the agent still knows how to begin. It first identifies the entity that the alert refers to, then follows relationships to the relevant services and resources. When several candidates emerge, it gathers more evidence, checks the sequence of events, and eliminates hypotheses that cannot explain the symptoms. If the evidence is insufficient, it knows when to stop instead of forcing a root cause.

Fault scenarios cannot be exhaustively enumerated, but the basic method of investigating them can be reused. STAROps is designed to keep an investigation moving and converge on an answer even when the fault is unfamiliar.

How STAROps Turns RCA into a System Capability

Because a scenario catalog can never cover every production fault, simply adding more workflows cannot be the primary goal. STAROps focuses on the investigation itself: how the agent understands the current system, decides what to examine next, demonstrates that a conclusion is reliable, and learns from failures in production.

First, Understand the System: UModel

Many RCA investigations go wrong at the first step because different systems use different names for the same object.

In application performance management (APM), a business service appears as a service. In distributed traces, it appears as a set of spans. In Kubernetes, it maps to a Deployment and its pods; further down, it maps to an ECS instance or a node. Logs, metrics, and change records use different fields as well. Without a unified model of these object relationships, the agent has to match objects using names and context. It can easily split one object into several or mistake a neighboring object for the root cause.

UModel organizes these objects and relationships. It records which pods run a service, which Deployment each pod belongs to, which node hosts it, which databases the service depends on, and where to query the relevant metrics, logs, and traces.

The immediate benefit is that the agent knows exactly what it is investigating. Whether the investigation moves down from a business endpoint to a service, pod, and node, or outward from a change to the affected services, it can follow explicit relationships. The model proposes hypotheses and interprets evidence; UModel provides a stable map of the system.

Static Topology Is Not Enough: Map the Investigation, Too

A console topology usually shows which objects exist and which ones depend on others. A fault investigation needs to capture something else as well: which objects have already been checked, where anomalies were found, which candidates are under suspicion, and which branches remain unexplained.

On top of UModel, STAROps maintains a dynamic investigation topology. The investigation starts with the alerting entity and expands as evidence accumulates. When a candidate is ruled out, that branch stops. If several pods on the same node degrade at the same time, the investigation converges on the host. If database connections slow down, the agent checks the SQL queries, connection pool, and upstream and downstream traces.

This graph serves as both an investigation map and a record of the process. Metrics, logs, traces, events, and changes are attached to the relevant objects. Candidate root causes are retained along with the evidence for and against each one. Engineers can see why the agent continued in a particular direction, add information at critical points, or take over the investigation.

Use a Benchmark to Expose Answers That Only Look Right

RCA agents face a tricky problem: an answer can be wrong even when the explanation sounds convincing. From the final report alone, it is often hard to tell whether the agent followed the evidence to the root cause or guessed a common answer from the alert's surface symptoms.

That is why RCA-Bench records more than a single root cause label. The public RCA-100 dataset contains 103 fault cases across six major categories and 28 fault types. Each case records the root cause entity, fault type, propagation path, and key evidence. In a queryable environment, the agent must decide for itself what to examine first and where to go next.

We evaluate three things separately: whether the agent found the right root cause entity, identified the right fault type, and supported its investigation with evidence. The first two are scored mainly from entity relationships and fault-type relationships in UModel. Deterministic rules currently account for about 82% of the overall score. The LLM is used only to help judge the direction of the investigation and whether the evidence is sufficient, so one model does not have complete authority over another model's score.

Production Failures Must Feed the Next Iteration

An offline benchmark can show whether a capability exists, but production introduces permissions, data integrations, customer-specific topologies, and version differences. A version that scores well offline may still fail in production because it cannot retrieve the data, chooses the wrong tool, or converges too early.

STAROps retains the task input, tool calls, query results, errors, evidence, execution time, and user feedback from production runs. For a low-scoring run or failure case, we first determine whether the problem lies in object identification, data retrieval, investigation direction, evidence quality, or the final conclusion.

Reproducible failures become regression cases. Whenever UModel, a tool, a Skill, a model, or an investigation strategy changes, we validate the change against those cases. This turns a one-off fix into a regression check that every later version must pass.

Head-to-Head Results: A Complete Report Does Not Mean an Accurate Root Cause

The published RCA agent benchmark uses a stratified evaluation set of 30 cases drawn from RCA-100. STAROps and OpenClaw + DeepSeek-V4-Pro receive the same task inputs and are evaluated with the same brise scorer.

Evaluation dimension	STAROps	OpenClaw + DeepSeek-V4-Pro	Difference
Overall score	75.23	51.02	+24.2
Root cause entity	90.	52.8	+37.2
Fault type	58.	33.3	+24.7
Investigation process	75	69.8	+5.2

The totals matter, but the breakdown matters more.

The gap in the investigation-process component is only 0.052, equivalent to 5.2 points in the table. This shows that a general-purpose agent can already perform multiple rounds of queries and produce a reasonably complete analysis. The real separation comes from identifying the root cause entity and the fault type. Choose the wrong object, and every subsequent metric, log, and trace centers on that object. Misclassify the fault type, and the remediation recommendation loses its foundation.

By major fault category, STAROps leads by 50.6, 29.9, 27.6, and 24.2 points in database, node, code, and resource cases, respectively. It trails the comparison system by 11.2 points in traffic cases, mainly because of rate-limiting scenarios. Rate limiting, DNS, CDN, third-party services, and public network paths are the areas STAROps will focus on next.

Case Studies

Hard cases do not necessarily involve obscure technologies. The real difficulty is often that the alert fires far from the root cause while several plausible explanations appear at the same time. The following two cases fit that pattern.

product-catalog Traffic Drops, but the Root Cause Is an Underlying Node

The high-node-CPU case begins with a traffic-drop alert for product-catalog::ListProducts.

If you look only at the business-level data, frontend is the most obvious suspect. Its traffic and error rate change at the same time, and it is the upstream entry point for product-catalog. The general-purpose ReAct agent concluded that the load balancer had failed, while OpenClaw attributed the problem to frontend HTTP 5xx errors. Both reports cite data and describe a propagation path that sounds plausible.

The actual root cause is Kubernetes node cn-hongkong.10.0.1.107. CPU utilization on that node rose from about 10.38% to 99.98%, affecting multiple pods on the same node. frontend traffic dropped by 66.66%, followed by a 62.48% drop in ListProducts traffic, which ultimately triggered the business alert.

STAROps did not stop at frontend. It moved from the alerting operation to its service, then followed the runs-on relationship to the pod and host node. Other nodes remained healthy while several pods on this node degraded at the same time. Together, those two pieces of evidence narrowed the investigation to the infrastructure layer.

In this evaluation, ReAct scored 15, OpenClaw scored 12, and STAROps scored 94. If the team had acted on either of the first two conclusions, it might have kept investigating the load balancer or scaled out frontend while leaving the node with saturated CPU untouched.

Node CPU stays near 100% 
↓ 
Multiple pods on the same node are affected 
↓ 
frontend and other business services degrade at the same time 
↓ 
product-catalog API traffic drops, triggering an alert

Frontend Checkout Slows Down, but the Root Cause Is a Slow SQL Query

The slow-SQL case begins with an alert on frontend::POST /api/checkout. At the same time, the environment shows changes in checkout latency, Kafka lag, Java Virtual Machine (JVM) garbage collection, thread count, and Deployment scaling. Any one of these signals, viewed in isolation, could look like a plausible root cause.

The general-purpose ReAct agent concluded that checkout contained a code defect, while OpenClaw located the problem in frontend's checkout logic. Both captured part of the anomaly, but their investigations stopped midway along the propagation path.

STAROps continued downstream through the traces. checkout depends on cart, which in turn depends on inventory. A slow inventory trace showed a SELECT query taking about 10.8 seconds, plus another 2.1 seconds to acquire a database connection. The slow query held up inventory, and the delay then propagated through inventory → cart → checkout → frontend, ultimately appearing as a slow response from the frontend endpoint.

There were several distracting signals along this path. Kafka lag was measured only in milliseconds and could not explain a timeout measured in seconds. The garbage collection, thread-count changes, and Deployment scaling occurred after the services had begun to degrade, making them more likely consequences of the slow query. Only after the signals were placed on a single timeline did the evidence converge on the slow SQL query in inventory.

ReAct and OpenClaw both scored 15 on this case, while STAROps scored 84. If the team had treated the problem as a code defect, it might have rolled back frontend or checkout while leaving the SQL query that was blocking the call chain in place.

inventory slow query and blocked connection acquisition 
↓ 
cart calls to inventory slow down 
↓ 
checkout call chain latency increases 
↓ 
frontend checkout slows down, triggering an alert

Conclusion

One of these faults occurred at the node-resource layer; the other occurred along the database access path. They have little in common on the surface, and STAROps used different data and investigation paths for each. But the method was the same: identify the object, then gather evidence along its relationships. When multiple candidates emerge, place them on a single timeline and determine which one explains the full propagation path and which ones are merely accompanying symptoms.

This is what generalization must solve. We cannot make an agent know every possible fault in advance. We can, however, ensure that it can move an investigation forward when the problem is unfamiliar and stop at the right boundary when the evidence is insufficient.

That is why RCA sits at the core of STAROps. Impact analysis, remediation recommendations, and change execution have a reliable starting point only when the agent consistently identifies the right object and cause. As agents gain more permission to act in production, the cost of a wrong root cause rises. A single misjudgment can escalate from misleading an engineer to triggering the wrong scale-out, rollback, or configuration change.

We expect the capability gap in AgenticOps to center increasingly on judgment quality. Tool calling and automated execution will become baseline capabilities. What will determine whether an agent is truly ready for production is whether it can build a complete evidence chain for an unfamiliar fault, distinguish the root cause from accompanying symptoms, and stop when the evidence runs out.

For STAROps, root cause identification is an engineering discipline that must be continuously calibrated against real faults. Keeping its judgments accurate and its limits explicit remains our most important task as we move AgenticOps into production.

Plaintext Stays Put, Logs Stay Useful: Sensitive Data Protection with Alibaba Cloud Observability

ObservabilityGuy — Fri, 11 Sep 2026 05:22:45 +0000

This article explains how Alibaba Cloud SLS secures sensitive AI logs via multi-stage masking and encryption while preserving analytical value.

AI Agents are filling logs with more sensitive data in more complex forms. Beyond traditional fields such as phone numbers, IP addresses, and API keys, AI coding Agents, retrieval-augmented generation (RAG) applications, and operations-focused AI Agents may also read source code, configuration files, environment variables, and database connection strings. They then write tool calls, terminal output, and model responses to audit logs, making it easier for sensitive information to enter the logging pipeline.

You cannot simply discard this data; it provides essential evidence for troubleshooting, auditing, and governance. To keep data both secure and useful, Alibaba Cloud Observability protects sensitive data across collection, ingestion, and transformation. Data can be masked or encrypted before it reaches a LogStore, then transformed after ingestion into secure datasets suitable for long-term retention and shared analysis.

Let's start with the overall design, then look at how each method works in practice.

Reduce Exposure Without Sacrificing Value: Protect Sensitive Data End to End

The Simple Log Service (SLS) approach to sensitive data protection comes down to two questions: how should you protect the content, and where in the data pipeline should that protection happen? The first determines whether to use masking, symmetric encryption, or envelope encryption. The second determines whether to process the data on the client side, in an ingest processor, or through data transformation.

You can combine these two dimensions flexibly. The same protection method can run during collection, before ingestion, or after data reaches a LogStore; at any of these stages, you can choose masking or encryption based on the risk. Match the combination to the data source, compliance requirements, and downstream analysis. Use client-side processing when plaintext must not leave the host, an ingest processor to govern direct writes from multiple sources, and data transformation to create a protected LogStore for long-term sharing or governance of existing data.

Choose Based on How the Data Will Be Used

Different types of sensitive data call for different treatment. SLS provides three types of SPL functions: masking, symmetric encryption, and envelope encryption. These options address three needs: preserving readability, allowing authorized recovery, and separating duties.

Masking: Keep Logs Readable and Reduce Privacy Risk

Use mask for personally identifiable information (PII) or credentials scattered through free text. It supports two matching modes:

buildin: matches built-in patterns for mobile phone numbers, email addresses, national ID numbers, bank card numbers, IP addresses, landline numbers, AccessKey pairs, license plates, international mobile subscriber identity (IMSI) values, vehicle identification numbers (VINs), and more;
keyword: matches field or key names such as password, token, authorization, and api_key.

You can replace an entire value with a placeholder or retain a prefix and suffix. Keeping the first three and last four digits of a phone number, for example, protects privacy while still allowing a rough cross-check against application records during troubleshooting. Masked logs remain readable, searchable, and aggregatable, making masking a good default for most logging scenarios. For implementation details, see the page titled Use the mask function for data masking.

Common use cases include:

phone numbers, email addresses, and national ID numbers embedded in AI gateway prompts;
client IP addresses and authentication tokens in application access logs;
bank card numbers, addresses, and user identifiers in LLM application logs.

Symmetric Encryption: Store Ciphertext and Decrypt It When Authorized

Use aes_encrypt when data must be recoverable later and key ownership is clearly defined. Content such as internal RAG Q&A, model requests, and customer solution documents can be stored as ciphertext, then decrypted during an authorized audit by the team that holds the key and initialization vector (IV).

If one team handles collection, analysis, and auditing, it can manage the key and IV centrally, making symmetric encryption a simple, high-performance choice. The boundary is equally clear: once the key is shared, every holder can decrypt the data. This makes symmetric encryption suitable within one organization or security domain, but not for broad distribution across teams.

Envelope Encryption: Separate Public and Private Keys for Collaboration on Highly Sensitive Data

Use envelope_encrypt when data collection and decryption are handled by different parties. It generates a random data encryption key (DEK) to encrypt the application data, then encrypts the DEK with an asymmetric public key. The collection side holds only the public key; a separate security or audit team safeguards the private key.

This pattern is well suited to highly sensitive content, including:

the full output returned after an AI coding tool reads a file;
command output;
entire log entries containing keys, connection strings, or configuration files.

Envelope encryption also provides predictable performance. A DEK is generated at the pipeline level, and asymmetric encryption applies only to that key; AES still encrypts the application data. This minimizes the effect on collection throughput.

Protection methods are not tied to specific processing stages. The following sections trace three points along the data path and use one representative method for each, but every stage can support other methods as well.

Client-Side Protection: Keep Plaintext at the Source

In stricter compliance environments, protection must happen at the collection source. Logs are processed as soon as they are collected, so plaintext never leaves the host or container or enters the downstream transport pipeline.

Client-side protection comes in two forms. LoongCollector processes host and container logs as they are collected, while LoongSuite Pilot provides on-device privacy controls for AI coding Agents. Together, these approaches contain plaintext before data leaves its source.

LoongCollector: Process Data as You Collect It Without Changing the Application

Note: All names, phone numbers, email addresses, public keys, private keys, and other sensitive data shown in this article are synthetically generated.

For applications running on Elastic Compute Service (ECS) instances, Container Service for Kubernetes (ACK) clusters, on-premises data center hosts, or containers, LoongCollector can run SPL during collection instead of acting solely as a collector. AI gateways, Alibaba Cloud Model Studio applications, RAG services, and model proxy layers typically produce large volumes of access and interaction logs. These logs combine structured fields with natural-language input. Processing them at the collection source significantly reduces plaintext exposure.

Consider an AI gateway access log. A raw entry might contain the caller's IP address, an API key, and the user's prompt:

model=qwen-max client_ip=203.0.113.55 prompt="My name is Zhang. My phone number is 138xxxxxxxx, and my email address is zhangwei@xxxx.com. Please look up my recent orders." tokens=128

In the LoongCollector collection configuration, mask can handle built-in PII patterns and keyword fields in a single pass:

After the log is written to SLS, it can still be used to analyze model call volume, token usage, and gateway access trends, but the sensitive values have been replaced:

This approach works well for host- and container-side data such as application logs, gateway logs, and model proxy logs. It requires no changes to application code: processing happens in the collection configuration, and logs reach SLS in a safe, masked form.

LoongSuite Pilot: Mask AI Coding Data Before Distribution

Another common client-side use case is auditing AI coding activity. Coding Agents running on a developer's machine often access source code, configuration files, and local environment variables. If a developer pastes a configuration snippet into a prompt, or an Agent reads a .env file and writes its contents into tool-call arguments, API keys, cloud-provider access key/secret key pairs (AK/SK), and private keys can enter the collection pipeline.

LoongSuite Pilot is designed for these on-device AI coding scenarios. It discovers and collects activity data from Claude Code, Cursor, Codex, Qoder, and similar tools, then normalizes the data into audit events for sessions, tool calls, model calls, and token usage. Its built-in rule-based masking engine scans and replaces sensitive content before sending data to any output channel. Once masking is enabled, the same rules apply whether the destination is SLS, JSONL, HTTP, or OpenTelemetry Protocol (OTLP).

Pilot's automatic masking covers these common credential types:

Masking type	Coverage	Replacement marker
Cloud AccessKey	Alibaba Cloud LTAI, AWS AKIA/ASIA, Tencent Cloud AKID	`[ACCESSKEY_MASKED]`
API key	OpenAI-compatible `sk-` keys; GitHub personal access tokens (PATs) with `ghp_`, `gho_`, or `ghs_` prefixes	`[APIKEY_MASKED]`
Database connection string	MySQL, PostgreSQL, MongoDB, and Redis URIs; password-bearing JDBC URLs	`[DATABASEURL_MASKED]`
Private key	PEM/OpenSSH private key blocks	`[PRIVATEKEY_MASKED]`

For example, in an AI coding session, an Agent reads a local .env file and produces the following tool result:

DB_URL=mysql://root:Abc@2025!@10.0.0.12:3306/prod
ALIYUN_AK=LTAI5tFakeExampleKey0001
-----BEGIN RSA PRIVATE KEY-----
MIIEvQIBADANBgkqh...
-----END RSA PRIVATE KEY-----

After LoongSuite Pilot masks the data, SLS receives:

mask.mode controls masking, with three supported values: none, all, and custom. Masking is disabled by default. Once enabled, sensitive credentials no longer reach downstream outputs in plaintext. Audit metadata such as session_id, tool_name, token usage, and execution duration remains available, so engineering managers and security teams can still analyze AI coding activity, tool-call risks, and cost distribution.

In summary, client-side protection is a good fit when:

plaintext must not leave an application host or developer machine;
the collection pipeline crosses network and team boundaries, making plaintext transmission unacceptable;
you want to protect sensitive data at the point where logs are created without changing application code.

Protect Data Before Ingestion: Govern Direct Writes Centrally

Not all data passes through LoongCollector. Many systems write directly to SLS through SDKs, web tracking, OpenAPI, or third-party collection pipelines. For such data, an ingest processor provides centralized server-side governance: after the data reaches SLS but before it enters a LogStore, the processor runs SPL to mask or encrypt it.

An ingest processor's key advantage is centralized configuration. Rules live in SLS, so clients do not need to be upgraded one by one. When a processing policy changes, you update the processor once to cover all direct writes. This is easier to roll out across multiple business lines, SDKs in different programming languages, and third-party systems.

Consider an AI gateway access log. The gateway joins the fields for each model call into a line separated by ## and writes it directly to SLS. The request field contains the complete model request body and may include arbitrary sensitive content, such as the user's prompt or national ID number. The other fields contain structured call metadata:

2026-06-25 11:17:25.471##c9d7ab4c-b312-9d08-9243-c7f7475905be##5000000276111136##API_KEY##qwen-max##text-generation##{"input":{"prompt":"My national ID number is 53010219200508011x. Please check my insurance policy status."}}##chat-app##3de69809a9d0fd22773f33c77dab12a8

Here, request contains the entire unstructured request body. To show how an ingest processor can apply one policy to all direct writes before storage, this example uses symmetric encryption for the entire request field: it first splits the line on the delimiter, then applies aes_encrypt only to the sensitive request body.

Encrypt a Highly Sensitive Request Body in an Ingest Processor

Ingest processor SPL configuration:

* | extend temp = split(content, '##')
  | extend time          = temp[1],
           request_id    = temp[2],
           caller_uid    = temp[3],
           source        = temp[4],
           model         = temp[5],
           api_type      = temp[6],
           request       = temp[7],      
           service_name  = temp[8],
           trace_id      = temp[9]
  | extend request = to_base64(aes_encrypt(
        cast(request as binary),
        cast('aZ4kP9mX2qL7nR1t' as binary),
        cast('Gh8dW3sB6vN0cT5e' as binary)))
  | project-away content, temp

After ingestion, request is Base64-encoded ciphertext, while call metadata such as request_id, region, model, service_name, and trace_id remains in plaintext:

Operations engineers can troubleshoot the request path using request_id and trace_id, while business operations teams can count calls by model and service_name. The request body itself remains encrypted.

Decrypt Data Only When Authorized

When the original data is needed, an authorized team member who holds the key and IV decrypts it at query time:

Data Transformation: Build Secure Datasets for Shared Analysis

Client-side processing and ingest processors protect data before storage. Data transformation addresses a different problem: how to reorganize data already in a LogStore into datasets that are safer and better suited to long-term analysis.

In practice, many teams maintain two LogStores:

Raw LogStore: retains data briefly, has strict access controls, and is used only for emergency troubleshooting or security investigations;
Protected LogStore: stores the output of a data transformation task; its sensitive fields are masked or encrypted, and it retains data longer to support analysis and audits for a broader audience.

This pattern works especially well for AI observability data. Consider AI coding activity logs collected by LoongSuite Pilot from tools such as Qoder. Each llm.response event contains structured metadata, including the session ID, model, and user ID, plus a gen_ai.output.messages field. That field contains the full text of the model's response for the current turn and may include internal system names, host details, configuration data, keys, or other sensitive content:

{
  "event.name": "llm.response",
  "gen_ai.session.id": "58bf461a-285f-468e-963c-00cd1706a9d9",
  "gen_ai.provider.name": "qwen",
  "gen_ai.response.model": "auto",
  "user.id": "1",
  "gen_ai.output.messages": "[{\"role\":\"assistant\",\"parts\":[{\"type\":\"text\",\"content\":\"Candidate salary table prepared: Zhang （phone number 138xxxxxxxx）...\"}],\"finish_reason\":\"end_turn\"}]",
  "trace_id": "a58ed78ac2573159b4fa8b38ab4c2697",
  "__time__": "1782367255"
}

Because gen_ai.output.messages contains a complete model response with no fixed structure, field-by-field masking cannot cover every risk. Yet an investigation may still need the original text. This example uses envelope encryption to show how data transformation can first derive the analysis field msg_len from the response length, then encrypt the sensitive body and write it to a protected LogStore:

* | extend msg_len = length("gen_ai.output.messages")
  | extend result = envelope_encrypt(
      cast("gen_ai.output.messages" as binary),
      '-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0Vh1QIpNurP0bB9uHRK3
ZL29SjY74OtYa5nmf9xUMKAuLrio5BN6LRZyG680uCkCThCMOK/T9ka5ZYtEIu8b
U7acNR1dL3LWsbZG3ScbqyISJ3pEBli2rCgOqBQK3EE+6HGavh6wo8FAhkSBiryU
eVKxSJSnC928RvlBWtyfdubVYQHPONhx0TbCj38Njjfi8kbKGgE92q9gTlL+spge
qvgDec9GV2VzxhLoTgxd6jdHYyfFzuQucb0hZtmWqus4E4MTnAXnUJOGizmI02AP
5d49P44kZXElBPRw4A0WISKBmwOzYCZqPSEWyHnj/qIaXAbz90B8TCUiPZlNouCG
TwIDAQAB
-----END PUBLIC KEY-----',
      '{"mode":"RSA"}',
      '{"mode":"CTR"}')
  | extend encrypted_dek = result['encrypted_dek'],
           enc_output_messages = result['encrypted_data']
  | project-away "gen_ai.output.messages"

In the record written to the protected LogStore, gen_ai.output.messages is removed. It is replaced by enc_output_messages, which contains the ciphertext, and encrypted_dek, which contains the public-key-encrypted DEK. The remaining metadata stays in plaintext:

The protected LogStore retains fields such as event type, session ID, model, user ID, and response length, so teams can still use it for alerts, trend analysis, and risk investigations. The model response itself is encrypted. For deeper investigations, authorized security personnel can decrypt selected records with the private key.

This strikes a balance between encryption and analyzability: rather than encrypt every field, extract the metadata needed for analysis first and encrypt only the highly sensitive body. Routine analysis uses the metadata, while the original sensitive content can be accessed only through authorized decryption.

How to Choose: Plaintext Boundary, Analysis Needs, and Decryption Permissions

Ask three questions when choosing an approach.

First, can plaintext leave the collection source? If not, process it at the source.

Second, must the logs remain readable for routine analysis? If so, preserve readability. Encrypt only when the content is highly sensitive as a whole and cannot be reliably protected field by field.

Third, who is allowed to decrypt the data? One team within a single security domain can manage keys centrally. Across teams, or when duties must be separated, decryption authority should remain under the independent control of the security or audit team.

The following table summarizes the options:

Scenario	Recommended processing stage	Recommended method
AI gateway logs on hosts or in containers; plaintext must not leave the host	LoongCollector on the client side	Data masking with mask; encrypt selected fields as needed
SDKs or third-party systems writing directly to SLS	Ingest processor	Masking or encryption before ingestion
Historical logs that require protection	Data transformation	Transform data from the source LogStore, then write masked or encrypted output to a new LogStore
Internal RAG Q&A content that requires controlled recovery	Client-side processing or ingest processor	Symmetric encryption with aes_encrypt
Entire blocks of highly sensitive content, such as tool output or configuration files	Client-side processing or data transformation	Envelope encryption with envelope_encrypt

Keep Logs Secure Without Losing Their Value

Logs make systems observable and support analysis and traceability. Sensitive data protection should preserve that value within a clear security boundary. SLS applies masking and encryption during collection, ingestion, and transformation, helping organizations reduce plaintext exposure while retaining the analysis they need.

Organizations can choose masking or encryption based on the data type. They can apply protection during collection, before ingestion, or after ingestion through data transformation, then choose symmetric or envelope encryption based on who controls the keys. Together, these choices define a clear, enforceable boundary that keeps data both useful and secure.

Alibaba Cloud Partners with Datadog Close Go's Last Observability Gap

ObservabilityGuy — Wed, 09 Sep 2026 02:13:59 +0000

This article introduces OpenTelemetry Go Compile-Time Instrumentation v1, enabling zero-code, build-time observability for Go applications.

Go was the last mainstream language without zero-code observability. In July 2026, the OpenTelemetry Go Compile-Time Instrumentation project shipped its stable v1 release. Launched jointly by Alibaba and Datadog and developed through a year and a half of community collaboration, it lets Go developers add distributed tracing and metrics collection to an application with a single command, without touching any business code.

This article walks through how this milestone works, how to use it, and when to choose it.

1. Why Go Has Long Lacked Zero-Code Observability

Java has -javaagent, Python has sitecustomize, Node.js has --require.NET has CLR Profiler. AI Agents in these languages inject agents dynamically at runtime, so developers get traces and metrics without changing a line of code.

Go can't do that, and the reason is fundamental: Go compiles to a static binary — no VM, no bytecode, no class-loading hooks. Once go build finishes, you have a self-contained machine-code file with nowhere to "attach" an agent at runtime.

That has long left Go developers with two options:

Manual instrumentation: call span.Start() / span.End() explicitly in every HTTP handler and around every DB call — intrusive, and easy to miss spots
eBPF observation from the outside: capture network calls in the kernel — zero-touch, but you see only L4/L7 protocol information, with no business semantics

A wide gap separates the two. For a platform engineering team running hundreds of Go microservices, "add tracing by hand to every service" isn't realistic. eBPF covers a lot of ground but can't reach inside the code — if you want to know which SQL query was slowest in a request, eBPF can only tell you "this TCP connection took 200 ms"; it gives you nothing at the database/SQL level.

Compile-time instrumentation fills exactly that gap: probes are injected during go build, and the resulting binary carries observability with it. No code changes, no external AI Agent.

2. Compile-Time Instrumentation: Injecting Agents at Build Time with -toolexec

The Go toolchain has a little-known but very powerful extension point: -toolexec.

When you run go build, the go command is really orchestrating lower-level tools such as compile and link. -toolexec lets you name a wrapper program, and the Go toolchain routes every invocation of compile/link through that wrapper first — much the way Unix strace or time wraps a command.

OpenTelemetry Go Compile-Time Instrumentation builds on this mechanism. Its core tool, otelc, acts as a wrapper around -toolexec. As the compiler processes each package's source, otelc does the following:

Parse the AST: read the current package's abstract syntax tree
Match rules: check whether a registered instrumentation rule applies (for example, "this is ListenAndServe in net/http")
Inject code: weave tracing/metrics code into the target function before compilation
Pass through to the compiler: hand the rewritten source to the original compile for normal compilation

The binary you end up with has the agent code built in. At runtime there is no extra agent process, no sidecar, and no eBPF program to load.

How this fundamentally differs from a Java agent: Java injects logic at runtime by rewriting bytecode, which incurs runtime overhead (every class load passes through a transformer). Go compile-time instrumentation does all the rewriting during the build, so there is no additional runtime overhead — what you get is an ordinary Go binary that happens to contain tracing code.

This is also what makes the approach so CI/CD-friendly: in your build pipeline you replace go build with otelc go build. No changes to your deployment architecture, no Pod spec edits, no DaemonSet.

3. What v1 Supports: Coverage from net/http to gRPC

As the first stable release, v1 focuses on the most heavily used library categories in the Go ecosystem:

Library	Type	Signals produced
net/http	HTTP client/server	Trace spans + HTTP semantic attributes (method, status_code, route)
Database/SQL	Database access	DB spans + statement summary + connection information
google.golang.org/grpc	RPC framework	Client/Server spans + gRPC semantic attributes
github.com/redis/go-redis	Redis client	Redis command spans
Go runtime	Runtime metrics	GC, goroutine, memory metrics

A few key design decisions:

Rule-based architecture. The instrumentation logic for each library is defined as a rule, and a rule describes the package paths and function signatures to match plus the code template to inject. This means the community can contribute new rules independently, without touching the core framework. After v1, supporting a new library essentially means submitting a new rule.

Semantic Conventions compliance. Every span and metric produced follows the OpenTelemetry Semantic Conventions — attribute names, span names, and metric units are fully standardized. Whatever backend you use (Jaeger, Tempo, Simple Log Service, ARMS), the data means the same thing.

Automatic discovery. By default, otelc scans your go.mod dependency tree and automatically finds and enables every library covered by a registered rule. You don't have to declare "instrument net/http" — if you use it, it gets instrumented.

v1 deliberately prioritized a focused, high-quality core over breadth. Every supported library has been through full correctness tests and performance benchmarks. Later releases will keep expanding coverage.

4. Get Started in 3 Minutes: Add Tracing with One Command

Install otelc:

go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest

Option 1: replace the build command directly

otelc go build -o myapp .

That's it. The resulting myapp already has tracing code built in. Set OTEL_EXPORTER_OTLP_ENDPOINT at startup and trace data will flow to your Collector automatically.

Option 2: leave the build command unchanged (CI/CD friendly)

otelc setup 
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'" 
go build -o myapp .

This fits better when you already have a complex Makefile or CI pipeline — you add two lines of setup to the build environment and leave the existing go build command alone.

Dockerfile integration example:

FROM golang:1.23 AS builder
RUN go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest
WORKDIR /app
COPY . .
RUN otelc go build -o /myapp .
FROM gcr.io/distroless/base
COPY --from=builder /myapp /myapp
ENTRYPOINT ["/myapp"]

Image size is unaffected: otelc is only used during the build, and the final image contains only the compiled output.

5. Choosing Among Three Paths: Compile-Time vs. eBPF vs. Manual

Go observability now has three complementary paths, not competing ones:

Dimension	Compile-time instrumentation	OpenTelemetry eBPF Instrumentation (OBI)	Manual instrumentation (Go API)
Prerequisites	Able to recompile the source	Linux kernel ≥ 4.x, privileges	Able to modify the source
Code intrusion	None	None	High
Runtime overhead	Very low (code is inlined)	Low (collected in kernel space)	Depends on the implementation
Depth of coverage	Function level (including third-party dependencies)	Protocol level (HTTP/gRPC/SQL)	Any granularity
Multi-language support	Go only	Go/Java/Python/Node.js, etc.	Go only
Deployment changes	Change the build command	Deploy a DaemonSet	Change code + redeploy
Best fit	Platform engineering teams rolling out observability uniformly	Existing services, multi-language clusters	Business logic that needs custom spans

How to decide:

If you can rebuild your services and want zero-code coverage with low overhead → compile-time instrumentation
If you have many existing services, don't want to recompile them one by one, or run a mixed-language cluster →OBI
If you need custom spans inside specific business logic (business semantics such as "user places order") → manual instrumentation

You can combine all three. Compile-time instrumentation covers the generic spans from the standard library and third-party dependencies, manual instrumentation adds business-semantic spans, and the two sets of spans stitch into one trace automatically. eBPF then serves as the fallback for older services you can't recompile yet.

In practice, for a typical Go microservice cluster the most pragmatic strategy is this: use compile-time instrumentation plus a little manual instrumentation for new services; cover existing services with eBPF first, then move them to compile-time instrumentation in CI over time.

6. Community Collaboration and Next Steps

How this project came about is itself an interesting case of open-source collaboration.

In early 2025, Alibaba and Datadog were each exploring Go compile-time instrumentation internally and discovered each other's work. Rather than build separately and then fight over the standard in the community, the two companies merged their efforts into the OpenTelemetry community and created a dedicated Special Interest Group (SIG) under CNCF to drive the work in a vendor-neutral way.

Over a year and a half, the project went all the way from PoC to stable. Key milestones:

SIG formed (2025 Q1): technical direction and governance structure settled

Core framework landed (2025 H1): rule engine, AST rewriting, test infra

Community growth (2025 H2): new contributors joined through CNCF LFX Mentorship — among them Azhar Momin, who went from mentee to approver

v1 released (2026 Q3): first stable version, covering 5 core library categories

What's on the roadmap next:

More instrumentation rules: Kafka, MongoDB, AWS SDK, and others
Registry integration: discover and install community-contributed rules through the OpenTelemetry Registry
Build performance optimization: reduce otelc's impact on compile time
Validation in more scenarios: production case studies and performance benchmarks

7.Final Thoughts

Go's observability gap is finally closed.

If you're a platform engineer running observability infrastructure for dozens or hundreds of Go services, otelc go build may be the highest-ROI change you can make: one command, full coverage, nothing intrusive.

If you maintain a library or are simply interested in OpenTelemetry, we'd welcome your rule contributions. Writing an instrumentation rule for a library is far easier than implementing an SDK wrapper from scratch — a rule is essentially a declarative description of what code to inject, in which function, at which position.

Resources:

Project repository: github.com/open-telemetry/opentelemetry-go-compile-instrumentation
Quick-start docs: Getting Started in the project README
CNCF Slack channel: #otel-go-compile-instrumentation
OBI: github.com/open-telemetry/opentelemetry-go-instrumentation

STAROps Synthetic Monitoring Intelligent Analysis in Practice: From Task Analysis to Cross-Domain APM Fault Demarcation

ObservabilityGuy — Fri, 04 Sep 2026 05:46:11 +0000

This article explains how STAROps integrates Synthetic Monitoring, SLS, UModel, and APM to automatically analyze external access anomalies and accurately pinpoint cross-domain faults.

When a user reports that "the site won't load" or "the API suddenly got slow", network monitoring usually flags the anomaly quickly. The hard part comes next: did the fault occur on the network path between the user and the server, or inside the server-side application? Does it hit one region or ISP only, or every user? Should you call the network team, the application team, or the owner of a downstream dependency?

Synthetic Monitoring is the tool that watches this external access experience. Probe nodes spread across regions and ISPs follow fixed rules to simulate real users visiting a website, an API, or a domain name, and they record data for each stage: DNS resolution, TCP connection, TLS handshake, and server response. Traditional synthetic monitoring reports, however, are better at showing what happened than at explaining why it happened.

STAROps brings intelligent O&M analysis to observability data. Its Agent automatically queries task configuration, synthetic monitoring logs, and application call chains, then links evidence that used to sit in separate systems. In the end it answers four questions: where the anomaly occurred, which user paths it affected, which side owns the problem, and what data backs the conclusion.

This article is for product managers, developers, and O&M engineers who are new to Synthetic Monitoring intelligent analysis. It explains why STAROps connects Synthetic Monitoring, SLS, UModel, and APM, and how that chain moves you from spotting an anomaly to explaining it and pointing at a fix. By the end, you will understand where traditional synthetic monitoring analysis breaks down, how STAROps assembles evidence across systems, and how a network problem is told apart from a backend one.

Getting Started: Key Concepts in the Analysis Chain

Synthetic Monitoring: Probe nodes actively visit a target website, API, or domain name and continuously check availability and performance from the user's point of view.
STAROps Agent: The intelligent analysis unit that queries data, aggregates evidence, judges faults, and generates reports. Referred to as the Agent below.
SLS: Alibaba Cloud Simple Log Service, which stores the raw execution record of every synthetic check. Think of it as the log warehouse you search for factual evidence during analysis.
UModel: A unified model that describes observable objects such as tasks and applications, plus the relationships between them. Think of it as a map that tells the Agent who each object is, where its data lives, and how the objects connect.
APM: Application Performance Monitoring, used to watch how a request executes after it reaches the backend. One complete request is a Trace, and each processing step within it is a Span.
Workspace: An isolated scope of observability data and resources. The Agent must pin down the Workspace before it can locate the right task, logs, and APM data.

1. Background: Gaps in Traditional Synthetic Monitoring Analysis

Figure 1: Main gaps in traditional synthetic monitoring analysis

Traditional analysis usually starts with a synthetic monitoring report: check availability rate, response time, and error codes, then switch by hand between logs, task configuration, and APM pages to look for a cause. These metrics work as a health overview, but they rarely add up to verifiable root cause evidence, so four gaps remain in troubleshooting.

First, task context is missing. The same error code can mean completely different things under different task types, target addresses, DNS configurations, assertion rules, and monitoring point coverage. Repeating the error code alone tells you nothing about whether the cause is a misconfiguration, a broken target service, or a failure at one protocol stage.

Second, raw evidence is missing. OpenAPI is the standard query interface the system exposes, and central-side metrics are already aggregated statistics; both suit stable queries and health overviews, but neither carries all the detail an ad-hoc drill-down needs. Stage durations, city and ISP, target IP, CNAME, raw DNS answers, response headers, assertion results, and Trace context usually live only in the raw synthetic monitoring records in the user's SLS.

Third, protocol stages are not opened up. A single HTTP check may run through DNS, TCP, TLS, request send, first-packet wait, HTTP response, response download, and content assertion. Look only at total duration and the final error code, and problems from different layers blur together.

Fourth, synthetic monitoring and the backend application are not linked. Synthetic Monitoring can tell you that external access failed or slowed down, but once the request enters the backend, the problem may sit in business logic, the database, the cache, or a downstream dependency. In APM, the server span marks where the backend receives and handles the request, and child Spans record internal steps such as database, cache, and downstream calls. Without that evidence, the Agent cannot reliably pinpoint a problem inside the backend.

Synthetic Monitoring intelligent analysis therefore needs a complete chain: task identification → data location → raw sample query → field filtering and combined analysis → protocol-stage drill-down → link diagram rendering → cross-domain APM alignment → final fault demarcation.

2. Core Capabilities

Figure 2: The five core capabilities of STAROps Synthetic Monitoring intelligent analysis

1. Synthetic Monitoring Task Analysis

Every synthetic monitoring task carries a unique taskId. Once you supply a taskId, a task name, or a target address, the Agent locates the matching synthetics.task entity — the structured description UModel keeps for that task. It then reads the task type, target, probe frequency, monitoring points, timeout, assertions, and Trace configuration, and queries raw synthetic monitoring samples within the specified time range.

The analysis does more than compute sample count, success rate, availability rate, response time, and error distribution. It also filters, aggregates, and compares by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Even when a task looks stable overall, single-task analysis still checks protocol stages, localized slow points, and configuration risks instead of simply reporting "running normally".

Coverage today includes HTTP/HTTPS, Ping, TCP, UDP, DNS, DNSTRACE, SMTP, POP3, FTP, Traceroute, MTR, API, Multi, Browser, WebSocket, and SSL. Each task type follows its own analysis path: Ping focuses on packet loss and RTT, API and Multi on failed requests and steps, Browser on the main document, page timing, and the resource waterfall.

RTT (round-trip time) is the time data takes to travel from the probe to the target and back. API, Multi, and Browser correspond to API requests, multi-step tasks, and browser page loads respectively.

2. Intelligent Workspace Inspection

Single-task analysis answers "why is this task failing"; Workspace inspection answers "which tasks in this group deserve attention first". A Workspace is an isolated observability space that holds the tasks, logs, and related resources belonging to one user or business scope.

Workspace inspection first rolls up all synthetic monitoring tasks into an overall health view, then drills down into the tasks that matter. Task selection weighs status severity, failure count, availability rate, latency, error concentration, failing-stage concentration, and how long the anomaly has lasted, so the ranking never rests on a single metric.

Inspection supports three cycle scenarios:

High-frequency monitoring: looks at the last 30 minutes for wide-scale unavailability, clustered timeouts, and high-priority risks, and analyzes the Top 1–3 tasks in depth.
Daily diagnosis: analyzes the last 24 hours against the same window the previous day, and reports trends, lingering risks, and priority tasks.
Weekly review: analyzes the last 7 days against the previous week, and identifies persistent anomalies, repeated fluctuations, recovered risks, and administration items.

3. Fine-Grained Protocol Diagnosis

The Agent picks protocol evidence to match the task type rather than explaining every task with one set of metrics:

HTTP/HTTPS: analyzes stage by stage along DNS → TCP → TLS → request send → first-packet wait → HTTP response → response download → assertion.
DNS: checks the query object, DNS Server, RCODE, CNAME, resolved IP, SOA, expected match, and whether the domain name exists.
WebSocket: additionally checks whether the HTTP Upgrade completed.
API and Multi: pinpoint the failing request or step.
Browser: separates main-document, resource-loading, page-performance, and operation-step anomalies.

If network protocols are new to you: DNS translates a domain name into an IP address, TCP establishes a reliable connection, and TLS encrypts the traffic and validates the certificate. RCODE is the result status returned by the DNS server, CNAME marks a domain alias, and SOA records the basic information of the domain's authoritative zone.

Diagnosis covers configuration problems as well as data problems. A DNS query object mistakenly set to a full URL, an HTTP 200 whose content assertion fails, a disabled task, a query window shorter than the probe interval, or no matching samples in SLS — each is flagged on its own and never lumped in as a target service anomaly.

4. Request Link Visualization

The synthetic monitoring access link diagram is generated dynamically from real samples. The first node is always the probe, and only the protocol stages the request actually reached appear after it. If the request fails at DNS, the diagram stops at DNS and does not draw TCP, TLS, or the backend service.

The diagram follows a "stage node → anomaly node" structure. A DNS anomaly, for example, renders as "probe → DNS resolution → resolution anomaly (NXDOMAIN)" rather than jumping straight from the probe to the anomaly. Normal nodes, affected paths, and the final root cause use different styles, so an anomaly propagation path is not mistaken for the root cause.

Figure 3: Synthetic monitoring access link diagram generated from the actual execution stages in STAROps task analysis

The diagram does not depend on APM. With no Trace available, it still shows the synthetic monitoring access path; when a backend Trace exists, an APM call chain diagram is added to show the backend entry point, API, business processing, downstream dependencies, and the specific point of failure. The trace_id ties the two diagrams to the same request.

5. Cross-Domain APM Fault Demarcation

When an HTTP, API, Multi, or Browser task carries Trace evidence such as trace_id, traceparent, or traceInfo, the Agent picks a failed or high-latency sample and queries the APM Trace itself, instead of merely telling you to look it up.

A Trace is the full path of a request from the probe into the backend and on to other components; a Span is one step along that path. Linked analysis aligns four kinds of Span:

Synthetic monitoring client span: the external view, from the probe sending the request to receiving the response.
Backend server span: the entry view, from the request reaching the application to the application returning a response.
Local business Span: the execution of a Handler, business method, or internal computation.
Downstream dependency Span: a call to a database, cache, RPC, or external HTTP service.

By comparing synthetic monitoring stage durations, the server span, child Spans, and the first concrete error, the Agent can place the fault on the network path, the path before the application entry, the backend service, or a downstream dependency — or state that evidence is insufficient. A 502 from a backend API may be only the surface symptom; if the first concrete error appears in a downstream DNS, database, or external service Span, the conclusion moves to the downstream dependency.

3. Synthetic Monitoring Architecture and Technical Approach

Figure 4: Analysis architecture combining UModel, SLS, and APM

Data Organization: UModel and SLS Working in Two Layers

Think of UModel as the map and SLS as the field notes. UModel holds relatively stable object information: which task this is, what it targets, which Workspace it belongs to, where its data lives, and how it relates to other objects. SLS holds the execution facts that keep arriving: which probe hit which target at what time, which protocol stages it went through, and whether it succeeded.

Creating a synthetic monitoring task produces a synthetics.task entity in UModel that holds metadata such as taskId, task type, target address, probe configuration, Workspace, SLS data coordinates, and Trace configuration. Every real probe run then writes one or more execution records into SLS. One task entity therefore maps to many synthetic monitoring samples; each probe run does not create a new entity.

When a request arrives, the Agent first queries this map through UModelSearch to locate the task entity and its log location, then enters the matching SLS Project and Logstore to query samples by taskId and time range. A Project is the isolated resource space in SLS, and a Logstore is the data container inside it that holds one class of logs.

Analysis Strategy: Overall Aggregation, Risk Screening, and Protocol Drill-Down

With the samples in hand, the Agent first sizes up the scope and distribution of the anomaly, then drills down step by step into specific protocol stages. Single-task analysis aggregates by time, region, ISP, monitoring point, target IP, DNS Server, error code, and failing stage. Workspace inspection instead rolls up all tasks first, then screens out the ones with low availability rate, high latency, concentrated errors, or persistent anomalies.

After screening, the Agent reads representative failed or high-latency samples and takes a different analysis path per task type. An HTTP task, for instance, is checked in order for DNS, TCP, TLS, request send, first-packet wait, response download, HTTP status code, and content assertion; a DNS task goes on to read RCODE, the number of answer records, CNAME, resolved IP, SOA, DNS Server, and the match result.

This "overall aggregation → risk screening → protocol drill-down → representative sample" approach separates resolution failures, connection problems, certificate issues, slow server-side processing, and content assertion failures, instead of explaining only the final error code.

Cross-Domain Linkage: From the Synthetic Monitoring Domain into the APM Domain

The analysis chain can continue into APM only when the synthetic monitoring sample carries Trace information. For any task with Trace enabled, the system generates a synthetic APM service and uses the same_as relationship in UModel to map synthetics.task to that service permanently. For the Agent, this relationship is the signpost from the synthetic monitoring domain into the APM domain.

same_as answers "which synthetic APM object corresponds to this synthetic monitoring task"; trace_id answers "which applications and call nodes did this particular request pass through inside the backend". The first is a stable task-level relationship, the second a precise per-request association. Note one thing in particular: same_as does not equate the synthetic monitoring task with the real backend application.

Inside APM, the Agent looks at the synthetic monitoring client span, the backend server span, local business Spans, and downstream dependency Spans together:

If synthetic monitoring duration rises and the server span rises with it, the problem leans toward the backend service.
If most of the time sits in a database or downstream call, you can narrow it to that dependency.
If the client span is slow while the server span is short, the problem more likely lies in the network, CDN, gateway, queuing, or another uninstrumented segment.
If DNS, TCP, or TLS already failed and no server span exists, the request never reached the application.

If a sample has a trace_id but complete backend Spans are missing, the Agent explicitly marks "insufficient APM evidence" rather than reading "no data found" as "backend is normal".

Design Principle

The STAROps Agent does not hand a fixed report to a large model for summarizing. It builds a query plan first, then constrains its conclusions with real data. UModel serves as the semantic control plane that identifies objects and relationships, and the user-side Logstore serves as the data plane that holds observed facts: indexed fields screen samples fast, and SPL queries then fetch and process finer protocol details. Here, an index is a pre-built field for fast retrieval, and SPL is the query language used to filter, aggregate, and transform log data.

Different tasks load different evidence models, normalizing heterogeneous logs into five layers: task, single execution, protocol stage, blast radius, and call chain. When a sample carries Trace information, the Agent follows trace_id into APM and aligns the external client span with the backend server span, business Spans, and downstream dependency Spans. The output is not an empirical guess a large model made from field names; it is a traceable fault demarcation conclusion constrained by raw samples, aggregated results, protocol semantics, and cross-domain Traces.

End-to-End Analysis Chain

Parse the user input and decide whether the scope is a single task or a Workspace.
Query synthetics.task to obtain the task configuration and SLS data coordinates.
Fix the time window and calibrate time units.
Aggregate overall results and screen for risk using indexed fields.
For abnormal tasks, read representative samples and unindexed protocol fields.
Enter the dedicated analysis for DNS, HTTP, Browser, API, Multi, or another type as appropriate.
If a Trace exists, pick a failed or high-latency sample and query the APM call chain.
Align task configuration, synthetic monitoring stages, blast radius, and Span evidence, then output the fault demarcation conclusion, link diagram, recommended actions, and data limitations.

The core strategy is "overall aggregation → risk screening → protocol drill-down → representative sample → cross-domain alignment". It keeps query volume under control across long time windows and large Workspaces while preserving the raw evidence needed to pinpoint the failing stage and the root cause.

4. How to Use

For task analysis, simply supply a taskId, a task name, or a Workspace, for example:

Analyze the synthetic monitoring tasks over the last 30 minutes.

You do not have to write log queries yourself. The Agent turns a natural-language question into concrete steps: task location, sample query, protocol analysis, and APM linkage.

The Agent then completes the following steps automatically:

Identify the scope and confirm whether this is single-task analysis or Workspace inspection.
Find synthetics.task through UModelSearch or the task query capability.
Locate the user-side SLS Project and Logstore.
Query raw synthetic monitoring samples within the fixed time window.
Run protocol-stage analysis according to the task type.
If Trace evidence exists, go on to query the APM Trace.
Output the complete Synthetic Monitoring intelligent analysis report.

For ongoing inspection, choose the high-frequency monitoring, daily diagnosis, or weekly review template. The inspection scans the synthetic monitoring tasks in the Workspace on a schedule, screens out risky tasks, and generates a report automatically.

A "Long-Running Task" in STAROps is an analysis task that repeats automatically on a schedule; a "Digital Employee" is an Agent pre-configured with skills, tools, and data permissions. Combine the two and synthetic monitoring inspection runs on schedule and pushes its results to the people you designate.

Synthetic Monitoring Task Analysis

Figure 5: Starting a synthetic monitoring task analysis in the STAROps console

In the STAROps console, you can ask about synthetic monitoring task details directly, for example "Analyze the synthetic monitoring tasks over the last half hour", "Analyze why a given synthetic monitoring task is failing", or "Show the synthetic monitoring tasks whose availability rate dropped sharply in the last day".

Synthetic Monitoring Inspection Tasks

Figure 6: Opening Long-Running Task in STAROps

Click "Long-Running Task" in the upper-left corner of the STAROps console.

Figure 7: Creating a synthetic monitoring inspection task and configuring the Workspace

Click "Create Task" in the upper-right corner of the page, select a Digital Employee and the Workspace the task belongs to, and enter a prompt such as "synthetic monitoring inspection".
Select one of the three synthetic monitoring inspection templates or combine them, and describe the inspection details in text; by default the inspection runs at Workspace level.

Figure 8: Configuring notification recipients and adding the job to Long-Running Task

4. Click "Configure Now" to add notification recipients, then click "Add to Long-Running Task" once you have made your selection.

5. Confirm the configuration is correct, then click "Confirm Execution".

Figure 9: Viewing the inspection report generated by the Long-Running Task

6. Click "Report" at the top of the page to view the detailed inspection results.

5. Fault Demarcation Logic

1. Locate the Layer Where the Request Stopped

An HTTP call chain runs DNS → TCP → TLS → request send → first-packet wait → HTTP response → download → assertion. The deepest stage the Agent can observe tells it whether the request reached the service entry point.

A DNS failure with no TCP evidence, for example, means the request never established a connection. An HTTP 502 shows that DNS resolution, the TCP handshake, and the response header return all completed, so the problem cannot be a first-packet wait timeout. An HTTP 200 with a failed assertion means the service is reachable and the failure sits in the response content or the business logic.

2. Determine the Scope of Impact and Temporal Pattern

Simultaneous failures across multiple cities and ISPs point to the target service, a public entry point, or a cross-network path. Concentrated failures in a single city or ISP suggest a regional network or ISP link issue. If only some monitoring points fail within the same city and ISP, the Agent drills down by client_id, DNS Server, local DNS, and egress path when these fields are available.

The Agent also separates continuous, sudden, intermittent, and recovered anomalies over time. It never generalizes a single spike into a continuous failure, and averages never stand in for P95, maximum values, or anomalous time buckets.

3. Prioritize DNS Evidence Over Outer Error Codes

The DNS error_code reflects the probe's local judgment, while RCODE is the protocol-layer response from the DNS server — the two are not interchangeable.

When raw_exception is present, the Agent reads rcode, rcode_name, and the answer record count first, to tell NXDOMAIN, SERVFAIL, REFUSED, FORMERR, NOTIMP, and NODATA apart (NODATA means RCODE is NOERROR but the target record is empty). If those fields are missing, it draws evidence from dns_raw, dns_raw_inner, message, SOA, EDE, and DNSSEC information, and states the confidence level of the conclusion explicitly.

DNS analysis also compares the DNS Server, city, ISP, monitoring point, CNAME, resolved IP, and target edge zone. Note that if error_code=0 but only an SOA record is returned without A, AAAA, or CNAME records, this only indicates that the current rule evaluation passed; it does not prove that the target record resolved normally.

4. APM Evidence Determines the Backend Boundary

If total synthetic monitoring duration rises while DNS, TCP, and TLS stay normal, and the server span rises with it, the problem lies in the backend service.
If a specific business sub-Span within the server span accounts for most of the duration, the application's internal processing is slow.
If the first concrete error, or most of the duration, appears in a downstream Dependency Span, the problem lies in that downstream dependency.
If the client span is markedly slow while the server span is very short, the problem more likely lies in the path before the application entry, the gateway, the CDN, or an uninstrumented component.
If DNS, TCP, or TLS fails and there is no server span, the request never entered the application.
If a trace_id exists but APM has no data or the Spans are incomplete, the evidence is insufficient, and you cannot conclude from it that the backend is healthy.

The gap between the client span and the server span is not pure network time; it may also include time spent in the CDN, gateway, ingress proxy, queueing, and uninstrumented components. The final conclusion therefore has to draw on synthetic monitoring stage fields, target access evidence, and Span parent-child relationships together.

6. How Cross-Domain APM Analysis Works

Figure 10: Analysis flow from a synthetic monitoring task into the APM call chain

A full cross-domain APM analysis takes four steps:

Starting from the synthetic monitoring task, the Agent uses UModelSearch to locate the synthetics.task entity, confirming the task type, target address, Workspace, and data coordinates.
The Agent queries the synthetic monitoring SLS samples to complete the analysis on the synthetic monitoring side; if the samples carry Trace evidence, it extracts a representative trace_id.
The Agent uses the trace_id to query the APM Trace, retrieving the server span, business processing Span, and downstream Dependency Span for the same request within the backend application.
The Agent aligns the evidence from the synthetic monitoring side and the APM side to form a cross-domain fault demarcation conclusion.

Evidence alignment follows these rules:

Synthetic monitoring is slow and the APM server span slows with it: the problem most likely lies in the backend service.
Synthetic monitoring is slow and the APM downstream Dependency Span slows with it: the problem most likely lies in a backend downstream dependency.
Synthetic monitoring is slow but the APM server span is very short: the problem leans toward the network link, the ingress layer, the gateway, or the path before the application entry.
The DNS, TCP, or TLS stage fails and there is no server span: the request never entered the application, so look first at the path before the application entry.
Trace evidence exists but no backend server span can be found: you cannot conclude that the backend is healthy. The Agent can only flag "insufficient APM evidence" and check Trace reporting, sampling, Region, Workspace, and APM integration.

The APM linkage presents evidence as two link diagrams: one shows the external synthetic monitoring access path, the other the backend Trace path, and trace_id aligns them as the same request. Together they show whether the request reached the service entry point and, once it was inside the application, exactly which Span failed.

Figure 11: Aligning the synthetic monitoring access path with the backend APM Trace via trace_id

7. From "Seeing the Anomaly" to "Knowing What to Do Next"

The value of STAROps is not longer reports; it is a shorter distance from alert to action. It organizes task configuration, raw SLS samples, monitoring point distribution, protocol stages, and APM Traces into one continuous chain of evidence. Users can start from a taskId, task name, or Workspace and work out step by step whether an anomaly comes from a global outage, a regional network, a single ISP, a specific monitoring point, the task configuration, or the backend application and its downstream dependencies.

For experienced engineers, these diagnostic capabilities cut down the constant switching between the Synthetic Monitoring console, SLS, and APM to piece evidence together by hand. For anyone handling a synthetic monitoring fault for the first time, they turn an expert's troubleshooting sequence into a repeatable analysis flow. At Workspace level, the Agent can also pick high-risk targets out of a large pool of tasks and keep running high-frequency monitoring, daily diagnosis, and weekly reviews.

Appendix: Glossary of Terms

Term	Definition
Synthetic Monitoring	Simulates user access from distributed probe nodes to proactively check the availability and performance of target services.
STAROps Agent	An intelligent analysis unit responsible for invoking tools, querying data, aggregating evidence, and generating diagnostic conclusions.
Workspace	A mutually isolated set of observable resources and data scopes.
taskId	The ID that uniquely identifies a synthetic monitoring task.
UModel	A unified model describing observable entities and their relationships, used to locate analysis targets and navigate across domains.
SLS	Alibaba Cloud Simple Log Service (SLS), used to store and query synthetic monitoring execution samples and related logs.
Project / Logstore	A Project is a resource isolation space in SLS, and a Logstore is a data container within it that stores a specific category of logs.
SPL	A query language used to filter, aggregate, and transform log data.
synthetics.task	The entity type in UModel that represents a synthetic monitoring task.
APM	Application Performance Monitoring, used to observe the execution of requests after they enter the backend application.
Trace / Span	A Trace is the complete call chain of a single request, and a Span is an individual processing step within that chain.
same_as	The UModel relationship that associates a synthetic monitoring task entity with its corresponding synthetic monitoring APM service.

Conclusion

Synthetic Monitoring captures the external user experience before a request reaches the service, APM observes how the request executes once inside the application, SLS preserves the factual evidence, and UModel locates the objects and connects the different data domains. STAROps organizes these four into a continuous chain that runs from task discovery through sample aggregation and protocol diagnosis to backend fault demarcation.

This practice turns Synthetic Monitoring from a system that only displays abnormal metrics into an analysis entry point that explains anomalies and supports decisions. Instead of a raw list of fields, users get verifiable answers: where the request stopped, who the anomaly affected, which side owns the fault, what the evidence is, and what to tackle first.

From Reinventing the Wheel to Embedding STAROps via OpenAPI: Why NIIMBOT Chose a Cloud AIOps Foundation

ObservabilityGuy — Fri, 04 Sep 2026 02:00:02 +0000

NIIMBOT integrated Alibaba Cloud's STAROps and UModel via OpenAPI into its SRE platform for automated, cross-domain AIOps diagnostics.

Background

When a technically mature team has moved its entire stack to the cloud and already built a complete observability stack, the key question is no longer "can we build it?" but "what should we build ourselves, and what should we hand to the cloud?" That was the decision facing NIIMBOT. With an established in-house SRE platform, should the company keep building its own observability data foundation and operations digital twin, or adopt a more complete set of ready-made capabilities?

NIIMBOT is dedicated to making things easier to manage through innovation. Since its founding in 2012, the company has continued to advance smart label printing hardware, cloud printing service platforms, and enterprise performance management systems, making label printing smarter and easier. NIIMBOT products are now used across retail, telecommunications, office settings, industry, healthcare and laboratories, and homes, reaching more than 22 million users worldwide.

Business Challenges

As the business expanded rapidly around the world, its systems grew exponentially in scale and complexity, pushing operational complexity to a new level. NIIMBOT runs its business systems on Alibaba Cloud and had already assembled a comprehensive observability stack: Real User Monitoring (RUM) for frontend experience data, Managed Service for Prometheus for container and cloud resource metrics, Application Real-Time Monitoring Service (ARMS) for application performance tracing, and Simple Log Service (SLS) for logs. A self-managed Grafana instance connected to these cloud data sources to provide unified dashboards. Even with this extensive toolset, three structural problems remained.

(1) No Clear Global Topology: Internal Calls and External Dependencies Sat in Separate Graphs

The business systems ran in the cloud, but one question remained hard to answer: which APIs does an application call, and which cloud resources does it depend on? ARMS showed some internal API call relationships, while RDS instances, Redis instances, message queues, and other external dependencies remained scattered across their own monitoring views. The two could not be combined into a single dynamic graph. As the business grew, mapping topology by hand became increasingly inefficient — dependencies documented today could be out of date after next week's release.

(2) Multi-Dimensional Observability Data Was Complete, but Disconnected

Metrics, logs, traces, and events were all being collected. The problem was that collecting everything is not the same as connecting everything. Troubleshooting an incident meant context-switching repeatedly among metric charts, traces, logs, and change events, matching timestamps by hand, with no single thread automatically linking the different data dimensions. The data was there; people still had to piece the clues together.

(3) Alert Noise Made Cross-Domain Root Cause Analysis Take Tens of Minutes

When an incident occurred, every component across the stack — frontend, backend, middleware, database, and containers — raised alerts at once, burying the true source of the problem. Root cause analysis depended heavily on experience: engineers had to decide by instinct where to begin, and locating and analyzing a cross-domain fault routinely took tens of minutes.

These challenges raised a question specific to NIIMBOT. The team was capable of building its own SRE platform, but it also understood the cost of automatically correlating multi-dimensional observability data and maintaining a dynamic, real-time view of topology. Both required substantial investment and continuous upkeep. Was this particular wheel really worth building in-house?

Solution: A Multi-Dimensional Data Foundation, a UModel Digital Twin, and OpenAPI Integration with the In-House SRE Platform

After technical discussions with Alibaba Cloud, NIIMBOT found that the Alibaba Cloud observability system already provided the two capabilities it needed most: automatic correlation across observability data dimensions and dynamic awareness of end-to-end topology. The correlation coverage was broad, and the topology updated automatically — exactly what the team had planned to build. NIIMBOT therefore decided to stop reinventing this wheel. Alibaba Cloud would provide the observability data foundation and operations digital twin; the in-house SRE platform would call STAROps diagnostics through OpenAPI and focus on orchestration and closed-loop workflows tailored to NIIMBOT's business. The solution was implemented in four layers.

(1) A Unified Multi-Dimensional Observability Data Foundation: Bringing Four Existing Collection Capabilities onto One Data Plane

Rather than replace the RUM, Prometheus, ARMS, and SLS deployments already in use, NIIMBOT used Cloud Monitor 2.0 (CMS 2.0) to collect, store, view, and analyze metrics, logs, traces, events, and changes in one place. Frontend experience data from RUM, container and cloud resource metrics from Prometheus, application performance and traces from ARMS, and business logs from SLS now converge on the same data plane. This provides a consistent foundation for topology modeling and intelligent diagnostics. In short, the data no longer sits in separate silos; it enters a shared correlation layer with consistent definitions.

(2) The UModel Operations Digital Twin: Automatic End-to-End Topology Modeling with Real-Time Updates

This capability is the main reason NIIMBOT shifted from building to adopting. UModel models business systems automatically and constructs an end-to-end topology spanning internal API calls and external cloud resource dependencies — frontend → backend → middleware → database → containers — in a single graph. The graph updates in real time as applications are deployed, dependencies change, or workloads scale, with no manual upkeep. Clicking any node brings up its associated observability data. The topology is not a static architecture diagram but a living map backed by current data. UModel's mature correlation coverage and dynamic topology awareness spare the team the ongoing work of building and maintaining these models itself.

(3) STAROps Intelligent Diagnostics: Automatic Cross-Domain Root Cause Analysis on the UModel Topology

With a unified data foundation and the UModel topology in place, STAROps can perform root cause analysis (RCA) automatically when an incident occurs. It follows upstream and downstream relationships in UModel, finds relevant nodes, retrieves their observability data, and quickly returns investigation steps and analysis results. What once required engineers to guess where to begin and query multiple systems by hand becomes AI-driven, cross-domain correlation. In practice, STAROps performs reliably on single-domain analysis of infrastructure resources, such as resource utilization levels for a Pod or an anomalous cloud resource metric. In more complex cases — frequent garbage collection in a Pod, for example, which spans the container layer and application-level JVM metrics — STAROps can start from application monitoring, follow the topology to the JVM metrics, and pinpoint the root cause. This is the capability that turns "the data is there, but people assemble the clues" into "AI connects the data dimensions automatically."

(4) STAROps Integrates with the In-House SRE Platform: Embedding Diagnostics via OpenAPI

NIIMBOT engineers no longer need to leave their familiar SRE platform for another console when troubleshooting. Through OpenAPI, the in-house platform calls STAROps directly as a diagnostic engine and receives a live stream of its analysis and diagnostic results. With one click inside their own SRE platform, engineers can follow the STAROps reasoning process and see its conclusions in real time, completing full-domain diagnosis without leaving the platform.

This model embeds capabilities instead of replacing platforms. NIIMBOT gains the full diagnostic power of the Alibaba Cloud observability system while preserving the autonomy of an in-house SRE platform built around its own business logic. The cloud provides the foundation; the customer's platform remains the interface.

Business Value: Let the Cloud Handle the Heavy Lifting and Refocus on the Business

(1) One-Click, Closed-Loop Full-Domain Diagnosis Inside the In-House SRE Platform

Previously, troubleshooting a cross-domain fault meant switching repeatedly among ARMS, Prometheus, SLS, and Grafana. Engineers matched timestamps and assembled clues by hand, and root cause analysis routinely took tens of minutes. Now, a single click in the NIIMBOT in-house SRE platform triggers diagnosis of a business system issue. STAROps follows the UModel topology, correlates multi-dimensional observability data across domains, and returns the root cause analysis. The operations team no longer has to sift through each data dimension or hunt for clues layer by layer.

(2) No More Reinventing the Wheel: A Mature Team Puts Its Effort Back into the Business

NIIMBOT once had to commit its own engineering resources to building and maintaining general-purpose operations capabilities such as topology modeling and multi-dimensional data correlation. That costly, maintenance-intensive work is now handled by the Alibaba Cloud observability system and UModel. Freed from reinventing the wheel, the team can focus again on cloud resource management and system architecture optimization — work that contributes more directly to its business. For mature teams, deciding what to build and what to hand to the cloud is a real strategic choice; NIIMBOT has made its own.

(3) Operations Evolves from Reactive Firefighting to Proactive Management

The operations team once spent its days waiting for alerts, chasing faults, and conducting post-incident reviews. With UModel's dynamically updated end-to-end topology and AI-driven analysis of multi-dimensional observability data, the team has moved from reactive response to proactive inspection and earlier problem detection. While keeping business systems stable, its role has evolved from incident responder to steward of system health, completing its transition to an AIOps operating model.

Looking Ahead: From Diagnostics That Work to Diagnostics That Know NIIMBOT Better

As STAROps becomes embedded in the in-house SRE platform, more core business systems will gain end-to-end topology modeling and intelligent diagnostics, extending the living map and one-click cross-domain diagnosis across the entire business. Each new system onboarded strengthens the AIOps capability of the NIIMBOT in-house SRE platform: wherever the business expands, AIOps coverage follows.

(1) From Trace-Guided Correlation to Automatic Analysis from Any Entry Point

STAROps already pinpoints root causes by correlating across domains along the UModel topology. The next step is to make the result independent of where an investigation begins. Whether an engineer starts from application monitoring or from an infrastructure resource such as a Pod or container, the system should extend upstream and downstream automatically and connect every observability data dimension. The goal is more seamless, intelligent cross-domain correlation, with root cause analysis that no longer depends on choosing the right entry point.

(2) Making the Embedded OpenAPI Integration Smoother

Streaming the diagnostic process back to the in-house SRE platform through OpenAPI is central to this integration. Guided by NIIMBOT's feedback from day-to-day use, the two sides will continue improving the responsiveness and completeness of the stream, making the analysis shown inside the platform more detailed and coherent — and turning embedded capabilities into a genuinely seamless experience.