Skip to the stories
Microsoft 365 & SharePoint

Why AI Agents and ML Models Should Share One Feature Repository

  • Lipsa Senapati
  • 590 views

Separate pipelines for ML models and AI agents drive up cost, cause data drift and fragment governance. One shared feature repository is the fix.

Why AI Agents and ML Models Should Share One Feature Repository

Here's the thing: the teams building predictive models and the teams building AI agents are often reading from the same enterprise data, just through two entirely different plumbing setups. Lipsa Senapati, a data intelligence developer, makes the case that this split is expensive and that a shared, governed feature repository is the practical way to close it.

Two teams, one data estate

The divide tends to follow team lines. Data science groups spend months writing extraction scripts, SQL transforms and statistical feature sets to train models for churn, lifetime value or operational risk. Software teams building agents for triage, customer service or workflow automation need much of the same operational state, but they usually fetch it through ad hoc API endpoints, point-to-point microservice calls and bespoke database lookups.

Nobody planned to build two retrieval stacks for the same customer record. It happened because the two groups were solving different problems on different timelines. That's how duplication gets baked in, and it's hard to unwind once both pipelines are in production.

What the duplication actually costs

Senapati points to three costs. The first is money. Running parallel transformations and storage for near-identical data inflates cloud compute and storage bills. The second is drift. When a model calculates a risk score one way and an agent derives a similar figure from a different query, the two can quietly disagree, and nobody notices until a decision goes wrong. The third is governance. Access rules, lineage and audit controls end up scattered across disconnected silos, which makes them harder to enforce consistently.

Anyone who has inherited a estate full of one-off integrations will recognise the pattern. Each shortcut looks reasonable on its own, and the total becomes technical debt that slows every later change.

Why feature stores matter here

In machine learning operations, a feature is clean, precalculated business context: an account age, a rolling 30-day transaction volume, a payment velocity metric or a risk score built from historical records. Feature stores became standard in ML engineering because they fixed two persistent problems. They removed duplicate transformation logic between training and serving environments, and they reduced the inconsistencies that appear when offline models and online inference engines compute the same thing differently.

Those are the same two problems that show up when agents are built on ad hoc data access. The fix that worked for models is worth borrowing.

Agents have the same data needs as models

Senapati's central argument is that the data requirements of autonomous AI agents look functionally identical to those of traditional ML models. An agent approving a workflow, routing an urgent ticket or triggering a business intervention needs accurate, point-in-time entity context. It can't reliably work that out in the moment.

If an agent has to compute rolling aggregations on the fly, or query fragmented transactional databases during a large language model call, two things tend to go wrong. Response latency climbs into seconds, which degrades the user experience, triggers tool timeouts and pushes up inference token costs. And the context payload itself becomes less trustworthy. Data assembled dynamically is often incomplete, stale or formatted inconsistently across prompt chains, which raises the risk of hallucinated or poorly grounded outputs.

The alternative is straightforward in principle. If the enterprise already has a governed feature repository that computes, tests and caches entity attributes for its models, agents should read from that same authoritative source rather than rebuilding the values themselves.

Four pillars for a unified repository

Senapati proposes a platform-first approach built on four pillars. Each one is a design decision teams can start examining now, whatever tooling they already run.

  1. Treat feature logic as shared, version-controlled software. Stop scattering SQL across notebooks and microservice codebases. Define each transformation once, in a central repository with continuous integration, automated unit tests and schema validation. When a business definition changes, the update should reach both model inputs and agent tools at the same time.

  2. Support dual-read paths. A production repository needs two integrated storage tiers. An analytical columnar lakehouse serves historical training, retrospective bias analysis and offline agent evaluation. A low-latency key-value store serves live inference. The article's design target is demanding: when an agent calls a tool, it should retrieve point-in-time entity state in fewer than 20 milliseconds.

  3. Standardise interfaces for different consumers. Traditional ML pipelines consume numeric tensors, Parquet files and tabular data frames. LLM agents need structured JSON and contextual entity summaries. A flexible serialisation layer on top of the central catalogue lets each framework read the same verified data without anyone rewriting pipeline logic.

  4. Enforce end-to-end lineage, drift detection and audit trails. Agents bring heavier audit and security obligations than a batch model does. A central repository can record an immutable trail from raw source data through transformed features and model inferences to agent actions. That traceability is what makes root-cause debugging and drift investigation practical, and it's what makes compliance reviews answerable.

What happens if teams wait

Many organisations have prioritised fast generative AI prototypes over careful data engineering. That's understandable, but it creates a timing problem. Point-to-point connections can survive a demo. They rarely survive a production environment where schemas change, maintenance backlogs grow and different services hold conflicting versions of the same entity.

My read is that the first sign of trouble will be inconsistency rather than outright failure: an agent and a model giving different answers about the same customer, with no clear way to explain why. Teams that already run a feature store for their models are in a good position to extend it. Teams that don't should start by inventorying which data their agents fetch directly, and which of those fetches duplicate something a model already computes. That list is a sensible first step toward one authoritative source.

Source: Lipsa Senapati · www.informationweek.com

Neighbouring cells