Shipping AI inside a multi-tenant property platform
Retrieval isolation, per-portfolio configuration, evaluation across thousands of small customers, and cost budgets that hold at platform scale.
A single-tenant AI feature is a demo. A property management platform serving thousands of operators — each with its own portfolio, fee schedule, state rules and document conventions — is a different engineering problem, and most of it has nothing to do with the model.
The constraints that decide whether the feature ships are isolation, configurability, measurability and unit cost. Get those wrong and the pilot works beautifully for the design partner and collapses on customer forty.
Isolation is a retrieval problem before it is a database problem
Row-level security on the tables is table stakes. The leak risk in an AI feature is the retrieval layer: an embedding index that mixes portfolios, a cache keyed on question text rather than tenant, a summary written once and reused across operators.
- Scope every retrieval query by tenant at the index level, not by filtering results after the fact
- Key caches, embeddings and derived summaries on tenant identity so nothing can be served across a boundary
- Never place another tenant's content in the context window, even as a few-shot example
- Log the tenant identifier on every model call so a leak is detectable after the fact, not merely improbable
Configuration without a fork per customer
Operators differ on late fee grace periods, income multiples, document requirements and what counts as an exception. If those differences live in prompts, you end up maintaining a prompt per customer and can no longer reason about behaviour at all.
Keep policy in versioned, typed configuration that code reads. Keep prompts generic and few. The model's job is to read documents and extract structure; the operator's rules apply afterward, in code, where they are reviewable and testable.
Evaluation when every customer is small
A platform rarely has one large evaluation set. It has thousands of tiny ones. Build a shared golden set covering the document types and edge cases you see across the base, and layer per-segment slices — by state, by document vendor, by portfolio size — so a regression that only hurts one segment is visible before release.
Track quality per segment in production too. Platform-wide accuracy averages hide the customer who is about to churn.
Unit economics decide the roadmap
At platform volume, the difference between a two-call and a five-call workflow is a line item. Set a cost and latency budget per run, enforce it in the harness, and measure cost per processed document rather than per month.
The usual wins are unglamorous: cache what is stable, route easy cases to a smaller model, and stop paying a model to do arithmetic that code can do for free.