Data Warehouse + lineage graph (foundation)
Every downstream AI initiative breaks because the agent can't resolve "is this the same customer?" across billing + CRM + support.
What
A warehouse that lands raw events from billing, CRM, support, helpdesk, telephony, and 1C-style accounting plus a lineage graph that lets agents join entities across those sources. A separate "golden record" mapping reconciles merchant/customer IDs across systems so an agent asked "is account X churning?" can stitch transactions + tickets + login telemetry without manual joining.
Why it matters
A large payments processor sized this as the single foundation that unlocks churn prediction, support copilots, and merchant self-service — none of which can be built reliably without it. Cost: low-six-figure USD equivalent. Without it, every downstream AI initiative reduces to a demo that breaks in production because the agent can't resolve identities.
End-to-end
- Inventory every source. Billing, CRM, HelpDesk, telephony, 1C, product DB, ad platforms.
- CDC each source into the warehouse with no transformation (Fivetran / Airbyte / custom).
- Build a
golden_recordtable that maps source IDs → canonical entity (merchant_id, customer_id, deal_id). - Author a lineage graph as a YAML or markdown file: which downstream table depends on which source, with column-level mapping for the load-bearing fields.
- Stand up the semantic / metric layer on top (see #142).
- Define a "source freshness" SLA per table; surface in a dashboard the agent reads before answering.
- Ship Q1 MVP for one critical question (e.g., churn). Q2 add SSoT. Q3-Q4 extend coverage.
Gotchas
- Vendor estimates run 3–4× inflated; one team challenged a vendor quote that was inflated ~4× over the independently-rebuilt cost. Always re-cost AI infra independently before signing.
Tools
- ClickHouse / Snowflake / BigQuery
- Fivetran / Airbyte / custom CDC
- dbt for tests + column-level lineage
- A
golden_recordtable or MDM-lite