90% Lower Pipeline Cost, Sub-10 ms Queries: Moving Aithon's Data Layer Off Databricks

Context360 is the data layer under everything Aithon does. It is both the input and the output of Aithon’s intelligence. This is the story of how that layer moved off a Databricks/Spark monolith and onto plain SQL running on AWS, writing directly to Postgres. The migration was also a chance to fix the data model itself, not just swap the engine.

The starting point: Slow, expensive, and mixed

The old layer ran on Databricks, Spark DLT, and Delta Lake. It had three structural problems.

It was slow. The full pipeline took one to two hours and ran once a day. It recalculated all data on every run instead of processing only the new rows. The silver layer alone took about 30 minutes, with 84+ JSON extractions per row, every run.

It was expensive. Databricks spend was about $10K a month and growing fast. It ramped from roughly $0.4K in August 2025 to a peak of about $15K in February 2026.

It was fragile. Every customer’s data sat in shared tables, and ownership depended on complex ID schemes.

Reads had their own problems. The API minted a Databricks token, read through Lakebase, and data only reached RDS through a sync job every 15 minutes. Each layer had a different freshness, so “which layer is stale?” was the first debugging question. The chatbot opened a Databricks SQL Warehouse session that could cold-start for 2 to 4 minutes.

The diagnosis: The wrong-sized tool

Spark, the engine under Databricks, exists for big data: datasets too large for one machine, spread across a cluster. The real test is simple. Does the data fit on one box? A single modern server handles hundreds of GB to several TB, so the bar for needing a distributed cluster is high. Below it, distributed compute is pure overhead: the cost of coordinating many machines, plus the platform fee, with no speed-up.

Context360 is nowhere near that bar. The production numbers:

That is the entire CRM core. Single-digit GB, fitting comfortably inside a single Postgres instance. The one genuinely large dataset is a people feed of about 70M rows. Even with it, the whole data layer is about 165 GB. Not close to a terabyte.

The clearest example: one job that took about an hour on Databricks now runs in minutes on Postgres, and only on the changed rows. The sync workers now use DuckDB, a single-node engine, to read S3. Tooling that fits the scale.

The new stack: dbt, Postgres, and AWS-native compute

	Before	After
Transform	Spark DLT, streaming	dbt: SQL, open source, incremental
Storage	Delta Lake → Lakebase → RDS (15-min lag)	RDS Postgres directly, no sync
Orchestration	DLT graph (all-or-nothing) + ~28 jobs	Step Functions + EventBridge
Compute	Spark clusters	ECS Fargate: runs, exits, pay-per-second
Refresh	Full refresh, daily	Incremental, every 3 h
Isolation	All orgs in shared tables, ID-tagged	A Postgres schema per org

The new layer is a straight line from source to app. ECS sync workers land raw source data in RDS. dbt transforms it in three layers (staging, intermediate, marts) and writes directly to RDS, with no sync step. The API, the web app, and the agents read straight from there. Nothing depends on Spark.

Nothing was thrown away, and nothing is copied from Databricks. The medallion was rebuilt in Postgres off the same sources. The new bronze layer reads the same Fivetran S3 landing the old one read, so it holds the same rows. The old gold and insights layers collapse into one marts layer. The “served subset” that insights used to hold as a second copy is now a filter inside dbt.

Orchestration is one Step Function, fired by EventBridge every 3 hours. Each stage runs as an isolated ECS Fargate task. A failure in one stage doesn’t undo the others, and the retry resumes from that stage. Compute starts, runs, and exits. There is no warm cluster.

Stage	What it does	Time
Staging	Bronze → typed, cleaned	~2 min
Intermediate	Normalize, resolve companies and identity, build canonical keys	~6 min
Shared marts	Company, person, source map, taxonomy	~3 min
Per-org marts	One private schema per customer, run in parallel	~2 min
Tests	Uniqueness, no fan-out, no cross-org leak, referential integrity	~1 min
Intelligence	LLM producers and deal intelligence, only if input changed	~5–10 min

The swap retired one DLT pipeline (about 90 model files), 28 Databricks jobs, and about 74.5K lines of Spark, DLT, and utility code. In their place: about 80 dbt models in readable SQL and 5 ECS compute units.

Remodeling the data, not just moving it

The migration also changed the data model in four ways. Each fixes a correctness or security problem in the old layer.

No row duplication. Salesforce represented access control by copying rows: the same opportunity once per viewer. At 500 users and 10K opportunities, that is 5M rows for 10K real deals. Now every entity is one row. Visibility lives in a thin join table at the opportunity-and-seller grain, a faithful port of Salesforce’s 7 sharing gates plus a HubSpot owner gate. 5M rows became 10K plus a grants table.

Deterministic IDs. Every ID is a hash of the source and the source ID. Same input, same ID, always. That makes the pipeline idempotent: rebuild a table from scratch and the graph edges still line up. The old DLT relied on streaming checkpoint state. Corrupt it and you got duplicates or missing rows, a real incident class. Now it’s delete+insert on a unique key. Run it twice, get the same answer.

Physical org isolation. Each customer gets their own Postgres schema. Only the shared Context360 schema is common to all, and the flow is one-way: orgs read from it, never the reverse. Leaking one tenant into another stops being a “remember the WHERE clause” problem and becomes structurally impossible. The rows live in different schemas.

Canonical company registry. The same company shows up under different names in different sources. Domain resolution collapses them into one master record, and a source map points every source row to it. The old layer left about 1,223 duplicate companies in about 23K records. Now there is one company, and any source resolves to it.

Why the app is faster

Two reasons. One on the read side, one on the write side.

Reads: three layers became one. The API now reads RDS Postgres directly. A plain connection, with no Databricks token, no Lakebase hop, and no warehouse cold-start. The agents read through a pooled, read-only RDS role. The marts carry indexes built by dbt. On production, queries return in under 10 ms.

Writes: incremental instead of full refresh. A full shared rebuild takes about 14 minutes. Incremental models process only the delta: the opportunity model ran in 0.3 seconds for the changed rows. With a 3-hour cadence, fresh data becomes intelligence within hours, instead of the full day a daily refresh forced.

Cost: from ~$10K to ~$0.7–1.3K a month

Component	Databricks	AWS replacement	AWS cost
DLT: the daily medallion refresh	~$3.9K	dbt on ECS Fargate, incremental	~$150–300
SQL Warehouse: served reads + agents	~$3.1K	Eliminated. API and agents read RDS directly	$0
Lakebase: Postgres mirror	~$1.9K	Deal intelligence reads RDS natively	$0
Jobs: ingestion, orchestration, whitespace	~$1.1K	ECS tasks, pay-per-second	~$300–600
Storage	<$0.1K	Same S3 + RDS, already paid	$0 incremental
Step Functions orchestration	None	Standard	~$1
Total (~30 days)	~$10K/mo		~$700–1,300/mo

That’s an 85–90% reduction, about $9K a month. The savings don’t come from killing an idle cluster. The DLT pipeline was already serverless. The real drivers:

• No platform markup. Databricks bills a DBU on every unit of compute, serverless or not. Fargate is raw AWS price. At this volume, the platform fee was where the money went.
• Incremental, not a daily full refresh. The CRM sources are not append-only. Records update in place, and DLT’s stream-oriented model fought incremental processing there. Capturing it meant moving to batch anyway, which is exactly the batch upsert, keyed by a deterministic ID, that now runs in dbt.
• No always-on SQL Warehouse. The warehouse that served reads was the genuinely warm resource, and the source of the agents’ 2 to 4 minute cold start. API and agents now read RDS directly.
• One operational world, no lock-in. ECS for everything, instead of ECS plus a Databricks platform floor.

The Databricks figures come from billing data, by product. The AWS figures are still partly estimated.

Cutting over without breaking anyone

The new native read path is the production path for every customer org. It got there without a customer data incident, through three rules.

Ship dark. Every change landed behind a flag that defaults off, byte-identical to the old behavior. Code could deploy to production long before any switch was flipped.

Dual-run, then gate. An automated gate runs each customer read both ways, new RDS path and old Databricks path, against the same live data and off live traffic. The rows must be byte-for-byte identical across 21 checks before an org is cleared.

Flip per org, on a human go. Cutover happens one org at a time, only after that org’s gate is green, and it’s reversible by flag.

Databricks now runs only as a thin bridge being retired: a sync job that copies the legacy tables to RDS every 15 minutes, and three remaining API triggers. Retiring both, and repointing the triggers to Step Functions and SQS, takes Databricks fully off.

The result is a data layer that fits the size of the data. Plain SQL, compute that only runs when there’s work, one read path, and customer data that can’t cross between tenants.

For Aithon, that means a data layer about 90% cheaper to run, queries under 10 ms, fresh data every 3 hours instead of once a day, and every customer’s data in its own schema.

That’s what Eversynced brought to Aithon’s data layer, and what we bring to every engagement. Eversynced embeds engineers into product teams as full members, not vendors. If you’re evaluating team augmentation, we’d like to talk.