{"id":83866,"date":"2026-10-14T14:28:29","date_gmt":"2026-10-14T08:58:29","guid":{"rendered":"https:\/\/2thenew.today\/blog\/?p=83866"},"modified":"2026-10-27T23:08:11","modified_gmt":"2026-10-27T17:38:11","slug":"incremental-intelligence-bringing-data-engineering-principles-to-ai-systems-2","status":"publish","type":"post","link":"https:\/\/2thenew.today\/blog\/incremental-intelligence-bringing-data-engineering-principles-to-ai-systems-2\/","title":{"rendered":"Incremental Intelligence: Bringing Data Engineering Principles to AI Systems"},"content":{"rendered":"

Introduction<\/strong><\/h1>\n

Every data engineer who has run dbt run –full-refresh<\/em><\/strong> on a 400-million-row table at 2 a.m. knows a specific dread: watching warehouse credits burn while reprocessing everything, when only a fraction of the data actually changed.<\/p>\n

dbt’s answer \u2014 incremental models, materializations, snapshots, exposures \u2014 is a decade of hard-won wisdom about one question: when something changes, how much of the world do you actually need to redo?<\/strong><\/p>\n

Here’s what nobody’s really written about yet: AI engineers are relearning this exact lesson right now, one expensive re-embedding job and one forgetful chatbot at a time. RAG pipelines, agent memory, and prompt versioning are all quietly reinventing dbt’s incremental philosophy \u2014 usually without knowing a mature vocabulary for it already exists.<\/p>\n

This post is that bridge: dbt’s core concepts, translated into AI-native equivalents, with working code.<\/p>\n

<\/h1>\n

The Problem AI Systems Are Rediscovering<\/strong><\/h1>\n

A very typical (and expensive) RAG pipeline:<\/p>\n

\"img\"<\/p>\n

Fix one typo in one of 5,000 documents, and this still re-embeds all 5,000 \u2014 the RAG equivalent of dbt run –full-refresh<\/strong><\/em> because one row changed. The same pattern shows up in agent memory (re-summarizing an entire conversation history every turn) and in evals (re-running the full suite for every prompt tweak, instead of only what’s affected). None of this is a knock on AI engineers \u2014 it’s a young field re-discovering an old, already-solved problem.<\/p>\n

<\/h1>\n

Translating dbt Concepts Into AI Systems<\/strong><\/h1>\n

\"img\"<\/p>\n

\"image-02\"<\/p>\n

Same insight as dbt’s is_incremental()<\/strong><\/em> macro: filter first, process second.<\/p>\n

\n

One caveat worth knowing before you ship this: updated_at timestamps only catch changes to source content. If your chunking strategy or embedding model changes, the source document can look “unchanged” while its vectors are stale. Store the content hash alongside chunking_version<\/strong> and embedding_model_version<\/strong>, and treat any of the three changing as a reason to reprocess. The hash catches content changes; the version fields make pipeline and model changes explicit.<\/span><\/p>\n<\/blockquote>\n

Incremental Strategies and Their AI Counterparts<\/strong><\/h1>\n

\"img\"<\/p>\n

\"img\"<\/p>\n

Use “append” where you needed “merge,” and you get duplicates \u2014 in dbt, duplicate rows; in an agent, a context window with three contradictory versions of the same fact.<\/p>\n

\ud83e\udded Quick self-check:<\/strong><\/em> Does the same fact ever get updated more than once? No \u2192 Append<\/strong>. Yes, and you only care about the current value \u2192 Merge<\/strong>. Yes, and you need to know what it used to be \u2192 you need SCD-2<\/strong>, not a simple merge (next section).<\/span><\/p>\n

<\/h1>\n

Practical Implementation Lessons<\/strong><\/h1>\n

Two production details matter once incremental embeddings are running:<\/p>\n

    \n
  • \n
      \n
    • Delete handling:<\/strong> If a source document is removed, its chunks and vectors should be deleted or tombstoned too. Otherwise, retrieval can continue surfacing content that no longer exists.<\/li>\n
    • Idempotent re-embedding and upserts:<\/strong> When the embedding model or chunking version changes, reprocess the affected documents and upsert using a deterministic key such as document_id + chunk_id + chunking_version + embedding_model_version<\/strong>. A retry should update the same vector rather than create duplicates.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

      <\/h1>\n

      Bringing Slowly Changing Dimensions to AI Memory<\/strong><\/h1>\n

      The single most underused idea in AI system design: not all memory should behave the same way when it changes.<\/p>\n

      \"image<\/p>\n

      SCD-1 facts<\/strong> (overwrite, no history needed): a corrected phone number, a fixed CRM typo. There’s no value \u2014 and real risk \u2014 in an agent “remembering” the old, wrong version.<\/p>\n

      SCD-2 facts<\/strong> (history matters): evolving product preferences, a patient’s medication history, a client’s risk tolerance over time. Treat these as SCD-1 and you lose the ability to explain why the AI is behaving the way it is now \u2014 an audit and compliance liability in regulated industries. Treat everything as SCD-2, on the other hand, and you get context-window bloat from facts that never needed history.<\/p>\n

      dbt’s snapshot pattern (dbt_valid_from \/ dbt_valid_to<\/em><\/strong>) maps directly onto an AI “memory snapshot”:<\/p>\n

      \"img\"<\/p>\n

      When an agent is wrong, “knowing what it believed and when helps identify the root cause”\u2014easy with snapshot history, difficult with a single mutable blob.<\/p>\n

      <\/h1>\n

      Bringing dbt Governance Patterns Into AI<\/strong><\/h1>\n
        \n
      • \n
          \n
        • Pre\/Post-hooks \u2192 guardrails:<\/strong>\u00a0dbt runs SQL before\/after a model executes. AI pipelines need the same two moments: pre-hooks (PII scrubbing, prompt-injection checks, before the LLM call) and post-hooks (safety classification, schema validation, memory writes, after the call). Treating guardrails as first-class pipeline steps \u2014 not an afterthought bolted on post-incident \u2014 is the whole point.<\/li>\n
        • Source freshness \u2192 context freshness:<\/strong>\u00a0dbt checks if raw data is stale; AI systems rarely check if retrieved context is stale \u2014 yet a RAG system citing three-week-old pricing will do so with exactly the same fluent confidence as correct information. A simple warn_after \/ error_after<\/strong><\/em> contract per source (docs: warn at 3 days; pricing: error at 6 hours) catches this before a user does.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
          \n

          # context_sources.yml<\/code><\/em><\/p>\n

          <\/code><\/em><\/span>sources:<\/code><\/em><\/span>
          \n\u00a0 \u00a0 - name: pricing_docs<\/span><\/code><\/span><\/em><\/span>
          \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0freshness:<\/code><\/em><\/span>
          \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 warn_after: {<\/span>count: 3<\/span>, period: hour<\/span>}<\/span><\/code><\/em><\/span>
          \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 error_after: {<\/span>count: 6<\/span>, period: hour<\/span>}<\/span><\/code><\/em><\/span><\/p>\n

          \u00a0 - name: product_docs<\/span><\/code><\/em><\/span>
          \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0freshness:<\/code><\/em><\/span>
          \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0warn_after: {count: 3<\/span>, period: day<\/span>}<\/code><\/em><\/span>
          \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0error_after: {count: 7<\/span>, period: day<\/span>}<\/code><\/em><\/span><\/p>\n<\/blockquote>\n

            \n
          • \n
              \n
            • Exposures \u2192 dependency declarations: <\/strong>dbt exposures document what downstream thing depends on a model, so a change is traceable before it ships. Most AI stacks have no equivalent \u2014 nobody knows what breaks if you swap an embedding model until something breaks. A simple ai_exposures.yml (which prompts\/models\/indexes feed which features, owned by whom) turns “who broke the chatbot?” into “oh, I broke it” \u2014 days earlier and much cheaper.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
              \n

              # ai_exposures.yml<\/code><\/span><\/em><\/p>\n

              <\/code><\/em><\/span>exposures:<\/code><\/em><\/span>
              \n\u00a0 \u00a0 - name: support_chatbot_widget<\/span><\/code><\/span><\/em><\/span>
              \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0type: application<\/span><\/code><\/code><\/em><\/span>
              \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0owner: support-eng<\/span><\/code><\/em><\/span>
              \n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0depends_on:<\/code><\/code><\/em><\/span><\/p>\n

              - prompt: refund_policy_V3<\/span>
              \n- index: kb_articles_embeddings<\/span>
              \n- model: text-embedding-3-large<\/span><\/code><\/span><\/p><\/blockquote>\n

                \n
              • \n
                  \n
                • dbt Jobs \u2192 CI for prompts:<\/strong>\u00a0The highest-leverage idea to steal: run a golden-set eval suite against any prompt or retrieval-config change before it merges, exactly like dbt’s CI job tests SQL changes before merge.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                  <\/h1>\n

                  Case Study: Building a More Reliable TicketMind<\/strong><\/h1>\n

                  A support-bot team ships v1 with: full ticket history stuffed into every prompt, a nightly full re-embed of 12,000 articles, customer facts as one mutable JSON blob, and no eval gate on prompt changes.The failures arrive on schedule: a 45-minute nightly job re-processing 11,950 unchanged articles for 50 new ones (Week 3); an unanswerable refund dispute because facts were overwritten with no history (Week 6); a “make it concise” prompt tweak that silently drops safety disclaimers, caught only after a customer complains (Week 9); and degrading answer quality as 40+ tickets of irrelevant history crowd out the context window (Week 11).<\/p>\n

                  The fix is incremental, fittingly: switch to incremental (merge) embedding, split customer facts into SCD-1 (overwrite) vs. SCD-2 (snapshot) tables, scope context to current merged facts instead of full history, add a CI eval gate on the refund-policy prompt, declare an exposure naming compliance as owner, and set a freshness contract on the knowledge base. None of it required new infrastructure \u2014 just applying discipline the data team already had.<\/p>\n

                  \"img\"<\/p>\n

                   <\/p>\n

                  A Practical Guide to Getting Started<\/strong><\/h1>\n
                    \n
                  • \n
                      \n
                    • Audit your \u201cfull refresh\u201d points \u2014 fix the most expensive one first.<\/li>\n
                    • Classify your top 10 facts as SCD-1 or SCD-2.<\/li>\n
                    • Write down a freshness contract per data source.<\/li>\n
                    • Declare exposures for your top 3 production AI features.<\/li>\n
                    • Add one CI eval gate on your most-changed prompt.<\/li>\n
                    • Formalize pre\/post-hooks instead of scattered if checks.<\/li>\n
                    • Document your memory schema like a dbt model.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                      None of this needed a new framework or a research paper \u2014 just borrowing ten years of data engineering discipline and pointing it at a problem AI teams are currently re-discovering the expensive way. Next time you’re designing agent memory or debugging a RAG system that “doesn’t feel fresh,” ask the dbt question instead of the AI question: not which model should I use, but what actually changed?<\/p>\n

                      Happy building (incrementally)!<\/p>\n

                       <\/p>\n","protected":false},"excerpt":{"rendered":"

                      Introduction Every data engineer who has run dbt run –full-refresh on a 400-million-row table at 2 a.m. knows a specific dread: watching warehouse credits burn while reprocessing everything, when only a fraction of the data actually changed. dbt’s answer \u2014 incremental models, materializations, snapshots, exposures \u2014 is a decade of hard-won wisdom about one question: […]<\/p>\n","protected":false},"author":2354,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":8,"footnotes":""},"categories":[6194],"tags":[8917,5519,8916],"class_list":["post-83866","post","type-post","status-publish","format-standard","hentry","category-data-engineering","tag-aimemory","tag-dbt","tag-incrementaldataprocessing"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t