Detect modified data in SQL Server tables
Solved Email & Outlook
MS
Michael Scofield
November 6, 2019
2 replies
6,870 views
Reviewed by moderators

Our nightly sync to the warehouse copies entire tables because nobody knows what changed since the last run. The tables grew and full copies stopped fitting the night.

What are the mechanisms for knowing which rows changed, from crude to proper?

Accepted Answer
Verified by Edwin J. Hoffer, Database Specialist ยท Reviewed November 2019

Crude to proper is exactly the right framing, four rungs on that ladder, each fixing the previous one's blind spot.

Rung one, timestamps you may already have: a last_modified datetime column maintained by the application or a trigger lets the sync take rows where modified since the last run. Blind spots: deletes leave no row to carry a timestamp and any write path that skips the column, bulk loads classically, silently escapes the sync.

Rung two, rowversion: a rowversion column updates automatically on every modification, no application cooperation required. The sync takes rows above the highest value it saw last time, comparing against MIN_ACTIVE_ROWVERSION to avoid in flight transactions. Same delete blindness, but the automatic maintenance closes the skipped write path hole and adding the column is one ALTER TABLE.

Rung three, Change Tracking, the built in answer to deletes: ALTER DATABASE enables it, ALTER TABLE per table, then CHANGETABLE(CHANGES table, last_version) returns the keys of inserted, updated and deleted rows since a version number, exactly the primitive an incremental sync wants. It records that a row changed rather than what the values were, join back to the source for current values. Its retention window must exceed the longest gap between syncs or the sync must handle the full reinitialize case.

Rung four, Change Data Capture, when the warehouse wants history rather than presence: CDC reads the transaction log and materializes every change with before and after images into queryable tables, feeding syncs that need the intermediate values or audit style loads. It costs a log reader running, storage for the change tables and edition awareness, Enterprise before SQL 2016 SP1. For plain which rows do I copy tonight, Change Tracking on rung three is the fit, CDC earns its weight only when the question becomes what were the values along the way.

Enabled Change Tracking on the six big tables and rebuilt the sync around CHANGETABLE. First night: 40 minutes instead of five hours, deletes finally propagating too. The ladder explanation is going in our wiki verbatim, credited.