What is Change Data Capture?
Change Data Capture (CDC) is a process of identifying and capturing only the changes to the data. So, a downstream system or system can act upon it. A common use case is data replication across different system
Why implementing CDC is Vital for an organization?
The concept of capturing only changes comes from the database where data replication often occurs from transactional-based relational databases to a data warehouse. Especially in high-velocity environments where time-sensitive decisions are made, Change Data Capture is an excellent fit to achieve low-latency, reliable, and scalable data replication.
Given the need to replicate data across multiple environments/systems, picking the perfect data integration tool and the right CDC method has become more critical for an organization.
Significance of implementing CDC:
- Increases efficiency by processing only changed data instead of Full Refresh.
- Integrations can run quickly with just delta load.
- Data Consistency can be maintained with frequent data sync between the systems.
Boomi CDC / Change Data Capture:
To address growing conflicts between systems over data synchronization, Boomi’s Change data capture helps maintain synchronized and reliable data across all systems. Boomi’s CDC uses a pull mechanism where it extracts the whole set of data every time and compares the delta outside of the source system and processes it.
This boomi CDC solution is best suitable for integration that involves pulling data from legacy systems that don’t have the flexibility to use either FLAG / LAST MODIFIED DATE to better identify data changes.
Boomi uses Find Changes shape to implement CDC. Below are the steps involved in implementing change data capture.
- Extract the entire data set every time from the source system.
- Configure profile and key columns used for comparison.
- Configure the Find Changes shape to determine which records should go down Add, Update or delete path.
- First run, passes down all records to Add path and creates a delta file to be used for comparison
- Subsequent runs will compare the current data set with the cached data from the previous run and passes the records to Add, Update or delete path correspondingly.
Reference: What is CDC?
Cover Photo by Mika Baumeister on Unsplash
