Change Data Capture
Logins & ACL
CHANGE DATA CAPTURE
Change Data Capture (or CDC) enables you to configure Enzo Server to forward changes from source systems (pipelines) to one or more destination systems (listeners).
The CDC capability is currently in Preview.
At a high level the CDC engine is designed to detect changes from source systems and stores them in an CDC Store. Separately, each listener holds a pointer to the last successfully sent change record from the CDC Store, and plays back all changes from that point on. This architecture ensures that source and destination systems are completely decoupled. In addition, since the CDC Store stores information as a JSON payload, it is possible to forward the change records to any destination system including generic webhooks.
Enzo Server pulls data changes from a source system and stores detected changes in the CDC Store.
It is also possible to extract all current records in their initial state before starting the CDC process. When configured to do so, a complete extract of the source system is performed and stored in an INIT Store. This makes it possible to build a complete replication mechanism between two or more systems.
Before you can setup a CDC pipeline and listener, you need to configure the appropriate adapters and the required Connection Strings. See the Configuration Settings section for information about configuring adapters. And see the Connection Strings section for details on how to create connection strings.
Two connection strings are needed to setup CDC: a loopback connection to Enzo Server, and a connection to an external SQL Server database that will be used for the CDC and the INIT Stores.
Make sure that the SQL Server connection string uses a Default Catalog.
Pipeline Adapter Config Setting
Before creating pipelines and listeners, you need to first create a configuration setting for the Pipeline Adapter. The configuration settings require two values: the connection strings you created previously. One of them points to Enzo Server directly (using the 'sa' account preferably) and the other points to a SQL Server database.
It is important for the storeConnectionString setting to point to a SQL Server database with the Default Catalog specified and poiting to an existing, but empty, database. The Pipeline Adapter will create the necessary tables automatically.
Create a Pipeline
To create a new pipeline, select the Pipeline Adapter, and click on the Pipelines tab. From here you can click on the NEW icon to create a new pipeline.
Pipeline ModeYou can choose between three modes for the pipeline:
Create a Listener
Once a pipeline has been created, you can create one or more listeners. Select the pipeline you want, then click on Add Listener.
You can choose between two processing modes for the listener:
When selecting the INIT option, every record from the CDC INIT Store will be sent first; once this operation completes events from the CDC Store will be forwarded.
Some listeners require additional parameter mapping. For example if the destination system is a SharePoint list, a keyField needs to be specified. This can be a constant value, or the value contained in a field from the event. The parameter screen provides additional details for available options.
SQL Server Listener
You can specify a relational database for a listener; when doing so you will need to create the table before saving the
listener, or the listener will fail. To help you create the proper schema for the destination table, a
VIEW/EXECUTE DDL STATEMENT button will be visible when choosing Database as the destination. Clicking
the button will show you the statement used to create the table, and you will have the option to create the
The DDL command will only work if the destination table is a SQL Server database.