In this article we will be looking at a new pipeline offering called Webhook Pipelines and explore specific use cases for using this new technology.

Until now, Enzo offered Replication Pipelines that can be used to replicate data changes from a source system to one or more destination systems, providing both Change Data Capture (CDC) and replay capabilities. With the introduction of Webhook Pipelines, clients can push messages into the pipeline architecture so that messages sent by client applications or systems can be forwarded to one or more listeners, with full replay capability.

Replication vs. Webhook Pipelines

Replication Pipelines are designed to pull data from source systems automatically, on a predefined schedule, and save those changes into a CDC Store (the Change Data Capture store). As a result, Replication Pipelines are best suited for monitoring source systems for changes automatically, and forward those changes to one or more destination systems (called Listeners). In summary, Replication Pipelines use a pull mechanism to extract data.



Webhook Pipelines on the other hand wait for HTTP requests to be submitted to the pipeline. HTTP messages sent to a webhook pipeline are then stored in the CDC Store discussed previously. This means that webhook pipelines use a push mechanism to receive data payloads. Once in the CDC store, Listeners can be configured to receive the data. Since the payload can be any JSON or XML payload, the complete payload is forwarded to listeners; this is thus more useful for messaging architectures. Example of destination systems are the Azure Service Bus, Azure Queues and custom HTTP endpoints.



Another key difference between Replication and Webhook pipelines is related to the INIT store. Replication Pipelines can be used to extract the initial state of a source system by exporting all current records in an INIT store, before ongoing changes are captured. Webhook pipelines do not have an INIT store by design.

Use cases for Webhook Pipelines

Application Development

Webhook pipelines can simplify application development by abstracting the destination messaging system architecture and complexity, since sending messages is done through simple HTTP requests. In addition, abstracting the underlying messaging architecture reduces long-term maintenance costs when the destination messaging architecture needs to change.

For example you can build an application that leverages Azure Queues (from the Azure Storage Service), but may choose to leverage the Azure Service Bus in the future. Using webhook pipelines allows you to add/remove listeners without changing a single line of application code.

Multicast Architecture

The pipeline architecture provides the ability to send messages to any number of listeners with an eventual consistency model. This allows an application to send a single message, and forward the message to multiple destination systems. For example, if an application sends a warning in the form of an HTTP post to a webhook pipeline, the message could be forwarded to two or more listeners without changing the application code. This is also very useful for transparently forwarding HTTP messages sent from vendor hosted systems.

Change Data Capture & Replay

A key feature of the Enzo Pipeline architecture is the ability to store messages into a scalable CDC Store, providing guaranteed delivery of messages to all listeners. In addition, each listener can choose to replay a portion of the CDC store if necessary. This provides destination systems the ability to receive some or all of the messages from the CDC store if necessary.

Sample HTTP Post

The following HTTP request shows a sample message sent to an Enzo Webhook Pipeline that will get stored into the CDC Store. Any listener configured to receive messages from this pipeline with get a copy of the JSON payload.

POST https://webhook.enzounified.com/api/HTTPEnzoPipeline HTTP/1.1
loginId: email@hotmail.com
pipelineId: 12345ab6-789a-1c23-45f6-7edd89a123fc
Content-Type: application/json

{"msg": "test message"}