Introduction

Enzo Server
  Installation
  Quick Start
  Core Features
    Async Calls
    ConnectionStrings
    Edge Cache
    HTTP Access
    Scheduling
    Views
  Advanced Capabilities
    Change Data Capture
  Administration
    Configuration Settings
    Logins & ACL
    Auditing
    Current Executions
    Linked Server
    SSL

  User Guides
     Sharding
     Sharding Overview

     SharePoint

   Adapters
    All Adapters
    ActiveCampaign
    Azure Bus
    Azure IoT Hub
    Azure Key Vault
    Azure Storage
    Bing Search
    ConstantContact
    Couchbase
    CSV
    DB (RDBMS)
    Dropbox
    Files
    Flight Aware
    FreshBooks
    FTP
    Geonames
    GoogleDrive
    GoogleSearch
    GoogleSheet
    GreenMile
    GpsInsight
    HttpEndpoint (REST)
    Intacct
    Insteon
    Loggly
    MailChimp
    Messaging (SMS/SMTP)
    MongoDB
    Motus
    MSAD
    MSMQ
    OneDrive
    OneDrive365 (Business)
    Pipeline
    Pressero
    RabbitMQ
    SalesForce
    Shard
    SharePoint
    Slack
    SQL Server
    SQLX
    Sudoku
    Teams
    Threading
    Twilio
    Twitter
    VellemanDMX
    Whiparound
    WMI
    Xls (Excel)
    ZIP


  SDK
    Overview
    Installation
    DevHost
    Create A Simple Adapter
    Handler Columns

    Advanced SDK Concepts
      Handler Decorators
      Dynamic Columns
      Table & Table Enumerators
      Virtual Tables

/adapters/pipeline




Pipeline


The Pipeline adapter allows you to define Change Data Capture (CDC) events from a source system to any destination adapter. This is an advanced adapter and should be used from the Enzo Manager application directly. Some of the advanced capabilities include multicasting events, and replaying events from a specific date/time.

This adapter is currently in Preview


Configuration Settings



Name Description Default
Boolean _isDefault Make this the default configuration. Defaults to true if configName is already the default.
String enzoConnectionString The central connection string to Enzo (loopback)
String storeConnectionString The central connection strong for the store engine

Handlers



ClearErrors Deletes the error log for a listener. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener

SQL Call

exec Pipeline.ClearErrors 'sharepoint-usstates', 'sql-listener'


Http Call

/bsc/pipeline/clearerrors

Output Columns

This call does not return data.

CreateListener Creates a new listener for the pipeline. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
name R The name of the listener
system R The destination system (AzureBusTopic,Twilio,SQL...)
config The name of the destination adapter configuration
remoteObject The name of the remote object, table or endpoint
mode The processing mode of the listener (each,batch) each
initialize When 1 send all initialization data False
deleteCDC When 1 send all delete operations False
upsertCDC When 1 send all upsert operations False
columns List of columns for forward (leave blank for all)
filter Destination SQL-Like filter
options JSON object providing options specific to the destination adapter

SQL Call

exec Pipeline.CreateListener 'sharepoint-usstates', 'sql-listener-usstates'


Http Call

/bsc/pipeline/createlistener

Output Columns

This call does not return data.

CreatePipeline Creates a new pipeline. details 

Arguments

Name Description Default
name R The name of the pipeline
adapter R The name of the source adapter
config R The configuration name of the source adapter
sourceTable R The source table to read from
interval The polling interval to the source system as a cron expression
mode The pipeline mode (CDC,Sync,CDCSync)
filter Adapter-specific filter if required (ex: Twitter)

SQL Call

exec Pipeline.CreatePipeline 'sharepoint-usstates', 'sharepoint', 'sp-config', 'USStates', 'az-config', '* * /5 * * ?', 'cdcsync'


Http Call

/bsc/pipeline/createpipeline

Output Columns

This call does not return data.

DeleteListener Deletes a listener in a pipeline. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener

SQL Call

exec Pipeline.DeleteListener 'sharepoint-usstates', 'sql-listener-usstates'


Http Call

/bsc/pipeline/deletelistener

Output Columns

This call does not return data.

DeletePipeline Deletes a pipeline and associated listeners. details 

Arguments

Name Description Default
name R The name of the listener

SQL Call

exec Pipeline.DeletePipeline 'sharepoint-usstates'


Http Call

/bsc/pipeline/deletepipeline

Output Columns

This call does not return data.

DisableListener Prevents a listener from forwarding events to the destination system. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener

SQL Call

exec Pipeline.DisableListener 'sharepoint-usstates', 'sql-listener'


Http Call

/bsc/pipeline/disablelistener

Output Columns

This call does not return data.

DisablePipeline Prevents a pipeline from gathering change events from the source system. details 

Arguments

Name Description Default
name R The name of the pipeline

SQL Call

exec Pipeline.DisablePipeline 'sharepoint-usstates'


Http Call

/bsc/pipeline/disablepipeline

Output Columns

This call does not return data.

EnableListener Allows a listener to forward events to the destination system. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener

SQL Call

exec Pipeline.EnableListener 'sharepoint-usstates', 'sql-listener'


Http Call

/bsc/pipeline/enablelistener

Output Columns

This call does not return data.

EnablePipeline Enables a pipeline from gathering change events from the source system. details 

Arguments

Name Description Default
name R The name of the pipeline

SQL Call

exec Pipeline.EnablePipeline 'sharepoint-usstates'


Http Call

/bsc/pipeline/enablepipeline

Output Columns

This call does not return data.

GetCreateDDL Returns the DDL command to execute in order to create the destination table details 

Arguments

Name Description Default
dbName The database name to connect to (leave blank for Azure SQL Databases)
schemaName R The schema name to use dbo
tableName R The table name to create
connectionStr R The Central Connection String name
fields R The list of fields selected from the source pipeline

SQL Call

exec Pipeline.GetCreateDDL 'datalake', 'twitterschema', 'twitter-capture1', 'localsqldb001', 'string id,int age' 


Http Call

/bsc/pipeline/getcreateddl

Output Columns

String sql
GetExecutionLog Returns the execution log for a given execution id details 

Arguments

Name Description Default
executionId R The execution id
pipelineName R The name of the pipeline

SQL Call

exec Pipeline.GetExecutionLog '87632ad6-9d4c-4ff1-bf61-91159f3857be'


Http Call

/bsc/pipeline/getexecutionlog

Output Columns

String executionId
DateTime eventDate
String message
String level
String pipelineName
String listenerName
String pipelineId
String listenerId
GetExecutions Returns the list of recent executions for a given pipeline or listener details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName The name of the listener

SQL Call

exec Pipeline.GetExecutions 'pipelineName'
exec Pipeline.GetExecutions 'pipelineName', 'listenerName'


Http Call

/bsc/pipeline/getexecutions

Output Columns

String ExecutionId
DateTime ExecutionDate
String ErrorMessage
String StackTrace
Int32 RecordCount
Int32 TotalMin
Int32 Year
Int32 Month
Int32 Day
Int32 Hour
Int32 Minute
String PipelineName
String ListenerName
String PipelineId
String ListenerId
GetJobStats Returns saved jobs with current statistics. details 

Arguments

Name Description Default
pipelineName The name of the pipeline to filter on

SQL Call

exec Pipeline.GetJobStats


Http Call

/bsc/pipeline/getjobstats

Output Columns

String pipelineName
String listenerName
String status
String lastSyncId
Int64 lastProcessedCount
Int64 lastErrorCount
Double lastTotalKB
Int64 totalProcessedCount
Int64 totalErrorCount
String lastRunTime
String resultMessage
String stackTrace
Boolean replaying
String pipelineIdentifier
String listenerIdentifier
String disabled
ListListeners Gets available listeners. details 

Arguments

Name Description Default
pipelineName The name of the pipeline
listenerName The name of the listener (starts with)

SQL Call

exec Pipeline.GetListeners 'sharepoint-usstates' 


Http Call

/bsc/pipeline/listlisteners

Output Columns

String pipelineNameThe name of the pipeline
String nameThe name of the listener
String identifierThe internal unique id for this pipeline
String ownerThe owner of the pipeline
Boolean enabledtrue when the pipeline is enabled
DateTime createdOnThe date the pipeline was created
String systemThe destination system (AzureBusTopic,Twilio,SQL...)
String configThe name of the destination adapter configuration
String remoteObjectThe name of the remote object, table or endpoint
String modeThe processing mode of the listener (each,batch)
Boolean initializeWhen 1 send all initialization data
Boolean deleteCDCWhen 1 send all delete operations
Boolean upsertCDCWhen 1 send all upsert operations
String columnsList of columns for forward (leave blank for all)
String filterDestination SQL-Like filter
String optionsJSON object providing options specific to the destination adapter
DateTime lastPingedOnLast date/time the listener sent a Ping while running
Boolean isInitTrue when the listener is currently processing items from the INIT log
Boolean isReplayingTrue when the listener is replaying from the CDC log as requested
Boolean hasRetriesTrue when the listener has items to retry at the next execution cycle
Boolean hasErrorsTrue when the listener has errors that will not be retried automatically
Boolean clearErrorsTrue when errors will be deleted at the next execution cycle, as requested
Boolean replayErrorsTrue when errors will be replayed at the next execution cycle, as requested
ListPipelines Gets list of pipelines. details 

Arguments

Name Description Default
name The name of the pipeline (starts with)

SQL Call

exec Pipeline.ListPipelines


Http Call

/bsc/pipeline/listpipelines

Output Columns

String nameThe name of the pipeline
String identifierThe internal unique id for this pipeline
String ownerThe owner of the pipeline
Boolean enabledtrue when the pipeline is enabled
DateTime createdOnThe date the pipeline was created
String adapterThe name of the source adapter
String configThe configuration name of the source adapter
String sourceTableThe source table to read from
String intervalThe polling interval to the source system as a cron expression
String modeThe pipeline mode (CDC,Sync,CDCSync)
String filterAdapter-specific filter if required (ex: Twitter)
String columnListList of columns available from the source system
String IDsThe Primary Key of the source system (if any)
Reinitialize Reinitializes a listener by replaying the INIT log and then the complete CDC log. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener

SQL Call

exec Pipeline.reinitialize 'pipeline001', 'listener001'


Http Call

/bsc/pipeline/reinitialize

Output Columns

This call does not return data.

Replay Replays a listener from a specific start time. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener
startTime R Replay date (in UTC). Send an empty string to replay the complete CDC log

SQL Call

exec Pipeline.replay 'pipeline001', 'listener001', '2019-03-02t03:41:12.4760949z'


Http Call

/bsc/pipeline/replay

Output Columns

This call does not return data.

ReplayErrors Replays the error log for a listener. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
listenerName R The name of the listener

SQL Call

exec Pipeline.ReplayErrors 'sharepoint-usstates', 'sql-listener'


Http Call

/bsc/pipeline/replayerrors

Output Columns

This call does not return data.

RunListener Runs a listener as soon as possible if it is enabled. details 

Arguments

Name Description Default
pipelineName R The name of the pipeline
name R The name of the listener

SQL Call

exec Pipeline.RunListener 'sharepoint-usstates'


Http Call

/bsc/pipeline/runlistener

Output Columns

This call does not return data.

RunPipeline Runs a pipeline as soon as possible if it is enabled. details 

Arguments

Name Description Default
name R The name of the pipeline

SQL Call

exec Pipeline.RunPipeline 'sharepoint-usstates'


Http Call

/bsc/pipeline/runpipeline

Output Columns

This call does not return data.

UpdatePipelineSchedule Updates a pipeline schedule. details 

Arguments

Name Description Default
name R The name of the pipeline
interval R The polling interval to the source system as a cron expression

SQL Call

exec Pipeline.UpdatePipelineSchedule 'sharepoint-usstates',  '* * /5 * * ?'


Http Call

/bsc/pipeline/updatepipelineschedule

Output Columns

This call does not return data.