Job Reader: Big Data
Use a Big Data Job Reader when reading from Azure CosmosDB or Google Cloud BigQuery system. These source platforms offer specific settings tailored to their unique characteristics.
Create/Edit a Big Data Job Reader
To read data from a Big Data source platform, select the desired Connection String and enter the SQL command using the syntax of the source platform. In addition, when reading from a NoSQL data source, use the Apply JSON/XML Column Transformation option to convert the data into rows and columns if desired. Big Data sources that store data in rows and columns already (such as Google BigQuery).
For CosmosDB data sources, a Database Name and Table is required, although DataZen can use the default settings specified by the Connection String.
It is highly recommended to specify a High Watermark column in the Replication Settings tab to avoid reading all the records of the source system everytime the job runs.