...
| Table of Contents |
|---|
Introduction
Introduction : In new datawarehouse architecture, the raw data is procssed by aws-glue jobs (etl) and stored in destination s3 bucket. To create or update the ETL jobs needs jenkins job which create the stack and required resources.
Create/Update the python script
Synapse-ETL-Jobs is Synapse project where all the python scripts are maintained. To create a new job or update an existing job which requires the processing of python script, first we need to do the changes in Synapse-ETL-Jobs and build. Every merge in the project creates a new version of tag. Always use the latest tag version.
...