Ultimate Azure Data Factory: Cloud Data Engineering
Ultimate Azure Data Factory: Cloud Data Engineering, Real world Modern Data Warehouse project for Data Engineers using Azure Data Factory, Sql, Data Lake, Databricks.
Welcome!
Data engineering is a thriving focus in the IT industry, with Microsoft’s Azure Data Factory emerging as a sought-after tool in cloud-based data engineering.
Join this course for a step-by-step journey into mastering Azure Data Factory (ADF). Using a real-world scenario of an e-commerce company grappling with data integration and insights, we’ll explore the data of an online wine retailer, showcasing how implementing a modern data warehouse with ADF can provide solutions.
Distinguishing itself from other Udemy offerings on Azure Data Factory and Data Engineering Technologies, this course guides you hands-on in transforming raw data into a Modern Data Warehouse using Azure Data Factory (ADF). Upon completion, you’ll gain proficiency in ADF, ready to tackle real-world data engineering projects.
Given the course’s focus on real-world business scenarios, it adopts a sequential approach mirroring how such requirements unfold in actual projects. This method ensures you not only implement business needs but also grasp the technical concepts explained at each stage of implementing data pipelines with Azure Data Factory (ADF).
This course covers more than just modern data warehouse concepts like architecture, medallion layers, and delta lake. You’ll also gain expertise in utilizing diverse Azure ecosystem solutions, including Azure Data Lake Storage, Azure SQL Database, and Azure Databricks. Additionally, you’ll learn to visually represent the completed data warehouse through Power BI reports.
This course enables you to grasp concepts and skills assessed in the Azure Data Engineer Associate Certification exam DP203. While it equips you with the necessary skills, it’s important to note that the course is not designed solely for certification passing but for comprehensive learning.
I appreciate your time, and I’ve crafted this course to be practical and focused. I aim for simplicity and conciseness, starting from the basics and ensuring proficiency in the technologies covered.
Currently the course teaches you the following:
Azure Data Factory
- Constructing a contemporary Data Warehouse architecture for a data engineering solution involves utilizing Azure Data Engineering technologies like Azure Data Factory (ADF), Azure Data Lake Gen2, Azure SQL Database, Azure Databricks, Azure KeyVault, and Microsoft PowerBI.
- Incorporating data from varied sources with diverse formats into Azure Data Lake Gen2 is achieved through the use of Azure Data Factory.
- Comprehending Azure concepts, including resources and their provisioning methods.
- Learning to incorporate and use tools such as Azure Storage Explorer, Azure Data Studio, and Visual Studio Code in the development workflow.
- Implementing Azure Data Factory (ADF) pipelines using different control flow activities such as Get Metadata, ForEach, If Conditions, etc.
- Using Parameters and Variables in Pipelines, Datasets and LinkedServices to create generic parameter driven pipelines in Azure Data Factory (ADF).
- Using parameters in conjunction with Azure KeyVault to create generic parameter driven piplines in Azure Data Factory (ADF).
- Implementing Mapping Data Flows to create transformation logic to handle a variety of transformation scenarios such as Filter, Conditional Split, Derived Column, Aggregate, Join, Select, and Sink transformation.
- Developing universal components in data pipelines, such as Flowlets, and mastering the swift development of data processing needs through pre-built pipeline templates.
- Learning how to implement error handling in data pipelines and controlling pipeline flow.
- Implementig data quality rules using the Assert transformation within a data pipeline.
- Implementing data pipelines to handle common slowly changing dimension scenarios such as SCD Type 1 and SCD Type 2.
- Implementing data pipleines to implement a Fact table.
- Learning how to debug data pipelines and resolving issues.
- Implementing pipeline scheduling using different types of triggers such as Event Trigger, Schedule Trigger and Tumbling Window Trigger in Azure Data Factory (ADF)
- Implementing Azure Data Factory pipelines to invoke Mapping Data Flows and executing them.
- Creating ADF pipelines to execute Databricks Notebook activities to carry out transformations and implement a Delta Lake table.
- Creating pipeline dependencies and using the Pipeline activity to orchestrate the ETL/ELT process.
- Implementing trigger dependencies to understand how to chain pipelines and orchestrate the data flow.
- Monitoring data pipelines, creating alert notifications, and reporting data factory metrics using Azure Data Factory Monitor.
- Understanding how to monitor Azure Data Factory pipelines using Azure Monitor using specific Data Factory metrics.
Modern Data Warehouse
- Understand the different types of Data Warehouse Architectures.
- Understand the concepts of a Delta Lake.
- Understand the Dimensional Model and a Star Schema based Data Warehouse.
- Understand the concept of Medallion Layers and how to implement it within the Azure Data Lake Storage.
Azure Databricks
- Understand the creation of an Azure Databricks Workspace, Databricks clusters, Mounting storage accounts, Creating Databricks notebooks, performing transformations using Databricks notebooks, and Invoking Databricks notebooks from Azure Data Factory.
- Understand the implementation of a Delta Lake table using Azure Databricks Notebook activity from an Azure Data Factory pipeline.
- Understand the concepts of Optimizing a Delta Lake Table, Time Travel, Vacuuming, and Delta Logs.
Azure Resources and Azure Storage Solutions
- Learn the different approaches to creating Azure Resources.
- Learn how to create an Azure Storage Account resource, creating containers, and how to upload data through the Azure Portal or through Azure Storage Explorer into the Azure storage resource.
- Learn how to create an Azure SQL Database resource, understand the Pricing Tiers, Creating an Admin User, Creating Tables, Loading Data, Querying the database and interacting with Azure Sql Database through Azure Data Studio.
