Azure Data Factory – Job-Oriented Professional Training Program

This Azure Data Factory – Job-Oriented Professional Training Program is designed to provide practical, industry-oriented training in cloud-based data integration, ETL/ELT, data pipelines, Azure storage, databases, data transformation, orchestration, monitoring, security, and deployment.

Students will learn how to design and implement scalable and production-ready data pipelines using Azure Data Factory (ADF) and integrate data from multiple sources such as SQL Server, Azure SQL Database, Azure Blob Storage, Azure Data Lake Storage, REST APIs, files, and other cloud/on-premises systems.

The course focuses on hands-on development, real-world data integration scenarios, pipeline automation, troubleshooting, monitoring, and an end-to-end data engineering project.

Students should have:

Basic IT Knowledge

• ☑ Basic computer knowledge
• ☑ Understanding of files and folders
• ☑ Basic command-line knowledge
• ☑ Basic understanding of cloud computing

Database Knowledge

• ☑ Basic SQL knowledge
• ☑ SELECT statements
• ☑ WHERE conditions
• ☑ JOINs
• ☑ GROUP BY
• ☑ Aggregate functions
• ☑ Basic understanding of tables and relationships

Data Concepts

• ☑ Basic understanding of databases
• ☑ Basic understanding of ETL concepts
• ☑ Basic understanding of structured and semi-structured data

Recommended

• ☑ Basic Azure knowledge
• ☑ Basic Python knowledge – helpful but not mandatory
• ☑ Basic understanding of data warehousing

No prior Azure Data Factory experience is required.

Required

• ☑ Web Browser – Chrome / Edge
• ☑ Microsoft Azure Account
• ☑ Azure Portal Access
• ☑ Azure Data Factory Studio
• ☑ Visual Studio Code
• ☑ Git
• ☑ GitHub account

Database Tools

Install at least one:

• ☑ SQL Server
• ☑ SQL Server Management Studio (SSMS)
• ☑ Azure Data Studio – where applicable
• ☑ DBeaver – optional

Data Processing / File Tools

• ☑ Microsoft Excel
• ☑ CSV / JSON file editors
• ☑ Visual Studio Code
• ☑ Azure Storage Explorer

Optional Tools

• ☑ Python
• ☑ Postman
• ☑ PowerShell
• ☑ Azure CLI
• ☑ GitHub Desktop

Azure Resources Required for Hands-On Training

Depending on the lab:

• ☑ Azure Data Factory
• ☑ Azure Storage Account
• ☑ Azure Blob Storage
• ☑ Azure Data Lake Storage Gen2
• ☑ Azure SQL Database
• ☑ Azure Key Vault
• ☑ Azure Synapse Analytics – optional
• ☑ Azure DevOps / GitHub – for CI/CD exercises

Important: Azure services may generate charges. Training should preferably use a controlled lab subscription, Microsoft Azure free/credit resources where eligible, or an organizational Azure account.

By completing this course, students will be able to:

• ☑ Understand Azure Data Factory architecture
• ☑ Create and manage ADF resources
• ☑ Build ETL/ELT pipelines
• ☑ Configure Linked Services and Datasets
• ☑ Work with Integration Runtime
• ☑ Use Copy Activity
• ☑ Build dynamic and parameterized pipelines
• ☑ Use expressions and dynamic content
• ☑ Implement Mapping Data Flows
• ☑ Integrate SQL Server and Azure SQL
• ☑ Work with Azure Blob Storage
• ☑ Work with Azure Data Lake Storage Gen2
• ☑ Integrate REST APIs
• ☑ Implement incremental data loading
• ☑ Build metadata-driven pipelines
• ☑ Schedule and automate pipelines
• ☑ Monitor and troubleshoot pipelines
• ☑ Optimize ADF performance
• ☑ Secure pipelines using Managed Identity and Key Vault
• ☑ Integrate ADF with Git
• ☑ Understand CI/CD for Azure Data Factory
• ☑ Build an end-to-end cloud data engineering project 

Cloud & Azure Fundamentals

• ☑ Introduction to Cloud Computing
• ☑ IaaS, PaaS and SaaS
• ☑ Azure fundamentals
• ☑ Azure Portal
• ☑ Azure Resource Groups
• ☑ Azure Regions
• ☑ Availability Zones
• ☑ Azure subscriptions
• ☑ Resource management

Data Engineering Fundamentals

• ☑ What is Data Engineering?
• ☑ Data sources
• ☑ Structured data
• ☑ Semi-structured data
• ☑ Unstructured data
• ☑ ETL vs ELT
• ☑ Batch processing
• ☑ Real-time processing concepts
• ☑ Data pipelines
• ☑ Data warehouses
• ☑ Data lakes

Practical Lab

• ☑ Create Azure account/resource group
• ☑ Explore Azure Portal
• ☑ Create basic Azure resources 

Introduction to ADF

• ☑ What is Azure Data Factory?
• ☑ ADF architecture
• ☑ Data integration concepts
• ☑ ADF components
• ☑ Azure Data Factory Studio
• ☑ Authoring interface
• ☑ Monitoring interface

Core Components

• ☑ Pipelines
• ☑ Activities
• ☑ Datasets
• ☑ Linked Services
• ☑ Integration Runtime
• ☑ Triggers
• ☑ Parameters
• ☑ Variables

Practical Labs

• ☑ Create Azure Data Factory
• ☑ Explore ADF Studio
• ☑ Create first pipeline
• ☑ Configure pipeline components 

Linked Services

• ☑ Linked Service fundamentals
• ☑ Azure SQL Linked Service
• ☑ Blob Storage Linked Service
• ☑ Data Lake Linked Service
• ☑ REST API Linked Service

Datasets

• ☑ Dataset fundamentals
• ☑ File datasets
• ☑ Database datasets
• ☑ Parameterized datasets
• ☑ Dynamic datasets

Integration Runtime

• ☑ Azure Integration Runtime
• ☑ Self-hosted Integration Runtime
• ☑ Integration Runtime architecture
• ☑ Connectivity concepts
• ☑ On-premises data integration

Practical Labs

• ☑ Connect ADF to Azure SQL
• ☑ Connect ADF to Blob Storage
• ☑ Configure Self-hosted Integration Runtime
• ☑ Create parameterized datasets

Copy Activity

• ☑ Copy Activity fundamentals
• ☑ Source configuration
• ☑ Sink configuration
• ☑ Source and sink datasets
• ☑ File-based data movement
• ☑ Database-to-database movement
• ☑ Cloud-to-cloud movement

Supported Data Scenarios

• ☑ SQL Server → Azure SQL
• ☑ SQL Server → Data Lake
• ☑ CSV → Azure SQL
• ☑ Blob → Data Lake
• ☑ REST API → Storage

Performance

• ☑ Parallel copy
• ☑ Batch size
• ☑ Data partitioning
• ☑ Copy performance optimization
• ☑ Fault tolerance
• ☑ Retry configuration

Practical Labs

• ☑ Build data-copy pipelines
• ☑ Load CSV files into Azure SQL
• ☑ Move SQL data into Data Lake
• ☑ Configure copy performance 

Control Flow Activities

• ☑ Execute Pipeline
• ☑ Lookup
• ☑ Get Metadata
• ☑ ForEach
• ☑ If Condition
• ☑ Switch
• ☑ Set Variable
• ☑ Append Variable
• ☑ Wait
• ☑ Web Activity

Pipeline Parameters

• ☑ Pipeline parameters
• ☑ Dataset parameters
• ☑ Variables
• ☑ Dynamic expressions
• ☑ Parameterized pipelines

Practical Labs

• ☑ Build parameterized pipelines
• ☑ Create dynamic ForEach pipelines
• ☑ Implement conditional processing
• ☑ Build reusable pipelines 

Dynamic Expressions

• ☑ Expression Builder
• ☑ String functions
• ☑ Date functions
• ☑ Logical functions
• ☑ Mathematical functions
• ☑ Collection functions
• ☑ Conversion functions

Dynamic Pipeline Design

• ☑ Dynamic file names
• ☑ Dynamic folder paths
• ☑ Dynamic table names
• ☑ Dynamic dates
• ☑ Runtime parameters
• ☑ Metadata-driven pipelines

Practical Labs

• ☑ Create dynamic file processing
• ☑ Build date-based pipelines
• ☑ Implement dynamic source and destination paths
• ☑ Build metadata-driven ingestion 

Mapping Data Flows

• ☑ Introduction to Mapping Data Flow
• ☑ Data Flow architecture
• ☑ Source transformation
• ☑ Select transformation
• ☑ Filter transformation
• ☑ Derived Column
• ☑ Aggregate
• ☑ Join
• ☑ Conditional Split
• ☑ Lookup
• ☑ Sort
• ☑ Union
• ☑ Sink

Data Transformation

• ☑ Data cleansing
• ☑ Data type conversion
• ☑ Null handling
• ☑ Derived columns
• ☑ Aggregation
• ☑ Joins
• ☑ Data validation

Practical Labs

• ☑ Create Mapping Data Flow
• ☑ Transform CSV data
• ☑ Join multiple datasets
• ☑ Clean and aggregate data
• ☑ Load transformed data into Azure SQL/Data Lake 

Trigger Types

• ☑ Manual triggers
• ☑ Schedule triggers
• ☑ Tumbling Window triggers
• ☑ Event-based triggers
• ☑ Storage event triggers

Scheduling

• ☑ Daily pipelines
• ☑ Hourly pipelines
• ☑ Time-zone considerations
• ☑ Dependency management
• ☑ Trigger parameters

Practical Labs

• ☑ Schedule daily data ingestion
• ☑ Build hourly pipelines
• ☑ Trigger pipelines when files arrive
• ☑ Implement dependent pipelines 

Incremental Data Loading

• ☑ Full load vs incremental load
• ☑ Watermark concept
• ☑ Last modified date
• ☑ Timestamp-based loading
• ☑ Incremental SQL queries
• ☑ Change tracking concepts

Metadata-Driven Architecture

• ☑ Metadata tables
• ☑ Configuration-driven pipelines
• ☑ Dynamic source mapping
• ☑ Dynamic sink mapping
• ☑ Reusable ingestion framework

Practical Project

• ☑ Build incremental data pipeline
• ☑ Implement watermark-based loading
• ☑ Build metadata-driven ingestion framework 

Azure Storage

• ☑ Azure Blob Storage
• ☑ Azure Data Lake Storage Gen2
• ☑ Containers
• ☑ Filesystems
• ☑ Directories
• ☑ Storage tiers
• ☑ Data organization

Data Lake Architecture

• ☑ Raw data layer
• ☑ Cleansed data layer
• ☑ Curated data layer
• ☑ Medallion architecture concepts
• ☑ File formats

File Formats

• ☑ CSV
• ☑ JSON
• ☑ Parquet
• ☑ Avro – introduction

Practical Labs

• ☑ Create Data Lake
• ☑ Build raw/processed/curated folders
• ☑ Load data into Data Lake
• ☑ Transform and organize datasets 

API Fundamentals

• ☑ REST API concepts
• ☑ HTTP methods
• ☑ GET
• ☑ POST
• ☑ Authentication
• ☑ Headers
• ☑ JSON responses

ADF REST Integration

• ☑ REST Linked Services
• ☑ REST datasets
• ☑ Pagination
• ☑ API parameters
• ☑ Dynamic API URLs
• ☑ API authentication concepts

Practical Lab

• ☑ Extract data from REST API
• ☑ Store API data in Azure Data Lake
• ☑ Build scheduled API ingestion pipeline 

Monitoring

• ☑ ADF Monitoring
• ☑ Pipeline runs
• ☑ Activity runs
• ☑ Trigger monitoring
• ☑ Integration Runtime monitoring

Troubleshooting

• ☑ Pipeline failures
• ☑ Activity failures
• ☑ Connection errors
• ☑ Authentication errors
• ☑ Data type errors
• ☑ Timeout issues

Performance Optimization

• ☑ Pipeline optimization
• ☑ Copy Activity optimization
• ☑ Parallelism
• ☑ Partitioning
• ☑ Data Flow optimization
• ☑ Integration Runtime optimization

Practical Labs

• ☑ Troubleshoot failed pipelines
• ☑ Analyze pipeline execution
• ☑ Optimize slow pipelines
• ☑ Implement retry and timeout strategies

Azure Security

• ☑ Azure Identity fundamentals
• ☑ Azure RBAC
• ☑ Role assignments
• ☑ Managed identities
• ☑ Service principals

ADF Security

• ☑ Secure Linked Services
• ☑ Managed Identity authentication
• ☑ Credentials management
• ☑ Secure connection configuration

Azure Key Vault

• ☑ Key Vault fundamentals
• ☑ Secrets
• ☑ ADF integration
• ☑ Secret-based authentication

Practical Labs

• ☑ Configure Managed Identity
• ☑ Store credentials in Key Vault
• ☑ Secure ADF pipelines
• ☑ Implement RBAC 

Source Control

• ☑ Git integration with ADF
• ☑ GitHub / Azure DevOps concepts
• ☑ Branching
• ☑ Pull Requests
• ☑ Collaboration

Deployment

• ☑ Development environment
• ☑ Test environment
• ☑ Production environment
• ☑ ARM/Bicep deployment concepts
• ☑ Automated deployment concepts
• ☑ CI/CD pipeline for ADF

Practical Labs

• ☑ Connect ADF to Git
• ☑ Create feature branches
• ☑ Publish ADF changes
• ☑ Implement environment-based deployment
• ☑ Build CI/CD workflow 

🚀 End-to-End Data Engineering Project

Students will design and implement a production-style data integration solution using Azure Data Factory.

Project Scenario

Multiple Data Sources → Azure Data Factory → Data Lake → Transformation → Azure SQL / Data Warehouse → Reporting

Data Sources

• ☑ SQL Server
• ☑ CSV files
• ☑ JSON files
• ☑ REST API
• ☑ Azure SQL Database

Project Tasks

• ☑ Create Azure Data Factory
• ☑ Configure Linked Services
• ☑ Create Datasets
• ☑ Configure Integration Runtime
• ☑ Build ingestion pipelines
• ☑ Implement incremental loading
• ☑ Create dynamic pipelines
• ☑ Implement Mapping Data Flows
• ☑ Store raw data in Azure Data Lake
• ☑ Create processed and curated data layers
• ☑ Load transformed data into Azure SQL
• ☑ Configure scheduled triggers
• ☑ Implement monitoring
• ☑ Configure error handling
• ☑ Secure credentials using Key Vault
• ☑ Implement Git integration
• ☑ Configure CI/CD deployment 

Source Systems

↓

SQL Server / CSV / REST API / Azure SQL

↓

Azure Data Factory

↓

Raw Data – Azure Data Lake Gen2

↓

Data Transformation – Mapping Data Flow

↓

Processed / Curated Data

↓

Azure SQL / Data Warehouse

↓

BI / Reporting Layer

Azure Data Integration

• ☑ Azure Data Factory
• ☑ Azure Data Factory Studio
• ☑ Integration Runtime
• ☑ Mapping Data Flow

Azure Storage

• ☑ Azure Blob Storage
• ☑ Azure Data Lake Storage Gen2
• ☑ Azure Storage Explorer

Databases

• ☑ SQL Server
• ☑ Azure SQL Database
• ☑ SQL

Azure Services

• ☑ Azure Key Vault
• ☑ Azure Resource Groups
• ☑ Azure RBAC
• ☑ Managed Identity
• ☑ Azure Monitor

Development & Version Control

• ☑ Git
• ☑ GitHub
• ☑ Azure DevOps – optional
• ☑ Visual Studio Code

Data Formats

• ☑ CSV
• ☑ JSON
• ☑ Parquet
• ☑ Avro – introduction

APIs

• ☑ REST APIs
• ☑ HTTP
• ☑ JSON

• ☑ Azure Data Factory Developer
• ☑ Azure Data Engineer
• ☑ Data Engineer
• ☑ Azure ETL Developer
• ☑ ETL Developer
• ☑ Cloud Data Engineer
• ☑ Data Integration Developer
• ☑ Azure Data Integration Engineer
• ☑ Junior Azure Data Engineer
• ☑ Associate Data Engineer
• ☑ BI / Data Engineer
• ☑ Cloud ETL Developer
• ☑ Data Pipeline Developer
• ☑ Azure Analytics Engineer – Entry Level 

Azure Data Factory Developer

• ☑ Design and develop Azure Data Factory pipelines
• ☑ Create data integration workflows
• ☑ Configure Linked Services and Datasets
• ☑ Implement Copy Activities
• ☑ Configure pipeline parameters and variables
• ☑ Implement triggers and scheduling
• ☑ Monitor pipeline execution
• ☑ Troubleshoot failed data pipelines
• ☑ Optimize pipeline performance
• ☑ Implement reusable pipeline components

Azure Data Engineer

• ☑ Design cloud-based data integration solutions
• ☑ Build ETL/ELT pipelines
• ☑ Integrate structured and semi-structured data
• ☑ Work with Azure Data Lake Storage
• ☑ Implement data transformation workflows
• ☑ Work with SQL and cloud databases
• ☑ Implement incremental data loading
• ☑ Design scalable data pipelines
• ☑ Implement security and access control
• ☑ Monitor and optimize data workloads

Data Integration Engineer

• ☑ Connect multiple data sources
• ☑ Build source-to-target data flows
• ☑ Implement data validation
• ☑ Transform and cleanse data
• ☑ Schedule automated data processing
• ☑ Handle pipeline failures
• ☑ Maintain data integration workflows
• ☑ Document data pipelines and mappings

After completing this program, learners should be able to design, develop, automate, monitor, secure and deploy cloud-based data integration pipelines using Azure Data Factory.

They will have practical experience working with Azure Data Factory, Azure Data Lake, Azure SQL, SQL Server, REST APIs, data transformation, incremental loading, metadata-driven pipelines, monitoring, security and CI/CD.

Final Skill Path

SQL & Data Fundamentals → Azure Fundamentals → Azure Data Factory → Linked Services & Datasets → Copy Activity → Pipeline Automation → Data Flow → Azure Data Lake → Incremental Loading → Metadata-Driven Pipelines → Monitoring → Security → Git & CI/CD → Real-World Project