Data Engineering
Learn to build the pipelines and storage that move and prepare data for analysis.
- Fees
- Contact us for fees
About this course
Data engineers build the systems that collect data from many sources, clean it, store it and deliver it, reliably and on schedule, to analysts and models. This course teaches those systems from the ground up: SQL and data modelling, Python pipelines, a warehouse, Spark for large data, Airflow for scheduling and Kafka for live streams.
You finish by designing, building and documenting one complete pipeline. The aim is not only to move data but to make it trustworthy: tested, monitored and easy to rerun when something breaks.
What you will be able to do
- Explain the data engineering lifecycle and the difference between batch and streaming pipelines.
- Write advanced SQL and design star-schema models for analytics.
- Build ETL and ELT pipelines in Python with logging, error handling and tests.
- Process large datasets with Apache Spark (PySpark) and store them in efficient formats such as Parquet.
- Schedule, monitor and retry workflows with Apache Airflow.
- Ingest and process live event streams with Apache Kafka.
- Add data-quality checks and document a pipeline so that another engineer can run it.
Syllabus
10 modules. Select a module to see its topics.
Module 1: Data engineering foundations
- The role and the lifecycle
- Data sources and formats (CSV, JSON, Parquet)
- Batch and streaming
- ETL and ELT
- The Linux command line and Git
Module 2: Python for pipelines
- Scripts and modules
- Files and APIs
- pandas transformations
- Logging and error handling
- Configuration and secrets
- Tests with pytest
Module 3: SQL and data modelling
- Joins, CTEs and window functions
- Normalisation
- Star and snowflake schemas
- Fact and dimension tables
- Slowly changing dimensions
- Indexes and query plans
Module 4: Databases and data warehouses
- Relational and NoSQL stores
- PostgreSQL
- How warehouses differ from transactional databases
- Columnar storage
- Partitioning
- Loading strategies
- A cloud warehouse such as Snowflake in outline
Module 5: Data lakes and cloud storage
- Object storage such as Amazon S3
- File formats (Parquet, Avro)
- The lakehouse idea
- Folder and partition layout
- Access control basics
Module 6: Batch processing with Apache Spark
- Spark architecture
- DataFrames and Spark SQL in PySpark
- Joins and aggregations at scale
- Performance tuning basics
- Writing results to Parquet
Module 7: Workflow orchestration with Apache Airflow
- DAGs
- Operators and sensors
- Scheduling and backfills
- Retries and alerts
- Connections and variables
- Testing a DAG
Module 8: Streaming with Apache Kafka
- Topics and partitions
- Producers and consumers
- Consumer groups
- Delivery semantics (at least once, exactly once)
- A simple stream-processing pipeline
Module 9: Data quality, testing and governance
- Validation rules
- Automated data tests
- Lineage and documentation
- Masking and access control
- Pipeline monitoring and cost control
- dbt for transformations in outline
Module 10: Capstone pipeline
- Choosing a business case and writing the design
- Ingesting data from two or more sources
- Storing, transforming and serving it for analysis
- Scheduling, testing and monitoring the pipeline
- Documentation and a short design review
Tools and technologies
- Python
- SQL
- PostgreSQL
- Apache Spark (PySpark)
- Apache Airflow
- Apache Kafka
- A cloud warehouse (Snowflake)
- Cloud object storage (Amazon S3)
- Docker
- dbt (in outline)
- Linux
- Git and GitHub
Projects you will build
Batch ETL pipeline that pulls data from an API and files into a warehouse.
Dimensional model (star schema) for a sales dataset, with analytical queries on top.
Daily pipeline scheduled in Airflow, with retries, alerts and a backfill.
Real-time event pipeline with Kafka feeding a live summary table.
Data-quality checks and a pipeline-health report.
Capstone: a documented, tested end-to-end pipeline for a chosen business case.
Before you start
- This outline starts from the basics: Python and SQL are taught in the first modules, so no earlier coding is assumed.
- Helpful but not required: basic computer skills, logical thinking and a laptop with enough memory for local tools such as Docker and Spark.
- Entry requirements, the minimum laptop specification and whether labs run in the cloud are confirmed by Pathway Career when you enquire.
Career paths
Examples of roles that use these skills. They are not a promise of a job.
- Junior data engineer
- ETL developer
- Data pipeline engineer
- Big data engineer
- Data warehouse engineer
- Analytics engineer
Questions about this course
Do I need to code before I start?
This outline assumes no coding experience: Python and SQL are taught from the basics in the first modules. Entry requirements are confirmed by Pathway Career when you enquire.
How is this different from Data Science and Data Analytics?
Data engineers build and run the data systems. Analysts and scientists use what those systems deliver. This course teaches the building side.
Are Spark, Airflow and Kafka all covered?
Yes, each has its own module with hands-on work. Pathway Career confirms the versions taught, and whether labs run locally or in the cloud, when you enquire.
Do I need a cloud account?
Cloud storage and warehouse exercises may need a free-tier or trial account, and cloud services can charge by use. Pathway Career confirms how this is handled for students when you enquire.
Does the course cover Snowflake in depth?
Snowflake appears here as an example warehouse. The separate Snowflake course covers the platform in depth.
Next step
Want to know more about Data Engineering? Send an enquiry to ask for the full syllabus, batch dates and fees.
Send an enquiryRelated courses
AI & Machine Learning
Learn how AI systems learn from data: core algorithms, training models and evaluating results.