Introduction to data engineeringSyllabus

Subject code

İFS-40182y

Credits

6

Student workload

180

Semester

Spring semester

Academic year

2025-2026

Subject type

Elective

Form of education

Full-time

Language of instruction

Azerbaijani

In-class hours

a) 30 saat - mühazirə b) 30 saat - seminar

Out-of-class hours

a) 72 saat - sərbəst iş b) 48 saat - hazırlıq

Introduction to data engineering teaches students the basic concepts, principles and modern technologies of data engineering. the subject covers data collection, storage, processing and integration, ETL processes, databases, big data platforms, data pipelines, cloud technologies, devops and dataops approaches, data security and data visualization. practical skills are developed using tools such as python, sql, mongodb, hadoop, spark and power BI in the laboratory exercises. at the end of the course, students acquire basic knowledge and skills in data management and development of analytical solutions for real projects.

• lecture
• interactive discussion
• seminar
• presentations
• project-based learning

1. 1

   CLO 1: It explains the basic concepts and principles of data engineering.

2. 2

   CLO 2: It applies methods of data collection, storage and processing.

3. 3

   CLO 3: Works with SQL and nosql databases.

4. 4

   CLO 4: Develops ETL processes and data pipelines.

5. 5

   CLO 5: It uses big data and cloud technologies.

6. 6

   CLO 6: Analyzes data with data analytics and visualization tools.

1. 1.
   Introduction: fundamentals of data engineeringLecture

   The topic introduces the field of data engineering, which is considered the foundation of modern data architecture. students learn the purpose of the field and the role of the data engineer in the process of collecting, storing, and processing large volumes of data. also, the fundamental differences between a data engineer who builds infrastructure and a data scientist who develops models and predictions from that data and the application areas of these professions in the industry are analyzed.

   • It explains the basic concepts and principles of data engineering.
2. 2.
   Data and their typesLecture

   Data, their internal structures and management methods, which are the main raw materials of data analytics and engineering, are studied in the subject. students analyze the differences, data sources, and ways of collection between structured data, which has a certain orderly pattern, and free-form unstructured data. the most commonly used transmission and storage formats in the industry (csv, json, xml), data quality measurement to ensure data reliability, and principles of raw data cleaning are also explored.

   • It applies methods of data collection, storage and processing.
3. 3.
   Transformation and integrationLecture

   The subject studies the processes of bringing raw data from different sources into a single format and preparing them for analytics. students analyze the ETL steps that are the foundation of data engineering, and the normalization and denormalization concepts used for database optimization. also, the principles of effective integration of data from heterogeneous sources in a single repository are investigated.

   • Develops ETL processes and data pipelines.
4. 4.
   Databases and SQLLecture

   In the subject, the management systems of relational databases and the basics of the SQL language, which is used to manipulate data, are studied. students analyze the principles of database design and modeling through table structures, primary and foreign keys. in the practical part, fundamental SQL commands and various SQL operations are mastered for reading and managing data.

   • Works with SQL and nosql databases.
5. 5.
   Nosql databasesLecture

   In the subject, nosql databases designed for storing large-volume, rapidly changing and unstructured data are studied. students analyze the differences from relational (sql) databases and the main features of non-relational systems. during the lesson, the 4 main types of nosql - document, key-value, column and graph structures are examined and the first practical operations on mongodb, a document-oriented database, are mastered.

   • Works with SQL and nosql databases.
6. 6.
   Data storage and data warehousingLecture

   Concepts of centralized storage and management of large volumes of corporate data for analytical purposes are studied in the subject. students analyze the fundamental differences between transaction-oriented OLTP (online transaction processing) systems and analytical-oriented OLAP (online analytical processing) systems. also, the working principles of hive and hbase technologies, which are the main pillars of the big data ecosystem, are investigated, according to the project needs, dedicated data showcases for specific business areas, data lake environments where raw and unstructured big data are stored.

   • It applies methods of data collection, storage and processing.
7. 7.
   Data pipeline: concept and componentsLecture

   Data pipeline systems that automate the process of processing raw data from the source and delivering it to the destination are studied in the subject. students analyze the differences between batch processing, which provides batch processing of data at specified time intervals, and real-time processing, which provides instant processing of data as it is generated. it also explores the working principles and industry role of orchestration tools that manage the sequence and dependencies of tasks in complex pipelines.

8. 8.
   Big data technologiesLecture

   The subject studies the big data ecosystem, a large and complex set of data that is impossible to process with traditional database systems. students analyze the 3v model, which defines the fundamental characteristics of big data, and the hadoop environment, which provides a distributed storage architecture. also, the working principles of the apache spark platform, which offers faster analytics thanks to the mapreduce programming model and in-memory computing technology used for parallel processing of data on clusters, are explored.

9. 9.
   Data security and privacyLecture

   In the subject, the principles of protecting data from outside interference, regulation of user rights and compliance with legal standards are studied in the corporate data infrastructure. students analyze the basics of data protection, encryption methods used to secure data being transmitted and stored, as well as authentication mechanisms that check system access permissions. also, the requirements of the GDPR (general data protection regulation) and other international normative acts, which regulate the protection of personal data in the digital environment at the global level, are examined.

10. 10.
    Cloud computing and data engineeringLecture

    In the topic, the role of cloud technologies in the formation and scaling of modern data architecture is studied. students analyze industry-leading cloud platforms (aws, azure, google cloud) and the key service ecosystems they provide for data engineering. also, the concept of serverless architecture, which minimizes infrastructure management and is based on a pay-as-you-go model, its working principles and advantages over traditional server systems are practically explored.

11. 11.
    Devops and dataops approachesLecture

    Modern methodologies that increase flexibility, quality and speed in software development and data project management are studied in the subject. students analyze devops principles that integrate programming and systems administration. it also explores the dataops approach, a data engineering adaptation of these principles, its impact on the quality of data pipelines, as well as ci/cd processes that ensure that code and data are automatically tested and integrated into the system.

12. 12.
    Data preparation for machine learningLecture

    In the subject, raw data preprocessing processes are studied for the correct and effective operation of machine learning models. students analyze the preprocessing phase, which removes outliers and biases in the data, and feature engineering, the process of extracting new meaningful variables from existing data to improve model accuracy. also, applied scaling and transformation methods to adapt the data distribution structure to the model are investigated to bring the numerical values ​​of different ranges to the same level.

13. 13.
    Data visualization and analytics platformsLecture

    The subject studies ways to clearly and effectively communicate the results of complex data analysis to business decision makers. students analyze the importance of transforming large volumes of data into visual graphs, the industry's leading analytical tools (tableau, power bi) and their capabilities. also, the principles of designing easy-to-read, interactive dashboards and reports, color and graphics selection methods are explored.

14. 14.
    Performance evaluation in data engineering systemsLecture

    Optimization methods and system monitoring are studied in the subject, which ensure fast, effective operation of large data systems and queries. students analyze the mechanisms of indexing, which speeds up database searches, partitioning, which divides huge tables into manageable chunks, and caching, which stores frequently accessed data in memory. it also explores the working principles of prometheus (metrics collection) and grafana (visual monitoring dashboards), tools widely used in the industry to monitor system resource usage, latencies, and errors in real-time.

15. 15.
    Data engineering project and course reviewLecture

    In this final topic of the course, theoretical and practical knowledge gained by students during the semester is combined on a real project. students present their individual or team data engineering projects, discuss them with colleagues and faculty members, and pass the evaluation stage. in the second part of the lesson, a general overview of the entire course is given and modern trends in the field of data engineering and future development prospects are analyzed.

No reading list has been added.