Introduction to data engineeringTopics

The topic introduces the field of data engineering, which is considered the foundation of modern data architecture. students learn the purpose of the field and the role of the data engineer in the process of collecting, storing, and processing large volumes of data. also, the fundamental differences between a data engineer who builds infrastructure and a data scientist who develops models and predictions from that data and the application areas of these professions in the industry are analyzed.

Data, their internal structures and management methods, which are the main raw materials of data analytics and engineering, are studied in the subject. students analyze the differences, data sources, and ways of collection between structured data, which has a certain orderly pattern, and free-form unstructured data. the most commonly used transmission and storage formats in the industry (csv, json, xml), data quality measurement to ensure data reliability, and principles of raw data cleaning are also explored.

The subject studies the processes of bringing raw data from different sources into a single format and preparing them for analytics. students analyze the ETL steps that are the foundation of data engineering, and the normalization and denormalization concepts used for database optimization. also, the principles of effective integration of data from heterogeneous sources in a single repository are investigated.

In the subject, the management systems of relational databases and the basics of the SQL language, which is used to manipulate data, are studied. students analyze the principles of database design and modeling through table structures, primary and foreign keys. in the practical part, fundamental SQL commands and various SQL operations are mastered for reading and managing data.

In the subject, nosql databases designed for storing large-volume, rapidly changing and unstructured data are studied. students analyze the differences from relational (sql) databases and the main features of non-relational systems. during the lesson, the 4 main types of nosql - document, key-value, column and graph structures are examined and the first practical operations on mongodb, a document-oriented database, are mastered.

Concepts of centralized storage and management of large volumes of corporate data for analytical purposes are studied in the subject. students analyze the fundamental differences between transaction-oriented OLTP (online transaction processing) systems and analytical-oriented OLAP (online analytical processing) systems. also, the working principles of hive and hbase technologies, which are the main pillars of the big data ecosystem, are investigated, according to the project needs, dedicated data showcases for specific business areas, data lake environments where raw and unstructured big data are stored.

Data pipeline systems that automate the process of processing raw data from the source and delivering it to the destination are studied in the subject. students analyze the differences between batch processing, which provides batch processing of data at specified time intervals, and real-time processing, which provides instant processing of data as it is generated. it also explores the working principles and industry role of orchestration tools that manage the sequence and dependencies of tasks in complex pipelines.

The subject studies the big data ecosystem, a large and complex set of data that is impossible to process with traditional database systems. students analyze the 3v model, which defines the fundamental characteristics of big data, and the hadoop environment, which provides a distributed storage architecture. also, the working principles of the apache spark platform, which offers faster analytics thanks to the mapreduce programming model and in-memory computing technology used for parallel processing of data on clusters, are explored.

In the subject, the principles of protecting data from outside interference, regulation of user rights and compliance with legal standards are studied in the corporate data infrastructure. students analyze the basics of data protection, encryption methods used to secure data being transmitted and stored, as well as authentication mechanisms that check system access permissions. also, the requirements of the GDPR (general data protection regulation) and other international normative acts, which regulate the protection of personal data in the digital environment at the global level, are examined.

In the topic, the role of cloud technologies in the formation and scaling of modern data architecture is studied. students analyze industry-leading cloud platforms (aws, azure, google cloud) and the key service ecosystems they provide for data engineering. also, the concept of serverless architecture, which minimizes infrastructure management and is based on a pay-as-you-go model, its working principles and advantages over traditional server systems are practically explored.

Modern methodologies that increase flexibility, quality and speed in software development and data project management are studied in the subject. students analyze devops principles that integrate programming and systems administration. it also explores the dataops approach, a data engineering adaptation of these principles, its impact on the quality of data pipelines, as well as ci/cd processes that ensure that code and data are automatically tested and integrated into the system.

In the subject, raw data preprocessing processes are studied for the correct and effective operation of machine learning models. students analyze the preprocessing phase, which removes outliers and biases in the data, and feature engineering, the process of extracting new meaningful variables from existing data to improve model accuracy. also, applied scaling and transformation methods to adapt the data distribution structure to the model are investigated to bring the numerical values ​​of different ranges to the same level.

The subject studies ways to clearly and effectively communicate the results of complex data analysis to business decision makers. students analyze the importance of transforming large volumes of data into visual graphs, the industry's leading analytical tools (tableau, power bi) and their capabilities. also, the principles of designing easy-to-read, interactive dashboards and reports, color and graphics selection methods are explored.

Optimization methods and system monitoring are studied in the subject, which ensure fast, effective operation of large data systems and queries. students analyze the mechanisms of indexing, which speeds up database searches, partitioning, which divides huge tables into manageable chunks, and caching, which stores frequently accessed data in memory. it also explores the working principles of prometheus (metrics collection) and grafana (visual monitoring dashboards), tools widely used in the industry to monitor system resource usage, latencies, and errors in real-time.

In this final topic of the course, theoretical and practical knowledge gained by students during the semester is combined on a real project. students present their individual or team data engineering projects, discuss them with colleagues and faculty members, and pass the evaluation stage. in the second part of the lesson, a general overview of the entire course is given and modern trends in the field of data engineering and future development prospects are analyzed.