How to Become a Data Engineer: Beginners Course

TL;DR

Data engineers process and manage data so it flows downstream reliably, secure and correctly formatted for data scientists and analysts. The role is growing in importance because 85 to 87% of big data projects fail due to unreliable infrastructure and poor data quality. In the US, the median data engineer salary is 90 to 150k a year.

Transcript

welcome to this data engineering crash course created by Justin Chow this course covers the essentials of data engineering you'll learn about databases Docker and analytical engineering you'll explore Advanced topics like data pipeline building with airflow and you'll engage in batch processing with spark and streaming data with Kafka the course cu... Read More

Key Insights

• The failure rate of big data projects is 85 to 87%, driven by unreliable data infrastructures and poor quality data, which is a primary reason the data engineering role has grown in importance.
• Data scientists were historically expected to build the necessary data infrastructure, but incorrect data modeling and redundant work led to high data scientist turnover, creating strong demand for dedicated data engineers.
• The average median salary for data engineers in the US is 90 to 150k a year, though the figure varies depending on location and whether the role is based in the states.
• Data engineers have a crucial role in data-driven decisions in the AI and ML space, especially with the rise of ChatGPT, and ensure efficient management and processing of data as it moves downstream.
• Docker is an open-source platform that simplifies building, shipping, and running applications inside containers, packaging an environment with every dependency so it runs the same on a coworker's machine or in the cloud.
• A Dockerfile is a text file of instructions (a blueprint) used to create a Docker image, which is a read-only, immutable, lightweight standalone executable package including code, runtime, libraries, environment variables, and config files.
• A Docker container is the runtime instance of a Docker image, completely isolated from the host and other containers with its own file system, and can be started, stopped, and deleted independently.
• The course covers databases, Docker, analytical engineering, data pipeline building with Airflow, batch processing with Spark, and streaming data with Kafka, culminating in a full end-to-end pipeline project.

Questions & Answers

Q: Why should you consider getting into data engineering?

Data engineering has grown in importance because 85 to 87% of big data projects fail due to unreliable data infrastructures and poor data quality. Previously data scientists were expected to build infrastructure, leading to incorrect data modeling, redundant work, and high turnover. This created strong demand for data engineers to process data and extract value, with a US median salary of 90 to 150k a year.

Q: What is the average salary for a data engineer in the US?

According to the course, the average median salary for data engineers in the US is 90 to 150k a year. The instructor notes that this number highly depends on the location and whether you are based in the states or not, so the figure will change depending on where you work. It reflects the growing importance of the role.

Q: What topics does this data engineering course cover?

The course covers the essentials of data engineering including databases, Docker, and analytical engineering. It then explores advanced topics like data pipeline building with Airflow, batch processing with Spark, and streaming data with Kafka. The course culminates in a comprehensive project where you create a full end-to-end pipeline, following the instructor to build and evolve a pipeline using open-source tools such as Airbyte.

Q: What is Docker and why is it used in data engineering?

Docker is an open-source platform that simplifies the process of building, shipping, and running applications inside containers. It lets you package your environment with every dependency into a container that can be passed to coworkers or run in the cloud, solving the 'it runs okay on my machine' problem. It ensures environments are reproducible and that applications work the same in production as they did in testing.

Q: What are the three main concepts of Docker?

The three main concepts are the Dockerfile, the Docker image, and the Docker container. The Dockerfile is a text file containing all the instructions, acting as a blueprint. That blueprint creates the Docker image, a lightweight standalone executable package. The image is then placed into a Docker container, which is the running instance of the image, isolated from the host and other containers.

Q: What is the difference between a Docker image and a Docker container?

A Docker image is a read-only, immutable, lightweight standalone executable package that includes code, runtime, libraries, environment variables, and config files. A Docker container is the runtime instance, or running version, of that image. For example, one image can run in three separate containers as isolated environments. You can stop or delete one container without affecting the others, since changes only happen inside a container.

Q: How does Docker solve the 'it runs on my machine' problem?

Docker puts your environment into a container that includes every dependency and silos everything together. When a coworker cannot run your project, you can pass them this container, or run it in the cloud, and everything inside the container will be seen on the other end exactly as it was during development. This keeps applications organized and ensures they work as developed across different machines and production.

Q: What impact do data engineers have on a company?

Data engineers play a crucial role in making data-driven decisions in the AI and ML space, especially with the rise of ChatGPT. They ensure efficient management and processing of data as it moves downstream, guaranteeing its quality, security, availability, and proper formatting for data scientists, analysts, and other teams. This leads to better insights and outcomes, driving innovation and competitiveness so the company and team can thrive.

Summary & Key Takeaways

• Created by Justin Chau, a developer advocate at Airbyte, this data engineering crash course covers the essentials including databases, Docker, and analytical engineering, plus advanced topics like pipeline building with Airflow, batch processing with Spark, and streaming with Kafka, ending in a full end-to-end pipeline project.

• Data engineering matters because 85 to 87% of big data projects fail due to unreliable infrastructures and quality data. Data scientists once had to build this infrastructure, causing incorrect data modeling, redundant work, and high turnover, which raised demand for data engineers whose US median salary is 90 to 150k a year.

• Docker is introduced as the first module: an open-source platform that packages applications and dependencies into isolated containers. Its three core concepts are the Dockerfile (a text-file blueprint), the Docker image (a read-only immutable executable package), and the Docker container (the isolated, running instance of an image).