Data engineer roadmap for 2027: SQL, pipelines and the cloud

A data engineer roadmap for 2027 has nine stages: SQL and data modelling, Python, one warehouse, batch pipelines and orchestration, transformation, streaming basics, cloud storage and compute, data quality and an end-to-end project. Google Cloud describes a Professional Data Engineer as someone who collects, transforms, stores and delivers data for diverse applications.

Last updated: 11 November 2026.

What does a data engineer do?

A data engineer builds and maintains the systems that move data from where it is created to where people and models use it. Google Cloud says a Professional Data Engineer designs and builds data infrastructure while optimising for performance and security, and administers data platforms.

AWS says its Certified Data Engineer - Associate (DEA-C01) validates the ability to ingest and transform data, orchestrate data pipelines while applying programming concepts, design data models, manage data life cycles and ensure data quality. Microsoft's Fabric Data Engineer Associate (DP-700) expects expertise in data loading patterns, data architectures and orchestration, and skill in SQL, PySpark and Kusto Query Language.

The three lists agree on five jobs: ingest, transform, orchestrate, model and check quality. The ETL and data pipelines explainer covers the first two in detail.

How is a data engineer different from a data analyst or a data scientist?

A data engineer prepares and delivers data, a data analyst turns it into business insight, and a data scientist applies analytical techniques to extract meaning from it. The table compares the three using each role's primary definition.

Microsoft's Power BI Data Analyst Associate page says the analyst collaborates with analytics engineers and data engineers to identify and acquire data, so the roles are a chain and not alternatives. The data analyst roadmap covers the analyst path, and a longer role comparison is in Data Analyst vs Data Scientist vs Data Engineer. Pay differs by employer, city and experience and is not compared here.

What are the stages of a data engineer roadmap?

The roadmap has nine stages, and each lists the tools named in the vendors' documentation, one project and the proof of skill. The table summarises them.

How do you build a pipeline you can test?

Start with a pipeline small enough to run on a laptop, and check the data before and after every step. DuckDB's website describes DuckDB as an analytical database system that can be deployed anywhere, with a PostgreSQL-inspired SQL dialect and the ability to read and write CSV, JSON, Parquet and Iceberg locally or in object storage; its projects are MIT-licensed and governed by the independent DuckDB Foundation. The script below loads a hypothetical five-row orders file into a raw table, builds a clean table and counts nulls and duplicate ids in both:

import duckdb, pathlib, tempfile

csv = """order_id,customer,city,amount,order_date
1001,Asha, pune ,1200,2026-11-01
1002,Bilal,Pune,850,2026-11-01
1003,Chitra,Chennai,2400,2026-11-02
1003,Chitra,Chennai,2400,2026-11-02
1004,Dev,MUMBAI,,2026-11-02
"""
path = pathlib.Path(tempfile.mkdtemp(), "orders.csv"); path.write_text(csv)
con = duckdb.connect()  # in-memory database
con.sql(f"CREATE TABLE raw_orders AS SELECT * FROM read_csv('{path}', all_varchar=true)")
con.sql("""CREATE TABLE fact_orders AS SELECT DISTINCT CAST(order_id AS INT) AS order_id, customer,
  upper(left(trim(city), 1)) || lower(substr(trim(city), 2)) AS city,
  CAST(amount AS INT) AS amount, CAST(order_date AS DATE) AS order_date
  FROM raw_orders WHERE amount IS NOT NULL""")
one = lambda sql: con.execute(sql).fetchall()[0][0]
def check(t):
    return {"rows": one(f"SELECT count(*) FROM {t}"),
            "null amounts": one(f"SELECT count(*) FROM {t} WHERE amount IS NULL"),
            "duplicate ids": one(f"SELECT count(*) - count(DISTINCT order_id) FROM {t}")}
print("raw :", check("raw_orders"))
print("fact:", check("fact_orders"))
print(con.execute("SELECT order_date, city, sum(amount) FROM fact_orders GROUP BY ALL ORDER BY 1, 2").fetchall())

With Python 3.12.13 and DuckDB 2.0.0 the output was:

raw : {'rows': 5, 'null amounts': 1, 'duplicate ids': 1}
fact: {'rows': 3, 'null amounts': 0, 'duplicate ids': 0}
[(datetime.date(2026, 11, 1), 'Pune', 2050), (datetime.date(2026, 11, 2), 'Chennai', 2400)]

The raw table shows the two problems, one missing amount and one repeated order, and the clean table shows both removed. The revenue query then returns 2050 for Pune on 1 November (1200 plus 850) and 2400 for Chennai on 2 November.

The sample has limits. It silently drops the order with a missing amount, whereas a real pipeline writes rejected rows to a table and raises an alert. It also pins a version for a reason: under DuckDB 2.0.0 the first attempt, which called fetchone() on a relation, raised an error, so the script uses the connection's execute and fetchall instead.

Which tools should you learn for batch, transformation and streaming?

Learn one tool for each of orchestration, transformation and streaming, and learn the concept before the product. Apache Airflow's documentation describes Airflow as an open-source platform for developing, scheduling and monitoring workflows such as batch-oriented data pipelines, with workflows defined entirely in Python. The documentation now writes a workflow as a "Dag" and defines it as a model that encapsulates everything needed to execute a workflow, including the schedule, the tasks and their dependencies. PyPI lists apache-airflow 3.4.0, and the repository's LICENSE file is the Apache License 2.0.

The dbt documentation says dbt transforms raw warehouse data into trusted data products: you write SQL select statements and dbt creates modular data models. Tutorials can mislead because dbt has two major versions. v2, the current Rust-based generation, is the default when you install dbt, and v1, the original Python-based generation, is still maintained. dbt's data tests cover non-null values, unique values, relationships to another model and accepted values, and each test returns the set of failing records, which makes it a ready-made quality check.

The Apache Kafka introduction says event streaming is the practice of capturing data in real time from sources such as databases, sensors, mobile devices, cloud services and applications. Kafka's downloads page lists 4.4.0, released on 22 October 2026.

Apache Spark describes itself as a multi-language engine for data engineering, data science and machine learning on single machines or clusters, and Apache Parquet is an open source, column-oriented file format for efficient storage and retrieval. Apache Iceberg is a format for huge analytic tables that lets engines such as Spark, Trino and Flink work safely with the same tables at the same time.

Is SQL enough to become a data engineer?

SQL is not enough on its own to become a data engineer: it is the first stage and not the whole job. The vendor skill lists add programming, orchestration, data modelling, lifecycle management and quality.

In the Stack Overflow Developer Survey 2026, among 12,061 respondents, PostgreSQL was used for extensive development work by 57.9%, SQLite by 42.1% and MySQL by 35.1%, so PostgreSQL is a sensible free place to start. The PostgreSQL project describes PostgreSQL as an open source object-relational database system that uses and extends the SQL language.

Data modelling belongs in stage 1 alongside SQL. The Kimball Group's dimensional modelling page lists a four-step design process (business processes, grain, dimensions for descriptive context and facts for measurements) and covers star schemas and the types of fact table. The SQL joins guide covers the joins that star schemas depend on. Databricks' documentation describes a medallion architecture of bronze (raw), silver (validated) and gold (enriched) layers, which is a naming convention for data quality stages and not a standard.

Is there demand for data engineering skills in India?

Demand shows up in several Indian reports. foundit's Insights Tracker June 2026 hiring snapshot for India, published in July 2026, lists Data Engineers at +32% year on year as the top trending role in the IT sector of global capability centres.

Its December 2025 snapshot lists Data Engineering at a 16% share of top skills for 4 to 6 years of experience and Data Architecture at 14% for 7 to 10 years. NASSCOM's 2026 study Agentic AI Adoption in Indian Enterprises, a survey of 200 India-headquartered legal entities, reports that 71% indicate a need for AI/ML and agent engineering capabilities, followed by 67% seeking data engineering skills.

Both sources describe employers' needs and not vacancy counts, and the foundit data is strongest above the fresher band. A pipeline project with tests and a README is the evidence a beginner can show. The cloud engineer roadmap and the DevOps engineer roadmap cover the storage, scheduling and deployment skills that data engineers share with them.

What are the limits of this roadmap?

A roadmap does not replace experience. Google Cloud's Professional Data Engineer lists no prerequisites but recommends 3+ years of industry experience, including 1+ years designing and managing solutions on Google Cloud, so it is not a first certification. Versions change quickly, as the dbt v1 and v2 split and the DuckDB 2.0.0 change show. The sample pipeline is local and tiny, and real pipelines also need access control, cost control and monitoring. Salary data is left out.

Where does this roadmap fit in a data analytics programme?

Certification Program in Data Analytics with AI and Gen AI is an online programme from E&ICT Academy, IIT Roorkee, run with Masai as the implementation partner, with live sessions and an optional campus immersion, covering SQL, Python, Google Sheets, prompt engineering, LLM tools and agentic workflows with LangChain and CrewAI, and a capstone project.

Its first module is Core Data Architecture and Analytical Foundations, and its topics include relational databases and SQL queries and Python foundations, which match stages 1 and 2. The topic list does not name orchestration or streaming tools, so stages 4 to 7 need separate practice.

Frequently asked questions

What does a data engineer do?

A data engineer builds and maintains the systems that collect, transform, store and deliver data. Google Cloud describes the role as collecting, transforming, storing and delivering data for diverse applications, and AWS lists ingesting and transforming data, orchestrating pipelines, designing data models, managing data life cycles and ensuring data quality as the core skills.

Is SQL enough to become a data engineer?

SQL alone is not enough to become a data engineer. SQL is the first stage, but AWS's Certified Data Engineer - Associate also covers programming concepts, pipeline orchestration, data modelling, data life cycles and data quality, and Microsoft's Fabric Data Engineer Associate expects PySpark and Kusto Query Language alongside SQL. Add Python, a warehouse, an orchestrator and quality checks.

What is the difference between a data engineer and a data analyst?

A data engineer collects, transforms, stores and delivers data, while a data analyst works with the available data to deliver insights and business value. Microsoft's Power BI Data Analyst Associate page says analysts collaborate with data engineers to identify and acquire data, so engineers supply the tables that analysts query.

Do data engineers need Python?

Data engineers need some programming, usually Python, after SQL. Apache Airflow workflows are defined entirely in Python, and AWS's data engineer certification refers to applying programming concepts. Microsoft's Fabric Data Engineer Associate mentions SQL, PySpark and Kusto Query Language. A first project in Python and SQL covers both.

Which tools should a beginner data engineer learn first?

A beginner should learn PostgreSQL or DuckDB for SQL, then Python, Apache Airflow for orchestration and dbt for transformation, and finally Apache Kafka for streaming basics. Learn the concepts of orchestration, transformation and streaming first, because each tool can be replaced by another.

Is dbt version 1 or version 2 the one to learn?

dbt's documentation says v2, the current Rust-based generation, is the default when you install dbt, and v1, the original Python-based generation, is still maintained. The language is the same SQL select statements, tests and configuration files, so concepts carry over, but check which version a tutorial uses before following it.

Next step

Build the stage 2 loader this week and run the checks on the raw and clean tables. The Certification Program in Data Analytics with AI and Gen AI page lists the SQL, Python and capstone modules for readers who want a guided route through the early stages, and the data analyst roadmap shows the path of the people who will query the data.