How to Learn Databricks: Free Edition Walkthrough

TL;DR

Databricks Free Edition requires only an email or a Google or Microsoft login, with no credit card, and it gives you a serverless starter warehouse for compute. The platform is built on Apache Spark, so it handles large data volumes, and it combines ingesting, analyzing, and visualizing data in one place for data engineers, analysts, and scientists working collaboratively.

Transcript

What's going on everybody? Welcome back to another video. Today we're going to be learning data bricks in under two hours. Now, for the past 5 weeks, we've been diving into data bricks. And in this video, we're just putting all that together so you can follow along really easily. We're going to start by taking a quick walkthrough of data bricks and... Read More

Key Insights

• Databricks Free Edition costs nothing and takes no credit card information. You sign up with a Google account, a Microsoft account, or just an email, state what country you are from, and click continue, which the presenter describes as about as seamless as it can possibly be.
• Databricks is built on top of Apache Spark, which is why it handles large amounts of data well. It is designed so that entire teams of data engineers, data analysts, and data scientists can ingest, analyze, and visualize data collaboratively in one platform.
• The workspace is the place where you work and collaborate. You can give teammates access to your workspace so that any data, code, or visualizations you have created there become available to them without sending files back and forth.
• Catalog in Databricks works like a schema in a database. Underneath it sit all your tables, views, and stored procedures, and the samples section contains different databases you can click into to view the tables and files available within Databricks.
• Jobs and pipelines split into three pieces: an ingestion pipeline handles the extraction step of ETL, an ETL pipeline handles transforming and loading data into a table inside Databricks, and a job orchestrates when those pipelines run, whether daily, weekly, or monthly.
• The Free Edition offers no compute options beyond a serverless starter warehouse sized at 2X-Small. It stays inactive until you open something and start working with real data, at which point the serverless compute activates and is usable for free.
• A second compute type called generic compute exists but is not visible in the interface. It applies only to the specific case of running Python in a notebook, while the serverless compute covers most other work in the platform.
• Genie is the AI system inside Databricks that lets you get insights from your data using natural language. The SQL editor also has AI integrated into it, which helps with writing the base queries you would otherwise type out by hand.

Questions & Answers

Q: How do I sign up for Databricks Free Edition?

Go to the Databricks Free Edition signup page, linked in the video description, and choose to sign up for the free edition. You can log in with a Google account, a Microsoft account, or just an email address. No credit card information is taken. After signing in, you enter your name and the country you are from, then click continue, and your account is ready. The presenter signed in with a Google account and described the process as about as seamless as it can possibly be.

Q: What is Databricks built on and why does that matter?

Databricks is built on top of Apache Spark, which means it is very good at handling large amounts of data. That foundation supports the platform's purpose: it is designed for entire teams of data engineers, data analysts, and data scientists to work with data collaboratively. Ingesting data, analyzing data, and visualizing data all happen in one place rather than being spread across separate tools, which is what separates it from working in something like SQL, R, or Tableau alone.

Q: What is the catalog section in Databricks used for?

Catalog works like a schema in a database. In a database schema you have all your tables, views, stored procedures, and related objects underneath it, and catalog is very similar. Going into a workspace or into the samples section shows different databases, and you can click into those databases to look at the tables inside them. Catalog is where you get access to view all of your data, files, and tables within Databricks in one organized place.

Q: What is the difference between an ingestion pipeline, an ETL pipeline, and a job?

Each does a slightly different thing. An ingestion pipeline covers the extraction part of the ETL process, meaning you are extracting data to bring it in. An ETL pipeline covers the entire process of transforming the data, loading it, and actually putting it into a table within Databricks. A job orchestrates the schedule: it is where you specify when the pipelines actually run, so you can time them to run daily, weekly, monthly, or whenever you want. Created pipelines and jobs appear listed at the bottom of that section.

Q: What compute do you get with the Databricks free edition?

Compute is very simple in the free edition because there are no options to choose from. You get a serverless starter warehouse, shown with its owner, a size of 2X-Small, and an active or inactive status. It stays inactive when you are not looking at tables or running queries, and the moment you open something and start working with real data, the serverless compute activates and you can use it for free. On the full Databricks product you can customize compute to your needs instead.

Q: Why is there a hidden generic compute type in Databricks?

There is one other type of compute besides the serverless warehouse that you cannot see in the interface. It is called generic compute, and it exists for one specific use case: running Python in a notebook. For most other work in the platform you will be using the serverless compute, but the presenter mentions generic compute because it is worth knowing it exists even though it does not appear among the visible compute options in the free edition.

Q: What can you do in the Databricks marketplace?

The marketplace lists companies that Databricks partners with and has connectors to, which makes working with those tools easier. Through partner connect integration you can connect to tools such as Fiverr, Power BI, Tableau, DBT, and Prophecy, and you can search to check whether a connector exists for a tool your team uses. The marketplace also has a products area with tables and files, meaning data, plus models, notebooks, and other resources you can search for and use free. If you cannot find the data you want there, you can get free data from a site like Kaggle and bring it into Databricks.

Q: How does Databricks help you organize and share SQL queries?

Queries you develop in the SQL editor can be saved directly within Databricks under the queries section. The presenter contrasts this with an earlier job where 20 or 30 queries were saved loose in a file explorer, requiring emailing a query to a colleague so they could pull it up and copy and paste it. In Databricks everything stays in one place, so organization and sharing are handled by the platform. The SQL editor itself supports multiple tabs and lets you select which workspace your queries point at.

Summary & Key Takeaways

• The video compiles five weeks of Databricks lessons into a single follow-along session. It covers a walkthrough of the platform, importing data from both flat files and connected data sources, using the SQL editor and notebooks, visualizing data, working with AI tools such as Genie and the AI assistant, and building a full project at the end.

• Databricks is built on top of Apache Spark, which makes it strong at handling large amounts of data. It is designed for whole teams of data engineers, data analysts, and data scientists to work with data collaboratively, covering ingestion, analysis, and visualization all in one place rather than across separate tools.

• The left-hand navigation covers the main areas: workspace for collaboration, catalog for databases and tables, jobs and pipelines for ingestion, ETL and orchestration, compute for the SQL warehouse, marketplace for partner connectors and free data, the SQL editor, saved queries, dashboards, and Genie for natural language insights.