Open in app

Sign In

Write

Sign In

Brian Lipp
Brian Lipp

28 Followers

Home

About

May 31, 2022

Cloud Data Platform Showdown: Databricks

Background Databricks was founded by the original developers of the Apache Spark Project. Databricks is dedicated to the Open Source community in many ways, they contribute to Apache Spark, and have created new Open Source Projects and several new projects such as MLflow and Delta Lake. On the other hand…

4 min read

Cloud Data Platform Showdown: Databricks
Cloud Data Platform Showdown: Databricks

4 min read


Jul 26, 2021

MLOps Basics Part 4

So far in our MLOps journey, we have created ML research and ML model-building pipelines as well as saved them in serialized form. Saving models this way allows us to now take that serialized ML model and load it into an application. We will now take the saved ML model…

6 min read

6 min read


Jun 9, 2021

MLOps Basics Part 3

In previous articles, we gained the basics of MLOps and set up our orchestrator. Now we will put together our Python applications and interactions with Databricks Spark. Why Databricks Spark? We will be using Databricks Spark as a general platform to run our code. In some cases, we will Spark to run code…

6 min read

MLOps Basics Part 3
MLOps Basics Part 3

6 min read


Jun 9, 2021

MLOps Basics Part 2

In our first article, we introduced the basics of MLOps, now we will talk about our core application in our tech stack, Airflow. Airflow will be the central orchestrator for all batch-related tasks. Swapping technologies This tech stack is designed for flexibility and scalability. There should be no issues using alternative tooling…

6 min read

MLOps Basics Part 2
MLOps Basics Part 2

6 min read


Jun 9, 2021

MLOps Basics Part 1

MLOps? MLOps (Machine Learning Operations) is the practice of combining the lessons learned from DevOps for the productionisation of machine learning. Its role is to fill the gap between the data scientist and the machine learning consumers. Machine Learning? Data Science? Machine Learning can be understood as the process of applying a set of techniques…

3 min read

MLOps Basics Part 1
MLOps Basics Part 1

3 min read


May 27, 2021

Decomposing The Lakehouse

“Do not collect weapons or practice with weapons beyond what is useful.” Miyamoto Musashi, Dokkodo Students of the Ichi school Way of Strategy should train from the start with the (normal) sword and the long sword in either hand. This is a truth: when you sacrifice your life, you must…

4 min read

Decomposing The Lakehouse
Decomposing The Lakehouse

4 min read


Mar 22, 2021

Setting up your Python Environment

When working on multiple Python projects it's common to run into issues with Python versioning, and package management. I am going to introduce two projects to help you tackle these common issues. I’m not going to take about the Conda project, simply because in my experience 90% of the time…

4 min read

4 min read


Feb 10, 2021

Choosing a data store, a guide for the curious

When starting a new project, it's a good idea to evaluate your data storage needs. I’m going to shy away from the term database and instead, I’ll use the term data store because oftentimes labels are loaded with baggage that will distract us. …

7 min read

Choosing a data store, a guide for the curious
Choosing a data store, a guide for the curious

7 min read


Nov 24, 2020

Modern Data Engineering

History Data Engineering is a relatively new concept, although the skills have been around for some time. If you Google around you will find that the skills, tools, and job responsibilities will vary significantly. My approach is a broad, modern approach to the data engineering role. Many hyperspecialized roles also exist…

5 min read

Modern Data Engineering
Modern Data Engineering

5 min read


Apr 29, 2020

Engineering for All: Faking

If you believe it, they believe it. 267th Ferengi Rule of Acquisition All war is deception Sun Tzu The dangers of testing Let's face it testing software can be hard. Even with the best intentions, our tests can easily break. This phenomenon is called brittle unit tests. Brittle Unit tests Unit testing has a very bad reputation…

5 min read

Engineering for All: Faking
Engineering for All: Faking

5 min read

Brian Lipp

Brian Lipp

28 Followers
Following
  • Netflix Technology Blog

    Netflix Technology Blog

  • Barr Moses

    Barr Moses

  • Rafi Kurlansik

    Rafi Kurlansik

See all (6)

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech

Teams