What is MLOps?

1. What is MLOps — Definition & Core Concepts

MLOps (Machine Learning Operations) is a set of practices, tools, and cultural philosophies that automate and streamline the entire machine learning lifecycle — from data preparation and model training to deployment, monitoring, and retraining.

The Three Pillars of MLOps

MLOps = ML + DevOps + Data Engineering

Notice that Data Engineering is literally one-third of MLOps. You are not starting from zero — you are starting from a position of strength.

Key Terminology You Need to Know

2. Why MLOps Exists — The Real Problems

Only 22% of companies that start ML projects successfully deploy them to production. The rest fail not because the model was bad, but because of engineering and operational challenges. MLOps exists to solve this "last mile" problem.

The Top 5 Problems MLOps Solves

The Business Impact — Real Numbers

3. The ML Lifecycle — End to End

The ML lifecycle is a continuous loop, not a straight line. Each stage feeds back into the others. Here is every stage explained in detail:

Stage 1: Problem Definition

Before writing any code, clearly define what business problem you are solving and how you will measure success.

# Example: E-commerce Churn Prediction

# Business Problem:
#   "We're losing 15% of customers monthly. Can we predict
#   who will churn so we can offer retention discounts?"

# ML Problem Translation:
#   Binary classification: Will customer churn in next 30 days? (Yes/No)

# Success Metrics:
#   - Precision > 80% (don't waste discounts on non-churners)
#   - Recall > 70% (catch most actual churners)
#   - Business: Reduce churn rate from 15% to 10%

# Data Available:
#   - Order history (3 years, 2M records)
#   - Customer profiles (500K customers)
#   - Support tickets (1.5M records)
#   - Website clickstream (stored in Snowflake)

Stage 2: Data Collection

This is where you shine. Collecting data from multiple sources, handling different formats, managing data lineage — this is pure data engineering.

# As a data engineer, you'd build this pipeline:
import pandas as pd
from sqlalchemy import create_engine

# Connect to your data warehouse
engine = create_engine("snowflake://user:pass@account/db/schema")

# Pull data from multiple sources
orders = pd.read_sql("""
    SELECT customer_id, 
           COUNT(*) as total_orders,
           AVG(order_value) as avg_order_value,
           DATEDIFF(day, MAX(order_date), CURRENT_DATE) as days_since_last_order,
           SUM(order_value) as lifetime_value
    FROM orders 
    WHERE order_date >= DATEADD(year, -2, CURRENT_DATE)
    GROUP BY customer_id
""", engine)

support = pd.read_sql("""
    SELECT customer_id,
           COUNT(*) as total_tickets,
           AVG(resolution_hours) as avg_resolution_time,
           SUM(CASE WHEN sentiment = 'negative' THEN 1 ELSE 0 END) as negative_tickets
    FROM support_tickets
    GROUP BY customer_id
""", engine)

# Merge all data sources
customer_data = orders.merge(support, on="customer_id", how="left")
customer_data = customer_data.fillna(0)  # Handle missing values

print(f"Dataset: {customer_data.shape[0]} customers, {customer_data.shape[1]} features")
# Output: Dataset: 487,523 customers, 9 features

Stage 3: Data Preparation

Clean, validate, and transform the raw data. This is 60-80% of real-world ML work — and data engineers own this skill.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Step 1: Remove duplicates and invalid records
customer_data = customer_data.drop_duplicates(subset="customer_id")
customer_data = customer_data[customer_data["total_orders"] > 0]

# Step 2: Handle outliers (cap at 99th percentile)
for col in ["avg_order_value", "lifetime_value"]:
    cap = customer_data[col].quantile(0.99)
    customer_data[col] = customer_data[col].clip(upper=cap)

# Step 3: Create target variable
# A customer "churned" if they haven't ordered in 30+ days
customer_data["churned"] = (customer_data["days_since_last_order"] > 30).astype(int)

# Step 4: Split into features (X) and target (y)
feature_cols = ["total_orders", "avg_order_value", "days_since_last_order",
                "lifetime_value", "total_tickets", "negative_tickets"]
X = customer_data[feature_cols]
y = customer_data["churned"]

# Step 5: Split into train/test sets (80/20)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# Step 6: Scale features (important for many algorithms)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
print(f"Churn rate: {y.mean():.1%}")

Stages 4-8: Training → Deployment → Monitoring

These stages are covered in depth in Chapters 3-8. For now, here is the quick summary:

4. MLOps Maturity Levels (0 to 3)

Google defined 3 levels of MLOps maturity. Most companies are at Level 0 or 1. Understanding where your target company sits helps you prepare for interviews.

Level 0: Manual Everything

• Data scientists work in Jupyter notebooks
• Models are trained manually, passed to engineers as pickle files
• No versioning of data or models
• Deployment takes weeks/months with custom scripts
• No monitoring — "if it breaks, someone will notice"

Your role: Introduce basic pipeline automation and version control.

Level 1: ML Pipeline Automation

• Automated training pipelines (run on schedule or trigger)
• Experiment tracking with MLflow or W&B
• Model registry with staging/production stages
• Feature store for consistent feature serving
• Basic monitoring (accuracy, latency)
• Manual deployment with some automation

Your role: Build and optimize the ML pipelines, feature stores, and data validation.

Level 2: CI/CD for ML

• Fully automated CI/CD pipeline: code change → test → train → deploy
• Automated data validation and model validation gates
• A/B testing and canary deployments for models
• Automated retraining triggered by drift detection
• Comprehensive monitoring dashboards
• Infrastructure as Code for all ML resources

Your role: Own the entire infrastructure — pipelines, deployment, monitoring, scaling.

Level 3: Full Automation with Governance

• Everything from Level 2 plus:
• Automated feature discovery and selection
• Model fairness and bias auditing built into the pipeline
• Compliance reporting (model cards, audit trails)
• Cost optimization for training and inference
• Multi-model orchestration (model ensembles, A/B/C testing)

Your role: Lead the platform team. Design the architecture for the entire ML platform.

5. Key Roles in MLOps Teams

Understanding the team structure helps you identify where you fit and how to position yourself.

6. DevOps vs MLOps — Side by Side

If you understand DevOps, you are 60% of the way to MLOps. Here is what is the same and what is different:

The Extra Dimensions in MLOps

The fundamental difference is that ML systems have three axes of change instead of one:

7. Core Principles of MLOps

Principle 1: Treat ML Pipelines as First-Class Citizens

ML pipelines should be version-controlled, tested, reviewed, and deployed just like application code. No more "scripts on someone's laptop."

# Example: ML Pipeline as Code (ZenML)
from zenml import pipeline, step

@step
def load_data() -> pd.DataFrame:
    """Load customer data from warehouse"""
    return pd.read_parquet("s3://data-lake/customers.parquet")

@step
def validate_data(df: pd.DataFrame) -> pd.DataFrame:
    """Run data quality checks"""
    assert df.shape[0] > 1000, "Too few records!"
    assert df["customer_id"].nunique() == df.shape[0], "Duplicates found!"
    return df

@step
def train_model(df: pd.DataFrame) -> "Model":
    """Train and return the best model"""
    # ... training logic ...
    return model

@pipeline
def churn_pipeline():
    data = load_data()
    validated = validate_data(data)
    model = train_model(validated)

# Run the pipeline
churn_pipeline()

Principle 2: Automate Everything That Can Be Automated

Manual steps = bottlenecks and errors. Automate data validation, training, testing, deployment, and monitoring.

Principle 3: Monitor Models Like Production Services

A model in production is a service. Monitor it with the same rigor as a web application — plus ML-specific metrics.

Principle 4: Reproduce Any Result at Any Time

If someone asks "why did the model predict X for customer Y on March 5th?", you should be able to reproduce the exact input features, model version, and prediction.

Principle 5: Version Everything

Code (Git) + Data (DVC) + Models (MLflow) + Pipelines (ZenML) + Infrastructure (Terraform) = Full reproducibility.

8. Real-World MLOps Examples

9. The MLOps Tech Stack — Complete Overview

Here is the full landscape organized by layer. Do not try to learn everything — focus on one tool per category.

10. Hands-On: Your First MLOps Setup

Let us set up a minimal MLOps environment on your laptop in 15 minutes. This uses MLflow for tracking and a simple scikit-learn model.

Step 1: Install Dependencies

# Create a new project
mkdir my-first-mlops && cd my-first-mlops
python -m venv venv && source venv/bin/activate

# Install core MLOps tools
pip install mlflow scikit-learn pandas numpy great-expectations evidently

Step 2: Create Sample Data

# generate_data.py
import pandas as pd
import numpy as np

np.random.seed(42)
n = 10000

data = pd.DataFrame({
    "customer_id": [f"C{i:05d}" for i in range(n)],
    "total_orders": np.random.poisson(12, n),
    "avg_order_value": np.random.normal(65, 25, n).clip(5),
    "days_since_last_order": np.random.exponential(20, n).astype(int),
    "support_tickets": np.random.poisson(2, n),
    "satisfaction_score": np.random.normal(3.8, 0.8, n).clip(1, 5),
})

# Create churn label (based on realistic factors)
churn_prob = (
    0.3 * (data["days_since_last_order"] > 30).astype(float) +
    0.2 * (data["support_tickets"] > 3).astype(float) +
    0.3 * (data["satisfaction_score"] < 3).astype(float) +
    0.2 * (data["total_orders"] < 5).astype(float)
)
data["churned"] = (np.random.random(n) < churn_prob).astype(int)

data.to_csv("customer_data.csv", index=False)
print(f"Generated {n} records. Churn rate: {data['churned'].mean():.1%}")

Step 3: Train with MLflow Tracking

# train.py
import mlflow
import mlflow.sklearn
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

# Load data
data = pd.read_csv("customer_data.csv")
features = ["total_orders", "avg_order_value", "days_since_last_order",
            "support_tickets", "satisfaction_score"]
X = data[features]
y = data["churned"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Set up MLflow experiment
mlflow.set_experiment("customer_churn")

# Run experiment
with mlflow.start_run(run_name="gradient_boost_v1"):
    # Hyperparameters
    params = {"n_estimators": 200, "learning_rate": 0.05, "max_depth": 4}
    mlflow.log_params(params)
    
    # Train
    model = GradientBoostingClassifier(**params, random_state=42)
    model.fit(X_train, y_train)
    
    # Evaluate
    y_pred = model.predict(X_test)
    metrics = {
        "accuracy": accuracy_score(y_test, y_pred),
        "f1": f1_score(y_test, y_pred),
        "precision": precision_score(y_test, y_pred),
        "recall": recall_score(y_test, y_pred),
    }
    mlflow.log_metrics(metrics)
    
    # Save model
    mlflow.sklearn.log_model(model, "model")
    
    print(f"Accuracy: {metrics['accuracy']:.3f}")
    print(f"F1 Score: {metrics['f1']:.3f}")
    print(f"View results: mlflow ui (then open http://localhost:5000)")

Step 4: Launch MLflow UI

# In your terminal:
mlflow ui

# Open http://localhost:5000 in your browser
# You'll see your experiment, parameters, metrics, and saved model!

11. Your Data Engineering Advantage

Let us be explicit about why your background is so valuable in MLOps: