P(y=1) = 1 / (1 + e^{-(b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ)})

Example: Customer Churn Prediction

Imagine a telecom company wanting to predict which customers are likely to cancel their subscription (churn). They collect data on:

• Monthly charges
• Contract length
• Customer service calls
• Usage patterns

Using logistic regression, they can estimate the probability of each customer churning and take proactive measures to retain high-risk customers.

# Python example using scikit-learn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np

# Assume X contains features and y contains labels (1 for churn, 0 for not churn)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train_scaled, y_train)

# Make predictions
y_pred = model.predict(X_test_scaled)
y_pred_prob = model.predict_proba(X_test_scaled)[:, 1]  # Probability of churn

# High-risk customers (probability > 0.7)
high_risk_customers = np.where(y_pred_prob > 0.7)[0]
        

Example: Handwritten Digit Recognition

Consider the task of recognizing handwritten digits (0-9). Each digit image can be represented as a set of pixel values.

When a new digit image needs to be classified, KNN:

1. Computes the distance between the new image and all training images
2. Identifies the K closest training images
3. Assigns the most common digit class among these K neighbors

# Python example using scikit-learn
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

# Load the digits dataset
digits = load_digits()
X = digits.data
y = digits.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train KNN classifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)

# Predict
y_pred = knn.predict(X_test)

# Example: predict a single digit
# import matplotlib.pyplot as plt
# plt.imshow(digits.images[0], cmap='gray')
# print(f"Prediction: {knn.predict([digits.data[0]])[0]}")
        

Example: Loan Approval Prediction

A bank wants to automate its loan approval process. They have data about applicants including:

• Income
• Employment duration
• Existing debt
• Credit history

A decision tree can create rules like:

• If credit_score > 700 and income > 50000: Approve
• If credit_score < 600: Deny
• If credit_score between 600-700 and debt_to_income < 0.3: Approve
• Otherwise: Deny

# Python example using scikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
import pandas as pd

# Assume df is a DataFrame with loan application data
X = df[['income', 'credit_score', 'employment_years', 'debt_to_income']]
y = df['loan_approved']  # 1 for approved, 0 for denied

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train decision tree
tree = DecisionTreeClassifier(max_depth=5)  # Limit depth to avoid overfitting
tree.fit(X_train, y_train)

# Make predictions
y_pred = tree.predict(X_test)

# Visualize the tree (requires graphviz)
# from sklearn.tree import export_graphviz
# export_graphviz(tree, out_file='loan_tree.dot', 
#                feature_names=X.columns,
#                class_names=['Denied', 'Approved'],
#                filled=True)
        

Example: Disease Diagnosis

Medical researchers want to classify patients as having or not having a particular disease based on various biomarkers.

The data is complex and not linearly separable in the original feature space. SVM with a radial basis function (RBF) kernel can find an effective decision boundary in the transformed space.

# Python example using scikit-learn
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Assume X contains biomarker features and y contains disease status (1 for present, 0 for absent)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features (important for SVM)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create and train SVM classifier
svm = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True)
svm.fit(X_train_scaled, y_train)

# Make predictions
y_pred = svm.predict(X_test_scaled)
y_pred_prob = svm.predict_proba(X_test_scaled)[:, 1]  # Probability of disease presence
        

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

F1-Score = 2 × (Precision × Recall) / (Precision + Recall)

Example: Evaluating a Medical Diagnostic Test

Consider a model that predicts whether patients have a disease:

• TP = 85 (correctly identified patients with the disease)
• TN = 900 (correctly identified healthy patients)
• FP = 30 (healthy patients incorrectly classified as having the disease)
• FN = 15 (patients with the disease incorrectly classified as healthy)

Evaluating the model:

• Accuracy = (85 + 900) / (85 + 900 + 30 + 15) = 0.956 (95.6%)
• Precision = 85 / (85 + 30) = 0.739 (73.9%)
• Recall = 85 / (85 + 15) = 0.85 (85%)
• F1-Score = 2 × (0.739 × 0.85) / (0.739 + 0.85) = 0.79 (79%)

# Python example for calculating evaluation metrics
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix
import numpy as np

# Assume y_true contains actual labels and y_pred contains predicted labels
conf_matrix = confusion_matrix(y_true, y_pred)
print("Confusion Matrix:")
print(conf_matrix)

# Calculate metrics
accuracy = accuracy_score(y_true, y_pred)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1-Score: {f1:.4f}")
        

# Python example for ROC curve and AUC
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt

# Assume y_true contains actual labels and y_pred_prob contains predicted probabilities
fpr, tpr, thresholds = roc_curve(y_true, y_pred_prob)
auc = roc_auc_score(y_true, y_pred_prob)

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f'AUC = {auc:.3f}')
plt.plot([0, 1], [0, 1], 'k--')  # Random guessing line
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()
    

Example: Complete ML Pipeline for Customer Churn Prediction

# Python example of a complete ML pipeline
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
import matplotlib.pyplot as plt
import seaborn as sns

# 1. Load data
df = pd.read_csv('telecom_customer_data.csv')

# 2. Exploratory Data Analysis (simplified)
print(df.head())
print(df.info())
print(df.describe())
print(df.isnull().sum())

# Check class distribution
plt.figure(figsize=(6, 4))
sns.countplot(x='Churn', data=df)
plt.title('Class Distribution')
plt.show()

# 3. Feature Engineering
# Add new feature: Average monthly bill over customer lifetime
df['Avg_Monthly_Bill'] = df['TotalCharges'] / df['tenure']
df['Avg_Monthly_Bill'].fillna(df['MonthlyCharges'], inplace=True)  # For new customers

# 4. Data Preprocessing Setup
# Identify different column types
numeric_features = ['tenure', 'MonthlyCharges', 'TotalCharges', 'Avg_Monthly_Bill']
categorical_features = ['gender', 'Partner', 'Dependents', 'PhoneService', 'InternetService', 
                       'OnlineSecurity', 'OnlineBackup', 'DeviceProtection', 'TechSupport', 
                       'StreamingTV', 'StreamingMovies', 'Contract', 'PaperlessBilling', 'PaymentMethod']

# Create preprocessing pipelines
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

# Combine preprocessing steps
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])

# 5. Split data
X = df.drop('Churn', axis=1)
y = df['Churn'].map({'Yes': 1, 'No': 0})  # Convert to binary

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# 6. Create and train model pipeline
model_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(random_state=42))
])

# 7. Hyperparameter tuning
param_grid = {
    'classifier__n_estimators': [100, 200],
    'classifier__max_depth': [None, 10, 20],
    'classifier__min_samples_split': [2, 5],
    'classifier__min_samples_leaf': [1, 2]
}

grid_search = GridSearchCV(
    model_pipeline,
    param_grid,
    cv=5,
    scoring='f1',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)

# Get best model
best_model = grid_search.best_estimator_
print(f"Best parameters: {grid_search.best_params_}")

# 8. Evaluate model
y_pred = best_model.predict(X_test)
y_pred_prob = best_model.predict_proba(X_test)[:, 1]

# Print evaluation metrics
print("Classification Report:")
print(classification_report(y_test, y_pred))

print("Confusion Matrix:")
conf_mat = confusion_matrix(y_test, y_pred)
sns.heatmap(conf_mat, annot=True, fmt='d', cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion Matrix')
plt.show()

print(f"ROC AUC Score: {roc_auc_score(y_test, y_pred_prob):.4f}")

# 9. Feature Importance Analysis
if hasattr(best_model['classifier'], 'feature_importances_'):
    # Get feature names after preprocessing
    ohe_feature_names = best_model['preprocessor'].transformers_[1][1]['onehot'].get_feature_names_out(categorical_features)
    feature_names = numeric_features + list(ohe_feature_names)
    
    # Get feature importances
    importances = best_model['classifier'].feature_importances_
    
    # Sort feature importances
    indices = np.argsort(importances)[::-1]
    
    # Plot feature importances
    plt.figure(figsize=(12, 8))
    plt.title('Feature Importances')
    plt.bar(range(len(indices[:15])), importances[indices[:15]], align='center')
    plt.xticks(range(len(indices[:15])), [feature_names[i] for i in indices[:15]], rotation=90)
    plt.tight_layout()
    plt.show()

# 10. Model Deployment (conceptual)
# In a real scenario, you would save the model and deploy it
import joblib
joblib.dump(best_model, 'churn_prediction_model.pkl')

# Then in production:
# loaded_model = joblib.load('churn_prediction_model.pkl')
# new_customer_data = pd.DataFrame([{...}])  # New customer data
# prediction = loaded_model.predict(new_customer_data)
# probability = loaded_model.predict_proba(new_customer_data)[:, 1]
        

Common Challenges and Best Practices

Overfitting: When a model performs well on training data but poorly on unseen data. Solutions include using simpler models, more training data, regularization, and early stopping.

Underfitting: When a model is too simple to capture the underlying pattern. Solutions include using more complex models, adding more features, or reducing regularization.

Imbalanced data: When classes are not equally represented. Solutions include resampling techniques (oversampling or undersampling), using appropriate evaluation metrics, or cost-sensitive learning.

Feature selection: Including irrelevant features can degrade model performance. Techniques like recursive feature elimination, feature importance, and L1 regularization can help select the most relevant features.