Summary

This week, we will discuss classifier calibration and quantile regression. We will learn how to improve probability estimates from classifiers and how to create prediction intervals using quantile regression techniques.

Learning Objectives

After completing this week, you are expected to be able to:

  • Use CalibratedClassifierCV to calibrate classifiers.
  • Evaluate classifier calibration via appropriate metrics and visualizations.
  • Use quantile regression (via QuantileRegressor and HistGradientBoostingRegressor) to estimate conditional quantiles and make prediction intervals.
  • Evaluate interval estimates using appropriate metrics.

Topics

  • Classifier Calibration
    • Brier Score
    • Log Loss
    • Expected Calibration Error (ECE)
    • Maximum Calibration Error (MCE)
    • CalibratedClassifierCV
      • Platt Scaling
      • Isotonic Regression
    • Reliability Diagrams
  • Quantile Regression
    • QuantileRegressor
    • HistGradientBoostingRegressor
    • Pinball Loss
    • Prediction Intervals
      • Coverage
      • Interval Length

Activities

Upcoming Deadlines