AIC & BIC: Model Comparison Criteria

AIC (Akaike) and BIC (Bayesian) Information Criteria measure model quality, penalizing for complexity to prevent overfitting.

• Lower AIC/BIC → better model. The absolute number means nothing on its own — only the gap between two models does.
• AIC = n·ln(RSS/n) + 2k    BIC = n·ln(RSS/n) + k·ln(n)
• BIC penalizes extra parameters more than AIC
• BIC tends to select simpler models

AIC or BIC — which one? AIC asks "which model predicts best?"; BIC asks "which model is most likely the true one?". Predicting a process → AIC. Deciding which variables really matter → BIC. When the two disagree, the disagreement is itself information: the extra variable helps prediction but is not strongly supported.

How big a difference counts? ΔAIC under 2 → the models are tied, take the simpler one. Between 2 and 10 → the lower one is genuinely better. Over 10 → decisive.

Key rule: AIC/BIC only makes sense comparing models on the same dataset. Same rows, same response — add or drop rows and the numbers are no longer comparable.