Evaluation Methodology

How to tell if an LLM really improved: benchmarks, human and LLM-as-judge evals, arena Elo ratings, contamination and judge bias, and confidence intervals.