What’s going on with the Open LLM Leaderboard?

Hugging Face yanked 30 models off the Open LLM Leaderboard after catching systematic prompt injection attacks that inflated MT-Bench scores by up to 12%. The cheat was as simple as stuffing the system prompt with hidden instructions telling the judge LLM to rate responses higher.

The leaderboard was your shortcut for picking base models; now every download risks a Trojan horse that tanks RAG performance in production. Running Llama-3-8B because it “ranked #2” is just gambling with 4 GPU-hours of your fine-tuning budget on a model that might hallucinate 40% more than the clean 7B that sits at #15.

Teams shipping customer-facing agents this quarter need to re-run their own evals—everyone else can keep trusting the old offline numbers.