UnBias-Plus Leaderboard

An open benchmark for evaluating fine-tuned LLMs on bias detection and neutral rewriting of news articles. Evaluated on an out-of-distribution dataset of 300 samples, equally distributed between biased and unbiased examples, using an LLM as judge.

Models

300

Test samples

Metrics

Overall model performance. Models compared on parse rate, bias reduction, contextual relevance, unbiased handling, segment quality, and inference speed. Higher is better for all metrics except latency.

Legacy still leads on raw bias reduction (60.6%) and unbiased handling (correct ID median 5.0)
Qwen3-8B V2-D3 is the strongest of the new 8B SFT lineage on bias reduction (57.0%), narrowly ahead of V2-D4, V2, V2-D1, and V2-D2
Qwen3-8B V2-D4 is the most reliable and cleanest new checkpoint: best parse rate (99.4%, tied with Legacy) and lowest hallucination rate (2.9%) of any model, though it trades off slightly lower bias reduction (54.3%)
Qwen3-4B ORPO (experimental) shows a severe reliability regression: parse rate drops to 80.9% (worst of all 10 models) and correct identification collapses to 2.0, the lowest recorded, not yet production ready
Qwen3-4B V2-D3 matches Qwen3-8B V2-D3's bias reduction closely (56.2% vs 57.0%) on a smaller model, but trails the V2-series 8B checkpoints on correct identification (3.0 vs 3.3-3.9), suggesting a capacity-related judgment gap rather than a recipe issue

Biased sample metrics. Computed on the biased half of the evaluation dataset. Bias reduction % = (original_bias - rewrite_bias) / original_bias on a 0-5 LLM judge scale. Global rewrite quality compares model output against the human reference rewrite.

Unbiased sample handling. Computed on the unbiased half of the evaluation dataset. A score of 5.0 = model correctly identified the article as unbiased and preserved the original text unchanged. Results are split: Qwen3-8B V2-D2, V2-D3, V2-D4, and Legacy score a median of 5.0, while Qwen3.5-4B, the original Qwen3-8B UnBias-Plus, Qwen3-8B V2, V2-D1, Qwen3-4B V2-D3, and Qwen3-4B ORPO all score a median of 2.0, meaning they label unbiased articles as biased more than half the time. Qwen3-4B ORPO is the most affected, with correct identification mean dropping to 2.0.

Segment-level metrics. Recall at words measures how many ground-truth biased words appear in at least one model segment. New models outperform Legacy here despite losing on global metrics. Lower is better for hallucination and duplicate rates.

What is UnBias-Plus?

UnBias-Plus is an open project by the Vector Institute (AIXpert) for detecting and neutralizing bias in news articles using fine-tuned large language models.

Models return structured JSON with severity, bias_found, biased_segments (each with original, replacement, severity, bias_type, reasoning), and unbiased_text.

For questions or collaboration, email shaina.raza@vectorinstitute.ai.

Evaluation design

All models are evaluated on an out-of-distribution dataset of 300 samples, equally distributed between biased and unbiased examples, using an LLM as judge.

Metrics explained

Bias reduction % measures how much bias the model removes: (original_bias - rewrite_bias) / original_bias on a 0-5 GPT judge scale.

Recall at words measures segment localization: what percentage of ground-truth biased words appear in at least one model segment (substring match).

Global rewrite quality compares the model rewrite against the human reference rewrite (0-5 LLM judge).

Correct identification scores whether the model correctly labeled and preserved unbiased articles (5.0 = perfect preservation, 2.0 = mislabeled as biased).

Citation

If you use UnBias-Plus, please cite the paper:

@article{radwan2026unbias,
  title={UnBias-Plus: Detect, Explain, and Rewrite Bias},
  author={Radwan, Ahmed Y and ElKady, Ahmed and Chaduvula, Sindhuja and Hafez, Mohamed and Krishnan, Amrit and Raza, Shaina},
  journal={arXiv preprint arXiv:2606.23412},
  year={2026}
}

Built by the Vector Institute AIXpert team.