See https://arxiv.org/abs/2409.11149 SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration

Install SAGED-bias

Install the latest version of SAGED-bias from PyPi using pip:

pip install sagedbias

We officially support Python 3.10, and 3.11.

Customize bias-benchmarking prompts and metrics with SAGEDbias

Benchmark Building: Scrape and Assemble

Benchmark Running: Generate and Extract

Diagnosis: Group, Summarize, and Compare

Pipeline - Build and Run Benchmark