huggingface/lighteval

[EVAL] Add kyrgyzLLM benchmark

Open

#1,036 opened on Nov 4, 2025

View on GitHub
 (1 comment) (0 reactions) (1 assignee)Python (514 forks)auto 404
good first issuenew-task

Repository metrics

Stars
 (2,496 stars)
PR merge metrics
 (PR metrics pending)

Description

Hi,

We just open-sourced the Kyrgyz LLM Evaluation Dataset.

Evaluation short description

  • Why is this evaluation interesting?

KyrgyzLLM-Bench is the first comprehensive benchmark suite for deep language understanding in Kyrgyz. It is interesting because it provides broad, culturally grounded coverage by combining native benchmarks (such as KyrgyzMMLU and KyrgyzRC) with carefully translated and post-edited international benchmarks (such as HellaSwag, WinoGrande, BoolQ, GSM8K, and TruthfulQA).

  • How used is it in the community? As the benchmark was released recently, its adoption by the community is just beginning. It is significant because it's the first comprehensive benchmark suite for deep language understanding, specifically in the Kyrgyz language, providing a new and essential tool for researchers and developers.

Evaluation metadata

Thanks you!

Contributor guide

Research direction
Examine existing benchmark implementations in lighteval to understand the integration pattern. Study the KyrgyzLLM Bench dataset structure and implement a new evaluation task following the same conventions.
Tech stack
python
Domain
aibackend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60