huggingface/lighteval

[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split

Open

#634 opened on Mar 21, 2025

 (3 comments) (0 reactions) (0 assignees)Python (514 forks)auto 404
featuregood first issuehelp wanted

Repository metrics

Stars
 (2,496 stars)
PR merge metrics
 (PR metrics pending)

Description

Solution/Feature

When creating a custom task, the few-shot samples can be specified via a split from the dataset. However, it would be convenient if there were a way to make it so I can specify samples from the split that will be used as the examples, similar to what LM Evaluation Harness does here in its yaml config:

test_split: train
fewshot_split: train
fewshot_config:
  sampler: id_sampler
  sampler_config:
    id_list: ["2022_21", "2022_88", "2022_143"]
    id_column: id
    exclude_from_task: true

Something along the lines of:

@dataclass
class LightevalTaskConfig:
    """Stored configuration of a given [`LightevalTask`].

    Arguments:
    ...
        hf_subset (str): Subset used for the current task, will be default if none is selected.
        hf_avail_splits (list[str]): All the available splits in the evaluation dataset
        evaluation_splits (list[str]): List of the splits actually used for this evaluation
        few_shots_split (str): Name of the split from which to sample few-shot examples
        few_shots_id_column (str): Column name in the dataset to use for the few_shot_id_list
        few_shots_id_list (list[str]): List of unique identifiers corresponding to the samples in the dataset, based on matching these IDs with the values in few_shots_id_column.
        few_shots_select (str): Method with which to sample few-shot examples
    ...
    """

The new arguments would be few_shots_id_column and few_shots_id_list, and wrote a rough description of what they would look like.

Contributor guide