ShishirPatil/gorilla

[Apibench] Resume interrupted LLM generations from last generation

Open

#484 opened on Jun 24, 2024

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (1,397 forks)batch import
apibench-datagood first issue

Repository metrics

Stars
 (12,996 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Describe the issue When we try to launch LLM generations on APIBench, if the generation is interrupted midway, currently the partial file is deleted and restart generation from start. What we should instead do is check where the generation stopped and resume from the last generated index. This is specially critical for large models where generation can take >1 hours.

ID datapoint 2. Provider: This affects all 3: TorchHub/HuggingFace/PyTorch Hub

File that needs to be edited: https://github.com/ShishirPatil/gorilla/blob/main/eval/get_llm_responses.py

Contributor guide

Research direction
Inspect get llm responses.py to detect where generation was interrupted and implement logic to resume from the last successful index instead of restarting.
Tech stack
python
Domain
aibackend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75