ggml-org/llama.cpp

server: exit failure if `--embedding` is set with an incoherent `--ubatch-size`

Closed

#6,263 opened on Mar 23, 2024

View on GitHubContributor guide
 (8 comments) (1 reaction) (0 assignees)C++ (21,764 forks)batch import
enhancementgood first issuehelp wanted

Repository metrics

Stars
 (124,134 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Context

there is no advantage to increase n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.

Proposition

Exit failure if --embedding is set and --ubatch-size != --batch-size in the server example. Probably also in the retrieval example in #6193.

Aldo probably KV bert.context_size must be taken into account.

Contributor guide

Research direction
In the server example, add validation that when embedding is set, ubatch size must equal batch size, otherwise print an error and exit. Also consider checking KV bert.context size consistency.
Tech stack
ccpp
Domain
backendapi
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++Git
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70