Storia-AI/sage

[FEATURE REQUEST] Add progress bar when indexing

Open

#59 opened on Oct 6, 2024

View on GitHubContributor guide
 (9 comments) (0 reactions) (0 assignees)Python (116 forks)auto 404
good first issuehacktoberfest🔥 difficulty: 1

Repository metrics

Stars
 (1,267 stars)
PR merge metrics
 (PR metrics pending)

Description

Is your feature request related to a problem? Please describe. Indexing (especially local) can take a long time, which causes users to give up.

Describe the solution you'd like We should display a progress bar (e.g. via tqdm).

Additional context Each embedder will have a different way of calculating progress. For instance, with the OpenAIBatchEmbedder, we could have two bars: the first one for chunking + issuing indexing jobs, and the second one based on the status of indexing jobs. For the other synchronous embedders (e.g. Voyage, Gemini, Marqo), we'd just have one.

Contributor guide

Research direction
Implement progress bars for each embedder using tqdm. For OpenAIBatchEmbedder, two progress bars: one for chunking and issuing jobs, another for job status. For synchronous embedders, one progress bar. Ensure progress updates are accurate.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Pythontqdm
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70