ggml-org/llama.cpp

Request: Nougat OCR Integration

Open

#3,294 opened on Sep 21, 2023

View on GitHubContributor guide
 (8 comments) (13 reactions) (0 assignees)C++ (21,764 forks)batch import
help wantedmodel

Repository metrics

Stars
 (124,134 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Request: Nougat OCR Integration

I suggest adding Nougat OCR into llama.cpp to enable the processing of scientific PDF documents. This can act as a first step towards adding multimodal models to this project!

Implementation: It seems that Nougat is based on standard transformer architecture (like Bart and Swin Transformer) and most of the work would be on figuring out how to add the image processing.

Let me know what you think! P.S.: Love this repo! I hope to add my own retrieval-pretrained transformer at some point to this repo.

Contributor guide

Research direction
Investigate the Nougat architecture and its dependencies. Determine how to incorporate image processing into the existing llama.cpp codebase, possibly by adding a new model type or integrating an external library. Review similar PRs or issues that added new models.
Tech stack
cpp
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Needs investigation
Prerequisites
C++CMakeGitBasic knowledge of transformer architectures
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40