ggml-org/llama.cpp

in situ auto-Frankenmerges

Closed

#4,718 opened on Dec 31, 2023

View on GitHubContributor guide
 (11 comments) (2 reactions) (0 assignees)C++ (21,737 forks)batch import
enhancementgood first issue

Repository metrics

Stars
 (124,043 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Feature Description

Modify llama.cpp to support on-the-fly "Frankenmerging" of the model in memory with itself.

Motivation

Frankenmerges, including auto-Frankenmerges, are becoming increasingly popular and appear to have properties that merit further study; it's Rich Sutton's "bitter lesson" in the small: stacking more decoder blocks means a greater total amount of computation in a single inference pass and, perhaps surprisingly, under the right circumstances, that greater accessible computation outweighs the 'noise' induced by performing fairly brutal surgery on the order of decoder blocks.

Right now experimentation is taking place at the level of building new models with mergekit. This is slow. The ability to mix-and-match decoder blocks on the fly in llama.cpp would speed up iteration and experimentation, helping better understand the tradeoff between greater available net computation and decoder surgery induced noise.

Possible Implementation

Something like this:

https://github.com/semiring/IRL-llama.cpp/blob/master/llama.cpp#L4346

Contributor guide

Research direction
Study the linked reference implementation in IRL llama.cpp, specifically around line 4346, to understand the proposed approach. Review existing llama.cpp architecture for loading and managing decoder blocks. Investigate how to dynamically reorder or replicate blocks during inference without modifying the model file. Consider performance implications and memory management. The issue has 11 comments; check if any maintainer feedback guides the implementation direction.
Tech stack
cpp
Domain
machine learningai
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++ programmingTransformer architecture understandingllama.cpp codebase familiarityModel merging concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15