unslothai/unsloth

[Feature request] Support GPTQ quantization

Closed

#39 opened on Dec 17, 2023

View on GitHubContributor guide
 (35 comments) (0 reactions) (0 assignees)Python (5,658 forks)batch import
feature requesthelp wantedon roadmap

Repository metrics

Stars
 (64,271 stars)
PR merge metrics
 (Avg merge 3d 15h) (525 merged PRs in 30d)

Description

So I have a GPTQ llama model I downloaded (from TheBloke), and it's already 4 bit quantized. I have to pass in False for the load_in_4bit parameter of:

model, tokenizer = FastLlamaModel.from_pretrained(

because if I don't, I get an error thrown saying:

The model is already quantized with gptq. You can't quantize it again with bitsandbytes

But, if I pass in False for load_in_4bit, this code makes bnb_config be None:

        bnb_config = None
        if load_in_4bit:
            bnb_config = BitsAndBytesConfig(
                load_in_4bit              = True,
                bnb_4bit_use_double_quant = True,
                bnb_4bit_quant_type       = "nf4",
                bnb_4bit_compute_dtype    = dtype,
            )

and that makes quantization_config be None as well:

quantization_config = bnb_config,

and that crashes here:

        if hasattr(self, "quantization_config"):
            output["quantization_config"] = (
                self.quantization_config.to_dict()

with the error message:

'NoneType' object has no attribute 'to_dict'

So I'm not sure how to LoRA train this llama model. Any thoughts?

Contributor guide

Research direction
Investigate how to integrate GPTQ quantization support in unsloth, ensuring it doesn't conflict with existing bitsandbytes quantization. Check the quantization config handling to allow GPTQ models to be loaded without setting load in 4bit=True.
Tech stack
python
Domain
machine learningai
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonMachine LearningHugging Face Transformers
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30