# LLaVA-OneVision
## 概述
LLaVA-OneVision 模型由 Bo Li、Yuanhan Zhang、Dong Guo、Renrui Zhang、Feng Li、Hao Zhang、Kaichen Zhang、Yanwei Li、Ziwei Liu 和 Chunyuan Li 在 [LLaVA-OneVision: Easy Visual Task Transfer](https://huggingface.co/papers/2408.03326) 中提出。
LLaVA-OneVision 是一个视觉语言模型，能够根据一张或多张图像/视频生成文本。该模型由 SigLIP 视觉编码器和 Qwen2 语言骨干网络组成。图像采用 anyres-9 技术进行处理，即将图像分割为 9 个块，以更好地处理高分辨率图像并尽可能多地捕捉细节。然而，为了更节省内存，视频每帧会被池化（pooling）为总长度为 196 个 token 的序列。LLaVA-OneVision 提供 0.5B、7B 和 72B 三种规模，并在基准评估中取得了卓越的性能。
该论文的摘要如下：
*我们推出了 LLaVA-OneVision，这是一个开放的大型多模态模型（LMMs）系列，它整合了我们在 LLaVA-NeXT 博客系列中关于数据、模型和视觉表示的见解。实验结果表明，LLaVA-OneVision 是首个能够同时在单图像、多图像和视频这三个重要的计算机视觉场景中突破开放 LMMs 性能边界的单一模型。重要的是，LLaVAOneVision 的设计支持跨不同模态/场景的强大迁移学习，从而产生新的涌现能力。特别是，通过从图像到视频的任务迁移，展示了强大的视频理解和跨场景能力。*
LLaVA-OneVision 架构。摘自原始论文。
提示：
- 我们建议用户在计算批量生成时使用 `padding_side="left"`，因为这能带来更准确的结果。只需确保在生成之前调用 `processor.tokenizer.padding_side = "left"` 即可。
- Llava-OneVision 针对图像使用不同数量的 patch，因此除了处理输入时执行的填充外，还必须在建模代码内部对输入进行填充。如果模型处于 `eval()` 模式，默认设置为“左侧填充”，否则为“右侧填充”。
### 使用聊天模板格式化提示
每个 **checkpoint** 都使用特定的提示格式进行训练，具体取决于底层的大语言模型骨干网络。为确保格式正确，请使用处理器的 `apply_chat_template` 方法。
**重要：**
- 您必须构建对话历史——直接传入普通字符串将无法正常工作。
- 每条消息应为一个字典，包含 `"role"` 和 `"content"` 键。
- `"content"` 应为包含不同模态（如 `"text"` 和 `"image"`）的字典列表。
以下是输入结构的示例。我们将使用 [llava-onevision-qwen2-7b-si-hf](https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-si-hf) 以及包含文本和图像的对话历史。每个 content 字段都必须是一个字典列表，如下所示：
```python
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-si-hf")
conversation = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "What’s shown in this image?"},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": "This image shows a red stop sign."},]
},
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the image in more details."},
],
},
]
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Note that the template simply formats your prompt, you still have to tokenize it and obtain pixel values for your images
print(text_prompt)
'<|im_start|>user\nWhat is shown in this image?<|im_end|>\n<|im_start|>assistant\nPage showing the list of options.<|im_end|>'
```
🚀 **附加内容：**如果你使用的是 `transformers>=4.49.0`，还可以从 `apply_chat_template` 获取向量化输出。有关如何使用它的更多详细信息，请参阅下方的 **使用示例**。
该模型由 [RaushanTurganbay](https://huggingface.co/RaushanTurganbay) 贡献。原始代码可在[此处](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/main)找到。
## 使用示例
### 单图像推理
以下是加载模型并以半精度（`torch.float16`）执行推理的方法：
```python
import torch
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
"llava-hf/llava-onevision-qwen2-7b-ov-hf",
device_map="auto"
)
# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
conversation = [
{
"role": "user",
"content": [
{"type": "image", "url": url},
{"type": "text", "text": "What is shown in this image?"},
],
},
]
inputs = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
inputs = inputs.to(model.device, torch.float16)
# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
'user\n\nWhat is shown in this image?\nassistant\nThe image shows a radar chart, also known as a spider chart or a star chart, which is used to compare multiple quantitative variables. Each axis represents a different variable, and the chart is filled with'
```
### 多图像推理
LLaVa-OneVision 支持以多张图片作为输入进行推理，这些图片可以属于同一个提示词，也可以属于不同的提示词（在批量推理中）。为此，您必须使用带有 "ov" 后缀的检查点。对于多张图片的情况，我们建议使用**嵌套的图片列表**作为输入。否则，每张图片都会被切分为 patch，从而消耗大量内存。以下是具体操作方式：
```python
import torch
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
# Load the model in half-precision
model = LlavaOnevisionForConditionalGeneration.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf", device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")
# Prepare a batch of two prompts, where the first one is a multi-turn conversation and the second is not
conversation_1 = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
{"type": "text", "text": "What is shown in this image?"},
],
},
{
"role": "assistant",
"content": [
{"type": "text", "text": "There is a red stop sign in the image."},
],
},
{
"role": "user",
"content": [
{"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
{"type": "text", "text": "What about this image? How many cats do you see?"},
],
},
]
conversation_2 = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/microsoft/kosmos-2-patch14-224/resolve/main/snowman.jpg"},
{"type": "text", "text": "What is shown in this image?"},
],
},
]
inputs = processor.apply_chat_template(
[conversation_1, conversation_2],
add_generation_prompt=True,
tokenize=True,
return_dict=True,
padding=True,
padding_side="left",
return_tensors="pt",
).to(model.device, torch.float16)
# Generate
generate_ids = model.generate(**inputs, max_new_tokens=30)
processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
['user\n\nWhat is shown in this image?\nassistant\nThere is a red stop sign in the image.\nuser\n\nWhat about this image? How many cats do you see?\nassistant\ntwo', 'user\n\nWhat is shown in this image?\nassistant\nThe image shows a whimsical scene of a snowman sitting by a campfire. The snowman is anthropomorphized, wearing a hat and']
```
### 视频推理
LLaVa-OneVision 还支持以视频作为输入进行推理，其中视频帧被视为多张图片。以下是具体操作方法：
```python
from huggingface_hub import hf_hub_download
import torch
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
# Load the model in half-precision
model = LlavaOnevisionForConditionalGeneration.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf", device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")
video_path = hf_hub_download(repo_id="raushan-testing-hf/videos-test", filename="sample_demo_1.mp4", repo_type="dataset")
conversation = [
{
"role": "user",
"content": [
{"type": "video", "path": video_path},
{"type": "text", "text": "Why is this video funny?"},
],
},
]
inputs = processor.apply_chat_template(
conversation,
num_frames=8
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt"
).to(model.device, torch.float16)
out = model.generate(**inputs, max_new_tokens=60)
processor.batch_decode(out, skip_special_tokens=True, clean_up_tokenization_spaces=True)
["user\n\nWhy is this video funny?\nassistant\nThe video appears to be humorous because it shows a young child, who is wearing glasses and holding a book, seemingly reading with a serious and focused expression. The child's glasses are a bit oversized for their face, which adds a comical touch, as it's a common trope to see children wearing"]
```
## 模型优化
### 使用 bitsandbytes 进行量化
该模型可以以 8 位或 4 位精度加载，在保持原始模型性能的同时大幅降低内存需求。首先请确保已安装 bitsandbytes，`pip install bitsandbytes`，并确保拥有该库支持的 GPU/加速器。
bitsandbytes 正在重构以支持 CUDA 之外的多种后端。目前，ROCm（AMD GPU）和 Intel CPU 的实现已趋于成熟，Intel XPU 正在开发中，预计将在第四季度/第一季度支持 Apple Silicon。有关安装说明和最新的后端更新，请访问 [此链接](https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backend)。
我们非常重视您的反馈，以便在正式发布前发现并修复问题！请查阅[这些文档](https://huggingface.co/docs/bitsandbytes/main/en/non_cuda_backends)以获取更多详情和反馈链接。
只需将上述代码片段更改为：
```python
from transformers import BitsAndBytesConfig, LlavaOnevisionForConditionalGeneration
# specify how to quantize the model
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.float16,
)
model = LlavaOnevisionForConditionalGeneration.from_pretrained(model_id, quantization_config=quantization_config, device_map="auto")
```
### 使用 Flash-Attention 2 进一步加快生成速度
首先，请确保已安装 flash-attn。有关该包的安装方法，请参阅 [Flash Attention 的原始仓库](https://github.com/Dao-AILab/flash-attention)。只需将上述代码片段更改为：
```python
from transformers import LlavaOnevisionForConditionalGeneration
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
model_id,
use_flash_attention_2=True
).to(0, device_map="auto")
```
## LlavaOnevisionConfig[[transformers.LlavaOnevisionConfig]]
#### transformers.LlavaOnevisionConfig[[transformers.LlavaOnevisionConfig]]
```python
transformers.LlavaOnevisionConfig(transformers_version: str | None = None, architectures: list[str] | None = None, output_hidden_states: bool | None = False, return_dict: bool | None = True, dtype: str | torch.dtype | None = None, chunk_size_feed_forward: int = 0, is_encoder_decoder: bool = False, id2label: dict[int, str] | dict[str, str] | None = None, label2id: dict[str, int] | dict[str, str] | None = None, problem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = None, vision_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None, text_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None, image_token_index: int = 151646, video_token_index: int = 151647, projector_hidden_act: str = 'gelu', vision_feature_select_strategy: typing.Literal['default', 'full'] = 'full', vision_feature_layer: int | list[int] = -1, multimodal_projector_bias: bool = True, tie_word_embeddings: bool = False, image_grid_pinpoints: list | None = None, vision_aspect_ratio: str = 'anyres_max_9')
```
[来源](https://github.com/huggingface/transformers/blob/v5.23.0/src/transformers/models/llava_onevision/configuration_llava_onevision.py#L26)
**参数：**
vision_config (`Union[dict, ~configuration_utils.PreTrainedConfig]`，*可选*)：视觉骨干网络的配置对象或字典。
text_config (`Union[dict, ~configuration_utils.PreTrainedConfig]`，*可选*)：文本骨干网络的配置对象或字典。
image_token_index (`int`, *可选*, 默认为 `151646`) : 用作输入图像占位符的图像标记索引。
video_token_index (`int`, *可选*, 默认为 `151647`) : 用作输入视频占位符的视频 token 索引。
projector_hidden_act (`str`，*可选*，默认为 `gelu`)：多模态投影器所使用的激活函数。
vision_feature_select_strategy (`Literal[default, full]`，*可选*，默认为 `full`)：用于从视觉骨干网络中选择视觉特征的特征选择策略。
vision_feature_layer (`Union[int, list[int]]`，*可选*，默认值为 `-1`)：用于选择视觉特征的层索引。如果提供了多个索引，则会将对应索引的视觉特征拼接起来，以形成视觉特征。
multimodal_projector_bias (`bool`，*可选*，默认为 `True`)：是否在多模态投影器中使用偏置。
tie_word_embeddings (`bool`，*可选*，默认为 `False`)：是否根据模型的 `tied_weights_keys` 映射来绑定权重嵌入。
image_grid_pinpoints (`List`, *可选*) : 用于处理高分辨率图像的可用分辨率列表。列表中的每个元素都应为 `(height, width)` 形式的元组或列表。
vision_aspect_ratio (`str`，*可选*，默认为 `"anyres_max_9"`)：处理图像特征时使用的宽高比。默认值为 "anyres_max_9"。
这是用于存储 LlavaOnevisionModel 配置的类。它根据指定的参数实例化 Llava Onevision 模型，并定义模型架构。使用默认值实例化配置将得到与 [llava-hf/llava-onevision-qwen2-7b-ov-hf](https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-ov-hf) 类似的配置。
配置对象继承自 [PreTrainedConfig](/docs/transformers/v5.23.0/zh/main_classes/configuration#transformers.PreTrainedConfig)，可用于控制模型输出。有关更多信息，请参阅 [PreTrainedConfig](/docs/transformers/v5.23.0/zh/main_classes/configuration#transformers.PreTrainedConfig) 的文档。
示例：
```python
>>> from transformers import LlavaOnevisionForConditionalGeneration, LlavaOnevisionConfig, SiglipVisionConfig, Qwen2Config
>>> # Initializing a CLIP-vision config
>>> vision_config = SiglipVisionConfig()
>>> # Initializing a Llama config
>>> text_config = Qwen2Config()
>>> # Initializing a Llava-Next llava-hf/llava-onevision-qwen2-7b-ov-hf style configuration
>>> configuration = LlavaOnevisionConfig(vision_config, text_config)
>>> # Initializing a model from the llava-hf/llava-onevision-qwen2-7b-ov-hf style configuration
>>> model = LlavaOnevisionForConditionalGeneration(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
```
## LlavaOnevisionProcessor[[transformers.LlavaOnevisionProcessor]]
#### transformers.LlavaOnevisionProcessor[[transformers.LlavaOnevisionProcessor]]
```python
transformers.LlavaOnevisionProcessor(image_processor = None, tokenizer = None, video_processor = None, num_image_tokens = None, vision_feature_select_strategy = None, chat_template = None, image_token = '', video_token = '