import os

%load_ext watermark
%load_ext autoreload
%autoreload 2

import torch
import requests
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration

%watermark -a 'Ethen' -d -u -v -iv

Author: Ethen

Last updated: 2026-01-26

Python implementation: CPython
Python version       : 3.10.14
IPython version      : 8.37.0

transformers: 4.57.3
requests    : 2.32.5
torch       : 2.6.0
PIL         : 12.0.0

Mixed Modality LLM Inputs - LLaVA¶

This document introduces a potent method for processing multimodal inputs. LLaVA (Large Language and Vision Assistant) [1] [2] demonstrated connecting pre-trained vision encoder with large language model through a MLP projection layer can achieve impressive multimodal understanding capabilities. Details are illustrated in the diagram below:

Given an input image $X_v$, they employ a pre-trained CLIP visual encoder ViT-L/14 to extract visual features $Z_v$ from the last two layers. To bridge visual and linguistic modalities, a MLP projection layer is used to maps image features into language model's word embedding space $H_v$.

As for training:

The model is first pre-trained on image text pairs to align the features of LLM and vision encoder, keeping both frozen, while only training the projection layer. This stage can be understood as training a visual tokenizer for the frozen LLM.
Next, they keep the vision encoder frozen, while training both projection layer and LLM.

Implementation¶

Our implementation will demonstrate using a pre-trained Llava model for generating image caption, and show how to implement the modality merging part.

model_name = "llava-hf/llava-1.5-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(model_name, dtype=torch.bfloat16, cache_dir="/data")
processor = AutoProcessor.from_pretrained(model_name, cache_dir="/data")
print("number of parameters: ", model.num_parameters())

Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.30it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.

number of parameters:  7063427072

image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
print(raw_image.size)
raw_image

(640, 480)