Exploring potential use instances of Phi-3-Imaginative and prescient, a small but highly effective MLLM that may be run domestically (with code examples)

Microsoft just lately launched Phi-3, a strong language mannequin, with a brand new Imaginative and prescient-Language variant known as Phi-3-vision-128k-instruct. This 4B parameter mannequin achieved spectacular outcomes on public benchmarks, even surpassing GPT-4V in some instances and outperforming Gemini 1.0 Professional V in all however MMMU.

This weblog put up will discover how one can make the most of Phi-3-vision-128k-instruct as a sturdy imaginative and prescient and textual content mannequin in your information science toolkit. We’ll reveal its capabilities by means of varied use instances, together with:

• Optical Character Recognition (OCR)
• Picture Captioning
• Desk Parsing
• Determine Understanding
• Studying Comprehension on Scanned Paperwork
• Set-of-Mark Prompting

We’ll start by offering a easy code snippet to run this mannequin domestically utilizing transformers and bitsandbytes. Then, we’ll showcase an instance for every of the use instances listed above.

Operating the mannequin domestically:

Create a Conda Python surroundings and set up torch and different python dependencies:

conda set up pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
pip set up git+https://github.com/huggingface/transformers.git@60bb571e993b7d73257fb64044726b569fef9403 pillow==10.3.0 chardet==5.2.0 flash_attn==2.5.8 speed up==0.30.1 bitsandbytes==0.43.1

Then, we will run this script:

# Instance impressed from https://huggingface.co/microsoft/Phi-3-vision-128k-instruct
# Import mandatory libraries
from PIL import Picture
import requests
from transformers import AutoModelForCausalLM
from transformers import AutoProcessor
from transformers import BitsAndBytesConfig
import torch
# Outline mannequin ID
model_id = "microsoft/Phi-3-vision-128k-instruct"
# Load processor
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
# Outline BitsAndBytes configuration for 4-bit quantization
nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Load mannequin with 4-bit quantization and map to CUDA
mannequin = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="cuda",
trust_remote_code=True,
torch_dtype="auto",
quantization_config=nf4_config,
)
# Outline preliminary chat message with picture placeholder
messages = [image_1]
# Obtain picture from URL
url = "https://pictures.unsplash.com/photo-1528834342297-fdefb9a5a92b?ixlib=rb-4.0.3&q=85&fm=jpg&crop=entropy&cs=srgb&dl=roonz-nl-vjDbHCjHlEY-unsplash.jpg&w=640"
picture = Picture.open(requests.get(url, stream=True).uncooked)
# Put together immediate with picture token
immediate = processor.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
# Course of immediate and picture for mannequin enter
inputs = processor(immediate, [image], return_tensors="pt").to("cuda:0")
# Generate textual content response utilizing mannequin
generate_ids = mannequin.generate(
**inputs,
eos_token_id=processor.tokenizer.eos_token_id,
max_new_tokens=500,
do_sample=False,
)
# Take away enter tokens from generated response
generate_ids = generate_ids[:, inputs["input_ids"].form[1] :]
# Decode generated IDs to textual content
response = processor.batch_decode(
generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
# Print the generated response
print(response)

This code will load the Phi-3 mannequin like we might any transfomersmannequin. We add bitsandbytes in order that its suits in a consumer-grade GPU reminiscence.
We use a easy immediate `<|image_1|>nWhat is proven on this picture?` the place we reference the picture and ask for an outline of what’s in it. This immediate get processed with the picture (Similar picture as this weblog’s thumbnail) and fed by means of the mannequin. This ends in the next output:

The picture reveals a single yellow flower with a inexperienced stem in opposition to a blue background.

As soon as the mannequin is loaded, the processing and prediction took 2s on an RTX3080.

Now that we’re all arrange, lets discover some use instances:

Optical Character Recognition (OCR)

We need to transcribe the textual content from a picture. To take action, change the message and the picture URL traces within the code snippet above with:

messages = [>nOCR the text of the image as is. OCR:"]
url = "https://i.postimg.cc/fRFvFVyv/Screenshot-from-2024-05-22-20-55-43.png"

Enter:

Output:

3 EXPERIMENTS
We present Place Interpolation can successfully lengthen context window as much as 32 occasions of the unique measurement, and such extension could be performed with solely a number of tons of of coaching steps. We present the ensuing fashions are sturdy LLMs with totally efficient lengthy context home windows. We reveal its efficiency in various duties together with language modeling, passkey retrieval, and lengthy doc-
ument summarization. We additionally current benchmark outcomes of the prolonged fashions on the unique LLaMA analysis benchmarks.

The result’s as anticipated! The total textual content transcription took 6 seconds, which is a bit gradual in comparison with common OCR.

Picture Captioning

The target right here is to caption a pure picture:

messages = [image_1]
url = "https://i.postimg.cc/q7Wjtj3n/Screenshot-from-2024-05-22-20-56-56.png"

Enter:

Output:

This picture reveals a luxurious toy resembling a personality from a well-liked animated tv collection. The toy has a spherical physique with a hood, and it’s carrying a pair of glasses. The character’s face isn’t seen within the picture.

Largely proper (no glasses 😞). Nonetheless, it’s spectacular that it acknowledged Kenny from an animated collection.

Desk Parsing

We need to see if the mannequin is ready to parse the desk right into a format that’s machine readable:

messages = [>nExtract the table content of the table in JSON format: "]
url = "https://i.postimg.cc/W1YwqXk8/Screenshot-from-2024-05-22-21-41-03.png"

Enter:

Output:

```json
[
{
"Benchmark": "Phi-3 Vision-128K-In1",
"MMMU": 40.2,
"MMBench": 80.5,
"ScienceQA": 90.8
},
{
"Benchmark": "LlaVA-1.6 Vicuna-7B",
"MMMU": 34.2,
"MMBench": 76.3,
"ScienceQA": 70.6
},
{
"Benchmark": "QWEN-VI Chat",
"MMMU": 39.0,
"MMBench": 75.8,
"ScienceQA": 67.2
}
]
```

It’s certainly in a position to extract the content material of the desk, in structured format, instantly from the screenshot.

Determine Understanding

We immediate the mannequin to explain a determine:

messages = [>nWhat is this figure about? "]
url = "https://i.postimg.cc/Y0sRQR4Q/Screenshot-from-2024-05-22-21-47-39.png"

Enter:

Output:

The determine is a bar chart evaluating the efficiency of varied machine studying fashions, with every mannequin represented by a pair of stacked bars indicating the efficiency of the bottom mannequin and the fine-tuned mannequin.

Temporary however right!

Studying Comprehension on Scanned Paperwork

The objective is to ask questions on scanned or photographed textual content. This differs from Optical Character Recognition (OCR) in that we’re not occupied with merely transcribing the textual content. As a substitute, we need to assess whether or not the mannequin can perceive the written content material and reply questions on it instantly.

messages = [>nWhat is this book about? "]
url = "https://i.postimg.cc/rmHxPbQY/Screenshot-from-2024-05-22-20-56-21.png"

Enter:

Output:

The picture reveals the duvet of a e-book titled ‘SAM TALLENT RUNNING THE LIGHT’ with a synopsis describing the story of a comic named Billy Ray Schafer who’s coping with private {and professional} challenges. The e-book seems to be a piece of fiction, probably a novel, because it features a quote from DOUG STANHOPE, ICONOCLAST, praising the e-book as good and one of the best fictional illustration of comedy.

The e-book title is fallacious however the the MLLM was in a position to perceive what the textual content within the image is about and summarize it in a single shot.

Set-of-Mark Prompting

Set-of-Mark (SoM) prompting makes use of interactive segmentation fashions to divide a picture into areas and mark them with symbols, enabling massive multimodal fashions to higher perceive and reply visually grounded questions.

To simplify issues on this instance, I marked the objects manually as an alternative of utilizing a mannequin after which referenced the mark (4) in my immediate:

messages = [>nWhat is object number 4? "]
url = "https://i.postimg.cc/fy0Lz798/scott-webb-p-0l-WFknspg-unsplash-2.jpg"

Enter:

Object quantity 4 is a cactus with orange flowers in a pot.

The MLLM was in a position to perceive my reference and reply my query accordingly.

So, there you will have it! Phi-3-Imaginative and prescient is a strong mannequin for working with pictures and textual content, able to understanding picture content material, extracting textual content from pictures, and even answering questions on what it sees. Whereas its small measurement, with solely 4 billion parameters, could restrict its suitability for duties demanding sturdy language expertise, most fashions of its class are at the very least twice its measurement at 8B parameters or extra, making it a standout for its effectivity. It shines in purposes like doc parsing, desk construction understanding, and OCR within the wild. Its compact nature makes it excellent for deployment on edge units or native consumer-grade GPUs, particularly after quantization. It will likely be my go-to mannequin in all doc parsing and understanding pipelines, as its zero-shot capabilities make it a succesful instrument, particularly for its modest measurement. Subsequent, I may even work on some LoRA fine-tuning scripts for this mannequin to see how far I can push it on extra specialised duties.

References: