Building a Multimodal Instructional Assistant with Florence-2

I built a tool that looks at a labeled diagram, reads the text printed on it, and writes both a caption and a set of study questions from what it finds. It runs two models on a single 4GB laptop GPU. Most of the interesting work turned out to be figuring out which performance tricks didn't work on Florence-2's custom code, and building around the ones that did.

What It Does and Who It Is For

Multimodal Instructional Assistant is a small Gradio app with two tabs. Upload one image and pick "Caption" or "Generate Questions," and it returns either a written explanation of the image or five numbered study questions built from the labels on it. Upload a folder's worth of images in the batch tab and it processes them one at a time, then hands back a CSV.

Under the hood, two models do the work. Florence-2-large, Microsoft's vision-language model at about 1.5GB, handles OCR and visual captioning. TinyLlama-1.1B-Chat turns Florence-2's raw output into something readable.

It's built for the case where you already have a diagram, a labeled chart, or a photo of a textbook figure, and you want a written explanation or a quiz out of it without typing either by hand. I tested it mostly on anatomy-style diagrams, heart diagrams labeled "Aorta," "Left Ventricle," and so on, but it works on anything with printed text, and falls back to a generic visual caption when there isn't any.

The Problem That Made Me Build It

The starting point was a plain vision captioning model pointed at a labeled diagram. It described the picture in general terms: a diagram showing a heart with several labeled parts, without engaging with a single label. Running OCR alone was worse. It returned the words themselves, "Aorta Left Ventricle Pulmonary Valve," with no sentence around them.

Neither output is useful for studying. A caption without the labels doesn't teach the vocabulary. A list of labels without sentences doesn't explain what they mean or how they relate to each other.

I wanted one thing: read what's printed on the image, and write a sentence or a question that actually uses it. And I wanted it running locally on a laptop GPU, since sending every diagram to a paid captioning API wasn't a habit I wanted to build.

Architecture: How It Works

Every image goes through Florence2Handler in model_handler.py. Two Florence-2 calls happen no matter what: <OCR> to pull raw text off the image, about 1 second on my RTX 3050 Laptop GPU, and <MORE_DETAILED_CAPTION> for a general visual description, about 2 seconds. If the OCR text comes back with three or more words, both outputs get handed to TinyLlama.

if ocr_text and len(ocr_text.split()) >= 3:
    try:
        return self.enhancer.enhance_caption(ocr_text, visual_caption)
    except Exception as e:
        warnings.warn(f"Phi-3-mini enhancement failed, falling back: {e}")

return visual_caption

TinyLlama runs through a standard Hugging Face pipeline("text-generation", ...), wrapped in a small system/user prompt template, and turns "Aorta, Left Ventricle, Pulmonary Valve" plus the visual caption into either a 2-3 sentence explanation or five numbered questions. If OCR comes back empty, which happens on photos or anything without printed text, the app skips TinyLlama and returns Florence-2's caption directly.

One detail that isn't obvious from the code alone: images get padded to a square with a white border before they reach Florence-2. Its DaViT vision backbone asserts on non-square feature maps, and a rectangular photo crashes with "only support square feature maps" otherwise.

Both models stay loaded on the GPU at once. Florence-2 takes about 0.83GB of VRAM, TinyLlama about 2.2GB, leaving roughly 1.26GB of headroom out of 4.29GB measured total on the laptop GPU. TinyLlama loads eagerly when the app starts, about 10 seconds, so the first real request doesn't stall on a cold model load.

The Hardest Technical Decision

I went in wanting the usual performance checklist: KV caching for the decoding loop, Flash Attention or SDPA, 4-bit quantization, torch.compile. Florence-2 pushed back on every one of them.

KV caching was the big one. Newer versions of transformers wrap decoder caches in an EncoderDecoderCache object, and Florence-2's custom generate() method doesn't know what to do with it. It crashes with 'NoneType' object has no attribute 'shape'. I pinned transformers==4.44.2 and set use_cache=False explicitly on every generate() call, single-image and batched alike.

generated_ids = self.model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=256,
    do_sample=False,
    num_beams=1,
    use_cache=False,  # EncoderDecoderCache incompatibility with Florence-2's custom generate()
)

The rest of the checklist went the same way. Florence-2's modeling code doesn't expose the flag transformers checks for before enabling SDPA, so attention always falls back to eager, regardless of what the GPU supports. Loading the model in 4-bit with bitsandbytes ran without errors, but the captions started describing objects that weren't in the image, so I turned quantization off rather than debug a hallucination that only showed up under quantization. torch.compile raised its own errors against Florence-2's custom model class and got turned off too.

The actual decision, once all four of those were off the table, was to stop trying to make Florence-2 itself faster and spend the budget on a second, much simpler model instead. TinyLlama runs through a completely standard pipeline, no custom generate(), no custom attention class, where caching and dtype handling work exactly as documented. The caption-quality problem got solved there instead of inside Florence-2.

What I Measured

The numbers below come from the Gradio UI's own elapsed-time readout on an RTX 3050 Laptop GPU (4GB), not a formal benchmark harness. Treat them as one machine's real numbers, not a claim about Florence-2 in general.

• Plain caption, no OCR text found: about 1s OCR plus about 2s visual caption, roughly 3 seconds total.
• Enhanced caption, OCR text present and TinyLlama runs: about 16 seconds total.
• Question generation with TinyLlama: about 25 seconds total. The longer prompt, five numbered questions instead of 2-3 sentences, costs more.
• App startup: Florence-2 loads in about 20 seconds, TinyLlama in about 10, so the app is usable roughly 30 seconds after python app.py.
• VRAM: 0.83GB for Florence-2, 2.2GB for TinyLlama, 1.26GB free out of 4.29GB measured total.

The LoRA fine-tuning script has its own numbers, from a smoke test on the 5-image dummy dataset in data/dummy_dataset: 10 steps, 5 epochs, batch size 2, gradient accumulation 4. Loss went 1.89, 0.29, 1.75, 1.25, 1.28, 0.84, 0.88, 0.44, 0.63, 0.22 across those 10 steps, for a total of about 194 trillion FLOPs (total_flos: 194156284170240.0 in the trainer state). The bounce is expected. Ten steps on five images checks that the LoRA target modules, q_proj, k_proj, v_proj, out_proj, fc1, fc2, rank 8, alpha 16, don't crash against Florence-2's BART-based language backbone. It isn't a training run that produced a usable adapter.

What I Would Do Differently

The batch tab doesn't use the tensor-batched generate_batch() method that model_handler.py still defines. It used to, before TinyLlama existed, batching every image into one forward pass through Florence-2. Once captions started depending on OCR text plus a possible TinyLlama call, that stopped being simple: you don't know whether an image needs the LLM step until you've already run OCR on it. The batch tab now loops over images one at a time, with a progress bar standing in for the parallelism that used to be there.

I'd split it into two passes instead. Batch all images through Florence-2's OCR and caption stage using generate_batch(), then batch the TinyLlama calls afterward for whichever images actually have OCR text. That keeps the GPU busy through the Florence-2 stage instead of finishing one image start-to-finish before starting the next.

I'd also revisit the KV cache question against a newer transformers release instead of sitting on 4.44.2 indefinitely, and try 8-bit quantization for Florence-2 since 4-bit was specifically the one that hallucinated. I haven't tried the middle ground.

Conclusion

Multimodal Instructional Assistant runs two models on one 4GB GPU by putting the performance budget where it actually pays off: a small second model with a completely standard inference path, instead of forcing every optimization flag onto a vision-language model whose custom generation code wasn't built to support them yet. KV caching, SDPA, quantization, and torch.compile all looked like free wins going in. None of them were, on this model, today. Finding that out early was worth more than a week spent trying to force any one of them to work.