What is multimodal AI?
Plain-English definition · Updated 2026-10-23. Tokenization rules below are vendor documentation snapshots — they change with model generations.
AITop is an independent guide. Prices and features change fast — always check the vendor's page.
Multimodal describes a model that handles more than one kind of data — usually text plus images, audio or video — in a single network, instead of chaining a transcriber, an image captioner and a language model together. OpenAI's GPT-4o (May 2024) made the term mainstream: it accepts any combination of text, audio, images and video as input and is trained end-to-end, so all inputs and outputs pass through the same neural network — which is why it can respond to audio in an average of 320 milliseconds, where the old three-model voice pipeline averaged seconds and lost tone, laughter and background sound along the way. For a buyer, multimodality shows up in two places: which inputs a tool takes natively (and what it can hear that a pipeline can't), and the bill — because images, audio and video are tokenized by rules that differ sharply per vendor, so the same 1024×1024 screenshot can cost a few hundred tokens on one API and well over a thousand on another.
What it actually means
A model's modality is the kind of data it can read or write. A text-only model sees characters; a multimodal model sees pixels, waveforms and frames as first-class input, all converted into the same token stream the model already consumes (see AI token). Two architectures get sold under the same word. Natively multimodal models — OpenAI's GPT-4o is the canonical example — are trained end-to-end across text, vision and audio; OpenAI's own launch material contrasts this with the voice mode that preceded it, which chained three separate models (speech-to-text, a text model, text-to-speech) and therefore could not observe tone, speaker count or background noise. Bolted-on multimodality is the pipeline version: vision or audio grafted onto a text model through adapters. Both accept your image; only one has actually "seen" it the way it sees text.
The distinction is not academic. A native multimodal model answers spoken audio with sub-second latency because there is no transcription round-trip; a pipeline burns time and drops information at every seam. And when a product page says "supports images", the honest follow-up question is how: one model may read a chart as native visual input, while another runs it through an OCR layer first — the difference shows up in what it gets wrong, not just in what it costs.
Why it matters when you're picking a tool
Because "multimodal" on a landing page is a capability claim with a meter attached, and the meter is where vendors differ most. Every major API converts non-text input into tokens, but the conversion rules are completely different — and they are published, so you can do the arithmetic before you commit:
OpenAI: tile-based. Image inputs are billed by 512-pixel tiles. At detail: low, an image is a flat 85 tokens regardless of size; at detail: high, the image is scaled and split into 512×512 tiles worth 170 tokens each, plus the 85-token base (the current 4o/4.1/4.5 generation; newer models use 70/140). OpenAI's documentation works the example for you: a 1024×1024 image at high detail costs 765 tokens — 4 tiles × 170 + 85.
Anthropic: pixel-area. Claude counts images at approximately (width × height) ÷ 750 tokens, with images resized to fit the model's native resolution cap first (1568 pixels on the long edge for most models, ~1,568 tokens; newer large models allow more). Anthropic's docs price a 1000×1000 image at about 1,334 tokens — no tiling, the bill simply scales with pixel area, which makes it cheap for small images and expensive for large ones.
Google: per-tile flat rate. Gemini counts any image ≤384 pixels in both dimensions at a flat 258 tokens; larger images are cut into 768×768 tiles at 258 tokens each — so the same 1024×1024 screenshot tiles into 4 pieces and bills roughly 1,032 tokens. Audio and video are metered by the second: 32 tokens per second of audio, 263 tokens per second of video. That last number is the one people miss when they feed an hour of meeting recording to a "multimodal" model.
Same image, three bills: 765 tokens on one API, ~1,334 on another, ~1,032 on a third — and those tokens then price at whatever your model's per-token rate is (see inference cost). A tool that looks cheaper per text token can be materially more expensive at your actual workload, because your workload is photographs.
The 2026 reality check
Every flagship chatbot is now multimodal, which is exactly why the term has stopped being a differentiator on its own. Since GPT-4o's May 2024 launch set the pattern — one network, text/audio/image/video in, any combination out — "multimodal" migrated from a feature bullet to table stakes across the major assistants. The differentiators have moved downstream of the label: which modalities are native rather than pipelined, what the input limits are (image counts, video length, file sizes), and how the output side works — a model that sees video but can only reply in text is multimodal in the input column only.
The billing rules are a moving target, and the vendor doc is the only source. Tile sizes, base token counts and per-second rates have all changed between model generations — the constants in this entry are documentation snapshots, not physics. Two habits keep you safe: check the token-counting page for the exact model you will deploy before quoting costs to a client, and use the vendors' own calculators or count-token API endpoints to price a realistic sample of your workload — ten representative images and one representative minute of audio will expose a billing surprise that no comparison table will.
Watch for the pipeline hiding under a multimodal badge. Product-level "multimodal" claims can still be pipelines internally: a voice assistant that transcribes then generates loses exactly what native models capture — tone, overlapping speakers, background audio — and adds latency at each seam. The test is behavioral, not linguistic: interrupt it mid-sentence and see whether it hears you, or ask it about sarcasm in a voicemail.
Quick checklist
- Ask "native or pipelined?" before "multimodal or not?" A natively multimodal model hears tone and interruption; a transcribe-then-answer pipeline does not, no matter what the feature matrix says. Latency on spoken input is the tell.
- Price your real inputs, not your text. Run your typical image sizes and audio lengths through each vendor's published tokenization rule — flat 85-token low detail, 85+170 per 512px tile, width×height÷750, or 258 per 768×768 tile — and multiply by the per-token rate. The cheapest model per text token is often not the cheapest per photograph.
- Meter audio and video by the second. At ~263 tokens per second, an hour of video input is roughly 950k tokens before you have asked it a single question. Workloads that sounded small in prompts are large in seconds.
- Check the input limits you will actually hit. Max images per request, max pixels per image, max video length and file-size caps differ per model and per API tier; a tool that accepts images is not a tool that accepts your images.
- Re-check the token rules per model generation. Tile constants and per-second rates are documentation, not contracts — archive the page you priced against, the way you would a quote.
Where you'll hit it
Multimodality is the quiet variable in three purchases you will meet on this site: the ChatGPT vs Claude vs Gemini comparison (where the same screenshot bills three ways and the vision quirks differ per family), the best AI transcription tools (where native audio input is the entire product, and the per-second metering above is your real cost driver), and the best AI image generators (where output modality, not input, is what you are paying for). Related terms: AI token is the metered unit everything above converts into; context window is the budget those image tokens share with your text; inference cost is how the token bill becomes an invoice; and AI agent is where multimodal inputs meet automated loops. Full list on the AI tools glossary.
The bottom line
Multimodal means one model that reads text, images, audio and video natively — and in 2026 every flagship has that, so the word itself no longer decides anything. What decides purchases now is which modalities are native rather than pipelined, what the input limits are, and above all the billing: images are tokenized by vendor-specific rules (tiles, pixel area, or flat per-tile rates) and audio by the second, so the same workload can cost twice as much on one API as on another. Read the token-counting doc for the exact model you will deploy, price a realistic sample of your inputs, and treat "multimodal" on a landing page as the start of the question, not the answer.