Multimodal inputs

Feed images, audio, video, and PDFs into any labeling or extraction agent.

Every labeler on the other pages takes text. To label other modalities, change the input argument and the model. The schema and the output_schema pattern stay the same.

from typing import Literal

from agno.agent import Agent
from agno.media import Image
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Classification(BaseModel):
    label: Literal["wildlife", "landscape", "sports", "architecture", "other"] = Field(
        ..., description="The primary scene type of the image"
    )


agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions="You classify images by scene type.",
    output_schema=Classification,
)

url = "https://storage.googleapis.com/generativeai-downloads/images/generated_elephants_giraffes_zebras_sunset.jpg"
result = agent.run("Classify this image.", images=[Image(url=url)]).content
# Classification(label='wildlife')

Input argument per modality

Modality	Import	Argument
Image	from agno.media import Image	images=[Image(url=...)]
Audio	from agno.media import Audio	audio=[Audio(content=...)]
Video	from agno.media import Video	videos=[Video(content=..., format="mp4")]
PDF	from agno.media import File	files=[File(url=...)]

All four classes accept url, filepath, or raw bytes via content. The cookbook examples fetch audio and video bytes first. Agno installs httpx.

import httpx

from agno.agent import Agent
from agno.media import Audio
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Transcript(BaseModel):
    text: str = Field(..., description="Verbatim transcript of all spoken audio")


transcription_agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions="Transcribe all spoken audio. Return only the transcript.",
    output_schema=Transcript,
)

url = "https://agno-public.s3.us-east-1.amazonaws.com/demo_data/QA-01.mp3"
response = httpx.get(url, timeout=30.0)
response.raise_for_status()

result = transcription_agent.run(
    "Transcribe this audio.",
    audio=[Audio(content=response.content)],
).content
# Transcript(text='...')

Bounding boxes

For region detection, return normalized coordinates so the result is resolution-independent.

from pydantic import BaseModel, Field


class BoundingBox(BaseModel):
    label: str = Field(..., description="What the box contains")
    x: float = Field(..., ge=0.0, le=1.0, description="Top-left x in [0, 1]")
    y: float = Field(..., ge=0.0, le=1.0, description="Top-left y in [0, 1]")
    width: float = Field(..., ge=0.0, le=1.0, description="Width in [0, 1]")
    height: float = Field(..., ge=0.0, le=1.0, description="Height in [0, 1]")

The per-field description on x, y, width, and height is load-bearing. Without it, and without the [0, 1] convention spelled out in the instructions, models return degenerate boxes (all-zero or whole-image). Spell out the coordinate system in both places.

The bounding boxes cookbook has runnable single-object, multi-object, and per-box confidence variants.

Transcription and diarization

Audio extraction covers transcription, speaker diarization, and timestamped segments. Each is a schema change over the same API.

Output	Schema shape
Flat transcript	{ text: str }
Speaker turns	{ turns: List[{ speaker, text }] }
Timestamped segments	{ segments: List[{ start_seconds, end_seconds, text }] }

Model choice

The cookbook uses gemini-3.5-flash across its text, image, audio, video, and PDF recipes. A replacement model must support the input modality and structured output.

Next steps

Task	Guide
Define the output schema	Data extraction
Assign labels to media	Classification
Review media labels	Quality pipeline

Developer Resources

• Image extraction cookbook
• Audio transcription cookbook
• Video extraction cookbook
• Document extraction cookbook
• Multimodal agents