Crawl4AI

Crawl4aiTools exposes a crawl function that fetches one or more URLs with Crawl4ai’s AsyncWebCrawler and returns extracted markdown content, optionally filtered by a search query using BM25.

Crawl4aiTools enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

Prerequisites

The following example requires the crawl4ai library, its browser runtime, and an OPENAI_API_KEY for the Agent’s default model.

uv pip install -U crawl4ai openai
crawl4ai-setup
crawl4ai-doctor
export OPENAI_API_KEY="your-openai-api-key"

Example

The following agent will scrape the content from the https://github.com/agno-agi/agno webpage:

from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")

Toolkit Params

Parameter	Type	Default	Description
max_length	Optional[int]	5000	Specifies the maximum length of the text from the webpage to be returned.
timeout	int	60	Timeout in seconds for web crawling operations.
use_pruning	bool	False	Enable content pruning to remove less relevant content.
pruning_threshold	float	0.48	Threshold for content pruning relevance scoring.
bm25_threshold	float	1.0	BM25 scoring threshold for content relevance.
headless	bool	True	Run browser in headless mode.
wait_until	str	"domcontentloaded"	Browser wait condition before crawling (e.g., “domcontentloaded”, “load”, “networkidle”).
proxy_config	Optional[Dict[str, Any]]	None	Proxy configuration passed to the browser.
enable_crawl	bool	True	Enable the web crawling functionality.
all	bool	False	Enable all available functions. When True, all enable flags are ignored.

Toolkit Functions

Function	Description
crawl	Crawls one or more URLs and returns the extracted content, optionally filtered by a search_query for relevance-based extraction.

Developer Resources

• Tools