WebSurferAgent

WebSurfer Agent

In browser-use tool and crawl4ai tool notebooks, we demonstrated how to create Agents with basic web surfing capabilities.

Now, we’re taking it a step further with WebSurferAgent—a powerful agent equipped with built-in web surfing tools right out of the box!

WebSurferAgent with browser-use tool

Warning: Browser Use requires Python 3.11 or higher.

Installation

To get started with the browser-use integration in AG2, follow these steps:

1. Install AG2 with the browser-use extra:

   pip install -U ag2[openai,browser-use]
   

   Note: If you have been using autogen or ag2, all you need to do is upgrade it using:

   pip install -U autogen[openai,browser-use]
   

   or

   pip install -U ag2[openai,browser-use]
   

   as autogen, and ag2 are aliases for the same PyPI package.

2. Set up Playwright:

   # Installs Playwright and browsers for all OS
   playwright install
   # Additional command, mandatory for Linux only
   playwright install-deps
   

3. For running the code in Jupyter, use nest_asyncio to allow nested event loops. bash pip install nest_asyncio

You’re all set! Now you can start using browsing features in AG2.

Imports

import os

import nest_asyncio

from autogen.agentchat import UserProxyAgent
from autogen.agents.experimental import WebSurferAgent

nest_asyncio.apply()

browser-use WebSurferAgent

Note: Browser Use supports the following models: Supported Models

We had great experience with OpenAI, Anthropic, and Gemini. However, DeepSeek and Ollama haven’t performed as well.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

There are two ways to start a chat session which is using only one agent with LLM configuration.

Recommended: Using the run Method

The new run method simplifies the process by eliminating the need for manual UserProxyAgent creation.

• ✅ Easier setup – No need to manually register tools

# The `web_tool="browser_use"` tells the agent to use the `BrowserUseTool` to surf the web.
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="browser_use")

run_response = websurfer.run(
    message="Get info from https://docs.ag2.ai/docs/Home",
    tools=websurfer.tools,
    max_turns=2,
    user_input=False,
)
run_response.process()

Manual Setup: Using initiate_chat Method

This method requires manually creating a UserProxyAgent and registering tools for execution.

• ⚠️ More setup required
• ⚠️ Must manually register tools

websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="browser_use")
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
# WebSurferAgent has a list of tools which are registered for LLM
# We need to register the tools for execution with the UserProxyAgent
for tool in websurfer.tools:
    tool.register_for_execution(user_proxy)

user_proxy.initiate_chat(
    recipient=websurfer,
    message="Get info from https://docs.ag2.ai/docs/Home",
    max_turns=2,
)

WebSurferAgent with crawl4ai tool

Installation

To get started with the crawl4ai integration in AG2, follow these steps:

1. Install AG2 with the crawl4ai extra:

   pip install -U ag2[openai,crawl4ai]
   

   Note: If you have been using autogen or ag2, all you need to do is upgrade it using:

   pip install -U autogen[openai,crawl4ai]
   

   or

   pip install -U ag2[openai,crawl4ai]
   

   as autogen, and ag2 are aliases for the same PyPI package.

2. Set up Playwright:

   # Installs Playwright and browsers for all OS
   playwright install
   # Additional command, mandatory for Linux only
   playwright install-deps
   

3. For running the code in Jupyter, use nest_asyncio to allow nested event loops. bash pip install nest_asyncio

You’re all set! Now you can start using browsing features in AG2.

Imports

import os

import nest_asyncio

from autogen.agentchat import UserProxyAgent
from autogen.agents.experimental import WebSurferAgent

nest_asyncio.apply()

Crawl4AI WebSurferAgent

Note: Crawl4AI is built on top of LiteLLM and supports the same models as LiteLLM.

We had great experience with OpenAI, Anthropic, Gemini and Ollama. However, as of this writing, DeepSeek is encountering some issues.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

There are two ways to start a chat session which is using only one agent with LLM configuration.

Recommended: Using the run Method

The new run method simplifies the process by eliminating the need for manual UserProxyAgent creation.

• ✅ Easier setup – No need to manually register tools

# `web_tool` parameter must be set to `crawl4ai` in order for the `Crawl4AITool` to be used.
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="crawl4ai")

run_response = websurfer.run(
    message="Get info from https://docs.ag2.ai/docs/Home",
    tools=websurfer.tools,
    max_turns=2,
    user_input=False,
)
run_response.process()

Manual Setup: Using initiate_chat Method

This method requires manually creating a UserProxyAgent and registering tools for execution.

• ⚠️ More setup required
• ⚠️ Must manually register tools

user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="crawl4ai")

# WebSurferAgent has a list of tools which are registered for LLM
# We need to register the tools for execution with the UserProxyAgent
for tool in websurfer.tools:
    tool.register_for_execution(user_proxy)

user_proxy.initiate_chat(
    recipient=websurfer,
    message="Get info from https://docs.ag2.ai/docs/Home",
    max_turns=2,
)

Edit this page