> ## Documentation Index > Fetch the complete documentation index at: https://docs.ag2.ai/llms.txt > Use this file to discover all available pages before exploring further. # autogen.agentchat.contrib.capabilities.vision_capability.VisionCapability

VisionCapability

```python theme={null} VisionCapability( lmm_config: dict[str, Any], description_prompt: str | None = 'Write a detailed caption for this image. Pay special attention to any details that might be useful or relevant to the ongoing conversation.', custom_caption_func: Callable = None ) ``` We can add vision capability to regular ConversableAgent, even if the agent does not have the multimodal capability, such as GPT-3.5-turbo agent, Llama, Orca, or Mistral agents. This vision capability will invoke a LMM client to describe the image (captioning) before sending the information to the agent's actual client.
The vision capability will hook to the ConversableAgent's `process_last_received_message`.
Some technical details:
When the agent (who has the vision capability) received an message, it will:
1. \_process\_received\_message:
a. \_append\_oai\_message 2\. generate\_reply: if the agent is a MultimodalAgent, it will also use the image tag.
a. hook process\_last\_received\_message (NOTE: this is where the vision capability will be hooked to.) b. hook process\_all\_messages\_before\_reply 3\. send:
a. hook process\_message\_before\_send b. \_append\_oai\_message Initializes a new instance, setting up the configuration for interacting with a Language Multimodal (LMM) client and specifying optional parameters for image description and captioning.
Parameters: | Name | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `lmm_config` | **Type:** dict\[str, typing.Any] | | `description_prompt` | **Type:** str \| None

**Default:** 'Write a detailed caption for this image. Pay special attention to any details that might be useful or relevant to the ongoing conversation.' | | `custom_caption_func` | **Type:** Callable

**Default:** None | ### Instance Methods #### add\_to\_agent ```python theme={null} add_to_agent(self, agent: ConversableAgent) -> None ``` Adds a particular capability to the given agent. Must be implemented by the capability subclass.
An implementation will typically call agent.register\_hook() one or more times. See teachability.py as an example. Parameters: | Name | Description | | ------- | -------------------------------------------------------------------------- | | `agent` | **Type:** [ConversableAgent](/docs/api-reference/autogen/ConversableAgent) |
#### process\_last\_received\_message ```python theme={null} process_last_received_message(self, content: list[dict[str, Any]] | str) -> str ``` Processes the last received message content by normalizing and augmenting it with descriptions of any included images. The function supports input content as either a string or a list of dictionaries, where each dictionary represents a content item (e.g., text, image). If the content contains image URLs, it fetches the image data, generates a caption for each image, and inserts the caption into the augmented content.
The function aims to transform the content into a format compatible with GPT-4V multimodal inputs, specifically by formatting strings into PIL-compatible images if needed and appending text descriptions for images. This allows for a more accessible presentation of the content, especially in contexts where images cannot be displayed directly.
Parameters: | Name | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `content` | The last received message content, which can be a plain text string or a list of dictionaries representing different types of content items (e.g., text, image\_url).

**Type:** list\[dict\[str, typing.Any]] \| str | Returns: | Type | Description | | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | str | str: The augmented message content Raises: AssertionError: If an item in the content list is not a dictionary. Examples: Assuming `self._get_image_caption(img_data)` returns "A beautiful sunset over the mountains" for the image. - Input as String: content = "Check out this cool photo!" Output: "Check out this cool photo!" (Content is a string without an image, remains unchanged.) - Input as String, with image location: content = "What's weather in this cool photo: `<img http://example.com/photo.jpg>`" Output: "What's weather in this cool photo: `<img http://example.com/photo.jpg>` in case you can not see, the caption of this image is: A beautiful sunset over the mountains " (Caption added after the image) - Input as List with Text Only: content = `[\{"type": "text", "text": "Here's an interesting fact."}]` Output: "Here's an interesting fact." (No images in the content, it remains unchanged.) - Input as List with Image URL: `python content = [ \{"type": "text", "text": "What's weather in this cool photo:"}, \{"type": "image_url", "image_url": \{"url": "http://example.com/photo.jpg"}}, ] ` Output: "What's weather in this cool photo: `<img http://example.com/photo.jpg>` in case you can not see, the caption of this image is: A beautiful sunset over the mountains " (Caption added after the image) |