Here are some tips for using non-OpenAI Models with AutoGen.

​
Finding the right model

Every model will perform differently across the operations within your AutoGen setup, such as speaker selection, coding, function calling, content creation, etc. On the whole, larger models (13B+) perform better with following directions and providing more cohesive responses.

Content creation can be performed by most models.

Fine-tuned models can be great for very specific tasks, such as function calling and coding.

Specific tasks, such as speaker selection in a Group Chat scenario, that require very accurate outputs can be a challenge with most open source/weight models. The use of chain-of-thought and/or few-shot prompting can help guide the LLM to provide the output in the format you want.

​
Validating your program

Testing your AutoGen setup against a very large LLM, such as OpenAI’s ChatGPT or Anthropic’s Claude 3, can help validate your agent setup and configuration.

Once a setup is performing as you want, you can replace the models for your agents with non-OpenAI models and iteratively tweak system messages, prompts, and model selection.

​
Chat template

AutoGen utilises a set of chat messages for the conversation between AutoGen/user and LLMs. Each chat message has a role attribute that is typically user, assistant, or system.

A chat template is applied during inference and some chat templates implement rules about what roles can be used in specific sequences of messages.

For example, when using Mistral AI’s API the last chat message must have a role of user. In a Group Chat scenario the message used to select the next speaker will have a role of system by default and the API will throw an exception for this step. To overcome this the GroupChat’s constructor has a parameter called role_for_select_speaker_messages that can be used to change the role name to user.

groupchat = autogen.GroupChat(
    agents=[user_proxy, coder, pm],
    messages=[],
    max_round=12,
    # Role for select speaker message will be set to 'user' instead of 'system'
    role_for_select_speaker_messages='user',
)

If the chat template associated with a model you want to use doesn’t support the role sequence and names used in AutoGen you can modify the chat template. See an example of this on our vLLM page.

​
Discord

Join AutoGen’s #alt-models channel on their Discord and discuss non-OpenAI models and configurations.

Non-OpenAI ModelsAnthropic Claude