Cerebras
Cerebras has developed the world’s largest and fastest AI processor, the Wafer-Scale Engine-3 (WSE-3). Notably, the CS-3 system can run large language models like Llama-3.1-8B and Llama-3.1-70B at extremely fast speeds, making it an ideal platform for demanding AI workloads.
While it’s technically possible to adapt AutoGen to work with Cerebras’
API by updating the base_url
, this approach may not fully account for
minor differences in parameter support. Using this library will also
allow for tracking of the API costs based on actual token usage.
For more information about Cerebras Cloud, visit cloud.cerebras.ai. Their API reference is available at inference-docs.cerebras.ai.
Requirements
To use Cerebras with AutoGen, install the autogen[cerebras]
package.
Getting Started
Cerebras provides a number of models to use. See the list of models here.
See the sample OAI_CONFIG_LIST
below showing how the Cerebras client
class is used by specifying the api_type
as cerebras
.
Credentials
Get an API Key from cloud.cerebras.ai and add it to your environment variables:
API parameters
The following parameters can be added to your config for the Cerebras API. See this link for further information on them and their default values.
- max_tokens (null, integer >= 0)
- seed (number)
- stream (True or False)
- temperature (number 0..1.5)
- top_p (number)
Example:
Two-Agent Coding Example
In this example, we run a two-agent chat with an AssistantAgent (primarily a coding agent) to generate code to count the number of prime numbers between 1 and 10,000 and then it will be executed.
We’ll use Meta’s LLama-3.1-70B model which is suitable for coding.
Importantly, we have tweaked the system message so that the model doesn’t return the termination keyword, which we’ve changed to FINISH, with the code block.
Tool Call Example
In this example, instead of writing code, we will show how Meta’s Llama-3.1-70B model can perform parallel tool calling, where it recommends calling more than one tool at a time.
We’ll use a simple travel agent assistant program where we have a couple of tools for weather and currency conversion.
We start by importing libraries and setting up our configuration to use
Llama-3.1-70B and the cerebras
client class.
Create our two agents.
Create the two functions, annotating them so that those descriptions can be passed through to the LLM.
We associate them with the agents using register_for_execution
for the
user_proxy so it can execute the function and register_for_llm
for the
chatbot (powered by the LLM) so it can pass the function definitions to
the LLM.
We pass through our customer’s message and run the chat.
Finally, we ask the LLM to summarise the chat and print that out.
We can see that the Cerebras Wafer-Scale Engine-3 (WSE-3) completed the query in 74ms – faster than the blink of an eye!