OpenLIT an open source product that helps developers build and manage AI agents in production, effectively helping them improve accuracy. As a self-hosted solution, it enables developers to experiment with LLMs, manage and version prompts, securely manage API keys, and provide safeguards against prompt injection and jailbreak attempts. It also includes built-in OpenTelemetry-native observability and evaluation for the complete GenAI stack (LLMs, Agents, vector databases, and GPUs).

For more info, check out the OpenLIT Repo

Adding OpenLIT to an existing AutoGen (Now AG2) service

To get started, you’ll need to install the OpenLIT library

OpenLIT uses OpenTelemetry to automatically intrument the AI Agent app when it’s initialized meaning your agent observability data like execution traces and metrics will be tracked in just one line of code.

pip install ag2 openlit

Collecting ag2
  Downloading ag2-0.4-py3-none-any.whl.metadata (24 kB)
Collecting openlit
  Downloading openlit-1.32.4-py3-none-any.whl.metadata (22 kB)
Collecting diskcache (from ag2)
  Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Collecting docker (from ag2)
  Downloading docker-7.1.0-py3-none-any.whl.metadata (3.9 kB)
Collecting flaml (from ag2)
  Downloading FLAML-2.3.2-py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: openai>=1.3 in /usr/local/lib/python3.10/dist-packages (from ag2) (1.54.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from ag2) (24.2)
Requirement already satisfied: pydantic!=2.6.0,<3,>=1.10 in /usr/local/lib/python3.10/dist-packages (from ag2) (2.9.2)
Collecting python-dotenv (from ag2)
  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from ag2) (2.5.0)
Collecting tiktoken (from ag2)
  Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: numpy<2.0.0,>=1.24.0 in /usr/local/lib/python3.10/dist-packages (from ag2) (1.26.4)
Collecting anthropic<0.22.0,>=0.21.0 (from openlit)
  Downloading anthropic-0.21.3-py3-none-any.whl.metadata (17 kB)
Collecting boto3<2.0.0,>=1.34.0 (from openlit)
  Downloading boto3-1.35.69-py3-none-any.whl.metadata (6.7 kB)
Collecting botocore<2.0.0,>=1.34.0 (from openlit)
  Downloading botocore-1.35.69-py3-none-any.whl.metadata (5.7 kB)
Requirement already satisfied: opentelemetry-api<2.0.0,>=1.27.0 in /usr/local/lib/python3.10/dist-packages (from openlit) (1.28.2)
Collecting opentelemetry-exporter-otlp<2.0.0,>=1.27.0 (from openlit)
  Downloading opentelemetry_exporter_otlp-1.28.2-py3-none-any.whl.metadata (2.3 kB)
Collecting opentelemetry-instrumentation<0.49,>=0.48b0 (from openlit)
  Downloading opentelemetry_instrumentation-0.48b0-py3-none-any.whl.metadata (6.1 kB)
Requirement already satisfied: opentelemetry-sdk<2.0.0,>=1.27.0 in /usr/local/lib/python3.10/dist-packages (from openlit) (1.28.2)
Requirement already satisfied: requests<3.0.0,>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from openlit) (2.32.3)
Collecting schedule<2.0.0,>=1.2.2 (from openlit)
  Downloading schedule-1.2.2-py3-none-any.whl.metadata (3.9 kB)
Collecting tiktoken (from ag2)
  Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting xmltodict<0.14.0,>=0.13.0 (from openlit)
  Downloading xmltodict-0.13.0-py2.py3-none-any.whl.metadata (7.7 kB)
Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (3.7.1)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (0.27.2)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (1.3.1)
Requirement already satisfied: tokenizers>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (0.20.3)
Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (4.12.2)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3<2.0.0,>=1.34.0->openlit)
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3<2.0.0,>=1.34.0->openlit)
  Downloading s3transfer-0.10.4-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.10/dist-packages (from botocore<2.0.0,>=1.34.0->openlit) (2.8.2)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/local/lib/python3.10/dist-packages (from botocore<2.0.0,>=1.34.0->openlit) (2.2.3)
Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.3->ag2) (0.7.1)
Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai>=1.3->ag2) (4.66.6)
Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api<2.0.0,>=1.27.0->openlit) (1.2.15)
Requirement already satisfied: importlib-metadata<=8.5.0,>=6.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api<2.0.0,>=1.27.0->openlit) (8.5.0)
Collecting opentelemetry-exporter-otlp-proto-grpc==1.28.2 (from opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
  Downloading opentelemetry_exporter_otlp_proto_grpc-1.28.2-py3-none-any.whl.metadata (2.2 kB)
Collecting opentelemetry-exporter-otlp-proto-http==1.28.2 (from opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
  Downloading opentelemetry_exporter_otlp_proto_http-1.28.2-py3-none-any.whl.metadata (2.2 kB)
Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.66.0)
Requirement already satisfied: grpcio<2.0.0,>=1.63.2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.68.0)
Collecting opentelemetry-exporter-otlp-proto-common==1.28.2 (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
  Downloading opentelemetry_exporter_otlp_proto_common-1.28.2-py3-none-any.whl.metadata (1.8 kB)
Collecting opentelemetry-proto==1.28.2 (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
  Downloading opentelemetry_proto-1.28.2-py3-none-any.whl.metadata (2.3 kB)
Collecting protobuf<6.0,>=5.0 (from opentelemetry-proto==1.28.2->opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
  Downloading protobuf-5.28.3-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Requirement already satisfied: setuptools>=16.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation<0.49,>=0.48b0->openlit) (75.1.0)
Requirement already satisfied: wrapt<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation<0.49,>=0.48b0->openlit) (1.16.0)
Requirement already satisfied: opentelemetry-semantic-conventions==0.49b2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-sdk<2.0.0,>=1.27.0->openlit) (0.49b2)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=2.6.0,<3,>=1.10->ag2) (0.7.0)
Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic!=2.6.0,<3,>=1.10->ag2) (2.23.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.26.0->openlit) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.26.0->openlit) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.26.0->openlit) (2024.8.30)
Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->ag2) (2024.9.11)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->anthropic<0.22.0,>=0.21.0->openlit) (1.2.2)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->anthropic<0.22.0,>=0.21.0->openlit) (1.0.7)
Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->anthropic<0.22.0,>=0.21.0->openlit) (0.14.0)
Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<=8.5.0,>=6.0->opentelemetry-api<2.0.0,>=1.27.0->openlit) (3.21.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<2.0.0,>=1.34.0->openlit) (1.16.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (0.26.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (3.16.1)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (2024.10.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (6.0.2)
Downloading ag2-0.4-py3-none-any.whl (366 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 366.1/366.1 kB 8.1 MB/s eta 0:00:00
Downloading openlit-1.32.4-py3-none-any.whl (229 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.1/229.1 kB 15.1 MB/s eta 0:00:00
Downloading anthropic-0.21.3-py3-none-any.whl (851 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 851.6/851.6 kB 8.5 MB/s eta 0:00:00
Downloading boto3-1.35.69-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 9.5 MB/s eta 0:00:00
Downloading botocore-1.35.69-py3-none-any.whl (13.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.0/13.0 MB 57.1 MB/s eta 0:00:00
Downloading opentelemetry_exporter_otlp-1.28.2-py3-none-any.whl (7.0 kB)
Downloading opentelemetry_exporter_otlp_proto_grpc-1.28.2-py3-none-any.whl (18 kB)
Downloading opentelemetry_exporter_otlp_proto_http-1.28.2-py3-none-any.whl (17 kB)
Downloading opentelemetry_exporter_otlp_proto_common-1.28.2-py3-none-any.whl (18 kB)
Downloading opentelemetry_proto-1.28.2-py3-none-any.whl (55 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 4.3 MB/s eta 0:00:00
Downloading opentelemetry_instrumentation-0.48b0-py3-none-any.whl (29 kB)
Downloading schedule-1.2.2-py3-none-any.whl (12 kB)
Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 42.3 MB/s eta 0:00:00
Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)
Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 3.0 MB/s eta 0:00:00
Downloading docker-7.1.0-py3-none-any.whl (147 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 147.8/147.8 kB 8.6 MB/s eta 0:00:00
Downloading FLAML-2.3.2-py3-none-any.whl (313 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 313.9/313.9 kB 19.8 MB/s eta 0:00:00
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.2/83.2 kB 6.9 MB/s eta 0:00:00
Downloading protobuf-5.28.3-cp38-abi3-manylinux2014_x86_64.whl (316 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.6/316.6 kB 21.0 MB/s eta 0:00:00
Installing collected packages: xmltodict, schedule, python-dotenv, protobuf, jmespath, flaml, diskcache, tiktoken, opentelemetry-proto, docker, botocore, s3transfer, opentelemetry-instrumentation, opentelemetry-exporter-otlp-proto-common, boto3, anthropic, ag2, opentelemetry-exporter-otlp-proto-http, opentelemetry-exporter-otlp-proto-grpc, opentelemetry-exporter-otlp, openlit
  Attempting uninstall: protobuf
    Found existing installation: protobuf 4.25.5
    Uninstalling protobuf-4.25.5:
      Successfully uninstalled protobuf-4.25.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.17.1 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 5.28.3 which is incompatible.
tensorflow-metadata 1.13.1 requires protobuf<5,>=3.20.3, but you have protobuf 5.28.3 which is incompatible.
Successfully installed ag2-0.4 anthropic-0.21.3 boto3-1.35.69 botocore-1.35.69 diskcache-5.6.3 docker-7.1.0 flaml-2.3.2 jmespath-1.0.1 openlit-1.32.4 opentelemetry-exporter-otlp-1.28.2 opentelemetry-exporter-otlp-proto-common-1.28.2 opentelemetry-exporter-otlp-proto-grpc-1.28.2 opentelemetry-exporter-otlp-proto-http-1.28.2 opentelemetry-instrumentation-0.48b0 opentelemetry-proto-1.28.2 protobuf-5.28.3 python-dotenv-1.0.1 s3transfer-0.10.4 schedule-1.2.2 tiktoken-0.7.0 xmltodict-0.13.0

import openlit

from autogen import AssistantAgent, UserProxyAgent

openlit.init()

OpenLIT will now start automatically tracking

LLM prompts and completions
Token usage and costs
Agent names and actions
Tool usage
Errors

Lets look at a simple chat example

import openlit

openlit.init()

import os

llm_config = {"model": "gpt-4", "api_key": os.environ["OPENAI_API_KEY"]}
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", code_execution_config=False)

# Start the chat
user_proxy.initiate_chat(
    assistant,
    message="Tell me a joke about NVDA and TESLA stock prices.",
)

user_proxy (to assistant):

Tell me a joke about NVDA and TESLA stock prices.

--------------------------------------------------------------------------------
assistant (to user_proxy):

Why don't NVDA and TESLA stock prices ever get coffee together?

Because whenever they heat up, they always take a steep drop before they can cool down! 

I hope this brings a smile to your face. Investing in stocks can be a rollercoaster sometimes. Please note that this is humor and doesn't reflect the actual dynamics of these companies' stock prices. TERMINATE.

--------------------------------------------------------------------------------
Replying as user_proxy. Provide feedback to assistant. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: hi
user_proxy (to assistant):

hi

--------------------------------------------------------------------------------
{
    "name": "openai.chat.completions",
    "context": {
        "trace_id": "0x35ae5952626492e432ae25af5bf92daa",
        "span_id": "0x44103383aa51d1b1",
        "trace_state": "[]"
    },
    "kind": "SpanKind.CLIENT",
    "parent_id": null,
    "start_time": "2024-11-21T01:53:47.597241Z",
    "end_time": "2024-11-21T01:53:48.506758Z",
    "status": {
        "status_code": "OK"
    },
    "attributes": {
        "telemetry.sdk.name": "openlit",
        "gen_ai.system": "openai",
        "gen_ai.operation.name": "chat",
        "gen_ai.endpoint": "openai.chat.completions",
        "gen_ai.response.id": "chatcmpl-AVqg7QSYB8CpEN1I5PYT1laBcds9S",
        "gen_ai.environment": "default",
        "gen_ai.application_name": "default",
        "gen_ai.request.model": "gpt-4",
        "gen_ai.request.top_p": 1.0,
        "gen_ai.request.max_tokens": -1,
        "gen_ai.request.user": "",
        "gen_ai.request.temperature": 1.0,
        "gen_ai.request.presence_penalty": 0.0,
        "gen_ai.request.frequency_penalty": 0.0,
        "gen_ai.request.seed": "",
        "gen_ai.request.is_stream": false,
        "gen_ai.usage.input_tokens": 580,
        "gen_ai.usage.output_tokens": 9,
        "gen_ai.usage.total_tokens": 589,
        "gen_ai.response.finish_reasons": [
            "stop"
        ],
        "gen_ai.usage.cost": 0.017939999999999998
    },
    "events": [
        {
            "name": "gen_ai.content.prompt",
            "timestamp": "2024-11-21T01:53:48.506257Z",
            "attributes": {
                "gen_ai.prompt": "system: You are a helpful AI assistant.\nSolve tasks using your coding and language skills.\nIn the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.\n    1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.\n    2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\nReply \"TERMINATE\" in the end when everything is done.\n    \nuser: Tell me a joke about NVDA and TESLA stock prices.\nassistant: Why don't NVDA and TESLA stock prices ever get coffee together?\n\nBecause whenever they heat up, they always take a steep drop before they can cool down! \n\nI hope this brings a smile to your face. Investing in stocks can be a rollercoaster sometimes. Please note that this is humor and doesn't reflect the actual dynamics of these companies' stock prices. TERMINATE.\nuser: hi"
            }
        },
        {
            "name": "gen_ai.content.completion",
            "timestamp": "2024-11-21T01:53:48.506314Z",
            "attributes": {
                "gen_ai.completion": "Hello! How can I assist you today?"
            }
        }
    ],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "openlit",
            "telemetry.sdk.version": "1.28.2",
            "service.name": "default",
            "deployment.environment": "default"
        },
        "schema_url": ""
    }
}
assistant (to user_proxy):

Hello! How can I assist you today?

--------------------------------------------------------------------------------
Replying as user_proxy. Provide feedback to assistant. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: exit

ChatResult(chat_id=None, chat_history=[{'content': 'Tell me a joke about NVDA and TESLA stock prices.', 'role': 'assistant', 'name': 'user_proxy'}, {'content': "Why don't NVDA and TESLA stock prices ever get coffee together?\n\nBecause whenever they heat up, they always take a steep drop before they can cool down! \n\nI hope this brings a smile to your face. Investing in stocks can be a rollercoaster sometimes. Please note that this is humor and doesn't reflect the actual dynamics of these companies' stock prices. TERMINATE.", 'role': 'user', 'name': 'assistant'}, {'content': 'hi', 'role': 'assistant', 'name': 'user_proxy'}, {'content': 'Hello! How can I assist you today?', 'role': 'user', 'name': 'assistant'}], summary='Hello! How can I assist you today?', cost={'usage_including_cached_inference': {'total_cost': 0.03731999999999999, 'gpt-4-0613': {'cost': 0.03731999999999999, 'prompt_tokens': 1068, 'completion_tokens': 88, 'total_tokens': 1156}}, 'usage_excluding_cached_inference': {'total_cost': 0.017939999999999998, 'gpt-4-0613': {'cost': 0.017939999999999998, 'prompt_tokens': 580, 'completion_tokens': 9, 'total_tokens': 589}}}, human_input=['hi', 'exit'])

Sending Traces and metrics to OpenLIT

By default, OpenLIT generates OpenTelemetry traces and metrics that are logged to your console. To set up a detailed monitoring environment, this guide outlines how to deploy OpenLIT and direct all traces and metrics there. You also have the flexibility to send the telemetry data to any OpenTelemetry-compatible endpoint, such as Grafana, Jaeger, or DataDog.

Deploy OpenLIT Stack

Clone the OpenLIT Repository

Open your terminal or command line and execute:
```
git clone git@github.com:openlit/openlit.git
```
Host it Yourself with Docker

Deploy and start OpenLIT using the command:
```
docker compose up -d
```

For instructions on installing in Kubernetes using Helm, refer to the Kubernetes Helm installation guide.

Configure the telemetry data destination as follows:

Purpose	Parameter/Environment Variable	For Sending to OpenLIT
Send data to an HTTP OTLP endpoint	`otlp_endpoint` or `OTEL_EXPORTER_OTLP_ENDPOINT`	`"http://127.0.0.1:4318"`
Authenticate telemetry backends	`otlp_headers` or `OTEL_EXPORTER_OTLP_HEADERS`	Not required by default

💡 Info: If the otlp_endpoint or OTEL_EXPORTER_OTLP_ENDPOINT is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.

Visualize and Optimize!

With the Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your AI application’s performance, behavior, and identify areas of improvement.

Just head over to OpenLIT at 127.0.0.1:3000 on your browser to start exploring. You can login using the default credentials - Email: user@openlit.io - Password: openlituser

Auto Generated Agent Chat: GPTAssistant with Code Interpreter Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent

On this page

Adding OpenLIT to an existing AutoGen (Now AG2) service
Lets look at a simple chat example
Sending Traces and metrics to OpenLIT
Deploy OpenLIT Stack
Visualize and Optimize!

Examples

Agent Observability with OpenLIT

Adding OpenLIT to an existing AutoGen (Now AG2) service

Lets look at a simple chat example

Sending Traces and metrics to OpenLIT

Deploy OpenLIT Stack

Visualize and Optimize!

Examples

​Adding OpenLIT to an existing AutoGen (Now AG2) service

​Lets look at a simple chat example

​Sending Traces and metrics to OpenLIT

​Deploy OpenLIT Stack

​Visualize and Optimize!

Adding OpenLIT to an existing AutoGen (Now AG2) service

Lets look at a simple chat example

Sending Traces and metrics to OpenLIT

Deploy OpenLIT Stack

Visualize and Optimize!