AG2 has been instrumental in helping Nexla build NOVA, our data engineer co-pilot. Nova empowers our customers to create data pipelines effortlessly using natural language in an engaging, interactive experience. Thanks to AG2’s human-in-the-loop capabilities, Nexla ensures strong governance, high accuracy, and a delightful user experience—making data engineering more intuitive and efficient than ever.Saket Saurabh, Co-founder & CEO, Nexla
In today’s fast-paced GenAI landscape, organizations are constantly searching for smarter, more efficient ways to manage and transform data. Nexla is a platform dedicated to the automation of data engineering, enabling users to get ready-to-use data with minimal hassle. Central to Nexla’s approach are Nexsets—data products that streamline the process of integrating, transforming, delivering, and monitoring data. Our mission is to make data ready-to-use for everyone, eliminating the complexities traditionally associated with data workflows.
With the introduction of Project NOVA, we’re leveraging AG2, a leading open-source agentic framework, to create powerful, production-grade agentic workflows that empower users to accomplish complex tasks with the simplicity of natural language.
The Challenge: Elevating Data Automation at Nexla
One of the primary challenges our customers face is the time and effort required to develop and manage complex data transformations. Even with a clear vision of the final data model, data transformation is a multi-step process that can be both time-consuming and technically demanding.The Solution: Harnessing AG2 for Project NOVA
AG2 provided us with the perfect foundation to build intelligent agents capable of handling complex data tasks far beyond basic conversational functions. This led to the creation of NOVA—Nexla Orchestrated Versatile Agents, a system designed to translate natural language into precise data transformations. NOVA simplifies data operations by breaking down complex tasks into manageable steps, enabling users to interact with their data intuitively and efficiently. By leveraging GenAI Agents built with AG2, we’ve also tackled the challenge of creating a common data model, allowing for seamless integration of diverse data sources to a unified data model. This innovation bridges the gap between user intent and data manipulation, paving the way for a unified and accessible data infrastructure across platforms and industries.Natural Language to Transforms
NOVA’s Natural Language to Transforms feature allows users to take a Nexset—a data product within Nexla—and describe, in plain language, the transformation they need. NOVA then automatically generates the required transforms, whether in Python or SQL, depending on the task. For example, a user could simply instruct, “Compute average speed and average duration for every origin-destination pair, hourly and by day of the week.” NOVA breaks down this request into a series of steps, applies the necessary transformations, and delivers the desired output. This allows users to focus on analyzing and utilizing the transformed data without getting bogged down in the complexities of coding.Natural Language to ELT (Extract, Load and Transform)
Next up is Natural Language to ELT, which allows users to build and execute ELT pipelines simply by providing natural language instructions. Users can input one or more Nexsets, a final data model, and an optional set of instructions, and NOVA does the rest. NOVA doesn’t just generate a static script—it allows users to interactively tweak the SQL logic as they go, ensuring that the final output is exactly what they need. This interactive, dynamic approach makes it easier than ever to handle complex ELT tasks, directly executing business logic on platforms like BigQuery or Snowflake and many other connectors that Nexla supports with precision and efficiency.Use Cases: Empowering Diverse Users
These features are designed with a broad range of users in mind:- Data Engineers: Automate routine data transformation tasks, freeing up time to focus on more strategic initiatives.
- Business Analysts: Generate insights quickly without the need for complex coding, enabling faster decision-making.
- Business Users: Interact with data naturally, transforming ideas into actionable queries without requiring deep technical expertise.
Why Nexla and AG2?
Nexla’s unique value proposition is its ability to integrate advanced AI-driven automation into existing workflows seamlessly. By building on the robust capabilities of AG2, we’ve ensured that NOVA is not only scalable but also reliable for production-grade applications. The flexibility and power of AG2 have been instrumental in allowing us to create agents that handle sophisticated tasks beyond basic interactions, making them an essential part of our platform’s evolution. Moreover, the scalability and reliability of AG2 have enabled us to deploy these features across large datasets and cloud platforms, ensuring consistent performance even under demanding workloads.Technical Deep Dive: The Architecture Behind NOVA
At the heart of NOVA’s success is a sophisticated agent architecture, powered by AG2:- Planner Agent: Analyzes user queries to determine the necessary steps for the ELT or transformation task, planning the workflow.
- Query Interpreter Agent: Translates the planner’s high-level steps into actionable SQL or Python for execution by the Data Transformer Agent.
- Data Transformer Agent: Generates the required SQL or Python logic, ensuring it aligns with the specific schema and data samples.
- Evaluator Agent: Reviews the generated logic for accuracy before execution, ensuring it meets the necessary requirements.
- API Agent: Manages interactions with databases and cloud services, executing the approved logic and creating Nexsets as needed.
NOVA Architecture Diagram
