Custom AI Agent for QA Automation

About The Project

Appvance is an innovative company focused on developing AI-driven tools and GenAI to help companies improve their Software Quality. Instead of using teams of engineers to develop high-quality, automated QA tests, they aim to develop a system that intakes test cases written in natural language and outputs sets of test cases that are fast, repeatable, and reliable.

The Challenge

Because security and privacy is a key consideration for Appvance's customers, it was important to develop a system that was not only accurate but also entirely self-hosted and deployable within a private cloud. Some key requirements for the project included:

Self-Hosted LLMs

To maintain informational security, we needed to avoid using any external commercial api providers for LLM inference, and setup high-performance, self-managed inference api's for a range of different models.

Speed and Reliability

Because of the size of customer sites, it was imperative to design a system that was able to handle a massive amount of throughput in a reliable manner.

Starbourne Labs' Solution

Leveraging prior experience with self-hosted inference and deep expertise in developing highly scale-able event driven applications, we were able to develop a secure, high performance, reliable system capable of interpreting natural language test cases, and translate them into Appvance's proprietary engine for test case generation.

Key aspects of our approach included:

Event driven micro-services architecture

A robust architecture enables scalability under heavy workloads, enabling Appvance to manage high volumes of data.

Innovative Architecture

A specialized multi-modal retrieval framework geared towards contextual understanding of large scale web applications with complex user interfaces.

Privately Hosted LLMs

High-performance inference across a range of multi-modal LLMs, managed within a private cloud.

Implementation Process

The project was executed in two main phases, the first focusing on core technical research and R&D, evolving into a process of iterative improvement, stacking additional features on top of the core system.

1. Prototype Development

Objective

Create a functioning, deployable MVP of the application capable of handling the baseline scenarios required to prove out the viability of the product.

Outcome

A fully functional and deployable version of the application leveraging commercial inference api's

2. Iterative Development and Feature Development

Objectives

1. Move towards a fully self hosted stack of LLMs

2. Improve the product to handle more complex test cases on more dynamic web pages, at higher speed

Outcomes

1. Migrate towards and deploy privately hosted LLMs

2. Successfully interpret complex test cases, handle ambigous scenarios, and increase reliability.

Throughout both phases, Starbourne maintained a close relationship with Appvance, participating in daily standups, contributing towards architecture discussions and decisions, and building rapport with the Appvance engineering team.

Conclusion

Starbourne Lab's ongoing collaboration with Appvance exemplifies our core strength in leading complex, highly technical initiatives utilizing cutting edge technology.

Not only did this project showcase our ability to break down complex problems and produce experimental projects while also ensuring scalability and security, but also highlighted the value of fostering deep, collaborative partnerships.