LLM experimentation framework for a global software company
Overview
We recently collaborated with an international software company, undertaking R&D for how they might integrate Large Language Models (LLM) into their products. In particular there was an interesting hypothesis to test; would their proprietary data combined with an LLM provide better responses than an off-the-shelf LLM? If yes, this could provide a ‘data moat’ and a competitive advantage for the future.
This project was an exciting opportunity for us to delve into the different methods of LLM integration and model training.
The Challenge
Our client was seeking to understand how LLMs could enhance their existing software products. The key challenge lay in objectively assessing the LLM's capabilities to perform specific tasks efficiently and effectively. With thousands of data points to consider, this was no small feat. The project demanded an extensive research and testing framework, capable of providing reliable, measurable outcomes. We could then use this framework to run experiments and test hypothesis in a scientific way.
Our Solution
To address these challenges, we embarked on a comprehensive journey:
1. Data Preparation and Protection: We dedicated significant effort to preparing the client's data, ensuring it was in a format conducive for LLM testing, while also maintaining strict protocols to protect the data and prevent any exposure of training materials.
2. Experimentation and Testing: Our team undertook a rigorous testing regime, employing multiple LLMs and vector search methodologies (RAG). We also undertook fine-tuning of two LLMs using our client’s data. Tests of our custom solutions were conducted against a baseline of an out-of-the-box LLM to see which performed better.
3. Cost Analysis: Understanding the economic impact of different approaches was crucial. We conducted a thorough analysis of the cost implications associated with each method, helping the client make informed decisions about their investment in LLM technology.
4. Result Evaluation: With the data gathered from these experiments, the client now possesses comprehensive insights into the efficacy of each LLM approach in addressing their specific business challenges.
Key Technologies
The project leveraged several key technologies:
· Amazon Web Services (cloud compute)
· PostgreSQL with pgvector (database management and vector search)
· OpenAI and Azure OpenAI (sourcing and deploying LLMs)
· NodeJS (experiment and testing framework)
Summary
With a short burst of a two month R&D project, our client is now able to make data-driven decisions about the benefit of integrating LLMs into their software products, before making expensive and time-consuming product roadmap changes.
The experimentation framework is reusable, so that as new LLM capabilities emerge new tests can easily be run by the client’s development team.
For us, it was very interesting to try out every different method of adding proprietary data into an LLM and see which performs best in which scenarios. This is great knowledge for us to take into future projects.
