Should you use OpenAI's embeddings? Probably not, and here's why.

Should you use OpenAI's embeddings? Probably not, and here's why.

When considering whether to use OpenAI's embeddings, there are a few key factors to take into account. The first is the language model (LLM) that will answer the question, and the second is the embeddings model that will pick the source material from the knowledge base.

Currently, OpenAI is the leading provider of LLMs, with GPT4 and GPT3.5 being the top choices. However, when it comes to embeddings, OpenAI's models are not necessarily superior. Benchmarks have shown that there are other models, such as the Instructor models (xl and large), that score higher than ada-002, OpenAI's embeddings model.

Of course, benchmarks alone don't determine the right choice for your specific needs. Factors like cost, performance, and speed also come into play. Additionally, you have to trust that OpenAI will continue to support and maintain the model you choose. If ada-002 is discontinued in the future, it could create problems if you've embedded millions of documents using that model. Similarly, if your usage increases and you find yourself embedding millions of queries per day, the cost of the OpenAI API could become a concern.

To address these uncertainties, I decided to conduct a comparison between different sets of embeddings. I randomized queries using both OpenAI's ada-002 embeddings and other options. Surprisingly, I couldn't tell the difference between the two sets. This led me to develop a recommended procedure for choosing embeddings:

1. Start with the lightest embedding model: Begin by trying the lightest model available. If it meets your requirements, there's no need to explore further.

2. Try a beefier model for comparison: If the lightest model doesn't work for you, try a more robust embeddings model. Conduct a blind comparison between the different options to determine which one performs better for your specific use case.

3. Consider OpenAI's ada-002 as a last resort: If you're already using a relatively large model like Instructor XL and still find that it doesn't meet your needs, then and only then should you consider a blind test against ada-002 from OpenAI. If you find that it outperforms other options, then it may be worth using.

In addition to the technical considerations of using OpenAI's embeddings, there are strategic implications for organizations that deploy large language models (LLMs) like GPT. Here are some managerial perspectives to keep in mind:

1. Utilization and Responsibility: With the power of LLMs comes the responsibility to ensure employees are educated on the risks and potential harms. Companies should clearly define and communicate their policy on LLM use, discouraging unofficial usage and concealing of such usage. Safeguards like writing system prompts that constrain responses and testing adversarial examples can also help prevent damage to the competitiveness of the organization.

2. Data Privacy: Using a cloud API for LLMs means potentially sending and storing sensitive data in the cloud. Opting for Azure OpenAI services can mitigate certain risks. Azure OpenAI ensures that data is not shared externally, even with OpenAI. By default, queries are stored for 30 days, but you can request Azure not to store your requests. If you choose to fine-tune a model, be sure to remove personally identifiable information and sensitive data from the training data to prevent any leakage in the output.

3. Reproducibility: LLMs, including OpenAI models, are non-deterministic, meaning that calling the same model with the same input can yield different outputs. This lack of reproducibility can pose challenges for auditing, testing, and even user experience. It's important to be aware of this limitation when using LLMs and consider potential workarounds or alternative approaches to ensure reproducibility when needed.

In conclusion, the decision to use OpenAI's embeddings should be carefully considered based on your specific requirements, weighing factors like cost, performance, and trust in the model's longevity. It may be worth exploring alternative options and conducting blind comparisons to find the best fit for your needs. Additionally, organizations deploying LLMs should be mindful of the strategic implications, such as educating employees on responsible usage, ensuring data privacy, and addressing challenges related to reproducibility. By taking these factors into account, you can make an informed decision and maximize the benefits of using large language models while mitigating potential risks.