🚨 The Silvia team just announced our latest engineering advancement. Every business wants access to the highest level of intelligence, but at the lowest cost possible. The rise of LLMs has made intelligence abundant, yet one of the hardest problems across startups and corporate America is predicting the compute cost associated with this intelligence. I have been dealing with this personally as we build @cfosilvia and the problem comes up in almost every conversation I have with CEOs, founders, and executives. Every business embraced AI about 18 months ago and things seemed great until the compute bills started to show up. The bills for internal compute usage were difficult to swallow, but things got outrageous if you had an AI product that allowed your users to consume compute without limits. I know this problem intimately because that is the situation that Silvia was in. Every question that was asked meant higher compute costs for our company. But we didn’t want to limit usage because users were getting genuine value out of the product. This challenge sent our team down a deep rabbit hole of cutting costs, while improving the experience for users. The second part was really important: we did not want to degrade the user experience by simply taking away access to the highest quality models. Thankfully, resource constraints breed innovation. We aren’t the biggest company, nor do we have the largest balance sheet, but we came up with a very novel solution that we are announcing today. The Silvia engineering team built a model router that cut costs by up to 29%, decreased latency, and improved the quality of answers for users. Trifecta! The way we do this is by reading the first 500 characters of a query and then predicting the level of effort that will be needed by a model to answer the query. The highest effort needs are routed to the most powerful models. The lowest effort needs are routed to different, better models for the query. A good example of this would be “what is the date?” You don’t need to use the latest Anthropic model to answer this query. In fact, sending a simple query like this to the most powerful model will make your compute costs increase and will actually increase the latency, which means a worse user experience for the Silvia user. By implementing the model router, the user gets a better experience and we get lower costs. Win-win. One of the interesting aspects of the implementation is that our model router runs on CPUs instead of GPUs. This allows us to read the query and predict the level of effort needed in less than 1 millisecond. This CPU implementation is why latency is not affected, nor is cost significantly increased by any potential additional GPU consumption. Another important point is that many of you have probably seen the news that OpenRouter is being purchased by Stripe for around $7 billion. This is a great outcome from what appears to be a very smart, capable team. Their model routing API is related (their product and our internal implementation both touch model routing), but you should think of OpenRouter as making it possible to do model routing for companies, while Silvia’s model router is a custom, intelligent system that specifically routes Silvia queries to the right model. They give access to the functionality of model routing to many companies, while our internal product does the real decision-making specific to our use case. Lastly, our implementation of a model router is a strategic bet that will allow us to become model-agnostic over time. We don’t care who created the different models, we just want to route a query to the model best positioned to answer. The large model labs will never allow their users to be model agnostic, but that would require the lab to potentially route a query to a competitor’s model. No bueno in their eyes. Instead, Silvia being an independent AI research lab gives us the power of being agnostic. We simply want the best experience for our users. Last week we announced that Silvia is now the most accurate AI tax product on the market, including beating OpenAI, Anthropic, Google, and xAI. Today we are announcing a custom, in-house model router that rivals the best technology anyone else has built. There will be many more engineering announcements to come. I truly believe we have assembled one of the best AI teams and we are currently the best AI research lab in finance. If you are interested in learning more about the technical details of the model router, you can read the engineering blog post here: https://www.cfosilvia.com/ai-lab/adaptive-model-routing Everyone wants the best intelligence and the lowest cost. Silvia just showed the world what is possible in this pursuit. I anticipate many other companies will build this custom solutions to achieve the same benefits.(Anthony Pompliano 🌪)