Miami-based AI startup Subquadratic has emerged from stealth mode with a bold assertion: it has overcome a mathematical bottleneck that has hindered the development of large language models (LLMs) for nearly a decade. Initial skepticism surrounded their claims, especially since the company provided limited evidence to substantiate its breakthrough. However, recent independent evaluations appear to lend credibility to their assertions, suggesting that their new model, SubQ, may indeed be a significant advancement in the field.
Subquadratic asserts that SubQ is not only faster and more cost-effective than existing models but also utilizes far less energy. The company claims that SubQ can process up to ten times the amount of text simultaneously compared to most other models, enabling it to efficiently handle data-intensive tasks such as analyzing extensive document sets or entire codebases. Moreover, SubQ reportedly matches the performance of leading models from tech giants like Google DeepMind, OpenAI, and Anthropic in critical areas, including coding tasks. Despite initial misgivings, Subquadratic has released further information and independent test results from Appen, which seem to validate many of their claims, thus reigniting interest in their technology.
The crux of Subquadratic’s innovation lies in its use of sparse attention, which significantly reduces the computational requirements typically associated with dense attention mechanisms found in traditional transformers. By focusing on the most relevant relationships between tokens, SubQ reduces the number of computations needed, avoiding the quadratic expansion problem that has plagued LLMs. Subquadratic’s co-founders express hope that their breakthrough could revolutionize the design and efficiency of LLMs, potentially rendering current transformer-based models obsolete in the coming years. As the field of AI continues to evolve, the implications of Subquadratic’s advancements could set a new standard for the future of language models.
Source: A startup claims it broke through a bottleneck that’s holding back LLMs via MIT Technology Review