Today's Large Language Models Will Never Be Responsible, Safe or Green

The Need for Investment in Revolutionary New AI Paradigms

TL;DR Version:

Today's large language models (LLMs) have impressive capabilities but must be revised. They struggle with hallucinations, lack real-world grounding, unreliable reasoning, opacity, and bias. These issues stem from their core architectures and training methodologies, making them unsafe and not environmentally sustainable.

Investment should focus on new AI paradigms that address these issues, such as task-specific models, embodied agents, hybrid neuro-symbolic systems, and interpretable models. These approaches emphasize domain-specific expertise, real-world interaction, transparent reasoning, and reliable behavior, creating AI systems that are safer, more robust, and aligned with human values.

The venture capital community must balance investments in current LLMs with bold bets on startups pioneering these innovative approaches. This diversified strategy will ensure AI evolves into a powerful, ethical, and beneficial societal tool.

While today's LLMs have significant limitations, exploring new paradigms and investing in innovative AI approaches is essential for developing responsible, safe, and green AI technologies.

I recently contacted an acquaintance, Roger, about being an advisor for 1Infinity’s first fund, which is focused on responsible, safe, and green AI. I contacted him because I knew he would be skeptical about the current implementation of generative AI, particularly large language models (LLMs).

After he congratulated me and told me he was sure I would be successful, he responded as follows:

“FWIW, I am extremely skeptical about generative AI. There are academic studies that state that “responsible” LLMs are an architectural impossibility.  

https://twitter.com/jacyanthis/status/1798543632523489586?s=46&t=ieCIZbe4BCo0gkUVaPus7Q

“Safe” is a really important goal and well suited to venture, as Microsoft, OpenAI, and Google are demonstrating that lack of interest in safety every day.

“Green” strikes me as impossible. As you must know, Microsoft has blown up its “carbon neutral by 2030” promise by growing its energy footprint by 30% over the past two years. Perhaps more concerning are the reports that every LLM prompt consumes half a liter of clean water. Given that most of the cloud server farms and fabs are in deserts, the threat to the environment is huge.”

And I one hundred percent agree with him. I have often expressed that the underlying technology was never designed for the current use patterns involving massive amounts of data. My and my three partners’ belief in this can be seen in our current pipeline of startups we have developed. Almost none are LLM companies or even support LLM technology in an attempt to make them responsible, safe or green. They are all focused on developing new net capabilities or combinations of new net capabilities as a pure deep-tech solution or as an application layer. Let us go into this a little deeper, as well as how

The Inherent Flaws of Today's LLMs

Large language models such as Gemini, ChatGPT-4.0, and PaLM 2 have captivated the world with their incredible fluency, broad knowledge, and apparent intelligence. These models, trained on vast amounts of online data, can engage in human-like dialogue, answer questions, write coherent text, and even code. To many, LLMs appear to be nearing human-level intelligence.

However, the technical foundation that today's state-of-the-art LLMs are built upon has inherent flaws and limitations that raise serious doubts about whether these models can be made safe, robust, and truly intelligent. Issues such as hallucinations, lack of grounding in the real world, inability to reason reliably, opaque and unpredictable behavior, and the potential to perpetuate biases and misinformation are not just minor bugs. They arise from fundamental choices in current LLMs' model architectures and training methodologies.

As transformative as LLMs have been, we need to look beyond the paradigm of giant neural networks trained to recognize statistical patterns in huge undifferentiated datasets scraped from the internet. The future of AI that is trustworthy and beneficial to humanity will require radically different approaches, such as task-specific models, embodied agents that interact with the real world, hybrid systems that integrate symbolic knowledge and reasoning with neural networks, and interpretable models whose behaviors are more predictable and controllable.

The venture capital community would be wise to balance their investments in scaling up LLMs with bold bets on startups pioneering these alternative, potentially revolutionary approaches to AI. Some of the most important breakthroughs will come not from the tech giants but from small teams questioning prevailing assumptions and architectures. A portfolio that includes incremental scaling and contrarian plays is needed as we chart the course of artificial intelligence in the pivotal years ahead.

Hallucinations and Lack of Grounding

One of the most concerning behaviors of LLMs is hallucination—generating statements that are fluent and plausible but factually incorrect, with no basis in the training data or real-world knowledge. LLMs frequently make up fake facts, references, quotes, and stories.

Hallucinations arise from several key underlying issues, as detailed in "Cognitive Mirage: A Review of Hallucinations in Large Language Models." The paper identifies three primary causes.

The first cause is data collection. The quality and range of knowledge within pre-trained corpora vary significantly. Information incorporated into LLMs can be incomplete or outdated, leading to unreliable outputs. When pertinent data is unavailable, the model may resort to heuristic judgments based on term frequencies, often resulting in inaccuracies. This issue is exacerbated in multilingual LLMs, especially in low-resource languages.

The second cause is the knowledge gap. LLMs' training data often lacks comprehensive and up-to-date information. The knowledge gap can lead to significant inaccuracies when models generate responses based on outdated or incomplete information. For example, LLMs might fabricate plausible but incorrect details when they encounter gaps in their training data.