OpenAI is doomed to failure. The odds of them generating the 600-800 billion needed to pay their debts as well as maintain their infrastructure is essentially 0%
I genuinely have no idea what their endgame is, it makes zero sense unless they are completely delusional.
So the paper the entire tech industry has entered a suicide pact over is supposedly "Attention is All You Need"
Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
Neural networks are firmly beyond me, but it was presented to me that this is THE paper that basically proves that LLMs work, that you can do stuff with them that we've seen, and most importantly that it works not only with text, not only with images, but also video... assuming you just throw more and more compute which means more and more power at it.
Supposedly this has held true so far. You just have to keep throwing more and more compute -- and more and more electricity -- at it.
There's also plenty of videos (some I've linked here already) talking about how big tech is basically counterfeiting revenue over this -- OpenAI is buying Nvidia cards on credit, Nvidia is using that sale (before it actually closes) to give OpenAI credit, which OpenAI is using to buy more Nvidia cards on credit. Add a few more companies to the mix -- the 3 (yes, only 3, thanks to consolidation and patent law) companies in the world that make RAM, the two that make GPUs, the two that make CPUs, et cetera, and you get where we're at. They're basically playing Three-card monte with revenue -- I'll take out a loan to buy his stuff he'll take out a loan based on my buying his stuff to buy their stuff, they'll take out a loan to invest in me so I can pay back my loan based on him buying their stuff....
The theory I've heard is they think there's something there, that if they can just power through this they'll hit that "Rapture of the Nerds" level where they can generate an actual working, thinking, remembering, human equivalent brain -- Artificial General Intelligence. Once they can do that, they can scale it infinitely and suddenly they destroy the concept of labor forever. Even moreso if they can make it think faster or better than a human -- Artificial Superhuman Intelligence. At that point they just teach Mr. Computer how to make new AI and new RAM and new CPUs/GPUs and step back and revel in the profits forever.
Yeah LLMs might not be able to do it, but it might be able to help them bridge the gap to do it in some other way. That is that LLMs will be a transitional technology that we use to jumpstart and empower the actual research, like how computers were a transitional stage to LLMs.
Of course if that doesn't pan out, eventually the game of Three-card monte ends, and boy howdy we're gonna see the mother of all bubbles burst.
