Aug 2, 2024 03:17 AM
https://www.thenewatlantis.com/publicati...lmost-over
EXCERPTS: This notion — that LLMs are “just” next-word predictors based on statistical models of text — is so common now as to be almost a trope. It is used, both correctly and incorrectly, to explain the flaws, biases, and other limitations of LLMs. Most importantly, it is used by AI skeptics like Marcus to argue that there will soon be diminishing returns from further LLM development: We will get better and better statistical approximations of existing human knowledge, but we are not likely to see another qualitative leap toward “general intelligence.”
There are two problems with this deflationary view of LLMs. The first is that next-word prediction, at sufficient scale, can lead models to capabilities that no human designed or even necessarily intended — what some call “emergent” capabilities. The second problem is that increasingly — and, ironically, starting with ChatGPT — language models employ techniques that combust the notion of pure next-word prediction of Internet text.
For firms like OpenAI, DeepMind, and Anthropic to achieve their ambitious goals, AI models will need to do more than write prose and code and come up with images. And the companies will have to contend with the fact that human input for training the models is a limited resource. The next step in AI development is promising as it is daunting: AI building upon AI to solve ever more complex problems and check for its own mistakes.
There will likely be another leap in LLM development, and soon. Whether or not it’s toward “general intelligence” is up for interpretation. But what the leap will look like is already becoming clear.
[...] Perhaps at a certain scale, the models would even learn how to “model” the process that created their training data, verbal intelligence. In other words, by studying trillions of specific selections of text, the model would learn to approximate intelligent reasoning itself. “What does it mean to predict the next token well enough?” asks then–OpenAI chief scientist Ilya Sutskever in a 2023 interview. “It’s actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token…. In order to understand those statistics … you need to understand: What is it about the world that creates this set of statistics?”
[...] Importantly, though, there are flaws. Sometimes, models simply memorize sequences of text, particularly ones they see repeatedly. Other times, infamously, they make up plausible-sounding “facts” that are false. Counterintuitively, the memorization of frequently encountered text is a case of the models’ failure, while the so-called “hallucinations” are, in a way, a case of their success. Language models are not intended to be a database of the text in their training data, for the same reason that it is neither expected nor desirable for you to memorize every word of a book you read. We do not want the models to memorize the training data — we want them to model it, to map the relationships and patterns within it. In this sense, all non-memorized LLM responses are hallucinations — that is, plausible-sounding responses. Some hallucinations are desirable, while others, particularly false information presented as fact, are undesirable.
Yet even when an LLM presents factual information in sequences of text that it did not memorize, it is still extremely difficult to know whether it truly “understands” the information. The reality that the models routinely output false information suggests, at a minimum, that their models of the world are flawed, or that they are not appropriately grounded.
[...] RLHF was essential to making ChatGPT a friendly, helpful, and knowledgeable assistant. But it also comes with tradeoffs. ...
[...] There are other drawbacks to the RLHF approach. It can make models more sycophantic, meaning they invent facts they assess the human might like to hear. RLHF can also make models more verbose, because human reviewers seem to prefer longer answers to more concise ones that contain the same information. RLHF can cause models to be mealy-mouthed, refusing to take positions, or inappropriately dodging questions using all-too-common phrases such as “as an AI language model, I cannot….” Google’s Gemini model caused a minor scandal with its refusal to answer questions such as whether the conservative activist Christopher Rufo has hurt society more than Adolf Hitler. (Gemini’s habit of producing racially skewed images, for instance depicting Nazis as black in the interest of diversity, was almost certainly not related to RLHF — it was because Google built its model to emphasize diversity, seemingly by tweaking user prompts behind the scenes automatically.) Meta’s Llama model refused to write code to “kill” a computer process — a term of art, in this context — because, the model said, killing is wrong.
In a technical sense, problems of this kind stem from what is called “overoptimization,” which is when a reward model overshoots the target of modeling human preferences...
[...] f we are to use language models to push back the frontiers of human knowledge, it seems likely that something beyond human preferences is required. The obvious candidate is AI models themselves. This approach is known by a variety of names, the most general of which is reinforcement learning from AI feedback (RLAIF). The concept is also sometimes referred to as “scalable oversight.” It is undoubtedly cheaper to use AI than humans for feedback, but some have suggested that it might also be better...
[...] Current language models are still making next-word predictions based on their statistical representations of the Internet. But as the approaches outlined here play an increasing role in the development of language models, this description will become increasingly unhelpful, and eventually it may fall apart altogether. If approaches like Constitutional AI are widely adopted, it may become more appropriate to think of future language models as the product of several AIs reasoning together and conversing among themselves, with the entire written corpus of human knowledge — our tweets and blogs, our poetry and prose, our wisdom and folly — as the foundation.
We do not know where this path will take us... (MORE - missing details)
EXCERPTS: This notion — that LLMs are “just” next-word predictors based on statistical models of text — is so common now as to be almost a trope. It is used, both correctly and incorrectly, to explain the flaws, biases, and other limitations of LLMs. Most importantly, it is used by AI skeptics like Marcus to argue that there will soon be diminishing returns from further LLM development: We will get better and better statistical approximations of existing human knowledge, but we are not likely to see another qualitative leap toward “general intelligence.”
There are two problems with this deflationary view of LLMs. The first is that next-word prediction, at sufficient scale, can lead models to capabilities that no human designed or even necessarily intended — what some call “emergent” capabilities. The second problem is that increasingly — and, ironically, starting with ChatGPT — language models employ techniques that combust the notion of pure next-word prediction of Internet text.
For firms like OpenAI, DeepMind, and Anthropic to achieve their ambitious goals, AI models will need to do more than write prose and code and come up with images. And the companies will have to contend with the fact that human input for training the models is a limited resource. The next step in AI development is promising as it is daunting: AI building upon AI to solve ever more complex problems and check for its own mistakes.
There will likely be another leap in LLM development, and soon. Whether or not it’s toward “general intelligence” is up for interpretation. But what the leap will look like is already becoming clear.
[...] Perhaps at a certain scale, the models would even learn how to “model” the process that created their training data, verbal intelligence. In other words, by studying trillions of specific selections of text, the model would learn to approximate intelligent reasoning itself. “What does it mean to predict the next token well enough?” asks then–OpenAI chief scientist Ilya Sutskever in a 2023 interview. “It’s actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token…. In order to understand those statistics … you need to understand: What is it about the world that creates this set of statistics?”
[...] Importantly, though, there are flaws. Sometimes, models simply memorize sequences of text, particularly ones they see repeatedly. Other times, infamously, they make up plausible-sounding “facts” that are false. Counterintuitively, the memorization of frequently encountered text is a case of the models’ failure, while the so-called “hallucinations” are, in a way, a case of their success. Language models are not intended to be a database of the text in their training data, for the same reason that it is neither expected nor desirable for you to memorize every word of a book you read. We do not want the models to memorize the training data — we want them to model it, to map the relationships and patterns within it. In this sense, all non-memorized LLM responses are hallucinations — that is, plausible-sounding responses. Some hallucinations are desirable, while others, particularly false information presented as fact, are undesirable.
Yet even when an LLM presents factual information in sequences of text that it did not memorize, it is still extremely difficult to know whether it truly “understands” the information. The reality that the models routinely output false information suggests, at a minimum, that their models of the world are flawed, or that they are not appropriately grounded.
[...] RLHF was essential to making ChatGPT a friendly, helpful, and knowledgeable assistant. But it also comes with tradeoffs. ...
[...] There are other drawbacks to the RLHF approach. It can make models more sycophantic, meaning they invent facts they assess the human might like to hear. RLHF can also make models more verbose, because human reviewers seem to prefer longer answers to more concise ones that contain the same information. RLHF can cause models to be mealy-mouthed, refusing to take positions, or inappropriately dodging questions using all-too-common phrases such as “as an AI language model, I cannot….” Google’s Gemini model caused a minor scandal with its refusal to answer questions such as whether the conservative activist Christopher Rufo has hurt society more than Adolf Hitler. (Gemini’s habit of producing racially skewed images, for instance depicting Nazis as black in the interest of diversity, was almost certainly not related to RLHF — it was because Google built its model to emphasize diversity, seemingly by tweaking user prompts behind the scenes automatically.) Meta’s Llama model refused to write code to “kill” a computer process — a term of art, in this context — because, the model said, killing is wrong.
In a technical sense, problems of this kind stem from what is called “overoptimization,” which is when a reward model overshoots the target of modeling human preferences...
[...] f we are to use language models to push back the frontiers of human knowledge, it seems likely that something beyond human preferences is required. The obvious candidate is AI models themselves. This approach is known by a variety of names, the most general of which is reinforcement learning from AI feedback (RLAIF). The concept is also sometimes referred to as “scalable oversight.” It is undoubtedly cheaper to use AI than humans for feedback, but some have suggested that it might also be better...
[...] Current language models are still making next-word predictions based on their statistical representations of the Internet. But as the approaches outlined here play an increasing role in the development of language models, this description will become increasingly unhelpful, and eventually it may fall apart altogether. If approaches like Constitutional AI are widely adopted, it may become more appropriate to think of future language models as the product of several AIs reasoning together and conversing among themselves, with the entire written corpus of human knowledge — our tweets and blogs, our poetry and prose, our wisdom and folly — as the foundation.
We do not know where this path will take us... (MORE - missing details)