"This broader, context-sensitive kind of learning and know-how is the more basic and ancient kind of knowledge, one which underlies the emergence of sentience in embodied critters and makes it possible to survive and flourish."

"But the diversity of responses highlights a deeper problem: as these LLMs become more common and powerful, there seems to be less and less agreement over how we should understand them."

"This leads to a troubling question: how can these systems be so smart, yet also seem so limited?"

"The underlying problem isn’t the AI. The problem is the limited nature of language."

"Once we abandon old assumptions about the connection between thought and language, it is clear that these systems are doomed to a shallow understanding that will never approximate the full-bodied thinking we see in humans."

"In short, despite being among the most impressive AI systems on the planet, these AI systems will never be much like us."

"A dominant theme for much of the 19th and 20th century in philosophy and science was that knowledge just is linguistic — that knowing something simply means thinking the right sentence and grasping how it connects to other sentences in a big web of all the true claims we know."

"The ideal form of language, by this logic, would be a purely formal, logical-mathematical one composed of arbitrary symbols connected by strict rules of inference, but natural language could serve as well if you took the extra effort to clear up ambiguities and imprecisions."

"It also motivated a lot of the early work in Symbolic AI, where symbol manipulation — arbitrary symbols being bound together in different ways according to logical rules — was the default paradigm."

"For these researchers, an AI’s knowledge consisted of a massive database of true sentences logically connected with one another by hand, and an AI system counted as intelligent if it spit out the right sentence at the right time — that is, if it manipulated symbols in the appropriate way."

"This is because language doesn’t exhaust knowledge; on the contrary, it is only a highly specific, and deeply limited, kind of knowledge representation."

"All language — whether a programming language, a symbolic logic or a spoken language — turns on a specific type of representational schema; it excels at expressing discrete objects and properties and the relationships between them at an extremely high level of abstraction."

"But there is a massive difference between reading a musical score and listening to a recording of the music, and a further difference from having the skill to play it."

"All representational schemas involve a compression of information about something, but what gets left in and left out in the compression varies."

"But there are nonlinguistic representational schemes which can express this information in an accessible way: iconic knowledge, which involves things like images, recordings, graphs and maps; and the distributed knowledge found in trained neural networks — what we often call know-how and muscle memory."

"One way of grasping what is distinctive about the linguistic representational schema — and how it is limited — is recognizing how littleinformation it passes along on its own."

"As Chomsky and his acolytes have pointed out for decades, language is just not a clear and unambiguous vehicle for clear communication."

"But humans don’t need a perfect vehicle for communication because we share a nonlinguistic understanding."

"but research suggests that the amount of background knowledge a child has on the topic is actually the key factor for comprehension."

"Understanding a sentence or passage depends on an underlying grasp of what the topic is about."

"The inherently contextual nature of words and sentences is at the heart of how LLMs work."

"Neural nets in general represent knowledge as know-how, the skillful ability to grasp highly context-sensitive patterns and find regularities — both concrete and abstract — necessary for handling inputs in nuanced ways that are narrowly tailored to their task."

"In LLMs, this involves the system discerning patterns at multiple levels in existing texts, seeing both how individual words are connected in the passage but also how the sentences all hang together within the larger passage which frames them."

"The result is that its grasp of language is ineliminably contextual; every word is understood not on its dictionary meaning but in terms of the role it plays in a diverse collection of sentences."

"In short, LLMs are trained to pick up on the background knowledge for each sentence, looking to the surrounding words and sentences to piece together what is going on."

"This allows them to take an infinite possibility of different sentences or phrases as input and come up with plausible (though hardly flawless) ways to continue the conversation or fill in the rest of the passage."

"But critics are right to accuse these systems of being engaged in a kind of mimicry. This is because LLMs’ understanding of language, while impressive, is shallow."

"A system like GPT-3 is trained by masking the future words in a sentence or passage and forcing the machine to guess what word is most likely, then being corrected for bad guesses. The system eventually gets proficient at guessing the most likely words, making them an effective predictive system."

"These skills, and the connected knowledge, allow the machine to explain how something complicated works, simplify difficult concepts, rephrase and retell stories, along with a host of other language-dependent abilities."

"Instead of a massive database of sentences linked by logical rules, as Symbolic AI assumed, the knowledge is represented as context-sensitive know-how for coming up with a plausible sentence given the prior line."

"“Abandoning the view that all knowledge is linguistic permits us to realize how much of our knowledge is nonlinguistic.”"

"But the ability to explain a concept linguistically is different from the ability to use it practically."

"The contextual knowledge is embedded in one form — the capacity to rattle off linguistic knowledge — but is not embedded in another form"

"These nuanced patterns of information are hard to express and convey in language but are still accessible to others. This is also the precise kind of context-sensitive information that neural networks excel at picking up and perfecting."

"Language is important because it can convey a lot of information in a small format and, especially after the creation of the printing press and the internet, can involve reproducing and making it available widely."

"But compressing information in language isn’t cost-free: it takes a lot of effort to decode a dense passage."

"Building a deep understanding is time-consuming and exhaustive, however the information is provided."

"This doesn’t make these machines stupid, but it also suggests there are intrinsic limits concerning how smart they can be."

"But they will undoubtedly seem to approximate it if we stick to the surface. And, in many cases, the surface is enough"

"Language may be a helpful component which extends our understanding of the world, but language doesn’t exhaust intelligence, as is evident from many species, such as corvids, octopi and primates."

"Rather, the deep nonlinguistic understanding is the ground that makes language useful"

"The goal is for AI systems to focus on the world being talked about, not the words themselves — but LLMs don’t grasp the distinction."

"There is no way to approximate this deep understanding solely through language; it’s just the wrong kind of thing."