Jan 29, 2025 11:45 PM
(This post was last modified: Jan 29, 2025 11:52 PM by C C.)
The brain holds no exclusive rights on how to create intelligence
https://www.thetransmitter.org/neuroai/t...elligence/
EXCERPTS: . . . many of the recent developments that have contributed to the explosive success of AI have diverged from neuroscience as a source of computational principles. A decade ago, inspired by the brain, recurrent neural networks (RNNs) seemed to be the way forward as AI tackled time-dependent problems such as speech recognition and natural language processing. This direction changed quickly, however, with the landmark transformer paper in 2017: “Attention is all you need.”
The introduction of the transformer architecture marked an important inflection point in the history of AI. Transformers are notable both in their surprising power and in how un-brain-like they are. They lack the recurrent connections of RNNs and operate in discontinuous time—that is, through discrete time steps without any “memory” of the states from the previous time step. They are also devoid of any form of working memory; they cleverly externalize working memory by iteratively increasing the length of the input at each iteration. Most notably perhaps, transformers lack any internal dynamics or ability to tell time...
[...] By design, transformers are, in a sense, timeless. To use an analogy with terms from the philosophy of time, transformers operate in a block universe where the past, present and future (in the case of bidirectional transformers) are all simultaneously available. By contrast, RNNs operate in a presentist universe in which only the current input is available, and computations unfold in continuous time.
The so-called attention mechanism of transformers sounds biological, but it does not really refer to what most cognitive neuroscientists would consider attention. It essentially assigns a value to the strength of the relationship between all word pairs in a sentence ... Furthermore, the implementation of the attention mechanism also lacks biological plausibility... (MORE - details)
https://www.thetransmitter.org/neuroai/t...elligence/
EXCERPTS: . . . many of the recent developments that have contributed to the explosive success of AI have diverged from neuroscience as a source of computational principles. A decade ago, inspired by the brain, recurrent neural networks (RNNs) seemed to be the way forward as AI tackled time-dependent problems such as speech recognition and natural language processing. This direction changed quickly, however, with the landmark transformer paper in 2017: “Attention is all you need.”
The introduction of the transformer architecture marked an important inflection point in the history of AI. Transformers are notable both in their surprising power and in how un-brain-like they are. They lack the recurrent connections of RNNs and operate in discontinuous time—that is, through discrete time steps without any “memory” of the states from the previous time step. They are also devoid of any form of working memory; they cleverly externalize working memory by iteratively increasing the length of the input at each iteration. Most notably perhaps, transformers lack any internal dynamics or ability to tell time...
[...] By design, transformers are, in a sense, timeless. To use an analogy with terms from the philosophy of time, transformers operate in a block universe where the past, present and future (in the case of bidirectional transformers) are all simultaneously available. By contrast, RNNs operate in a presentist universe in which only the current input is available, and computations unfold in continuous time.
The so-called attention mechanism of transformers sounds biological, but it does not really refer to what most cognitive neuroscientists would consider attention. It essentially assigns a value to the strength of the relationship between all word pairs in a sentence ... Furthermore, the implementation of the attention mechanism also lacks biological plausibility... (MORE - details)