Research Despite impressive output, generative AI doesn’t have coherent understanding of world

[C C]

https://www.eurekalert.org/news-releases/1063781

INTRO: Large language models can do impressive things, like write poetry or generate viable computer programs, even though these models are trained to predict words that come next in a piece of text.

Such surprising capabilities can make it seem like the models are implicitly learning some general truths about the world.

But that isn’t necessarily the case, according to a new study. The researchers found that a popular type of generative AI model can provide turn-by-turn driving directions in New York City with near-perfect accuracy — without having formed an accurate internal map of the city.

Despite the model’s uncanny ability to navigate effectively, when the researchers closed some streets and added detours, its performance plummeted.

When they dug deeper, the researchers found that the New York maps the model implicitly generated had many nonexistent streets curving between the grid and connecting far away intersections.

This could have serious implications for generative AI models deployed in the real world, since a model that seems to be performing well in one context might break down if the task or environment slightly changes.

“One hope is that, because LLMs can accomplish all these amazing things in language, maybe we could use these same tools in other parts of science, as well. But the question of whether LLMs are learning coherent world models is very important if we want to use these techniques to make new discoveries,” says senior author Ashesh Rambachan, assistant professor of economics and a principal investigator in the MIT Laboratory for Information and Decision Systems (LIDS).

Rambachan is joined on a paper about the work by lead author Keyon Vafa, a postdoc at Harvard University; Justin Y. Chen, an electrical engineering and computer science (EECS) graduate student at MIT; Jon Kleinberg, Tisch University Professor of Computer Science and Information Science at Cornell University; and Sendhil Mullainathan, an MIT professor in the departments of EECS and of Economics, and a member of LIDS. The research will be presented at the Conference on Neural Information Processing Systems... (MORE - details, no ads)

[stryder]

It reminds me of a time I used the London Underground to go to a conference.

I'd meticulously wrote down the directions from the station I needed in regards to which roads to follow to reach my destination.

When it came to the day though I wandered out of a different exit from the same station which meant all my directions were useless.

Luckily I'd checked out the road surface on Google Map beforehand and remembered certain details of architecture that led me to my destination, it proved it's always a good idea to use as much to your advantage as possible to make sure the outcome is what your looking for.

[confused2]

This could have serious implications for generative AI models deployed in the real world, since a model that seems to be performing well in one context might break down if the task or environment slightly changes.

It is always useful to have some understanding of the question you are asking .. current AIs don't have godlike power.

As in the above route finding example..

Finding the shortest route between two points on a map is 'tricky'. I remember some fanfare in the 1980's when Fredman and Tarjan came up with a good algorithm.
https://www.cs.upc.edu/~mjserna/docencia...-mst-h.pdf
The usual AI method of picking the best answer doesn't generally involve diving off into computationally intensive algorithms on the way. Given enough time it is possible (but highly unlikely) that an AI could work out a shortest route program but it would be a long and tedious process - not something it could do in the course of answering "how to get from this car park to the nearest kebab shop". Without a 'shortest path' algorithm a map would be useless and the AI gives routes from its database of routes that have worked in the past.

Likewise chess playing computers will analyze possible moves ahead of the current situation .. a general purpose AI will pick moves from a database of known good moves. Being able to beat a general purpose AI doesn't mean you can beat a chess computer - they're doing completely different things.

I use an AI (Pi) to generate snippets of Python code .. mostly because I can't be bothered to learn Python. It gives the best match to what it thinks I'm asking for without actually checking the code. On one occasion I got fed up of its bad answers and wrote the code myself - then asked it what it thought of it - it liked it - it must have parsed it in some way to know whether it was good or bad - I can't guess how though.