[ netdynamic // tech news ]

METR’s Updated AI Time Horizon Plot Reveals Gaps

AI evaluation nonprofit METR has released an update to its widely discussed time horizon plot, and the findings show a mixed picture. Some AI capabilities continue to improve along an exponential curve, but models still lag behind humans on tasks requiring judgment, even as they excel at coding. The original version of the chart, which suggested the length of tasks frontier models could complete doubled roughly every four months, was widely shared and often misread because of the complexity of its underlying measures.

The method works by having models attempt a large suite of tasks that human testers had also completed. Each task’s difficulty is measured by how long it took people to finish, and models are rated by the longest tasks they can complete successfully. The updated plot adds more tasks, including some that would take humans far longer than those in the original set, as well as more open-ended problems. This matters because a central question in AI right now is whether progress on verifiable tasks, where success can be checked automatically, will carry over to tasks where outcomes are harder to measure. Reinforcement learning has made models strong at coding and math precisely because those domains offer clear right and wrong answers. Judging something like choosing a research direction is far harder, which is why earlier evaluations largely left such tasks out.

Those harder tasks matter most for automating AI research itself, a topic drawing intense attention after a separate METR report suggested AI might be speeding up progress at Anthropic by about a factor of two. In one new task, models were asked to design AI evaluations usable in METR’s own research; none of the outputs were good enough for staff to find useful. In another, models scored philosophical arguments, a test of judgment whose relevance to AI R&D employees themselves disagreed about, and models again trailed humans. Still, newer models are improving on these open-ended tasks, moving from clearly below the bar to borderline performance within a year. Researchers remain divided on what the gap means, with some believing research taste will be a bottleneck and others expecting verifiable-task training to eventually transfer. The report offers encouraging signs, but predicting how quickly models will close the gap remains uncertain.


Source: The most misunderstood graph in AI just got a major update via MIT Technology Review