Despite the increasing prevalence of artificial intelligence (AI) tools like Claude and ChatGPT, comprehensive understanding of how users engage with these technologies remains limited. AI companies such as Anthropic and OpenAI frequently release usage reports, but critics argue that these documents often omit crucial data. Anka Reuel, a PhD candidate in computer science at Stanford’s Trustworthy AI Research (STAIR) Lab, points out that there is no independent verification of the data provided by these companies. To address this issue, Reuel is leading a new initiative called the AI Observatory, a public platform designed to aggregate and analyze real conversations with popular AI models sourced from user consent across seven existing datasets.
The AI Observatory aims to fill a significant gap in the current understanding of AI interactions, especially given that critical decisions regarding the benefits and risks of AI are often made with limited data. Initial findings suggest that AI usage varies widely across different models and has evolved over time. For instance, while the Anthropic Economic Index predominantly focuses on productivity-related interactions, the AI Observatory discovered that nearly half of the conversations would have been excluded from this analysis. These omitted discussions often delve into personal topics such as health and relationships, highlighting a broader spectrum of AI usage that corporate reports tend to overlook.
Further analysis revealed that AI interactions have become more elaborate, with longer conversations and increased small talk over time, indicating a growing desire for companionship among users. Interestingly, the frequency of sensitive content, such as hate speech and harassment, appears to have declined, possibly reflecting improved safety measures implemented by AI platforms. The research also found notable differences in how various AI models are utilized—Grok and Gemini were popular for information retrieval, while ChatGPT was favored for educational assistance. Despite these insights, researchers acknowledge that the AI Observatory’s dataset represents only a fraction of the vast amount of data available to major companies. As such, while the findings provide a more nuanced understanding of AI usage, they may still underrepresent sensitive interactions. This ongoing research emphasizes the necessity for independent analysis in the evolving landscape of AI technology.
Source: We still don’t know how people are really using AI via MIT Technology Review