The Future of AI Assessment: Real-World Tasks and Human Baseline Models

As artificial intelligence continues to advance, companies like OpenAI are exploring innovative strategies to refine their models. A recent initiative has unfolded that involves third-party contractors submitting real examples of their work tasks. This effort aims to create a comprehensive benchmark against which AI performance can be measured. The goal is to develop a human baseline for evaluating AI’s capabilities in various fields.

The Structure of Real-World Tasks

OpenAI’s new approach hinges on collecting detailed accounts of actual job tasks from contractors, offering fascinating insights into how AI can be trained more effectively. According to internal documents, contractors are encouraged to provide tangible outputs from their professional experiences—essentially, the real work they’ve done, rather than just a summary. Each submission includes two key components: the task request, which is what a manager or colleague might ask, and the task deliverable—the completed work in response.

For example, one scenario detailed the requirements of a Senior Lifestyle Manager tasked with creating an itinerary for a family’s yacht trip to the Bahamas. This concrete task highlights how AI can be trained to understand nuanced, complex requirements that human professionals handle regularly. By gathering such specific examples, OpenAI aspires to measure and compare AI capabilities with established human performance.

Interestingly, contractors can also submit fabricated examples designed to illustrate how they would tackle various scenarios. This flexibility could assist in assessing potential AI responses to unconventional tasks, broadening the scope of training data for future models.

The Legal and Ethical Considerations

However, this undertaking is not without its challenges. Intellectual property concerns loom large as contractors share their work with OpenAI. Legal experts warn that accepting submissions at this scale could expose AI labs to various risks, including potential violations of nondisclosure agreements or misappropriation of trade secrets. Evan Brown, an intellectual property lawyer, emphasizes the challenges this poses for AI labs in discerning what constitutes confidential information.

OpenAI has advised contractors to sanitize their submissions by removing personal information and proprietary data. Specific tools, like those designed for “scrubbing” confidential data, are mentioned as resources for contractors to navigate this complex landscape. Nonetheless, the onus remains on contractors to determine what is safe to share, creating a significant trust dynamic between the AI labs and their collaborators.

This approach reflects a broader trend in the AI industry where the quality of training data directly impacts the performance of generative AI systems. The reliance on real-world examples not only creates a more relatable framework for task assessment but also enhances the credibility of the AI models. As the quest for achieving artificial general intelligence (AGI) progresses, the lessons learned from this initiative could inform future developments across various sectors, ultimately shaping how humans and AI interact in the workplace.