Large language models like GPT-4 still need humans to tell it the kind of content it’s being trained on. This process is called labeling. Human beings read content, and give it labels like news article, medical report, jokes, etc. These labels give GPT-4 the ability to understand the type of text someone like you prompts it with and it’s able to generate new content that reads with a news article, medical report, or jokey tone.

Researchers at Yale, Carnegie Mellon, and Berkeley who were studying if AIs like GPT-4 train themselves for power domination and deception discovered GPT-4 is better at labeling content than the human labelers enabling GPT-4 to be so powerful.

They Were Trying to Be Cheap

The researchers needed to label over 500 thousand pieces of content and was provided an estimate of $500,000 at 20,000 man hours of work. Instead of coming up with the money, they used GPT-4 to do the labeling for them.

They ran a test were they hired 3 expert labelers and 3 crowdsourced labelers (people off the street) to annotate 2,000 pieces of content and prompted GPT-4 to do the same task.

GPT-4 outperformed the crowdsourced labelers 2-to-1. GPT-4 performed on par with the expert labelers in content that encouraged non-physical harm, spying, and betrayal. GPT-4 also did the task a lot cheaper than humans coming in at an estimated $5,000 for labeling the 500k+ pieces of content.

These results squarely put crowdsourced labeling in the crosshairs of being obsoleted by AI. Companies would normally resort to using Amazon’s Mechanical Turk service to outsource a gargantuan human task on the cheap. Labeling is still a tedious job, and tapping into a globally cheap and flexible workforce can still be costly.

This experiment shows one task that can still be a net benefit while being performed by AI.