🔍 Read the full analysis: How To Think About ChatGPT And Lasting Data Science Skills on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT performed better on coding, statistics and prediction tasks than a group using conventional online resources. But on a later technical test without ChatGPT, the researchers found no evidence that the AI group had learned more.
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher on data science tasks than colleagues using Google and sites such as Stack Overflow, but showed no evidence of stronger technical knowledge on a follow-up test without the chatbot. Published October 26 in Nature Human Behaviour, the study suggests that AI assistance can expand what workers accomplish while it is available, without necessarily building skills they can use on their own.
The trial allocated 986 BCG consultants to ChatGPT or a control group. Of those, 573 began the survey and 487 submitted work on both assigned tasks; that group formed the main analysis sample. Participants completed two 90-minute tasks chosen from coding, statistics and prediction. The ChatGPT group received 15 to 20 minutes of prompting training and used GPT-4. The control group received instruction in using Google and online resources including Stack Overflow and Khan Academy.
Compared with the control group, ChatGPT users scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction, according to the study. They also finished coding and prediction about nine and 14 minutes faster on average, respectively; the authors reported no significant time difference for statistics. Scores were compared with results from 44 BCG data scientists who completed the tasks without AI. On average, consultants using ChatGPT remained below that expert benchmark on all three tasks.
Afterward, participants answered five technical questions about topics including probability, machine learning and coding, without access to ChatGPT. The authors reported no evidence that the ChatGPT group did better than the control group on this test. They also found that the ChatGPT group became more confident that the tool could solve problems it could not. The paper describes that confidence result as exploratory, not as a definitive finding.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Gains Did Not Equal Skill Gains
The distinction matters to employers deciding whether generative AI is a productivity aid, a form of training, or both. In this experiment, workers with ChatGPT completed unfamiliar tasks more successfully, but the follow-up test did not show that they had retained an advantage when the tool was removed. The findings therefore support a narrower conclusion than “AI trained the workforce”: the tool improved performance during use, while evidence of independent learning was absent on the measured test.
That has practical implications for how companies evaluate training and work quality. A task completed with AI may show what an employee can deliver with assistance, not what that employee can reliably do unaided. Employers assessing capability may need to distinguish AI-supported output from independent expertise, and set review procedures that account for tasks the model can mishandle. The trial does not establish how every workplace should do that, but it cautions against treating short-term performance gains as proof of lasting skill development.
The study also raises a separate concern about confidence. If workers become more confident in a tool’s ability to answer questions it cannot solve, that could affect when they seek review or verify results. Because the authors call this part of their analysis exploratory, it should be treated as a signal for further study, not a settled estimate of workplace risk.
Products Worth Considering
Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black
Bose SoundLink Max Portable Bluetooth Speaker, IP67 Waterproof, Black
Samsung Odyssey G3 G30D 27-Inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
A Controlled Test at BCG
The researchers preregistered the trial on March 13, 2024, before data collection, which took place in late March and early April that year. Consultants had at least a bachelor’s degree, and most reported no coding experience, according to the authors. The work was designed with BCG data scientists and OpenAI’s Economic Impacts Research team.
The tasks were deliberately constructed so ChatGPT could help but could not correctly answer every question independently. Participants had to clean, merge and filter datasets in Python; assess statistical models and plots; or build a prediction model about the predictability of international football matches. This design tested whether users could work with the chatbot’s assistance rather than simply copy a complete, correct answer.
One subgroup had prior coding experience. Among these participants, whom the paper calls “competent coders,” the ChatGPT group scored five percentage points below the data scientists on coding and statistics; the authors found no evidence of a difference from the experts on those tasks. On prediction, the subgroup remained 20 points below. These subgroup results do not alter the overall finding that the full ChatGPT group remained below the expert average.
“The authors describe the effect as “temporary upskilling.””
— Wiles, Krayer and co-authors, as described in their paper
Limits of the Follow-Up Evidence
The study tested a particular group of consultants, a short training session, GPT-4 and three specially designed tasks. It does not establish whether different training, longer practice, other AI systems or repeated use would produce lasting skills. The follow-up consisted of five technical questions, so it is a limited measure of what participants retained or could apply later.
The main analysis included 487 people who submitted work on both tasks, fewer than the 986 initially allocated. The supplied study summary does not explain all reasons for that reduction or how it may affect interpretation. Nor does this experiment measure long-term workplace outcomes such as job performance, error rates or whether workers later become better at learning technical material.
The confidence finding is also explicitly described by the authors as exploratory. Its size and durability are not established by the information reported here. The results should not be read as proving that ChatGPT can never teach lasting data science skills; they show that this trial found no advantage on its immediate unaided test.
Testing Longer-Term Skill Retention
The paper reports the trial and its follow-up results; the supplied material does not identify a scheduled next experiment or a further milestone. Questions for future research include whether more extensive instruction, guided practice, feedback or repeated unaided exercises help workers retain skills after AI use.
For employers, the immediate next step is to judge AI-supported work and independent competence as separate measures. The study offers evidence about performance in its specific tasks, but does not prescribe a company training policy. Further trials across occupations and longer periods would be needed to determine when AI use contributes to lasting learning.
Key Questions
What did the BCG trial find?
Consultants using ChatGPT scored higher than the control group on coding, statistics and prediction tasks. The advantage was 49 percentage points for coding, 20 for statistics and 17 for prediction.
Did ChatGPT users perform as well as data scientists?
Not on average. The ChatGPT group remained below the scores of 44 BCG data scientists on all three tasks, although participants with coding experience came close on coding and statistics.
Did the consultants retain their advantage without ChatGPT?
The researchers reported no evidence of an advantage for the ChatGPT group on five technical questions completed without the tool. That test was limited and does not settle what longer or different training might achieve.
Does the study prove AI cannot teach lasting skills?
No. It tested one short training, one model and a set of tasks with a particular group of consultants. Its result is narrower: this trial did not find an unaided-test advantage immediately after the tasks.
Source: ThorstenMeyerAI.com
