When Data Scientists Met AI Engineers
A few weeks ago, I attended an event called "Future Leaders - Alumni Network" in London and since then I've been left disturbed.
At one point, the discussion was about:
“How exactly will AI impact jobs and society?”
I realised that, despite being able to sketch on a whiteboard how GenAI, RAG, the WINS framework, and Agentic AI all work — and even why CxOs are drawn to these technologies — I couldn’t give a clear, academically grounded answer to that question.
But this week, whilst enjoying the half-term break from day job, I discovered Paul Krugman’s “Hot Dog and Bun” thought experiment.
Krugman’s Hot Dog and Bun Thought Experiment
Imagine an economy that produces only two things: hot dogs and buns. Each hot dog requires exactly one bun. They are perfect complements.
Suppose both sectors employ 100 workers, each producing 100 units a day — 10 000 hot dogs and 10 000 buns. Output is perfectly matched, labour fully employed, and wages equal across both industries.
Now imagine a technological improvement doubles productivity in the hot-dog sector. With the same 100 workers, it can produce 20 000 sausages.
At first, this seems like a triumph of innovation — until we notice that bun production remains capped at 10 000. Since every finished product requires both components, total output of hot dogs is still limited to 10 000.
The productivity gain doesn’t increase national output; it reallocates labour value. Employment and wages fall in the automated (hot-dog) sector, while labour in the slower (bun) sector becomes more valuable. The total number of hot dogs stays the same, but income and importance shift from the fast sector to the slow one.
Krugman’s insight was elegant: when technology accelerates one part of a complementary system, the value of labour migrates to the part that hasn’t caught up yet.
The Modern Parallel: Data Scientists and AI Engineers
That simple economic pattern explains a shift many of us have quietly witnessed in the technology consulting world.
A few years ago, data scientists were the heroes of the AI revolution — the innovators building models, training algorithms, and promising transformation. But most consultancies discovered an awkward truth: they didn’t have enough client projects to apply those models. There were more boring projects than the cool projects that Data Scientists could do. Most of Data Science projects were more proofs of concept than production systems.
Then came AI engineers — the complementary sector. They didn’t invent intelligence; they integrated it. They wrapped APIs around foundation models, built RAG pipelines, and made AI usable for real business problems.
Almost overnight, the value of labour shifted. While data scientists focused on invention, AI engineers controlled the bottleneck to delivery. Demand, wages, and visibility followed. Just as in Krugman’s model, progress in one layer (model building) made another layer (integration) economically pivotal.
The Economics of Complementary Progress
In economics, productivity gains in one area only raise total output when complementary sectors evolve alongside.
In technology, this translates to:
Innovation doesn’t create value until integration catches up.
Every time one side accelerates — model research, infrastructure, automation — the other side becomes the bottleneck and temporarily gains labour value. That is why, when GenAI exploded, the ability to operationalise models suddenly became more valuable than the ability to train them.
What Happens Next
But this balance won’t last forever. Integration is already being abstracted by orchestration frameworks, auto-evaluation tools, and no-code AI builders.
When that happens, today’s integrators will feel what yesterday’s data scientists did — the experience of being overtaken by the next layer. The next bottleneck may move toward AI governance, verification, and human–AI collaboration design — areas where automation still struggles to replace human judgement.
This is not the end of jobs; it is the migration of human value. Each cycle pushes us upward: from building, to connecting, to overseeing, to designing meaning.
What It Means for All of Us
Each wave of technology doesn’t destroy work; it reshuffles where human judgement matters. The challenge for every professional is not to defend one title, but to stay close to the next imbalance — the part of the system that technology hasn’t yet automated.
Krugman’s thought experiment and a related article about it, helped me see that AI’s impact on jobs isn’t about loss, but about movement — the ongoing migration of value across layers of progress.
That’s the answer I wish I’d given that evening:
AI won’t simply take jobs. It will keep moving where work matters most.