TLDR;

• We tested AI-Assisted Development (AIAD) at scale.
• The raw lab results looked like multiple-x productivity gain, but the reality was a bit more subdued. 
• Even that translates into a ridiculous ROI – think millions saved for the price of a SaaS subscription. 
• Oh, and it also lets engineers do things they previously couldn’t (or wouldn’t). 
• Bottom line: AI in dev is not optional anymore – it’s table stakes. 

Disclaimer

I am a pragmatic individual, sometimes to the point of skepticism. I question everything, all the time, me included. Not in a conspiracy theory way, but in a “show me the numbers” kind of way. You know, the way that pisses off most people, because you are questioning their sacred beliefs developed on TikTok. 

Why do I do this? Because that’s the only way to keep learning and growing. Moving on.

What is AIAD

AI-Assisted Development (AIAD) is what happens when developers let AI agents help them plan, architect, create, and maintain software. 

The average dev spends ~70% of their time typing code. AI can now take a big chunk of that grunt work. The result? Devs feel faster. But “feeling faster” is not the same as measured productivity. So how do you measure it without bias? One possibility: ask your local psychic. 

The Testing Framework

Fortunately, at CJ – where among other things, I’m responsible for the Engineering department – we had a lot of open-minded engineers that wanted to give AI a go and see what the hype was about. 

We chose a few dozen volunteers, from all walks of life: from technically-inclined individuals with no software experience, to engineering leaders with 25+ of expertise. 

We wanted coverage across: 

Project types 

• PoC 
• Feature development  
• Refactoring / Migration / Upgrade  
• Discovery / Debugging  
• Test development  
• Designing & architecting  
• Optimization / Tuning / Health-checks  
• Infrastructure development. 

Project phases 

• Planning  
• Coding 
• Review/Pairing
• Testing
• Deployment
• Maintenance 

Because not all projects are equal. Debugging is research-heavy, migration is code-heavy, and if you let people choose their “test projects,” they’ll pick the ones where AI shines. 

Data Gathering

The instructions to our volunteers were simple:  

1. Estimate your project like you normally do.
2. Execute the project with AI, as much as possible. 
3. Compare what you estimated with what it actually took.  
4. Bonus points: tell us how it felt – what you loved, what you hated, and whether your coffee was any good that morning. 
5. To record all this we used a secret, bleeding-edge technology, very expensive and hard to get: an Excel spreadsheet. 

Timeline: 90 days. 

Tools: Claude Code Sonnet 4.0, with a sprinkle of Opus 4.1. 

Raw Results

As expected, the results were all over the place: 

• Some projects: 10% faster.
• Others: 10x faster.
• Average across the board: ~500% (aka 5x velocity). 

So, using AI, on average, the developer experienced a velocity increase of 5x. 
How a Business Executive reads that: “OMG, that means Engineering will complete 5 years’ worth of projects in just 1 year.” 

My reaction? “Ummm … not so fast.”

Normalized Results

Not so fast, because despite our best efforts this is still a lab experiment, and its findings do not directly map to the real world. Once you adjust for the real-world variables (aka death by a thousand cuts), the 500% ends up being normalized to a much more realistic (conservative?) 20%. 

How to read that: if tomorrow everyone in Engineering starts using AI, the velocity will increase by minimum 20%. 

Beyond the Numbers

Velocity isn’t the whole story. Here’s what AI adds:

• Opportunity cost reduction → Engineers can run side quests (lower-priority tasks) concurrently instead of being stuck in single-threaded mode.
• Skill enablement → AI levels up engineers on tech they’ve never touched before, flattening learning curves.
• Pseudo-engineering → Non-engineers can build internal tools and apps that make their jobs way more productive without waiting six months for a roadmap slot.

Conclusions

So, just 20%? Yes – and that’s still outstanding.

• If you’ve got 100 engineers, that’s like suddenly hiring 20 more.
• Cost of those 20 humans: about $1.7M/year.
• Cost of AI usage for 100 engineers: ~$100K/year.
• ROI: 17x.

Not bad. Not bad at all.

YMMV (Your Mileage May Vary)

Like all things in life, results depend on:

• Your tech stack
• Project types & phases
• Engineer buy-in
• Your adoption framework
• App purpose
• Expectations

So no, it’s not a magic wand. But AI isn’t going away. It’s cheap, powerful, and – if you’re not using it – you’re already behind.