Did AI coding actually improve efficiency? Do not treat three types of evidence as a single conclusion

One developer states that AI helped them build a prototype in ten minutes; another states that reviewing AI modifications is slower than writing them from scratch. Both statements can be true simultaneously. Prototype building, mature project modification, and long-term maintenance are not the same type of work. Without specifying the task, completion criteria, and timing method, "efficiency gains" easily turn into talking past each other.

For technical leads, a more useful question is: on our tasks, which part of the effort did AI reduce, and where did it shift the work? This article uses several public studies from METR as a thread to explain how to read seemingly conflicting conclusions and provides an evaluation method that teams can adopt. Data is current as of 2026 9 5; there are no internal EveryInfra efficiency experiments here, nor rankings of any models.

What exactly did "slower by 19%" measure

In the study published in July 2025, METR had 16 developers familiar with their respective open-source projects complete 246 real tasks, randomly assigning whether they were allowed to use the AI tools available at the time. Under these research conditions, task completion time with AI increased by 19%. The study involves tools from early 2025 and specific working scenarios, not a uniform result for all development activities in 2026.

This limitation is not meant to downplay the results, but to preserve their utility. A maintainer familiar with a large repository already knows project conventions, historical trade-offs, and error-prone areas. Even if the first version of modifications provided by the model looks reasonable, it may still need to be re-examined against this background. On the other hand, entering an unfamiliar project, performing one-time data processing, or quickly validating ideas may involve different constraints.

Therefore, using this experiment to directly declare that "AI coding is useless" goes beyond the evidence; interpreting it to mean that "anyone who can write prompts will definitely gain efficiency" is similarly unfounded. It serves better as a reminder that subjective feelings and the time required to complete tasks do not necessarily align.

In the Hacker News discussion at the time, readers also raised questions about sample size, project familiarity, and how subsequent rework should be timed. These comments help complement the evaluation questions, but cannot be treated as new experimental results. In particular, a vivid empirical ratio should not be written as a statistical rule applicable to all projects.

Why subsequent materials from 2026 cannot simply extract a new percentage

By the experimental design update in February 2026, METR explicitly pointed out that subsequent experiments encountered selection bias in participants and tasks, and multi-agent concurrency also made time tracking more difficult. Some developers were reluctant to participate in task assignments that required giving up AI. Researchers noted that the assistance provided by tools may be increasing, but existing data is insufficient to reliably estimate the growth magnitude.

This is not "the old experiment being overturned." The old experiment still describes the scenario it studied at the time; the subsequent materials remind us that in the new environment, even the research subjects and measurement methods are changing. People who use AI more actively may not have entered the sample, and tasks best suited for AI may not have been submitted.

For enterprise pilots, this issue is very practical. Counting only users willing to showcase results easily misses failed or abandoned tasks; having only members unfamiliar with AI participate may turn friction from the learning phase into long-term effects. Both choices can alter the conclusion.

Our recommendation is to first record how tasks enter the pilot, then look at the average results. Keep track of who did not participate, which tasks were excluded, and what the reasons for exclusion were. There is no need to collect irrelevant personal information here; a log categorized by task type is more explanatory than showing only a few successful demos.

Working fast, producing much, and delivering high value are not the same metric

METR further distinguished speed and value in the technical worker survey from May 2026. The survey covered 349 technical workers, and respondents' self-reported median work value multiples ranged between 1.4 and 2 times depending on the phrasing; the self-reported median speed multiple was 3 times. This refers to subjective estimates in the questionnaire, not team productivity gains confirmed by randomized experiments.

Why might these two metrics differ? Imagine a team using AI to quickly generate multiple internal demo pages. If these pages help confirm an important requirement, the value may be high; if they do not enter any decisions, the number of code files and pages increases, but business value may not follow. This is a hypothetical example, not a research sample from a specific company.

Conversely, AI helping engineers check documentation fewer times and reducing frequent tool switching may improve the work experience, even if the project's final delivery date does not change significantly. Experience improvements are worth recording, but should not be rewritten as having saved an equivalent proportion of human resources budget.

We prefer to ask three separate questions clearly: whether the same task was completed faster, how much manual effort was required to reach the same standard, and whether the completed task was more worth doing. Combining them into a single "efficiency improvement multiple" instead discards the information most helpful for teams to adjust their working methods.

For your team's evaluation, define "completion" first

Before integrating new tools, fix an agreed-upon completion standard. For example, an API integration task can require: the happy path works, critical failures are explainable, necessary tests pass, and the handover person understands the limitations. Specific projects can add or remove conditions, but cannot lower requirements after seeing AI output just to make results look better.

Next, record two types of time separately. The first is calendar time from task start to acceptance, including waiting time; the second is actual human effort time, including understanding requirements, preparing context, review, and rework. When multiple agents run concurrently, the runtime minutes of individual machines cannot be directly added up as human work hours, nor can all waiting time be counted as saved time.

A record sufficient to begin a pilot can contain just these fields:

• Task type, complexity assessment, and members' familiarity with the codebase.
• Tools and versions used, whether AI assistance was involved, and who is responsible for final sign-off.
• Time of first version generation, time of reaching established completion standards, and actual human effort.
• Issues found during review, number of rework iterations, and defects discovered subsequently.
• Business purpose of the task; whether it was ultimately adopted, and why if not.
• Interrupted, failed, and voluntarily abandoned tasks must not disappear from records.

This checklist is our evaluation recommendation, not a replication of METR's experiments, nor does it guarantee significant conclusions from small samples. Its primary purpose is to uncover whether work is specifically getting stuck in generation, verification, or handover stages.

If current friction mostly stems from mismatched interfaces, switching to a stronger model may not solve the problem. For example, an SDK constructing requests does not mean the service supports the same parameters or response methods. The on-site OpenAI SDK and Gemini migration boundary discusses this kind of compatibility judgment; these integration constraints should also enter pilot task completion criteria.

Small samples should aid decisions, not manufacture promotional figures

After evaluating for a period, break results down by task type. If low-risk, well-structured tasks saved human labor while cross-module modifications brought more reviews, the next step may be adjusting the scope of AI usage rather than universally increasing or stopping usage across all tasks.

When reviewing results, also check whether task difficulty and quality requirements changed. If a task previously required handling errors completely and now only completes the happy path, the two should not be directly compared. If a task previously yielded a handoff-ready plan and now generates ten unadopted drafts, output growth should not be judged solely by count.

Tool costs can be treated as a dimension, but should be viewed alongside human review, failure rework, and maintenance investment. When reliable records are lacking, acknowledge that calculation is temporarily impossible, rather than packaging API call counts or generated lines of code as return on investment. For how to compare task adaptation, failure behavior, and cost metrics together, refer to the API Selection Guide.

METR also has a science communication note specifically discussing how to present research findings that are easily misunderstood. The same applies to enterprise content: retaining unappealing limitations is often more effective than picking the most striking number for helping readers judge suitability.

"Does AI coding improve efficiency?" will not have a uniform answer just because another opinion piece is published. However, teams can make the question more answerable: under which task types, tool versions, and completion standards did manual effort decrease, did quality change, and were results truly adopted? Obtaining such bounded answers first before deciding to expand tool usage is more useful than arguing over an efficiency multiple detached from tasks.