Evaluating GitHub Copilot for engineering teams

Overview

While I was an Engineering Manager at JustGiving, the engineering organisation proposed GitHub Copilot as an AI coding assistant for its engineers. I was one of several people who contributed to evaluating it. This case study describes how the evaluation was approached, what we learned and the principles that came out of it.

It is not a product review. The tools have changed since and will keep changing, so a list of features would date quickly. The more durable question is how an engineering organisation decides whether a new capability is worth adopting, and on what terms.

Neither of the easy answers was satisfactory. Adopting the tools immediately would have meant accepting claims we had not tested. Rejecting them would have meant ignoring something engineers were curious about. A serious evaluation sat between the two.

The challenge

Interest in AI-assisted development was growing quickly across the industry. Vendors published productivity claims, engineers shared experiences that ranged from enthusiastic to dismissive, and it was reasonable for leaders to ask what all of this meant for their own teams.

Inside the organisation there was genuine curiosity. Engineers wanted to try the tools, which is a healthy instinct and one worth supporting. There was also pressure to form a view: what to tell teams, and what the implications were for security and code ownership.

The difficulty was that most of the available information was opinion. Published studies described other organisations, other codebases and often carefully chosen tasks. Individual anecdotes were sincere but hard to generalise. Leadership needed something more reliable than either: evidence from our own engineers, working on our own systems, under our own review and security standards.

Evaluation objectives

The evaluation set out to answer six questions.

• Impact on delivery. Does the tool help engineers complete real work, and for which kinds of work?
• Effect on code quality. Is the code that results as correct, readable and maintainable as code written without it?
• Effect on review practices. What changes for the reviewer when part of a change was generated?
• Security concerns. What does the tool send outside the organisation, and what might it introduce into the codebase?
• Developer experience. Do engineers find it useful, and does that vary between people and tasks?
• Organisational fit. Does it work with our languages, tooling, policies and ways of working?

Agreeing the questions first mattered. Without them, an evaluation tends to confirm whatever the people running it already believed.

Evaluation approach

Real work rather than synthetic tests

Engineers used Copilot on their normal work in existing codebases. Benchmarks and greenfield exercises flatter a coding assistant, because they avoid the things that make real software difficult: unfamiliar conventions, legacy decisions and context spread across many files.

Small-scale adoption

For comparison, I also evaluated Cursor. Copilot went forward to the trial because it integrated with the workflows and tooling engineers were already using, which is the organisational fit question in practice. The trial involved a limited group rather than the whole organisation. A small group is easier to observe and support, and the cost of being wrong is contained. An organisation-wide rollout would have created habits and expectations before we understood the consequences, and those are much harder to change afterwards.

Observation and feedback

Evidence came from the engineers doing the work: what they used it for, where suggestions helped, where they were discarded and what reviewers noticed. My contribution was in these activities, gathering and weighing that evidence with engineers and bringing it into the wider adoption discussions.

Quantitative and qualitative evidence

We considered both, and treated numbers with caution. Measures such as suggestion acceptance rate or lines of code produced are easy to collect and say little about whether the software is better. I do not have productivity figures to present here, and I would be sceptical of any single figure offered as the answer. The most useful evidence was qualitative: specific accounts from engineers and reviewers of what changed in their work.

What we learned

Developer productivity

The benefit engineers described most consistently was faster implementation of routine tasks. They also described less context switching, and found the assistance useful when working in unfamiliar areas. These are engineers’ own accounts of their work rather than measurements.

Code quality

The tools can improve speed. They do not guarantee quality. A suggestion that looks plausible still has to be read, understood and tested, and review remained essential.

Developer experience

Some engineers benefited more than others. The value depended on experience, the use case and individual workflow. That matters when deciding how to adopt a tool: one engineer’s experience, positive or negative, is a poor guide to a team’s.

Knowledge and learning

An assistant can be a useful learning aid. The same quality carries a risk: accepting a suggestion without understanding it.

Risks and trade-offs

• Over-trust. Fluent, confident output invites less scrutiny than it deserves.
• Security. Two separate concerns: what code and context leave the organisation, and whether generated code introduces vulnerabilities or insecure patterns.
• Licensing. Uncertainty about the provenance of generated code and how closely it may resemble existing licensed work.
• Reduced understanding. A team can come to own code that nobody on it fully reasoned through.
• Increased review burden. If code is produced faster than it can be reviewed properly, the constraint moves to the reviewer, and review quality is the first thing to suffer.
• Maintenance. Code is cheap to generate and expensive to keep. More of it is not an improvement in itself.

None of these was a reason to refuse the tools. They were the reason guardrails mattered. Stating the risks openly gave engineers a clear basis for using the tools, rather than leaving each person to guess what was acceptable.

Adoption principles

The guidance I contributed to rested on a small number of principles. They were deliberately simple, so that they would be remembered and would outlast any particular tool.

Engineers remain accountable

AI-assisted development does not remove accountability. An engineer is responsible for the software they ship, regardless of how it was produced. “The tool suggested it” is not an explanation for a defect.

Review standards do not change

A generated change is reviewed in the same way as any other. The author should be able to explain it, and the reviewer should expect them to.

Security expectations do not change

The same policies apply to what may be shared with a third party and to what may be merged. A new tool does not create an exception.

Understanding matters more than generation

If an engineer cannot explain a piece of code, it is not ready to merge. This protects the codebase and the engineer’s own development.

Use evidence rather than assumptions

Judge the tool on what it does for our work, not on vendor claims or on a strong opinion in either direction.

Adopt tools intentionally

Know what problem a tool is meant to solve and how we would recognise that it had helped. Be willing to stop using it if it does not.

Outcomes

The outcomes were qualitative, and I think it is more honest to say so than to attach a number to them.

A better understanding of the tools

The organisation had a first-hand view of where the tools helped and where they did not, based on its own work rather than on published claims.

More informed decisions

Discussions about adoption could draw on evidence from our own work rather than on assumptions.

Practical guidance

Engineers had guidance on how to use the tools and guardrails for where care was needed. I contributed to both.

Shared expectations

Ownership of AI-assisted code was explicit. Engineers and reviewers knew what was expected of them.

A considered approach to adoption

Adoption was approached deliberately: evaluate the tool against agreed questions, trial it with a limited group and agree expectations for its use.

Key lessons

AI is most valuable when solving real problems

The benefits engineers described were tied to specific, real tasks. That is a better starting point for adoption than the technology itself.

Adoption is organisational as well as technical

Installing a tool is the easy part. Agreeing expectations, review practices and ownership is where most of the work lies.

Guardrails increase confidence

Clear expectations give engineers a basis for using a tool. Without them, each person is left to guess what is acceptable. I see guardrails as something that enables adoption rather than restricts it.

Human judgement remains essential

Any benefit depends on an engineer deciding what to ask for and whether the result is good enough.

Leadership should focus on outcomes rather than excitement

The useful questions were about delivery, quality and the people doing the work, not about how impressive the technology was.

Reflection

This evaluation was a collaborative effort and reflects a particular organisation at a particular time. The tools have moved on, and some of the specific observations would be different today. I would expect the principles to hold.

The interesting question is not whether AI can generate code. It can. The interesting question is how an engineering organisation introduces a new capability while preserving accountability, quality and learning. That is a leadership problem, and the tool is the smaller part of it.