What Heat Pumps Actually Achieve: Measured Efficiency vs the Rating on the Box

Anyone pricing a heat pump eventually hits the same wall. Every running-cost estimate needs one efficiency number, and the sources hand you a different one depending on where you look. Our own heat pump payback calculator asks you to type a COP into a box, and the answer you get out of it swings by thousands of pounds or dollars depending on what you type.

So this report gathers what the field studies actually metered, in real occupied houses, and puts each figure next to a plain statement of what it includes. The short version is that three British fleets, all metered to the same declared boundary, returned annual efficiencies of 2.36, 2.81 and 3.86, and the difference between them is not the equipment. It is where the measurement boundary was drawn and whose houses were in the sample.

Key figures

• Across 428 air-source heat pumps monitored in British homes between September 2020 and September 2023, the median seasonal performance factor at the SPFH4 boundary was 2.78, with a mean of 2.81, as reported by Energy Systems Catapult in December 2024.
• The same 428 systems returned a median SPF of 2.93 at the narrower SPFH2 boundary, so choosing which components to count moved the headline number by 0.15 without changing a single installation.
• In the DOE Cold Climate Heat Pump Challenge field validation, published by PNNL in January 2025 across 22 field-tested units, the median heat-pump-only COP in the 0 to 5 degree Fahrenheit outdoor bin was 1.9, with periods of auxiliary heat and defrost excluded from that figure.
• Vermont's Department of Public Service metered 77 cold-climate heat pumps and found they operated at 88% of their average nameplate HSPF, with individual installations ranging from 57% to 119% of nameplate.
• The US federal test procedure computes the labelled HSPF2 using the heating bin hours of a single climate region, Region IV, under 10 CFR Appendix M1 to Subpart B of Part 430.
• Under EU ecodesign law, the only seasonal efficiency a manufacturer must declare is the one for "average climate conditions", defined in Commission Regulation (EU) No 813/2013 as the temperature conditions characteristic for the city of Strasbourg.

What the field studies measured

The table below is every annual, whole-season figure in this report, each with the boundary it was measured to. The boundary matters enough that the figures in the last column should not be read as a ranking.

The three British rows are the cleanest comparison in the whole dataset, because all three used the same SEPEMO H4 boundary in the same country. The DECC analysis of the Renewable Heat Premium Payment scheme reported a mean SPFH4 of 2.36 across 297 air-source systems. A decade later, Energy Systems Catapult's Electrification of Heat report found a median of 2.78 and a mean of 2.81 across 428 systems. The open dataset at HeatpumpMonitor.org, built from systems metered to MID-approved Class 1 electricity and Class 2 heat meters, sits far above both.

That is not this report's own arithmetic. The same three-way comparison was made in peer review by Rosenow, Lea and Boni in Energy and Buildings, and is summarised in the CIBSE Journal write-up of that paper, which notes that an SPF of 2.8 leaves running costs roughly level with a gas boiler while 3.86 saves around 26%.

The boundary is not a detail

European field trials report seasonal performance against the SEPEMO system boundaries, numbered H1 upward. Each step outward adds another electrical consumer to the denominator. The Electrification of Heat report sets the arithmetic out directly: SPFH2 counts the heat pump unit alone, SPFH3 adds the back-up and immersion heaters to both sides, and SPFH4 adds the circulation pump's electricity as well.

Because the same monitored systems were reported at several boundaries, the effect can be isolated exactly. Nothing about the houses or the hardware changes between these columns.

In the RHPP fleet, widening the boundary from H2 to H5 cost 0.36 of seasonal performance, close to 14% of the headline figure. In the Electrification of Heat fleet, the same widening from H2 to H4 cost only 0.15, around 5%.

That difference is the useful finding, and it is easy to miss. There is no fixed derating factor for "including the backup". The penalty is small when the backup heater rarely runs and large when it runs often. Fraunhofer ISE's German monitoring makes the same point from the other side: in the WP-QS im Bestand project, which tracked 77 systems in existing buildings over four years, electric heating rods accounted for just 1.3% of the electrical work in the air-to-water systems.

Where the backup does most of the work, the boundary stops being a correction and becomes the answer. The Center for Energy and Environment's Minnesota field study measured weather-normalised annual heat-pump-only COPs of 2.75, 2.78 and 2.51 at its three ducted sites. In the same report, modelling twelve Minnesota cases with a propane furnace behind a 10 degree Fahrenheit lockout, the annual whole-system COP came out between 1.09 and 1.33. Same equipment, same state, less than half the number, because the propane went inside the boundary.

Where the sources genuinely disagree

Cold-weather performance is where published figures diverge most, and it is tempting to reconcile them into one curve. That would be a mistake, because the numbers are not measuring the same thing. The clearest example is PNNL's field validation of the DOE Cold Climate Heat Pump Challenge, published in January 2025, which reports its COPs with periods of auxiliary heat and defrost stripped out by design, so that the compressor's own behaviour is visible. That is the opposite editorial choice to the SPFH4 figures above, which exist precisely to fold those periods in. This table shows what sits inside each headline cold-weather figure.

No ratio between any two of those rows means anything. Dividing PNNL's 1.9 by Catapult's 2.37 would compare an instantaneous compressor-only figure with auxiliary heat deliberately stripped out against a whole-system average that has auxiliary heat deliberately left in, at different temperatures, on different continents. The honest statement is the one the table makes: each figure is correct for what it measures, and the measurements are not interchangeable.

What can be said is the direction, from a single source at a time. Within the Electrification of Heat dataset alone, the 30-minute median whole-system COP peaked at 3.37 when the outdoor temperature was 10 degrees Celsius, fell to 2.37 at minus 3 and to 2.30 at minus 5. Ground-source systems in the same trial held 3.10 at minus 5.

What the rating on the box is actually measuring

None of the field figures above are what a manufacturer prints on a label, and the reason is structural rather than dishonest. Each rating standard bakes in a climate and freezes a set of operating conditions so that two machines can be ranked against each other.

Every entry in that table is verifiable in the governing document. 10 CFR Appendix M1 to Subpart B of Part 430 instructs testers to "use the rating conditions specified in table 8 of AHRI 210/240-2024 and the fractional heating bin hours specified for Region IV in table 16 of AHRI 210/240-2024 to determine the heating efficiency metric, HSPF2". On the European side, Commission Regulation (EU) No 813/2013 defines average climate conditions as "the temperature conditions characteristic for the city of Strasbourg", and that is the only declaration a manufacturer is obliged to make.

The exclusions are the more interesting half. Fairey, Parker, Wilcox and Lombardi documented several of them in ASHRAE Transactions in 2004, noting that the procedure "specifically requires that strip resistance heaters be prevented from operating during the frost accumulation test" even though most air-source heat pumps do run strip heat during defrost, and that it "assumes there is no defrost operation below 17F where defrost operation will, in fact, be most often triggered". They add that the single labelled value "was never envisioned" to predict performance across all climates.

Modern variable-speed equipment adds a further wrinkle. Bruce Harley's 2022 report for the Northwest Energy Efficiency Alliance observes that a modern heat pump's behaviour is governed by its firmware, and that "during standard rating tests, the critical control firmware is intentionally disabled". The report argues for load-based testing under CSA EXP07, which rates equipment across eight North American climates instead of one.

The cold-climate qualification specs are explicitly lab documents too. NEEP's ccASHP Specification version 4.0, effective January 2023, requires a COP at 5 degrees F of at least 1.75 at maximum capacity and states that "lab testing results OR engineering data for each system must be reported". ENERGY STAR's cold climate criteria use the same 1.75 threshold alongside a requirement to hold at least 70% of rated capacity at 5 degrees F. PNNL noted that most field units in the DOE Challenge did clear that 1.75 bar in practice.

What a buyer actually sees is thinner still. Carrier's consumer-facing efficiency guide explains the SEER2 and HSPF2 scales and says standard heat pumps begin to lose efficiency below 25 to 30 degrees F, but offers no boundary definition and no field figure to set against the label.

The one place rated and measured can be compared

There is exactly one comparison in this dataset where a rated number and a measured number were deliberately built to the same definition. Vermont's Department of Public Service and Cadmus metered 77 cold-climate heat pumps across 65 service accounts and computed an in-situ seasonal efficiency that, in their words, is "comparable to nameplate HSPF" and, "consistent with nameplate HSPF, the ratio does not include standby mode energy".

Against an average nameplate of 11.9 HSPF, they measured 10.7 kBtu/kWh. Their per-site realization ratio came to 88% of nameplate. Two slightly different figures appear in that report and both are right: the executive summary quotes roughly 90%, which is simply 10.7 divided by 11.9, while the body quotes 88%, which is the average of each site's own ratio. The range is the part worth carrying away either way. Individual installations landed anywhere from 57% to 119% of their own label. Some beat the rating outright.

A separate line of evidence lands nearby without being the same evidence. Hugh Henderson's white paper for NYSERDA and the New York Department of Public Service modelled seasonal COP for New York conditions using lab data for 3,583 central air-source units and found the seasonal COP averaged 87.7% of the dimensionless HSPF. That is a bin model over catalogue data, not metered houses, so it corroborates the Vermont result rather than confirming it.

And a genuine counterweight sits in the Minnesota work. CEE's overall conclusion was that "ccASHP performed to their rated specifications for both system capacity and efficiency". The same report is candid about why an installed figure often lands lower anyway, noting that the test methods "are intended to compare unit performance, and they are not intended to represent the installed efficiency of any specific installation". A blanket claim that field performance always falls short of the label does not survive the evidence.

What moves the number more than the equipment

Across the datasets, the strongest lever is flow temperature. The Electrification of Heat trial found a mean operating flow temperature of 39.7 degrees Celsius and a clear pattern of lower flow temperatures producing higher SPFs. The CIBSE Journal summary of the HeatpumpMonitor analysis puts numbers on it: systems reaching an SPF of 4.0 or better ran an average flow temperature on the coldest day of 36.6 degrees Celsius, against 39.5 degrees for systems around 3.5.

The effect shows up independently in southern Europe. Mouzeviris and Papakostas computed SCOP under EN 14825 for four Greek cities in a 2020 conference paper, rather than relying on the standard's three continental zones, and found the difference between 35 and 45 degree water outlet temperatures was worth 0.60 to 0.75 SCOP points for units without weather compensation.

Flow temperature is a building question as much as a machine question. Getting it down means enough emitter surface and a low enough heat loss, which is where a room-by-room sizing calculation and an honest heating load figure earn their keep, and where the fabric numbers from a wall U-value build-up feed straight in.

Population effects are the other half. The North American studies span very different intents: the NREL field validation instrumented 12 centrally ducted variable-speed units in IECC climate zones 5 and 6, while Cadmus's New York supplement metered 19 sites and had to sort them into supplemental, primary-with-backup and whole-home use before any average meant anything. A compilation of eleven North American studies assembled by Massachusetts state senator Will Brownsberger reaches the view that seasonal COPs generally land materially below rated performance, which is a defensible reading of the North American data even though the Minnesota and Vermont results above show it is not universal.

On the question that started most of these trials, the field data is fairly settled. Gibb, Rosenow, Lowes and Hewitt concluded in a 2023 Joule commentary that "well below 0 degrees C, heat pump efficiency is still significantly higher than fossil fuel and electric resistive heating systems at an appliance level", and that even near minus 30 degrees C field data shows efficiencies up to double those of resistive heating, while noting more analysis is required. The Catapult figures were widely reported at the interim stage, including by pv magazine, which quoted a mean COP of 2.44 on the coldest days.

So what number should you actually use

Put a whole-system annual figure into a payback calculation, not a peak and not a label. If you are in a mild maritime climate with a well-commissioned system on low flow temperatures, the metered evidence supports something in the high 2s to high 3s at the H4 boundary. If a fossil or resistance backup will carry part of your load below a lockout, the number that governs your bill is the one with that backup counted in, and Minnesota shows it can land near 1.2.

The practical move is to run it twice. Take the pessimistic case and the optimistic case, put both through the payback calculator, and look at the gap between the two break-even years. If the decision flips between them, the thing worth spending money on is not a better machine. It is a lower flow temperature and a better commissioned install, which is what every one of these datasets is quietly pointing at.

FAQ