Rethinking AI data centre cooling

Brooks Henderson of Phononic explains why cooling must be viewed as part of the power delivery and efficiency equation, with thermal management emerging as a core design parameter alongside power and performance.

The AI revolution has created a new bottleneck in computing infrastructure, and it is not necessarily processing power.

For decades, semiconductor innovation focused primarily on delivering more transistors, greater bandwidth and higher computational throughput. Today, however, the industry’s ability to exploit those advances increasingly depends on something far more fundamental: heat.

As GPUs approach kilowatt-level power consumption, high-bandwidth memory (HBM) stacks become denser, and co-packaged optics move closer to compute devices, thermal management is emerging as one of the defining engineering challenges of the AI era.

According to solid state cooling specialist Phononic, thermal physics is now setting many of the practical performance limits for AI infrastructure. The company argues that the next major gains in compute performance, energy efficiency and infrastructure utilisation may come not from ever more powerful silicon alone, but from smarter and more precise thermal control.

The march toward the megawatt rack
The scale of AI infrastructure is changing at an extraordinary pace. What only recently were considered high-density racks consuming tens of kilowatts are rapidly giving way to systems measured in hundreds of kilowatts. Looking further ahead, the industry is actively discussing one-megawatt AI racks as next-generation deployments push power densities to unprecedented levels.

That growing power demand ultimately feeds a common goal: delivering more AI performance.

As power is distributed from the rack level down to individual devices, its consumption is concentrated in increasingly demanding GPUs, CPUs, networking silicon and memory subsystems. The result is a steady increase in thermal density throughout the data centre.

Modern AI accelerators already consume several hundred watts, with future generations moving closer to the kilowatt range. At the same time, memory bandwidth requirements continue to rise, driving ever more sophisticated HBM architectures and advanced packaging technologies.

The problem is straightforward. Semiconductor devices are designed to operate within tightly controlled temperature envelopes. Once those limits are approached, protective mechanisms begin to take effect.

GPUs can reduce operating frequencies. Memory devices can enter protective refresh modes. System performance is throttled to prevent degradation and preserve reliability.

In other words, the silicon itself is often capable of delivering more performance than the thermal environment allows.

Current cooling strategies largely address this challenge through over-provisioning, employing increasingly sophisticated air and liquid cooling infrastructure to ensure acceptable operating conditions under worst-case scenarios.

Phononic believes that approach is becoming increasingly inefficient.

Targeting the hotspots
The company’s alternative strategy focuses on what it describes as intelligent, localised thermal management.

At the heart of its approach is solid state thermoelectric technology, which differs significantly from conventional mechanical cooling systems.

Unlike compressor-based cooling solutions, thermoelectric devices can both cool and heat. This capability may seem counterintuitive in an AI data centre, but maintaining components within their optimal thermal operating window often requires precise temperature control rather than simply achieving the lowest possible temperature.

Just as importantly, thermoelectric devices are themselves semiconductor components.

This creates opportunities for integration and telemetry that are difficult to achieve with traditional bulk cooling systems. Temperature data, operating conditions and system performance information can be captured directly at the point where thermal management occurs.

For data centre operators, this introduces a new level of visibility into how compute, memory and optical subsystems are behaving in real time.

Rather than viewing cooling as an entirely separate infrastructure layer, the technology effectively embeds thermal intelligence directly into the computing environment.

The HBM challenge
Nowhere is the thermal challenge more apparent than in high-bandwidth memory.

HBM has become one of the critical enablers of modern AI systems. Training and inference workloads require enormous amounts of memory bandwidth, driving continuous advances from HBM3 through HBM4 and into future generations.

However, each performance increase brings a corresponding thermal challenge.

According to Phononic, first-generation cooling solutions were designed around heat flux levels typically associated with HBM3 implementations. Future HBM4 platforms are expected to demand significantly higher cooling capability as power densities continue to increase.

The challenge becomes even more pronounced as packaging technologies evolve.

Advanced 2.5D and 3D integration schemes place compute and memory devices in extremely close physical proximity, delivering substantial gains in performance and bandwidth while simultaneously concentrating heat generation.

Traditional system-level cooling methods can struggle to address these localised hotspots efficiently.

Phononic’s approach combines thermoelectric cooling hardware with software-defined thermal management. Firmware continuously monitors conditions and dynamically adjusts cooling behaviour according to workload requirements.

This allows operators to optimise for different objectives.

Some applications may prioritise maximum performance. Others may focus on maximising reliability and component lifespan. Still others may seek the lowest possible energy consumption. The ability to adapt cooling behaviour to specific operational requirements represents an important departure from fixed cooling architectures.

The hidden cost of thermal throttling
One of the most significant consequences of inadequate thermal management is performance loss through throttling.

As HBM temperatures rise, memory devices can transition into protective operating modes designed to prevent long-term damage. While these mechanisms preserve reliability, they also reduce available performance.

For AI workloads, this can have a substantial impact.

Modern training and inference systems are highly dependent on memory bandwidth. Any reduction in memory performance can create bottlenecks that diminish the utilisation of expensive accelerators. Phononic argues that targeted hotspot cooling offers a direct path to improving effective silicon utilisation. By keeping memory devices within optimal temperature ranges, throttling events can be reduced or eliminated. GPUs and memory systems maintain peak performance for longer periods, allowing operators to extract greater value from existing hardware investments.

This relationship highlights an increasingly important shift in data centre thinking.

Raw chip specifications alone no longer define actual performance. Instead, realised performance increasingly depends on maintaining thermal conditions that allow devices to operate as intended.

From this perspective, cooling becomes an active contributor to computational output rather than simply a background utility.

Beyond device cooling
Improved thermal management also creates opportunities at the system level.

A conventional approach to AI infrastructure often requires highly conservative cooling strategies. Operators maintain aggressively low coolant temperatures to ensure hotspots never approach critical thresholds.

While effective, this approach consumes significant energy.

According to Phononic, targeting localised hotspots can allow operators to increase liquid cooling temperatures elsewhere in the system while maintaining device-level reliability.

The result is the potential for meaningful improvements in Power Usage Effectiveness (PUE) and overall energy efficiency.

Rather than cooling the entire system to accommodate a handful of high-temperature regions, resources can be applied precisely where they are needed most.

For hyperscale operators managing thousands of AI servers, even modest efficiency improvements can translate into substantial operational savings.

Co-packaged optics raise the stakes
Thermal management challenges extend well beyond GPUs and memory.

The rapid emergence of co-packaged optics (CPO) represents another major shift in AI infrastructure design.

As networking bandwidth requirements continue to accelerate toward 1.6T and beyond, traditional pluggable transceivers are increasingly unable to keep pace. The industry is therefore moving optical functions closer to switching silicon through co-packaged architectures.

While this approach addresses bandwidth constraints, it introduces new thermal complexities.

Historically, many optical components resided on switch faceplates where thermal isolation was relatively straightforward. Co-packaged optics bring sensitive photonic devices much closer to high-power electronics.

At the same time, external laser sources are becoming more powerful and generate additional heat loads of their own.

Unlike digital electronics, optical devices often exhibit extreme sensitivity to temperature variation. Small thermal fluctuations can directly affect performance and reliability.

This creates a strong requirement for thermal co-design.

Cooling can no longer be treated as an afterthought once systems have been architected. Instead, thermal considerations must be integrated alongside photonic and electrical design from the outset.

Phononic already has extensive experience in optical temperature control, with millions of thermoelectric cooling devices deployed in laser applications. The company sees this expertise becoming increasingly relevant as co-packaged optics move into mainstream AI infrastructure.

Rethinking cooling as part of power delivery
For power electronics engineers, perhaps the most important shift is conceptual.

Cooling has traditionally been viewed as an external support system, separate from power conversion and distribution infrastructure.

That boundary is beginning to disappear.

Phononic argues that thermal management should increasingly be considered part of the broader power optimisation equation.

Data centre designers typically operate within fixed power budgets. Every watt consumed by cooling infrastructure is a watt unavailable for computation.

However, intelligent thermal management introduces new possibilities. Faster response times, predictive operation and precise hotspot control allow systems to use cooling resources more efficiently.

The energy saved can either reduce operating costs or be redirected toward additional computational capacity.

This transforms cooling from a passive energy consumer into an active mechanism for power allocation and optimisation.

For power engineers, that means thermal design decisions may soon have direct implications for overall system performance and utilisation.

The rise of the thermal fabric
One of Phononic’s most ambitious concepts is what it calls the “Thermal Fabric.”

The idea builds on the growing convergence of telemetry, thermal control and data centre orchestration.

Phononic’s thermal kits combine solid state cooling devices with onboard firmware capable of monitoring temperature, voltage and operating conditions. These individual systems can then communicate through standard management interfaces and Redfish APIs.

Once connected, thermal management becomes part of a broader infrastructure control system. Instead of optimising cooling device by device, operators gain the ability to optimise thermal performance across entire racks or data centres.

Thermal information can potentially be integrated with workload scheduling, virtualisation platforms and power management systems.

This opens the door to new forms of optimisation.

Workloads could be distributed not only according to compute availability but also according to thermal headroom.

Cooling resources could be allocated dynamically where they provide the highest value. Data centres could make energy decisions based on real-time thermal intelligence.

The result would be a much tighter integration between compute, power and cooling domains than exists today.

Economics drive adoption
For hyperscalers, however, technology must ultimately be justified economically. Phononic believes the strongest case for intelligent cooling lies in the combination of performance gains and infrastructure efficiency.

Reducing throttling directly increases workload throughput. In AI environments increasingly measured by tokens generated and processed, higher utilisation translates directly into higher economic value.

At the same time, more efficient cooling can reduce operational expenditure and improve facility efficiency.

Importantly, the benefits operate at multiple levels.

Some value is realised at the semiconductor level through improved device utilisation. Some is captured at the package level through hotspot management. Additional gains emerge at the rack and infrastructure level through reduced cooling overhead.

This layered benefit structure is likely to become increasingly important as AI deployments continue to scale.

Thermal management takes centre stage
Looking ahead, Phononic believes thermal management will become a first-class design parameter alongside power and performance.

The trend appears difficult to avoid.

AI systems are simultaneously increasing power density, adopting advanced packaging technologies and integrating sensitive photonic components. Each development amplifies thermal complexity.

Moreover, hotspots are no longer confined to individual chips. Future infrastructures will contain thermal challenges across GPUs, memory, networking devices, optical components, power supplies and voltage regulation systems.

Managing those hotspots effectively will require faster response times, greater visibility and deeper integration between thermal systems and broader infrastructure management platforms.

In many ways, the industry’s traditional view of cooling is already becoming obsolete. The emerging AI data centre is not simply a collection of processors supported by cooling equipment. It is a tightly coupled system in which compute, power and thermal behaviour continuously influence one another.

As the industry advances toward megawatt-scale AI racks and increasingly sophisticated architectures, the ability to optimise those interactions may become one of the most important competitive advantages available.

The next leap in AI performance may therefore come not from adding more silicon, but from ensuring existing silicon can operate at its full potential. In that future, thermal intelligence could prove just as valuable as computational intelligence itself.

Watch the original interview here.