A New System For Keeping Supercomputers Cool

Researchers have developed a new type of cooling system for high-performance radars and supercomputers that circulates a liquid coolant directly into electronic chips through an intricate series of tiny microchannels.

Conventional chip-cooling methods use finned metal plates called heat sinks, which are attached to computer chips to dissipate heat. Such attachment methods, however, do not remove heat efficiently enough for an emerging class of high-performance electronics, said Suresh V. Garimella, who is principal investigator for the project and the Goodson Distinguished Professor of Mechanical Engineering at Purdue University.

New advanced cooling technologies will be needed for high-performance electronics that contain three-dimensional stacks of processing chips instead of a single, flat-profile chip. Too much heat hinders the performance of electronic chips or damages the tiny circuitry, especially in small ‘hot spots’.

“You can pack only so much computing power into a single chip, so stacking chips on top of each other is one way of increasing performance,” said Justin A. Weibel, a research associate professor in Purdue’s School of Mechanical Engineering, and co-investigator on the project.

“This presents a cooling challenge because if you have layers of many chips, normally each one of these would have its own system attached on top of it to draw out heat. As soon as you have even two chips stacked on top of each other the bottom one has to operate with significantly less power because it can’t be cooled directly.”

The solution is to create a cooling system that is embedded within the stack of chips. The work has been funded with a four-year grant issued in 2013 totaling around $2m from the U.S. Defense Advanced Research Projects Agency (DARPA). New findings are detailed in a paper appearing on Oct. 12 in the International Journal of Heat and Mass Transfer.

“I think for the first time we have shown a proof of concept for embedded cooling for Department of Defense and potential commercial applications,” Garimella said. “This transformative approach has great promise for use in radar electronics, as well as in high-performance supercomputers. In this paper, we have demonstrated the technology and the unprecedented performance it provides.”

A fundamental requirement stipulated by DARPA is the ability to handle chips generating a kilowatt of heat per square centimeter, more than ten times greater than in conventional high-performance computers.

“This number of 1,000W per square centimeter is sort of a Holy Grail of microcooling, and we’ve demonstrated this capability in a functioning system with an electrically insulated liquid,” Garimella said.

Much of the integration and testing of the system was performed by Purdue doctoral student Kevin Drummond. Key to fabrication of the devices used in the demonstration were teams led by co-investigators David Janes, a professor of electrical and computer engineering, and Dimitrios Peroulis, a professor of electrical and computer engineering and Deputy Director of the Birck Nanotechnology Center in Purdue’s Discovery Park.

The team has presented preliminary findings in several conference papers during the course of the project. The researchers received a best paper award last year in the emerging technologies category at the IEEE-ITherm conference, and additional papers will be published, Garimella said.

HFE-7100

The system uses a commercial refrigerant called HFE-7100, a dielectric, or electrically insulating fluid, meaning it won’t cause short circuits in the electronics. As the fluid circulates over the heat source, it boils inside the microchannels.

“Allowing the liquid to boil dramatically increases how much heat can be removed, compared to simply heating a liquid to below its boiling point,” he said.

The team created an elaborate testing apparatus that simulates the heat generated by real devices. An array of heaters and temperature sensors allow the researchers to test the system under a range of conditions, including the effects of hot spots. The testing system was fabricated at the Birck Nanotechnology Center.

The new approach improves efficiency by eliminating the need to attach cooling devices to chips.

“Any time you are attaching heat sinks to the chip there are a lot of resistances and inefficiencies associated with that interface,” Garimella said.

This interfacial, or ‘parasitic’, thermal resistance limits the performance of heat sinks.

“We are going to a technology that eliminates those interfaces because the cooling is occurring inside the chips,” Weibel said.

Using ultra-small channels allows for high performance.

“It’s been known for a long time that the smaller the channel the higher the heat-transfer performance,” Drummond said. “We are going down to 15 or ten microns in channel width, which is about ten times smaller than what is typical for microchannel cooling technologies.”

The new design solves one major obstacle to perfecting such systems: although using ultra-small channels increases the cooling performance, it is difficult to pump the required rates of liquid flow through the tiny microchannels. The Purdue team overcame this problem by designing a system of short, parallel channels instead of long channels stretching across the entire length of the chip. A special “hierarchical” manifold distributes the flow of coolant through these channels.

“So, instead of a channel being 5,000 microns in length, we shorten it to 250 microns long,” Garimella said. “The total length of the channel is the same, but it is now fed in discrete segments, and this prevents major pressure drops. So this represents a different paradigm.”

Peroulis and his students handled fabrication of the channels, a task made especially difficult by the need for “high aspect ratios,” meaning the microscopic grooves are far deeper than they are wide. The channels were etched in silicon with a width of about 15 microns but a depth of up to 300 microns.

“So, they are about 20 times as deep as they are wide, which is a non-trivial challenge from a fabrication perspective, particularly for repeatable and low-cost manufacturing processes,” Peroulis said.

Janes and his students designed and built the intricate heating and sensing portions of the testing apparatus.

“It is a complex task to be able to simulate the generation of hotspots and different heating scenarios while simultaneously having an accurate measure of the temperatures” Janes said.

Other members of the team focused on computational models to describe the physics of the cooling technology.

The new journal paper was authored by Drummond; doctoral student Doosan Back; Michael D. Sinanis, a manufacturing engineer and process development manager; Janes; Peroulis; Weibel and Garimella. Although the team has recently completed the DARPA-funded project, the overall research is ongoing.

The technology has evolved from work originating in the Purdue-based National Science Foundation Cooling Technologies Research Center. The center, formed in 1999, is a consortium of corporations, government laboratories and the university working to overcome heat-generation problems in electronic systems by developing new compact cooling technologies. More than 60 undergraduate students and about 100 graduate students have performed research through the center. The center also has involved about 15 Purdue faculty members from a variety of fields, from electrical engineering to chemistry.

In 2011, Garimella received the NSF Industry/University Cooperative Research Center Association’s Alexander Schwarzkopf Prize for Technological Innovation on behalf of the cooling-research center. Earlier, Indiana’s 21st Century Research and Technology Fund provided $3.8 million to help commercialize an advanced cooling system for hybrid and electric cars. Research in the center conducted in partnership with Toyota in 2016 led to an R&D 100 Award, given annually to the top 100 innovations recognized by R&D Magazine.