Is Hot IT a False Economy? An Analysis of Server and Data Center Energy Efficiency as Temperatures Rise

As demand for digital services grows, there is need to improve efficiency and reduce the environmental impact of data centers. The largest energy consumer in any data center is the IT, followed by the systems dedicated to cooling. Aiming to improve efficiency, and driven by metrics like PUE, there is a trend towards running data centers hotter to reduce the cooling energy. There is little research investigating the effect this will have on the IT beyond failure rates. To ensure overall efficiency is improving, we must view the data center as a system of systems, taking a holistic view rather than focusing on individual sub-systems. In this paper we use industry standard benchmarks and a wind-tunnel to profile typical enterprise IT. We analyze the effect of environmental conditions on IT efficiency, showing minor increases in temperature or pressure impact the efficiency of servers. Using an idealized, simulated data center case study we show that the interaction between cooling systems, server behavior and local climate are non-trivial and increasing temperatures has potential to worsen efficiency.


I. INTRODUCTION
D ATA centers are dedicated facilities for housing infor- mation communication technology (ICT) hardware.They provide the fundamental infrastructure that underlies our increasingly digital society.In 2020, data centers accounted for roughly 1% of global energy demand with a further 1.1-1.4% consumed by the data transmission networks associated with them [1].While the growth in energy demand has slowed considerably since the 90% growth boom years of the early 2000s [2], thanks to improvements in energy efficiency, there is still an increasing demand for data center capacity tied to the massive growth in digital utilization.51% of the world's population had access to the world wide web in 2018, and that number is predicted to grow to 66% by 2023 [3].The number of connected devices is also expected to rise to an estimated 29.3 billion [3] with the growth of the Internet of Things (IoT) and smart cities [4].More people and devices online will mean continual growth in the infrastructure that supports this digital environment.
Data centers have a large environmental impact in terms of resources consumed (energy, water, critical raw materials), greenhouse gasses emitted [5], and e-waste generated.One industry accepted practice for reducing energy consumption is widening temperature and humidity ranges, particularly in conjunction with economizers.It is often perceived that operating at higher temperature set-points reduces cooling energy and consequently overall energy consumption.But the effect of this on the IT equipment, specifically servers, has not been well examined and any change in IT load impacts the amount of cooling required.
This paper studies the impact temperature and pressure have on servers and DC cooling energy and the interconnection between the two.We aim to determine whether there is always a net benefit to energy savings in raising inlet temperature across all climate zones.Our primary contributions are novel environmental testing of servers using a wind tunnel, a multi-generational analysis of server behavior under different environments and applying this profiling data to a global case study.
We begin by analyzing the power and performance of 1 U enterprise rack servers with five different CPU generations at varying temperatures and pressures to determine the impact environmental conditions can have on IT.We then extrapolate these findings to a simulated, idealized data center based in London to evaluate the overall effect on a DC and compare different cooling systems.Finally, we extend this simulation to identical data centers in other geographical regions to assess the relationship between the DC and its local climate.
The rest of this paper is arranged as follows.Section II describes the rationale behind this work, emerging trends in data centers and the industry's reluctance to extend operating temperatures.Section III provides an overview of the methodology used to test the servers.Section IV presents the findings of our experiments, followed by a case study presented in Section V examining the interplay of IT and cooling energy in a DC.This is expanded on in Section VI to include the climate in 23 other locations worldwide.Finally, Section VII discusses conclusions and future work.

II. BACKGROUND
Since its inception in 2006 and promotion by the Green Grid in 2007, PUE has been the primary metric used by data centers to continuously measure and improve their site's infrastructural energy efficiency [6].PUE is a measure of the ratio of energy consumed by IT hardware against energy consumed by the entire data center and officially become part of an ISO standard in 2016 [7].Today, some hyper-scale data centers are able to achieve yearly PUEs as low as 1.1 across all their sites [8], [9], while the average PUE globally is down from 2.5 in 2007 to 1.59 in 2020 [10].
Some of these efficiency gains are attained by the implementation of best practices in cooling and airflow management such as those specified in The Green Grid Data Center Maturity Model [11] and the EU Code of Conduct on Data center Efficiency [12].Traditionally, data centers have used air-based cooling systems based on Direct Expansion (DX) or chiller systems.A DX system is comprised of a Computer Room Air Conditioning (CRAC) and air-cooled condenser as a means of removing heat.A chiller cooling solution utilizes a chilled water-glycol mix comprising of a cooling tower or dry cooler for heat rejection which feeds chilled liquid to a Computer Room Air Handling (CRAH).Chiller systems typically incur greater capital expenditures than DX systems but benefit more from economies of scale.
Measures to improve efficiency for air-cooled systems include the use of blanking panels, proper management of raised floor plenums or their alternatives, and implementing aisle containment to prevent the hot and cold air from mixing by creating a physical separation between the upstream (cold) and downstream (hot) portions of a server.This can be achieved with room-based CRAC/CRAH units, although in-row units are becoming more common in aisle-contained DCs also [13].
As well as minimizing the increase in air temperature between the outlet of the CRAC or CRAH unit and the inlet of the servers, the separation of the aisles allows for a pressure gradient to be formed across the servers, potentially aiding in the flow of cool air through the servers [14].This pressure differential can be minimal, requiring the server fans themselves to drive flow, or it can be raised to differentials of 15-25 Pa -putting a greater load on the CRAC but taking the load off the server fans to drive the flow [15].
Today, some data centers also utilize water/liquid-based cooling systems within the room and close to the IT, such as direct-tochip cooling, rear-door heat exchangers and immersion cooling which can be more energy efficient, especially for greater power density -although those are still not widely adopted [16].
Where ambient conditions allow, hyperscalers like Microsoft, Google and Meta are adopting free and evaporative cooling to reduce energy consumption [17] as well as custom IT [18] designed for greater efficiency.
Traditional enterprise data centers lag behind: the industry's average PUE is stagnant around 1.58 [19].Data center environmental bodies have identified over-cooling to be one of the key reasons why this is still the case, and in recent years, these bodies have worked collectively to increase the allowable  C.Even when newer equipment that can support wider ranges is installed, ambient temperatures are often not modified to align with them.This reluctance to extend operating ranges is mainly attributed to concerns over IT hardware reliability, reduced response times to manage cooling failures, warranty issues, and potential decreased server efficiency.A lack of research, industry consensus, and historical data compound these concerns.
1) IT Reliability and Failure Rates: In 2011, ASHRAE investigated relative server failure rates when raising server temperature above 20 • C based on reliability data from multiple hardware vendors [20].Continuous operation at 27 In contrast, a 2008 study by Intel [22] using 900 blade servers over a 10-month period found server reliability was not significantly affected by temperature or humidity, even with relatively poor air quality.In a more recent white paper [23], after running Intel data centers at higher temperatures (32.7 • C) for eight years (2013-2021), they found no increase in component failure.
Pinheiro et al. [24] studied the effect of temperature on hard disk failures in Google data centers and found a drop in disk failure rates with increased temperatures, except for at very high temperatures (45 • C).In 2012, El-Sayed et al. [25] analyzed more data on hard disk replacements at different Google data centers and observed lower disk failure rates than previous models predicted below 50 • C.They have also noted no correlation with higher temperatures for Dynamic Random Access Memory (DRAM) failures and node outages.
2) Concerns Towards Design Set Points, Safety Margins and Hot Spots: With increased inlet temperature, the hot aisle temperature needs to be continuously monitored to ensure that the data center's temperature does not exceed the design parameters for nearby equipment, cables, and hardware.Some rack-based equipment such as power distribution units (PDUs) and network switches have distinct operating envelopes with an upper limit of 45 • C [21] which needs to be considered; though these are typically higher than the same operating ranges for servers.
In addition to hot spots, another common concern among data centers is reduced safety margins.Most servers have a critical temperature threshold and will shut down when that threshold is reached to avoid serious damage.Similarly, CRAC units have temperature thresholds and built-in failure modes that force the unit to shut down when that threshold is crossed.As the ambient temperature in a data center increases, equipment will be operating closer to the maximum temperature, reducing the time for graceful shutdown or taking protective measures in the case of critical failure event [26].
Nonetheless, hotspots and safety margin concerns can be controlled if data centers follow the best practice in cooling and airflow management [12].One of the most effective methods is hot and cold aisles containment.It prevents hot spots and hot air recycling by physically separating the IT exhaust air from the supply air in the DC which can effectively reduce the mixing of hot and cold air and allow for a safe temperature increase.
3) Decreased Efficiencies: Server power tends to increase with temperature, particularly in older hardware, caused by increased fan activity and an increase in silicon electrical leakage current [27], [28], [29].El-Sayed et al. [25] observed a significant increase in power when ambient temperature increased, attributing this mostly to fan power, in an 11th generation Dell PowerEdge R710 2 U rack server (released in 2009).In 2018, Wang et al. [30] observed a correlation between CPU temperature and server power by intermittently cooling a CPU in isolation.Later, they profiled a single server's power consumption under load and up to 45 • C to generate a temperature-aware model [31].Both tests showed increased power consumption under hotter inlet temperatures or reduced cooling.
ASHRAE studied the power consumption against temperature for multiple vendors for class A2 and A3 devices [32], showing that server fan power consumption rises notably past the recommended range (25 • C).Operating at the top of the allowable range could cause A2 class servers to use up to 20% more power than operating at 20 • C.But newer A3 class servers are more efficient and designed for higher inlet temperatures.ASHRAE's data indicate that as temperature rises, the increase in power consumption of a class A3 device is generally only 50% of that of a class A2 device.
The ASHRAE analysis highlights the variation of power consumption among models, vendors and form factors.For instance, 1 U rack servers are at a disadvantage in that they tend to be less efficient than larger servers at higher inlet temperatures because of the smaller size and higher rotational speed of their fans [21].Therefore, these servers are more likely to exhibit higher increases in energy consumption when the inlet temperature rises.Blade servers, on the other hand, typically use larger fans at lower rotational speeds to achieve the same volume of airflow.

III. METHODOLOGY
We will benchmark servers under varying environmental conditions to determine the impact of pressure and temperature on server efficiency.Using an industry-standard benchmarking  tool (SERT) we can monitor power and performance during the benchmark and generate an aggregate efficiency score.To control and vary the environmental conditions we build an experimental low-speed wind-tunnel to emulate the range of temperatures a server might experience in a data center.

A. Wind-Tunnel
The low-speed, closed-loop wind-tunnel was designed to simulate the various thermal conditions seen within a data center and push that envelope beyond ASHRAE-recommended standards.The tunnel, shown in Fig. 1, is based upon the design proposed in [33] and is comprised of the test chamber, a heating and cooling system, flow conditioning geometry, and a fan to modify the pressure drop across the server under test (SUT).
Servers are placed in the center of the test chamber such that the only passage air will take is through the server.The test chamber geometry is shaped so that the air flow through the chamber and into the front of the server is laminar.The depth and width of the chamber are variable and can accommodate servers and blade chassis from 1 U to 12 U.A flow contraction, shown in Fig. 2, based on the design by Bell and Mehta [34] was utilized upstream of the server, with accommodations made to ensure the laminar airflow is maintained for any size of server the tunnel can accommodate.
Maintaining a 5th-order contraction shape was computationally and experimentally determined by Bell and Mehta to be the best in the range of configurations and geometries tested to minimize boundary layer separation, pressure drop and ensure a where H e is exit height, or the height of the server being tested, and H i is inlet height, the full height of the test chamber.The contraction is made from a fixed length of aluminium sheeting.Given the variable nature of the exit height H e , the position of H i must move laterally along the length of the test chamber.To maintain the 5th-order polynomial in (1), the center position of the aluminium sheet sits on a curved slider.
Bell and Mehta determined the minimal acceptable ratio between contraction length and inlet height, a, to be 0.89.
To determine the length of the aluminium contraction sheet and the position of both the end point at H i and the center-point, we treat the contraction sheet as a Bezier curve numerically approximated with Gauss quadrature for a 5th-order polynomial This is transformed from a range of −1 to 1 to a range of 0 to aH i and combined with the Gauss quadrature with (1) leading to Where H i is 12 U or 534 mm based on the height of the tunnel.This was in turn determined by a combination of the standardized ducting size, the expansion ratio of the WAD, and the depth of 12 U of servers, which lead to an L c of 553 mm providing center-point and H i locations along the length test chamber.Under ideal circumstances, deviations in the cross-section of the tunnel should be kept between 10-15 • to minimize any pressure drop or boundary wall separation [35], [36] -practically, this would result in an inordinately long test chamber, so a Wide Angle Diffuser (WAD) was installed upstream of the contraction and server, see Fig. 3.
Typically, a WAD would use honeycomb screens to enforce laminarization on the flow, but we opted for a series of overlapping rows and columns of rods as designed by Barratt and Kim [37], as this allowed for a greater rate of expansion while also minimizing pressure drop and turbulence.
The effectiveness of the WAD is determined by the porosity of the honeycomb, ideally, to minimize turbulence ≥ 0.99.
where d is the diameter in millimeters of rod and S is spacing between rows or columns.Given the physical limitations of the very small diameter columns and rows, a 3 mm steel wire and 38 mm spacing was chosen, resulting in = 0.994.The wire was placed 32 mm apart along the length of the WAD, exactly out of phase to maintain the 38 mm spacing between rods and columns.
Inside the test chamber, the temperature is measured at each end (upstream and downstream of the server) and a pressure differential is measured across the server.The temperature in the tunnel is maintained by a liquid-air heat exchanger attached to an air handling unit.A constant pressure difference across the server analogous to data center aisle containment is maintained by a variable speed fan, placed upstream of the server on the return ducting.

B. SERT Benchmark
SERT is a benchmarking tool created by the Standard Performance Evaluation Corporation (SPEC) [38].It is a benchmark divided into three workloads targeting the primary components of a server: CPU, memory and storage.An efficiency score is calculated for each benchmark as a normalized performance score per Watt of power used.These are amalgamated into a SERT metric using a 65:30:5 weighing ratio of CPU to memory and storage workloads that gives a coarse but directly comparable measurement for the efficiency of a server.

C. Test Details
Our previous research highlighted the substantial impact CPU generation has on the server's overall energy efficiency [39].For this test, we have chosen a selection of typical enterprise servers of each generation spanning a wider range (2012-2017) to determine what effect ambient conditions have, if any, on newer hardware generations.The configurations tested are shown in Table II.We chose to test multiple generations of servers to determine if there were any trends within the wider industry reflected at the server level.In order to compare between generations, the specifications must be similar; however, architectural differences between generations of CPU mean that the technical specifications like clock speed are not directly comparable in a way that indicates performance.To counteract this, we chose CPUs that occupy similar positions in their respective product ranges for their generation.We have both low and high-specification versions of the most recent servers, the R640 with Xeon scalable CPUs.For each of the servers tested, the total memory capacity, disk and power supply configuration remain Temperature variations for the servers are kept within the recommended environmental envelope defined by the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) [20].The PowerEdge R600 servers typically fall into the ASHRAE A2 classification per the Table I.It is possible for them to operate as A3 devices given lower specification levels and certain restrictions.In our case only the PowerEdge R640 with a Intel Xeon Silver 4116 could fall into this category.
To account for any fluctuations in SERT test results, benchmark tests are repeated a minimum of three times per scenario in order to avoid any spurious results.Using the wind-tunnel, each server was benchmarked at three different temperature ranges (20-23.3• C, 23-26.6 • C and 26.6-30 • C) and two differential pressures (<5 Pa and >15 Pa) across the server.The temperature regions were chosen to span SERT's valid ambient temperature range rather than the higher ASHRAE A2 to avoid invalidating the tests and to determine whether the relatively minor changes to the temperature that a DC might be able to achieve without retrofitting specific hardware would have any significant impact.The pressure values were chosen to be at the extremes that could be experienced by the server in an aisle contained environment.Either hot and cold aisles are kept around the same pressure with just enough pressurization in the cold aisle to avoid any flow from the hot side, or the cold aisle can be pressurized, potentially taking some load off the server fans.At low pressure differentials in the wind-tunnel, the server fans play a greater role in driving airflow, relegating the main fan to compensate for the back-pressure of the tunnel geometry.

IV. TESTING RESULTS
Fig. 4 shows the SERT efficiency score of each server configuration for the environmental conditions tested.It is evident that server efficiency has improved significantly over the years, with SERT scores increasing with each generation.Each Pow-erEdge server chassis (R620, R630, R640) is represented twice in Fig. 4. All hardware in the chassis is identical except for the CPU.This demonstrates both how component choice can impact efficiency and the significant impact the CPU has on the server.As inlet temperatures increase, server efficiency decreases for every server tested; 1-2% from 20 • C to 30 • C (p-value < 0.01).In contrast, changes in pressure differential across the server had no significant effect on the efficiency score.

A. Efficiency Detail
The two influences on efficiency scores are performance and power consumption.Fig. 5 shows the power consumption and CPU workload performance for each server and the temperature range limited to the CPU workload.The frequency of each measurement is displayed as the width of the plot, giving an indication of the underlying distribution.The figure shows that the distributions of both power and score are multi-modal due to the server power and performance varying with target utilization in the benchmark, the median power, and score for each target utilization are also shown in the figure .Fig. 5 shows the decline in efficiency as temperature increases is driven by increased power consumption, not reduced performance.The performance of each server increases proportionally to utilization, but power does not.In the older servers, the rate of power increase is greater at higher utilization, whereas the newer R640 server had larger power increases from 0% to 25%.Server power for earlier generations is convex with respect to utilization and concave in the later generations.Considering the relative change in performance, this indicates that aside from environmental considerations, workload allocation strategies between generations of servers offers an opportunity for optimization.
The effects of temperature and pressure are most exaggerated at the extremes of utilization: when the server is idling or running at 100%.Fig. 6 shows the change in power consumption for each server at both extremes compared to the mean power at 20-23.3 • C at low pressure.Increasing inlet temperatures resulted in increased power consumption for every server under load.At idle, the impact is much less though still trending upwards at higher temperatures.Additionally, every higher pressure scenario has slightly improved power consumption against its low-pressure counterpart, though the effect is much smaller than temperature.
The R640 Xeon Silver 4116 is the only A3-class server and shows minimal impact of temperature or pressure under load.It is much more efficient and less sensitive to the environment than the higher power Gold 6148 (Fig. 5) despite its lower performance due to very low power consumption.This indicates that lower power density servers coupled with higher temperatures could realize savings in the cooling system without incurring the reductions in efficiency seen in other servers.

B. Effects on CPU
We can observe how the CPU temperatures respond to changes in the environment to give an indication of whether setpoint changes are influencing CPU behavior.If environmental conditions were to affect the lifespan of the CPU, we would expect to see increased CPU temperatures over the range.Additionally, one potential reason why a hotter environment will cause increased power consumption is due to increased leakage current in the CPU at higher CPU temperatures.Fig. 7 shows the power and temperatures of the CPU during benchmarks reported by the internal registers.There are distinct effects due to the temperature and pressure of the environment, but these are relatively small compared to increasing utilization.Additionally, the changes in CPU temperatures do not perfectly mirror the changes in CPU power and neither does CPU power completely mirror the changes in overall server power (Fig. 6).Therefore, the increase in power consumption is not wholly attributed to CPU leakage current, and there are other components in the server also contributing to the increase.
Every CPU remained below its specified maximum temperature (TCase Max), with the exception of the E5-2690 and E5-2699 v4 at 100%, but there is no increase at higher ambient temperatures.Since there are no changes in performance either (Fig. 5) we can conclude this did not incur any throttling.

C. Internal Server Cooling System
There was no change in CPU temperatures with ambient conditions, indicating the server cooling system has increased fan speeds.This is confirmed in Fig. 8, where fan speed response to increasing utilization is shown in RPM and as a percentage difference normalized against the mean for each utilization in 20-23.3• C at low pressure.A portion of the increased server power can be attributed to increased fan activity; in some cases, fans were spinning 25-50% faster than the base case.Due to the cubic relationship between fan speed and power, this could mean up to a 300% increase in fan power alone.Here, we can see a more exaggerated impact due to pressure than seen elsewhere, since with a higher pressure differential across the server the fans do not need to spin as quickly to generate the airflow.Notably, the A3 server (R640 -Silver 4116) had almost no fan response compared to the other servers.
Changes in fan speed and internal temperatures also impact the change in air temperature across the server, known as delta-T.We noted a typical 1-4 • C range on each of the server's delta-T, though the relationship between CPU temperature, CPU power and fans speed is complex.For the newer servers, R640 and R630 Xeon 2699 v4 the delta-T decreased, indicating the smaller heat flux between the heatsink and that air was impacting cooling to some degree.Meanwhile, older servers, particularly the oldest R620 E5-2690 specification, exhibited the largest increases in delta-T (>4 • C) due to the significant increase in power consumption of that server in the hotter environment.

D. Benchmark Findings
Overall, we see that age, specification, and utilization all impact the server's efficiency and power consumption; these effects are compounded to a lesser but still significant extent by environmental factors like temperature and pressurization.Utilization is the most significant factor overall, and each generation behaves differently as this increases.The other factors compound this behavior in non-trivial ways, but a general trend is power consumption increases with ambient temperature by 2-6% at the highest temperatures, and a saving of 2-4% is available by utilizing higher pressure differentials across servers in aisle contained DCs.

V. CASE STUDY: LONDON
The second half of this paper deals with the real-world implications of the findings of Section IV.If running data centers at higher inlet temperatures do not significantly impact failure rates or performance then the overall energy efficiency improvement depends on the saving in the cooling system outweighing the increased power from IT. Higher inlet temperatures open the door for massive savings and the capability to run data centers in a wider range of climates.

A. Methodology
A simulated idealized homogeneous data center, loosely based on DC3 at LDeX [40], [41] was the basis of our analysis.The Green Grid's Liquid Cooling Total Cost of Ownership Calculation Tool [42] was used to estimate the cooling costs as they account for IT power consumption, power density, room setpoint, different types of air cooling, and local climate.There are few tools available that can perform this analysis, this one was chosen for its ability to estimate these attributes and because its calculations are based on historical trends and established scientific principles.
Using all six server types considered in Section IV, the model data center was comprised of 700 homogeneous rack cabinets, with power density ranging from 10 kW per cabinet to 18.4 kW per cabinet depending on the server.This puts the scenarios considered in the median to upper quartile of power density as reported by Uptime Institute [43].The total IT power consumption is estimated at 7-13MW for a 300,000 sqft area facility.These six homogeneous data centers were then considered with DX or chiller cooling technology at set-points of 21.6 • C, 25 • C, and 28.3 • C, taking the respective power consumptions and delta-Ts recorded for those servers at the set-points into account to calculate IT power consumption and power density for sizing the cooling systems.
The relationship between IT power consumption and cooling power consumption for these various temperature and hardware scenarios was considered at loads of 25%, 50%, 75%, and 100%, with the cooling systems themselves being sized based on 100% IT consumption proportionally scaled to match load percentage.All experimentally derived figures used in this analysis were of the low pressure results, as this seemed most analogous to the capacity of the Green Grid tool.
A total of 144 scenarios were evaluated across four variables: 1) 6 Server configurations: See Table II.

B. Observations
Like for an individual server, as utilization goes up at the DC level, so does the IT energy consumption and thus energy for the cooling systems (Fig. 9).As in Section IV the impact of utilization and temperature varies with server type.The small increase in power consumption with temperature set-point seen at the server level compounds when considered for the DC and has varying levels of impact on cooling requirements.In Fig. 9, we see the overall impact of increasing set-points on the DCs energy consumption.Particularly with DX cooling, we see very little change or even increases in total power for the DC as inlet temperature set-point increases.
1) Difference in Cooling System Behavior Between DX and Chiller: When comparing DX and chiller cooling systems for this case study, the chiller system consumes considerably less energy than DX.Chiller cooling systems typically incur a greater capital expenditure but conversely benefit greatly from economies of scale; the larger the DC, the smaller that capital expenditure is as a proportion of operating costs.
From a performance standpoint, the behavior of the chiller is more susceptible to changes in temperature setpoint than the DX system.As the temperature setpoint rises, the energy consumed by the chiller decreases at a greater rate of change than the DX, leading to a knock-on effect of a greater decrease in total facility energy consumption also.the setpoint to 28 • C. The net effect is shown as a solid line for each method of cooling and for all servers.In many cases, for DX cooling, any savings from reductions in cooling power are negated by the increased IT energy (although significantly this is not always the case, such as the R640 -Gold 6148 at 100% utilization, or the R630 E5-2690 v3 at 50% utilization, both of which result in a net facility saving.)On the other hand, increasing the temperature set point for the chiller is such a large saving that the net total is always less than at 21.6 • C.This speaks to the complexity of the interplay between IT and cooling systems in DC environments and how an awareness of the behavior of the IT is imperative to predicting the requirements of the systems in place to cool it.

VI. CASE STUDY: IMPACT IN GLOBAL MARKETS
As the demand for internet access grows globally, so does the requirement for its underlying infrastructure.Emerging markets in South America, Africa, and Asia face unique challenges afforded by the varied climates of their high population centers, but they also benefit from existing research and lessons learned in their creation and development.This portion of the case study considers transporting the previously discussed model DC to 24 locations across these three continents to analyze how energy consumption varies with climate.The locations were chosen to focus on both emerging markets as well as established markets in hotter climates.
ASHRAE International Climate Zones were developed by ASHRAE for Standard 90.1-2007 [44] to combine seasonal temperature and humidity information for a geographical location into a classification system for use in building management.The original release had a limited number of locations cataloged, but this database was built upon in future releases.For the purposes of this case study, standard 90.1-2016 and its database of 6,443 global locations was used to maintain consistency with the version used in the Green Grid tool.We selected 24 locations included in the 90.1-2016Standard [45], and these are shown in Fig. 11 and Table III.
We have previously considered the interplay between the use and type of server and the impact this IT load has on cooling energy consumption, for these two types of cooling.By reducing the variation in servers down to their raw IT loads in order to more generally consider trends, we can analyze the other side of this coin -the relationship between IT energy consumption and cooling energy consumption as it relates to the different climates and cooling systems.This is in a sense analogous to a partial PUE for a DC in each of these climates, insofar as the gradient of each climate zones scatter points in Fig. 12 represent the ratio of cooling energy consumption to IT energy consumption.A shallow gradient suggests increasing IT power has a minimal impact on cooling and thus would be analogous to a low or "good" partial PUE, whereas a steep gradient demonstrates any change to IT load results in a significant increase in cooling load or a "bad" partial PUE.
At a glance, Fig. 12 seems to suggest all climates see a shallower gradient for this cooling/IT relationship as setpoint increases, although, in practice there is more nuance to this conclusion.
Fig. 13 shows the change in cooling energy consumption against IT load as the setpoint increases from 21.6 • C for both methods of cooling.For DX cooling, the greatest savings occur when increasing temperature setpoint in the warmest countries, such as those with climate zones 0B (hottest and dry) and 0 A (hottest and humid).Conversely, when using a chiller system, the greatest savings to cooling energy consumption occur in the coolest climates.
This can perhaps be explained by considering the way in which these cooling systems maintain a setpoint within the DC.DX systems cool the air for a DC directly using refrigerant cycles within the air handling unit itself and will be running almost constantly against the significant difference in temperature between the setpoint and the ambient air temperature of climates such as Dubai or India.Increasing the setpoint within the DC, in turn, minimizes the temperature gradient a Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.DX system is constantly fighting against and thus lowers the cooling energy requirements.While the same could potentially be true for the refrigerant cycles in a chiller system, the presence of an intermediate coolant -be it water or glycol -means any savings are vastly overshadowed by the benefits of creating a greater temperature gradient between the coolant reservoir and the setpoint.If a water tower full of 10 • C coolant that has reached that temperature because of the ambient air temperature now only needs to cool a DC to 28.3 • C instead of 21.6 • C, the duty cycle of the compressors may drop significantly.
The flip side to this behavior occurs when increasing the setpoint for chiller systems in warmer climates.That some of these climates have average temperatures greater than these setpoints likely means thermal mass of the coolant becomes less of a contributing factor for savings, and in turn the increased IT energy load results in a higher demand for cooling than is saved by increasing setpoint.A similar behavior is likely responsible for the increased cooling seen for 28.3 • C setpoint for the singular 4 C climate represented in this study -that is to say, any energy savings made decreasing the cooling load by increasing setpoint are negated by the increased cooling load requirements of hotter IT.
It is important to take these savings and losses in context, however, for a DC of this size, we calculate chiller cooling to be consistently more efficient than DX cooling.Fig. 14 shows the absolute annualized energy consumption of the total DCcooling and IT -for each of the 24 cities considered in this study and for both DX and chiller cooling.This is taken at 50% utilization of a fabricated server that occupies the median power draw of all six studied servers.This is best summarized by comparing the 'best' DX cooled example -Puerto Montt at 28.6 • C at a total annualized energy cost of 28.07 GWhwith the 'worst' chiller cooled example -Caracas as 28.6 • C at a total annualized energy cost of 19.48 GWh.This shows that in every location studied (for a data center of this size with this configuration) the greatest saving to be had would be a change of cooling system from DX to chiller.Even if this idealized data center did exist, such a change would incur heavy capital expenditure to retrofit the scale of which might outweigh the savings available.The important takeaway is not to underestimate the complexity of the interlocking systems that make up a data center -server type and age, load utilization and power density, type of cooling and setpoint, and geographic location are among a number of factors that need to be taken into consideration when analyzing the energy efficiency of a DC.

VII. CONCLUSION
We have profiled a number of typical enterprise rack servers while in varying environmental conditions within the manufacturer's guidelines and found: 1) regardless of generation, the server's efficiency is impacted by the environment 2) the compute performance remained unaffected 3) temperature had a much larger effect than pressurization, but both effects are measurable.We analyzed how this behavior would impact DCs using a case study, to determine whether the increase in server power would offset gains from increasing DC temperatures.In general, cooling energy decreases with higher inlet temperatures and water-based chillers are more efficient but even in our idealized example, interactions between many factors influenced the DC-level power consumption, including utilization rate, cooling system type, server specification and local climate.Overall there were no universal, guaranteed savings from increased inlet temperatures, increases in IT energy can offset any beneficial decreases in cooling energy and therefore raising temperatures is potentially a false economy.This is particularly true with older IT hardware, but also depends on the local climate and cooling technologies, where we found different responses between DX and WC cooling in hotter climates.Our findings indicate that relationships and interactions between the different systems in the DC are complex and warrant further investigation including more profiling and real-world experiments.While there are intuitive guidelines for increasing efficiency, each DC must collect data and evaluate these changes individually, as it is easy for changes made with the intention of saving energy to have the opposite effect overall.

Fig. 1 .
Fig. 1.Low-speed server wind-tunnel render, showing cut-away test chamber and flow conditioning geometry.

Fig. 2 .
Fig. 2. Variable depth contraction cross section showing the relationship between inlet height (H i ), outlet height (H e ), contraction length (L c ). H e varies in steps of 1 U to accommodate servers of different dimensions.

Fig. 3 .
Fig. 3. Banked wide angled diffuser (WAD) cross-section demonstrating expansion ratio and column/row spacing used to achieve laminar flow going into the test section.

Fig. 5 .
Fig. 5. Distributions of server power and normalized CPU workload scores for each server tested within the temperature ranges and the median power and score for each utilization.

Fig. 6 .
Fig. 6.Mean and standard deviation for the change in power due to the environment for each server at extreme utilizations (idle and 100%).The reference point for each server is the mean power during low pressure tests at 20-23.3 • C. Separate reference points are used for each utilization.

Fig. 7 .
Fig. 7. Mean and 95% confidence intervals for CPU power and temperature as reported by internal CPU performance registers for each target benchmark utilization under the different environmental conditions.

Fig. 8 .
Fig. 8. Mean point estimate and 95% confidence intervals of absolute fan speed and percentage change at different target benchmark utilizations.Percentage change is calculated using the mean at each utilization for each server at 20-23.3 • C and ≤5 Pa pressure as a reference point.

Fig. 9 .
Fig. 9. Cooling and IT energy combined to show total annualized energy for a model data center in London using different servers, cooling technologies (DX:Top, Chiller:Bottom), and set points.In all cases, IT uses proportionally more energy than cooling and with DX cooling (top row) the increase in IT load often outpaces any decrease in cooling when increasing the temperature.

Fig. 10
illustrates the change in energy consumption of the DX and chiller cooling systems as well as IT as a result of raising

Fig. 10 .
Fig. 10.Change in annual energy consumption of IT and cooling infrastructure (DX and WC) at varying total DC utilization when increasing the setpoint from 21.6 • C to 28.6 • C.

Fig. 11 .
Fig. 11.ASHRAE climate zone data applied to a world map, annotated with the 24 cities considered in the case study.

Fig. 12 .
Fig. 12. Modelled cooling energy for DCs comprising the range of servers at different IT loads in each of the chosen locations.

Fig. 13 .
Fig. 13.Expected change in annual energy for each cooling system across the range of servers when increasing the set-point from 21.6 • C.

Fig. 14 .
Fig. 14.Summary table of median total annual energy for the six servers at 50% utilization, comparing DX and WC cooling systems at varying operating temperatures by city.The shared color map indicates that despite different responses between the two cooling technologies, WC cooling uses less energy regardless of the local climate.

TABLE I ASHRAE
[20]CENTER EQUIPMENT ENVIRONMENTAL SPECIFICATIONS operating temperature and humidity ranges in data centers.In 2011, ASHRAE updated its guidelines to define two additional classes of operation, A3 and A4, providing higher allowable temperature boundaries for up to 40 • C and 45 • C, respectively, as shown in TableI[20].

TABLE II SERVER
SPECIFICATIONS TO BE TESTED the same, but memory generation and speeds change between those servers supporting DDR3 and DDR4.

TABLE III EVALUATED
LOCATIONS FOR THE CASE STUDY