As NVMe approaches price parity with SATA SSDs, we have been receiving a growing number of requests for Minimus servers with NVMe SSD’s. While designing these servers, we’ve learned some interesting facts about NVMe and why it may be time to ditch SATA. Here is what we found:
Most HPC buyers look to get the most performance for the lowest budget. As mentioned in an earlier post 'Server Cost Breakdown By Components', processors are normally the most expensive component in a cluster, this is especially true for non-GPU / Co-processor accelerated clusters. Picking the right CPU is critical.
Written by Kaitlin Krull from Modernize.
There are a couple of important points for those managing a data center:
- Pricing can change fast, especially when a new line of processors chooses to support only certain standards. For example, when Intel E5-26xx v4 called out DDR4-2400 vs the DDR4-2133 for Intel E5-26xx v3.
- While a specific model may go up or down, the overall trend in memory prices are downward. In some cases this is as much as 30% in 4 months.
- Keeping on top of current memory is a constant battle. For example, we just updated minimusservers.com with significantly lower pricing despite being online for just over 2 months.
- The current price for DDR4-2400 ECC memory is around $4.70 per GB.
Note: this is Minimus pricing, pricing form other sources may be higher.
Figure 1: Price trends of 4GB DDR4-2400 from Jan 2015 to June 2016
Image credit: pcpartpicker.com
Comparing the pricing trends for two specific models of 16 GB DDR4 2133 from the same brand (Kingston), the pricing for one model is a lot more stable than the other.
Figure 2: Volatile price of the Kingston KVR21R15D4/16 (16 GB DDR4-2133)
Image credit: pcpartpicker.com
Figure 3: Relatively stable price trend of the Kingston KTH-PL421/16G (16 GB DDR4-2133)
Image credit: pcpartpicker.com
There are several conflicting factors that affect the daily price variance of memory:
- The cost of producing memory, like nearly all silicon based computer components, is largely from enormous upfront capital costs amortized over time. The marginal cost is actually quite low.
- When a new specification is released, pricing starts out high because of low demand, little competition, and low manufacturing yield. As a product matures and gains adoption, prices drop.
- Supply and demand does play a big role. Prices can change daily if demand is high or market conditions change.
- Manufacturers often increase their prices on specific models to incentivize customers to buy different models that cost less to produce. Searching against a specific configuration, you can see that memory costs will vary as much as 300% despite the same performance specification.
Check out the Minimus formula for cost savings and see a price comparison:
Even if you aren’t considering raising your data center set point, consider that inlet air temperature varies significantly from top to bottom of the rack. Airflow engineering attempts to even out airflow across the room, but a variation of 10°F is still not uncommon.
To find the effect of higher temperatures, GRC decided to measure the power of a server with varying input air – and you should, too. Using a power meter and thermocouple, we gradually increased the amount of recirculation to increase the input temperature into the server.
*Note: Raising the air set point is to save power on the chiller. We can’t test that affect because GRC data centers are chiller-less.
The following data is taken from a 2009 Dell R410, populated with a single, low power processor. Since the chassis is lightly loaded, the fans aren’t working strenuously. Fan power in other chassis is shown in Table 3 below.
Table 1: Dell R410 (2009) lightly loaded with a single processor, representing relatively low fan power
|Ambient Temperature (ºC)||Ambient Temperature (ºF)||Power (Watts)||Increase (∆ 27ºC)|
Table 2: Distributed Computing Load and Air Temperature
We found the server power increased logarithmically with input temperature. We also measured the difference between servers in air versus servers submerged in GRC cooling equipment.
Table 3: Power Savings from Submerging in Oil
|Brand||Model||Size||Power Savings in Oil*|
|Dell||1950, gen III||1U||30%|
|Dell||C6100c||2U, 8 CPU||17%|
With your servers, how much power could you save by running the fans slower? How much if you submerged the servers and removed the fans altogether?
To learn more about Green Revolution Cooling's solutions call us at +1 512-692-8003 or contact us here.
Transformer Efficiency: Is it worth it?PUE often encompasses all of the losses from the utility down to the server. This includes transformers that convert power down to 400 or 208, 3 phase power. To get an understanding of your "total" efficiency, transformer efficiency and costs have to first be understood.
Here are few things of interest to data center operators:
- There are several factors that affect transformer inefficiency (electrical losses):
- Hysteresis due to charging the field in the iron core (which is NOT proportional to load).
- Line losses through the coil (which are proportional to load).
- There are several different type of transformers:
- "Standard temperature rise" – More efficient at partial loads, where data centers actually operate.
- "Low temperature rise" – This means there is a low surface temperature (less heat coming off the transformer). These transformers have larger iron cores, which make them more efficient at transferring energy at high loads, but less efficient at partial loads, where data centers actually operate.
- “DOE 2016” – In the US, the Department of Energy released a new efficiency standard for all transformers sold. These transformers are more efficient across the board.
Figure 1: Transformer efficiency curves for standard, low temperature, and DOE 2016 type transformers
- Different buyers get different pricing* based on volume, type of organization, application, relationship with sales rep, etc. Big companies can end up spending as much as 40% less than small businesses.
- CPUs are a dominant cost and are rarely negotiable
- Memory and storage pricing is relatively constant between vendors after discounts
Server Cost BreakdownNote: the prices in the below table have been sourced from popular electronics vendor websites, and are not necessarily indicative of Minimus pricing.
|Component||Quantity||Price Estimate||Extended Total|
|CPU: E5-2650v4 (12 CORE)||2||$1,116.00||$2,232.00|
|Memory: 16GB, DDR4-2400, ECC||8||$95.00||$760.00|
|Storage: Boot SSD, 120GB||1||$119.00||$119.00|
|Storage: 480GB, Medium Endurance SSD||2||$391.97||$783.94|
|Network Card: None||0||$0.00||$0.00|
|Motherboard: Dual Socket E5-26xx v4, 8 Memory DIMMS, On-board Network: 2x RJ45 Gbe LAN Ports, 1x RJ45 IPMI LAN Port||1||$299.00||$299.00|
|CPU Heat Sink||2||$14.00||$28.00|
|Sheet Metal Case||1||$100.00||$100.00|
|Assembly Labor and Test||1||$150.00||$150.00|
About GRC and Minimus ServersGreen Revolution Cooling has introduced Minimus Servers in partnership with hardware giants Supermicro and Gigabyte. The Minimus platform offers reliable, purpose-built servers for a fraction of the cost of OEM servers. The core of Minimus’s price-effectiveness is that customers get exactly the configuration they need, but without the added costs of extraneous features and brand-related premiums that are normally tacked onto OEM offerings.
To learn more about Minimus Servers contact us at email@example.com or call +1 512-692-8003.
Designed in partnership with high-quality OEM manufacturers like Supermicro and Gigabyte, the Minimus Server is a reliable, low-cost server that offers savings of 50% to 60% compared to Dell.com.
Why Minimus? Why now?For years, Green Revolution Cooling has been helping customers design and build their own custom hardware so that they are able to take full advantage of their immersion cooling technology. Since GRC has deployed thousands upon thousands of these servers, paired with the fact that they boast a less than 1% failure rate, we're making the Minimus architecture widely available to the entire market. In conjunction with immersion cooling, Minimus Servers make data centers more efficient, more cost-effective, and easier to deploy - simple as that.
How do Minimus Servers cost a fraction of average OEM prices? Take a look at the image below.
Don't pay for unnecessary components with fully customizable Minimus Servers
How to save 50% to 60% off over Dell.comThe Minimus offers you the ability to pick and choose the parts and features you need for your application like processors, memory, etc., while eliminating all the extraneous parts and features such as fans, intricate chassis, architecture for redundant power supplies, hard drive caddies, etc.
The result is a lean, low-cost server that is custom-built for your application.
High ReliabilityThousands of Minimus-designed servers have already been deployed in data centers around the world and are performing efficiently. In addition, they have proven to be some of the most reliable servers in application. This high reliability comes from the use of premium components from quality manufacturers like Supermicro, Gigabyte, Intel, and others, used in conjunction with naturally protective immersion cooling environments. All of these factors are what contribute to the Minimus’s unmatched reliability.
The new metrics for data center efficiency and sustainability
As always, our All You Need to Know About... series just scratches the surface of a different subject for each entry, and to dive deep into all the data and specifics you'll want to download our corresponding White Paper "A Paradigm Shift in Data Center Sustainability".
Read the full White Paper on Data Center Sustainability
The Green Grid defines the following metrics for measuring the efficiency of a data center in terms of its water use and carbon emissions:
- WUEsite = Annual Site Water Usage / IT Equipment Energy (in L/kWh)
- WUEsource = Annual Energy Source Water Usage + Annual Site Water Usage / IT Equipment Energy (in L/kWh)
- CUE = Total CO2 Emissions Caused by the Total Data Center Energy / IT Equipment Energy (kgCO2eq/ kWh)
Download A Paradigm Shift in Data Center Sustainability White Paper
Have questions? Want to see how the CarnotJet system can cut your data center energy and costs? Contact us via a email at firstname.lastname@example.org or by phone at +1(512) 692-8003.
Today's post continues our series All You Need to Know About... with a "deep dive" into the magical liquid that makes it all possible – ElectroSafe. We'll look at all aspects of our coolant from fire safety to what makes it the perfect liquid for immersion cooling. For further reading, download the ElectroSafe Coolant Fact Sheet and FAQs for a brief one-page overview with information such as the NFPA fire diamond and coolant characteristics such as evaporation rate and density, then give our White Paper on the subject, Submerged Servers in the Data Center a read to discover how safe ElectroSafe really is.