Lately, I have been contemplating when we should come out of stealth mode, and let the crypto-currency mining community know what GRC is doing to help folks take advantage of this opportunity. GRC has acquired significant experience in the space by deploying our patented liquid immersion cooling technology for a number of Bitcoin miners. GRC’s ability to cool these powerful, heat-generating mining computers at over 100kw per rack proved to be most valuable – especially since our technology is extraordinarily power efficient. After all, low electricity cost is one of the key factors in achieving a profitable mining operation. Upon observing that many people wishing to participate in this gold rush have limited IT and Data Center operations experience, we decided to do something a little different. We have also been building containerized data centers for the Air Force, so we decided to mash together our crypto-currency experience with our containerized datacenter experience and voila – the HashTankTM was launched late this summer. The HashTankTM is, first and foremost, a turn-key Bitcoin mining operation. We deliver to our customers a 40 foot ISO shipping container complete with 432 S9 Antminers, racks, power distribution, and cooling infrastructure. The customer just needs to provide a cement pad upon which we place the HashTankTM, as well as power, water, and internet connections. We’ve also formed a partnership with a power company that can provide low cost power rates.
Green Revolution Cooling provides the Texas Advanced Computing Cluster with new GPU-based servers that leverage the cost benefits available with immersion cooling.
Green Revolution Cooling (GRC) today announced that it will be delivering a custom GPU-based cluster to the Texas Advanced Computing Center (TACC). TACC has been a long-time advocate and user of GRC’s immersion cooling technology dating back to 2009, when the technology was first brought to market. The installation has expanded over the years to include more racks and this new order will further build on the existing deployment.
GRC's Liquid immersion computing system is the ultra-efficient computing cluster at John Paul Catholic University in San Diego, CA.
[October 24, 2017 – Austin, TX] Green Revolution Cooling (GRC), the leader in immersion cooling, today announced the latest installation of its liquid immersion computing system at John Paul Catholic University (JPCU) in San Diego, CA. The high density cluster made up of industry standard servers optimized for immersion, and built in partnership with Supermicro, exploit the enhanced performance and compelling economic value inherent with GRC liquid immersion technology.
“Virtually limitless cooling, industry leading efficiency, and lower upfront cost, made it an absolute no brainer for us,” said Kevin Meziere, VP of Technology at JPCU “the ability to cool over 100kW [per rack] effectively and efficiently, gives us the ability to be more responsive to future technology trends and adopt more powerful hardware without concerns of cooling capacity and costs. Being a downtown campus every square inch counts, so the ability to pack equipment more densly is a huge win.”
PIC in Barcelona, Spain shares key performance and reliability results from its immersion cooled data-intensive cluster.
[April 19, 2017 – Austin, TX] Green Revolution Cooling (GRC), a leader in immersion cooling, today announced key performance and reliability results from its installation at Port d’Informació Científica (PIC) in Barcelona, Spain. The ultra-efficient cluster installed in October 2015 has since been used to process dozens of Petabytes of data from CERN's Large Hadron Collider, and leading-edge astrophysics projects.
What’s a CIO to do? Business Unit leaders are notoriously unpredictable in forecasting their demands for new applications and IT support. Yet, it is critical that you respond quickly to these demands so the company can capture the incremental revenue, cost reduction, and/or customer service improvement that justifies the IT investment. In the past, one way of dealing with this challenge was the over-provisioning of datacenters – essentially assuring that there was always capacity on demand.
A 2015 research report from IDC1 indicates that the market is increasingly abandoning big brand, OEM servers in favor of “whitebox” (unbranded) servers made using parts from ODM suppliers.
- HP’s server sales went down almost 30% between 2010 and 2015.
- Dell’s sever sales were pretty much flat with a 4.5% annual (server) revenue growth. Dell’s sales seem to be increasingly focused on whitebox style servers for hyperscale companies.
- IBM’s server sales are down almost 70% during the same 5 year period (2010-2015).
A separate study released by Lawrence Berkley National Labs2 shows that while the major players are in decline, the market overall is getting bigger with white box servers taking over the market.
There are a couple of important points for those managing a data center:
- Pricing can change fast, especially when a new line of processors chooses to support only certain standards. For example, when Intel E5-26xx v4 called out DDR4-2400 vs the DDR4-2133 for Intel E5-26xx v3.
- While a specific model may go up or down, the overall trend in memory prices are downward. In some cases this is as much as 30% in 4 months.
- Keeping on top of current memory is a constant battle. For example, we just updated minimusservers.com with significantly lower pricing despite being online for just over 2 months.
- The current price for DDR4-2400 ECC memory is around $4.70 per GB.
Note: this is Minimus pricing, pricing form other sources may be higher.
Figure 1: Price trends of 4GB DDR4-2400 from Jan 2015 to June 2016
Image credit: pcpartpicker.com
Comparing the pricing trends for two specific models of 16 GB DDR4 2133 from the same brand (Kingston), the pricing for one model is a lot more stable than the other.
Figure 2: Volatile price of the Kingston KVR21R15D4/16 (16 GB DDR4-2133)
Image credit: pcpartpicker.com
Figure 3: Relatively stable price trend of the Kingston KTH-PL421/16G (16 GB DDR4-2133)
Image credit: pcpartpicker.com
There are several conflicting factors that affect the daily price variance of memory:
- The cost of producing memory, like nearly all silicon based computer components, is largely from enormous upfront capital costs amortized over time. The marginal cost is actually quite low.
- When a new specification is released, pricing starts out high because of low demand, little competition, and low manufacturing yield. As a product matures and gains adoption, prices drop.
- Supply and demand does play a big role. Prices can change daily if demand is high or market conditions change.
- Manufacturers often increase their prices on specific models to incentivize customers to buy different models that cost less to produce. Searching against a specific configuration, you can see that memory costs will vary as much as 300% despite the same performance specification.
Check out the Minimus formula for cost savings and see a price comparison:
Even if you aren’t considering raising your data center set point, consider that inlet air temperature varies significantly from top to bottom of the rack. Airflow engineering attempts to even out airflow across the room, but a variation of 10°F is still not uncommon.
To find the effect of higher temperatures, GRC decided to measure the power of a server with varying input air – and you should, too. Using a power meter and thermocouple, we gradually increased the amount of recirculation to increase the input temperature into the server.
*Note: Raising the air set point is to save power on the chiller. We can’t test that affect because GRC data centers are chiller-less.
The following data is taken from a 2009 Dell R410, populated with a single, low power processor. Since the chassis is lightly loaded, the fans aren’t working strenuously. Fan power in other chassis is shown in Table 3 below.
Table 1: Dell R410 (2009) lightly loaded with a single processor, representing relatively low fan power
|Ambient Temperature (ºC)||Ambient Temperature (ºF)||Power (Watts)||Increase (∆ 27ºC)|
Table 2: Distributed Computing Load and Air Temperature
We found the server power increased logarithmically with input temperature. We also measured the difference between servers in air versus servers submerged in GRC cooling equipment.
Table 3: Power Savings from Submerging in Oil
|Brand||Model||Size||Power Savings in Oil*|
|Dell||1950, gen III||1U||30%|
|Dell||C6100c||2U, 8 CPU||17%|
With your servers, how much power could you save by running the fans slower? How much if you submerged the servers and removed the fans altogether?
To learn more about Green Revolution Cooling's solutions call us at +1 512-692-8003 or contact us here.
- Different buyers get different pricing* based on volume, type of organization, application, relationship with sales rep, etc. Big companies can end up spending as much as 40% less than small businesses.
- CPUs are a dominant cost and are rarely negotiable
- Memory and storage pricing is relatively constant between vendors after discounts
Server Cost BreakdownNote: the prices in the below table have been sourced from popular electronics vendor websites, and are not necessarily indicative of Minimus pricing.
|Component||Quantity||Price Estimate||Extended Total|
|CPU: E5-2650v4 (12 CORE)||2||$1,116.00||$2,232.00|
|Memory: 16GB, DDR4-2400, ECC||8||$95.00||$760.00|
|Storage: Boot SSD, 120GB||1||$119.00||$119.00|
|Storage: 480GB, Medium Endurance SSD||2||$391.97||$783.94|
|Network Card: None||0||$0.00||$0.00|
|Motherboard: Dual Socket E5-26xx v4, 8 Memory DIMMS, On-board Network: 2x RJ45 Gbe LAN Ports, 1x RJ45 IPMI LAN Port||1||$299.00||$299.00|
|CPU Heat Sink||2||$14.00||$28.00|
|Sheet Metal Case||1||$100.00||$100.00|
|Assembly Labor and Test||1||$150.00||$150.00|
About GRC and Minimus ServersGreen Revolution Cooling has introduced Minimus Servers in partnership with hardware giants Supermicro and Gigabyte. The Minimus platform offers reliable, purpose-built servers for a fraction of the cost of OEM servers. The core of Minimus’s price-effectiveness is that customers get exactly the configuration they need, but without the added costs of extraneous features and brand-related premiums that are normally tacked onto OEM offerings.
To learn more about Minimus Servers contact us at firstname.lastname@example.org or call +1 512-692-8003.