ARMy of Servers and Project Moonshot

In my first blog post about building out servers with ARM processors, I had mentioned that one could build a high-density scale-out server infrastructure by fitting 20 blades into a 3U chassis:

How would 20 ARM-based server nodes be able to fit into a 3U chassis? It has been done before, even with nodes that pull more than 15-20W each. Both Sun Microsystems and Compaq had a 3U blade server chassis that supported 20 blades (with one UltraSPARC IIe/IIi or one Pentium M processor, 1-2GB of RAM and one 2.5″ hard drive bay) and an Ethernet switch or pass-through module. One can use the same blade setup, use smaller and more efficient power supplies and cooling (as the need for power and cooling with be a lot less than 20W per blade), update the switch to support Gigabit downstream ports and 10Gbps Ethernet uplink ports, and reduce the chassis depth. I would even bet that one could find a way to fit 20 nodes in 2U of space without sacrificing any functionality or availability.

Well, HP has taken that idea and ratcheted it up to a very impressive scale. Project Moonshot crams 72 server nodes into a 2U half-width tray housing eighteen Calxeda EnergyCards and four external 10Gb/s XAUI ports. Four of those trays can slot into a 4U SL6500 chassis, for a total of 288 nodes (72 nodes per 1U). Each EnergyCard contains four EnergyCore processors with up to 4GB per processor and 4 SATA ports, all while drawing 25 watts. In turn, each EnergyCore processor comes in both two and four Cortex-A9 core configurations and has a high-throughput fabric switch built-in. The fabric switch provides multiplexed access to five 10Gb/s XAUI ports and six 1Gb/s SGMII ports, all of which is wrapped around by three 10GbE MAC ports. Each EnergyCore also provides five SATA ports, several PCIe controllers and an SD/eMMC controller (say, for booting an operating system).

While such an impressive setup may not immediately fit into common enterprise workloads, don’t be surprised to see these things popping up at places where companies need an enormous amount of light/moderate duty workloads that can be scaled out across thousands of threads; and, where the cost of power and cooling are at a premium. Both Ubuntu and Fedora can be used, though Windows Server 8 might be an option if Microsoft deems it to be worth the time and money.

Having fun with the Portland I-405 Loop

Outside of the sometimes stupefying drivers and traffic issues during rush hour, I love driving and cruising… and no, it didn’t spontaneously happen when I got my S5. It started ever since I got my license and a car. I guess it was a way for me to get out of the house, clear my mind and see the sights of the Portland metro area and the Willamette Valley. Of course, my love for it grew even more each time I got a better car: a 1999 Ford Taurus SHO, my 2004 Audi A4 and now the Blue Bomber, er Dalek-5, 2011 Audi S5.

Anyway, one of the routes that I like to take when I just want to just drive is the Portland Loop that is I-405. The first loop starts when I merge on to I-405 N from US Hwy 26 E, then drive up until I-405 N ends in a fork. The three-way fork gives you the option to continue on to I-5 N, out to Kirby Drive or the first kink in the loop, I-84 by way of I-5 S.

Of course, it makes sense as drivers that are dropped on to I-405 N would need a way to get on to I-5 S and I-84, but there’s an easier way for those coming from the Sunset Tunnel. Anyway, I continue on down I-5 S for a while, passing by the Rose Garden exit, the exit for Hwy 99E and come back to another fork in the road. This time, it is to either continue down I-5 S or head back up on to I-405 N. Continue on I-405 N and pass the Everett exit and I’ve just finished the first lap of the Portland Loop, I-405 N edition.

The second version of the loop starts off with merging on to I-405 S from US Hwy 26 E, then almost immediately cut over to I-5 N. If it were any other day that I take that route, I would normally jump off to I-84 E, but I would continue on I-5 N to drive this loop. I drive on, passing by the Rose Garden and take the I-405 S exit. To wrap up the first lap of the Portland Loop, I-405 S edition, I continue on down, skipping the US Hwy 26 W exit and jump back on to I-5 N.

If I am absolutely bored out of my mind and just want to kill time, I will do about 2-3 laps of the I-405 N edition. I usually do not do the I-405 S edition unless if I already coming down I-5 S from Vancouver or a jaunt out on Marine Drive. I have also not timed any of my lap runs, though I have considered recording the runs if I could get my hands on a GoPro HD camera mounted on the dash of my S5.

You don’t have to have a car from one of the Bavarian car maker to enjoy the Portland Loop. In fact, it can even be done in a cargo van, a city car, a motorcycle or a proper EV.

Facebook and Open Compute Project

I just want to get this out in the open, I am not a Facebook (or Twitter) user nor have I been thrilled with the fast-and-loose nature of Facebook’s handling of public or private user information.

In an interesting twist to Facebook’s “openness”, Facebook has released details and documents of their Open Compute Project that is the basis for their recent datacenter located in Prineville, Oregon. The specifications have been released by Facebook under the Open Web Foundation Agreement, while the design and implementation files are released under the Creative Commons Attribution 3.0 license.  Being a hardware-nut and have always been interested in datacenter design and architecture, the released server, rack and power component specifications made my day.

To me, it is very nice to see what was done to make the datacenter as lean and mean as possible, mostly the power supply used by each server node. The 450W power supply has two inputs, a nominal 277VAC input as primary and a 48VDC input as a backup. For typical servers, you would have two separate AC or DC power supplies that should pull power from two different sources (such as independent UPS or circuits). There is inherent inefficiencies with this setup as you have to deal with additional losses due to having two sets of conversion components, increased cooling, and an external interposer.

Right now, the two server boards that have had their designs and specifications released are both dual socket boards. The first board is a high-memory capacity board that can take two AMD Opteron 6100-series processors and has 24 memory slots evenly distributed between the two sockets. The second board uses Intel Xeon 5500/5600-series processors and has the common 18 memory slots (9 per socket, 3 slots per channel) setup found in most 1U and 2U servers. Both boards are custom designs and only have the required components included, such as SATA, USB and Gigabit Ethernet. Efficiency, cost savings and simplicity are the reasons for the stark nature of these boards, which is the priority for large scale-out compute systems.

Another unique feature of both boards is how power is provided; rather than using the more common ATX-style connectors (which are available for testing purposes), the designs call for a edge-mounted connector that is more likely found in Cisco Catalyst 6500-series modules or various blade servers. Again, simplicity and efficiency are critical.

As intriguing as the hardware is, it may not be as practical in more common small or medium business environments. Nonetheless, some of the design elements are already found in blade servers or industrial applications. It is the datacenter design elements that will have more of an impact in the near future, as IT continues its march towards cost and energy efficiency.

Edit: BTW, how cool are LED lights powered over Ethernet?

A Rack of ARMs: 16 cores in 1U

If you have been following my blog recently, you will probably have seen my advocacy of using servers with ARM processors in large scale-out environments due to their small form factor and very low power consumption. Both of which allows a single rack to host hundreds, if not thousands, of compute threads and increase application resilience; since, one or two downed nodes in a cluster of hundreds of nodes doesn’t have the same impact as one or two large virtualization hosts within a cluster of tens of nodes.

One new offering in the ARM server market is ZT System’s R1801e 1U rackmount chassis that houses eight nodes (or “System on Modules” in ZT-speak), one 80GB SSD per node, consolidated network by way of two integrated 4+1 Gigabit Ethernet switches, and is pre-loaded with Ubuntu Server. Each node has an ARM processor with dual Cortex-A9 cores, 1GB of DDR3 memory, 1GB of flash and the typical USB and SATA connections found in common systems. Read the rest of this entry »

A new look at cloud scale-out with ARM processors

With the rise of the smartphones and dependencies on web applications and the cloud, there has been one processor architecture that has taken the charge: ARM. Processors based on the ARM architecture have found their way into Apple iPhone, iPad and iPod Touch devices, Android smartphones and tablets, upcoming Chrome OS netbooks and tablets, home network routers and home multimedia hubs. Some of the reasons for the proliferation of ARM-based processors include: low cost, low-to-very-low power consumption, decent processing power, and open development environment. Read the rest of this entry »