"COLLECTIVE INTELLIGENT BRICKScreating distributed applications much easier and less likely to fail because of problems such ascongestion and timeouts. In addition, scalability over a wide range also requires highcommunication performance. Electronics of one prototype brick. IceCube-prototype hardware implementation Storage server prototype The IceCube prototype shown in Figure 1 is a 27-brick, 26- terabyte system. It was developed bythe IBM Almaden Research Center in collaboration with IBM Engineering & TechnologyServices (Mainz, Germany), the IBM Thomas J. Watson Research Center, and many externalvendors. IceCube successfully implements the ideas presented above and should be seen, not as aprotoproduct, but as a proof of concept for brick-based architecture implemented as a 3D cube.7.1. Bricks Following the brick architectures definition, each brick contains a switch, a central processingunit (CPU), and storage for the software and user data. The specific choices made concerning theCPU, operating system (OS), and other components are incidental, as long as the desired brickinterfaces and performance requirements are met. Figure 2 shows the electronics of each brick. At the core of the brick is an eight-port Ethernetswitch chip which supports 1 Gb/s per port. Six of its ports are connected to communicationsdevices on the surfaces of the brick. These are newly invented system-level capacitive couplers,described below. The seventh port is connected to the CPU by an Ethernet-PCI (PeripheralComponent Interface) network interface chip (NIC). It is known that Transmission Control Protocol/Internet Protocol (TCP/IP) processing insoftware puts substantial loads on the CPU [7]; this is in addition to the storage software loaditself. Therefore, a high-performance x86 processor was chosen, supported by a chipset that waswidely used in 2002 and is based on an integrated Northbridge/Southbridge controller chipDEPT OF IT, PDCE 2009-2010 Page 25 COLLECTIVE INTELLIGENT BRICKS(Silicon Integrated Systems SiS735). The CPU consists of 32-bit AMD Athlon** XP x86processors with a worst-case die heat flux of about 50 W/cm^sup 2^. In each brick are 12 laptop 2.5-in. drives, each with an 80-GB capacity, offering a total storagecapacity of 960 GB per brick for user data. The software is stored on a 256-MB flash memory.The BIOS (basic input/output system) uses a modified version of Phoenix Technologies code.The 12 disks are controlled by three Advanced Technology Attachment (ATA) RAID 0/1controllers, operating all drives as ATA masters. The brick was designed primarily by IBM Engineering & Technology Services in Mainz,Germany. The overall function was subdivided into a passive PCI backplane board, a CPUmother/daughter board which plugs into the PCI backplane, a communications board, a 12-waydisk controller board, and two power supply boards. Since two of the boards contained severalPCI devices each (a situation not found in a standard personal computer), the PCI bus on thebackplane is supported with an auxiliary bus which disambiguates the PCI addressing. Figure 1(b) exposes the inside of the brick. The 12 disks, only one of which is visible, areclamped side by side between two thick aluminum plates, using thermal conducting material as abuffer. A Fourier analysis of the vibration spectra of operating disks determined that vibrationcoupling between the disks is barely detectable. Component cooling is described below in thethermal architecture section. The rectangular blue structures visible on the brick surfaces are capacitive couplers, which areused to connect the brick to all of its neighboring bricks. At the top and the bottom of each brickis a standard floating power connector, which carries 48 VDC and provides wiring for a controlcircuit discussed later.The dimensions of a brick are 20 20 25 cm at 8.9 kg (roughly 8 8 10 in. at nearly 20pounds). Capacitive coupling is an excellent match for brick architectures and has generated considerableinterest; therefore, it is discussed in some detail. DEPT OF IT, PDCE 2009-2010 Page 26 COLLECTIVE INTELLIGENT BRICKSCapacitive couplers are insulated flat metal plates mounted on all six surfaces of a brick. A pairof plates consisting of two neighboring bricks forms a small capacitor that can transmit high-speed electrical signals between bricks [8]. Using this technology with the brick architecture hasbrought the cost of a 10-Gb/s port down to considerably less than $100, including the per-portcost of the switch chip itself, the couplers, and the internal brick wiring. This cost is substantiallyless than that of conventional solutions using centralized switches and long wires or fibers. Another reason for inventing this interbrick connection is the mechanical insertion problem. Forexample, if there is a hole in the top layer of bricks, one should be able to simply drop a brickinto it. With conventional connectors, this would be awkward, because the connections have tobe made on multiple sides simultaneously. Couplers provide an elegant solution to this problembecause they slide past each other when bricks are inserted. Capacitive coupling is only one of avariety of related contactless schemes that could be used for this purpose; others includeinductive coupling and electromagnetic (i.e., wireless or optical) transmissions. The alignment of capacitive couplers is not very critical and can be aided by simple mechanicalstructures [9]. It is expected that one can scale this technology to support thousands of brickswithout requiring expensive high-precision mechanical components. A pair of coaxial cables transmits differential dc-balanced signals from conventional transceiversto the back of the metal plates. The transmitting and receiving paths are identical. No specialamplifiers or compensation networks are required as long as the capacitances are large enoughand there are no parasitic reactances. The required minimum capacitances are 70 pF for 1-Gb/slinks and 25 pF for 3.125-Gb/s links.6 Such values can readily be achieved by coating flat metalplates of less than one cm^sup 2^ area with thin dielectric layers and having the plates touchgently. Eye diagram of a capacitive coupler used for 10-Gb/s transmissions. In Figure 1(b), several capacitive coupler plates are visible. They are designed for 10-Gb/stransmissions over four parallel channels. This corresponds to the industry standards used for 10- Gb/ s Ethernet, Fibre Channel, and InfiniBand**. There are a total of 16 = 4 . 2 . 2 plates percoupler. This is required to support four- channel bidirectional transmission, which doubles theDEPT OF IT, PDCE 2009-2010 Page 27 COLLECTIVE INTELLIGENT BRICKSnumber of wires, and each is differential, which doubles it again. Thus, the total number of wiresis 4 2 2 = 16, and the total number of pads is 16. The carrier for the metal plates is a flat ceramicsubstrate. The standard XAUI (10 Gb/s Attachment Unit Interface) protocol [10] for Ethernet, InfiniBand,and Fibre Channel physical interfaces employs an 8B/10B encoding scheme [11] to avoid thetransmission of low-frequency signals over optical channels. Fortuitously, this also solves thelow-frequency cutoff problem of capacitive coupling. The 8B/10B encoding scheme guaranteesthat the run length of zeros or ones is limited to five each. The 3-dB low-frequency cutoff waschosen at 1/50 of the data bit rate. Figure 3 shows the measured eye pattern for one of the channels used with 10-Gb/s links at 3.17Gb/s, which is slightly faster than that required for 10-Gb/s Ethernet links. The eye is wide open,as one would expect for a clear communication channel. Preliminary 3D electromagnetic fieldmodel studies indicate that the technology may be extendable to 40-Gb/s links. IceCube base-power, cooling, and control The collection of bricks sits on a base that provide\\s power, cooling, and certain controlfunctions to the bricks. Thermal architecture This section describes the details of the IceCube thermal architecture [4] (Figure 4). A verticalarray of cold rails (R) serves as the thermal backbone of the system. Bricks slide down along therails and are clamped to the rails to make good thermal contact. Liquid coolant removes heatfrom the cold rail. Coolant circulates through the cold rails, is supplied to the bottom ends of therails, and returns through tubes (S) internal to the cold rails. Bricks can be added while thesystem is running; this was even done during the assembly of the prototype. For lower-poweredsystems, the liquid coolant within the cold rails could be replaced with a one- way, upward flowof chilled air. DEPT OF IT, PDCE 2009-2010 Page 28 COLLECTIVE INTELLIGENT BRICKSFigure 4 Thermal architecture for a 3D brick-based system. A liquid-cooled intelligent brick system is quiet. This is an important consideration, since it isdifficult for large, fan- cooled systems to adhere to the noise limits (<75 dBA) set by governmentregulations. Nearly all of the heat is removed through the cold rails and very little through the surface of thebricks. This means that the cube can scale to very large dimensions, at least in the horizontaldimension. Vertical scaling is limited because the cold rail must remove all of the heat from itscolumn of bricks. Floor loading sets another limit to vertical scaling. It may be possible to handle power densities of several hundred kW/m^sup 3^ with practicaltemperature differences (~20C) between the top and bottom of the cold rails and carefullydesigned brick internals. Nuclear reactor cores cool thermal power densities of approximatelyone hundred MW/m^sup 3^ with similar temperature differentials [12]. DEPT OF IT, PDCE 2009-2010 Page 29 COLLECTIVE INTELLIGENT BRICKS8.POWER SYSTEM A 208-VAC-to-48-VDC redundant and modular power supply located in the base provides thebricks with power. The power supply can be remotely controlled via Ethernet and is interlockedwith the cooling system. Conventional hot-swap floating power connectors on the top andbottom surfaces of each brick carry 48-VDC power vertically along each of the nine columns ina tapped-bus arrangement. The lower bricks in a column must carry the current for the bricksabove them. The lower voltages required for the electronics are created locally within the bricksusing dc/dc converters. Power distribution at 48 VDC is suitable for (large) storage server applications, in which the totalpower consumption per column is of the order of l kW (assuming 200 W per brick, stacked fourto five high). Future compute bricks, which may dissipate 1-2 kW each, will have to distributehigher-voltage power along the columns, complicating conformance with safety regulations. Single points of failure The base of the prototype contains several single points of failure because it was not an essentialproject objective to eliminate them, although this could have been done with additionalengineering effort. The failure points are the integrity of the coolant distribution system, thecontrol modules, and parts of the 208-VAC power distribution system. The 48-VDC system isredundant. A future system would contain redundant control modules, and the cooling systemcould be compartmentalized in various ways so that a leak would not render the systeminoperative. Note that insertion or removal of a brick does not require opening the coolingsystem loop. There have been no problems with the IceCube cooling system in more than oneyear of operation. Since all bricks in a given column share power and cooling, there is a higher probability ofcorrelated failures for vertically stacked bricks. Software recognizes this and avoids storingredundant data on bricks in the same column. DEPT OF IT, PDCE 2009-2010 Page 30 "