Myrinet 2e4o31

MYRINET

Abstract: Myrinet, ANSI/VITA 26-1998, is a high-speed local area networking system designed by Myricom to be used as an inter connect between multiple machines to form computer clusters. Myrinet has much less protocol overhead than standards such as Ethernet, and therefore provides better throughput, less interference, and lower latency while using the host U. Although it can be used as a traditional networking system, Myrinet is often used directly by programs that "know" about it, thereby bying a call into the operating system. Keywords: Myrinet, myricom, Ethrenet Latency, Throughput, switching, Duplex, Gigabit Ethernet. Introduction: Myrinet is a cost-effective, high-performance, packet-communication and switching technology that is widely used to interconnect clusters of workstations, PCs, servers, blade servers, or single-board computers. Clusters provide an economical way of achieving high performance, high availability. Myrinet physically consists of two fibre optic cables, upstream and downstream, connected to the host computers with a single connector. Machines are connected

via low-overhead routers and switches, as opposed to connecting one machine directly to another. Myrinet includes a number of fault-tolerance features, mostly backed by the switches. These include flow control, error control, and "heartbeat"

monitoring on every link. The newest, "fourth-generation" Myrinet, called Myri10G, s a 10 Gbit/s data rate and is inter-operable with 10 Gigabit Ethernet on PHY, the physical layer (cables, connectors, distances, signaling). Myrinet Overview: Myricom Myrinet is the market leader in high-performance, high-availability, cluster interconnects. Myricom shipped its first Myrinet products in August 1994 and there are now thousands of Myrinet installations, ranging in size up to more than 1,000 hosts. These sites include many of the world's premier cluster-computing systems. Characteristics that distinguish Myrinet from other networks include: 









Full-duplex 2+2 Gigabits/s or 10+10 Gigabits/s data rate links, switch ports, and interface ports Flow control, error control, and "heartbeat" continuity monitoring on every link Low-latency, cut-through, crossbar switches, with monitoring for highavailability applications Switch networks that can interconnect hundreds of hosts with a single, full-bisection network component, can scale to tens of thousands of hosts, & that can also provide alternative communication paths between hosts Host interfaces that execute a control program to interact directly with host processes ("OS by")

for low-latency communication, and directly with the network to send, receive, and buffer packets

--Packet Format

Myrinet is an American National Standard -- ANSI/VITA 26-1998. The link and routing specifications are public, published, and open. Myrinet components :

( up to 24payload bytes) –--Header Arbitrary length – CRC, error-control Myrinet Switch Counters

 

 

Myrinet Switches Myrinet software for most common hosts and operating systems. Myrinet Interfaces Myrinet link fiber or cables

Myrinet Switches: Myrinet switches provide the high speed cluster message ing network for ing messages between computer nodes and for I/O. The Myrinet switches have a few counters that can be accessed from an Ethernet connection to the switch. These counters can be accessed to monitor the health of the connections, cables, etc. The following information refers to the 16-port, the clos64 switches, and the Myrinet2000 switches. Myrinet Packets:

The Myrinet switches maintain counters for the following types of received packets between reboots or clears: •

•

•

•

•

good_high, good_low - A 64-bit counter of the number of good packets received in on this port. bad_high, bad_low - A 64-bit counter of the number of non-zero CRC8 packets received on this port. timeouts - The number of times that the switch didn't see a tail for too long coming in on a port. Timeouts are on the receive side, so they indicate that a packet went too long without a tail, or that the packet was blocked for too long. bad_routes - The number of packets that entered this port (receive), but had a routing byte that was off either end (outofrange) of the switch. dead_routes - The number of packets that entered this port (receive), but had a routing byte destined for a valid port that was down, disconnected, or off.

•

•

•

•

bad_packets- Received packets with non-zero CRC bytes at the tail. missed_beats - A port is not connected. Connected ports receive heart beat signals from their opposites, and when these stop arriving, the counter goes up. If you have HA features enabled, the port will also shut itself down (portDownOnMissedBeat). A missed beat is no beat in 10 usecs, but this is polled for by the xbar microcontrollers around every 10 msecs and updated in the monitoring card probably about every 100 msecs. port_state - A mask of the following bits:  #define SW_PORT_ON 0x01  #define SW_PORT_BEAT 0x02  #define SW_PORT_TRANSMIT 0x04  #define SW_PORT_RECEIVE 0x08

•

control is in 2 parts. o The 0xff bits are an internal state machine state of the following values:

case 0: /* port was in reset. initialize control */ case 1: /* port is initialized */ case 2: /* port is off, but automatic shutdown is enabled */ case 3: /* port is on, automatic shutdown is on */ case 10: /*port was disabled by . */

o The 0xff00 bits are a bit mask of the following bits: #define HW_CTRL_XMIT_OFF 0x80 #define HW_CTRL_RECV_OFF 0x40 #define HW_CTRL_MISS_BEAT_OFF 0x20 #define HW_CTRL_ILL_OFF 0x10 System Requirements for Switch Monitoring : Clos-64 or 16 port switch requirements •

The Linux kernel must the RARP protocol. Kernel configuration defaults to "no" so a new kernel may have to be configured. This can be a a module or as part of the kernel but as a module has to be loaded. It's not very big. 

RARP can be confirmed by looking for a /proc/net/rarp. If the file exists, then RARP is ed. If not, a module may have to loaded or it is not ed.

RARP might be configured as a module. The command > lsmod can be used to list modules such as rarp.o that have been installed. If there is a file in /lib/modules/[release number]/rarp.o then that release is configured as a module.

RARP can be added with the command: > insmod /lib/modules/[release number]/rarp.o. IP addresses must be assigned for each switch. This implies that at the switches must be connected to an Ethernet hub. Each switch has its hardware MAC address printed on the switch. The IP address must be broadcast from a Linux host using the rarp command. This command informs the Myrinet switch its IP address. •

•

•

•

•

Setting the IP address in the switch: /sbin/rarp -s [IP address] [MAC Address] The "ping" command can be used to that the Myrinet switch has assigned its IP address. The Myrinet switch will store that IP address until it is powered off or rebooted with the bob_test -reboot command. If it is powered off or rebooted, the rarp command will have to be repeated. The switch will not recognize a changed IP address with a second rarp command. The switch will have to be powered off or rebooted to change the previous address. the corresponding 16port or 64-port version of the Myrinet Switch Query Program, bob_test, from Myrinet at There are several precompiled versions of the 64-port version. There is no Alpha version. The Makefile requires some simple modifications to make it work. It does build correctly on the Alpha platform using gcc.

Myrinet2000, or M3, Switches:

Myrinet 2000 Switches are considerably different from the Clos-64(m2) and 16port switches. First, these switches get their IP addresses from the dhd demon, not from the RARP command. So, the switches' IP and MAC addresses must be in the dhd's configuration file, /etc/dhd.conf, which is generated from the Cluster Tools on lant.. Dhd must be restarted when the file is modified, it does not respond to a SIGHUP or notice that the file is changed. Once the m3 switch has its address it is accessed by HTML browsing, SNMP commands, or one of several small programs supplied by Myrinet. Note that these programs do not all run correctly so use with caution. Since then hardware upgrades and software improvements have made Myrinet the network of choice for many cluster builders and until recently there was hardly an alternative when a fast, lowlatency network was required. Like Infiniband, Myrinet uses cut-through routing for an efficient utilisation of the network. Also RDMA is used to write to/read from the remote memory of other host adaptor cards, called Lanai cards. These cards interface with the PCI-X bus of the host they are attached to. The latest Myrinet implementation only uses fibers as signal carriers. This gives a high flexibility in the connection and much headroom in the speed of signals but the fiber cables and connectors are rather delicate which can lead to damage when cluster nodes have to be serviced. Myrinet offers ready-made 8--256 port switches. The 8 and 16 port switches are full crossbars. In principle all larger networks are build form these using a Clos network topology. An example for a 64port systems is shown in Figure.

Figure. An 8×16 Clos network using 8 an 16 port crossbar switches to connect 64 processors. A Clos network is another example of a logarithmic network with the maximum bisectional bandwidth of the endpoints. Note that 4 ports of the 16×16 crossbar switches are unused but other configurations need either more switches or connections or both. Myricom provides benchmark measurements of its networks at which for the newest Myinet2000 switches give quite good results: about 230 MB/s for a Ping-pong experiment with latencies around 7 µs for small messages with the MPI version that Myricom distributes with its networks.

Myrinet Interfaces: Myrinet packets may be of any length, and thus can encapsulate other types of packets, including IP packets, without an adaptation layer. Each packet is identified by type, so that a Myrinet, like an Ethernet, may carry packets of many types or protocols concurrently. Thus, Myrinet s several software interfaces. Myricom supplies two alternative lowlevel message-ing systems for Myrinet: GM, which is used in most

Myrinet clusters today, and a new system called MX (Myrinet Express). Other software interfaces such as MPI, Sockets, DAPL, VI, and PVM are layered efficiently over GM or MX, and are available from Myricom and from third parties. Both GM and MX provide "Ethernet emulation," in which any protocols carried over Ethernet, typically T/IP and UDP/IP, can be carried over Myrinet. The GM and MX systems provide protected -level access to the Myrinet (secure in multi-, multiprogramming environments); reliable, ordered delivery of messages; network mapping and route computation; and other features that robust and error-free communication. The Software and Customer page provides a complete outline of Myrinet software, with links to documentation, and the Performance Measurements page shows the performance achieved with the various software interfaces. Myrinet Express (MX) Software: Myrinet Express (MX) is a new messageing system from Myricom that provides lower latency with middleware layers such as MPI. MX includes Ethernet emulation and for all protocols that run over Ethernet, including T/IP and UDP/IP. MX was developed for the Myri-10G series of NICs, but MX-2G is already available on several operating systems for the PCI-X series of Myrinet NICs. MX was developed for the Myri10G series of NICs and is fully compatible with MX-2G at the API and application levels. Myrinet-2000 and Myri-10G employ the same network architecture and protocols, and, when used with MX software , are fully compatible with respect to applications and application-programming interfaces.

Myrinet Components and Software: Myricom supplies Myrinet components and software in two series: Myrinet-2000 and Myri-10G. Myrinet-2000 is a superior alternative to Gigabit Ethernet for clusters, whereas Myri-10G offers performance and cost advantages over 10-Gigabit Ethernet. Myri-10G uses the same physical layers (PHYs: cables, connectors, signaling) as 10-Gigabit Ethernet, and is highly interoperable with 10-Gigabit Ethernet. In fact, Myri-10G NICs are both 10-Gigabit Myrinet NICs and 10-Gigabit Ethernet NICs. Myrinet-2000 and Myri-10G employ the same network architecture and protocols, and, when used with MX software , are fully compatible with respect to applications and applicationprogramming interfaces. Myricom supplies Myrinet software for most common hosts and operating systems. The software is supplied "open source," and other Myrinet software is available from third parties. You or an integrator install the NICs and software in the hosts, and connect the network with cables and switches. The software maps the network, and uses whatever communication paths are available from host to host. No switch programming or routing-table configuration is necessary. Software Stack:

Application

Host

MPI, Sockets ,etc

Kernal agent

level API

GM

Myrinet control program NIC

Fabric Management System: Myricom's Fabric Management System (FMS) provides centralized diagnostic monitoring of the Myrinet fabric from a command-line or web interface. FMS consists of one FMS server process to manage the fabric, and many Fabric Management Agent (FMA) processes, one running on each Myrinet node in the fabric. It is an important diagnostic tool for ing the health of the Myrinet hardware. The Fabric Management System (FMS) for Myrinet networks may be used with either the MX or GM low-level firmware. FMS can be used with GM-1 and supersedes Mute, the previous management software provided by Myricom. Performance: Myrinet is a lightweight protocol with little overhead that allows it to operate with throughput close to the basic signaling speed of the physical layer. On the latest 2.0 Gbit/s links, Myrinet often runs at 1.98 Gbit/s of sustained throughput, considerably better than what Ethernet offers, which varies from 0.6 to 1.9 Gbit/s, depending on load. However, for supercomputing, the low latency of Myrinet is even more important than its throughput performance, since, according

to Amdahl's law, a high-performance parallel system tends to be bottlenecked by its slowest sequential process, which is often the latency of transmission of messages across the network in all but the most embarrassingly parallel supercomputer workloads. Deployment: According to Myricom, 141 (28.2%) of the June 2005 TOP500 supercomputers used Myrinet technology. In the November 2005 TOP500, the number of supercomputers using Myrinet was down to 101 computers, or 20.2%, in November 2006, 79 (15.8%), and by November 2007, 18 (3.6%), a long way behind Gigabit Ethernet at 54% and Infiniband at 24.2%. Technology and Reliability: Myrinet components are implemented with the same advanced technology -- fullcustom-VLSI CMOS chips -- as today's workstations, servers, PCs, and singleboard computers. This use of CMOS technology is one reason why Myrinet performance has advanced and will continue to advance in step with advances in the hosts, without changes to the network architecture and software interfaces. These CMOS-based Myrinet components are also extremely reliable. The MTBF of current-production Myrinet switches and NICs exceeds 5 million hours per port. Myrinet exhibits a very low bit-error rate, and is highly robust with respect to host, switch, and cable faults. Myrinet can map itself continuously, and use alternate routes to circumvent faults (e.g., disconnections and powering-down). The hardware computes and checks a CRC for each packet on each link. The NICs

provide parity checking both in their memory and across host IO buses.

Reality: Myrinet is the clear market leader in highperformance, high-availability, cluster interconnect. Myricom shipped its first Myrinet products in August 1994. Including the installations supplied by Myricom's OEM customers and by Myrinet resellers and integrators, there are now many thousands of Myrinet installations, ranging in size to more than 2,000 hosts and more than 4,000 processors. These sites include many of the world's premier cluster-computing systems. A total of 141 (28.2%) of the June-2005 TOP500 supercomputers use Myrinet technology.

Conclusion: Myrinet is a cost-effective, high-performance, packet communication and switching technology that is widely used to interconnect clusters of workstations, PCs, or single-board computers. Conventional networks such as Ethernet can be used to build clusters, but do not provide the performance or features required for high-performance or highavailability clustering.

References: – A. Gulati, D. K. Panda, P. Sadayappan, and P. Wyckoff, NIC-based Rate Control for Proportional Bandwidth Allocation in Myrinet Clusters, IP ‘01 – S. Senapathi, B. Chandrasekharan, D. Stredney, H.-W. Shen, and D. K. Panda, QoS-aware Middleware for Cluster-based Servers to

Interactive and Resource-Adaptive Applications, HPDC ’03 – D. Buntinas, D. K. Panda, J. Duato, and P. Sadayappan, Broadcast/Multicast over Myrinet using NIC-Assisted Multidestination Messages, CANPC ‘03 – D. Buntinas, D. K. Panda and P. Sadayappan, Fast NIC-Based Barrier over Myrinet/GM, IPDPS ‘01.

– D. Buntinas, D.K. Panda, and W. Gropp, NIC-Based Atomic Operations on Myrinet/GM, SAN-1 – D. Buntinas and D. K. Panda, NICBased Reduction in Myrinet Clusters: Is It Beneficial? SAN-2 – W. Yu, D. Buntinas, and D. K. Panda, High Performance and Reliable NIC-Based Multicast over Myrinet/GM-2, IP ’03