Configuring ATM Networks

This article describes how to configure Linux-based PCs and an asynchronous transfer mode (ATM) switch to build an ATM network.

by Wayne J. Salamon

The Linux ATM software (device driver and utilities) is developed and supported by Werner Almsberger in Switzerland as part of the Linux-ATM API software set (see Resources). This software contains device drivers for the following ATM adapters: Efficient ENI-155P, SMC ATM Power 155, Rolf Fiedler's TNETA1570 board, Zeitnet ZN1221/ZN1225 and the IDT 77901/77903 155 and 25 Mbps adapters. Also, a driver for the Fore PCA-200E ATM adapter is available separately (see Resources). The two adapters I have experience with are the Efficient ENI-155p and the Fore PCA-200E.

The National Institute of Standards and Technology (NIST) uses ATM and Fast-Ethernet networks as interconnects in its scalable cluster computing initiative. One research area is evaluating the benefits of ATM and Fast-Ethernet networks in this cluster environment.

In this article, I will tell you how to obtain and install the ATM support software and device drivers. I will also describe how to configure the ATM connections on the PCs and the switch to be used for IP network traffic.

The ATM interface cards I use are ENI-155P ATM adapters produced by Efficient Networks and PCA-200EPC adapters from Fore Systems. These cards are installed in standard Pentium or Pentium-Pro-based PCs running Linux. The ATM switch I used for this article is a Fore ASX-1000, although the information I give applies to all of the Fore ATM switches. This switch can be set up to allow the Linux workstations to use IP over both Switched Virtual Circuits (SVC) and Permanent Virtual Circuits (PVC).

Obtaining and Installing the Linux-ATM Software

The ATM software is available from http://lrcwww.epfl.ch/linux-atm/. The software is packaged as a compressed, gzipped tar file. Each version of the software is tied to a specific version of the Linux kernel. For this article, I used version 0.35 running on Linux kernel 2.1.90. The size of the ATM software distribution is roughly 500KB. The device driver for the Fore PCA-200E adapter can be obtained by anonymous FTP from ftp://os.inf.tu-dresden.de/pub/pca200e/. Refer to the README file in the PCA200 distribution for further information.

The driver portion of the Linux-ATM software, as well as the changes to the Linux kernel, are shipped as one large patch file. Therefore, adding support to the Linux kernel for ATM is straightforward: apply the kernel patch, configure and rebuild the kernel in the usual way. The ATM configuration items you must have are:

Asynchronous Transfer Mode (ATM) (CONFIG_ATM)
Classical IP over ATM with ATMARP (CONFIG_ATM_ATMARP)
Device driver, one of the following:
Efficient Networks ENI155P (CONFIG_ATM_ENI)
ZeitNet ZN1221/ZN1225 (CONFIG_ATM_ZATM)
Rolfs TI TNETA1570 (CONFIG_ATM_TNETA1570)
IDT 77201 (NICSTAR) (CONFIG_ATM_NICSTAR)

I recommend starting with a fresh Linux kernel source tree before applying the ATM patch. Refer to the USAGE file that is part of the Linux-ATM software, as things may change. All of the device drivers in the distribution can be built as kernel modules or as part of the kernel object itself. If you are using a Fore PCA-200E adapter, you do not select a driver during the kernel configuration. The PCA-200E device driver is built as a module separately, as specified in the README file included in the PCA200 distribution.

After the kernel is patched, rebuilt and installed, you are ready to build the ATM support software. Again, refer to the instructions in the USAGE file. One change I recommend is installing the support files in /usr/local/atm-version/bin and creating a soft link from /usr/local/atm to the actual install directory. By using the soft link, you can change ATM software levels and back them out, if needed, without changing the configuration scripts.

Configuring the ATM Device Interface

You are now ready to configure the IP over ATM. First, you must decide what type of ``virtual circuits'' to use to connect the machines. ATM is a point-to-point, switched technology; in order for two hosts to communicate, a virtual circuit must be established between them.

Switched Virtual Circuits (SVCs) are connections that are established dynamically and torn down when the connection is no longer needed. However, a high latency is associated with establishing a connection. Also, SVCs are deleted after a timeout period if no traffic is sent over the connection. Therefore, the latency associated with SVCs is not always predictable. I encountered several problems when using SVCs, such as connections not being established or sometimes failing to remain open.

Permanent Virtual Circuits (PVCs) are established and kept open. Thus, no latency is associated with establishing the connection, as there is when using SVCs. The disadvantage of PVCs is that the switch must be configured to establish all the connections between the hosts. When you have several hosts and each host needs to communicate with all the others, the number of PVCs required within the ATM switch grows rapidly. Specific configuration information for SVCs and PVCs is discussed later, but I will jump ahead a bit in order to complete the IP configuration now. The steps to configure the ATM interface are as follows:

Start the ATM software daemons with these commands:
```
 atmsigd -b
 ilmid -b
 atmarpd -b
```
Create the ATM device name:
```
 atmarp -c atm0
```

Configure the ATM interface for IP:

 ifconfig atm0 ipaddr netmask netmask mtu mtu

Add the route for the ATM subnet:

 route add -net network netmask netmask atm0

Create a permanent ATM ARP (address resolution protocol) cache entry for the ARP server:
```
 atmarp -s arpserver arpsrvnsap arpsrv
```

ipaddr is the IP address of the ATM interface, netmask is the network mask and network is the IP address of the network to which we are connecting. arpserver is the IP address of the ATM ARP server and arpsrvnsap is the ATM address of the ARP server. The ATM ARP server is used to convert an IP address to an ATM network service access point (NSAP) address. (The NSAP address is similar to a media access control (MAC) address and is 20 octets long.) The NSAP address is needed to establish SVCs between nodes. You can also create an /etc/hosts.atm file to contain the IP to NSAP mapping, allowing for quicker IP to NSAP translations. For my network, I use the Fore switch as the ARP server. The atmarpd daemon maintains a cache of IP to NSAP mappings. The atmarp command makes the ARP cache entry permanent when the arpsrv option is used.

One final note: if you are going to use PVCs only, you do not need to start the atmsigd and ilmid daemons. Listing 1 contains a complete example of configuration commands.

Configuration of Switched Virtual Circuits

The ATM switch configuration commands I use apply to the entire family of Fore ATM switches, because they all have a similar command interface.

When using SVCs, a host must pass information to the ATM switch, declaring its intent to set up a connection with another host. The term for connection setup is ``signaling''. The ATM protocol used between a host and a switch is the user-network-interface (UNI) signaling standard. There are several revisions of the UNI standard. The Fore ATM switch supports UNI 3.0, 3.1 and 4.0. The Linux-ATM software also supports these versions.

However, there are standards and there are implementations. In setting up our SVCs, I encountered several problems with UNI 3.0 signaling. The UNI 3.1 signaling was more stable and reliable. To change the signaling on the Fore ATM switch, each port must be changed individually, using the switch control processor (SCP) command interface.

First, log on to the SCP via a TELNET session or by using a terminal attached to the serial port on the Fore switch. The command syntax used here is the same as Fore's. Required parameters are shown between ``<'' and ``>''; options are enclosed in brackets (``['' and ``]''); modifiers to options are enclosed in parentheses. One of the modifiers must be chosen.

Change to the UNI configuration menu:

localhost::> conf uni

The switch prompt is shown in italics, while the command is shown in normal text.

The command show will list the current UNI status for each port. If the port is already configured for UNI 3.1, no change needs to be made. Otherwise, you must first delete the current configuration. The syntax for the delete command is del port vpi, where port is the switch port and vpi is the virtual path identifier (usually 0). To delete the signaling on port 1A1 for VPI 0, you would enter this command:

localhost::configuration uni> del 1a1 0

Now you are ready to configure the port for UNI 3.1. The syntax for the new command is:

new <port><vpi> [auto | uni30 | uni31] [-ilmi (up | down)]

The ilmi option is used when you want the port to respond to integrated local management interface (ILMI) requests. ILMI is used by the hosts to obtain the ATM NSAP address assigned to the host. You usually want to have ILMI active for the port, so the command for port 1A1, VPI 0 is:

localhost::configuration uni> new 1a1 0 uni31 -ilmi up

Now that the ATM switch ports have been configured, the software on the workstation must be set up. The key portions of the Linux-ATM software are three daemons: atmsigd to handle signaling (UNI), ilmid to handle ATM address registration and atmarpd to map ATM addresses to IP addresses. Listing 1 is the startup script I use to start the ATM daemons and to configure the ATM interface on a host. This script can be called from a system startup script (/etc/rc.d/rc.local, for example) to configure the ATM interface at boot time.

The ATM signaling daemon, atmsigd, must be compiled specifically for the version of signaling you wish to use and must be compatible with the signaling version the ATM switch port has been configured to use. The default version used in the ATM software is UNI 3.0. If you've configured the switch to use UNI 3.1, having the hosts use UNI 3.0 will most likely work, due to backward compatibility. However, I recommend you configure the Linux-ATM software to use the same version as the switch, UNI 3.1.

To have the signaling daemon use UNI 3.1, edit the Rules.make file in the ATM source directory ( /usr/src/atm if you follow the steps in the USAGE file). You need to change the STANDARDS line to specify the version of signaling to support. For UNI 3.1, this line should be: STANDARDS=-DUNI31.

Special Configuration for ENI-155p ATM Cards

If you are using Efficient ENI-155p ATM cards, the number of simultaneous virtual channels available is limited. The ENI card performs the segmentation and reassembly (SAR) of ATM cells by using memory on the adapter card as a buffer. The host ATM software allocates buffer space for each virtual channel. If you attempt to open more SVCs than are supported by the available buffer space, you will receive this error message from the ATM ARP daemon:

atmarpd:IO: [2]connect: No buffer space available

When IP over ATM is used, the device driver sends packets to the ATM card using an ATM Adaptation Layer (AAL). While several adaptation layers are available, AAL-5 is used for IP. The AAL-5 packet is a type of service data unit (SDU) and is somewhat analogous to an Ethernet frame. The AAL-5 packets are divided into individual ATM cells by the Efficient ATM adapter.

The MTU (maximum transmission unit) size for the ATM interface depends on the SDU size. The IP over ATM (Classical IP) specification says that the MTU should be no larger than 9180 bytes. There are also 8 bytes for an AAL-5 trailer, so the SDU for IP over ATM is 9188 bytes in the default configuration. The amount of buffer space needed on the card depends on the maximum SDU size.

The Linux-ATM software allocates three times the maximum SDU size, rounded up to the nearest power of two. In the default configuration, this allocation results in 32KB of buffer space being reserved for each ATM connection (9180 x 3 = 27540, rounded to 32768 bytes). Also, using classical IP causes two SVCs to be made: the initiating machine opens an active connection to the target machine and the target machine opens an active connection back, that is, a passive connection on the initiator. Therefore, these two connections result in the allocation of two buffers on the ATM card, for a total of 64KB.

The default configuration allows a host to have a maximum of fourteen simultaneous connections when using the ``client'' version of the ENI-155p ATM card, which has 512KB of memory and 504KB of memory available for the SAR buffers. These fourteen connections allow communication using IP over ATM to seven other hosts when using SVCs. If you set up PVCs, you can communicate with fourteen other hosts. When using an ARP server, you have one less connection available, reducing the host count by one as well. The ``server'' version of the ENI 155p card has 2MB of memory, with 2040KB for SAR buffers, allowing for more simultaneous connections.

To increase the number of simultaneous connections for classical IP, you need to change the size of the maximum SDU set on the ATM interface. By using the allocation rule given above, you can estimate the amount of memory needed for the connections. For example, if you want to use 16KB for each connection, the maximum SDU would be 16384 divided by 3, which is 5461 bytes. I'll use an SDU of 4352 bytes for my example in this article.

The maximum SDU is specified as an option to the ATM ARP daemon. However, when the SDU is changed, the IP interface must also be configured to have an MTU of the same size as the SDU, minus 8 bytes for the AAL-5 trailer. Therefore, in my example the MTU is 4344 bytes.

A potential problem occurs when changing the maximum SDU for the interface: the ATM ARP daemon (atmarpd) may not communicate with the ARP server on the Fore switch. Our switch would accept only connections with an SDU of 9188 bytes. The fix for this problem is to create a permanent ARP cache entry on the host, specifying the maximum SDU of 9188 bytes, for the connection to the ARP server. The steps for configuring the ATM software on the workstation are as follows:

Configure the IP interface for your MTU size, 4344 bytes in my example:
```
ifconfig atm0 ipaddr netmask netmask mtu 4344
```
Create a permanent ATM ARP cache entry for the ARP server with SDU size of 9188:
```
atmarp -s arpserver arpsrvnsap qos \
ubr:sdu=9188 arpsrv
```
Configure the SDU (MTU plus 8 bytes) on the ATM interface:
```
atmarp -q network ubr:sdu=4352
```

Refer to Listing 1 for a complete example of configuring the ATM software for the Efficient adapter.

Using IP over Permanent Virtual Circuits

To establish a PVC, the following steps must be performed.

On the workstation, add an ATM ARP entry on each node specifying the PVC (vpi.vci pair) used to connect to each of the other hosts.
Create the PVC on the switch.

As an example, the following commands executed on the appropriate host will set up a PVC between nodes named node1 and node2, on interface 0, using a vpi of 0 and a vci of 70:

node1: atmarp -s node2 0.0.70
node2: atmarp -s node1 0.0.70

The PVC is identified by three numbers, separated by two periods. The numbering scheme is interface.vpi.vci, where interface is 0 for the first ATM adapter, 1 for the next, etc. The default interface for the atmarp command is 0. The vpi (virtual path identifier) and vci (virtual channel identifier) are the standard ATM PVC identifiers. The host name (node1 and node2) can be used if there is an entry for it in the /etc/hosts file; otherwise, use the IP address of the host.

The commands above tell node1 to communicate with node2 over PVC 0.0.70 and for node2 to communicate with node1 over PVC 0.0.70. The atmarp command links the IP address of the target host to the PVC. You could choose a different PVC for each connection, but it is simpler to think in terms of one PVC connecting two machines.

The vpi.vci pair must not be in use on the host. Also, any ATM ARP cache entries must be deleted for the target host before creating the PVC. (These cache entries are created when SVCs are opened to the destination host.) To delete an ARP cache entry on node1 for node2, you would use this command:

node1: atmarp -d node2

Next, the switch must be configured to complete the PVC between the hosts. It is helpful to understand the port naming convention used by the Fore switch. The port names consist of three identifiers:

Board: the number of the switch board (same as the SCP number); each SCP controls one switch board.
Network Module: the slot (A, B, C, or D) in the switch board containing the port.
Port: the physical port number on the network module.

For example, port 1b3 refers to the first switch board, the second network module (module b) on board 1 and the third physical port on the second network module. The Fore ASX-200 switch has only one switch board, while the ASX-1000 switch has four. There is a maximum of four network modules per switch board and a maximum of six physical ports per module.

You must now create the virtual channels on the ATM switch. In our example, you would enter these commands on SCP 1:

localhost::> conf vcc 
localhost::configuration vcc> new 1a1 0 70 1a2 0 70
localhost::configuration vcc> new 1a2 0 70 1a1 0 70

1a1 is the switch port for node1 and 1a2 is the switch port for node2.

The switch completes the PVC based on the input port to output port virtual channel connection (VCC) mapping. Note that the PVC vpi.vci (0.70) matches the vpi.vci given to the atmarp commands on the hosts.

The above commands will connect two ports on the same ATM switch board. The Fore ASX-1000 switch has up to four switch boards. If you are connecting machines on different switch boards, the procedure is more complicated, as you must connect each port to the switch fabric and connect the fabric to each port. Thus, if you wish to connect a machine on port 1a1 to a machine on port 3a1, the following commands are required:

On SCP 1:

localhost::> conf vcc 
localhost::configuration vcc> new 1a1 0 70 1e3 0 70
localhost::configuration vcc> new 1e3 0 70 1a1 0 70

On SCP 3:

localhost::> conf vcc
localhost::>configuration vcc> new 3a1 0 70 3e1 0 70
localhost::>configuration vcc> new 3e1 0 70 3a1 0 70

On the Fore switch, the fabric connections are slot e. Therefore, port 1e3 refers to a connection from switch board 1 to switch board 3. Likewise, 3e1 refers to a connection from switch board 3 to switch board 1. Fore refers to these ports as ``intra-fabric'' ports.

Testing the Connections

Once the Classical IP setup is complete, all of the standard network tests can be performed. The simplest test is done by using the ping command to test the connection. One difference between SVC and PVC connections is a large latency for the first ping response when using SVCs. The reason for the latency is the setup time needed to establish the SVC. After the SVC is established, the latency for SVC and PVC connections should be the same.

After verifying the basic connectivity, you can run some network performance tests over the ATM connection. I have used the Netperf tool (see Resources) as well as some benchmarks developed locally. The maximum throughput performance is very good, around 132Mbps. This number is close to the maximum payload data rate for an OC-3 ATM network.

Conclusion

I have given instructions needed to set up the switch and hosts on an ATM network with Linux. The configuration steps given are specific to IP over ATM connections using the Classical IP standard. In addition to Classical IP, LAN Emulation (LANE) can be used to carry IP over ATM. LANE is supported by the Linux-ATM software as well, but configuration of LANE is beyond the scope of this article. For more information, refer to the documentation in the Linux-ATM distribution.

Hosts can communicate in several other ways using an ATM interface without relying on Classical IP. The ATM software supports ``native'' ATM sockets, where applications can communicate directly over an ATM connection, bypassing the IP software completely.

If you are interested in learning about ATM technology but don't have ATM hardware, the Linux-ATM software can be of help. The software has the capability to emulate an ATM device using TCP/IP to make the actual connection. By taking advantage of this support, you can get a head start on configuring ATM for Linux and learning the ATM programming interface.

NIST

Resources

Wayne J. Salamon is a Computer Scientist in the High Performance Systems and Services Division at the National Institute of Standards and Technology in Gaithersburg, MD. He has worked on system software for PCs, UNIX workstations and IBM mainframes for the past 12 years. When not doing computer stuff, he appears to play guitar, though only when connected to vacuum tube amplifiers. Wayne can be reached at wsalamon@nist.gov.