Introduction

• ATM
• APE 25

• Development strategy
• Driver initialization
• Transmit buffer management
• Receive buffer management

• Data structures and bindings
• Binding the standalone driver to the protocol.

• Classical IP over ATM
• LAN Emulation

 
Is it........

A radically new and innovative networking technology that will
 
Permit fixed and variable bit rate real-time traffic to co-exist painlessly with non-real- time bulk data transfer.

• Provide virtually unlimited scalability, satisfying all future networking requirements of humankind.

Name one thing that illustrates better than even the US Internal Revenue Code the ill effects of design by large committees of competing interests.

What can/will ATM provide'
 

• A high performance LAN or Intranet solution.
• High bandwidth virtual paths in the Internet infrastructure.
• World-wide end-to-end services that obviate IP??

• Connection-oriented "best-effort" transport for 53 byte cells
• Permanent or switched connections (PVCs and SVCs)
• QOS guarantees for constant and variable bit rate streams.
• Allocation of "leftover" bandwidth to variable bit rate streams
• Transport layer services called ATM Adaptation Layers "AAL's"

APE 25 Overview

  Supports AAL1 (CBR) and AAL5

• Adapter provided segmentation and reassembly
• Scatter/Gather facility on both send and receive (not yet used by us)
• Each connection endpt (VPI, VCI) is called a Logical Channel (LC)

Receive traffic management
 

• Eight received frame queues are provided
• Eight bits in the SISR indicates which queues are non-empty
• Each LC is bound to a specific receive queue
• Driver could (but we don't) give priority based service.

Transmit traffic management

The APE-25 provides eight transmission queues (TMQs).
Associated with each TMQ is

• A peak transmission rate (bps)
• A priority (0/1 = low/high) (not related to the CLP)
• Zero or more Logical Channels (LCs)

• Peak bit rate of an LC is the peak bit rate of its TMQ
• Each transmission consumes one cell credit
• Credits may be consumed at the peak rate
• Credits are renewed at the sustainable rate

• Service high priority TMQ's first
• Among TMQ's of equal priority service higher rate first.
• Your mileage may vary!

  Development strategy
 

• Use loadable modules to minimize:
•  kernel rebuilds
• system reboots
 
•  Start with a standalone driver
• chararacter device
• ioctl interface

• Export control of the board to the application level
 
ioctl(atmfd, AIO_RDCREG, parms);
 
•  Build application level tools to read/dump NIC state

  apestat utility: dump the APE25's control registers:

• Use conditionally included printk's extensively

Device driver organization

• Initialization

  Generic device driver initialization
  Acquire / Initialize PCI data

Register as a Linux character device

Register interrupt handler

 

Driver component initialization
  Connection management structures

Transmit management structures

Receive management structures

 

NIC state initialization
  Load microcode

Load miscellaneous state registers

Enable the NIC

 

• Connection management

•   Transmit management

•   Receive management

Driver Initialization
 

• Register as a character device for ioctl service

 
/* Register as a bogo character device.. for */
/* ioctl access for testing. */

printk("IBM APE25: Module initialization\n");
rc = unregister_chrdev(APE25_MAJOR, "ape25");

if (rc = register_chrdev(APE25_MAJOR, "ape25", &atm_fops))
{
    printk( "APE25: cannot register; rc %d\n", rc);
    return 1;

• PCI device identification

for (pci_dev = pci_devices; pci_dev; pci_dev = pci_dev->next)
{

    if (vendor==PCI_VENDOR_ID_IBM &&
        device==PCI_DEVICE_ID_ATM_APE25)

    {
        ape->pci_bus = pci_dev->bus->number;
        ape->pci_devfn = pci_dev->devfn;
        ape->pci_vendor = pci_dev->vendor;
        ape->pci_device = pci_dev->device;
        found = 1;
        break;
    }
}

• PCI device configuration

pcibios_write_config_word(bus, devfn, PCI_COMMAND, 0x117); pcibios_write_config_word(bus, devfn, PCI_STATUS, 0xfd7f); pcibios_write_config_byte(bus, devfn, PCI_LATENCY_TIMER, 0x40);

• Hook the APE 25's Interrupt

rc = request_irq(ape->pci_irq, ape25_irq,
SA_INTERRUPT, "ape25", ape);

• Reset the APE25

WT_CNTLREG(ape, MODE_REG, RESET_MODE);
i = 1;
while (i > 0)
{
    if (!(RD_CNTLREG(ape, MODE_REG) & RESET_MODE))
    break;
    i += 1;
}

• IPL the pico processors
• Allocate and initialize the DMA buffer pools
• Initialize the XRAM data structures (VP Table, LCI Table, LC Table, Cell buffer pools, and Transmit Request Queues

• Initialize remaining APE 25 Regs

WT_CNTLREG(ape, IDLE_CELL_HDR_HI, 0);
WT_CNTLREG(ape, IDLE_CELL_HDR_LO, 1);
WT_CNTLREG(ape, ERROR_REG_MASK, 0x8cff);
WT_CNTLREG(ape, OAM_MASK, 0xecea);
WT_CNTLREG(ape, OAM_B1_MASK1, 0xff40);
WT_CNTLREG(ape, OAM_B1_MASK2, 0xff41);
WT_CNTLREG(ape, PCI_ERROR_MASK, 0xFFFF);
WT_CNTLREG(ape, CERR_CNT, 0x00FF);
WT_CNTLREG(ape, SERR_CNT, 0x00FF);
WT_CNTLREG(ape, FR_TIMEOUT_VAL, 0x0032);
WT_CNTLREG(ape, ACONFIG_REG, 0xffff);
 
•   Activate the APE 25

#define OS2_MODE 0x1149
WT_CNTLREG(ape, MODE_REG, OS2_MODE);

Producer/Consumer perspective on buffer management

 

List	Producer	Consumer
Transmit Ready	Driver send function	APE-25
Transmit Complete	APE-25	Driver Xmit IRQ
Free	Driver Xmit IRQ	Driver Send function

 
 

Buffer List management
 

• List elements are buffer headers (TFD's) architected by the APE - 25
• Single forward link structure
• Link value of "1" indicates end-of-list
• External head and tail pointers
• Every list must be initialized with a single "placeholder"

Mutual exclusion mechanism
 

• Single producer and single consumer per list
• Producer adds to the tail and may alter

• Consumer removes from the head and may alter

• Consuming the last (and only) element is illegal!

Synchronization mechanism

Must prevent:
 

Possible options for the send routine

• Busy-wait - both conditions are "self-correcting"
• Sleep - and be awakened by the Xmit IRQ
• Drop the frame

Transmit details

Standalone driver buffer management
 

• TFD's are permanently associated with a single buffer
• Buffers of size 1 KB are allocated by default.
• No LC can own more than 8 buffers at any time.
• The chaining capabilities of the APE 25 are not used.

The FDL (Free Descriptor List)
 

• All but one buffer is on this list after initialization
• Buffers consumed from front of list by atm_send5u
• Buffers produced to end of list by TC Int Handler
• The last and only buffer is never consumed.
• No mutex required between driver and interrupt handler

The TCL (Transmit Complete List)

Receive Buffer Lists and Lifecycle

1 - Free list:                                 Buffers available for use by APE 25.
 

2 - APE-25 Reassembly            Buffers presently owned by the APE 25
        Queue                                   for the reassembly of frames

3 - Recv Ready Lists:                   Buffers containing assembled frames. The APE
                                                       25 defines 8 logical receive queues.

4 - Per LC Ready Lists:                Buffers containing assembled frames for a
                                                        particular LC
 

 
 

Receive mutual exclusion mechanism

 

	Recv Routine		Recv IRQ
Producing to	Free List	no problem	LC RBLm	no problem
Consuming from	LC RBLm	sleep() wakeup by Recv IRQ	RRLn	Should not occur.. but must be checked

 

The Receive Buffer Header (RBH)

 
 

31.............24	23.............16	15.............8	7................0
Data Buffer Address
Next RBH Address
Buffer Length		Status
LCI		Control
SDU Length		OAM / CPL0 / TUC

The Receive Free List (RFL)

  Buffers available for the APE 25 to use for frame assembly.

 

• Fixed size data buffers are used.
• The multibuffer frame capability of APE 25 is not used.
• Data buffers are bound to RBH's and assigned to the (RFL) during system initialization.
• Buffers are consumed from front of list by APE 25 each time a frame assembly starts.
• Buffers are produced onto the end of the list by atm_recv5u after a frame is copied back to user space.

The Receive Ready Lists (RRL's)

• During initialization a free RBH is assigned to each RRL
• None of these RBH's is attached to a data buffer
• Frames are produced onto RRL ends by APE 25
• Frames are consumed from RRL front by receive int handler and then..
• Produced onto the end of the target LC Queue

Mutex problems in consuming buffers:

• APE 25 has no interaction with these lists
• The APE 25s same buffer management strategy is carried over
• A free "dummy" RBH is assigned to each LC list at initialization
• A data buffer is not attached to the RBH

Per LC Received buffer list management

• As frames arrive they are appended to the destination LC's received buffer list by the driver's receive interrupt handler.
• Frames are removed from the front of the list via ioctl receive data requests.
• After data is copied to application the RBH and the data buffer are returned to the free list.
• The rebinding technique described earlier is used here.

Reviewing the buffer lifecycle

2 - APE 25 produces on to RRL_n via the RRLn_LFDA register

3 - Driver (interrupt service routine) consumes from RRL_n via ape->srrl[n] and produces to LC_RBL_k via ape->slcrbl[k]

4 - Driver (atm_recv5u) consumes from LC_RBL_k via ape->elcrbl[k] and produces to the free list via ape->erfl.

 
The driver manages a pool of 256 RBH's
 

• RBH's used in the receive processing above are allocated from this this...during system initialization

• Its principal use is in conjunction with Linux / ATM support

Overview:
 

• Werner Almesberger of the LRC coordinates development
• Current version is 0.36 ( < production release )
• Services

From init_module() the driver must call
 

struct atm_dev *atm_dev_register(
char *type,
struct atmdev_ops *ops,
unsigned long flags)
 

The first parameter is the device name .... "ape25"

The second is a pointer to the device driver operations vector table:

static struct atmdev_ops atm_ops =
{

ape25_open, /* open */
ape25_close, /* close */
ape25_ioctl, /* ioctl */
ape25_getsockopt, /* getsockopt */
ape25_setsockopt, /* setsockopt */
ape25_send, /* send */
:
:

• For receive operations, the address of a *push() function is provided by the protocol in the VCC structure.

•  Thee most significant change to the driver is moving to the use of standard skbuffs

Receive Buffer Management:
 
 

2 - APE 25 produces on to RRL_n via the RRLn_LFDA register

3 - Driver (interrupt service routine) consumes from RRL_n via ape->srrl[n]. RBH is produced to free RBH list. Skbuff is "pushed" to protocol.

4 - Protocol returns skbuff to driver. Driver consumes RBH from free RBH list and produces matched pair to the free list via ape->erfl.

TCP/IP over ATM
 

• Classical IP over PVC's (no signaling required.. but won't scale)
• Classical IP over SVC's (IEFT)
• LAN Emulation (ATM Forum)

Our problems included:

• Sleeping on lack of transmit resources
• Failure of ilmid to convey valid (vpi, vci) range to the switch.

Scope of the undertaking

Time: One to two person-months given

• Designer/Implementer with very gray hair/beard
• But no previous with Linux drivers or ATM drivers

Watch out for:

• Missing / erroneous information in the NIC spec.
• Assumptions relating to atomicity

• Assumptions relating to non-preemptability

• Export QoS support to the protocol
• Linux 2.1x compliant buffer management