FreeBSD - NeterionTrac - Trac

Wiki Navigation

Neterion FreeBSD Drivers

General Information

Driver name	if_nxge.ko Device Drivers Download
Supported adapters	All Neterion Adapters listed here.
Supported OS	FreeBSD 6.2 and 7.0
Hardware Architectures	x86, x86-64

Release Notes and Readme files

For information such as driver requirements, change logs, available downloads, and known issues, see the Release Notes.

For additional information regarding installation, performance suggestions, known issues, etc., please see the Readme.

Back to TOC

Measuring baseline performance

While GbE NICs typically run at GbE line rate on modern systems, 10GbE NICs cause "fast network, slow host" scenario and may or may not run at full 10GbE on a particular system. Typical bottlenecks on a particular system include PCI bus speed, system memory bandwidth, and/or CPU utilization. Also, network performance is very application-specific. For suggestions to improve performance, please refer to the Readme.

To measure baseline tcp throughput for your Neterion 10GbE Adapter, please connect two similar servers with Neterion NICs back-to-back, apply performance tuning tips from the next paragraph and run the following commands:

NTTCP:

This tool is available as port for FreeBSD and can be installed from /usr/ports/benchmarks/nttcp/ which requires internet access.

In Remote host, start nttcp server instance:

	# nttcp -i

In test host, start nttcp client instace(s) for Tx or Rx:
Transmit performance test:

	# nttcp -l <BUFFER_LENGTH> -w <WINDOW_SIZE> -t <XFRAME_IP_ADDRESS_OF_HOST1>

Example:

	# nttcp -l 65536 -w 227 -t 192.168.10.10

Receive performance test:

	# nttcp -l <BUFFER_LENGTH> -w <WINDOW_SIZE> -r <XFRAME_IP_ADDRESS_OF_HOST1>

Example:

	# nttcp -l 65536 -w 227 -r 192.168.10.10

iperf:

This tool is available as port for FreeBSD and can be installed from /usr/ports/benchmarks/iperf/ which requires internet access.

In remote host, start iperf server instance:

	# iperf -s

In test host, start iperf client instance(s)

	# iperf -c <XFRAME_IP_ADDRESS_OF_HOST1> -l <BUFFER_LENGTH> -W <WINDOW_SIZE> -t <DURATION_OF_TEST> -P <NUMBER_OF_STREAMS>

Example:

	# iperf -c 192.168.10.10 -l 64k -w 256k -t 3600 -P 4

Netperf:

This tool is available as port for FreeBSD and can be installed from /usr/ports/benchmarks/netperf/ which requires internet access.

In Remote host, start "netserver" server instance. "netserver" can be started either as a standalone or as a daemon

Starting netserver as a standalone server:

	# netserver -p <PORT_NUMBER>

Starting netserver as a daemon:

*01* Add this line to the /etc/services file:
	netperf 12865/tcp
This is to have 12865 TCP port for listening to netperf requests.
*02* Add the following line to the /etc/inetd.conf file:
	netperf stream tcp nowait root /opt/netperf/netserver netserver
*03* Restart inetd daemon.
	# kill -HUP <PROCESS_ID_OF_inetd_DAEMON>

In test host, start netperf client instace:

	# netperf -H <XFRAME_IP_ADDRESS_OF_HOST1>

Example:

	# netperf -H 192.168.10.10

For detailed procedure to run netperf please visit Netperf documentation page http://www.netperf.org/netperf/training/Netperf.html.

Please e-mail your performance results (along with server platform description) to support@neterion.com. Our support team will be able to confirm whether your results are expected for a given platform.

Back to TOC

Utilities

Hardware Statistics:

# getinfo nxgeX hwstats

Software Statistics:

# getinfo nxgeX swstats

Device Statistics

# getinfo nxgeX devstats

Driver statistics

# getinfo nxgeX drvstats

Driver Version

# getinfo nxgeX version

# getinfo nxgeX registers

Reading a specific register

# getinfo nxgeX getregister <REGISTER_OFFSET>

Writing to a register

# getinfo nxgeX setregister <REGISTER_OFFSET> <REGISTER_VALUE>

PCI Configuration Space:

# getinfo nxgeX pciconf

Device Configuration:

# getinfo nxgeX devconf

Querying Rx Buffer Mode

# getinfo nxgeX getbufmode

Changing Rx Buffer Modes

# getinfo nxgeX setbufmode <NEW_BUFFER_MODE>
Example:
    # getinfo nxge0 setbufmode 2
    Buffer mode changed to 2

Printing Usage

# getinfo
Usage: getinfo <INTERFACE> [hwstats] [swstats] [devstats] [drvstats] [version] [registers] [getregister offset] [setregister offset value] [pciconf] [devconf] [getbufmode] [setbufmode]
        INTERFACE   : Interface (nxge0, nxge1, nxge2, ..)
        hwstats     : Prints hardware statistics
        swstats     : Prints software statistics
        devstats    : Prints device statistics
        drvstats    : Prints driver statistics
        version     : Prints driver version
        registers   : Prints register values
        getregister : Read a register
        setregister : Write to a register
        pciconf     : Prints PCI configuration space
        devconf     : Prints device configuration
        getbufmode  : Prints Buffer Mode
        setbufmode  : Changes buffer mode

Back to TOC

Frequently Asked Questions (FAQ):

Q1: What version of driver am I using?

A: When driver is loaded, it prints device revision and driver version in /var/log/messages.

Copyright(c) 2002-2007 Neterion Inc.
nxge0: <Neterion Xframe 10 Gigabit Ethernet Adapter> mem 0xff4f0000-0xff4f7fff,0xff200000-0xff2fffff,0xff4fb800-0xff4fbfff irq 28 at device 3.0 on pci1
nxge0: Ethernet address: 00:0c:fc:00:0d:b8
nxge0: [FILTER+ITHREAD]
nxge0: XframeII 10 Gigabit Eth Revision 2 Driver v2.0.0.10992
nxge0: Serial Number SXT0544003
nxge0: Adapter is on 64 bit PCIX(M1) 133MHz Bus
nxge0: Using Line Interrupts
nxge0: TSO Enabled
nxge0: LRO Enabled
nxge0: Rx 1 Buffer Mode Enabled
nxge0: link state changed to DOWN
nxge0: link state changed to UP

In the above example,
 * Driver version: 2.0.0.10992
 * Adapter: "XframeII"
 * Revision: 2
 * Serial number: SXT0544003
 * MAC address: 00:0c:fc:00:0d:b8

Alternatively, we can get driver version (during runtime) by using getinfo tool.
# getinfo nxgeX version
Example:
# ./getinfo nxge0 version
===================================================================
Driver Version: 2.0.0.10992
===================================================================

Q2: How do I get PCI configuration information about Neterion adapter?

A: Use the getinfo tool that comes with the driver. 

To get pci configuration space for xge0 interface:
# getinfo nxge0 pciconf
===================================================================
PARAMETER               OFFSET  VALUE
===================================================================
device_id               0x0002  0x5832
vendor_id               0x0000  0x17D5
status                  0x0006  0x0230
command                 0x0004  0x0146
pciClass                0x0009  0x0000
revision                0x0008  0x0002
bist                    0x000F  0x0C00
header_type             0x000E  0x0000
latency_timer           0x000D  0x0040
cache_line_size         0x000C  0x4010
base_addr0_lo           0x0010  0x000C
base_addr0_hi           0x0014  0x0000
base_addr1_lo           0x0018  0x000C
base_addr1_hi           0x001C  0x0000
not_Implemented1        0x0020  0xB80C
not_Implemented2        0x0024  0x0000
cardbus_cis_pointer     0x0028  0x0000
subsystem_id            0x002E  0x6020
subsystem_vendor_id     0x002C  0x17D5
rom_base                0x0030  0x0000
rsvd_35                 0x0035  0x0000
capabilities_pointer    0x0034  0x0040
rsvd_38                 0x0038  0x0000
max_latency             0x003F  0x0501
min_grant               0x003E  0x01FF
interrupt_pin           0x003D  0xFF01
interrupt_line          0x003C  0x011C
msi_control             0x0042  0x018A
msi_next_ptr            0x0041  0x8A60
msi_cap_id              0x0040  0x6005
msi_lower_address       0x0044  0x0000
msi_higher_address      0x0048  0x0000
msi_unused              0x004E  0x0000
msi_data                0x004C  0x0000
vpd_addr                0x0052  0x0000
vpd_next_cap            0x0051  0x0000
vpd_cap_id              0x0050  0x0000
vpd_data                0x0054  0x0001
rsvd_b0                 0x0058  0x0000
pcix_command            0x0062  0x1025
pcix_next_cap           0x0061  0x2580
pcix_cap                0x0060  0x8007
pcix_status             0x0064  0x0118
===================================================================

You can also get PCI configuration information without using getinfo by using the pciconf command.
The following example gets PCI configuration space for xge0 interface:

# pciconf -l | grep 17d5
nxge0@pci1:1:0: class=0x020000 card=0x601c17d5 chip=0x583117d5 rev=0x04 hdr=0x00
nxge1@pci1:3:0: class=0x020000 card=0x602017d5 chip=0x583217d5 rev=0x02 hdr=0x00
# pciconf -r pci1:3:0 0:100
583217d5 02300146 02000002 00004010
ff4f000c 00000000 ff20000c 00000000
ff4fb80c 00000000 00000000 602017d5
fc500000 00000040 00000000 01ff011c
018a6005 00000000 00000000 00000000
00000000 00000001 00000000 00000000
10258007 5fe30118

Q3: How do I get the number of interrupts the device received since system startup?

A: Use "vmstat -i". 

# vmstat -i
interrupt                          total       rate
irq1: atkbd0                        1789          0
irq12: psm0                           20          0
irq14: ata0                       395905          0
irq18: uhci2                        9041          0
irq20: fxp0                       238469          0
cpu0: timer                    780208849       1910
cpu1: timer                    780208826       1910
irq256: nxge0                   15247486         37
Total                         1576310385       3860

Q4: How do I enable TSO feature in FreeBSD 7.0 driver?

A:  
 * Standalone driver: Set enable_tso to 1 in xge.conf before loading driver.
 * Driver in Kernel tree: Set kenv hw.xge.enable_tso to 1 before loading driver.

Q5: How do I enable LRO feature in FreeBSD 7.0 driver?

A:
 * Standalone driver: Set enable_lro to 1 in xge.conf before loading driver.
 * Driver in Kernel tree: Set kenv hw.xge.enable_lro to 1 before loading driver.

Q6: How do I enable LRO feature in FreeBSD 6.2 driver?

A: Enabling LRO requires two steps:
 * Enabling LRO support in FreeBSD kernel network stack.
 * Enabling LRO support in Neterion driver for FreeBSD.

 * Enabling LRO support in FreeBSD kernel network stack.
   Using LRO feature in driver requires applying a patch in FreeBSD kernel network stack 
   which requires compiling kernel (Please read the next FAQ about compiling FreeBSD kernel). 
   This is essential since driver once LRO enabled, sends packets with size more than interface 
   MTU to network stack during receive operation. But the network stack should accept them 
   rather than discarding them as oversize packets. Please comment out the following lines 
   in "ether_input" function in "src/sys/net/if_ethersubr.c" to enable LRO support in FreeBSD 
   network stack. 

        if (m->m_pkthdr.len >
            ETHER_MAX_FRAME(ifp, etype, m->m_flags & M_HASFCS)) {
                if_printf(ifp, "discard oversize frame "
                                "(ether type %x flags %x len %u > max %lu)\n",
                                etype, m->m_flags, m->m_pkthdr.len,
                                ETHER_MAX_FRAME(ifp, etype,
                                                m->m_flags & M_HASFCS));
                ifp->if_ierrors++;
                m_freem(m);
                return;
        }


 * Enabling LRO support in Neterion driver for FreeBSD.
   Enabling LRO in driver is a compile-time option. Uncomment the following line in driver's 
   Makefile and build the driver.
   (To uncomment the following line, just remove the "#" character in the line).

   #LRO_CFLAGS = -DCONFIG_LRO -DXGE_HAL_CONFIG_LRO

Q7: How do I compile the FreeBSD kernel?

A: We assume that the FreeBSD kernel sources are at "/usr/src/" directory. 
Please follow the steps mentioned below to compile the kernel.

 1. Update the kernel source with required changes.

 2. Change directory to "/usr/src/sys/<arch>/conf/" where <arch> is the architecture of the 
    machine in which we compile the kernel. Please run "sysctl hw.machine" to find which 
    archicture are you using. Following is the example from an amd64 machine.
    # sysctl hw.machine
    hw.machine: amd64

 3. Have a copy of the GENERIC kernel configuration file.
    # cp GENERIC MYKERNEL

 4. Run config command for the MYKERNEL to build system configuration files.
    # config MYKERNEL

 5. Change directory to "/usr/src/sys/<arch>/compile/MYKERNEL/"
    # cd /usr/src/sys/<arch>/compile/MYKERNEL"

 6. Run "make cleandepend" followed by "make depend" and finally "make" in the same directory 
    to complete kernel compilation.
    # make cleandepend && make depend
    # make

 7. Install the newly build kernel
    # make install

 8. Reboot the machine to boot with new kernel.
    # reboot

Q8: Why do many packets get lost when I run flood ping to Xframe on FreeBSD?

A: This can be due to using default value (200) for the sysctl "net.inet.icmp.icmplim". This sysctl
stores the maximum number of ICMP "Unreachable" and also TCP RST packets that will be sent back
every second. For running flood ping it is suggested to use a higher value.

Example:
Getting current value for sysctl "net.inet.icmp.icmplim":
    # sysctl net.inet.icmpl.icmplim
    net.inet.icmp.icmplim: 200

Setting a new value 2000000000 to sysctl "net.inet.icmp.icmplim":
    # sysctl net.inet.icmp.icmplim=2000000000
    net.inet.icmp.icmplim: 200 -> 2000000000

Q9: How do I enable MSI feature?

A: MSI (Message Signaled Interrupts) are supported in FreeBSD 7.0 driver.
 * Standalone driver: Set enable_msi in xge.conf to 1 before loading driver.
 * Driver in Kernel tree: Set kenv hw.xge.enable_msi to 1 before loading driver.
If MSI feature is enabled, driver prints interrupt type in log during load.
    MSI enabled : nxge0: Using MSI Interrupts
    MSI disabled: nxge0: Using Line Interrupts

Q10: How do I enable/disable TSO feature?

A: TSO can be enabled/disabled either through ifconfig command or before loading driver.
 * Through ifconfig command: Run "ifconfig nxgeX tso" to enable TSO and "ifconfig nxgeX -tso" to disable it.
 * Driver in Kernel tree: Set kenv hw.xge.enable_tso to 1 before loading driver.
 * Standalone driver: Set enable_tso in xge.conf to 1 before loading driver.

Q11: How do I enable/disable LRO feature?

A: LRO can be enabled/disabled either through sysctl or before loading driver.
 * Through sysctl: Run "sysctl dev.nxge.X.enable_lro=1" to enable LRO and "sysctl dev.nxge.X.enable_lro=0" to disable it.
 * Before loading driver:
     * Driver in Kernel tree: Set kenv hw.xge.enable_lro to 1 before loading driver.
     * Standalone driver: Set enable_lro in xge.conf to 1 before loading driver.

Last updated on Wed Nov 19 14:00:44 2008

Download in other formats:

Plain Text

Navigation

Wiki Navigation

Table of Contents

General Information

Release Notes and Readme files

Measuring baseline performance

Utilities

Frequently Asked Questions (FAQ):

Q1: What version of driver am I using?

Q2: How do I get PCI configuration information about Neterion adapter?

Q3: How do I get the number of interrupts the device received since system startup?

Q4: How do I enable TSO feature in FreeBSD 7.0 driver?

Q5: How do I enable LRO feature in FreeBSD 7.0 driver?

Q6: How do I enable LRO feature in FreeBSD 6.2 driver?

Q7: How do I compile the FreeBSD kernel?

Q8: Why do many packets get lost when I run flood ping to Xframe on FreeBSD?

Q9: How do I enable MSI feature?

Q10: How do I enable/disable TSO feature?

Q11: How do I enable/disable LRO feature?

Download in other formats: