Readme for Neterion's FreeBSD Drivers_
Table of Contents
Introduction
FreeBSD Driver for Neterion's Xframe 10GbE Server/Storage adapters
- Drivers support all Neterion's Xframe 10GbE adapters, FreeBSD 6.2, 7.0, and 8.0
- Supports both i386 and amd64 architectures
- FreeBSD 6.2 Features: Jumbo frames (up to 9600), LRO (Large Receive Offload), 2 and 3 buffer modes, Promiscuous mode, and multiple Xframe devices.
- FreeBSD 7.0/8.0 Features: Same as above. Also, Extended Message Signaled Interrupts (MSI-X), TCP Segmentation Offload (TSO), and 5 buffer mode.
Installation Instructions
Identifying the Adapter:
The Xframe adapter is identified by the board ID number on
the adapter.
Look for a label that has a barcode and a number, for example,
SXT0425072. The globally unique serial number identifies where
the card is built ('T' for Thailand), and the manufacturing date
(04 for 2004, 25 for work week 25). The factory-burned MAC
address (hardware address) shows up on the board above the
serial number, (similar to 000CFC000449 -- 00:0C:FC:00:04:49).
Driver Source Package:
freebsd (FreeBSD 6.2 packages)
This directory contains FreeBSD driver sources for Xframe device(s), Makefile,
the shell script (param.sh) used to update the configuration parameters,
and configuration file (xge.conf).
xge (FreeBSD 7.0/8.0 package)
This directory contains FreeBSD driver sources for Xframe device(s), Makefile,
and Xframe Hardware Abstraction headers and sources (xgehal and include folders).
xgehal
This directory contains the Xframe HAL sources. The driver uses HAL to perform
operations on the Xframe hardware.
include
The include subdirectory contains HAL header files.
In FreeBSD 7.0/8.0 package, include and xgehal will be within xge directory.
The source code for the getinfo tool (used to get statistics, pciconfig and
register dump) are included in the freebsd directory (or src/tools/tools/xge for 7.0/8.0).
Building the driver:
The driver is distributed in the source form. Driver and installation
utility executables need to be built for the target platform.
In the directory containing Makefile for building Neterion driver for FreeBSD,
(FreeBSD 6.2: freebsd, or FreeBSD 7.0/8.0: src/sys/modules/xge)
# make clean
# make
Please unload the previously installed Neterion drivers before
proceeding to the following steps.
# make uninstall
Loading the driver:
Use "kldload" to load driver module if_xge.ko.
# kldload ./if_xge.ko
Run "kldstat" and find an entry for if_xge kernel module to make
sure that the driver installation was successful.
# kldstat | grep if_xge
3 1 0xc22cc000 26000 if_xge.ko
Enabling interface and assigning IP address:
# ifconfig <INTERFACE> <IP_ADDRESS> up
<INTERFACE> will be similar to xge0, xge1 etc. and can be
found by executing "ifconfig -a".
Neterion adapters typically have MAC addresses starting with
"00:0C:FC" or "00:11:25".
Example:
# ifconfig xge0 10.2.2.40 up
Enables the xge0 interface and assigns to it the IP
address 10.2.2.40.
xge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
inet6 fe80::211:25ff:fe7b:c71%xge0 prefixlen 64 scopeid 0x5
inet 10.2.2.40 netmask 0xff000000 broadcast 17.255.255.255
ether 00:11:25:7b:0c:71
media: Ethernet autoselect (10GBASE-SR <full-duplex>)
status: active
Disabling the interface:
# ifconfig <INTERFACE> down
Example:
# ifconfig xge0 down
Unloading the Driver:
For FreeBSD 6.2, run the below command within freebsd directory.
For FreeBSD 7.0/8.0, run the below command within src/sys/modules/xge directory.
# kldunload if_xge.ko
Features:
a. Jumbo frames:
Xframe I/II supports MTU up to 9600 bytes, modifiable using ifconfig command.
b. LRO (Large Receive Offload):
For FreeBSD 6.2:
Apply patch in freebsd directory and compile the kernel. LRO is a compile-time
option, which can be enabled in the driver by compiling the driver with
following macros defined. Uncomment the below line in Makefile.
CFLAGS_LRO = -DCONFIG_LRO -DXGE_HAL_CONFIG_LRO
For FreeBSD 7.0/8.0:
Standalone driver: Set enable_lro in xge.conf to 1 before loading driver.
Driver in Kernel: Set kenv hw.xge.enable_lro to 1 before loading driver.
Through sysctl: Set "sysctl dev.nxge.X.enable_lro=1" to enable LRO.
c. Buffer modes:
For FreeBSD 6.2:
Supports 2 & 3 buffer modes.
For FreeBSD 7.0/8.0:
The driver supports 2, and 5 buffer modes. Driver uses 5 buffer mode for
all MTUs > 4074. 2 buffer mode can only be used for all MTUs <= 4074.
Standalone driver: Set buffer_mode in xge.conf to 2 or 3
Driver in Kernel: Set kenv hw.xge.buffer_mode to 2 or 3
(2: Rx 2 buffer mode, 3: Rx 3 buffer mode)
Buffer modes can be changed using getinfo tool. 1 buffer mode will be used
by the driver by default if none of the buffer mode macros are enabled in
Makefile.
# getinfo <INTERFACE> setbufmode <NEW_BUFFER_MODE>
Examples:
Changing buffer mode to 2: # getinfo nxge0 chgbufmode 2
Changing buffer mode to 3: # getinfo nxge0 chgbufmode 3
Changing buffer mode to 1: # getinfo nxge0 chgbufmode 1
Current buffer mode can be found using getinfo tool.
# getinfo <INTERFACE> getbufmode
Example (executed while driver was using 2 buffer mode):
# getinfo nxge0 getbufmode
Rx Buffer Mode: 1
d. TSO (TCP Segmentation Offload in FreeBSD 7.0/8.0):
TSO can be enabled/disabled either through ifconfig command or before loading driver.
* Through ifconfig command: Run "ifconfig nxgeX tso" to enable TSO and "ifconfig nxgeX -tso" to disable it.
* Driver in Kernel tree: Set kenv hw.xge.enable_tso to 1 before loading driver.
* Standalone driver: Set enable_tso in xge.conf to 1 before loading driver.
e. LRO (Large Receive Offload in both FreeBSD 6.2 and FreeBSD 7.0/8.0):
FreeBSD 6.2: This requires modifying Kernel. Please refer to Q6 in FAQs.
FreeBSD 7.0/8.0: LRO can be enabled/disabled either through sysctl or before loading driver.
* Through sysctl: Run "sysctl dev.nxge.X.enable_lro=1" to enable LRO and "sysctl dev.nxge.X.enable_lro=0" to disable it.
* Before loading driver:
* Driver in Kernel tree: Set kenv hw.xge.enable_lro to 1 before loading driver.
* Standalone driver: Set enable_lro in xge.conf to 1 before loading driver.
xge.conf:
xge.conf:
xge.conf could be loaded by executing the shell script param.sh. Essentially, the driver
parameters can be changed as follows:
a. By modifying the xge.conf and then executing param.sh shipped with the source code.
b. Using the command kenv and adding a prefix of "xge.hw".
For example, to change "ring_queue_max" from default value to 10; execute
kenv xge.hw.ring_queue_max=10
Make sure to reload the driver after changing the parameters.
Performance Suggestions
TCP Tuning
TCP Tuning a. sysctl net.inet.tcp.sendspace=786432 b. sysctl net.inet.tcp.recvspace=786432 c. sysctl kern.ipc.maxsockbuf=8388608
Usage & Troubleshooting:
For general information and support, please visit Neterion support website at http://www.neterion.com/support/support.html
Make sure that the operating system identifies the Xframe adapter. Note that Neterion vendor ID is 0x17D5 and Xframe I/II adapters can be fixed to both PCI and PCI-X slots.
The rest of this section details troubleshooting tips and information. Some of them are general and some are more specific. For online Troubleshooting tips and faqs, please visit http://trac.neterion.com/cgi-bin/trac.cgi/wiki/TitleIndex?anonymous
Loading the driver and initializing the device:
The first time FreeBSD identifies the device it stores the corresponding device/vendor IDs in an enumerated tree of PCI devices. Note that Neterion vendor id is 0x17d5. After cold reboot FreeBSD finds the device and tries to load the corresponding driver. If it fails, try to switch the card and/or cable. And, in parallel, send us the "tail" of the "/var/log/messages". We also need traces, register dump and statistics(Use getinfo tool at src/tools/tools/xge) .
Collect information
If you're reporting a problem to Neterion, please describe: host(s) and adapter(s), switch (if used), software version.
ARP
If this is a basic connectivity issue (e.g., cannot connect, cannot ping), make sure first that ARP works Do you see ARPs coming through a switch (in case switch is used)? Do you see frame drops at the switch when pinging? Do you see frame counts increasing via statistics?
Have you tried
a. A previous driver release;
b. A different adapter in the same PCI slot;
c. A different PCI slot;
d. Back-to-back setup so that the switch is excluded from the equation.
e. To replace the cables?
f. To use a different PCI slot?
Start clean
a. Have you tried to reboot the switch? Cold-reboot the host?
b. Make sure that the latest released driver gets loaded after the host
reboot, and that the 10GE interface shows up via ifconfig.
LEDs
Do you see LED going green after everything is connected and drivers loaded? How/when does the color changes?
ifconfig
Run "ifconfig -a" at the command prompt and check whether the output Looks as expected. Include the output in your problem report. Note for instance that "all-foxes" i.e, FF:FF:FF:FF:FF:FF MAC address could explain a general connectivity issue, if that's what you see.
Log
Please attach the generated log, with traces enabled. Note that the driver's logging facility is configurable at compile-time. Errors and traces can be compiled out on a per-component basis. The components are: HAL fifos and rings, device, link layer driver, etc., see XGE_IO_COMPONENT_HAL_??? in the Makefile. Here's how you enable all except data path traces: TRACE_CFLAGS = -DXGE_IO_DEBUG_MODULE_MASK=0xffffffbf \ -DXGE_IO_DEBUG_TRACE_MASK=0xffffffb1 \ -DXGE_IO_DEBUG_ERR_MASK=0xffffffbf Recompile with traces and include the log in the report.
Statistics
To print hardware and software statistics for interface instance 0 (i.e., nxge0),
run:
# getinfo nxgeX hwstats
# getinfo nxgeX swstats
# getinfo nxgeX devstats
# getinfo nxgeX drvstats
# getinfo nxgeX pciconf
to print hardware statistics and PCI statistics.
The getinfo tool generates log file in the working directory. Once done, ping a
few times, and collect the statistics again (Ping both from this and the remote
machines).
Many counters could be of interest. For example, "rmac_vld_frms" counts all
valid incoming Ethernet frames seen by the adapter. The "rmac_discarded_frms"
counts frames discarded by the adapter. Information could be derived from the
fact that (for instance) both counters stay constant during ping, if that is
what happening. For detailed description of the Xframe counters, please refer
to the "Xframe User Guide".
Please include the statistics into your problem report.
Xframe registers
Use getinfo to dump all Xframe BAR0 space registers. Include this register dump into your problem report. # getinfo nxgeX registers
PCI configuration space
Use getinfo to retrieve PCI vendor, device, etc. Include the PCI configutation space in your problem report. # getinfo nxgeX pciconf
netstat
Run netstat to see whether anything "percolates" up the stack. Send us the ICMP and Ethernet counters.
Device configuration
Use getinfo to retrieve device configuration information like fifo/ring memblock size, fifo length, ring size, mtu size etc. Include the Device configuration output in your problem report. # getinfo nxgeX devconf
Xframe & Driver configuration: xge.conf
Driver uses persistent configuration in a form of a text file. Via xge.conf user can change configurable parameters without re-building the driver. The user can tune-up driver performance or adjust memory footprint for a certain deployment scenario(s). This file contains a set of (name, value) pairs. The value -1 has a special meaning: use hard-coded default. (e.g., ring_main_max = -1; This is interpreted as: use the number of RxD blocks #defined in the driver's source.) Generally, the (-1) cancels overriding the hardcoded configuration and allows driver/HAL to decide values based on platform or MTU etc. Please note that changing values without having knowledge on these parameters can lead to system crash/panic or driver may not get loaded. The xge.conf contains the following attributes.
enable_tso
Enables TSO (TCP Segmentation Offload) feature in driver. Range: 0 - 1
enable_lro
Enables LRO (Large Receive Offload) feature in driver. Range: 0 - 1
enable_msi
Enable MSI (Message Signaled Interrupts) feature in driver. Range: 0 - 1
buffer_mode {{ Sets Rx buffer mode. Range: 1 - 3
VALUE BUFFER MODE
1 1 buffer mode 2 2 buffer mode 3 3 buffer mode }}}
fifo_max_frags
Sets the maximum number of Tx buffers that would be allocated for each TxDL. Range: 1 - 256. (-1: 18)
latency_timer
Sets the latency timer register in the PCI configuration space. For details, please refer to PCI specification. Range: 8 - 255. (-1: BIOS DEFAULT)
max_splits_trans
Sets the maximum number of split transactions the device is permitted to have
outstanding at one time. This value must be tuned with in-depth knowledge of the
system. For details, please refer to PCI-X specification.
Range: 0 - 7. (-1: BIOS DEFAULT)
Register Value Maximum Outstanding
0 1
1 2
2 3
3 4
4 8
5 12
6 16
7 32
mmrb_count
Sets the maximum byte count the device uses when initiating a sequence with one
of burst memory read commands. For details, please refer to PCI-X specification.
Range: 0 - 3. (-1: BIOS DEFAULT)
MMRB VALUE BYTE COUNT
0 512 B
1 1 KB
2 2 KB
3 4 KB
shared_splits
Sets the number of Outstanding Split Transactions that is shared by Tx and Rx requests. The device stops issuing Tx requests once the number of Outstanding Split Transactions is equal to the value of shared_splits. If the shared_splits is less than Maximum Outstanding Split Transactions field in the PCIX Command register, the device is still able to issue Rx read requests. A value of zero indicates that the Tx and Rx share all allocated Split Requests, i.e. the device can issue both types (Tx and Rx) of read requests until the number of Maximum Outstanding Split Transactions is reached. For details, please refer to PCI-X specification. Range: 0 - 31
stats_refresh_time_sec
Sets the time interval in seconds for automatic statistics transfer by the Xframe.
isr_polling_cnt
Sets the maximum number of times the driver's ISR "polls" for Tx and Rx completions. During the processing of completed descriptors by the driver, the device continues to DMA new completions. The value greater than zero "allows" the driver to loop back to check whether any newly completed descriptors were accumulated, and if yes, process them without leaving the interrupt context. Range: 0 - 65536.
mc_pause_threshold_q0q3 / mc_pause_threshold_q4q7
Sets the threshold value for pause frame generation for queue 0 through 3 and
queue 4 through 7. The threshold value indicates portion of the individual
receive buffer queue size. When the amount of data in the receive queue exceeds
the threshold, a pause frame is transmitted, provided that pause frame
generation was enabled. The threshold value is calculated as such,
Threshold % = Threshold_value / 256
The threshold value indicates portion of the individual receive buffer queue
size. Thresholds have a range of 0 to 255, allowing 256 possible watermarks in a
queue.
Range: 0 - 254.
rmac_pause_gen_en
Enables the received pause generation. Range: 0 - 1
rmac_pause_rcv_en
Enables the receive pause. Range: 0 - 1
rmac_pause_time
Sets the "pause time" value to be inserted in the outgoing pause frames. The value has units of pause quanta (one quanta = 512 bit times). Range: 16 - 65535.
tmac_util_period
Sets the sampling period over which the transmit link utilization is calculated.
Range: 0 - 0xF.
The different sampling periods are as follows:
0x0: 6.4 us 0x1: 25.6 us 0x2: 102.4 us
0x3: 409.6 us 0x4: 819.2 us 0x5: 1.6 ms
0x6: 6.6 ms 0x7: 13.1 ms 0x8: 26.2 ms
0x9: 52.4 ms 0xA: 104.9 ms 0xB: 209.7 ms
0xC: 419.4 ms 0xD: 838.9 ms 0xE: 1.7 sec
0xF: calculation disabled.
rmac_util_period
Sets the sampling period over which the receive link utilization is calculated.
Range: 0 - 0xF.
The different sampling periods are as follows:
0x0: 6.4 us 0x1: 25.6 us 0x2: 102.4 us
0x3: 409.6 us 0x4: 819.2 us 0x5: 1.6 ms
0x6: 6.6 ms 0x7: 13.1 ms 0x8: 26.2 ms
0x9: 52.4 ms 0xA: 104.9 ms 0xB: 209.7 ms
0xC: 419.4 ms 0xD: 838.9 ms 0xE: 1.7 sec
0xF: calculation disabled.
rmac_bcast_en
Enable/disable frames containing broadcast addreess to be passed to the host. Range: 0 - 1 (0 - Disable, 1- Enable)
fifo_memblock_size
Sets the memory block size used for allocating fifo descriptors. Setting value to page size ensures by-page allocation of descriptors. 128K bytes is the maximum supported block size. Range: 4096 - 131072 (Multiples of page size are adviced)
ring_memblock_size
Sets the memory block size used for allocating ring descriptors. Setting value to page size ensures by-page allocation of descriptors. 128K bytes is the maximum supported block size. Range: 4096 - 131072 (Multiples of page size are adviced)
fifo_reserve_threshold
Sets the minimal number of free transmit descriptors that will be preserved in the fifo at all times. Range: 0 - 8192.
fifo_queue_max
Sets the maximum number of transmit descriptor lists (TxDL's) for the fifo 0. Up to 8192 transmit descriptor lists can be supported by Xframe at any given time, and each FIFO can be configured to have any fraction of this total depth, provided the total does not exceed 8192. Range: 2 - 8192 (-1: 2048)
fifo_queue_initial
Sets the initial number of transmit descriptor lists (TxDL's) for the fifo 0. Up to 8192 transmit descriptor lists can be supported by the firmware at any given time, and each FIFO can be configured to have any fraction of this total depth, provided the total does not exceed 8192. Range: 2 - 8192 (-1: 2048)
fifo_queue_intr
Sets the interrupt mode for the fifo 0. Setting this value to 1 causes "interrupt per transmit operation", or more exactly, per each completed list (TxDL) of transmit descriptors. Range: 0 - 1 (1 - Use list interrupt, 0 - Do not use list interrupt)
fifo_tti_timer_val_us
Sets the time interval in microseconds, at which transmit timer interrupt is to be generated. Note that unless fifo_tti_timer_ci_en is set, the timer interrupt is generated only in presence of the transmit traffic. Note also that timer interrupt and utilization interrupt are two separate interrupt sources. Note that Xframe "counts" time in PCI clock cycles. Range: 0 - 65535.
fifo_tti_timer_ci_en
Enable/disable generation of a continuous interrupt. Setting value to 1 results in generation of continuous interrupt at fixed fifo_tti_timer_val_us intervals of time, independently of whether there is a transmit traffic or not. Note that for the purposes of periodic and/or slow-path processing HAL provides a separate so-called "scheduled interrupt" Range: 0 - 1 (0 - Disable, 1 - Enable)
fifo_tti_timer_ac_en
Enable/disable auto-cancel. Resets the timer if utilization interrupt was generated during the interval. Range: 0 - 1 (0 - Disable, 1 - Enable)
ring_queue_indicate_max_pkts
Sets the maximum number of Rx packets in this ring that can be delivered to the link layer during the processing of receive interrupt. Range: 1 - 65536.
fifo_tti_urange_a/fifo_tti_urange_b/fifo_tti_urange_c
Sets the transmit link utilization range A/B/C. Values are in percentage. Range: 1 - 100
fifo_tti_ufc_a/fifo_tti_ufc_b/fifo_tti_ufc_c/fifo_tti_ufc_d
Sets the frame count for the transmit utilization range A/B/C/D. Interrupt will be generated each time the line is utilized not more than fifo_tti_urange_a percent in the transmit direction and number of transmitted frames is greater than or equal to fifo_tti_ufc_a. Range: 0 - 65535
ring_queue_rti_urange_a/ring_queue_rti_urange_b/ring_queue_rti_urange_c
Sets the receive link utilization range A/B/C for the ring XGE_LL_RING_MAIN_QID (ring main). Range: 0 - 127
ring_queue_rti_ufc_a/ring_queue_rti_ufc_b/ring_queue_rti_ufc_c/ring_queue_rti_ufc_d
Sets the frame count for the receive link utilization range A/B/C/D for the ring main. Interrupt will be generated each time when (and only when) the line is utilized no more than ring_queue_rti_urange_a percent inbound, and number of received frames is greater or equal ring_queue_rti_ufc_a. Range: 0 - 65535 Example: With settings ring_queue_rti_urange_a = 10 & ring_queue_rti_ufc_a = 1 interrupt will be produced for each received packet if the inbound link utilization is less or equal 1Gbps.
ring_queue_rti_timer_ac_en
Enable/disable auto-cancel. Resets the timer if utilization interrupt was generated during the interval. Range: 0 - 1 (0 - Disable, 1 - Enable)
ring_queue_rti_timer_val_us
Interval of time, in microseconds, at which receive timer interrupt is to be generated. The timer interrupt is generated only in presence of the inbound traffic. Range: 0 - 65535
ring_queue_max
Sets the maximum number of RxD blocks for the ring main. Xframe supports up to 64 RxD blocks. Each RxD block has fixed size of 4KB. Each block contains a fixed number of receive descriptors (RxDs). Size of the receive descriptor depends on the configured receive buffer mode. For instance, one-buffer mode descriptor is 32 bytes long. That means, single RxD block contains 127 (128 - 1) such descriptors. Range: 1 - 64. (-1: 16)
ring_queue_initial
Sets the initial number of RxD blocks for the ring XGE_LL_RING_MAIN_QID ("ring
main"). It can grow up to ring_main_max.
Range: 1 - 64. (-1: 16)
ring_queue_dram_size_mb
Sets the size of the ring within the Memory Controller's buffer. Each ring can be assigned a value between 0 and 64, corresponding to a size of 0 to 64 MB. The sum of all ring sizes must be 64 MB. The queues default to 8 MB each. Range: 0 - 64MB (Xframe I), 0 - 32 (Xframe II).
ring_queue_backoff_interval_us
Sets the time (in microseconds), after which Xframe tries to download RxDs posted by the host for the ring main after the previous attempt to get RxDs had failed. Note that the "backoff" does not happen if host posts receive descriptors in the timely fashion. Range: 1 - 125000
NOTE: Parameters fifo_reserve_threshold, fifo_queue_max and fifo_queue_initial along with a hardcoded parameter tx_level_low (hardcoded in xgell.h) are inter-related. For transfer to happen, fifo_reserve_threshold value should be less than (fifo_queue_max - tx_level_low). For example, for fifo_queue_max = 2047, tx_level_low = 16, fifo_reserve_threshold should be less than (2047 - 16) = 2031. That means, we have to set fifo_reserve_threshold to a value less than or equal to 2030 for transfer to happen.
Known Issues
1. FreeBSD 8.0 driver is currently in Beta development and test.
2. Both ring_queue_initial and ring_queue_max are set to 2 by default.
Using higher values to ring_queue_initial and ring_queue_max may cause
memory allocation failures in some systems.
NOTE: Both ring_queue_initial and ring_queue_max should have same values.
To use a different value, <NEW_VALUE>,
Standalone driver:
Update configuration file xge.conf before loading driver.
ring_queue_initial <NEW_VALUE>
ring_queue_max <NEW_VALUE>
Driver in Kernel:
Create/update kenv before loading driver.
# kenv hw.xge.ring_queue_initial=<NEW_VALUE>
# kenv hw.xge.ring_queue_max=<NEW_VALUE>
Available Downloads
For latest available Neterion drivers or further support please contact your network adapter provider or check the Neterion support downloads site at http://www.neterion.com/support/xframe_customers.html
===================================================================
Neterion Inc. Proprietary COPYRIGHT (c) 2002-2008 Neterion Inc, ALL RIGHTS RESERVED
