FreeLists / openbeosnetteam / [openbeosnetteam] net buffers

["Axel Dörfler" <axeld@xxxxxxxxxxxxxxxx>]

Hi there,

I had a long talk with Marcus yesterday, as we were discussing the 
various concepts (mbuf vs. bone_data vs. sk_buff) as well as the needs 
of current (Gigabit ethernet) hardware.

To summarize, I give you a short overview over each technique combined 
with a short pro/con list.

1) mbuf/BSD: fixed 256 byte allocations of buffers as well as 2k 
allocations for larger chunks of memory (so called mbuf clusters). The 
256 byte structure is divided into a header and a data section, if the 
mbuf has more buffers chained together and/or external buffers (those 
2k buffers), the data region of the 256 byte structure is not used.
pro:
* very flexible, can be built around existing data buffers
* can share buffers
* cheap prepending/appending/splitting
* relatively cheap allocator (only 2 different buffer sizes)
con:
* very messy API
* is likely to fragment data into several mbufs (there is no space left 
for headers)
* always copies the user data at least once
* no linear buffer
* potentially wastes memory (fixed sized mbuf structure with external 
data)

2) bone_data/BeOS: a buffer consists of several smaller data buffers, 
joined via iovecs. Does not copy or allocate the data buffers, but only 
puts them into the queue. Has a separate structure to put the header 
information into.
pro:
* very flexible, can be built around existing data buffers
* can share buffers
* very cheap prepending/appending/splitting
* may not need to copy the data at all in the case of out-of-band data 
and a new networking card
* clean API
con:
* fragments the data a lot - every header added is placed in a separate 
chunk of memory
* many allocations, two for each buffer
* needs a special treatment for userland buffers
* no linear buffer

3) sk_buff/Linux: variable sized allocations, single buffer per packet, 
leaves space in the beginning for additional headers, and space at the 
end for additional data. Also has a separate structure for the header.
pro:
* linear buffer
* no fragmentation of data
* clean API
* very cheap prepending/appending *if* there is enough space left
* must always copy data at least once
con:
* needs a more complex allocator
* splitting the buffer is expensive (fragmenting existing packets)
* appending/prepending can get very expensive, too (if the original 
reserved space was not enough) - this might be lifted by intelligent 
stack design (which would then potentially wastes more memory)
* does not allow to share data buffers (only allows for read-only 
clones)

I've thought a bit about it, and I think best suited for us might be a 
mix between mbufs and sk_buffs:
- a separate header structure with a list of attached buffers
- fixed sized 2k buffers with a header and reserved space to place 
headers

Since a standard ethernet frame is 1500 bytes long, and a jumbo frame 
is 9000 bytes, we would always need a single buffer in the ethernet 
frame case (and can still leave a generous amount of space for 
headers), and 5 for the jumbo frames at maximum.
The buffers have reference counters and may be shared between multiple 
packets.

pro:
* clean API
* little fragmentation of data, perfect for high speed ethernet
* can share buffers
* simple allocator
* flexible (would work with variable buffer sizes as well)
* very cheap prepending/appending if there is enough space left
* still cheap prepending/appending if a new buffer is needed
* splitting a buffer is cheap
* must always copy data at least once
con:
* since a buffer has a header, it cannot be built from an existing data 
buffer
* usually a linear buffer, but you can't count on it
* potentially wastes a lot of memory - if this starts to become a 
problem, we can easily introduce a second default buffer size; it 
shouldn't matter that much for the non-server case, though

The current net_buffer API would defnitely look different than it does 
now, but what do you think? Does this sound like something we want to 
pursue? I haven't worked out the details yet, but the only questionable 
thing is how to design the buffer sharing properly (I have an idea for 
now, but that's not that clean).

Bye,
   Axel.