Subject: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc.

From: Costa Sapuntzakis <[email-masked]>
Date: Thu, 24 Feb 2000 13:56:46 -0800 (PST)

The TCP RDMA option reduces the overhead of receiving data over
TCP-based protocols such as NFS and HTTP.  

It enables the construction of a simple hardware accelerator that
copies data directly from the incoming packet into application
buffers, avoiding expensive copies in the protocol stack.  Even
without hardware acceleration, the option enables the protocol stack
to decrease the number of copies it must do.

The TCP RDMA option is an annotation and requires no modifications to
higher layer protocols. It can be used with popular protocols such as 
HTTP, NFS, and CIFS, along with new protocols.

The TCP option also provides a bit to indicate application-level
message boundaries. The bit enables out-of-order processing of the TCP
receive queue, potentially decreasing service times in the presence of
packet drops and improving performance on parallel systems.

A draft describing the TCP RDMA option can be found at:
ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt

-Costa

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 24 Feb 2000 14:59:38 -0800 (PST)

> TCP-based protocols such as NFS and HTTP.  

Do you have any data (simulation, implementation) to back up this claim?
Or did you mean to say "provides a capbility which an implementation can
use to try to reduce the overhead"?
 
> to decrease the number of copies it must do.

This seems to be an overstatement as well. Are you saying that an
implementation that currently has a single copy in its receive path
(from kernel to user space) can "reduce" the number of copies without
any hardware acceleration? That would imply that the number of
copies could be reduces to zero which I have a hard time understanding
(unless you add hardware acceleration).

> HTTP, NFS, and CIFS, along with new protocols.

How will the higher layer protocol peer know what RDMA offsets to use?
For this to provide benefits to e.g. NFS it seems like you'd want the
NFS peer to have offsets that would depend on which file is being written.
Otherwise the only benefit would be for the NIC to be able to separate all
the protocol headers from the data so that the data can be placed on 
e.g. contigiuos pages in memory.

> ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt

There is no DNS entry for ftpeng.cisco.com so I can't access the document.

Thanks,
   Erik

---

From: "David S. Miller" <[email-masked]>
Date: Thu, 24 Feb 2000 15:57:10 -0800

Date: Thu, 24 Feb 2000 14:59:38 -0800 (PST)
   From: Erik Nordmark <[email-masked]>

As an aside I think the RDMA proposal has a lot of holes too.  For
example, there are in-kernel HTTP accelerators that do the complete
client header parse and initial packet response in the hw interrupt
handler.  There are no user buffers involved, and static response
data is DMA'd directly from the filesystem page cache.

   > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt

   There is no DNS entry for ftpeng.cisco.com so I can't access the
   document.

Here is what I get:

? host -a ftpeng.cisco.com
Trying null domain
rcode = 0 (Success), ancount=1
The following answer is not authoritative:
The following answer is not verified as authentic by the server:
ftpeng.cisco.com        84574 IN        CNAME   ftp-eng.cisco.com
For authoritative answers, see:
cisco.com       38435 IN        NS      NS1.cisco.com
cisco.com       38435 IN        NS      NS2.cisco.com
Additional information:
NS1.cisco.com   78995 IN        A       192.31.7.92
NS2.cisco.com   67536 IN        A       192.135.250.69
rcode = 0 (Success), ancount=1
The following answer is not authoritative:
The following answer is not verified as authentic by the server:
ftp-eng.cisco.com       84574 IN        A       198.92.30.33
For authoritative answers, see:
CISCO.com       38435 IN        NS      NS1.CISCO.com
CISCO.com       38435 IN        NS      NS2.CISCO.com
Additional information:
NS1.CISCO.com   78995 IN        A       192.31.7.92
NS2.CISCO.com   67536 IN        A       192.135.250.69

---

From: "David S. Miller" <[email-masked]>
Date: Thu, 24 Feb 2000 16:10:26 -0800

Date: Thu, 24 Feb 2000 17:09:51 -0700
   From: "Justin T. Gibbs" <[email-masked]>

   In the case of a server response, RDMA benefits the client, not the
   server, so I fail to see why your example is problematic.  Zero
   copy send is not what this standard addresses.

With client memory bus bandwidth in the multi-gigabyte per second
range, who needs to avoid the single copy?  How much NFS and web
surfing does one need to do before this is would really come into
play?

And the bus speeds will just be faster by the time something like
this could be deployed widely.

For example, look at SACK, only within the past year are there a
decent number of systems out there implementing it.  Now how many
years ago did it enter RFC state?  And there are still stacks out
there even in their current development sources not implementing any
form of it.

Later,
David S. Miller
[email-masked]

---

From: "Justin T. Gibbs" <[email-masked]>
Date: Thu, 24 Feb 2000 17:09:51 -0700

In the case of a server response, RDMA benefits the client, not the
server, so I fail to see why your example is problematic.  Zero copy
send is not what this standard addresses.

--
Justin

---

From: "Justin T. Gibbs" <[email-masked]>
Date: Thu, 24 Feb 2000 17:50:52 -0700

>surfing does one need to do before this is would really come into

This is a very different argument than the one you "implied" before.
I can certainly say that I'd rather make use of my memory bandwidth,
regardless of how much I happen to have, in a more constructive manner
than copying data for no good reason.

RDMA is a general purpose feature.  Don't shoe horn it into just an
option to accelerate a few protocols.

>this could be deployed widely.

And the networks will be faster by then too and the demand for
pulling more rich content will increase, etc.

>there even in their current development sources not implementing any

The performance impact of RDMA is quite a bit larger than SACK, so
I don't know that your example is relevant.  All the big vendors
implement zero-copy in some shape or form and since RDMA is a scheme
to make zero-copy work in more cases, I'm sure it will be picked up
if the proposal is deemed sane.

--
Justin

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 24 Feb 2000 16:53:53 -0800 (PST)

> data is DMA'd directly from the filesystem page cache.

Even without the above special acceleration, section 2 in the draft
overestimates the number of copies involved in most OSes nowadays.

> send is not what this standard addresses.

Even without RDMA, people have done zero copy receive with TCP/IP before. 
What I see in RDMA is a way to provide message boundary in TCP so that apps
do not need to spend time on that (I call them lazy apps). I don't see right
away the other claims in the draft.  It seems to me that the authors should
re-evaluate their claims and provide support to them.  I don't see any
arguments in the draft.

If message boundary is the only benefit, I think the SCTP proposal is more
interesting.  Check draft-ietf-sigtran-sctp-06.txt.

							K. Poon.
							[email-masked]

---

From: "David S. Miller" <[email-masked]>
Date: Thu, 24 Feb 2000 17:01:34 -0800

Date: Thu, 24 Feb 2000 17:50:52 -0700
   From: "Justin T. Gibbs" <[email-masked]>

   The performance impact of RDMA is quite a bit larger than SACK,

It depends who you are.

For someone over a satellite link, I think SACK benefits them
much more than RDMA.

Later,
David S. Miller
[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 24 Feb 2000 17:34:14 -0800 (PST)

> if the proposal is deemed sane.

Can you elaborate on this?  Suppose TCP "blindly" does zero copy everything to
an app's buffer (for example, to a web browser's receive buffer) without
RDMA.  Then the browser app looks at the data and displays it.  What is the
difference RDMA makes in this case?  Yes, RDMA can separate different messages
in the buffer.  But this can also be done by the browser app, not by TCP.

I guess this is what I suggest the authors to add to the draft.  It is not
clear to me how RDMA can make a difference, especially in those cases the
authors claim a big performance difference.

							K. Poon.
							[email-masked]

---

From: Zachary Amsden <[email-masked]>
Date: Thu, 24 Feb 2000 17:56:26 -0800

> surfing does one need to do before this is would really come into

With network bandwidth approaching memory bus bandwidth, it becomes an issue.  
A single copy receive uses 3x the bus bandwidth of a zero copy implementation 
(or worse if not aligned properly), and also poisons many more cachelines.

-- 
Zachary Amsden  [email-masked]  (650) 933-6919  09U-510  Core Protocols

---

From: "David S. Miller" <[email-masked]>
Date: Thu, 24 Feb 2000 18:03:31 -0800

From: Zachary Amsden <[email-masked]>
   Date: Thu, 24 Feb 2000 17:56:26 -0800

   With network bandwidth approaching memory bus bandwidth, it becomes
   an issue.  A single copy receive uses 3x the bus bandwidth of a
   zero copy implementation (or worse if not aligned properly), and
   also poisons many more cachelines.

Before we discuss this point further, does anyone have real hard
evidence that cpu cycles on the client side are the issue for
the vast majority of systems out there?

In my experience cpu cycles are abundant on client machines.  Server
side is where getting precious cpu cycles back seems more important.
Client side cpu cycles are typically being expent on tasks such as
IDCT transforms to decode audio/visual streams, executing Java code,
but not copying the data from the network.

Next, have any other implementors investigated the cpu usage gains
obtainable from deferring TCP receive packet processing to user
context?  I have seen it make significant improvements, and whats more
this helps out all systems, this means old applications and
unsophisticated hardware.  (I am referring to Jakobson's idea of
nearly 10 years ago, and it appears once again he was right.)

Finally, what does RDMA do in the presence of SACK options?  Which
set of options gets kicked out?  Should the RDMA options go in
and we just drop the SACK blocks?  Or the other way around?

Later,
David S. Miller
[email-masked]

---

From: Charles Esson <[email-masked]>
Date: Fri, 25 Feb 2000 10:11:23 +1100

1) Given that the first paragraph says section 10 of rfc2026 is
irrelevant,
under what conditions is the document published?

---

From: Zachary Amsden <[email-masked]>
Date: Thu, 24 Feb 2000 19:32:40 -0800

> side is where getting precious cpu cycles back seems more important.

Agreed for the most part.  But reducing latency and cache misses are just as 
important on servers, and it would be interesting to see whether RDMA would 
make a difference there.  Of course for file/web serving, the server receive 
fast path can all be done in-kernel anyways, and output volume dwarfs input 
volume.

I can come up with a rigged scenario where RDMA should make a difference:

Group server or database which needs large amounts of external data over the 
network via NFS/your protocol of choice.  Internet search engines would 
possibly fall into this category.  In this case, input/output ratio is 
drastically different than a typical server, and memory bandwidth used by 
copying could become an issue.

However, this is a pretty limited use, and would only need to be deployed on a 
few machines.

I don't see RDMA being beneficial for typical web/file servers, or clients, 
other than getting a couple % extra performance.

---

From: "Justin T. Gibbs" <[email-masked]>
Date: Thu, 24 Feb 2000 22:13:07 -0700

>in the buffer.  But this can also be done by the browser app, not by TCP.

You seem to be saying that in the common case zero copy is achievable.
Most implementations I've seen require the network driver to make
a guess about where the payload will be in an incoming packet so the header
can be stripped off and the payload dmaed to an aligned area.   A page
flip is then performed to get the data where the user wants it,
imposing the restriction that your  payload be page sized so you don't
leave gaps in the user's destination buffer.  Certainly, with a more
intelligent network adapter that knows every protocol you can determine
exactly where the data is in each packet.  If you add connection tracking
and sequence number sniffing to the nic with a mechanism to register user
buffers to connections, you can get zero copy every time*.  Unfortunately
this is not very general purpose solution.  The point of RDMA seems to be
to allow nic manufacturers to add support for a single tcp option that, at
the very least, allows the nic to align the payload for you.  Add RID
registration with the nic and you get the payload exactly where you want it
too.  All without too much state information kept by the nic.

* This technique has been implemented with custom firmware on  Alteon
  Gig-E cards for a product I work on.
  single protocol
--
Justin

---

From: Alan Cox <[email-masked]>
Date: Fri, 25 Feb 2000 12:39:05 +0000 (GMT)

> leave gaps in the user's destination buffer.  Certainly, with a more

Perhaps its about time the world put together an official, sane, ring buffer
style mmap socket api. A lot of the requirement to align data is coming
from the existing socket API. 

>   Gig-E cards for a product I work on.

Nifty.

---

From: Charles Esson <[email-masked]>
Date: Fri, 25 Feb 2000 19:55:31 +1100

I must have missed something.

If we don't have this, you can take the destination port, convert to a
table address, use the sequence number,
do some calculations and come up with a buffer address and an offset. If
you want to mess up the layering
of your stack, they are all things you can do now.

or using RDMA

You tell the would the contents of your table, have the server play with
the data so detailed,
the server sends back info in this option.

--->An attacker plays with the data so returned.

The client instead of doing the port to table translation to determine
where to send the data
from the packet , does a port to table translation to find the table to
detect malicious corruption's
of the option.

How can you assume anything in the option is nothing but rubbish until
such
a translation and check is done?

1) Haven't you only changed the reason to do the port to table
translation.
2) Have completely messed up the layering of your stack. Ok people do it,
but as an option?
3) Provided a whole new farm yard of bugs and potential attacks?

As I read more and more RFC's one question I am constantly asking, why are
people making this terrific
protocol more and more complicated.

Isn't it difficult enough to implement already ( I think it is anyway).

> to decrease the number of copies it must do.

1) I think your a bit game if you assume the data in the option is
anything other than a story that aims to mislead.
2) If you want to play this game why can't you use the port, sequence
number and a table? All things available
to you without advertising the state of your system.

> packet drops and improving performance on parallel systems.

Out of order processing assumes the data that didn't arrive one day will.
Is that a valid
assumption on the internet? What happens with packets that are received
more than once,
do you DMA over the previous data thus corrupting the changes that have
been made
by the application? Do you make the rule the application may not alter the
DMA buffers?
Or do you check for this condition? Is this check in your acceleration
hardware?

I assume you believe you still only move the sequence number forward if
all data has
been received to that sequence number. Is this done in the acceleration
hardware?

By the time you have checked that the option is valid, updated your
received
sequence number, and made sure you are not DMAing  over data the
application already
has, what have you gained that could not be had without this option and a
restructure of a stack.

P.S. Maximum of two copies in the stack I wrote, one from card to
ip_buffer, and an
optional copy from ip_buffer to application_buffer. I don't believe I am
partially smart.
I deceive the introduction is misleading.

---

From: Zachary Amsden <[email-masked]>
Date: Fri, 25 Feb 2000 09:06:55 -0800

> of your stack, they are all things you can do now.

Yes, you can do that, and it isn't that tricky for simple protocols.  For any 
bulk data protocol higher than TCP with fixed headers, determining the payload 
offset is pretty straigtforward for a single transfer per connection.

Once you start with multiple transfers per connection, variable headers, or 
many different protocols, it becomes harder to do all this work in silicon.  
The advantage I see in RDMA is giving a generic payload pointer for the NIC to 
separate protocol data and payload.

I wonder whether it would be better to do this at the IP layer to enable it to 
be used for UDP and other protocols as well.

Of course RDMA is only going to help in cases where you have receive bandwidth 
issues, and such a scenario isn't likely to be the case for web/file servers 
or desktop clients.

The scenario that it gives the most benefit for is a "middleman" server that 
needs to do lots of I/O to a network storage device while servicing requests.  
In this case, any network I/O running locally may well be over a different 
protocol like UDP or ST.  (ST however, has no need for RDMA acceleration, as 
it already has a buffer transfer design).

---

From: Costa Sapuntzakis <[email-masked]>
Date: Fri, 25 Feb 2000 10:20:42 -0800 (PST)

Erik,

> use to try to reduce the overhead"?

Thanks for keeping me honest here. I have yet to code up an
implementation and generate numbers. The claims are, thus, perhaps
overstated.

> (unless you add hardware acceleration).

The copy that is being eliminated is from the TCP receive buffer to the
buffer cache.

As for the DNS entry problem, I'm a bit baffled since it works
from two external non-Cisco machines for me. I'll be forwarding this to
our internal tech guys. The domain name ftp-eng.cisco.com might
work better.

-Costa

---

From: Costa Sapuntzakis <[email-masked]>
Date: Fri, 25 Feb 2000 10:45:34 -0800 (PST)

Hi,

Your input on this is very much appreciated. I'd like to clarify a couple
things, though. Some of these things may not need clarification. :)

The TCP RDMA option is not about accelerating sending of data. It's
about speeding up the receiving of bulk data.

The TCP RDMA option isn't proposing to only about accelerating servers.
It's about accelerating the receiver of data. In the case of NFS READ
RPCs, that's the NFS client.

I agree that for specific problem domains, SCSI, NFS, HTTP, people may
want to build specialized server hardware.

Perhaps explaining why I decided to explore this space will help.
Today, you have specialized silicon that for simple bus protocols
(SCSI parallel interface and ATA) will directly take transfer blocks
between the device and the buffer cache. This is not currently done
with TCP, to the best of my knowledge. The best TCP implementations
do zero copy to a TCP receive buffer.

However, in the case of most storage protocols, you don't want
the data in the receive buffer. You want it in the buffer cache, so
there is a copy to the buffer cache.

So, NFS has a  CPU overhead hit as compared to optimized storage host bus
adapters. The goal was to eliminate part of this hit, by getting rid of an
extra copy.

Now, this proposal doesn't fix the interrupt overhead problem. 
Optimized FC/SCSI NICs have one interrupt/transfer or less.

-Costa

---

From: Costa Sapuntzakis <[email-masked]>
Date: Fri, 25 Feb 2000 10:50:32 -0800 (PST)

Hi, 

I don't think it's really a matter of SACK vs. RDMA. I think they are both
useful and somewhat orthogonal.

There is a concern that both RDMA and SACK are large options and there
is only 40 bytes of TCP options space.

Perhaps there is a way of expanding the TCP options space (yet another
TCP option!) that will enable both to co-exist peacefully.

-Costa

---

From: Alan Cox <[email-masked]>
Date: Fri, 25 Feb 2000 19:00:09 +0000 (GMT)

> be used for UDP and other protocols as well.

Congratulations. You are probably the first person in the world to find a sane
use for the IPv6 flow id 8)

Alan

---

From: Aman Singla <[email-masked]>
Date: Fri, 25 Feb 2000 11:52:53 -0800

> it already has a buffer transfer design).

Thats IMO an excellent point. If you're going to have to change your
client as well as the server, and add an additional formatted header
(through the TCP RDMA option), why do you want to constraint yourself
to doing it in the context of TCP only!
Why not do it using STP? a protocol that has been designed to do
RDMAs - addressing various issues which come up when designing RDMA
s/w, firmware, h/w..
The BDS fileserver using STP does something very similar to what you're
proposing to do with NFS. There's even a prototype doing SCSI using
RDMAs of the very same kind using STP.

thanks,

:a

---

From: Alan Cox <[email-masked]>
Date: Fri, 25 Feb 2000 20:04:25 +0000 (GMT)

> there is a copy to the buffer cache.

That assumes you require a copy to move from the network layer to the
buffer/page cache. Is that a warranted assumption. I think not.

> Optimized FC/SCSI NICs have one interrupt/transfer or less.

Interrupt overhead is old hat. Mitigation schemes are well understood

Alan

---

From: Michael Krause <[email-masked]>
Date: Fri, 25 Feb 2000 12:03:38 -0800

>this could be deployed widely.

It ain't free and there are plenty of reasons to avoid copying data since 
many applications do not always touch the data being moved.  Also, http and 
web traffic are probably not the best examples to illustrate the value of 
RDMA Read and Write operations.  Storage which was the context that started 
this effort certainly benefits from a combination of send for 
control  messages and RDMA for data movement - doesn't matter whether this 
in user-space / kernel-space, client or server.  Those buffers may be only 
used by the server to reflect a data set to a set of users without ever 
touching the buffers themselves.  Also, one could use this technology with 
storage devices to bypass the server and send data to one or more NICs for 
remote access - RDMA is still quite good for this type of operation and 
does not involve touching the data.

Mike

---

From: Jamshid Mahdavi <[email-masked]>
Date: 25 Feb 2000 14:19:11 -0800

Regarding RDMA and SACK, I don't know if there is exactly competition
for options space, since SACK primarily appears on acks, and my quick
reading of RDMA only appears on data.

(Of course, you'd still have to worry about bidirectional simultaneous
bulk data transfers, but I don't think anyone really does this.  Most
bidirectional apps are request/response, which would have little clash
between SACK and RDMA).

That being said, I'm inclined to agree with most of the other people
who question the necessity of RDMA, for most of the other reasons
which have already been stated.

--J

---

From: Zachary Amsden <[email-masked]>
Date: Fri, 25 Feb 2000 16:01:20 -0800

> reading of RDMA only appears on data.

Not only that, the presence of multiple SACK options means you have a serious 
traffic problem, and in that case I can't see RDMA helping your performance 
tremendously in that case.

-- 
Zachary Amsden  [email-masked]  3-6919  31-2-510  Core Protocols

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 25 Feb 2000 16:49:16 -0800 (PST)

> The copy that is being eliminated is from the TCP receive buffer to the

Most performing implementation (include the BSD 4.2 networking code from 
1980-something) just do a single copy from the kernel buffer to the
applications buffer in user space (assuming the NIC does DMA i.e. the device
driver doesn't copy from a buffer on the NIC to memory).
I that case you are not saving any copies. Thus the only potential performance
benefit is to more easily place the DMA'ed data in a "good" memory location.
But a NIC that parses protocol headers could accomplish the same thing without
requiring the sender to include any new TCP options.

> our internal tech guys. The domain name ftp-eng.cisco.com might

It must have been some transient with the http proxies around here.
Sorry for the false alarm.

  Erik

---

From: James Carlson <[email-masked]>
Date: 25 Feb 2000 20:49:02 -0500

> RPCs, that's the NFS client.

If you want to go fast on NFS client (and server!) operations, why not
use the default UDP mode of operation?  You just set your receive
offset to PAGE_SIZE-sizeof(READ3resok)-28-sizeof(layer_two), and
you're all set to page flip.  If you want to get fancy, you build this
offsetting magic into the device DMA.

As for TCP with arbitrary (unspecified) applications, it would make
more sense to create a new "map in TCP segment" system call than to
use RDMA.  Have this new call map in (read-only) raw system buffers to
the user's address space containing the received in-order data (with
packet headers and other nonsense), and copy out along a struct uio
array pointing to the actual data the application should be examining.
That gets you a zero data copy interface from driver to application
without having to modify any peers.

(I hacked a copy of 4.4BSD to do zero data copy on transmit as well
for an embedded system we're using here.  It was rather easy to do --
I just replaced m_copydata with a new "m_refcopy" function that
created read-only reference copies of the buffers for retransmit
purposes.  TCP ack handling and m_free/m_prepend needed minor tweaks
to get the reference counter logic right.)

> adapters. The goal was to eliminate part of this hit, by getting rid of an

That of course depends strongly on system architecture.  In any event,
RDMA doesn't appear to help.  You could just as easily create a nifty
Ethernet driver that copies only (say) the first 54 bytes of memory
into main memory and leaves the rest in the queue until it determines
the exact destination for the payload (or hits an exception case that
requires harder work).  This would eliminate the extra memory copy and
result in perfectly aligned receive data.  Such "fast path" code is
quite common in well-designed systems.

I hope describing the 54-bytes-first algorithm here stops anyone from
patenting the idea.  ;-}

-- 
James Carlson, System Architect                     <[email-masked]>
IronBridge Networks / 55 Hayden Avenue   71.246W   Vox:  +1 781 372 8132
Lexington MA  02421-7996 / USA           42.423N   Fax:  +1 781 372 8090
"PPP Design and Debugging" --- http://people.ne.mediaone.net/carlson/ppp

---

From: Alan Cox <[email-masked]>
Date: Sat, 26 Feb 2000 02:50:15 +0000 (GMT)

> without having to modify any peers.

I was thinking something along the lines of

	recvmsgiov(fd, iovec, etc...)

the difference being the kernel gets to fill in the iovec..

---

From: James Carlson <[email-masked]>
Date: Fri, 25 Feb 2000 22:21:01 -0500 (EST)

> the difference being the kernel gets to fill in the iovec..

Yep.  That's what I was thinking.  You just said it better.  ;-}

---

From: Lloyd Wood <[email-masked]>
Date: Sat, 26 Feb 2000 18:01:09 +0000 (GMT)

On 25 Feb 2000, Jamshid Mahdavi wrote:

> bulk data transfers, but I don't think anyone really does this.

Two words: delayed acks.

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: Lloyd Wood <[email-masked]>
Date: Sat, 26 Feb 2000 17:59:51 +0000 (GMT)

On Fri, 25 Feb 2000, Zachary Amsden wrote:

> traffic problem,

Not necessarily.

'Packet Reordering is not Pathological Network Behavior'
 Bennett, Partridge, Sheetman, IEEE/ACM Transactions on Networking,
 December 1999, (7):6, pp. 789-798

See also:
http://www.ir.bbn.com/%7Eshectman/metrics.html

multiple sack options do not necessarily equate to multiple losses.

L.

> and in that case I can't see RDMA helping your performance

---

From: Rick Jones <[email-masked]>
Date: Sat, 26 Feb 2000 11:54:15 -0800 (PST)

> That of course depends strongly on system architecture.  In any
> event, RDMA doesn't appear to help.  You could just as easily create
> a nifty Ethernet driver that copies only (say) the first 54 bytes of
> memory into main memory and leaves the rest in the queue until it
> determines the exact destination for the payload (or hits an
> exception case that requires harder work).  This would eliminate the
> extra memory copy and result in perfectly aligned receive data.
> Such "fast path" code is quite common in well-designed systems.
> 
> I hope describing the 54-bytes-first algorithm here stops anyone
> from patenting the idea.  ;-}

I cannot recall the specifics in terms of how many bytes it would
bring-in, but the "Slider" FDDI card for the HP 9000 735 and 755
workstations did just that - DMA-in some number of bytes and notify
the driver - enough to figure-out what was going-on, and then the
driver would program the remaining DMA. I think that the 735 and its
Slider FDDI card shipped about 1993ish.

rick jones
it's all just a question of datastructure management...

---

From: Zachary Amsden <[email-masked]>
Date: Sat, 26 Feb 2000 16:25:18 -0800

> the difference being the kernel gets to fill in the iovec..

And enabled by a socket option to indicate the app is willing to accept the 
kernel placement of headers and data.  Still needs hardware support to 
separate header and payload, though.

---

From: Bill Sommerfeld <[email-masked]>
Date: Sat, 26 Feb 2000 19:28:03 -0500

> Slider FDDI card shipped about 1993ish.

While we're engaged in a game of "who did it first..", I was under the
impression that apollo's token ring did the same sort of thing, about
a decade before that, albeit with an RDMA-like hack in the link layer.
The original apollo m68k pagesize was 1k..

As I understand it, packet headers were PIO'ed into one pool, and then
the packet data was DMA'ed into another pool of page-aligned,
page-flippable buffers.  One of the apollo old-timers like Paul Leach
would know for sure..

					- Bill

---

From: Alan Cox <[email-masked]>
Date: Sun, 27 Feb 2000 13:18:17 +0000 (GMT)

> separate header and payload, though.

It certainly makes it easier. Your average $30 tulip card can take a good
guess it this especially for predictible protocols like NFS over UDP.

However guessing wrongly is not fatal since for the case where you DMA then
map into the application you can map in some of the header bits without any
real security impact and point the iov further down the packet

---

From: James Carlson <[email-masked]>
Date: 27 Feb 2000 11:42:04 -0500

> separate header and payload, though.

Actually, no, it doesn't require any special hardware support.  Who
cares to separate the header and payload at all?  If it's not
specifically NFS that we're talking about (which has its own problems
due to demand-paging), then it's sufficient to map the entire received
packet into user space -- headers and all.  All that's really needed
at the application level is a list of data-start addresses and lengths
(as would be present in a regular uio vector).  The standard recvmsg()
call already provides an interface like this.  The extra bit of magic
is that for zero-copy the user doesn't get to put his own buffers into
msghdr.msg_iov; the kernel picks.  How about SO_RCVZCOPY to enable ...?

---

From: Zachary Amsden <[email-masked]>
Date: Sun, 27 Feb 2000 23:52:53 -0800

> msghdr.msg_iov; the kernel picks.  How about SO_RCVZCOPY to enable ...?

No, that situation doesn't require any hardware support.  However, a zero-copy 
receive path is not the only element of RDMA - RDMA was designed (I suppose 
from the discussion here) specifically to address header/payload issues for 
storage protocols.  Clearly one can do zero-copy receive with changes to the 
API and no hardware/firmware modifications.  But with no special hardware 
support, flipping the payload into some page with alignment constraints will 
require another copy.

There is one exception to my last statement that I know of:  If you pre-adjust 
the hardware receive buffers to make the payload align on a page boundary, you 
can flip the page into the buffer cache for (hopefully) the common case.  
However, this requires the ability to tune these header offsets and will only 
work for one protocol at a time (mostly).

Realistically, who is going to be running a storage system that requires so 
much bandwidth that avoiding receive copies is necessary, and runs on generic 
NICs with no firmware/ASIC modifications possible?  So I think using modified 
hardware is completely reasonable in those circumstances.

In any case, zero-copy receive with no payload separation, as you propose, 
would work fine with the API changes previously suggested, but that is a 
separate discussion :)