Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc.

From: Alan Cox <[email-masked]>
Date: Sun, 27 Feb 2000 16:09:39 +0000 (GMT)

> understand all the protocols that are amased over TCP. To do it in silicon
> it will probably make sense for a subset.

You dont want to do it in silicon. Forget doing all this in silicon. We have
this funky stuff called software. The silicon needs no RDMA support to do
sensible work in the API and the underlying OS are sensibly designed.

> You can look at it as a simple way to enable the protocl stack and the
> application to completely separate the protocol state machine (defined by
> headers and/or trailers) from the payload.

The two are tied together. You have to parse the TCP option stream to get
the ident in the first place. You can't act on the RDMAID until you
have checked the packet is syntactically valid and you've processed
the options including handling the SACK data mixed in with it.

It might also be fragmented of course.

> As for the vulnerability to attacks with a good size RDMAID and some
> imagination you can get the same level of protection as with the TCP
> sequence number (even a bit better because sequence numbers can be guessed
> from context).

The tcp sequence number protects against ordering errors not against DMAing
crap into the wrong buffer.

Different game, different cost if you lose.

Alan

---

From: "David R. Cheriton" <[email-masked]>
Date: Sun, 27 Feb 2000 14:17:24 -0800

>sensible work in the API and the underlying OS are sensibly designed.

I think it would be useful to get more quantitative.  I like to
think all of us in this discussion are well aware that some level
of performance can be achieved in software.  I'm not seeing where
"sensibly designed" leads us WRT to OSs, because, if they arent now
by whatever definition you are using, it seems pretty academic, as
they say.
  Clearly, data is being received from hardware and software does not
get to touch it until it has been stored to some memory.  My 
assumption is that the storage system memory is arranged in fixed
size pages of disk/file pages.  Without hardware RDMA to the storage
level, I believe one requires an extra copy, from whatever the
hardware delivers to what the storage system expects.  Either
you use twice the bandwidth in the storage system memory system or
or else you have a separate memory system for the network, and
have software/processor power adequate to copy between at wire
speed (with all the associated support facilities for this processor.)
 Unless there is something wrong with this reasoning,
it seems like a cost issue of providing the above hardware resources
vs. providing a NIC chip that can RDMA.  

My guessitimate is that the software-only approach would be easily
10 times more expensive here at the higher speed rates, of 10 Gbps.
If there is serious doubt about the merits of real hardware support,
we should try to quantify costs further at these speed ranges, IMHO.

>It might also be fragmented of course.
What you need to do >>in the common case<< before processing the RDMA
option is relatively simple and a very small portion of the
overall protocol state machines, so I presume your comment is
asking for more careful wording? Or do you really disagree with the
fundamental point?

>Different game, different cost if you lose.
It would help me to have a more careful definition of the types of
attacks you have in mind.  In an unsecure network with intruders,
presumably I can end up with bad data in the right buffer
or right data in the wrong buffer without using RDMA.
Do you view we have made things worse, and if so, how?
or are you objecting to us not making things better?

>Alan
David Cheriton

---

From: Alan Cox <[email-masked]>
Date: Sun, 27 Feb 2000 22:41:01 +0000 (GMT)

> vs. providing a NIC chip that can RDMA.  

or a NIC chip that has more memory on the chip and sends you the header then
you asynchronously give a DMA location for the rest of the buffer.

You've now got rid of RDMA, instead your little bit of extra silicon is 
generic, and will work with stuff like appletalk even. What does the
extra RAM and PCI glue cost you - probably not a lot.

That is why I prefer software solutions

> asking for more careful wording? Or do you really disagree with the

Yes

The sequence of operations you must execute is complex. 

You have to

Check the packet is long enough
Check the header is IPv4
Check the IPV4 IHL is valid
Check the protocol field
Check the source/dest addresses are legal
Work out if dest is for us or another node
Perform IP fragmentation (checks at least cannot be deferred defrag can)
Walk the IP options (I guess you could defer this)
Get the tcp header
Check the tcp header lengths are legal and fit the packet length
Check the tcp ports to figure out the socket
Parse the tcp options 
Check the RDMA identifier, arbitarily aligned of course
Check the RDMA identifier, port and addresses match
Perform sequence space checks

Then you can deliver the packet.

Alan

---

From: "David R. Cheriton" <[email-masked]>
Date: Sun, 27 Feb 2000 17:50:17 -0800

Alan,

Given you are concerned about Appletalk and PCI, perhaps we just
need to agree you are addressing a different performance level of
storage system.  I see no reason for hardware support at Appletalk
and PCI speeds either.  The RDMA option, even implemented in
a hardware NIC, would not preclude software processing of Appletalk
or anything else.

  I think it would be only fair for this group to expect you to
provide a more fully worked out design and evaluation of your
proposed NIC if this is really to be taken as a serious alternative
to an RDMA option.  I for one dont find it a competitive design
(it is also one that was previously considered.)

Regarding the complexity of TCP processing in hardware, people are 
doing this, so I regard this as a done deal.  (I'd hate
to be a high-speed NIC vendor that cant do this.)
The only issue is how to handle the next level of 
protocols that have high performance requirements, i.e. storage.
Also, NICs with hardware enhancements for protocol support,
such as the Intel Gigabit Ethernet NIC have been very successful
both in performance and market.  So, expect more there.

Finally, the RDMA option is targeted to allow use of standard
 Internet protocols for SANs in place of  specialized protocols
and networks such as  FibreChannel, especially for non-local 
access.
I.e. an extra option rather than an extra protocol stack and network.
I hope the discussion can recognize that broader issue in considering
alternatives and the general need for the RDMA option.
I.e. can SCSI over TCP really compete with FC without the RDMA option?
(In this sense, the subject line should have put SCSI first.)

DRC

---

From: Alan Cox <[email-masked]>
Date: Mon, 28 Feb 2000 12:52:11 +0000 (GMT)

> storage system.  I see no reason for hardware support at Appletalk

No we are addressing the same level of performance in a general purpose OS
the difference is my case doesn't need weird protocol hacks. When working
with a dedicated storage system RDMA becomes even less interesting because
you code the stack to the needs of the filer. You don't even need an MMU
on such kit

> (it is also one that was previously considered.)

I don't find RDMA a credible useful solution at the protocol level. It
doesn't offer any visible advantage, it complicates the stack futher thus
punishing the majority in the interest of the few.

I'm interested in why you think such a NIC wouldnt work, Providing you have
interrupt mitigation I see no reason for it to fail. I'd be interested
to know what the flaws in that technique were in your eyes.

> to be a high-speed NIC vendor that cant do this.)

But can you do it at $6 a part in volume ? Thats what the other 99.9% of the
people care about.

> protocols that have high performance requirements, i.e. storage.

ST already has this sort of stuff figured out.

> both in performance and market.  So, expect more there.

Intel provide no useful documentation so that is hard to evaluate.

> and networks such as  FibreChannel, especially for non-local 

So is ST and ST is proven technology. If you are going to break the protocol
to add hacks to it you might as well design the protocol properly based on the
past twenty years of learning where TCP is hard to get right. You need IP to
be compatible you don't need TCP.

If you want to solve the generic problem then you don't need RDMA anyway

Alan