Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc.

From: Vernon Schryver <[email-masked]>
Date: Sun, 27 Feb 2000 17:44:12 -0700 (MST)

> From: "David R. Cheriton" <[email-masked]>

> ...
>   Clearly, data is being received from hardware and software does not
> get to touch it until it has been stored to some memory.  My 
> assumption is that the storage system memory is arranged in fixed
> size pages of disk/file pages.  Without hardware RDMA to the storage
> level, I believe one requires an extra copy, from whatever the
> hardware delivers to what the storage system expects.  Either
> you use twice the bandwidth in the storage system memory system or
> or else you have a separate memory system for the network, and
> have software/processor power adequate to copy between at wire
> speed (with all the associated support facilities for this processor.)
>  Unless there is something wrong with this reasoning,
> it seems like a cost issue of providing the above hardware resources
> vs. providing a NIC chip that can RDMA.  

Depending on how you are counting copies, that reasoning has been wrong
in commercial UNIX systems for more than 10 years.
Do you use the RDMA bits before IP checksum, the TCP checksum, and the
medium FCS or checksum have been checked?  If not, if you receive the
entire link layer frame into some kind of temporary buffer or FIFO,
probably in the "network interface card/controller," to check the trailing
FCS and before using the RDMA bits, then commercial UNIX systems have been
doing as you say to save copies since the late 1980's.  As I said before,
such systems were a part of what killed Protocol Engines Inc.

If you do use the RDMA bits in the TCP header after 50-60 bytes of
the frame have arrived, but before the frame FCS, aren't you worried
about bit rot in the RDMA?

> My guessitimate is that the software-only approach would be easily
> 10 times more expensive here at the higher speed rates, of 10 Gbps.
> If there is serious doubt about the merits of real hardware support,
> we should try to quantify costs further at these speed ranges, IMHO.

By "expensive," are you talking about dollars or bits/second?

Regardless, if you look at the number of CPU cycles or gates in custom
silicon required to support incoming page flipping in old, existing
implementations, I bet you'll find that they are less "expensive" than
any likely RDMA implementation.  Power of 2 modular arithmetic is awfully
cheap compared to parsing and validating TCP options.

> ...
> It would help me to have a more careful definition of the types of
> attacks you have in mind.  In an unsecure network with intruders,
> presumably I can end up with bad data in the right buffer
> or right data in the wrong buffer without using RDMA.
> Do you view we have made things worse, and if so, how?
> or are you objecting to us not making things better?

Is it possible for a bad guy to use RDMA to put bad data into memory
that is not a buffer?

If the RID does no more than choose from a safe list of buffers, then how
does RDMA usefully differ from the old FDDI, ATM, and HIPPI implementations
that put incoming page-flippable data in buffers that get into user space
with the data having been seen on the system bus the absolute minimum
number of times for any scheme, including RDMA, once?
Systems I've worked on have done mbuf allocation in the network interface
hardware, including putting page-flippable payloads into page-mbufs that
can eventually be flipped into user space.  And of course, take care of
the TCP or UDP checksum.

Given the recently described extensions to readv(), absolutely
all data received by a system like that would be page-flippable,
and without needing the silicon or CPU cycles to parse RDMA options
or requiring the sender to send RDMA options or even know that the
receiver is being fast.

Vernon Schryver    [email-masked]

---

From: Costa Sapuntzakis <[email-masked]>
Date: Mon, 28 Feb 2000 00:42:27 -0800 (PST)

Ok, so doing NFSv2/v3 header/data splitting is easy on an in-order
TCP stream because NFS has fixed-length trailers. Here's a little
technique:

1) Assume that all READ/WRITE transfers are powers of 2
2) Assume all RPCs larger than 4k are WRITE RPCs and all responses larger
than 4k are READ responses
3) Take the message size from the first 4 bytes of the RPC/TCP
encapsulation
4) Round the message size down to the nearest power of 2 (call this
quantity data_size)
5) The data is the last data_size bytes of the message. Put
the last data_size bytes in a separate aligned buffer.

Note, to do this with NFS/TCP, your NIC has to do some primitive
level of TCP processing (at least keep track of flows). It also
needs to understand RPC/TCP message boundaries.

Are there significantly simpler approaches than this? 

NFSv4 doesn't seem to have fixed length trailers and neither
does CIFS in all cases. And it looks like it will be costly to parse 
NFSv4 headers. 

RDMA still has the following features:

- Per-packet (Works with arbitrary out-of-order reception of TCP
segments)
- Fixed header that's generic across all protocols (NFSv4, v5, AFS,
DFS, CIFS, etc..) 
- No page flipping necessary on solicited transfers
- Message boundary bit (which is admittedly orthogonal to RDMA) allows
out-of-order processing on TCP receive buffer. Decreases parsing latency,
esp. in the face of packet drops.

The following measures should improve security/safety:

- NIC should ascertain that TCP segment is in receive window

- NIC needs to check that the RID is valid for a given TCP conn
  for safety/security reasons
  
  If the NIC does header/data splitting, it needs to keep track of
  per-flow information because most file block transfesr will span
  multiple TCP segments. So the NIC will probably have a notion of
  a TCP flow #.

  The brute force approach way to check RID validity is to use a CAM to
  map from (RID, TCP flow #) -> buffer address.

  If that's too expensive, then RIDs can be hashed and the flow # and
  buffer address stored in the bucket. The flow # is verified before
  using the buffer address.

-Costa