Subject: Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc.

From: David Robinson <[email-masked]>
Date: Thu, 24 Feb 2000 17:42:05 -0800 (PST)

To efficently determine boundaries within a packet stream work
must be done somewhere.  In the RDMA proposal it is up to the
clients to do the work to make the server's job easier.  In traditional
intelligent NIC cards the server does the work by parsing the headers.

It seems that the design of RDMA is backwards as it relies on changes
to the many clients to enable efficiency on the server. A traditional
intelligent NIC card with a modest amount of hardware/firmware
can handle 99+% of requests from unmodified clients.  The existence
proof is checksumming NICs and NFS accelerator boards.

For an efficient IP storage device it will have to deal with legacy IP
client stacks (no RDMA) and a competitive IP storage vendor will
implement the smart NIC described above. Why is RDMA more compelling?

	-David

---

From: [email-masked]
Date: Fri, 25 Feb 2000 12:12:25 +0200

That is not completely accurate. You will need appreciably more silicon to
do what you suggest.   And you can do it only with information that "passes
through the protocol" .
The good thing about the  proposal is that it can TAG whatever the
application wants (and that can be several layers away from the protocol).
You can't "page-flip" to buffers that you are not aware of. And page
flipping wherever is applicable assumes  also page boundaries for buffers.

Julo

Julian Satran - IBM Research

Vernon Schryver <[email-masked]> on 25/02/2000 04:23:47

Please respond to Vernon Schryver <[email-masked]>

To:   [email-masked], [email-masked]
cc:    (bcc: Julian Satran/Haifa/IBM)
Subject:  Re: TCP RDMA option to accelerate NFS, CIFS, SCSI, etc.

> From: Erik Nordmark <[email-masked]>

> > A draft describing the TCP RDMA option can be found at:
> > ftp://ftpeng.cisco.com/pub/rdma/draft-csapuntz-tcprdma-00.txt
>
> There is no DNS entry for ftpeng.cisco.com so I can't access the
document.

ftpeng.cisco.com resolves for me to 198.92.30.33, and the URL works
ftpeng.cisco.com does not answer ICMP Echo-Requests.  It also seems that
Cisco is filtering ICMP TTL Exceeded.

Oh, well.  I predict that soon traceroute and ping will be as
effective as if the Internet were run by the old line telco managers
who went great lengths to keep their technical problems quite.
The recent security hassles will be a handy (and quite silly)
excuse.  (Yes, of course, Cisco has every right to filter however
they want.  I'm talking about technical sense, not rights.)

I'm even less impressed about the proposal than Erik Nordmark,
perhaps because more than 10 years ago I saw systems shipped by
more than one competitor of Sun Microsystems that paged flipped
NFS/UDP and user TCP data.  (well, one of the other vendors might
have been a little more recent 10 years.)

The motive for the proposal seems to be that while only a very few
CPU instructions are needed to page flip, the functions of those CPU
instructions are very hard in hardware.  I don't agree.  In today's
world of ASIC's, silicon to figure out where to drop incoming TCP
segments or NFS/UDP/IP fragments based only on old fashioned TCP and
RPC/XRD/UDP headers is nothing to write home about.  It wasn't even
all that big a deal more than 10 years ago, as everyone involved with
or who watched Protocol Engines Inc. remembers.

Hashing is almost as cool (and easy) in hardware as in software.

Vernon Schryver    [email-masked]

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 25 Feb 2000 09:35:30 -0700 (MST)

] From: Charles Esson <[email-masked]>

] I must have missed something.
]
] If we don't have this, you can take the destination port, convert to a
] table address, use the sequence number,
] do some calculations and come up with a buffer address and an offset. If
] you want to mess up the layering
] of your stack, they are all things you can do now.

Standards committees don't like hashing.  It looks complicated and
insufficiently deterministic on an overhead projector.

] or using RDMA
] ...

] --->An attacker plays with the data so returned.
] ...

That's a very good point.  The tagging in RDMA cannot be used until
after it has been validated by the receiver.  The validating consists
of looking at sequence numbers, RPC/XDR headers, etc. to figure
out where the data can and should go, and then checking that the
sender guessed right.  Why not skip the last part and ignore the
RDMA tag?  Then why send the RDMA tag?

 .......

> From: Pete Zaitcev <[email-masked]>

> Very well, but what about its companion document (SCOT)?
>  http://search.ietf.org/internet-drafts/draft-satran-scot-00.txt
> It is published, isn't it? It was somewhat disturbing to see the
> notice, but on the other hand it was honest. IBM could just as
> easily come up silently with a silly software patent for RDMA option
> or for SCSI over TCP idea as such.

The IETF's protections against patent games are well intended, but nothing
to worry about if you want to play them and nothing to rely upon if you
don't.  The history of IETF patent games demonstrates that the IETF is
powerless to limit them (or worse), and that they're harder to play than
the players hope.  (E.g. PPP CCP and PPP 48-bit FCS, respectively)

  ......

} From: "Justin T. Gibbs" <[email-masked]>

} ...
} >Can you elaborate on this?  Suppose TCP "blindly" does zero copy everything to
} >an app's buffer (for example, to a web browser's receive buffer) without
} >RDMA.  Then the browser app looks at the data and displays it.  What is the
} >difference RDMA makes in this case?  Yes, RDMA can separate different messages
} >in the buffer.  But this can also be done by the browser app, not by TCP.
}
} You seem to be saying that in the common case zero copy is achievable.
} Most implementations I've seen require the network driver to make
} a guess about where the payload will be in an incoming packet so the header
} can be stripped off and the payload dmaed to an aligned area.   A page
} flip is then performed to get the data where the user wants it,
} imposing the restriction that your  payload be page sized so you don't
} leave gaps in the user's destination buffer.

That is required only if you stick to the current API.  Obvious, 
minor changes in the direction of some operating systems that existed
before UNIX are sufficient to relax the page boundary requirement.
To use RDMA, you have to change the API.

}                                               Certainly, with a more
} intelligent network adapter that knows every protocol you can determine
} exactly where the data is in each packet.  If you add connection tracking
} and sequence number sniffing to the nic with a mechanism to register user
} buffers to connections, you can get zero copy every time*.  Unfortunately
} this is not very general purpose solution.

Only standards committees and some academics care about "every protocol"
or optimizing absolutely every application.  The rest of us (including
academics) only care about optimizing the important stuff.

Also as you say, looking at sequence numbers in the interface and relaxing
the sockets API rules about not touching any bytes in the buffer except
those that are actually received lets you avoid copies all of the time.
I don't see why that is not a general purpose solution, if you want one.

}                                             The point of RDMA seems to be
} to allow nic manufacturers to add support for a single tcp option that, at
} the very least, allows the nic to align the payload for you.  Add RID
} registration with the nic and you get the payload exactly where you want it
} too.  All without too much state information kept by the nic.

I've been hearing since the mid-1980's proposals to do TSP lookups in the
network interface instead of software because it is so incredibly difficult
to find the right TSP quickly in software.  I think those ideas are similar
to the RDMA idea.  They assume facts not in evidence, that there is a
problem that needs to be solved, and that the solution is not worse than
the nominal problem.  There are reasons why such proposals appear in
standards committees before implementations.

 ......

] From: Lloyd Wood <[email-masked]>

] Note the mentions of SCSI and SCSI/TCP and the tie-in with the
] proposed IP Storage efforts (recent ietf general list discussion).
]
] I'd still like to know _why_.

] ...
] SCSI DMA over TCP? What _is_ all this aiming for - trying to build
] distributed RAID arrays with really poor performance that are subject
] to WAN outages and DoS attacks?

Why put SCSI over an protocol that measures RTT's, worries about
congestion in routers, and that expects the error rates that come
with 5000 miles of wire and 20 routers in the path?  Does anyone
really think that TCP/IP or even IP with it's 64K bit packet limit
are remotely close to the right protocol, particularly given the
existing and commercially available alternatives?

A standards committee is the venue of first and last resort for
such ideas, especially a committee that is related to currently
trendy things like the SuperInfoHypeWay.

 ....

) From: [email-masked]

) That is not completely accurate. You will need appreciably more silicon to
) do what you suggest.   And you can do it only with information that "passes
) through the protocol" .

Significantly silicon more than what to do what?  Since the comment
was addressed to me, I'll assume one 'what' was looking at sequence
numbers, port numbers, and so forth to page flip.  Clearly it takes more
silicon to support page flipping in hardware than to not support page
flipping in hardware.  I will not agree that the required silicon is a
big deal, not because I have a clue about floor plans and so forth (I
don't), but because at a previous employeer I fought to keep the hardware
guys from throwing in gates to do it.  They had the silicon to spare and
had heard so much about the wonderfulness of page flipping that they wanted
to get in on the fun.

Doing things in hardware is ok only if you absolutely must.  Software is
always better when it is good enough, because it is soft.

) The good thing about the  proposal is that it can TAG whatever the
) application wants (and that can be several layers away from the protocol).
) You can't "page-flip" to buffers that you are not aware of. And page
) flipping wherever is applicable assumes  also page boundaries for buffers.

That's important only if you stick close to the sockets or UNIX read()
API.  If you are not ultra-conservative, and if you know a little of the
history of file and device I/O API's, or of you think about such things
for 10 seconds, then RDMA tagging becomes less interesting.
To use RDMA tagging, you must abandone the UNIX read() API.  If you change
the API, then you may as well think about the whole problem instead of
only a corner.  If you let the operating system tell the application where
the incoming data arrived, then you don't need elaborate hints from the
sender to the receivers hardware to say where the receiving software will
want the data.

) Vernon Schryver <[email-masked]> on 25/02/2000 04:23:47
)
) Please respond to Vernon Schryver <[email-masked]>

I did not write that!

 .....

) From: Alan Cox <[email-masked]>
)
) > flip is then performed to get the data where the user wants it,
) > imposing the restriction that your  payload be page sized so you don't
) > leave gaps in the user's destination buffer.  Certainly, with a more
)
) Perhaps its about time the world put together an official, sane, ring buffer
) style mmap socket api. A lot of the requirement to align data is coming
) from the existing socket API. 

The IETF should not get involved in API's.  There are plenty of other
standards committees in that arena, as well as big commercial outfits
including one in the U.S. Pacific Northwest.  In other words, do you think
the IETF would be more successful arguing with Microsoft about winsock
than the IETF has been in dealing with Microsoft's obviously completely
stupid and wrong PPP ideas?

If you do get involved in standardizing such things, then *PLEASE* don't
limit yourself to #$%$#@! ring buffers!  The ancient Execelan and preceding
(I've a mental block against the name starting with 'I') ring buffer notion
was ok as an initial hack, but WRONG for something to go fast.  To start,
you don't need pointers or indeces that must be written by both the
interface and the host.

---

From: "Howard Berkey" <[email-masked]>
Date: Fri, 25 Feb 2000 11:13:08 PST

It is unclear to me what real-world benefit this REALLY offers over 
existing methods.  If your stack is already jumping through the hoops 
necessary to (a) support zero-copy on recieve and (b) do it in a 
robust, secure manner, then RDMA seems to me to be of dubious value and 
an invitation to attack.

I must be missing something.

Howard

---

From: Zachary Amsden <[email-masked]>
Date: Fri, 25 Feb 2000 11:52:34 -0800

> 
> IP options do not interact well with today's routers. They take
> the packet off of the fast path and dump it into software processing mode.
> Otherwise, the IP options would be right place to do this.
> 
> -Costa

Yes, but are the types of uses we are talking about realistically going to be 
run over routed networks?  It's basically a mechanism for accelerating 
high-bandwidth storage access, which is likely to be a switched network.

Of course, IP options don't play well with the fast path in the TCP/IP stack 
either, but that can be changed much more easily than the router fast path (I 
assume it is in a hardware CAM or something similar).

If the goal is just high-bandwidth storage I/O, there are much lighter ways of 
doing this than using TCP.

-- 
Zachary Amsden  [email-masked]  3-6919  31-2-510  Core Protocols

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 25 Feb 2000 15:50:49 -0700 (MST)

> From: Costa Sapuntzakis <[email-masked]>

> ...
> Today, you have specialized silicon that for simple bus protocols
> (SCSI parallel interface and ATA) will directly take transfer blocks
> between the device and the buffer cache. This is not currently done
> with TCP, to the best of my knowledge. ...

It might be good to investigate the history of Protocol Engines Inc.,
including its goals, the reasons for its failure as a business, and what
it achieved technically.  A skewed history might be:
 1. founded to make silicon for XTP, a nominally faster protocol than TCP.
 2. when XTP protocol and the XTP chips got bogged down, shifted to making
    chips to help TCP go wire speed over FDDI.
 3. other people made TCP go wire speed over FDDI without any special
    silicon or new to protocols.  That took some wind out of XTP's sails,
    and tore the sails driving PEI's TCP acclerator chips.
  4. standard standards committee problems with XTP didn't help PEI's other
    sails.

If you ask me, SCSI/IP and RDMA have striking parallels to #1 and #2. 
I bet you'll meet parallels to #3 before any real deployment.  You've
started to see #4 in some of the suggested improvements to RDMA today.
It's not that the suggestions are not good ideas.  That problem is that
committees cannot say no to good ideas, while the one thing that matters
above all in any design task is saying no to almost everything.

Protocol Engines and XTP were based on the unexamined assumption that TCP
is very difficult to implement and an unavoidably slow protocol.  Most
people just knew those "facts" 15 years ago.  I think RDMA suffers a
similar problem.  Instead of starting by assuming that a new protocol is
needed for a new goal, if you actually look within the existing boundaries,
you'll often find a solution.  Often the inside solution is better than
any possible extension of the protocol.  Protocol extensions require more
bandwidth and more processing on both sender and receiver.  They also have
problems gaining enough marketshare to survive.

Please don't misunderstand me.  Greg didn't include my name among
the authors on one of the XTP specs because I said XTP was a stupid
idea.  I still like lots of XTP.  I also think that many of the
XTP ideas can be *and have been* applied to TCP implementations.

> However, in the case of most storage protocols, you don't want
> the data in the receive buffer. You want it in the buffer cache, so
> there is a copy to the buffer cache.

Which NFS implementation written in the last 10 or at least 5 years and
intended to be fast doesn't move data between the buffer cache near the
disk and the buffer cache near the application with zero (0) copies?
Page flipping to and from buffer caches is especially easy, because
buffer caches tend to be page aligned, and file systems like to move
data in page-sized or larger chunks.

> So, NFS has a  CPU overhead hit as compared to optimized storage host bus
> adapters. The goal was to eliminate part of this hit, by getting rid of an
> extra copy.

How can you have fewer than zero copies?

> Now, this proposal doesn't fix the interrupt overhead problem. 
> Optimized FC/SCSI NICs have one interrupt/transfer or less.

Interrupts are killers, and so for that last 5 or 10 years, a competetive
NFS system has had about 0.1 interrupts per packet.  The trick is not
reducing the ratio of interrupts/packet, but reducing it only so far that
things don't slow down, and increasing the ratio when the total system
(client & server) moves into a regime that requires more interrupts.

] From: Michael Krause <[email-masked]>

] It ain't free and there are plenty of reasons to avoid copying data since 
] ...
] touching the buffers themselves.  Also, one could use this technology with 
] storage devices to bypass the server and send data to one or more NICs for 
] remote access - RDMA is still quite good for this type of operation and 
] does not involve touching the data.

There are other, much easier ways to separate data and control
information in the receiver than being forced to parse optional
new bits in TCP or IP headers.

For 10 years, network interfaces in commercial UNIX systems have been
putting the headers (including RPC/XDR) of incoming NFS traffic in one
place (a "small mbuf") and the data in another place (the buffer cache)
without extra copies, and without parsing any headers, not to mention
new header bits with the nasty problems of TCP or IP options.
And this despite the fact that the RPC/XDR stuff is between variable length
(recall the NFS group list) and a hard to predict length.

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 26 Feb 2000 12:02:31 -0700 (MST)

> From: James Carlson <[email-masked]>

> ...
> > The TCP RDMA option isn't proposing to only about accelerating servers.
> > It's about accelerating the receiver of data. In the case of NFS READ
> > RPCs, that's the NFS client.
>
> If you want to go fast on NFS client (and server!) operations, why not
> use the default UDP mode of operation?  You just set your receive
> offset to PAGE_SIZE-sizeof(READ3resok)-28-sizeof(layer_two), and
> you're all set to page flip.  If you want to get fancy, you build this
> offsetting magic into the device DMA.

There's a simpler and much better way, motivated by the observation that
trailers are rare in IETF application and transport protocols.  Put the
first packetsize(mod pagesize) bytes in one queue or buffer and the
remaining multiple of pagesize bytes in another queue or buffer.  Notice
that this trick automatically covers variable sized TCP/IP headers due to
options.  Because the typical NFS/UDP/IP header contains 4(mod 8) bytes
and IP fragments must be 0(mod 8), another more sublte kludge of a trick
is needed if your MTU is less than 8K+about 200 and your NFS readsize is
a multiple of page sizes larger than 4K.

> ...
> That of course depends strongly on system architecture.  In any event,
> RDMA doesn't appear to help.  You could just as easily create a nifty
> Ethernet driver that copies only (say) the first 54 bytes of memory
> into main memory and leaves the rest in the queue until it determines
> the exact destination for the payload (or hits an exception case that
> requires harder work). ...

> I hope describing the 54-bytes-first algorithm here stops anyone from
> patenting the idea.  ;-}

That's been shipped commercially for many years, as has the modulo pagesize
trick.  Given the number of customers that bought proprietary UNIX source,
there should be no problem proving prior art.  Commercial FDDI chipsets
such as Motorola's CAMEL have supported the 54-btes-first idea for more
than 5 years.  The CAMEL can be told to set asside the the first 64 bytes.
That covers the usual 3-bytes of padding that does not appear on the wire,
the 13 byte MAC header, the 8-byte LLC header, and 40 bytes of TCP/IP
headers without any options.  The CAMEL can even put the pair of streams
of headers and data into different ring buffers.  Unfortunately, we all
know that the TCP/IP headers no longer have a fixed length of 40.
That limits the utility of the idea.

The problem with page flipping for the last 6 or 7 years has not been
teaching your DMA hardware to do it with the information already in the
bit streams, but with increasing page sizes, and then only with input.
16 KByte pages became too small for fast computers years ago, but there
not many media that have 64 KByte frames, and then there's the IPv4 packet
size problem and the TCP/IPv4 maximum segment size problem.
That's why the letting the operating system pick the user-space address
of input buffers such as was done in systems 30 years ago or using your
IOVEC idea is necessary.

It's always been trivial (modulo paging system overhead, especially
multi-processor locking and page table entry caching) to page flip on
output.  The hassles have always been on input.  And yes, those paging
system overheads can amount to many microseconds of latency and even
computation, and that amounts to a big deal if you care about speed.

I think all of that is obvious to anyone who has ever looked seriously
about making NFS or TCP go fast.  10 years ago it and related ideas were
considered big deal trade secrets, but that was 10 years ago.  It is also
obviously more effective, cheaper, easier, compatible, and interoperable
than RDMA.  It is why I think the RDMA proposal is like most such that
I've seen over the years, including the Motorola CAMEL header separating,
based on having a cool idea for a solution instead actually looking at
the problems.

---

From: Vernon Schryver <[email-masked]>
Date: Sat, 26 Feb 2000 19:34:35 -0700 (MST)

> From: Bill Sommerfeld <[email-masked]>

> ...
> While we're engaged in a game of "who did it first..", I was under the
> impression that apollo's token ring did the same sort of thing, about
> a decade before that, albeit with an RDMA-like hack in the link layer.
> The original apollo m68k pagesize was 1k..
>
> As I understand it, packet headers were PIO'ed into one pool, and then
> the packet data was DMA'ed into another pool of page-aligned,
> page-flippable buffers.  One of the apollo old-timers like Paul Leach
> would know for sure..

If you allow changes in the on-the-wire bits, then don't forget the
4.2 BSD (or earlier) "trailers", which put the headers in a little
mbuf, and the data into a string of enormous, 512 byte page mbuf.

I don't think the point has been "who did it first," but "let's not
start improving it before considering the state of the art."