Subject: Re: Framing Discussion

From: Michael Krause <[email-masked]>
Date: Fri, 22 Dec 2000 07:30:20 -0800

At 05:34 PM 12/20/2000 -0800, Mohan Parthasarathy wrote:
>If every TCP segment has a iscsi header, which is a waste, then this
>problem is relatively simple in identifying which iscsi command
>a given TCP segment belongs to and also indexing into the right
>offset inside the buffer.

Why would such a solution be a waste?  As a percentage of bandwidth, this 
falls very much into the noise category even with 1500 Byte segments.  Most 
high-speed protocols including new ones such as InfiniBand use a small 
header per packet to allow hardware a simple mechanism to understand where 
to place the data (also applies to software implementations which can do 
better with minor hardware assists).

At the first BOF, I spoke about aligning the protocol with InfiniBand since 
that will eventually become the server point of attach in the coming 
years.  The suggestion was made to include the same RDMA semantics (if they 
are supported) as InfiniBand.  It was further suggested there an in other 
e-mail that a simple 8-byte header with a 4-byte CRC be associated with 
each segment and that these fields be contained within the data payload so 
that TCP is not impacted.  The contents of this header would contain a 
8-bit op-code, VA, length, etc. allowing the responder to target the memory 
on the host / controller if all fields were valid.  If there was an 
out-of-order delivery, the data could be spilled to temporary memory either 
in the host or the adapter and upon recovery, delivered to the correct 
target buffer without requiring host processor intervention with a little 
creativity.  This 12-bytes of overhead for SCSI operations would have 
minimal impact on link utilization and overall solution efficiency.  I 
believe these same concepts have been stated in the various RDMA proposals 
that have been distributed and given the eventual movement to InfiniBand 
for servers and the new SRP (SCSI RDMA Protocol), one might want to create 
an iSCSI solution that can easily bridge into these other technologies.

Note: The arguments about adapter complexity, impact to OS, etc. are rather 
moot in many ways.  The work will be done to support InfiniBand over the 
next couple of years and thus the cost to implement / support is going to 
be fairly minimal.  It should also be noted that many of these changes have 
already be done using PCI / PCI-X based solutions that support VIA, 
Scheduled Transfer, Oracle, MPI, Sockets Direct, etc. so the ability to 
deploy solutions in the highly desirable I/O interconnect independent way 
is available today as well.  One does need to wait or rely upon InfiniBand 
to make all of this happen.  It should also be noted that many companies 
will be working to have Linux support for this type of technology in the 
upcoming year so solutions should be available to all by the time iSCSI 
ramps to volume in 2002.

Mike

---

From: Michael Krause <[email-masked]>
Date: Fri, 22 Dec 2000 07:42:02 -0800

At 04:58 PM 12/20/2000 -0800, Douglas Otis wrote:

>problem of the block size being less than the page size.  Unless the SCSI
>application is forced to allocate in pages and you have a means to force the
>alignment of these blocks as they are delivered by the network, then this
>MMU technique is not available.

Many file systems allocate in nice multiple 4KB quantities making much of 
this fairly straight to implement and for partials these are either placed 
in a multiple 4KB buffer or these tend to align on power of 2 quantities 
buffers so the performance impacts are mitigated.

>You are suggesting that you use a standard TCP, find the PDU, process the
>iSCSI headers, place the data according to the tag within the iSCSI header,
>and because this process is located on the network adapter, it does not
>impact TCP.  I think I understand the logic.  As you expect the PDU to be
>"often" segment aligned, framing is not an issue.

Correct.

> > A zero-copy TCP is a content directed placement of the data.  Content
> > directed placement implicitly implies header / data split including the
> > iSCSI protocol headers.  Don't see what the problem is.
>
>Not at all.  The content of the payload does not impact the placement with
>Zero Copy TCP.  Here you expand the definition of this interface to now
>include iSCSI.  Rather than using TCP as the boundary between the adapter
>and the system, iSCSI is now such a boundary.

For a completely off-loaded solution, this is correct.  It is possible to 
only off-load the TCP portion and provide a scatter operation where the 
iSCSI header is targeted to a different control set of buffers and the data 
goes the appropriate data buffers - this is done in many of today's 
implementations so the logic is well-understood.

>Fine, but you really should not call this part of TCP.  It is not.  You
>should call it an iSCSI adapter interface.  If you wish such an interface to
>be application specific, so will its interface.  Clearly SCSI dictates a
>different interface to that of TCP.  If you advocate the placement of the
>application on the adapter, then you are no longer discussing TCP.  You are
>discussing the back-end of a full or partial processing of iSCSI.  TCP is
>hidden within this application running on the adapter.  I had advocated
>using a pointer to an application routine to serve this function of placing
>the application on the adapter and should the value be less than 1024, it
>would indicate a pre-defined IANA designator for such an embedded
>application.  Perhaps the value of 1 could indicate a normal IP stack as the
>application as you bind the application to the port.  If this is the desire
>of the work group, as it seems to be, then declaring details of the
>application adapter interface is the next step.  What group makes adapters?

InfiniBand is looking to standardize the interface to InfiniBand-based TOE 
(TCP Off-load Engine) endnodes so that  a well-defined wire protocol can be 
created for all IHVs to implement above the InfiniBand transport 
protocol.  I think there might be some benefit to do the same thing for 
iSCSI as for TOE in an appropriate forum while allowing this workgroup to 
proceed with the iSCSI definition which defines the operational model and 
wire protocol being used.  In fact, if the iSCSI workgroup sets up some 
basic rules as outlined in another response from me today or along the 
lines of the various RDMA proposals, then the wire protocol defined by the 
iSCSI workgroup in essence defines much of this needed interface.  Need to 
think about what else if anything might be needed.

Mike

---

From: Mohan Parthasarathy <[email-masked]>
Date: Fri, 22 Dec 2000 10:18:51 -0800 (PST)

> better with minor hardware assists).
It is a waste, if there is a better solution to tackle the same thing.
If this can make things simpler, we should consider this as an option.
But how does the sender insert this header ? Typically the target
side would just dump all the data to TCP - which contains a iscsi
header followed by data. TCP then would segment the data depending
on the MSS. How do you insert such an header for each MSS sized
data ?

The only RDMA proposal on Infinaband i have seen is the one from
Microsoft. I am not sure which one you are referring to.

> 8-bit op-code, VA, length, etc. allowing the responder to target the memory 

If VA is the virtual address, then this has security problems. How do
you prevent arbitrary packets over-writing memory ? Actually all
you need is a tag to identify the buffer pool, offset within a
stream, and some smarts in the h/w to associate the tag to a
buffer pool. Is this not sufficient ?

> upcoming year so solutions should be available to all by the time iSCSI 
I am not sure which proposal you are talking about. Infiniband working
group is looking at the proposal from microsoft for RDMA. I think it
is more specific to Infiniband.

-mohan

---

From: Michael Krause <[email-masked]>
Date: Fri, 22 Dec 2000 13:17:23 -0800

>on the MSS. How do you insert such an header for each MSS sized

This is a simple gather operation.  Most OS understand how to gather 
different segments into a single byte stream and also account for 
additional headers.  Nothing special here.

>Microsoft. I am not sure which one you are referring to.

RDMA is defined within the InfiniBand architecture (within the workgroup I 
co-chair) - there is nothing for Microsoft to do here.  There have been a 
couple of drafts submitted to IETF on how to support RDMA over TCP which is 
what I'm referring to in this e-mail exchange.  The objective of a TCP RDMA 
solution should be to align as much of the semantics as possible so that 
bridging is simplified and products can easily inter-operate given the 
volume potential of iSCSI solutions relative to potential InfiniBand 
storage relative to InfiniBand servers in the long-run.

>buffer pool. Is this not sufficient ?

A VA is an address but what that address is a local issue and is only used 
as a key to find the real address.  As such, I don't see any security issue 
beyond that associated with any datea exchange on a network and the 
techniques even used today within FC products can be applied so the 
technology impacts are actually well understood and can be implemented by 
multiple vendors.

>is more specific to Infiniband.

I believe you are referring to the InfiniBand application workgroup which 
has been looking at how to bootstrap InfiniBand via SRP (SCSI RDMA 
Protocol) which is one possible mechanism to support storage over an 
InfiniBand fabric - there are several mechanisms being defined for 
boot.  This is a T10 not a Microsoft effort.  InfiniBand is simply trying 
to accelerate some of this work but T10 will own and drive it to 
completion. There is an upcoming T10 meeting Houston in January focused on 
SRP that people might want to attend concerning InfiniBand.

Mike

---

From: Mohan Parthasarathy <[email-masked]>
Date: Fri, 22 Dec 2000 17:09:10 -0800 (PST)

> additional headers.  Nothing special here.
Sure it is possible but generally not the optimized path in the stack.
But there are other problems with this proposal i guess. As TCP
is stream oriented it need not obey boundaries of the various
segments. The effect is you won't see a iscsi header on every
datagram.
> technology impacts are actually well understood and can be implemented by 

Normal packets - i.e not using RDMA, can't write anywhere. But i can see
how that can be done with RDMA i.e restict the region accessed by
the NIC for a given RDMA transaction.  Does the FC products do this ?

> InfiniBand fabric - there are several mechanisms being defined for 

This is Winsock direct proposal. It came to the IBTA working group
i think. I have the document with me. This proposes to bypass the
normal tcp/ip stack and use RDMA to achieve high throughput.

-mohan

---

From: Douglas Otis <[email-masked]>
Date: Wed, 27 Dec 2000 13:39:47 -0800

Michael,

> buffers so the performance impacts are mitigated.

Here we have a sticky problem of alignment.  You envision an intelligent
adapter forcing this alignment on an application specific basis.  This
intelligent adapter will recognize content of the TCP stream, remove
prefixed iSCSI headers, and pad-out non-page-sized data transfers.  The
intelligent adapter will use content of this TCP stream to direct iSCSI
headers into different buffers than SCSI data and keep a relationship
between header and data.  One means would be to extract the tag within the
header and use the pre-allocated buffer to place data.  In David Black's
view, this interface would be the same as a SCSI adapter.  In that manner,
the header-data association is automatic.  Your desire is to allow a simple
merge of InfiniBand hardware with IP technology.  Presently InfiniBand does
not automate IP nor SCSI, so you are trying to solving more than an SCSI-TCP
specific problem.

Julian has presented his view of adding a suffix to the IP header as a means
of framing and adding InfiniBand-IP differential information.  Those fearful
of this approach could use standard TCP and suffer additional buffer
requirements or retransmissions upon a segment loss.  Will there be enough
performance loss to justify such a transition and will InfiniBand be
motivation for this transition?  InfiniBand is agnostic about the protocol
and has not breached the area of IP or SCSI with perhaps the exception of
using IPv6 size addresses.  With such short-comings of TCP, why use a
modified TCP to instantiate a new SCSI interface?  Solutions being hardware
based, especially for those within InfiniBand, what benefit is derived using
this modified TCP?  You will be forced to handle two such protocols.  Normal
and the enhanced for wide screen.

In Julian's words:

  "Many of the new applications could benefit from using functions
   available from a new transport (like SCTP) but no vendor is going
   to "bet the business" on an all new and radically different
   transport protocol - unless there is a transition plan that
   allows him to start by using a mature transport protocol and then
   migrate it to a new protocol."

This transition plan is going from TCP to TAF-TCP, where the application
interacts at the datagram level to align retransmit and where additional
headers are added ahead of the transport headers.  To review iSCSI, it is
SCSI on iSCSI on TCP on TAF on IP.  How is it good for either IP or
InfiniBand to add additional application specific layers?  To deploy,
network equipment is impacted by this transition so you are not just
stepping on your own TOEs.  It would appear that even Julian agrees that
SCTP is a solution, but he can not see the transition.  RFC 2960 does
provide the framing for RDMA, VI, SCSI, IPC, gaming, multi-media, audio, and
every feature being sought with this TAF extension.  Which provides the
superior solution and allows the cleanest transition?

See:
http://www.haifa.il.ibm.com/satran/ips/draft-satran-transport-adaptation-fra
mework-00.txt

Doug