Subject: RE: Framing Discussion

From: "Douglas Otis" <[email-masked]>
Date: Thu, 21 Dec 2000 16:24:37 -0800

Julian,

Although the IPS workgroup declares it is using TCP as the transport layer,
there is a strong desire to view this transport as framed rather than as a
byte stream.  Clever schemes to mark the end of a frame or the beginning of
a PDU (some hope at the segment boundary) are still unresolved.  At least
this is openly the case.  I have given up asking how such a scheme of
placing SCSI data into SCSI buffers delivered in out-of-sequence TCP
segments can be done without modifying TCP.  David has clearly declared such
a question out-of-scope.  I will blink and say that there is an application
running iSCSI on an adapter and that the interface for this adapter is akin
to some existing SCSI adapter product.  At least this limits much of the
standard's damage to the adapters.  I still hold only a dim hope with
respect to damage as I have yet to see clarification on the framing scheme.
If you wish to shed additional clarity on this subject, your comments are
always welcome.  Clearly we see this subject from different perspectives but
I would not say I see the truth.

As SCSI is just one of three applications to be supported by this new
framing scheme, then I guess we will see three additional interfaces exposed
at these adapters.  One for network use, one for SCSI, one for VI and yet
another for IPC.  I expect that in the time it will take these products to
mature to the point of not causing disruption with various routing
equipment, SCTP will have supplanted TCP in these applications.  The need
already expressed within this WG are reasons SCTP will be available before
Bill has his version ready.

Doug

> JP,
>
> I think that many people on this list have already discussed the so called
> out of order processing but Doug Otis seems adamant that he is the holder
> of THE truth.
> The reason why framing recovery is deemed necessary is to keep to
> a minimum
> the reassembly memory
> on the NIC cards.  Data can be placed in host memory without being
> delivered (i.e., the ULP made aware that data is there).
>
> Julo
>
> Douglas Otis <[email-masked]> on 20/12/2000 20:55:56
>
> Please respond to Douglas Otis <[email-masked]>
>
> To:   JP Raghavendra Rao <[email-masked]>, [email-masked]
> cc:
> Subject:  RE: Framing Discussion
>
>
>
>
> JP,
>
> The VM page flipping will require blocks to be greater than or equal to
> pages in size and that these blocks are aligned at page boundaries.
> Neither
> of these assumptions are true. The goal is clear, the NIC will examine the
> content and then direct data payload to the system through the NIC
> interface.  The desire is to keep the buffers on the NIC small and thus
> allow out of sequence processing of the TCP stream.  This must be
> seen as a
> modification to the normal TCP implementation.  Such operation should
> include a complete description of the API to allow consideration of NIC
> design, inter-operability and security requirements.
>
> Doug
>
>
> > >> The design goal behind the framing discussion
> > >> is avoidance of that copy.  In contrast to the
> > >> typical case for NICs and TCP/IP, read data
> > >> buffers for SCSI data are usually *not*
> > >> interchangeable.
> > >>
> > >Why are they not interchangeable ? This is an
> > >assumption not stated anywhere.  Is there
> > >a list of other assumptions that is documented ?
> > >
> >
> > Typically, the original data buffer is with the application that
> > made the SCSI READ request - and this buffer may not have been
> > allocated with the constraints that should be ready for a simple
> > swapping. Assuming that the constraints are met, it would require
> > VM page flipping, which is considered to be an implementation hack.
> >
> > A protocol level solution to locate the buffer and its offset for
> > the incoming TCP segment is probably a better thing to have.
> >
> > -JP
> >
> >
>
>
>
>
>

---

From: [email-masked]
Date: Fri, 22 Dec 2000 10:56:32 +0200

Dear Mr. Cheng,

You can't fit data to tags if you have lost an iSCSI data header.  However
you could (in theory) place them in anonymous buffers and tag them later
(once you have recovered the header). In this case you either copy them or
have the storage-end do some scather-gather on the fragments you have
placed in anonymous buffers.

Julo

"Y P Cheng" <[email-masked]> on 21/12/2000 18:21:12

Please respond to "Y P Cheng" <[email-masked]>

To:   "'Ips@Ece. Cmu. Edu'" <[email-masked]>
cc:
Subject:  RE: Framing Discussion

> I agree with David. For the sake of clarity however I would like to add
> that even for those cases in which buffers are anonymous (like
> some appliance fileservers or disk controllers) the data that comes
> from the wire are not necessarily entire storage data blocks and have
> to be placed somewhere within a storage buffer. That particular
> placement can be made itself anonymous only by "placing" a
> "scather-gather" burden on the storage end of the wire
> and that is either expensive or outright impossible.
>
> Julo
>

Julo,

You lost me on the very last statement.  For appliance fileservers or disk
controllers, there is typically an small RTOS running.  The anonymous
non-storage data are received into a small bucket, say 256 bytes, which is
passed back to application for processing.  These types of data are known
as
control data like mode-select and login, etc. The component interacting
with
wire is given a number of these small buckets. There is no scatter-gather
burden on the storage end.

In the context of TCP/iSCSI implementation, the wire interacting component
needs to sort out between solicited and unsolicited iSCSI PDUs. For
solicited PDU, it places the data based on target task tags.  For
unsolicited control data, it places them into these small buckets.  For
unsolicited disk block data -- like streamed writes -- it places the data
into anonymous buffers prepared by the application.  Scatter/gather
handling
is a part of the "wire interacting component" design.  No extra burden.  In
fact, the HBA for an iSCSI initiator or target is specialized in move data
to destination buffers quickly.  Its main task is parsing the TCP/IP
segments.

This is the way I understand how an appliance fileserver or disk controller
works.  Either big companies have total different designs or I must have
misunderstood your statement.

Y.P. Cheng, Connectcom Solutions.

---

From: Wayland Jeong <[email-masked]>
Date: Fri, 22 Dec 2000 10:58:00 -0800

I have a question regarding the single bit TCP option. Forgive me if this
has been described somewhere else, but I don't completely understand how
this works.

The single bit option in the TCP header will inform the receiver that a PDU
header is present in the current TCP segment. Furthermore, I believe that
this option also indicates that the PDU header is aligned with the current
segment (i.e. the header can be found at a known offset). This must be the
case since the option does not include a pointer to the location of the
header within the segment.

Now, here is my question. If you cannot make any guarantees regarding the
alignment of PDU's and TCP segments being sent from the originator, then how
can you make any assumptions about when you might get this alignment bit?
Let's take an example. A receiver TOE that is parsing iSCSI PDU's notices a
dropped TCP segment. The iSCSI TOE must abandon the current PDU being
re-assembled and attempt to find the next alignment point. Everything that
comes in off the wire between the dropped segment and the next aligned
segment must be squirreled off into NIC memory somewhere until the dropped
segment is re-sent. Now, one of the problems we are trying to solve is the
problem of supplying a large amount of high-bandwidth memory on the NIC in
order to save-off an RTT's worth of wire-speed data waiting for information
to perform re-alignment. If a generalized receiver cannot make an assumption
of when it might receive this alignment through the bit in the TCP options
part of the header, then, how do you avoid having this worst-case
re-assembly memory?

Now, it might be the case that the receiver assumes that the sender always
packages its PDU's aligned. In this case, some segments may be less that
MSS, but that's okay. Thus, the receiver is assured that it will receive the
aligned bit in the TCP header and thus the next aligned PDU within a PDU's
worth of TCP data. This is fine, but it requires that both the sender and
the receiver play nicely together. If this is the case, is it the assumption
that if this option was agreed upon during negotiation, then the receiver
can assume that the sender is going to both use this option and ensure
alignment? Also, what about things in the network that terminate TCP like
NAT's, firewalls and PEP's. Surely they cannot be expected to keep this
assumption regarding PDU alignment.

-Wayland