Subject: Re: iSCSI: RE: Framing Discussion

From: [email-masked]
Date: Wed, 27 Dec 2000 08:22:58 +0200

Wayland,

Let's ignore for the first approximation anything that is not data.
Our assumption when outlining iSCSI was that all send data  (including most
of the iSCSI headers can be rebuilt from context).   The only context
information a sender has to keep is an association of a sender TCP sequence
number with either an iSCSI header or a SCSI buffer address and length.
The details will certainly vary with implementation.
In a loosely integrated solution the TCP stack will keep this context for
you.

In a tightly integrated solution you will have the TCP, on resend,
call-back the iSCSI layer to rebuild
headers and restate data addresses in which case iSCSI has to keep in its
tables some relation between the TCP sequence number and a specific data
piece (Task Tag, Data-packet-# etc.).

Regards,
Julo

Wayland Jeong <[email-masked]> on 26/12/2000 22:08:41

Please respond to Wayland Jeong <[email-masked]>

To:   Wayland Jeong <[email-masked]>, [email-masked]
cc:
Subject:  iSCSI: RE: Framing Discussion

I think I now understand the assumptions regarding the single-bit TCP
option
for indicating the presence of a PDU header in the current segment (thanks
Costa). It boils down to a hardware implementation that is tuned for the
best case (i.e. small amount of re-assembly memory on the NIC to park
fragmented PDU's with the assumption that the next aligned PDU is coming
shortly) and a method for dropping things into software when we are talking
to an ill-behaved NIC. Okay, I'll buy that.

Now, I have another dumb question. For direct data placement, the
discussions have been centered mostly around the need for alignment when
parsing PDU's on the receiving iSCSI TOE. One potential issue that has not
been discussed is the problem of how to handle re-transmission on the
sending iSCSI TOE.

From previous discussions, I am assuming that our goal is to avoid having a
network BWDP worth of memory on the NIC. The receiver can avoid this memory
by recovering PDU alignment in the TCP stream and using the self-describing
headers in the wire protocol (either iSCSI offsets or a RDMA shim layer) to
put the data directly in the buffer cache. On the sending side, we can DMA
directly from iSCSI descriptor CDB's into the TCP pipe using a hardware
path. But, unless we keep all of those un-acked TCP segment buffers around
in the NIC, it will be difficult to recover the context when we have to
re-transmit.

Let's suppose that we have an iSCSI TCP connection in which we have
multiple
outstanding I/O's. Thus, the byte stream has interleaved within it commands
and data from different I/O's. When we detect a dropped segment either
through normal TCP congestion or via SACK, how do we map the missing byte
block to the appropriate context? If we keep the segments around, then we
could match the missing segment easily and re-transmit. But that would
require the NIC to implement a BWDP's worth of transmit buffer memory.

To have the iSCSI TOE re-transmit directly from the buffer cache, it seems
that we would need some sort of context that would allow us to map a byte
window to a specific, meaningful point somewhere in the middle of a CDB
context. Essentially, you need enough context to be able to re-construct
the
TCP fifo since the memory in this fifo has since been effectively
re-allocated. Maybe this isn't too hard, but it sure sounds like a
difficult
problem for hardware to solve. But, as the software folks around here keep
telling me, "it's just gates" ;-)

-Wayland

---

From: Wayland Jeong <[email-masked]>
Date: Wed, 27 Dec 2000 14:11:23 -0800

>> Yeesh!! Thank goodness all I have to do is drop-in an embedded processor
>> into our chip. I'll let the firmware folks deal with this problem.
>> Certainly, this path does not have to be high-performance since
>> we are going into congestion control anyway, but we have to deal with it.
>> We can keep a
>> context stack for the current open TCP sessions which contain mappings of
>> TCP sequence numbers to specific CDB context locations (either command or
>> offset within the gather list). We can keep this stack as deep as the
>> maximum number of outstanding (i.e. not ack'd) TCP segments which for
>> Randy's example (1.25Gbs and RTT of 100ms) is not too bad (about 8K
>> entries). We can recover the contexts needed to re-build this missing TCP
>> segment and re-construct entire PDU(s) so that any necessary
>> digests can be re-calculated. We can then stage this data in memory
>> somewhere and pull-out the exact TCP block that we need to re-send.
>> Lovely.
>
> You forget the most important fact of an iSCSI adapter.  The microcode
does
> the transmit knows exactly how it creates a TCP segment.  First, it would
> not mix multiple PDU's into a single segment.  
>
Yes. The iSCSI adapter can make this choice. But, my concern is in talking
to a proxy like a firewall that terminates TCP. Such a proxy may and
probably will re-package your TCP stream especially if you are sending less
than MSS size segments which you inevitably will if you are trying to align
PDU's at the sender. If a proxy re-bundles your TCP stream and that stream
hits the target with a missing segment, that segment could cross alignment
boundaries. Normally, this will not be the case, but if you design an iSCSI
TOE that cannot handle TCP ACK's with non-PDU aligned sequence numbers then
your TOE has some inherent limitations. At a minimum, it shouldn't break.
Thus, some amount of context and an ability to handle the situation, albeit
on a very slow path, is required.

> Second, there is no TCP stack
> in the iSCSI adapter.   While a command or status PDU is created on the
fly,
> all data PDUs are retransmitted from the application buffers which are not
> freed until an exchange is complete. 
>
I'm not sure what you mean by application buffer. We want to immediately
free the NIC buffers since one of the goals discussed in Randy's framing
document is to avoid having a BWDP worth of high-speed memory on the NIC.
Current Fibre Channel HBA's are single chip solutions with no off-board
memory and only a handful (4-8) transmit FC frame buffers on-chip. If you
are referring to the buffer cache, then you have the same problem which is
how to map the TCP sequence numbers back to your SCSI protocol-level
constructs.

> Third, there is a separate TCP stream
> to each connection.  The task of retransmit is to find the exchange number
> by using the end-point and segment sequence number. The microcode keeps
> track the starting sequence numbers for multiple open exchanges.  However,
> one can take an easy way out by having only one open exchange per
endpoint.
> Of course, this slows things down.  But, don't be too surprised that some
> implementation might just do that.   As I said before, to keep track of a
> few thousand exchanges, there is a lot of memory needed.  However, the
> algorithm to go through the exchange context to retransmit a missing
segment
> is straight forward and does not take a rocket scientist!  (Or should I
say
> an iSCSI scientist?)
Yes, it does simplify things if you only have a single outstanding I/O, but
speaking from experience, your performance will stink.

> 
> In summary, transmit is easy because you got to call the shots as long as
> rules are followed.  Can't say the same for receive when old TCP
> implementations must be honored.
>
Thanks again for the input. This discussion really isn't too relevant to the
efforts by this group to standardize a mapping of iSCSI to TCP. These
implementation details are more about the complexity associated with
designing high-performance implementations of iSCSI. By high-performance, I
mean implementations that can meet or exceed current Fibre Channel
performance and can be brought to market at cost parity with FC HBA's.

-Wayland