Subject: RE: iSCSI: RE: Framing Discussion

From: Wayland Jeong <[email-masked]>
Date: Wed, 27 Dec 2000 11:09:59 -0800

[ stuff deleted ]

>> Let's suppose that we have an iSCSI TCP connection in which we
>> have multiple outstanding I/O's. Thus, the byte stream has
>> interleaved within it commands and data from different I/O's.
>> When we detect a dropped segment either through normal TCP congestion
>> or via SACK, how do we map the missing byte block to the
>> appropriate context? If we keep the segments around, then we
>> could match the missing segment easily and re-transmit. But that would
>> require the NIC to implement a BWDP's worth of transmit buffer memory.
>>
>> To have the iSCSI TOE re-transmit directly from the buffer cache, it
seems
>> that we would need some sort of context that would allow us to map a byte
>> window to a specific, meaningful point somewhere in the middle of a CDB
>> context. Essentially, you need enough context to be able to re-construct
>> the TCP fifo since the memory in this fifo has since been effectively
>> re-allocated. Maybe this isn't too hard, but it sure sounds like
>> a difficult problem for hardware to solve. But, as the software
>> folks around here keep telling me, "it's just gates" ;-)
>
> Yes, multiple I/O's and interleaved data streams require a context manager
> who maps the missing segment back to its large exchange table to determine
> how to retransmit the dropped segment.  No, Wayland, I would not do it in
> hardware.  It is all in microcode.  The microcode size is actually not
that
> big. On the contrary, the exchange table can be a few hundred KB's.  All
you
> need is a very very fast microengine with small number of gates, a true
> RISC.  Please keep asking the "dumb" questions.  I am mostly impressed by
> your questions.
>
Thanks for the reply. You'll have to let me know which questions don't
impress you ;-)

Yes, I am assuming that the re-transmit process will be handled in
firmware/micro-code. It's still gates though, they just happen to be in
someone's uP core ;-) It still seems like a tough problem in the general
case. 

Let's assume a worst case scenario. The iSCSI PDU size is greater than the
TCP MSS and the network MTU and you are talking to a firewall that is
re-packaging your TCP stream. No matter how much you try to send nicely
aligned PDU's, the firewall is going to take your less than MSS size TCP
segments and package them up so that you get full-size TCP segments by the
time it hits the target. The target detects a missing segment and keeps the
left edge of the window constant for three consecutive ACK's. Furthermore,
we are using the SACK option in TCP to optimize our performance over LFN's.
Thus, we are presented with the exact blocks that are missing.
Unfortunately, these missing blocks have fragments of PDU's from different
I/O's (could be command, could be data). Even worse, since we chose a PDU
size greater than MSS, some segments might be part of a PDU that does not
contain an iSCSI header and does contains a digest covering the entire PDU. 

Yeesh!! Thank goodness all I have to do is drop-in an embedded processor
into our chip. I'll let the firmware folks deal with this problem.
Certainly, this path does not have to be high-performance since we are going
into congestion control anyway, but we have to deal with it. We can keep a
context stack for the current open TCP sessions which contain mappings of
TCP sequence numbers to specific CDB context locations (either command or
offset within the gather list). We can keep this stack as deep as the
maximum number of outstanding (i.e. not ack'd) TCP segments which for
Randy's example (1.25Gbs and RTT of 100ms) is not too bad (about 8K
entries). We can recover the contexts needed to re-build this missing TCP
segment and re-construct entire PDU(s) so that any necessary digests can be
re-calculated. We can then stage this data in memory somewhere and pull-out
the exact TCP block that we need to re-send. Lovely.

I'm not saying it's impossible, but I am saying that implementing Fibre
Channel looks like a walk in the park compared to this stuff.

BTW, regarding the current iSCSI draft. I didn't see a Login/Text key
associated with negotiating the iSCSI PDU size. Is it assumed that an iSCSI
implementation should handle any PDU size?

> Y.P. Cheng, Connectom Solutions.
>
-Wayland

---

From: "Y P Cheng" <[email-masked]>
Date: Wed, 27 Dec 2000 12:32:03 -0800

> Certainly, this path does not have to be high-performance since
> we are going into congestion control anyway, but we have to deal with it.
> We can keep a
> segment and re-construct entire PDU(s) so that any necessary
> digests can be re-calculated. We can then stage this data in memory
> somewhere and pull-out the exact TCP block that we need to re-send.
> Lovely.

You forget the most important fact of an iSCSI adapter.  The microcode does
the transmit knows exactly how it creates a TCP segment.  First, it would
not mix multiple PDU's into a single segment.  Second, there is no TCP stack
in the iSCSI adapter.   While a command or status PDU is created on the fly,
all data PDUs are retransmitted from the application buffers which are not
freed until an exchange is complete.  Third, there is a separate TCP stream
to each connection.  The task of retransmit is to find the exchange number
by using the end-point and segment sequence number. The microcode keeps
track the starting sequence numbers for multiple open exchanges.  However,
one can take an easy way out by having only one open exchange per endpoint.
Of course, this slows things down.  But, don't be too surprised that some
implementation might just do that.   As I said before, to keep track of a
few thousand exchanges, there is a lot of memory needed.  However, the
algorithm to go through the exchange context to retransmit a missing segment
is straight forward and does not take a rocket scientist!  (Or should I say
an iSCSI scientist?)

In summary, transmit is easy because you got to call the shots as long as
rules are followed.  Can't say the same for receive when old TCP
implementations must be honored.

---

From: Douglas Otis <[email-masked]>
Date: Wed, 27 Dec 2000 15:28:41 -0800

Y.P.,

With respect to networks, there is more than just SCSI.  Yes, we can create
a network interface that solves immediate needs of SCSI, but at the
potential expense of adding a unique solution per each application demanding
content directed services.  IPS advocates are not allowed to discuss impact
of this interface such as if this connection is shared with IPs or ports or
how an application is indicated.  Once vendor specific details of SCSI are
solved, this same connection may also then be called upon to create another
unique interface for VI or IPC.  Are application/vendor unique solutions
beneficial if there is only a resemblance to the original protocol where
different schemes employed?  As these application standards change, be sure
to include Flash memory for your embedded processor.  Is this how WinModem
was created where application and hardware interface is blurred by vendor
unique solutions?

Doug

---

From: "Y P Cheng" <[email-masked]>
Date: Thu, 28 Dec 2000 10:28:35 -0800

> With respect to networks, there is more than just SCSI.  Yes, we
> can create a network interface that solves immediate needs of
> SCSI, but at the potential expense of adding a unique solution
> per each application demanding content directed services.
> IPS advocates are not allowed to discuss impact of this
> interface such as if this connection is shared with IPs or ports or
> different schemes employed?  As these application standards
> change, be sure to include Flash memory for your embedded processor.
> Is this how WinModem was created where application and hardware
> interface is blurred by vendor unique solutions?
>
> Doug

Doug,

What I said that "You can send anything you want, as long as rules are
followed" doesn't meant it is an incompatible solution requiring special
API.  An iSCSI adapter is both a NIC and a SCSI adapter.  As a NIC, it is
capable of RDMA to support VI and Winsock Direct.  As an iSCSI adapter it
adhere strictly to SCSI API and the IPS specifications syntactically and
semantically.  It has device drivers for different OS's providing SCSI
services with the exception of making TCP connections for logins.  However,
having said all these, each iSCSI adapter still has a unique implementation
including the ways it sends and resends TCP frames and supports multiple TCP
connections concurrently.  Being stream-oriented, TCP does give us a lot of
freedom how to segment the byte stream and how to send ACKs.

There are a lot of issues with traditional TCP implementations to support
networks with long latency delays and high probability of dropping frames.
But, if someone puts a new implementation inside an adapter, he has the
freedom to add new options like TCPRDMA and MSGBNDRY, even the SMTP protocol
to overcome the problems.  Not all implementations are created equal.  Not
everyone supports new options.  The magic word is interoperability.  The
secret is how two adapters both supporting new options talk to each other.
I thought the TCP option negotiation permits that already.  What I expect
from the IPS effort is to tell me how these new options should look like.

Yes, almost all adapters today have a flashable ROM that supports field
upgrades with new implementations for new TCP options approved by IPS WG.

Y.P.

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 28 Dec 2000 15:00:13 -0800

Y.P.

So you agree that each application requiring content directed services will
necessitate an additional vendor unique hardware interface.  The T10 group
does have the required parameters to support SCSI, and  Microsoft, Compaq,
and Intel have parameters to support VI and most OS implementations already
support network devices at the transport level or below.  With Microsoft
wishing the addition of IPC, there is now at least 4 vendor unique hardware
interfaces expected of this network connection if such level of service is
required.  Do you see an advantage discussing all these interfaces together
than each separately?  If we can find a non-application specific means of
providing these services, then the world is spared these hardware variants.
I would expect a great deal of complexity in attempting to nail down all the
details required to interface so many clever independent hardware solutions.

With one interface requiring the application to reconstruct the datagrams
for sending a retry, and with the standard method handling such details
below the application, there will be a significant change required to
support both modes of operation.  iSCSI is already a transport layer in
itself by allowing a separate retry mechanism from that of either SCSI or
TCP.  iSCSI combines multiple connections and implements its own flow
control and error detection.  Combine that with a layer between IP and TCP
to inject the concept of iSCSI PDUs as an element contained and aligned with
TCP.  From this alignment or framing, out of sequence network segments can
have the data extracted from application specific encapsulation and placed
within third-hand buffers.  This allows a simple ASIC which uses embedded
memory rather than external memory.  Perhaps more importantly, the number of
drivers to support this plethora becomes another concern if to see
standardization.

Doug