Subject: Re: Extended RETR - Draft posted
From: Alexey Melnikov <[email-masked]>
To: Solar Designer <[email-masked]>
Cc: Anders Rundegren <[email-masked]>, [email-masked]
Date: Tue, 01 Feb 2000 00:56:39 -0700
List-Id: <ietf-pop3ext.imc.org>

Solar Designer wrote:

> Yes, it seems we need that.  There's one more issue that you'd need
> to decide on -- are we calculating CRLF's the way they're transferred
> over POP3 (would be more essential for the protocol), or the way line
> separators are typically stored on a UNIX-based server (to allow for
> a simple seek).

There are servers that 
1). construct message on the flight, storing it in some different
format.
2). store headers separately from body.
3). store message with CRLF even if they run on Unix
4). store message in bytestuffed form (as received from SMTP)

Whatever rule you choose for calculation of offset you will not satisfy
ALL implementations, 
so there is no point doing that.

BTW, ESMTP Checkpoint/Restart extension (RFC 1845) that does exactly the
same thing for SMTP (and suffers from the same dot stuffing problem)
says:

   Any octets added by any SMTP data-stuffing algorithm do
   not count as part of this offset. In the case of data transferred
   with the DATA command the offset must also correspond to the
   beginning of a line.

In checkpoint/restart offset starts from 0.
I would suggest to use the same for consistency.

> > The model should be very simple, please correct me if I'm wrong:
> >
> > msg-offset-bytestuff-server-client-bytedestuff-msg
> 
> Unfortunately, there may be a few more conversions:
> 
> server_mailbox -> skip(*)_to_offset -> bytestuff -> LF_to_CRLF -> ...
> ... transfer -> bytedestuff -> convert_to_client_mailbox_format
> 
> (*) we haven't defined the "skip" for the server_mailbox "layer", yet.
> 
> > Example:
> >
> > Message number is '1'.
> > Message size is 217,743 bytes.
> > Offset 113,435 starts with a '.'.
> >
> > Lets assume our transfer got interrupted after 113,434 bytes. Now, the
> 
> To illustrate the case of having to byte-stuff, you'd want the
> transfer to be interrupted after 113,435 bytes.

Would it be easier to use TOP-like command? Something like

   TOP <msgnum> <lines> <start_from_line>
  
Alexey

Date: Mon, 31 Jan 2000 23:16:29 -0700
From: Alexey Melnikov <[email-masked]>
To: Anders Rundegren <[email-masked]>, POP3 Extensions Mailing List <[email-masked]>
Subject: Re: Extended RETR - Draft posted
Sender: [email-masked]

Anders Rundegren wrote:
> 
> The draft for the extended RETR command has now been posted. Feedback
> is, as always, very appreciated. I'm not sure about the section dealing
> with extended response codes, it definitely needs some more work. Since
> English is not my native language I ask you to look over this too.
> 
> http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt

Few comments:

1). 1. Introduction:

>Many clients connecting to an ISP, or even ISP itself,
>***have their modems improperly setup*** ...

I am not sure why you want to blame modem configuration for all
problems.
I use to have 19200 dialup connection to my ISP and a very noisy phone
line.
Sometimes connection was dropped every 5 minutes.

2). 2.2 The offset argument

        If the specified offset is beyond the size of the message being
        retrieved, or in any other way erroneous in its syntax, 
        ***the response from the server should start with "-ERR".****

I suggest to change the ending to something like the following:
        the server should send negative response code.

3). 3. Extended Response Codes

Change
  -ERR message exists [OFFSET-OVERRUN]
and
  -ERR no such message [NON-EXISTENT]

to 
  -ERR [OFFSET-OVERRUN] message exists
and
  -ERR [NON-EXISTENT] no such message 
respectively

This is according to ABNF in RFC 2449.

I also doubt the usefulness of [NON-EXISTENT] response.
Good behaving client must check the validity of the message number when
it reconnects.
Also if another client deleted the interrupted message while the client
reconnects, the client is risking to resume downloading the next
message. This is not what you expect from the client. In this case some
persistent message Id should be used (UIDL?).

Alexey

From: Solar Designer <[email-masked]>
Subject: Re: Extended RETR - Draft posted
To: [email-masked] (Anders Rundegren)
Date: Mon, 31 Jan 2000 07:19:29 +0300 (MSK)
Cc: [email-masked]
Sender: [email-masked]

> > "The offset is given in octets. Everything transferred with a standard
> > RETR command should be accounted for."
> 
> The sentence following the above sentence in the draft states:
> 
> "This means that top headers, followed by messageparts will be
> transferred, in that order."

Yes, but I didn't read it as "byte-stuffing will not affect offsets",
the "everything" seemed to imply the opposite.

> > I am assuming the "everything" applies to byte-stuffing.
> 
> Read the above quote.

So it needs to be clearer.

> > The question: what offset should it request this time?  It could have
> > stored the amount of data it has received, along with the incomplete
> > message.  However, that amount is one octet larger than what it would
> > be if the transfer wasn't interrupted the first time.  Obviously, the
> > server, according to this definition of the extension, expects to see
> > offsets into the data that would be "transferred with a standard RETR
> > command" -- that is, without any previously recovered transfers.
> 
> OK, this is something I thought about when I was drafting the memo. I
> decided not to include an explanation because I saw it as obvious.
> Having given it some more thought, I think a more thorough explanation
> and an example might be in order. I will probably add a section
> explaining how byte-stuffing and calculation of offsets should be done.

Yes, it seems we need that.  There's one more issue that you'd need
to decide on -- are we calculating CRLF's the way they're transferred
over POP3 (would be more essential for the protocol), or the way line
separators are typically stored on a UNIX-based server (to allow for
a simple seek).

> The model should be very simple, please correct me if I'm wrong:
> 
> msg-offset-bytestuff-server-client-bytedestuff-msg

Unfortunately, there may be a few more conversions:

server_mailbox -> skip(*)_to_offset -> bytestuff -> LF_to_CRLF -> ...
... transfer -> bytedestuff -> convert_to_client_mailbox_format

(*) we haven't defined the "skip" for the server_mailbox "layer", yet.

> Example:
> 
> Message number is '1'.
> Message size is 217,743 bytes.
> Offset 113,435 starts with a '.'.
> 
> Lets assume our transfer got interrupted after 113,434 bytes. Now, the

To illustrate the case of having to byte-stuff, you'd want the
transfer to be interrupted after 113,435 bytes.

> received data has been bytedestuffed and saved, thus the client executes
> 'RETR 1 113435'. The server opens the message and jumps to this offset.
> It now bytestuffs the '.' as usual and transmits the data to the client
> who bytedestuff the data and adds it to the incomplete data it received
> earlier.
> 
> > "       a) Less consumption of bandwidth
> >         b) Less resources used on the server-side
> >         c) Less time used online for the client"
> > 
> > I would consider (c) the most important reason.  Bandwidth isn't that
> > important here, as POP3 transfers are typically within one ISP, from
> > a mailbox at the ISP to the ISP's dialup line, which is busy anyway.
> 
> You have a point about bandwidth, in some cases, not being a problem.
> However, taking in account the fast evolution on the internet, this
> might in the future apply to other connections than modems, i.e. a
> company's internal network, since attachments might be much larger than
> they are today.

Yes, that's a valid point as well, but, in my opinion, (c) is more
important.

> > (b) is not fully achieved with your definition of the offsets, as the
> > server has to read the entire message from the mailbox to account for
> > byte-stuffing.
> 
> Well, this shouldn't be a problem with the above explained model. The
> server just need to do an fseek() to the specified offset.

I wish it was that simple.

To allow for a simple seek, we would need to define that offsets are
into the message as stored in a BSD-style mailbox.  (I am assuming
that performance is less critical on non-UNIX servers, and that they
can choose the mailbox format that allows for better performance if
they care.)  The calculation of offsets on the client end would get
more complicated, then -- unless the client uses single LF's for its
mailboxes as well.  Maybe it's not that bad, after all.

Signed,
Solar Designer

Date: Mon, 31 Jan 2000 04:18:12 +0100
From: Anders Rundegren <[email-masked]>
To: Solar Designer <[email-masked]>
CC: [email-masked]
Subject: Re: Extended RETR - Draft posted
Sender: [email-masked]

> "The offset is given in octets. Everything transferred with a standard
> RETR command should be accounted for."

The sentence following the above sentence in the draft states:

"This means that top headers, followed by messageparts will be
transferred, in that order."

> I am assuming the "everything" applies to byte-stuffing.

Read the above quote.

> The question: what offset should it request this time?  It could have
> stored the amount of data it has received, along with the incomplete
> message.  However, that amount is one octet larger than what it would
> be if the transfer wasn't interrupted the first time.  Obviously, the
> server, according to this definition of the extension, expects to see
> offsets into the data that would be "transferred with a standard RETR
> command" -- that is, without any previously recovered transfers.

OK, this is something I thought about when I was drafting the memo. I
decided not to include an explanation because I saw it as obvious.
Having given it some more thought, I think a more thorough explanation
and an example might be in order. I will probably add a section
explaining how byte-stuffing and calculation of offsets should be done.

The model should be very simple, please correct me if I'm wrong:

msg-offset-bytestuff-server-client-bytedestuff-msg

Example:

Message number is '1'.
Message size is 217,743 bytes.
Offset 113,435 starts with a '.'.

Lets assume our transfer got interrupted after 113,434 bytes. Now, the
received data has been bytedestuffed and saved, thus the client executes
'RETR 1 113435'. The server opens the message and jumps to this offset.
It now bytestuffs the '.' as usual and transmits the data to the client
who bytedestuff the data and adds it to the incomplete data it received
earlier.

> "       a) Less consumption of bandwidth
>         b) Less resources used on the server-side
>         c) Less time used online for the client"
> 
> I would consider (c) the most important reason.  Bandwidth isn't that
> important here, as POP3 transfers are typically within one ISP, from
> a mailbox at the ISP to the ISP's dialup line, which is busy anyway.

You have a point about bandwidth, in some cases, not being a problem.
However, taking in account the fast evolution on the internet, this
might in the future apply to other connections than modems, i.e. a
company's internal network, since attachments might be much larger than
they are today.

> (b) is not fully achieved with your definition of the offsets, as the
> server has to read the entire message from the mailbox to account for
> byte-stuffing.

Well, this shouldn't be a problem with the above explained model. The
server just need to do an fseek() to the specified offset.

// Anders

From: Solar Designer <[email-masked]>
Subject: Re: Extended RETR - Draft posted
To: [email-masked] (Anders Rundegren)
Date: Sun, 30 Jan 2000 06:44:35 +0300 (MSK)
Cc: [email-masked]
Sender: [email-masked]

> The draft for the extended RETR command has now been posted. Feedback
> is, as always, very appreciated. I'm not sure about the section dealing
> with extended response codes, it definitely needs some more work. Since
> English is not my native language I ask you to look over this too.
> 
> http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt

The initial impression is that this clearly needs more work, perhaps
resulting in a different approach to define offsets within a message.

"The offset is given in octets. Everything transferred with a standard
RETR command should be accounted for."

I am assuming the "everything" applies to byte-stuffing.  Now, let's
assume a transfer got interrupted right before a dot in the message.
The client requests that the transfer is continued from that offset,
and the operation succeeds.  The server byte-stuffs that dot (as it
is now at the beginning of a line being transferred), and transfers
more of the message to the client.  Unfortunately, the client gets
disconnected once again.  It wants to recover the transfer.

The question: what offset should it request this time?  It could have
stored the amount of data it has received, along with the incomplete
message.  However, that amount is one octet larger than what it would
be if the transfer wasn't interrupted the first time.  Obviously, the
server, according to this definition of the extension, expects to see
offsets into the data that would be "transferred with a standard RETR
command" -- that is, without any previously recovered transfers.

With this definition of the RETR extension, the client would have to
detect if the first character it receives after a transfer continues
is a dot _and_ it is not at the beginning of a _message_ line.  It
should then subtract one from the data size it is calculating, for
the case of having to continue the transfer once again.  This should
be at least documented in the same RFC, but I suggest that we better
avoid this extra complexity somehow.

A more obvious requirement of this definition is that the client
either stores the amount of data it has received for each incomplete
message, or re-calculates that amount once it wants to finish the
transfer.  The former is fine if we only want to recover transfers
while the client process is running, but might require extending
mailbox format for some clients if we want that functionality even
after an OS restart.  The latter puts some code from POP3 servers,
which used to be server-specific, into clients as well.

Finally, speaking of your reasoning for the extension --

"       a) Less consumption of bandwidth
        b) Less resources used on the server-side
        c) Less time used online for the client"

I would consider (c) the most important reason.  Bandwidth isn't that
important here, as POP3 transfers are typically within one ISP, from
a mailbox at the ISP to the ISP's dialup line, which is busy anyway.
(b) is not fully achieved with your definition of the offsets, as the
server has to read the entire message from the mailbox to account for
byte-stuffing.

Signed,
Solar Designer

Date: Sat, 29 Jan 2000 13:34:50 +0100
From: Anders Rundegren <[email-masked]>
To: [email-masked]
Subject: Extended RETR - Draft posted
Sender: [email-masked]

The draft for the extended RETR command has now been posted. Feedback
is, as always, very appreciated. I'm not sure about the section dealing
with extended response codes, it definitely needs some more work. Since
English is not my native language I ask you to look over this too.

http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt

Thanks!

// Anders Rundegren

Date: Sun, 09 Jan 2000 12:30:58 -0800
From: Chris Newman <[email-masked]>
Subject: Re: POP3 "resume download" extension
To: Keith Moore <[email-masked]>, [email-masked]
Cc: Anders Rundegren <[email-masked]>, "[email-masked]" <[email-masked]>
Sender: [email-masked]

--On Friday, December 17, 1999 1:22 -0500 Keith Moore <[email-masked]> 
wrote:
> fwiw, a lot of folks are likely to push back on defining any new extension
> to pop. there is a feeling that two mail access protocols are too many,

Let me explain my theory for why we can't get rid of POP easily.  The 
primary customer for POP servers is large ISPs providing individual user 
service.  These ISPs need to put as many mail users on a server as possible 
in order to keep management costs to a minimum so the per-user margin is 
profitable.  Most of the server operating systems which can scale large 
enough with enough reliability and performance are based on the BSD-socket 
API.  When machines get that big, two major limiting factors on scalability 
are a small fixed limit on TCP/IP connections (max file descriptors) and 
the exponential performance cost of select and poll.  In this scenario, 
TCP/IP connections are expensive and need to be kept as short lived as 
possible.  IMAP is anethema, as would be any extension to POP which might 
result in longer-running connections.

The excessively ugly /dev/poll interface in Solaris 8 (pre-release) fixes 
half the problem but that OS still has a small hard-coded limit on the 
number of file descriptors instead of making RAM the only limiting factor 
as it should.  So we have to live with protocols that use short-lived 
TCP/IP connections like POP and HTTP 1.0 (with the shared Internet 
suffering as a result).

> if you do define an extension for partial message retrieval I think it 
> should allow partial message retrieval (e.g. byte count) rather than 
> just resume.  that way, you don't have to drop a connection to 
> get the server to stop sending you a message.

I happen to support the idea of a resume download extension, but I'm 
opposed to adding a partial message retrieval extension to POP.  My reasons 
are subtle (mostly gut reaction), but I'll attempt to explain:

First, I don't believe most clients would bother to do a pipelined chunking 
download of POP mail.  It simply adds too much complexity to the common 
case (download all mail and delete from server) in order to optimize an 
uncommon case (user cancel of large message).  Simply closing the socket 
and re-opening is much less error prone, and wasting a few round-trips 
simply isn't a performance concern when you're interacting with the user. 
In IMAP, connections are usually long-lived so a pipelined chunking 
download makes more sense, but even though the facility is present in the 
protocol it's largely unused.  Furthermore, the interactions between 
dot-stuffing and byte ranges would make the POP3 version tricker to 
implement than the IMAP version.  That's far too much complexity to save a 
few round trips.

Second, I don't think POP3 is the right protocol to start experimenting 
with new user cancel facilities.  Most of the times the IETF has tried an 
explicit user cancel facility in a protocol, the result has largely failed 
(e.g., telnet TCP urgent data & IMAP pipelined/chunked partial fetch).  A 
largely stable full standard protocol is the wrong place to experiment.

Finally, I think a byte-range fetch could be abused.  In particular, some 
clients will try to use it to fetch pieces of a large message (something 
which IMAP does far better).  This would require a series of round trips to 
see if enough data has been fetched.  So if you're going to add a 
byte-range fetch, you also need to add a separate facility to at least 
fetch to the end of the first text part (possibly with a size limit as 
well) at the same time to minimize the potential for abuse.  But now the 
extension adds one or two commands and a bit of server-side MIME parsing 
which is really pushing the complexity barrier for a POP extension.

A "resume download" extension is simpler and doesn't have the above issues. 
About all it can be used for is to reduce retransmission of data after a 
user cancel or network failure -- thus making a POP client signficantly 
more likely to succeed at downloading a large message over a flaky 
connection.  That has an acceptable complexity:benefit ratio in my book, 
byte-range fetch does not.

		- Chris