Subject: Re: Extended RETR - version 2
From: <[email-masked]>
To: [email-masked]
Date: Mon, 28 Feb 2000 16:29:13 -0800 (PST)
List-Id: <ietf-pop3ext.imc.org>

On Mon, 28 Feb 2000, Mike Gahrns wrote:

> Re: Extended RETR - version 2>> Not necessarily.  I think a strong case can
> be made that while the volume
> >> may increase, the quality is also increasing as well.  As I mentioned
> >> earlier, I have no objections to this extension from a technical point of
> >> view, just pointing out that before we extend POP, lets make sure that we
> >> are solving a real problem.  A large part of POP's success is due to its
> > >simplicity.
> >Exactly what are you referring to by "quality", please specify.
> >// Anders
> *** Sorry for the delay in responding. I did not have email access last
> week...
> This comment was in regards to a previous statement that the number of line
> errors that occur and cause a message to be RETR'd again will only increase,
> as the use of the internet continues to grow beyond everybody's
> expectations.   The point I was trying to make was that although volume will
> no doubt increase, the quality of the lines/connections is also increasing.
> I would bet that in most of the world today, the quality of the connection
> is at a point where line quality is a non-issue, and for those parts of the
> world where line quality is still an issue, I would suspect that quality
> will rapidly improve.

It has a long way to go. I'm just between silicon valley and San
Francisco, one of the most wired parts of the world. I'm only 2.5 miles
from my local PacBell switch.

While I wait for DSL I'm using AOLs dialup pool. I regularly get
connection drops. Whether that's due to faulty equipment at AOL or whether
it's due to genuine line-quality issues doesn't really matter to me as an
end-user. As higher bandwidth connectivity rolls into an area the POTS
providers are pushed down market and the quality of service drops (at the
low end of the dialup market, anyway).

Dialup isn't particularly reliable in general right now. Were I trying to
download megabyte+ mails via POP3 I'd be retrying after connection drop
fairly often.

Cheers,
  Steve

From: "Mike Gahrns" <[email-masked]>
To: "Anders Rundegren" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: Extended RETR - version 2
Date: Mon, 28 Feb 2000 16:13:36 -0800
Sender: [email-masked]

Re: Extended RETR - version 2>> Not necessarily.  I think a strong case can
be made that while the volume
>> may increase, the quality is also increasing as well.  As I mentioned
>> earlier, I have no objections to this extension from a technical point of
>> view, just pointing out that before we extend POP, lets make sure that we
>> are solving a real problem.  A large part of POP's success is due to its
> >simplicity.
>Exactly what are you referring to by "quality", please specify.
>// Anders
*** Sorry for the delay in responding. I did not have email access last
week...
This comment was in regards to a previous statement that the number of line
errors that occur and cause a message to be RETR'd again will only increase,
as the use of the internet continues to grow beyond everybody's
expectations.   The point I was trying to make was that although volume will
no doubt increase, the quality of the lines/connections is also increasing.
I would bet that in most of the world today, the quality of the connection
is at a point where line quality is a non-issue, and for those parts of the
world where line quality is still an issue, I would suspect that quality
will rapidly improve.

Date: Sat, 26 Feb 2000 13:02:41 +0100
From: Anders Rundegren <[email-masked]>
To: [email-masked]
Subject: Re: Extended RETR - version 3
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------C761A1235DE3A859830F0320
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

I've attached version 3 of the draft describing the extended RETR
command. It incorporates some changes, suggested by Solar Designer and
Jochen Topf. Some suggested changes hasn't been incorporated, see my
reasoning for this below.

"Offsets equal to message sizes are allowed"
--------------------------------------------
I didn't add this to the draft since I found it redundant. If a message
transfer being resumed doesn't end with a '.' on a new line followed by
a CRLF pair, the transfer would be considered to be incomplete anyway.
There should be no need for the server to "double-check" this. Also, it
introduces further complexity to the extension.

"CAPA tag re-definition"
------------------------
This re-definition of the CAPA tag introduces furthermore complexity and
I fail to see the use of being able to specify the availability of the
extension to individual users maildrops.

I'm not sure on how to best solve the problem with message numbering not
being the same between sessions, for now I've included a note to
implementors that the use of UIDL is required to ensure validity of the
downloaded message.

Please continue with all feedback! I find comments posted on this list
to be extremely helpful in "bug-tracking" the specification of the
extension.

// Anders
--------------C761A1235DE3A859830F0320
Content-Type: text/plain; charset=us-ascii;
 name="extended-retr-draft-v3.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="extended-retr-draft-v3.txt"

INTERNET-DRAFT                                            INTERNET-DRAFT
                                                            A. Rundegren
                                                       26 February, 2000

                  Extended RETR - Extension to POP3
                    <draft-rundegren-pop3-01.txt>           

Status of this Memo

        This document is an Internet-Draft and is in full
        conformance with all provisions of Section 10 of RFC2026.

        Internet-Drafts are working documents of the Internet
        Engineering Task Force (IETF), its areas, and its working
        groups. Note that other groups may also distribute working
        documents as Internet-Drafts.

        Internet-Drafts are draft documents valid for a maximum of
        six months and may be updated, replaced, or obsoleted by
        other documents at any time. It is inappropriate to use
        Internet-Drafts as reference material or to cite them other
        than as "work in progress."

        The list of current Internet-Drafts can be accessed at
        http://www.ietf.org/ietf/1id-abstracts.txt

        The list of Internet-Draft Shadow Directories can be
        accessed at http://www.ietf.org/shadow.html.

        Distribution of this document is unlimited. Please send
        comments to the author.

Abstract

        This memo introduces an extension to the RETR command in the
        POP3 protocol. The extension makes it possible for an
        interrupted RETR command to be continued. This is
        particularly useful when downloading large messages over
        unstable connections.

A. Rundegren                                                    [page 1]
-- LINE -- 58 --
INTERNET-DRAFT                                         26 February, 2000

Table of Contents
        1.  Introduction  ...........................................  2
        2.  The Extended RETR Command  ..............................  2
          2.1  RFC 1939 Specification  ..............................  2
          2.2  The Offset Argument  .................................  3
	  2.3  Bytestuffing  ........................................  4
        3.  Extended Response Codes  ................................  5
        4.  The CAPA Tag  ...........................................  5
        5.  Security Considerations  ................................  6
        6.  Acknowledgements  .......................................  6
        7.  References  .............................................  6
        8.  Author's Address  .......................................  6

1. Introduction

        When email was first developed it was intended to transfer
        readable text messages. Since the introduction of MIME (RFC
        2045) it has become more common to transfer other message
        parts, more commonly known as attachments. These attachments
        are often much larger than the readable textparts.

        The current POP3 specification requires messages to
        re-downloaded from scratch in the case of a premature
        disconnect during a message transfer.

        Such disconnects happens for various reasons, including poor
        phone line quality, modem setup, network packet loss and
        temporary network problems.

        By extending the RETR command, the download-and-delete model
        that characterizes the POP3 protocol becomes more complete in
        handling todays requirements of email. The benefits are:

        a) Less time used online for the client
        b) Less resources used on the server-side
        c) Less consumption of bandwidth

2. The Extended RETR Command

        This section will give a detailed view on how the second
        argument of the extended RETR command should be implemented.

2.1 RFC 1939 Specification

        The RETR command (as specified by RFC 1939), takes one argument
        which specifies the message to be retrieved by the client. The
        allowed responses are "+OK" and "-ERR", without the quotes,
        possible followed by a space (decimal code 032) and a comment.

A. Rundegren                                                    [page 2]
-- LINE -- 116 --
INTERNET-DRAFT                                         26 February, 2000

2.2 The Offset Argument

        The second argument to the extended RETR command specifies the
        offset in the message of where the transfer should start. The
        offset is given in octets, and is calculated the way message
        sizes are , as defined in section 11 of RFC 1939. In order to
        ensure a valid multi-line response, with all lines terminated
        with a CRLF pair, the offset MUST NOT point into the middle of
        a CRLF pair.

        As specified by RFC 1939, responses can not be more htan 512
        characters long, including the terminating CRLF.

        Implementors should note that there is no guarantee that
        the message numbering stays the same between sessions. In
        order to ensure the validity of a downloaded message,
        the use of a UIDL command is required.

        The syntax of the extended RETR command would be:

            RETR msg offset

        Arguments:
            A message-number (required) which may NOT refer to a
            message marked as deleted. An offset (optional) of where
            to start the transfer, specified in octets. If the offset
            argument is omitted, the message will be sent as specified
            by the RETR command in RFC 1939.

        Description:
            If the POP3 server issues a positive response, then the
            response given is multi-line. After the initial +OK, the
            POP3 server sends the message corresponding to the given
            message-number, starting from the offset provided, being
            careful to byte-stuff the termination character (as with
            all multi-line responses).            

        Possible Responses:
            +OK message follows
            -ERR no such message

        Examples:
            C: RETR 1 345628
            S: +OK 1253512 octets
            S: <the POP3 server sends the message here, starting with
                the 345629th octet of the message>
            S: .

A. Rundegren                                                    [page 3]
-- LINE -- 174 --
INTERNET-DRAFT                                         26 February, 2000

        If the specified offset is beyond the size of the message being
        retrieved, points into the middle of a CRLF pair, or in any
        other way erroneous in its syntax, the server should respond
        with a negative response code. This would however not
        distinguish the type of error. In the following section you
        can read about the use of extended response codes to clarify
        errormessages.

2.3 Bytestuffing

        This section explains how bytestuffing should be performed when
        resuming a previously interrupted transfer. When data is to be
        transferred from the server to the client, the server should
        be careful to byte-stuff all data. If the previous transfer
        was interrupted in the middle of a line, where the remaining
        part of the line starts with a '.' character, this means that
        data, that would normally not be bytestuffed (since it's not in
        the beginning of a line), must be byte-stuffed. Any octets added
        by any data-stuffing algorithm do not count as part of the offset
        used by the Extended RETR command.

        Examples:
            Message number is 1.
            Message size is 217,743 octets.
            Offset 113,435 starts with a '.'.

            Lets assume, that when transferring the above message, an
            interrupt occurs after exactly 113,435 octets. When
            resuming the transfer, the client executes "RETR 1 113435".
            The server then byte-stuffs the data before transferring it
            to the client, who then byte-destuffs the data before
            appending it to the previously transferred data.

A. Rundegren                                                    [page 4]
-- LINE -- 232 --
INTERNET-DRAFT                                         26 February, 2000

3. Extended Response Codes

        The extended RETR command does not require the implementation
        of extended response codes (RFC 2449) to be present in order
        to work. However, this memo specifies the extended response
        codes to be used if implemented.

        If the specified offset is beyond the size of the message being
        retrieved, the extended response code should be:

            -ERR [OFFSET-OVERRUN] message exists

        If an error occured because the message was not present on the
        server, the extended response code should be:

            -ERR [NON-EXISTENT] no such message

        For all other type of errors, a simple "-ERR" is sufficient.

4. The CAPA Tag

        The extended RETR command introduces a new CAPA tag (RFC 2449).
        This tag tells the client that the server is able to continue
        interrupted RETR commands. Its implementation is mandatory if
        the server supports this extension and the CAPA command.

        CAPA tag:
            EXT-RETR

        Arguments:
            none

        Added commands:
            n/a

        Standard commands affected:
            RETR

        Announced states / possible differences:
            both / no

        Commands valid in states:
            TRANSACTION

        Specification reference:
            [POP3, this document]

        Discussion:
            The extended RETR capability indicates that the server is
            capable of excepting a second argument to the RETR command
            in order to resume a previously interrupted RETR command.

A. Rundegren                                                    [page 5]
-- LINE -- 290 --
INTERNET-DRAFT                                         26 February, 2000

5. Security Considerations

        This extension is believed not to introduce additional security
        issues. Use of the extended RETR command sends messages in the
        clear over the network, unless an encryption layer has been
        previously negotiated.

6. Acknowledgements

        The extension introduced in this memo is based on the RETR
        command as specified in RFC 1939. I would like to thank
        R. Gellens, C. Newman and L. Lundblade for their comments
        and suggestions during the drafting of this document.

7. References

        1)  Post Office Protocol - Version 3 (RFC 1939).
            J. Myers, M. Rose.

        2)  Multipurpose Internet Mail Extensions, Part One (RFC 2045).
            N. Freed, N. Borenstein.

        3)  POP3 Extension Mechanism (RFC 2449).
            R. Gellens, C. Newman, L. Lundblade.

8. Author's Address

        Anders Rundegren
        Skimmelgatan 1
        212 35 Malmo
        Sweden

        email: [email-masked]

A. Rundegren                                                    [page 6]
-- LINE -- 348 --

--------------C761A1235DE3A859830F0320--

Date: Fri, 25 Feb 2000 20:24:26 -0800
To: [email-masked]
From: Randall Gellens <[email-masked]>
Subject: The SYS and AUTH Response Codes
Sender: [email-masked]

I just sent in a draft defining the SYS and AUTH response codes and 
the AUTH-RESP-CODE capability.   Before it shows up in the I-D 
directories, it can be obtained from 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-pop-err-00.txt>.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Advertising is 85 percent confusion and 15 percent commission.
                                                 --Fred Allen

From: Solar Designer <[email-masked]>
Subject: Re: Extended RETR - version 2
To: [email-masked] (Jochen Topf)
Date: Sun, 20 Feb 2000 16:31:07 +0300 (MSK)
Cc: [email-masked]
Sender: [email-masked]

> 
> I have no historical data at the moment to tell us, whether the numbers are
> getting better or worse. But I am quite sure that they are getting better.
> From my view there is no need for the extended RETR, but maybe others see
> different statistics here.

It's not obvious to me whether we need this extension or not, but I
don't see any harm in it either, and I do hear some user complaints
about wasting their money downloading a large message from scratch.
(Fortunately, my job doesn't involve user support.)

If someone has volunteered to define the extension anyway, it seems
better to ensure there're no hidden uncertainties left.

> Would be nice to have some more statistics from other parts of the world
> and other types of ISPs.

The statistics posted here so far only show the overhead caused by
re-transmissions as seen by ISP's.  It's likely that some users are
getting disconnected more often than some of the others.  While we
see under 1% POP3 sessions interrupted during a message transfer on
average for an ISP, there may well be over 10% such sessions for a
given user.  If that user once receives a huge message, a disconnect
during the transfer of that message is even more likely.

If we look at the statistics, it's also important to note that most
of the POP3 sessions are to empty mailboxes, and thus didn't have a
real chance to crash.  It makes more sense to count POP3 sessions
that transferred at least one message and use that when calculating
the percentage of crashed transfers.

> Another thing: The offset and bytestuffing issue is really the killer here.
> If this is not documented properly and there is only one implementation
> doing it differently the whole thing is worthless, because it will lead

I think there should be at least two independent implementations
before this becomes an RFC (if ever).  Then new implementations could
be checked against those two.

> to corrupted email. I suggest adding a paragraph mentioning that the offset
> is calculated by using CRLF line endings, just to make sure.

I think the reference to section 11 of RFC 1939 covers that.

> And another caveat: The RFC should mention that there is no guarantee that
> the message numbering stays the same from one POP connection to the next.

Agreed.

> And a third caveat: There might be server implementations that send some
> error message when dieing unexpectedly. For instance when the server gets
> a signal it might send a "-ERR I am out of here" message or similar. If this
> can ever happen while the server is sending the mail, the client would

That would violate RFC 1939 anyway.

Signed,
Solar Designer

Date: Sat, 19 Feb 2000 10:48:14 +0100
From: Jochen Topf <[email-masked]>
To: [email-masked]
Subject: Re: Extended RETR - version 2
Sender: [email-masked]

Anders Rundegren <[email-masked]> wrote:
:> >Where in the world is the above statistics from?

: Correction: the above question was meant to be referring to geographic
: location (please excuse my malformed translation).

Germany.

I have no historical data at the moment to tell us, whether the numbers are
getting better or worse. But I am quite sure that they are getting better.
>From my view there is no need for the extended RETR, but maybe others see
different statistics here.

Would be nice to have some more statistics from other parts of the world
and other types of ISPs.

Another thing: The offset and bytestuffing issue is really the killer here.
If this is not documented properly and there is only one implementation
doing it differently the whole thing is worthless, because it will lead
to corrupted email. I suggest adding a paragraph mentioning that the offset
is calculated by using CRLF line endings, just to make sure.

And another caveat: The RFC should mention that there is no guarantee that
the message numbering stays the same from one POP connection to the next.
For instance, if there is another process accessing the same mailbox at the
same time it might have deleted a message. So if the client wants to use
extended RETR it MUST use UIDL to find the right message number, which further
complicates the matter and is an easy trap for the implementor to fall into.

And a third caveat: There might be server implementations that send some
error message when dieing unexpectedly. For instance when the server gets
a signal it might send a "-ERR I am out of here" message or similar. If this
can ever happen while the server is sending the mail, the client would
probabely not notice this, take this message as part of the mail, and ask
for a retransmit with the wrong offset resulting in a corrupted mail. I don't
know of any server with this behaviour, but it is a possible szenario. So
when the author of the server implements the extended RETR, he has to make
sure, that this can never happen with his server.

Jochen
-- 
Jochen Topf - [email-masked] - http://www.remote.org/jochen/

From: Solar Designer <[email-masked]>
Subject: Re: Extended RETR - version 2
To: [email-masked] (Anders Rundegren)
Date: Sat, 19 Feb 2000 10:33:58 +0300 (MSK)
Cc: [email-masked]
Sender: [email-masked]

> I'm attaching the revised version of the draft (I haven't uploaded it to
> IETF since I thought you might have more comments, which I hope). I've

Below are my proposed changes to the draft.  If the reasoning behind
some is non-obvious, just ask. :-)

--- extended-retr-draft-v2.txt	Sat Feb 19 08:05:15 2000
+++ extended-retr-draft-v2s.txt	Sat Feb 19 10:25:29 2000
@@ -81,23 +81,19 @@
         parts, more commonly known as attachments. These attachments
         are often much larger than the readable textparts.
 
-        Many clients connecting to an ISP, or even the ISP itself,
-        have their modems setup in a way that doesn't handle a noisy
-        phoneline very well. When line-quality isn't high enough, this
-        often results in a premature disconnect.
-
-        Using commands currently found in the the POP3 protocol, these
-        messages will have to be re-downloaded from scratch taking a
-        lot of unnecessary bandwidth from the ISP, not to mention
-        resources within the system itself.
+        Currently, POP3 requires messages to be re-downloaded from
+        scratch should there be a premature disconnect during a
+        message transfer. Such disconnects happen for various reasons,
+        including poor phone line quality, modem setup, network packet
+        loss, and temporary network problems.
 
         By extending the RETR command, the download-and-delete model
         that characterizes the POP3 protocol becomes more complete in
-        handling todays requirements of email. Consider the benefits:
+        handling todays requirements of email. The benefits are:
 
-        a) Less resources used on the server-side
-        b) Less time used online for the client
-        c) Less consumption of bandwidth
+        a) Less time used online for the client
+        b) Less consumption of bandwidth
+        c) Less resources used on the server side

 2. The Extended RETR Command
@@ -109,7 +105,8 @@
 
         The RETR command (as specified by RFC 1939), takes one argument
         which specifies the message to be retrieved by the client. The
-        possible responses are "+OK" and "-ERR", without the quotes.
+        allowed responses are "+OK" and "-ERR", without the quotes,
+        possibly followed by a space and a comment.

 A. Rundegren                                                    [page 2]
@@ -121,11 +118,14 @@
 
         The second argument to the extended RETR command specifies the
         offset in the message of where the transfer should start. The
-        offset is given in octets. All data that is transferred with a
-        standard RETR command should be accounted for. This means that
-        top headers, followed by messageparts will be transferred, in
-        that order. As specified by RFC 1939, responses can not be more
-        than 512 characters long, including the terminating CRLF.
+        offset is given in octets, and is calculated the way message
+        sizes are, as defined in section 11 of RFC 1939. In order to
+        ensure a valid multi-line response, with all lines terminated
+        with a CRLF pair (and never a single LF), the offset MUST NOT
+        point into the middle of a CRLF pair.
+
+        As specified by RFC 1939, responses can not be more than 512
+        characters long, including the terminating CRLF.
 
         The syntax of the extended RETR command would be:
 
@@ -140,7 +140,7 @@
 
         Description:
             If the POP3 server issues a positive response, then the
-            response given is multi-line. After the intial +OK, the
+            response given is multi-line. After the initial +OK, the
             POP3 server sends the message corresponding to the given
             message-number, starting from the offset provided, being
             careful to byte-stuff the termination character (as with
@@ -154,15 +154,24 @@
             C: RETR 1 345628
             S: +OK 1253512 octets
             S: <the POP3 server sends the message here, starting with
-                the 345628th octet of the message>
+                the 345629th octet of the message>
+            S: .
+               ...
+            C: RETR 1 1599140
+            S: +OK 0 octets
             S: .
 
+        Offsets equal to the message size are allowed, as shown in the
+        last example. This can be used by a client to check if it has
+        a complete message and continue the transfer if not, with one
+        command.
+
         If the specified offset is beyond the size of the message being
-        retrieved, or in any other way erroneous in its syntax, the
-        server should respond with a negative response code. This would
-        however not distinguish the type of error. In the following
-        section you can read about the use of extended response codes
-        to clarify errormessages.
+        retrieved, points into the middle of a CRLF pair, or is in any
+        way erroneous in its syntax, the server should respond with a
+        negative response code. This would however not distinguish the
+        type of error. In the following section you can read about the
+        use of extended response codes to clarify error messages.

 
@@ -209,10 +218,10 @@
         to work. However, this memo specifies the extended response
         codes to be used if implemented.
 
-        If the specified offset is beyond the size of the message being
-        retrieved, the extended response code should be:
+        If the specified offset is in any way erroneous, the extended
+        response code should be:
 
-            -ERR [OFFSET-OVERRUN] message exists
+            -ERR [INVALID-OFFSET] message exists
 
         If an error occured because the message was not present on the
         server, the extended response code should be:
@@ -237,13 +246,15 @@
 
         The extended RETR command introduces a new CAPA tag (RFC 2449).
         This tag tells the client that the server is able to continue
-        interrupted RETR commands. Its implementation is mandatory.
+        interrupted RETR commands. Its implementation is mandatory if
+        the server supports this extension and the CAPA command.
 
         CAPA tag:
             EXT-RETR
 
         Arguments:
-            none
+            one of ALWAYS, YES or NO in TRANSACTION state, an optional
+            ALWAYS specifier in AUTHORIZATION state
 
         Added commands:
             n/a
@@ -252,7 +263,7 @@
             RETR
 
         Announced states / possible differences:
-            both / no
+            both / yes
 
         Commands valid in states:
             TRANSACTION
@@ -262,9 +273,21 @@
 
         Discussion:
             The extended RETR capability indicates that the server is
-            capable of excepting a second argument to the RETR command
+            capable of accepting a second argument to the RETR command
             in order to resume a previously interrupted RETR command.
 
+            If a server supports this extension for at least some of
+            the maildrops, it MUST announce the extension while in
+            the AUTHORIZATION state. If the extension is supported
+            for all of the maildrops, the server SHOULD also include
+            the ALWAYS specifier, so that clients don't have to
+            reissue the CAPA command after entering the TRANSACTION
+            state. If announced in the AUTHORIZATION state, the
+            server MUST announce the capability in the TRANSACTION
+            state as well. If the extension is not supported for the
+            particular maildrop, the argument should be set to NO.
+            Otherwise, the argument can be set to either ALWAYS or
+            YES, which should be treated the same by clients.

 
@@ -293,8 +316,10 @@
 
 5. Security Considerations
 
-        Use of the extended RETR command sends messages in the clear
-        over the network.
+        This extension is believed not to introduce additional
+        security issues. Use of the RETR command sends messages in
+        the clear over the network, unless an encryption layer has
+        been previously negotiated.

 6. Acknowledgements

Date: Sat, 19 Feb 2000 06:26:30 +0100
From: Anders Rundegren <[email-masked]>
To: Mike Gahrns <[email-masked]>
CC: [email-masked]
Subject: Re: Extended RETR - version 2
Sender: [email-masked]

> >Where in the world is the above statistics from?

Correction: the above question was meant to be referring to geographic
location (please excuse my malformed translation).

> Not necessarily.  I think a strong case can be made that while the volume
> may increase, the quality is also increasing as well.  As I mentioned
> earlier, I have no objections to this extension from a technical point of
> view, just pointing out that before we extend POP, lets make sure that we
> are solving a real problem.  A large part of POP's success is due to its
> simplicity.

Exactly what are you referring to by "quality", please specify.

// Anders

From: "Mike Gahrns" <[email-masked]>
To: "Anders Rundegren" <[email-masked]>, <[email-masked]>
Subject: Re: Extended RETR - version 2
Date: Fri, 18 Feb 2000 10:43:45 -0800
Sender: [email-masked]

Re: Extended RETR - version 2Anders writes:
>> A quick grep over the logfiles of a large ISP (>1 Mio POP accesses a day)
>> tells me that less then 0.1% of the connections die while sending
something
>> to the client. There are another about 0.2% where the client didn't
properly
>> close the connection with QUIT, but they happen while waiting for the
next
>> command.
>Where in the world is the above statistics from?
>A few years/months from now this number is probably going to increase,
>given the nature of the Internet, i.e. everything escalates beyond
>everyone's imagination. :)

Not necessarily.  I think a strong case can be made that while the volume
may increase, the quality is also increasing as well.  As I mentioned
earlier, I have no objections to this extension from a technical point of
view, just pointing out that before we extend POP, lets make sure that we
are solving a real problem.  A large part of POP's success is due to its
simplicity.

From: Solar Designer <[email-masked]>
Subject: Re: Extended RETR - version 2
To: [email-masked] (Jochen Topf)
Date: Thu, 17 Feb 2000 14:48:17 +0300 (MSK)
Cc: [email-masked]
Sender: [email-masked]

> 
> : Sorry, but I'm a bit doubtful about this extension. How serious
> : is the problem with a "noisy phoneline" really ? I'm just a naive
> : user (i.e not running an ISP) but I have never experienced this
> : problem, neither when living in Sweden nor in Australia.
> 
> A quick grep over the logfiles of a large ISP (>1 Mio POP accesses a day)
> tells me that less then 0.1% of the connections die while sending something
> to the client. There are another about 0.2% where the client didn't properly
> close the connection with QUIT, but they happen while waiting for the next
> command.

Here's some data from a small ISP in Moscow, Russia (the log is for
about a day):

cannabis!root:~# grep -c 'Authentication passed' /var/log/popper
28042
cannabis!root:~# grep -c 'Authentication failed' /var/log/popper
699
cannabis!root:~# grep -c 'Session crashed' /var/log/popper
198
cannabis!root:~# grep -c 'Connection timed out' /var/log/popper
46
cannabis!root:~# grep -c 'Another MUA active' /var/log/popper
7

About 0.7% of the sessions crashed while sending something to the
client, and almost 1% didn't terminate with a QUIT.  There may also
be a few cases where people don't interrupt a slow connection (to
re-dial or re-connect over a network) because they know they wouldn't
be able to continue the transfer of a large message.

I'm not sure if the problem is worth an extension to the protocol,
but as long as this is an optional protocol feature and there're not
going to be any clients that will refuse to support older servers, I
don't see any problem with defining the extension.

Signed,
Solar Designer

Date: Thu, 17 Feb 2000 12:48:41 +0100
From: Anders Rundegren <[email-masked]>
To: [email-masked]
Subject: Re: Extended RETR - version 2
Sender: [email-masked]

> A quick grep over the logfiles of a large ISP (>1 Mio POP accesses a day)
> tells me that less then 0.1% of the connections die while sending something
> to the client. There are another about 0.2% where the client didn't properly
> close the connection with QUIT, but they happen while waiting for the next
> command.

Where in the world is the above statistics from?

A few years/months from now this number is probably going to increase,
given the nature of the Internet, i.e. everything escalates beyond
everyone's imagination. :)

// Anders

Date: Thu, 17 Feb 2000 11:56:00 +0100
From: Jochen Topf <[email-masked]>
To: [email-masked]
Subject: Re: Extended RETR - version 2
Sender: [email-masked]

Torbjorn Tornkvist <[email-masked]> wrote:
:>  Many clients connecting to an ISP, or even the ISP itself,
:>  have their modems setup in a way that doesn't handle a noisy
:>  phoneline very well. When line-quality isn't high enough, this
:>  often results in a premature disconnect.

: Sorry, but I'm a bit doubtful about this extension. How serious
: is the problem with a "noisy phoneline" really ? I'm just a naive
: user (i.e not running an ISP) but I have never experienced this
: problem, neither when living in Sweden nor in Australia.

A quick grep over the logfiles of a large ISP (>1 Mio POP accesses a day)
tells me that less then 0.1% of the connections die while sending something
to the client. There are another about 0.2% where the client didn't properly
close the connection with QUIT, but they happen while waiting for the next
command.

These numbers are very rough, just to give a sense of the order of magnitude.
They will probabely vary between different ISPs depending on types of users
etc.

Jochen
-- 
Jochen Topf - [email-masked] - http://www.remote.org/jochen/

To: Anders Rundegren <[email-masked]>
Cc: [email-masked], Alexey Melnikov <[email-masked]>, Solar Designer <[email-masked]>
Subject: Re: Extended RETR - version 2
From: Torbjorn Tornkvist <[email-masked]>
Date: 17 Feb 2000 08:45:01 +0100
Lines: 16
Sender: [email-masked]

>  Many clients connecting to an ISP, or even the ISP itself,
>  have their modems setup in a way that doesn't handle a noisy
>  phoneline very well. When line-quality isn't high enough, this
>  often results in a premature disconnect.

Sorry, but I'm a bit doubtful about this extension. How serious
is the problem with a "noisy phoneline" really ? I'm just a naive
user (i.e not running an ISP) but I have never experienced this
problem, neither when living in Sweden nor in Australia.

Cheers /Tobbe
--
Torbjörn Törnkvist , tel: +46 8 692 22 15 , fax: +46 8 654 70 71
Bluetail AB , Hantverkargatan 78 , SE-112 38 Stockholm , Sweden            
Email: [email-masked] , Web: http://www.bluetail.com/~tobbe

Date: Thu, 17 Feb 2000 01:26:53 +0100
From: Anders Rundegren <[email-masked]>
To: [email-masked]
CC: Alexey Melnikov <[email-masked]>, Solar Designer <[email-masked]>
Subject: Extended RETR - version 2
Sender: [email-masked]

This is a multi-part message in MIME format.
--------------BB1DF73CF35D8C026969FE38
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

I'm attaching the revised version of the draft (I haven't uploaded it to
IETF since I thought you might have more comments, which I hope). I've
done some changes based on your feedback and the one thing that struck
me was Alexey's comment below:

> BTW, ESMTP Checkpoint/Restart extension (RFC 1845) that does exactly the
> same thing for SMTP (and suffers from the same dot stuffing problem)
> says:
> 
>    Any octets added by any SMTP data-stuffing algorithm do
>    not count as part of this offset. In the case of data transferred
>    with the DATA command the offset must also correspond to the
>    beginning of a line.

This is a very good point. I've added a section explaining bytestuffing,
in which I also added a sentence that clarifies that offsets are
calculated "unstuffed".

// Anders
--------------BB1DF73CF35D8C026969FE38
Content-Type: text/plain; charset=us-ascii;
 name="extended-retr-draft-v2.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="extended-retr-draft-v2.txt"

INTERNET-DRAFT                                            INTERNET-DRAFT
                                                            A. Rundegren
                                                       17 February, 2000

                  Extended RETR - Extension to POP3
                    <draft-rundegren-pop3-01.txt>           

Status of this Memo

        This document is an Internet-Draft and is in full
        conformance with all provisions of Section 10 of RFC2026.

        Internet-Drafts are working documents of the Internet
        Engineering Task Force (IETF), its areas, and its working
        groups. Note that other groups may also distribute working
        documents as Internet-Drafts.

        Internet-Drafts are draft documents valid for a maximum of
        six months and may be updated, replaced, or obsoleted by
        other documents at any time. It is inappropriate to use
        Internet-Drafts as reference material or to cite them other
        than as "work in progress."

        The list of current Internet-Drafts can be accessed at
        http://www.ietf.org/ietf/1id-abstracts.txt

        The list of Internet-Draft Shadow Directories can be
        accessed at http://www.ietf.org/shadow.html.

        Distribution of this document is unlimited. Please send
        comments to the author.

Abstract

        This memo introduces an extension to the RETR command in the
        POP3 protocol. The extension makes it possible for an
        interrupted RETR command to be continued. This is
        particularly useful when downloading large messages over
        unstable connections.

A. Rundegren                                                    [page 1]
-- LINE -- 58 --
INTERNET-DRAFT                                         17 February, 2000

Table of Contents
        1.  Introduction  ...........................................  2
        2.  The Extended RETR Command  ..............................  2
          2.1  RFC 1939 Specification  ..............................  2
          2.2  The Offset Argument  .................................  3
	  2.3  Bytestuffing  ........................................  4
        3.  Extended Response Codes  ................................  4
        4.  The CAPA Tag  ...........................................  5
        5.  Security Considerations  ................................  6
        6.  Acknowledgements  .......................................  6
        7.  References  .............................................  6
        8.  Author's Address  .......................................  6

1. Introduction

        When email was first developed it was intended to transfer
        readable text messages. Since the introduction of MIME (RFC
        2045) it has become more common to transfer other message
        parts, more commonly known as attachments. These attachments
        are often much larger than the readable textparts.

        Many clients connecting to an ISP, or even the ISP itself,
        have their modems setup in a way that doesn't handle a noisy
        phoneline very well. When line-quality isn't high enough, this
        often results in a premature disconnect.

        Using commands currently found in the the POP3 protocol, these
        messages will have to be re-downloaded from scratch taking a
        lot of unnecessary bandwidth from the ISP, not to mention
        resources within the system itself.

        By extending the RETR command, the download-and-delete model
        that characterizes the POP3 protocol becomes more complete in
        handling todays requirements of email. Consider the benefits:

        a) Less resources used on the server-side
        b) Less time used online for the client
        c) Less consumption of bandwidth

2. The Extended RETR Command

        This section will give a detailed view on how the second
        argument of the extended RETR command should be implemented.

2.1 RFC 1939 Specification

        The RETR command (as specified by RFC 1939), takes one argument
        which specifies the message to be retrieved by the client. The
        possible responses are "+OK" and "-ERR", without the quotes.

A. Rundegren                                                    [page 2]
-- LINE -- 116 --
INTERNET-DRAFT                                         17 February, 2000

2.2 The Offset Argument

        The second argument to the extended RETR command specifies the
        offset in the message of where the transfer should start. The
        offset is given in octets. All data that is transferred with a
        standard RETR command should be accounted for. This means that
        top headers, followed by messageparts will be transferred, in
        that order. As specified by RFC 1939, responses can not be more
        than 512 characters long, including the terminating CRLF.

        The syntax of the extended RETR command would be:

            RETR msg offset

        Arguments:
            A message-number (required) which may NOT refer to a
            message marked as deleted. An offset (optional) of where
            to start the transfer, specified in octets. If the offset
            argument is omitted, the message will be sent as specified
            by the RETR command in RFC 1939.

        Description:
            If the POP3 server issues a positive response, then the
            response given is multi-line. After the intial +OK, the
            POP3 server sends the message corresponding to the given
            message-number, starting from the offset provided, being
            careful to byte-stuff the termination character (as with
            all multi-line responses).            

        Possible Responses:
            +OK message follows
            -ERR no such message

        Examples:
            C: RETR 1 345628
            S: +OK 1253512 octets
            S: <the POP3 server sends the message here, starting with
                the 345628th octet of the message>
            S: .

        If the specified offset is beyond the size of the message being
        retrieved, or in any other way erroneous in its syntax, the
        server should respond with a negative response code. This would
        however not distinguish the type of error. In the following
        section you can read about the use of extended response codes
        to clarify errormessages.

A. Rundegren                                                    [page 3]
-- LINE -- 174 --
INTERNET-DRAFT                                         17 February, 2000

2.3 Bytestuffing

        This section explains how bytestuffing should be performed when
        resuming a previously interrupted transfer. When data is to be
        transferred from the server to the client, the server should
        be careful to byte-stuff all data. If the previous transfer
        was interrupted in the middle of a line, where the remaining
        part of the line starts with a '.' character, this means that
        data, that would normally not be bytestuffed (since it's not in
        the beginning of a line), must be byte-stuffed. Any octets added
        by any data-stuffing algorithm do not count as part of the offset
        used by the Extended RETR command.

        Examples:
            Message number is 1.
            Message size is 217,743 octets.
            Offset 113,435 starts with a '.'.

            Lets assume, that when transferring the above message, an
            interrupt occurs after exactly 113,435 octets. When
            resuming the transfer, the client executes "RETR 1 113435".
            The server then byte-stuffs the data before transferring it
            to the client, who then byte-destuffs the data before
            appending it to the previously transferred data.

3. Extended Response Codes

        The extended RETR command does not require the implementation
        of extended response codes (RFC 2449) to be present in order
        to work. However, this memo specifies the extended response
        codes to be used if implemented.

        If the specified offset is beyond the size of the message being
        retrieved, the extended response code should be:

            -ERR [OFFSET-OVERRUN] message exists

        If an error occured because the message was not present on the
        server, the extended response code should be:

            -ERR [NON-EXISTENT] no such message

        For all other type of errors, a simple "-ERR" is sufficient.

A. Rundegren                                                    [page 4]
-- LINE -- 232 --
INTERNET-DRAFT                                         17 February, 2000

4. The CAPA Tag

        The extended RETR command introduces a new CAPA tag (RFC 2449).
        This tag tells the client that the server is able to continue
        interrupted RETR commands. Its implementation is mandatory.

        CAPA tag:
            EXT-RETR

        Arguments:
            none

        Added commands:
            n/a

        Standard commands affected:
            RETR

        Announced states / possible differences:
            both / no

        Commands valid in states:
            TRANSACTION

        Specification reference:
            [POP3, this document]

        Discussion:
            The extended RETR capability indicates that the server is
            capable of excepting a second argument to the RETR command
            in order to resume a previously interrupted RETR command.

A. Rundegren                                                    [page 5]
-- LINE -- 290 --
INTERNET-DRAFT                                         17 February, 2000

5. Security Considerations

        Use of the extended RETR command sends messages in the clear
        over the network.

6. Acknowledgements

        The extension introduced in this memo is based on the RETR
        command as specified in RFC 1939. I would like to thank
        R. Gellens, C. Newman and L. Lundblade for their comments
        and suggestions during the drafting of this document.

7. References

        1)  Post Office Protocol - Version 3 (RFC 1939).
            J. Myers, M. Rose.

        2)  Multipurpose Internet Mail Extensions, Part One (RFC 2045).
            N. Freed, N. Borenstein.

        3)  POP3 Extension Mechanism (RFC 2449).
            R. Gellens, C. Newman, L. Lundblade.

8. Author's Address

        Anders Rundegren
        Skimmelgatan 1
        212 35 Malmo
        Sweden

        email: [email-masked]

A. Rundegren                                                    [page 6]
-- LINE -- 348 --

--------------BB1DF73CF35D8C026969FE38--

Date: Mon, 14 Feb 2000 11:52:24 +0000
From: Alexey Melnikov <[email-masked]>
To: Mike Gahrns <[email-masked]>
CC: [email-masked]
Subject: Re: Extended RETR - Draft posted
Sender: [email-masked]

> I don't have any problems
> with the concept proposed in the draft, and it is clearly a nicer way to
> implement what POP originally did, but from my own personal experience, the
> problem of dropping a download partway through a message is relatively rare.
> If this hold true for many others, I suspect it may be hard to garner wide
> spread support for any POP extension.
> 
> Perhaps line qualities in different parts of the world may make this a wider
> spread problem for others.

This is very true for some country I use to live in :-).

Alexey

From: "Mike Gahrns" <[email-masked]>
To: "Anders Rundegren" <[email-masked]>, <[email-masked]>
Subject: Re: Extended RETR - Draft posted
Date: Tue, 8 Feb 2000 11:16:02 -0800
Sender: [email-masked]

Extended RETR - Draft postedIn general, there has been a strong opposition
to extending POP, in an attempt to keep it as simple as possible.  I think
to get wide spread support for a POP extension proposal such as this, it
would need to solve a very wide spread problem.  I don't have any problems
with the concept proposed in the draft, and it is clearly a nicer way to
implement what POP originally did, but from my own personal experience, the
problem of dropping a download partway through a message is relatively rare.
If this hold true for many others, I suspect it may be hard to garner wide
spread support for any POP extension.

Perhaps line qualities in different parts of the world may make this a wider
spread problem for others.
----- Original Message -----
From: Anders Rundegren
To: [email-masked]
Sent: Saturday, January 29, 2000 4:34 AM
Subject: Extended RETR - Draft posted

The draft for the extended RETR command has now been posted. Feedback
is, as always, very appreciated. I'm not sure about the section dealing
with extended response codes, it definitely needs some more work. Since
English is not my native language I ask you to look over this too.
http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt
Thanks!
// Anders Rundegren

Date: Sun, 06 Feb 2000 01:18:24 +0100
From: Anders Rundegren <[email-masked]>
To: Solar Designer <[email-masked]>, Alexey Melnikov <[email-masked]>
CC: [email-masked]
Subject: Re: Extended RETR - Draft posted
Sender: [email-masked]

Is there anyone having additional comments except what has already been
said?

I will revise the draft based on the feedback I've been getting so far.
However, this will not happen until next weekend due to extreme workload
at work.

> > Example:
> >
> > Message number is '1'.
> > Message size is 217,743 bytes.
> > Offset 113,435 starts with a '.'.
> >
> > Lets assume our transfer got interrupted after 113,434 bytes. Now, the
> 
> To illustrate the case of having to byte-stuff, you'd want the
> transfer to be interrupted after 113,435 bytes.

Yes, true, sorry about that...

> > > "       a) Less consumption of bandwidth
> > >         b) Less resources used on the server-side
> > >         c) Less time used online for the client"

I didn't put them in any specific order though.

> To allow for a simple seek, we would need to define that offsets are
> into the message as stored in a BSD-style mailbox.

I guess I should explore other systems. ;)

// Anders

Date: Fri, 4 Feb 2000 16:53:06 +0000
To: [email-masked]
From: Paul Overell <[email-masked]>
Subject: RFC2449 syntax
Sender: [email-masked]

Some comments on the syntax given in RFC2449.

In RFC2449 para 3.  General Command and Response Grammar

>
>     greeting     =  "+OK" [resp-code] *gchar [timestamp] *gchar CRLF                          
>

This fails to require a space after the +OK, I suggest

      greeting     =  "+OK" [SP [resp-code] *gchar [timestamp] *gchar] CRLF                          

>
>     text         =  *schar / resp-code *CHAR 
>

This syntax is correct if the server supports RESP-CODES but is otherwise overly
restrictive, I suggest either

     text         =  (*schar / resp-code *CHAR) /       ; if RESP-CODES supported
                     *CHAR                              ; otherwise

More controversial:

>
>      dot-stuffed  =  *CHAR CRLF                  ;must be dot-stuffed
>

This prohibits downloading messages containing 8 bit characters.  RFC1939 does not
explicitly outlaw 8bit characters, however, it does say messages "are assumed to
conform to ... [RFC822]", which could be interpreted as an implicitly outlawing
8bit characters.  

As ESMTP permits the transmission of 8bit MIME messages (and it is not unknown
using SMTP) are we to prevent users from collecting them via POP3?  I suggest:

        dot-stuffed = *OCTET CRLF               ; must be dot-stuffed

Regards
-- 
Paul Overell                                             T U R N P I K E

From: Solar Designer <[email-masked]>
Subject: Re: Extended RETR - Draft posted
To: [email-masked] (Alexey Melnikov)
Date: Wed, 2 Feb 2000 13:55:57 +0300 (MSK)
Cc: [email-masked]
Sender: [email-masked]

> 
> > Yes, it seems we need that.  There's one more issue that you'd need
> > to decide on -- are we calculating CRLF's the way they're transferred
> > over POP3 (would be more essential for the protocol), or the way line
> > separators are typically stored on a UNIX-based server (to allow for
> > a simple seek).
> 
> There are servers that 
> 1). construct message on the flight, storing it in some different
> format.
> 2). store headers separately from body.
> 3). store message with CRLF even if they run on Unix
> 4). store message in bytestuffed form (as received from SMTP)
> 
> Whatever rule you choose for calculation of offset you will not satisfy
> ALL implementations, 

Yes, indeed.

> so there is no point doing that.

Not necessarily: it might make sense to make life easier for those
servers for which it would be harder to switch to a more efficient
(for our POP3 extension) mailbox format.

I agree that this is probably not worth the extra complexity this
would bring into the specification of such a POP3 extension.

> BTW, ESMTP Checkpoint/Restart extension (RFC 1845) that does exactly the
> same thing for SMTP (and suffers from the same dot stuffing problem)
> says:
> 
>    Any octets added by any SMTP data-stuffing algorithm do
>    not count as part of this offset. In the case of data transferred
>    with the DATA command the offset must also correspond to the
>    beginning of a line.
> 
> In checkpoint/restart offset starts from 0.
> I would suggest to use the same for consistency.

> Would it be easier to use TOP-like command? Something like
> 
>    TOP <msgnum> <lines> <start_from_line>

TOP would be more consistent with current POP3, as we already have
line number arguments, but don't have octet ones, yet.

However, as you have pointed out, it would be inconsistent with the
SMTP extension.

Signed,
Solar Designer