Subject: [TOOLS-DEVELOPMENT] Priority Decision Needed: Non-ASCII characters in email addresses

From: Russ Housley <[email-masked]>
Date: Wed, 8 Jun 2016 12:52:42 -0400

Jari:

>> We had an interesting error report this morning from an IETF
>> community member who used an email address in the ngāpukapuka.nz
>> domain to register for IETF 96.  It turns out that smtplib in Python
>> 2.7.8 is not happy with non-ASCII characters in addresses.
>> 
>> I have also checked Mailman and discovered that it is also unable to
>> digest non-ASCII characters in email addresses.  Having discussed the
>> matter with Glen, I wanted to bring it to your attention as well.
> 
> Yes.  It looks as if smtplib has been fixed for python 3.5, but I suspect
> that a backport to 2.7 is unlikely:
> 
> http://bugs.python.org/issue22027
> 
> So we'll either have to do all the work of building the appropriate EAI
> strings in the application, and finagle them into the message, or we'll
> have to postpone a fix until we move up to python 3.5 or higher.  I'm
> adding python 3.5 upgrade to the PLAN file.  The django framework can run on
> python 3.x, but we haven't done any work to make our own code 3.5-friendly.
> 
> I suspect this step (going to 3.5) will be a substantial piece or work.
> 
> If EAI support is sufficiently urgent (i.e., can't wait for python 3.x
> conversion), we'll have to code it up ourselves, without lib support.

So far, we have had one complaint.  I expect more.

As you can see, Henrik has identified a way forward.  Sadly, it will require real work.

Please advise on the priority of this effort.  The IETF has done a lot of work to make internationalized domain names happen.  Now, we need to eat our dog food.  Do we need to eat that meal in the next few weeks or the next few months?

Russ

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Wed, 8 Jun 2016 09:57:32 -0700

On 6/8/16 9:52 AM, Russ Housley wrote:
> Jari: > >>> We had an interesting error report this morning from an IETF >>>
community member who used an email address in the ngāpukapuka.nz >>>
domain to register for IETF 96.  It turns out that smtplib in >>> Python
2.7.8 is not happy with non-ASCII characters in >>> addresses. >>> >>> I
have also checked Mailman and discovered that it is also unable >>> to
digest non-ASCII characters in email addresses.  Having >>> discussed
the matter with Glen, I wanted to bring it to your >>> attention as
well. >> >> Yes.  It looks as if smtplib has been fixed for python 3.5,
but I >> suspect that a backport to 2.7 is unlikely: >> >>
http://bugs.python.org/issue22027 >> >> So we'll either have to do all
the work of building the appropriate >> EAI strings in the application,
and finagle them into the message, >> or we'll have to postpone a fix
until we move up to python 3.5 or >> higher.  I'm adding python 3.5
upgrade to the PLAN file.  The >> django framework can run on python
3.x, but we haven't done any >> work to make our own code 3.5-friendly.
>> >> I suspect this step (going to 3.5) will be a substantial piece or
>> work. >> >> If EAI support is sufficiently urgent (i.e., can't wait
for python >> 3.x conversion), we'll have to code it up ourselves,
without lib >> support. > > So far, we have had one complaint.  I expect
more. > > As you can see, Henrik has identified a way forward.  Sadly,
it will > require real work. > > Please advise on the priority of this
effort.  The IETF has done a > lot of work to make internationalized
domain names happen.  Now, we > need to eat our dog food.  Do we need to
eat that meal in the next > few weeks or the next few months? > > Russ

I'd actually put this before format work, if only because I think the
infrastructure needs to be in place to handle nonASCII characters in all
places.

-Heather

---

From: Jari Arkko <[email-masked]>
Date: Wed, 8 Jun 2016 19:57:33 +0300

Thanks guys for this summary.

> Please advise on the priority of this effort.  The IETF has done a lot of work to make internationalized domain names happen.  Now, we need to eat our dog food.  Do we need to eat that meal in the…

My gut feeling is to put this on the “few months” bucket. There should be very few issues that demand bug fixes in matter of weeks. Now, if that few months starts to be more like half a year or a year, I’d reconsider though.

jari

---

From: Russ Housley <[email-masked]>
Date: Wed, 8 Jun 2016 13:34:28 -0400

Heather:

> >>> domain to register for IETF 96.  It turns out that smtplib in
> >>> Python 2.7.8 is not happy with non-ASCII characters in
> >>> addresses.
> >>> 
> >>> I have also checked Mailman and discovered that it is also unable
> >>> to digest non-ASCII characters in email addresses.  Having
> >>> discussed the matter with Glen, I wanted to bring it to your
> >>> attention as well.
> >> 
> >> Yes.  It looks as if smtplib has been fixed for python 3.5, but I
> >> suspect that a backport to 2.7 is unlikely:
> >> 
> >> So we'll either have to do all the work of building the appropriate
> >> EAI strings in the application, and finagle them into the message,
> >> or we'll have to postpone a fix until we move up to python 3.5 or
> >> higher.  I'm adding python 3.5 upgrade to the PLAN file.  The
> >> django framework can run on python 3.x, but we haven't done any
> >> work to make our own code 3.5-friendly.
> >> 
> >> work.
> >> If EAI support is sufficiently urgent (i.e., can't wait for python
> >> 3.x conversion), we'll have to code it up ourselves, without lib
> >> support.
> > 
> > Please advise on the priority of this effort.  The IETF has done a
> > lot of work to make internationalized domain names happen.  Now, we
> > need to eat our dog food.  Do we need to eat that meal in the next
> > few weeks or the next few months?
> 
> I'd actually put this before format work, if only because I think the infrastructure needs to be in place to handle nonASCII characters in all places.

Thanks.  I hope that they can move in parallel.

Russ

---

From: Russ Housley <[email-masked]>
Date: Thu, 9 Jun 2016 09:09:36 -0400

> My gut feeling is to put this on the “few months” bucket. There should be very few issues that demand bug fixes in matter of weeks. Now, if that few months starts to be more like half a year or a y…

There are at least three places where changes are needed in the system:

1) Mailman
2) Datatracker
3) IETF Meeting Registration

The IAOC usually opens registration for the next meeting very shortly after the end of the current meeting.  So, we know that all three of these cannot be ready to handle internationalized email addresses by the end of the Berlin meeting.  I think we should target the end of the Seoul meeting.  If there are no big surprises, we  can have it ready when we open registration for Chicago.

It would be really bad for the registration system to put an address on the attendees list and have Mailman choke.  I have asked Glen to investigate the current version of Mailman and determine whether it supports internationalized email addresses.  For the above list, I am assuming that an upgrade of some sort will be needed.  I am also assuming that there is a version of Mailman that will properly handle internationalized email addresses.  (Glen if that turns out to be incorrect, please raise the alarm on this list.)

Russ

---

From: Glen <[email-masked]>
Date: Tue, 14 Jun 2016 09:39:18 -0700

All -

> It would be really bad for the registration system to put an address on the attendees list and have Mailman choke.  I have asked Glen to investigate the current version of Mailman and determine whe…

I am, as requested, raising a "maybe" alarm:

We are on Mailman 2.1.17 at present.  The next version of OpenSuse to
which I hope to move us later this year (42.1) includes Mailman 2.1.21
(just released in February).  2.1.22 was released in April, and 2.1.23
is pending. (See https://launchpad.net/mailman/+milestones for details
and links).

I have read the entire changelog for every release from 2.1.17
forward, and I see NO references to UTF8, or Unicode, or International
Character support, or anything similar, when it comes to email
addresses.  They talk about their templates and translations, but
nothing about actual email addresses.

I therefore believe that all 2.X versions of Mailman equally lack
support for internationalized email addresses.

I note also a 3.0 branch of Mailman - but it seems to have lost steam
with no new releases in the past 18 months.

Now... there are hints that Mailman "might" support these characters
in some fashion already.  For example I see this:
https://wiki.list.org/DOC/4.90%20My%20list%27s%20moderator%20interface%20and%20plain%20format%20digests%20have%20garbled%20non-ascii.%20Change%20English%20charset%20to%20utf-8.
and I see references to character set bug fixes in sync_lists, a
Python script.  But I think that may be orthogonal to
internationalized email address support... I think it may refer only
to message bodies.  At that link, I see:

"Mailman's character set for English is us-ascii. This causes
non-ascii characters in posts and message headers and footers and
other places to be replaced by question marks and/or numeric HTML
entities in various contexts if the web interface and in plain format
digests. This can be avoided by changing Mailman's character set for
English to UTF-8. This has not been done in the GNU Mailman
distribution for mainly superstitious reasopns, but it is believed to
be safe. This is done by placing the line:

add_language('en', 'English (USA)', 'utf-8', 'ltr')

in mm_cfg.py and restarting Mailman.  The one known downside of doing
this is that Python's email library which is used by Mailman will
base64 encode charset=utf-8 message bodies which makes the raw message
body impossible to read by eye or search with simple tools like grep.
"

But it's unclear as to whether this would work or not, and I'd want
everyone to review this before we tried the above, even if we thought
it would work.

Absent that being some kind of magic bullet, it appears that there are
no differences in newer versions of Mailman with respect to
internationalized email addresses at this time.

Glen