Subject: [Enum] One byte good, two bytes bad?

From: "Conroy, Lawrence (SMTP)" <[email-masked]>
Date: Sun, 13 Jun 2004 07:28:37 -0400

Hi Folks,
  Whilst editing the BCP/"Experiences" draft, I came across one issue 
that I don't really understand.

As far as I can tell, the characters in all of the NAPTR fields are/can 
be within the US-ASCII
equivalent range of UTF-8, EXCEPT possibly for the SED-like 
substitution sub-field of the RegExp
field (i.e. any explicit text used to build a URI that allows UTF-8), 
AND possibly the Replacement
field that holds a domain name in a non-final NAPTR.
For all of these other fields, the characters are single-byte.

Here's the question:
The replacement field holds a domain - but...
Does IDN support mean that this domain name could include multi-byte 
characters?
(I think that it can, but I'm hopelessly confused on how IDN works and 
whether such domain
 names are to be expected in a NAPTR).

Here's the "heads up":
As a final observation, multi-byte characters in the substitution 
expression text can spoil your
ENUM client's whole day if you have done what just about everyone has 
so far - assumed that it's
all US-ASCII and used single byte string processing routines. I think 
that you might confuse the
subsequent byte(s) of the UTF-8 character with a delimiter (i.e. whilst 
looking for "!" - 0x21).

all the best,
  Lawrence
_______________________________________________
enum mailing list
enum at ietf.org
https://www1.ietf.org/mailman/listinfo/enum

---

From: Jim Reid <[email-masked]>
Date: Sun, 13 Jun 2004 08:58:18 -0400

>>>>> "lwc" == Conroy, Lawrence (SMTP) <lwc at roke.co.uk> writes:

    lwc> Does IDN support mean that this domain name could
    lwc> include multi-byte characters? 

No. The whole point of IDN is backwards compatibility with the
installed base. Hostnames -- as distinct from domain names -- can only
be composed from the characters A-Z, a-z, 0-9 and -. IDN uses Punycode
to convert between strings written in Unicode and "legal host name"
ASCII. This ensures IDN names won't upset existing web browsers, mail
software and so on. You and I might get presented with a domain name
of xn--qwertyuiop.co.jp but someone in Japan with an IDN-aware browser
sees this name as <hiragana-or-katakana=characters>.co.jp.

Now to be pedantic -- this is an IETF list after all -- domain names
in NAPTR records could contain multi-byte characters. [Not all domain
names are hostnames.] But not because of IDN. The DNS protocol is
8-bit clean. And anyone is able to enter whatever names they want into
the zone files they control. This would be unwise: who knows what a
SIP client or server (say) might do when presented with an illegal
name in a NAPTR record? And as you say multi-byte characters can make
life unpleasant for anything doing regexp substitution.

It might be worth writing up something on this along the lines of
"non-ASCII characters in NAPTRs considered harmful". While multi-byte
characters in NAPTRs shouldn't happen, it's not hard to envisage
circumstances where they will, either by accident or design. They may
even be useful in some cases. Another worry could be regexp processing
that eats the "xn--" prefix at the start of an IDN-encoded label.

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 14 Jun 2004 03:32:38 -0400

On Jun 13, 2004, at 13:22, Conroy, Lawrence (SMTP) wrote:
Here's the question:
The replacement field holds a domain - but...
Does IDN support mean that this domain name could include multi-byte 
characters?
(I think that it can, but I'm hopelessly confused on how IDN works and 
whether such domain
 names are to be expected in a NAPTR).
No, the domain name in a URI always (on the protocol level, which I 
claim we talk about here) always include the punycode encoded domain 
name, i.e. ASCII only.

For the rest of the URI, I claim ASCII only, but, how to 
internationalize a URI in general is a separate discussion which is 
held in the W3C and it goes under the name of IRI. Yes, there is a 
draft on IRI's as well.

See draft-duerst-iri-08.txt
Conclusion (my guess): It is "ok" to assume all bytes in the regexp is 
ASCII, BUT, if any multibyte stuff exists, it will be UTF-8 encoded.

    paf