Subject: Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

From: "Matthew A. Miller" <[email-masked]>
Date: Mon, 17 Apr 2017 11:41:29 -0600

On 17/03/27 22:48, Tim Bray wrote:
> First of all, let me say that I’m delighted with, and fully support, the
> promotion of the status of UTF-8 in the JSON RFC to MUST.  I suspect
> this steps way outside the JSONbis charter, but that’s a problem for
> chairs and ADs, not yr humble editor.
> 
> Comments on Matt's proposed text:
> 
> 1. How about a very short historical note, along the lines of: “Previous
> specifications of JSON, including the predecessor RFCs, have not
> required the use of UTF-8 for use with the application/json media type. 
> However, implementors of JSON-based software have overwhelmingly chosen
> to use the UTF-8 encoding, to the extent that it is the only realistic
> way to achieve interoperability in software which generates or consumes
> JSON.”
> 
> ... moving on...
> 
> ​​
> O
> ​​
> ​​
> n Mon, Mar 27, 2017 at 1:04 PM, Matthew A. Miller
> <[email-masked]
> <mailto:[email-masked]>> wrote:
> ​​
> 
>     ​​
>     JSON text SHOULD be encoded in UTF-8 (Section 3 of [UNICODE]); JSON
>     ​​
>     text MAY be encoded in UTF-16 or UTF-32 if the generator is certain
>     ​​
>     the intended recipients can process it. JSON text MUST NOT be encoded
>     ​​
>     in any encoding other than UTF-8, UTF-16, or UTF-32. When used with
>     ​​
>     media type "application/json" the JSON text MUST be encoded as UTF-8.
> 
> 
> ​2. Seriously, why the “JSON text MAY be encoded in… can process it ​”
> phrase?  It’s a distraction, and if people want to do that, we can’t
> stop them, but we shouldn't waste RFC space talking about practices that
> are not remotely interoperable.  The I in IETF stands for Internet, and
> JSON on the Internet is UTF-8, end of story.
> ​​
> 
>     Recipients that wish to support Unicode encodings other than UTF-8
>     can do this using a detection mechanism that is based on the fact
>     that the first character will always have a Unicode code point
>     greater than 0 and less than 128, thus the UTF-16/32 variants can
>     be detected by inspecting the first octets for nulls.
> 
> 
> ​3. Is it just me, or does it feel really dorky to talk mysteriously
> about this detection mechanism without providing details?  On top of
> which, anyone who's writing the kind of software that might lead one to
> consult ​an RFC first shouldn't bloody well use anything but UTF-8.  If
> people really want to have this, I think we owe the world an outline of
> the algorithm, maybe in an appendix. I'll volunteer to make my best
> effort to draft it and try to get consensus that it's correct..  If we
> can't, that's a powerful symbol that we shouldn't have this language. 
> But that's my fallback position; my real request to the group is that we
> just take this out.
> 

[ /me doffs hat ]

Thinking about this more, putting an encoding detection algorithm as an
appendix seems like a reasonable compromise to me.  To start, how about
removing the detection text from Section 8.1 and have an appendix that
starts with that text plus the table?

Assuming the above, what does everyone think of the following for
Section 8.1?

"""
JSON text SHOULD be encoded in UTF-8 (Section 3 of [UNICODE]). JSON
text MUST NOT be encoded in any encoding other than UTF-8, UTF-16,
or UTF-32. When used with media type "application/json" the JSON
text MUST be encoded as UTF-8.

Previous specifications of JSON have not required the use of UTF-8
with the "application/json" media type. However, the vast majority
of JSON-based software implementations have chosen to use the UTF-8
encoding, to the extent that it is the only encoding that achieves
interoperability.

Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a JSON text.  In the interests of interoperability,
implementations that parse JSON texts MAY ignore the presence of a
byte order mark rather than treating it as an error.
"""

- m&m

Matthew A. Miller

---

From: Nico Williams <[email-masked]>
Date: Mon, 17 Apr 2017 12:56:28 -0500

> starts with that text plus the table?

Or we could even just assert that such an algorithm is possible, and
that implementors MAY implement one.

> Assuming the above, what does everyone think of the following for

+1.

Nico
--

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Mon, 17 Apr 2017 19:08:30 +0000

On 4/17/17, 1:56 PM, "json on behalf of Nico Williams" <[email-masked] on behalf of [email-masked]> wrote:

    
    
    +1.

I’ve argued before for having the encoding detection algorithm put into an appendix. If it were, I also vote +1 to the rewording.

	Tony Hansen

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 17 Apr 2017 22:47:25 +0200

> that implementors MAY implement one.

Indeed.

Broken record mode:

— writing up the algorithm sounds like encouraging implementation.
  We *don’t* want people to implement this!
  (The whole interminable non-UTF-8 saga probably just was a nod from the RFC 4627 authors to the remnants of UTF-16 land, which mostly have died off since.  Why resurrect?)

- there have been about 15 attempts to define this algorithm on the mailing list.
  All were wrong.
  An Internet Standard should contain tried and true material, not errata fodder.

- an implementer is in a much better position to get this right than the standard, because they can write unit tests.

>> Assuming the above, what does everyone think of the following for

+1

Grüße, Carsten

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 18 Apr 2017 14:22:22 +0900

> - an implementer is in a much better position to get this right than the standard, because they can write unit tests.

I completely agree with Carsten. As far as I know, and as far as we have 
been told on this list, if some JSON isn't in UTF-8, then it simply will 
not interoperate.

In my view, the only reason to still have a MAY for UTF-16/32 is that 
this will avoid questions like: "I have a JSON parser in language FOO, 
it can take a string or an input stream as an argument. In FOO, strings 
are UTF-16, but the JSON RFC doesn't seem to allow this. What should I do."

This may look far-fetched to some of you (and I agree it is), but I 
remember having dealt with such questions when I was at W3C.

Regards,   Martin.

---

From: Tim Bray <[email-masked]>
Date: Mon, 17 Apr 2017 23:29:08 -0700

My position is the same as Carsten’s.

On Mon, Apr 17, 2017 at 1:47 PM, Carsten Bormann <[email-masked]> wrote:

>   (The whole interminable non-UTF-8 saga probably just was a nod from the
> RFC 4627 authors to the remnants of UTF-16 land, which mostly have died off
> since.  Why resurrect?)
>
> - an implementer is in a much better position to get this right than the
> standard, because they can write unit tests.
>
> >
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: John Cowan <[email-masked]>
Date: Tue, 18 Apr 2017 07:55:28 -0400

On Tue, Apr 18, 2017 at 1:22 AM, Martin J. Dürst <[email-masked]>
wrote:

In my view, the only reason to still have a MAY for UTF-16/32 is that this
> will avoid questions like: "I have a JSON parser in language FOO, it can
> take a string or an input stream as an argument. In FOO, strings are
> UTF-16, but the JSON RFC doesn't seem to allow this. What should I do."

The answer to that is something like this:  "In the IETF, we do interchange
standards.  What you do with JSON internally to your system is your own
business."

-- 
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
Said Agatha Christie / To E. Philips Oppenheim
"Who is this Hemingway? / Who is this Proust?
Who is this Vladimir / Whatchamacallum,
This neopostrealist / Rabble?" she groused.
        --George Starbuck, Pith and Vinegar

---

From: Pete Cordell <[email-masked]>
Date: Tue, 18 Apr 2017 14:01:28 +0100

On 18/04/2017 06:22, Martin J. Dürst wrote:
>>   (The whole interminable non-UTF-8 saga probably just was a nod from
>> the RFC 4627 authors to the remnants of UTF-16 land, which mostly have
>> died off since.  Why resurrect?)
>>
>> - an implementer is in a much better position to get this right than
>> the standard, because they can write unit tests.
>

+1

If we do do this, I think we could add some example test messages to 
helps with the development, e.g.:

     {"Example":1}
     {}
     "Example"
     ""
     "U+0100" (where U+0100 is the UTF form of the character, not ASCII)
     1

> are UTF-16, but the JSON RFC doesn't seem to allow this. What should I do."

IMO the answer to that is, "that's why it says 'JSON text _SHOULD_ be 
encoded in UTF-8'".

I agree with John Cowen, that use of UTF-16/32 is purely for internal 
scenarios; not on the Internet.  As such, I believe the IETF is going 
beyond its remit to say you can use UTF-16/32 for your internal purposes 
that I know nothing about and care nothing about, but you're not allowed 
to use other encodings that maybe more natural for your system.

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Tue, 18 Apr 2017 18:24:38 +0000

On 4/17/17, 4:47 PM, "json on behalf of Carsten Bormann" <[email-masked] on behalf of [email-masked]> wrote:
 
    Broken record mode:

    

It’s exactly >because< it’s hard that it should be done once. Here is my suggestion for the wording of the appendix.

Appendix N

As stated above, implementations generating interoperable JSON MUST NOT send UTF-16, and implementations receiving interoperable JSON is NOT REQUIRED to accept UTF-16. If an implementation needs to recognize UTF-16 JSON to deal with legacy files, the following algorithm will differentiate. (provide algorithm here)

	Tony Hansen

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Apr 2017 20:34:33 +0200

On Apr 18, 2017, at 20:24, HANSEN, TONY L <[email-masked]> wrote:
> 
> It’s exactly >because< it’s hard that it should be done once.

I was trying to say it should not be done at all.

Grüße, Carsten

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 19 Apr 2017 16:36:16 +0900

>     "U+0100" (where U+0100 is the UTF form of the character, not ASCII)

I don't mind adding examples, although it's getting a bit late for this.

However, if we want something like

I suggest to use the facility that the RFC Editor and others have worked 
long and hard now, and use the actual character:

"Ā" (string with the single character LATIN CAPITAL LETTER A WITH MACRON 
(U+0100))

Maybe a longer string would be better.

Regards,    Martin.

---

From: Pete Cordell <[email-masked]>
Date: Wed, 19 Apr 2017 10:50:16 +0100

> "Ā" (string with the single character LATIN CAPITAL LETTER A WITH MACRON

I'd missed the memo on non-ASCII characters in RFCs.  This sounds great 
to me.

Pete Cordell
Codalogic Ltd

---

From: Pete Cordell <[email-masked]>
Date: Thu, 27 Apr 2017 10:15:11 +0100

I seem to have killed this thread off again.  Sorry about that.

Any conclusions?

Cheers,

Pete.

---

From: "Matthew A. Miller" <[email-masked]>
Date: Thu, 27 Apr 2017 10:21:33 -0600

I see consensus for text in Section 8.1 pending an Appendix on encoding
detection, but nothing for the Appendix itself.

Looking deeper in the threads again, it appears to me that:

* There is consensus to say "just use UTF8" in many (most) scenarios
* There is rough consensus to say "always use UTF8", but concern this
change goes beyond the charter
* There is no consensus on a detection algorithm

While the charter calls for "absolute minimal changes", it calls out RFC
7159 (and its -bis) as documenting "interoperability concerns when
exchanging JSON over a network".  Documenting this interoperability is
one of its primary goals.

Therefore, the argument is made that a change to the effect of "JSON
text MUST be encoded as UTF-8" is considered an "absolute minimum
change" in order to achieve the goal of network interoperability.

To the working group:

* Is there strong objection to mandating only UTF-8?
* Does anyone have suggested text to that effect?

- m&m

Matthew A. Miller
JSONbis Chair

---

From: Pete Cordell <[email-masked]>
Date: Fri, 28 Apr 2017 15:23:45 +0100

Thanks.  I agree with your summary.

If you removed the sentence which mentions "UTF-16, or UTF-32" from the 
text that you proposed on 17 April, I'd be happy with that text.

Thanks again,

Pete.