Subject: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

From: Matthew Miller <[email-masked]>
Date: Mon, 13 Mar 2017 15:06:13 -0600

Hello JSONbis,

The security directorate review discussion has raised the issue of
encoding detection.  The original table from RFC 4627 was removed from
RFC 7159 due to a lack of consensus.  In this latest round, there have
been a number of comments have been made supporting (and against) adding
more guidance than is currently present.

The chair asks for a call on the following from the working group:

1) Does the working group think adding any text on how to detect the
encoding worthwhile?

2a) If such text is worthwhile, is the following proposed text from Nico
Williams acceptable (to be appended to Section 8.1)?

"""
   Implementors MAY count the number of ASCII NULs in the first four
   bytes of any JSON text to detect which of UTF-8, UTF-16, or UTF-32
   the text is encoded in:

    - if the count is zero, then the text is encoded in UTF-8
    - if the count is one or two, then the text is encoded in UTF-16
    - if the count is three, then the text is encoded in UTF-32

   This results from a) JSON texts having to start with an ASCII
   character, b) no unescaped NULs being allowed in JSON strings, and c)
   any type being allowed at the top-level, thus the first character may
   be a double-quote and the second may be any permissible, unescaped
   Unicode codepoint.  An ASCII character requires a NUL-valued byte in
   UTF-16 encoding, three in UTF-32, and none in UTF-8.

"""
2b) If such text is worthwhile but Nico's proposal is not worthwhile,
what would be acceptable?

Please respond by March 16.

Thank you in advance,

-- 
Matthew A. Miller
JSONbis chair

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 22:12:09 +0100

For the record…

> On 13 Mar 2017, at 22:06, Matthew Miller <[email-masked]> wrote:
> 
> Hello JSONbis,
> 

No, that would be a regression into maintaining the fiction that UTF-16 and UTF-32 versions of JSON are being used in interchange.

> “""

Not sure if I’m allowed to note that after saying no above, but not all JSON documents have four bytes.

> what would be acceptable?

Again, not worthwhile, but maybe it wouldn’t hurt to mention that implementations that want to guard against erroneously encoded input can detect ASCII NULs in the input and even use those to predict whether the encoder was using one of the UTF-16s or one of the UTF-32s.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 22:50:42 +0100

> 1) Does the working group think adding any text on how to detect the

Yes.

>    Implementors MAY count the number of ASCII NULs in the first four

s/MAY/can/

>    UTF-16 encoding, three in UTF-32, and none in UTF-8.

...and add that if the number of octets is less than 4, it's UTF-8.

> ...

I personally would prefer the table that I posted, as directly 
identifies byte ordering as well:

>            00 00 00 xx  UTF-32BE
>            00 xx 00 xx  UTF-16BE
>            xx 00 00 00  UTF-32LE
>            xx 00 xx 00  UTF-16LE
>            xx xx xx xx  UTF-8

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Mon, 13 Mar 2017 16:50:52 -0500

On Mon, Mar 13, 2017 at 10:12:09PM +0100, Carsten Bormann wrote:
> No, that would be a regression into maintaining the fiction that
> UTF-16 and UTF-32 versions of JSON are being used in interchange.

I would support saying that JSON texts SHOULD be UTF-8.  I might even
support that they MUST be UTF-8, but I suspect there won't be consensus
for that, though IF there is, then I agree with you.

> Not sure if I’m allowed to note that after saying no above, but not

Why wouldn't you be?

> all JSON documents have four bytes.

True!  The empty string at the top-level, with no whitespace around it,
is a two-byte text when encoded in UTF-8.

Indeed, just two bytes will do for some cases; at most four are needed.
Byte order can be detected as well, also in two to four bytes, but byte
order also is pointless to think about if UTF-8 were REQUIRED (which it
isn't [yet]).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 13 Mar 2017 17:15:06 -0500

On Mon, Mar 13, 2017 at 10:50:42PM +0100, Julian Reschke wrote:
> I personally would prefer the table that I posted, as directly identifies
> byte ordering as well:
> 
> >           00 00 00 xx  UTF-32BE
> >           00 xx 00 xx  UTF-16BE
> >           xx 00 00 00  UTF-32LE
> >           xx 00 xx 00  UTF-16LE
> >           xx xx xx xx  UTF-8

WFM.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 23:19:04 +0100

> ...and add that if the number of octets is less than 4, it's UTF-8.

No.

(It’s not hard to get this right in a program, because you can write unit tests.
It is much harder to get this right in text.
Let’s NOT write this text.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 23:23:07 +0100

> Let’s NOT write this text.)

"No" in the sense of "that's incorrect", or "it's correct, but we 
shouldn't state it"?

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 23:24:30 +0100

On 13 Mar 2017, at 23:23, Julian Reschke <[email-masked]> wrote:
> 
> "No" in the sense of "that's incorrect", or "it's correct, but we shouldn't state it"?

Incorrect.

(Counterexample left as an exercise to the reader.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 23:38:50 +0100

> (Counterexample left as an exercise to the reader.)

OK. Let's try again.

If the number of octets is less than 4, it's either UTF-8 or no JSON at all?

---

From: Peter Cordell <[email-masked]>
Date: Mon, 13 Mar 2017 22:44:49 +0000

> identifies byte ordering as well:

However, the table is not correct because the second 00 in the UTF-16 
cases could be non-zero when we have a string that starts with a 
non-ASCII range character.

Maybe a table in which the first n-bytes match the follow can be used:

     00 00        UTF-32BE
     00 xx        UTF-16BE
     xx 00 00     UTF-32LE
     xx 00 xx     UTF-16LE
     xx xx        UTF-8

But overall I prefer the "UTF-8 REALLY REALLY SHOULD be used unless the 
system knows otherwise" text proposed earlier.  Nothing other than UTF-8 
should be transported over a network these days (beyond file sharing 
etc).  UTF-8 won!

Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 14 Mar 2017 00:56:02 +0100

> If the number of octets is less than 4, it's either UTF-8 or no JSON at all?

OK, it seems I need to give away the answer to the exercise:

33 00

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 14 Mar 2017 07:01:23 +0100

> OK, it seems I need to give away the answer to the exercise:

Yep, I was too focused on top-level strings that start with non-ASCII...

Best regards, Julian

---

From: Tim Bray <[email-masked]>
Date: Mon, 13 Mar 2017 23:17:37 -0700

My position is identical to Carsten's. If you want guaranteed
interoperability, use I-JSON. Let's not weaken this document with dubious
handwaving that suggests anything but UTF-8 is a sane choice.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 14 Mar 2017 07:44:40 +0100

On 2017-03-14 07:17, Tim Bray wrote:
> interoperability, use I-JSON. Let's not weaken this document with
> dubious handwaving that suggests anything but UTF-8 is a sane choice.
> ...

I'm sympathetic with the intent.

However, if the spec allows encodings other than UTF-8, it needs to give 
sufficient information that this needs to be done by inspecting the 
payload, not out-of-band data. Carsten has demonstrated that it's not 
simple, so let's either write down the exact steps, or at least give 
readers something to start with, such as:

"Character encoding detection can be done based on the fact that the 
first character is always US-ASCII, so the UTF-16/32 variants can be 
detected by inspecting the first octets for zeros."

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 14 Mar 2017 07:50:10 +0100

On 14 Mar 2017, at 07:44, Julian Reschke <[email-masked]> wrote:
> 
> "Character encoding detection can be done based on the fact that the first character is always US-ASCII, so the UTF-16/32 variants can be detected by inspecting the first octets for zeros."

Prefix this with “If UTF-16 and UTF-32 forms need to be supported,”, and I’m happy.

Grüße, Carsten

---

From: Peter Cordell <[email-masked]>
Date: Tue, 14 Mar 2017 08:46:28 +0000

> Maybe a table in which the first n-bytes match the follow can be used:

Seems I need to correct my correction to allow for 0100;LATIN CAPITAL 
LETTER A WITH MACRON and so on:

     00 00        UTF-32BE
     00 xx        UTF-16BE
     xx 00 00 00  UTF-32LE
     xx 00 00 xx  UTF-16LE
     xx 00 xx     UTF-16LE
     xx xx        UTF-8
     xx EOF       UTF-8 (For Carsten's recent comment)

I think that's roughly where we got last time, and sufficient doubt had 
crept in that the UTF-8 should be used if unless the system knows the 
encoding is otherwise.

Pete Cordell
Codalogic Ltd
Rules for Describing JSON Content, http://json-content-rules.org

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 14 Mar 2017 11:46:57 +0100

>    xx EOF       UTF-8 (For Carsten's recent comment)

Wow, people are still posting incorrect match tables :-)

I think that Julian (with my amendment maybe) nailed it, no need to make ever more wronger proposals here.

(Again, in the age of unit testing, all this is much easier to get right for a coder than for a spec writer.)

Grüße, Carsten

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 14 Mar 2017 20:03:26 +0900

> Prefix this with “If UTF-16 and UTF-32 forms need to be supported,”, and I’m happy.

That would be fine with me, but I think it would be great if we could 
also rewrite the "JSON text SHALL be encoded in UTF-8, UTF-16, or 
UTF-32" phrase in the current draft as per my separate proposal (as 
modified by Peter Cordell at 
https://mailarchive.ietf.org/arch/msg/json/MR93bNIj1sDEutFIPupHTRYrJPI). 
Starting with UTF-8 only rather than starting with UTF-8, UTF-16, and 
UTF-32 together makes it clearer where interoperability is even if for 
somebody who reads the text in detail, there's not much that changes.

Regards,   Martin.

---

From: Peter Cordell <[email-masked]>
Date: Tue, 14 Mar 2017 11:43:05 +0000

> Wow, people are still posting incorrect match tables :-)

Well, it was posted to show the rat hole that specifying detection is, 
and illustrate why we might not want to go there.

> I think that Julian (with my amendment maybe) nailed it, no need to make ever more wronger proposals here.

I think Julian's table missed the case of a string starting with a 
character like U+0100.  If that did turn out to be the route chosen, it 
seemed prudent to point out the omission earlier rather than later. 
(I'll resist an update with xx 00 EOF UTF-16LE.)

 > (Again, in the age of unit testing, all this is much easier to get
 > right for a coder than for a spec writer.)

Only if you don't miss cases to test!

Pete Cordell
Codalogic Ltd

---

From: "Paul Hoffman" <[email-masked]>
Date: Thu, 16 Mar 2017 13:33:06 -0700

This whole thread feels like a regression from RFC 7159. The abstract of 
this document says:

    This document removes inconsistencies with other specifications of
    JSON, repairs specification errors, and offers experience-based
    interoperability guidance.

Opening up to new encodings reduces interoperability.

> 1) Does the working group think adding any text on how to detect the

No. Adding a multi-step test that is easy to get wrong reduces 
interoperability.

--Paul Hoffman

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Mar 2017 21:40:53 +0100

> Opening up to new encodings reduces interoperability.

+1

> No. Adding a multi-step test that is easy to get wrong reduces

-1 - this came up as unclear during IESG review, and it *is* unclear. If 
the media type doesn't have a charset parameter, the spec actually 
should state how detect the encoding.

Best regards, Julian

---

From: "Matthew A. Miller" <[email-masked]>
Date: Sat, 18 Mar 2017 13:26:52 -0600

Hello JSONbis,

After a lengthy discussion about encoding, I see no consensus on a
change.  Therefore, the current text for Section 8.1 will remain as it is.

Thank you all,

- m&m

Matthew A. Miller
JSONbis Chair

---

From: Julian Reschke <[email-masked]>
Date: Sun, 19 Mar 2017 10:25:07 +0100

> change.  Therefore, the current text for Section 8.1 will remain as it is.

A very big -1 on this.

I agree there was no consensus on a normative change, but I heard many 
people agreeing that we could clarity here.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 19 Mar 2017 11:29:34 +0100

...and here is a concrete proposal:

Original text:

> 8.1.  Character Encoding
>
>    JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32 [UNICODE]
>    (Section 3).  The default encoding is UTF-8, and JSON texts that are

Change 1:

Say "MUST" instead of "SHALL", as it's the more common form of 
expressing this requirement.

Change 2:

Replace "[UNICODE] (Section 3)" by "Section 3 of [UNICODE]".

That said, this citation isn't as stable as it should, as [UNICODE] 
refers to <http://www.unicode.org/versions/latest/> and unless I'm 
missing something, there's no guarantee that future versions will have 
the relevant bits in Section 3.

>    encoded in UTF-8 are interoperable in the sense that they will be
>    read successfully by the maximum number of implementations; there are
>    many implementations that cannot successfully read texts in other
>    encodings (such as UTF-16 and UTF-32).

Change 3:

Add "Text encoded in character encodings other than UTF-8, UTF-16, or 
UTF-32 can not be used with the media type "application/json".

(this explains the implications of the SHALL/MUST)

>    Implementations MUST NOT add a byte order mark (U+FEFF) to the
>    beginning of a JSON text.  In the interests of interoperability,
>    implementations that parse JSON texts MAY ignore the presence of a
>    byte order mark rather than treating it as an error.

Finally, change 4:

Add a new paragraph:

"Recipients that wish to support Unicode encodings other than UTF-8 can 
do this using a detection mechanism that is based on the fact that the 
first character will always have a Unicode code point less or equal than 
127, thus the UTF-16/32 variants can be detected by inspecting the first 
octets for nulls."

I believe none of these changes affects anything normative, but that 
they absolutely clarify the spec. In particular, having them in the spec 
would have avoided this whole discussion we just had.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 19 Mar 2017 14:34:47 +0100

On 19 Mar 2017, at 11:29, Julian Reschke <[email-masked]> wrote:
> concrete proposal

Maybe we can build consensus on this text change, as it is not a normative change.

(I’d rather have some additional change that makes UTF-8 the SHOULD, which I also sensed a good level of consensus for.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Sun, 19 Mar 2017 16:49:51 +0100

> (I’d rather have some additional change that makes UTF-8 the SHOULD, which I also sensed a good level of consensus for.)

I'm sympathetic to that, but it IMHO would mean to have another Last Call.

Best regards, Julian

---

From: "Matthew A. Miller" <[email-masked]>
Date: Mon, 20 Mar 2017 10:26:20 -0600

Thank you for the suggested changes, Julian.  To consolidate the
changes, I believe the following is your suggested text for all of
Section 8.1:

"""
JSON text MUST be encoded in UTF-8, UTF-16, or UTF-32 Section 3 of
[UNICODE].  The default encoding is UTF-8, and JSON texts that are
encoded in UTF-8 are interoperable in the sense that they will be
read successfully by the maximum number of implementations; there are
many implementations that cannot successfully read texts in other
encodings (such as UTF-16 and UTF-32).  Text encoded in character
encodings other than UTF-8, UTF-16, or UTF-32 cannot be used with
the media tye "application/json".

Recipients that wish to support Unicode encodings other than UTF-8
can do this using a detection mechanism that is based on the fact
that the first character will always have a Unicode code point less
or equal than 127, thus the UTF-16/32 variants can be detected by
inspecting the first octets for nulls.
"""

Does the working group object to this change?

- m&m

Matthew A. Miller
JSONbis Chair

---

From: John Cowan <[email-masked]>
Date: Mon, 20 Mar 2017 12:44:15 -0400

On Mon, Mar 20, 2017 at 12:26 PM, Matthew A. Miller <
[email-masked]> wrote:

> inspecting the first octets for nulls.

Two minor corrections: for "media tye" read "media type", and for
"less than or equal to 127" read "greater than 0 and less than 128".

--
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
Fundamental thinking is ha-ard.  Let's go ideology-shopping.
                        --Philosopher Barbie

---

From: Julian Reschke <[email-masked]>
Date: Mon, 20 Mar 2017 17:46:35 +0100

> [UNICODE].  The default encoding is UTF-8, and JSON texts that are

Put "Section 3 of Unicode" into parentheses.

> encodings (such as UTF-16 and UTF-32).  Text encoded in character

Replace "in other encodings (such as UTF-16 and UTF-32)" by just "in 
other encodings". The proposed text implies that there encodings other 
than UTF-16 or UTF-32 which could be allowed.

> Does the working group object to this change?

+1, with the comments above.

Best regards, Julian

---

From: Peter Cordell <[email-masked]>
Date: Mon, 20 Mar 2017 17:58:03 +0000

> than UTF-16 or UTF-32 which could be allowed.

If that's what you want, instead of:

     there are many implementations that cannot successfully read
     texts in other encodings (such as UTF-16 and UTF-32).

Or:

     there are many implementations that cannot successfully read
     texts in other encodings.

I think it would be even clearer to say:

     there are many implementations that cannot successfully read
     texts encoded in UTF-16 or UTF-32.

Regards,

Pete Cordell
Codalogic Ltd

---

From: Julian Reschke <[email-masked]>
Date: Mon, 20 Mar 2017 19:02:11 +0100

WFM.

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Tue, 21 Mar 2017 04:20:02 +0000

I like most of what Matt has below. However, I would prefer that the statement about inspecting the first octets for nulls were more explicit. I think it was Julian who posted a nice little chart the other day on how to determine which utf-16/32 variants was present based on the null pattern. If you don’t want it in the main text, then at least put it into an appendix. Otherwise, the code will be re-created many times, and probably often incorrectly.

	Tony Hansen

On 3/20/17, 12:26 PM, "json on behalf of Matthew A. Miller" <[email-masked] on behalf of [email-masked]> wrote:

    """

    

    - m&m
    
    Matthew A. Miller
    JSONbis Chair

---

From: "Matthew A. Miller" <[email-masked]>
Date: Thu, 23 Mar 2017 08:40:52 -0600

Hello JSONbis,

It looks like we have consensus for the following text for all of
Section 8.1:

"""
JSON text MUST be encoded in UTF-8, UTF-16, or UTF-32 Section 3 of
[UNICODE].  The default encoding is UTF-8, and JSON texts that are
encoded in UTF-8 are interoperable in the sense that they will be
read successfully by the maximum number of implementations; there are
many implementations that cannot successfully read texts encoded in
UTF-16 or UTF-32. Text encoded in character encodings other than UTF-8,
UTF-16, or UTF-32 cannot be used with the media type "application/json".

Recipients that wish to support Unicode encodings other than UTF-8
can do this using a detection mechanism that is based on the fact
that the first character will always have a Unicode code point
greater than 0 and less than 128, thus the UTF-16/32 variants can
be detected by inspecting the first octets for nulls.
"""

Please speak now if you have any objections.

Thank you all,

--
- m&m

Matthew A. Miller
JSONbis Chair

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Thu, 23 Mar 2017 18:56:03 +0000

I would still like to see Julian’s small table on differentiating between UTF variants added as an appendix.

	Tony Hansen

On 3/23/17, 10:40 AM, "json on behalf of Matthew A. Miller" <[email-masked] on behalf of [email-masked]> wrote:

    Hello JSONbis,

    """

    
    
    Thank you all,
    
    --
    - m&m
    
    Matthew A. Miller
    JSONbis Chair

---

From: Julian Reschke <[email-masked]>
Date: Thu, 23 Mar 2017 20:32:18 +0100

> I would still like to see Julian’s small table on differentiating between UTF variants added as an appendix.

FWIW, Carsten pointed out that the table wasn't accurate. Also, it seems 
some participants feel that having the table would actually encourage 
people to implement this, which some of the participants think is a bad 
idea in the first place.

Best regards, Julian

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Thu, 23 Mar 2017 20:02:28 +0000

>    people to implement this, which some of the participants think is a bad 

Ok, Nico’s simplification’s to the algorithm would be better. But it kind of proves my point about how it is easy to get wrong.

Which will lead to greater interoperability issues?

*) A greater chance of people implementing it CORRECTLY just because the info is there?
*) People deciding to implement it and then implementing it wrong because the info is NOT there. 

Which set would be larger? I don’t know. But I’d rather err on the side of overall increased interoperability.

	Tony Hansen

---

From: John Cowan <[email-masked]>
Date: Thu, 23 Mar 2017 16:49:21 -0400

On Thu, Mar 23, 2017 at 4:02 PM, HANSEN, TONY L <[email-masked]> wrote:

But I’d rather err on the side of overall increased interoperability.

In that case, discourage implementation of anything but UTF-8.

-- 
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
The Unicode Standard does not encode idiosyncratic, personal, novel,
or private use characters, nor does it encode logos or graphics.

---

From: "Matthew A. Miller" <[email-masked]>
Date: Thu, 23 Mar 2017 16:37:36 -0500

On 17/03/23 15:49, John Cowan wrote:
> 
> On Thu, Mar 23, 2017 at 4:02 PM, HANSEN, TONY L <[email-masked]
> <mailto:[email-masked]>> wrote:
> 

Our sponsoring AD and I have been discussing this option.  Alexey is
willing to go through another IETF last call if we can get consensus.

To start, to what degree is UTF-8 encouraged?

0) don't encourage UTF-8 more than already is
1) SHOULD encode as UTF-8 for all usages
2) SHOULD encode as UTF-8 where the media type is 'application/json'
3) MUST encode as UTF-8 where the media type is 'application/json'
4) other -- please specify

- m&m

Matthew A. Miller
JSONbis Chair

---

From: Pete Cordell <[email-masked]>
Date: Thu, 23 Mar 2017 22:08:45 +0000

> 4) other -- please specify

(3) is where I'd like to be and where I believe most implementations are 
based on list discussion.  This would seem to align the bis RFC with 
what's deployed, which seems the best approach for interoperability.

Also (1) for when not 'application/json' media type.

Pete Cordell
Codalogic Ltd

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 Mar 2017 18:00:22 -0500

> 4) other -- please specify

(1) and (3)

I don't mind (4): MUST encode in UTF-8 for all uses, though I don't
think we need to go that far.

Nico
--

---

From: John Cowan <[email-masked]>
Date: Thu, 23 Mar 2017 19:29:58 -0400

On Thu, Mar 23, 2017 at 6:08 PM, Pete Cordell <[email-masked]>
wrote:

> based on list discussion.  This would seem to align the bis RFC with what's
> deployed, which seems the best approach for interoperability.
>

+1

---

From: "Manger, James" <[email-masked]>
Date: Mon, 27 Mar 2017 00:30:02 +0000

> can do this using a detection mechanism

Not quite. The detection only distinguishes UTF-8/16/32, not "encodings other than UTF-8".

> that is based on the fact
> greater than 0

Not quite. It is based on the fact that the (unescaped) character U+0000 cannot appear *anywhere* in valid JSON, and all valid first characters have code points less than 128.

> by inspecting the first octets for nulls.

Might as well be explicitly that inspecting up to the first 4 octets is sufficient. "null" is a special term in JSON so re-using it for a 0x00 octet isn't ideal.

> Text encoded in character encodings other than UTF-8,

The sections starts with "MUST be encoded in UTF-8, UTF-16, or UTF-32" so this sentence adds nothing. It just slightly muddies the water. Drop it.

I'll go for option 3) MUST encode as UTF-8 where the media type is 'application/json'.
Suggested text:

"""
  8.1. Character Encoding

  When exchanged as bytes, JSON text MUST be encoded in UTF-8.

  Earlier editions also allowed UTF-16 and UTF-32 encodings. However, those
  choices do not produce interoperable JSON as they are not supported by
  many implementations. UTF-16 and UTF-32 encodings of JSON can be distinguished
  from a UTF-8 encoding by the presence of an octet with value 0 within the first 2 octets,
  since valid first characters all have code points less than 128 and unescaped U+0000 characters
  are not allowed anywhere.

  <paragraph on the byte order mark>
"""

--
James Manger

-----Original Message-----
From: json [mailto:[email-masked]] On Behalf Of Matthew A. Miller
Sent: Friday, 24 March 2017 1:41 AM
To: [email-masked]
Subject: Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

Hello JSONbis,

"""

Thank you all,

--
- m&m

Matthew A. Miller
JSONbis Chair

---

From: John Cowan <[email-masked]>
Date: Sun, 26 Mar 2017 20:49:48 -0400

> Not quite. The detection only distinguishes UTF-8/16/32, not "encodings

It doesn't say "encodings other than UTF-8".  It says "Unicode encodings
other than UTF-8", viz. UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32LE, and
UTF-32BE.  If you wish to support any of these, you can use a detection
mechanism.

-- 
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
BALIN FUNDINUL          UZBAD KHAZADDUMU
BALIN SON OF FUNDIN     LORD OF KHAZAD-DUM

---

From: "Manger, James" <[email-masked]>
Date: Mon, 27 Mar 2017 01:22:03 +0000

Apologies. Perhaps “Unicode encodings” means UTF-8/16/32 — and doesn’t include UTF-1/5/6/7/9/18/EBCDIC, SCSU, BOCU-1, … — though explicitly list the 3 is clearer to me.

From: John Cowan [mailto:[email-masked]]
Sent: Monday, 27 March 2017 11:50 AM
To: Manger, James <[email-masked]>
Cc: [email-masked]
Subject: Re: [Json] FW: Call for Consensus: Proposed Text for "8.1 Character Encoding"

It doesn't say "encodings other than UTF-8".  It says "Unicode encodings other than UTF-8", viz. UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32LE, and UTF-32BE.  If you wish to support any of these, you can use a detection mechanism.

--
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]<mailto:[email-masked]>
BALIN FUNDINUL          UZBAD KHAZADDUMU
BALIN SON OF FUNDIN     LORD OF KHAZAD-DUM

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 27 Mar 2017 13:05:43 +0900

On 2017/03/23 23:40, Matthew A. Miller wrote:
> [UNICODE].

Independent of any other changes, please put "Section 3 of [UNICODE]" in 
parentheses, or otherwise make this sentence make more sense.

Regards,   Martin.

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 27 Mar 2017 13:18:36 +0900

> 4) other -- please specify

The more UTF-8, and the less of anything else, the better for 
interoperability.

So ideally, that would be "MUST use UTF-8".

The next would be "SHOULD use UTF-8, but MAY use UTF-16/32 if they are 
sure the intended recipient can process it". (This would go together 
with (4) above.)

The next would be (3).

I'm fine if we need another IETF last call to get closer to the above. 
I'm also fine if we don't need another last call, even if the consensus 
falls somewhere else.

I still don't see why some people insist on keeping UTF-16/UTF-32. As 
far as I see it, the fact that it has been mentioned in earlier RFCs 
isn't relevant content-wise (because it's just confusing to people to 
mention these encodings when all they need to do is use UTF-8), and also 
isn't relevant process-wise (it's a good idea to remove usused/unusable 
options when rewriting a spec, and the IETF does this all the time).

Regards,   Martin.

---

From: "Matthew A. Miller" <[email-masked]>
Date: Mon, 27 Mar 2017 15:04:36 -0500

Hello JSONBis,

I believe I see consensus for:

* MUST encode as UTF-8 where the media type is 'application/json'
* SHOULD encode as UTF-8 for all (other) usages.

In an attempt to expedite things, here is a proposal for text that I
think matches the consensus:

"""
JSON text SHOULD be encoded in UTF-8 (Section 3 of [UNICODE]); JSON
text MAY be encoded in UTF-16 or UTF-32 if the generator is certain
the intended recipients can process it. JSON text MUST NOT be encoded
in any encoding other than UTF-8, UTF-16, or UTF-32. When used with
media type "application/json" the JSON text MUST be encoded as UTF-8.

- m&m

Matthew A. Miller
JSONBis Chair

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 28 Mar 2017 13:16:38 +0900

Looks good. Ship it!    Regards,   Martin.

On 2017/03/28 05:04, Matthew A. Miller wrote:
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
Prof. Dr.sc. Martin J. Dürst
Department of Intelligent Information Technology
College of Science and Engineering
Aoyama Gakuin University
Fuchinobe 5-1-10, Chuo-ku, Sagamihara
252-5258 Japan

---

From: Tim Bray <[email-masked]>
Date: Mon, 27 Mar 2017 21:48:51 -0700

First of all, let me say that I’m delighted with, and fully support, the
promotion of the status of UTF-8 in the JSON RFC to MUST.  I suspect this
steps way outside the JSONbis charter, but that’s a problem for chairs and
ADs, not yr humble editor.

Comments on Matt's proposed text:

1. How about a very short historical note, along the lines of: “Previous
specifications of JSON, including the predecessor RFCs, have not required
the use of UTF-8 for use with the application/json media type.  However,
implementors of JSON-based software have overwhelmingly chosen to use the
UTF-8 encoding, to the extent that it is the only realistic way to achieve
interoperability in software which generates or consumes JSON.”

... moving on...

​​
O
​​
​​
n Mon, Mar 27, 2017 at 1:04 PM, Matthew A. Miller <
[email-masked]> wrote:
​​

> ​​
> ​​
> ​​
> ​​
> ​​

​2. Seriously, why the “JSON text MAY be encoded in… can process it ​”
phrase?  It’s a distraction, and if people want to do that, we can’t stop
them, but we shouldn't waste RFC space talking about practices that are not
remotely interoperable.  The I in IETF stands for Internet, and JSON on the
Internet is UTF-8, end of story.
​​

> be detected by inspecting the first octets for nulls.

​3. Is it just me, or does it feel really dorky to talk mysteriously about
this detection mechanism without providing details?  On top of which,
anyone who's writing the kind of software that might lead one to consult
​an RFC first shouldn't bloody well use anything but UTF-8.  If people
really want to have this, I think we owe the world an outline of the
algorithm, maybe in an appendix. I'll volunteer to make my best effort to
draft it and try to get consensus that it's correct..  If we can't, that's
a powerful symbol that we shouldn't have this language.  But that's my
fallback position; my real request to the group is that we just take this
out.

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Mar 2017 07:33:52 +0200

On 2017-03-28 06:48, Tim Bray wrote:
> promotion of the status of UTF-8 in the JSON RFC to MUST.  I suspect
> this steps way outside the JSONbis charter, but that’s a problem for
> chairs and ADs, not yr humble editor.
>
> specifications of JSON, including the predecessor RFCs, have not
> required the use of UTF-8 for use with the application/json media type.
> However, implementors of JSON-based software have overwhelmingly chosen
> to use the UTF-8 encoding, to the extent that it is the only realistic
> way to achieve interoperability in software which generates or consumes
> JSON.”
>
> ... moving on...

If we do this, we'll have to add it to the "changes from 7159" section.

> ​3. Is it just me, or does it feel really dorky to talk mysteriously
> about this detection mechanism without providing details?  On top of
> which, anyone who's writing the kind of software that might lead one to
> consult ​an RFC first shouldn't bloody well use anything but UTF-8.  If
> people really want to have this, I think we owe the world an outline of
> the algorithm, maybe in an appendix. I'll volunteer to make my best
> effort to draft it and try to get consensus that it's correct..  If we
> can't, that's a powerful symbol that we shouldn't have this language.
> But that's my fallback position; my real request to the group is that we
> just take this out.

That was proposed before; it seems some participants are opposed to 
saying "too much" about the detection, leading it to be implemented more 
than before.

Best regards, Julian

---

From: Pete Cordell <[email-masked]>
Date: Tue, 28 Mar 2017 09:57:25 +0100

> way to achieve interoperability in software which generates or consumes

If I ruled the world, I'd strip it down even more.  Merging with Matt's 
text, I'd go with something like:

"""
JSON text SHOULD be encoded in UTF-8 (Section 3 of [UNICODE]).  When
used with media type "application/json" the JSON text MUST be encoded
as UTF-8.

     Previous specifications of JSON, including the predecessor RFCs,
     have not required the use of UTF-8 for use with the
     application/json media type. However, implementors of JSON-based
     software have overwhelmingly chosen to use the UTF-8 encoding, to
     the extent that it is the only realistic way to achieve
     interoperability in software which generates or consumes JSON.

But I'm very happy with Matt's proposal, so please consider this more 
towards the "editorial typo" class than a blocker :-)

On:
 > ​2. Seriously, ... we shouldn't waste RFC space talking about
 > practices that are not remotely interoperable.  The I in IETF stands
 > for Internet, and JSON on the Internet is UTF-8, end of story.

The title of the RFC is "_The_ JavaScript Object Notation (JSON) Data 
Interchange Format".  To me that sounds like a fairly definitive 
definition of JSON.  As the IETF often writes stuff that's more general 
than just usage on the Internet, I think it's reasonable to believe that 
this text covers all JSON usages; and people would be justified to argue 
that.  If it is just for the Internet, we could tweak the title, add 
something to that effect to the Introduction, and just say MUST be UTF-8.

Thanks,

Pete Cordell
Codalogic Ltd