Subject: Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

From: "Matthew A. Miller" <[email-masked]>
Date: Mon, 8 May 2017 10:02:32 -0600

Any other comments from the Working Group?  All things being on the
table, is mandating UTF-8 an acceptable outcome?

Is Pete's suggestion adequate?  I suspect not as it's very similar to
earlier text that was rejected, but confirmation would be helpful.

- m&m

Matthew A. Miller

On 4/28/17 8:23 AM, Pete Cordell wrote:
> Thanks.  I agree with your summary.
> 
> If you removed the sentence which mentions "UTF-16, or UTF-32" from the
> text that you proposed on 17 April, I'd be happy with that text.
> 
> Thanks again,
> 
> Pete.
> 
> On 27/04/2017 17:21, Matthew A. Miller wrote:
>> I see consensus for text in Section 8.1 pending an Appendix on encoding
>> detection, but nothing for the Appendix itself.
>>
>> Looking deeper in the threads again, it appears to me that:
>>
>> * There is consensus to say "just use UTF8" in many (most) scenarios
>> * There is rough consensus to say "always use UTF8", but concern this
>> change goes beyond the charter
>> * There is no consensus on a detection algorithm
>>
>> While the charter calls for "absolute minimal changes", it calls out RFC
>> 7159 (and its -bis) as documenting "interoperability concerns when
>> exchanging JSON over a network".  Documenting this interoperability is
>> one of its primary goals.
>>
>> Therefore, the argument is made that a change to the effect of "JSON
>> text MUST be encoded as UTF-8" is considered an "absolute minimum
>> change" in order to achieve the goal of network interoperability.
>>
>> To the working group:
>>
>> * Is there strong objection to mandating only UTF-8?
>> * Does anyone have suggested text to that effect?
>>
>>
>> - m&m
>>
>> Matthew A. Miller
>> JSONbis Chair

---

From: Julian Reschke <[email-masked]>
Date: Mon, 8 May 2017 18:37:30 +0200

> earlier text that was rejected, but confirmation would be helpful.

Here's a question that might be relevant wrt what we say in Section 8.

Is "JSON text" the same thing as what can be labelled 
"application/json"? Is "our" spec (as opposed to ECMA's) solely about 
what gets transferred as "application/json", or does it define more?

Best regards, Julian

---

From: Pete Cordell <[email-masked]>
Date: Wed, 10 May 2017 13:53:47 +0100

> what gets transferred as "application/json", or does it define more?

What would your input be for each of those two options?  (I know what 
mine would be.)

Cheers,

Pete.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 May 2017 15:08:39 +0200

> What would your input be for each of those two options?  (I know what

I believe we should have separate names for JSON represented as a 
sequence of characters (such as in a string variable in a programming 
language) and for a JSON-shaped octet sequence inside a 
"application/json"-typed (HTTP) message. For the latter, enforcing UTF-8 
IMHO is attractive.

I think it's ok for the spec to talk about both, but it really needs to 
be clear what we are talking about in each section.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 10 May 2017 15:38:25 +0200

On May 10, 2017, at 15:08, Julian Reschke <[email-masked]> wrote:
> 
> I believe we should have separate names for JSON represented as a sequence of characters (such as in a string variable in a programming language) and for a JSON-shaped octet sequence inside a "application/json"-typed (HTTP) message. For the latter, enforcing UTF-8 IMHO is attractive.

Do we do this a lot for other media types?
(Just wondering whether there are good parallels to draw.)

Ceterum censeo: the really interesting third level that we should be talking about is the (generic) data model.
(But that is not possible for this round of making the specification into Internet Standard.)

Grüße, Carsten

---

From: Pete Cordell <[email-masked]>
Date: Wed, 10 May 2017 15:04:27 +0100

> be clear what we are talking about in each section.

It's an interesting thought on JSON in programs.  It would be strange to 
be able to say it was not valid JSON if it was encoded in a string 
inside a Shift-JIS encoded Ruby program for example.

My ISO layers are rusty, but it looks like we can talk about JSON 
character sequences somewhere above the transport layer, and JSON 
encoded messages somewhere below the transport layer.  The latter 
possibly being transport specific.

To me it would seem discussion of "JSON" (without any further 
refinement) ought to be independent of the lower layer transport 
encoding aspects.

"application/json" would be one transport specific encoding (for which I 
think most are happy with only UTF-8).  JSON inside a Shift-JIS encoded 
Ruby program is in effect another form of transport for a JSON message.

Cheers,

Pete.

---

From: "Matthew A. Miller" <[email-masked]>
Date: Wed, 10 May 2017 11:13:06 -0600

> Ruby program is in effect another form of transport for a JSON message.

I believe in essence it is within our purview to set expectations of
what transits a wire protocol.

That phrasing is broad and likely vague.  To get a more specific, I
believe it is within scope to cover instances where a media type is
specified and that media type is "application/json" (e.g. HTTP bodies),
but I think it is also within scope for this document to essentially say
"where a protocol says 'use JSON here' then it is encoded as".  I don't
believe it is within scope to dictate JSON be encoded in any particular
manner when placed into a storage medium, or when embedded within other
content (e.g., the Shift-JIS encoded Ruby program even if said program
were transmitted over a network protocol).

Assuming the Working Group finds that scope acceptable and finds UTF-8
only acceptable, here is a starting proposal for text:

"""
8.1.  Character Encoding

When transmitting over a network protocol, JSON text MUST be
encoded in UTF-8 (Section 3 of [UNICODE]).

Previous specifications of JSON have not required the use of UTF-8
when transmitting JSON text. However, the vast majority of
JSON-based software implementations have chosen to use the UTF-8
encoding, to the extent that it is the only encoding that achieves
interoperability.

Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a JSON text.  In the interests of interoperability,
implementations that parse JSON texts MAY ignore the presence of a
byte order mark rather than treating it as an error.
"""

If you find this acceptable, please indicate that.  Otherwise, please
provide suggested changes.

- m&m

Matthew A. Miller

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 10 May 2017 11:42:50 -0600

> On May 10, 2017, at 11:13 AM, Matthew A. Miller <[email-masked]> wrote:
> 
> """
> 8.1.  Character Encoding
> 
> beginning of a JSON text.  

Suggestion: change "JSON text" here to "network-transmitted JSON text"
Rationale: it's not clear that this restriction applies to the JSON described up 2 paragraphs.

> In the interests of interoperability,

Other than my nit, +1

— 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Wed, 10 May 2017 15:05:16 -0400

On Wed, May 10, 2017 at 1:13 PM, Matthew A. Miller <
[email-masked]> wrote:

> provide suggested changes.

Emphatic +1

-- 
John Cowan          http://vrici.lojban.org/~cowan        [email-masked]
The Imperials are decadent, 300 pound free-range chickens (except they have
teeth, arms instead of wings, and dinosaurlike tails).  --Elyse Grasso

---

From: Pete Cordell <[email-masked]>
Date: Wed, 10 May 2017 21:30:34 +0100

> provide suggested changes.

A strong +1 to the spirit of the proposal.

I realise that the term "network protocol" is necessarily vague, but I 
wonder if it might be possible to avoid some confusion with the likes of 
NetBEUI (if that's still around) with a phrasing something like:

     When transmitted as the syntax of a network protocol, or as a
     payload of a network protocol intended to be interpreted as part of
     a protocol, JSON text MUST be encoded in UTF-8 (Section 3 of
     [UNICODE]).

Thanks,

Pete

---

From: Tim Bray <[email-masked]>
Date: Sat, 13 May 2017 10:37:01 -0700

​I’m meh on the network-or-not language.

I have one source of heartburn.  The stated purpose of 7159bis is to
achieve harmony with ECMA 404.  That document does *not* constrain JSON
encoding.  I’m fine  with saying that JSON has to be UTF-8, but I think we
should consult with ECMA and see if the goal of spec harmonization is still
possible if we make the UTF-8 restriction.

I’m much more positive about the UTF-8 restriction than the ECMA harmony
(which I think is ridiculous political bushwah) but at least the WG should
know the context before making a call.​

On Wed, May 10, 2017 at 1:30 PM, Pete Cordell <[email-masked]>
wrote:

> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)