Subject: Re: [Json] Call for Consensus: Proposed Text for "8.1 Character Encoding"

From: "Matthew A. Miller" <[email-masked]>
Date: Tue, 18 Jul 2017 10:27:43 +0200

Hello all,

I see consensus for the updated text below.  A note to the RFC editor is
added to ensure publication does not move forward until section 8.1 is
updated, and the change noted in the appendix.

"""
8.1.  Character Encoding

When transmitting over a network protocol, or as a payload of a
network protocol intended to be interpreted as part of a protocol,
JSON text MUST be encoded in UTF-8 (Section 3 of [UNICODE]).

Previous specifications of JSON have not required the use of UTF-8
when transmitting JSON text. However, the vast majority of
JSON-based software implementations have chosen to use the UTF-8
encoding, to the extent that it is the only encoding that achieves
interoperability.

Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a networked-transmitted JSON text.  In the interests
of interoperability, implementations that parse JSON texts MAY
ignore the presence of a byte order mark rather than treating it
as an error.
"""

Thank you all that participated in the discussion.

- m&m

Matthew A. Miller
JSONbis Chair

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 18 Jul 2017 17:45:15 +0900

Hello Matt, others,

This looks good. Ship it.

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Jul 2017 10:47:01 +0200

On 2017-07-18 10:27, Matthew A. Miller wrote:
> ...

1) I'm OK with the change.

2) I'm *not* OK with the way it's being executed:

- there should have been a new Internet Draft to review

- we are making an important normative change (that can make existing 
usages of JSON non-compliant) without getting IETF consensus for that change

- we're putting the RFC Production Center in charge to maintain our text 
(because the note to the RFC Editor does not say what to put into 
Appendix A)

Best regards, Julian

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 18 Jul 2017 10:34:07 +0100

Hi Julian,

> (because the note to the RFC Editor does not say what to put into 

I added a note for Appendix A. If people want to wordsmith it, it is not
too late for that.

---

From: Pete Cordell <[email-masked]>
Date: Tue, 18 Jul 2017 10:41:25 +0100

> JSON text MUST be encoded in UTF-8 (Section 3 of [UNICODE]).

I struggle here with "over a network protocol".  Perhaps we have a 
different idea of what "over a network protocol" means, but these days 
everything goes over a network protocol at some point in my mind.

Here's another stab at some wording in place of the first proposed 
paragraph:

     When transmitted as part of an IETF defined protocol, JSON text
     MUST be encoded in UTF-8 (Section 3 of [UNICODE]).

     Protocols defined outside the IETF are strongly encouraged to adopt
     a similar "MUST be encoded in UTF-8" requirement, but mandating this
     recommendation is beyond the scope of this specification.

Regards,

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Tue, 18 Jul 2017 09:39:39 -0700

> On May 13, 2017, at 10:37 AM, Tim Bray <[email-masked]> wrote:
> 
> I have one source of heartburn.  The stated purpose of 7159bis is to achieve harmony with ECMA 404.  That document does *not* constrain JSON encoding.  I’m fine  with saying that JSON has to be UTF-8, but I think we should consult with ECMA and see if the goal of spec harmonization is still possible if we make the UTF-8 restriction.

Sorry, I didn’t see this when it was originally posted.

I don’t believe that this would be a problem WRT ECMA-404 harmony. as long as the UTF-8 requirement applies to specific use cases such as “network protocols” or “IETF-specified protocols”.  ECMA-404 is only concern with defining which sequences of Unicode code points are well-foirmed JSON texts. It does not concern it self with how the code points might be encoded for transmission or storage. 

Concretely, within ECMAScript programs JSON texts are typically encoded as UTF-16 strings and UTF-8 encodings have to be translated to UTF-16 before the built-in JSON.parse function can be applied to them. But those requirements are part of ECMA-262, the  ECMAScript Language Specification, and not part of ECMA-404. 

Allen Wirts-Brock
Interim Chair, Ecma TC39

---

From: Tim Bray <[email-masked]>
Date: Tue, 18 Jul 2017 19:24:19 -0700

Oops, OK, I can bash out a quick draft, it's straightforward, will do so
unless chairs yell Stop.

However, I'm not going to be in favor of taking this to RFC status until we
hear that ECMA will proceed with the reconciliation.

On Tue, Jul 18, 2017 at 1:47 AM, Julian Reschke <[email-masked]>
wrote:

> (because the note to the RFC Editor does not say what to put into Appendix
> A)
>
> Best regards, Julian
>
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

-- 
- Tim Bray (If you’d like to send me a private message, see
https://keybase.io/timbray)

---

From: Julian Reschke <[email-masked]>
Date: Wed, 19 Jul 2017 07:42:48 +0200

> Oops, OK, I can bash out a quick draft, it's straightforward, will do so 

...you can use what I sent yesterday to the list.

> However, I'm not going to be in favor of taking this to RFC status until 
> we hear that ECMA will proceed with the reconciliation.

That's indeed another thing we need to make sure it happens.

Best regards, Julian

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 19 Jul 2017 07:20:24 +0100

> On 2017-07-19 04:24, Tim Bray wrote:
 (snip)
> That's indeed another thing we need to make sure it happens.

This will be my job in AUTH48.

---

From: Pete Cordell <[email-masked]>
Date: Wed, 19 Jul 2017 11:14:51 +0100

Since nobody else seems to struggle with "over a network protocol", I'll 
withdraw this proposed wording change and support Matt's proposed wording.

Roll the presses!

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------