Subject: Re: [secdir] [Json] secdir review of draft-ietf-jsonbis-rfc7159bis-03

From: Elwyn Davies <[email-masked]>
Date: Sun, 12 Mar 2017 23:07:40 +0000

(with half a Gen-art hat on...)
Does the WG really want to revisit the anguished discussions that resulted in the changes to Section 8.1 of draft-ietf-json-rfc4627bis between versions 07 and 08 back in late November 2013?
See https://www.ietf.org/mail-archive/web/json/current/msg02053.html and many, many messages beore this.
Cheers,Elwyn

Sent from Samsung tablet.
-------- Original message --------From: Peter Cordell <[email-masked]> Date: 12/03/2017  09:06  (GMT+00:00) To: Ned Freed <[email-masked]>, Julian Reschke <[email-masked]> Cc: [email-masked], John Cowan <[email-masked]>, [email-masked], [email-masked], [email-masked], Benjamin Kaduk <[email-masked]> Subject: Re: [Json] secdir review of draft-ietf-jsonbis-rfc7159bis-03 
On 11/03/2017 15:41, Ned Freed wrote:
>> On 2017-03-11 03:08, John Cowan wrote:
>> >
>> > On Thu, Mar 9, 2017 at 12:53 AM, Benjamin Kaduk <[email-masked]
>> > <mailto:[email-masked]>> wrote:
>> >
>> >     If that's what's supposed to happen, it should probably be more
>> >     clear, yes.  (But aren't there texts that have valid
>> interpretations
>> >     in multiple encodings?)
>> >
>> >
>> > Not if the content is well-formed JSON and the only possible encodings
>> > are UTF-8, UTF-16, and UTF-32.  It suffices to examine the first four
>> > bytes of the input.  If there are no NUL bytes in the first four bytes,
>> > it is UTF-8; if there are two NUL bytes, it is UTF-16; if there are
>> > three NUL bytes, it is UTF-32.  This works because the grammar requires
>> > the first character to be in the ASCII repertoire, and the NUL
>> > *character* (U+0000) is not allowed at all.
>
>> Good explanation. Maybe the spec should include it.
>
> +1
>
> This exact issue just came up in a media type review, where someone
> specified a charset parameter because they weren't aware of this algorithm.
>
> It would be very helpful to have this text in the RFC.

Although it does need slightly more detail to take into account 
endian-ness in the case of UTF-16 and -32.

The XML spec may offer some example text:

https://www.w3.org/TR/2008/REC-xml-20081126/#sec-guessing

Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 07:23:20 +0100

On 2017-03-13 00:07, Elwyn Davies wrote:
> Does the WG really want to revisit the anguished discussions that
> resulted in the changes to Section 8.1 of draft-ietf-json-rfc4627bis
> between versions 07 and 08 back in late November 2013?
>
> Cheers,
> Elwyn

No, but on the other hand, we should acknowledge that apparently the 
text both about what's mandatory and how auto detection works is not as 
clear as it should.

Best regards, Julian

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 13 Mar 2017 16:51:58 +0900

> text both about what's mandatory and how auto detection works is not as

It looks to me as if at the time of the above message in the WG, the 
chairs were successful in presenting a consensus, probably at a stage 
when the participants in the discussion where getting tired.

It seems that when put in the wider context of the IETF, that compromise 
now looks somewhat shaky.

My personal opinion is that we could try to fix this by changing the 
following:

    JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32 [UNICODE]
    (Section 3).  The default encoding is UTF-8, and JSON texts that are
    encoded in UTF-8 are interoperable in the sense that they will be
    read successfully by the maximum number of implementations; there are
    many implementations that cannot successfully read texts in other
    encodings (such as UTF-16 and UTF-32).

to something like the following:

    JSON text SHOULD be encoded in UTF-8 [UNICODE]
    (Section 3).  JSON texts that are
    encoded in UTF-8 are interoperable in the sense that they will be
    read successfully by the maximum number of implementations.

    There are
    many implementations that cannot successfully read texts in other
    encodings (such as UTF-16 and UTF-32). JSON text MAY be encoded in
    UTF-16 or UTF-32 [UNICODE] (Section 3) if the sender is sure that
    the intended recipients can read them.

That should then go together with a MIME registration that only lists UTF-8.

Regards,   Martin.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 16:53:29 +0100

+1000

Grüße, Carsten

> On 13 Mar 2017, at 08:51, Martin J. Dürst <[email-masked]> wrote:
> 
> It looks to me as if at the time of the above message in the WG, the chairs were successful in presenting a consensus, probably at a stage when the participants in the discussion where getting tired.
> 
> It seems that when put in the wider context of the IETF, that compromise now looks somewhat shaky.
> 
> My personal opinion is that we could try to fix this by changing the following:
> 
> >>>>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 16:56:33 +0100

+1000

Grüße, Carsten

---

From: Stefan Hagen <[email-masked]>
Date: Mon, 13 Mar 2017 16:58:38 +0100

Yes, tired is a good match && I second the proposed change.
"Stefan"

---

From: Stefan Hagen <[email-masked]>
Date: Mon, 13 Mar 2017 17:01:52 +0100

---

From: Peter Cordell <[email-masked]>
Date: Mon, 13 Mar 2017 16:20:14 +0000

>    the intended recipients can read them.

My only thought is to perhaps reflect that JSON isn't only transmitted, 
and JSON can be used for file based configuration etc, (even if this 
isn't strictly IETFs concern).  So perhaps s/sender/encoder/ in the last 
sentence, plus a few other tweaks yielding something like:

     There are many implementations that cannot successfully read texts
     in other encodings (such as UTF-16 and UTF-32 [UNICODE]).  JSON
     text MAY be encoded in other encodings if the encoder is sure that
     the intended recipients can read them.

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
http://json-content-rules.org/
---------------------------------------------------------------------

---

From: Peter Cordell <[email-masked]>
Date: Mon, 13 Mar 2017 16:33:33 +0000

Pete.

---

From: "Matthew A. Miller" <[email-masked]>
Date: Mon, 13 Mar 2017 11:54:31 -0600

>     the intended recipients can read them.

/me doffs hat

I like this change myself.

/me dons hat

As I recall, the table was removed mostly because the vast majority of
implementations did not support any encoding other than UTF-8, and no
one (that I recall) reported implementing the detection table.

- m&m

Matthew A. Miller
< http://goo.gl/LM55L >

---

From: Nico Williams <[email-masked]>
Date: Mon, 13 Mar 2017 13:36:29 -0500

> That should then go together with a MIME registration that only lists UTF-8.

+1.

I would restore the text from an older draft about how counting the
number of NULs in the first four bytes can be used to determine which of
UTF-8, 16, or 32 the text is encoded in, modified to avoid all talk of
BOMs (which had no consensus):

   Implementors MAY count the number of ASCII NULs in the first four
   bytes of any JSON text to detect which of UTF-8, UTF-16, or UTF-32
   the text is encoded in:

    - if the count is zero, then the text is encoded in UTF-8
    - if the count is one or two, then the text is encoded in UTF-16
    - if the count is three, then the text is encoded in UTF-32

   This results from a) JSON texts having to start with an ASCII
   character, b) no unescaped NULs being allowed in JSON strings, and c)
   any type being allowed at the top-level, thus the first character may
   be a double-quote and the second may be any permissible, unescaped
   Unicode codepoint.  An ASCII character requires a NUL-valued byte in
   UTF-16 encoding, three in UTF-32, and none in UTF-8.

Note the "MAY".  The idea is that we should be sending (and storing)
JSON texts encoded in UTF-8 per the text I +1'ed above, so we shouldn't
bother recommending or requiring the use of this UTF detection scheme.
But there's no harm in describing it.

See https://www.ietf.org/mail-archive/web/json/current/msg02053.html

The encoding detection algorithm can be optimized a bit since if neither
the first nor the second bytes are NUL-valued then the encoding must be
UTF-8.

The algorithm can also be combined with an understanding of byte
ordering to detect encoding by looking only at the first two bytes in
some cases, but whatever.

ASIDE:

  I don't see any reason not to also include text discussing byte-order
  detection in the UTF-16 and UTF-32 cases, but I don't care very much
  about it, and given the prior lack of consensus for it, there's no
  need to relitigate that point now.  Also, it's pretty obvious how to
  detect byte-order anyways, at least given the text on how to detect
  encoding: just looking at the first two bytes will tell you what the
  byte order is!

  If the first byte is zero-valued, then the byte order is big-endian,
  if the first is not zero-valued but the second is, then the byte order
  is little-endian, and if both, the first and second bytes are
  zero-valued, then the byte order is big-endian; if neither the first
  nor the second bytes are zero-valued then the encoding is UTF-8 and
  byte-order is irrelevant.

Cheers,

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 20:07:56 +0100

On 13 Mar 2017, at 18:54, Matthew A. Miller <[email-masked]> wrote:
> no

There even was one that implemented something for UTF-16/32: Jackson.
The author said:

"Most other
libraries just punt the problem by requiring caller to pass in Reader that
does byte->char decoding”

(He did not say whether his code was ever exercised outside of his test cases :-)

He also mentioned that, by contrast, most XML libraries do implement auto-detection (as they should).

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 13 Mar 2017 20:51:27 +0100

On 2017-03-13 18:54, Matthew A. Miller wrote:
> ...
> ...

Well, if we allow UTF-16 and UTF-32 - even if we discourage their use - 
we should say how to detect those...

Alternatively we could forbid them, but that would be a normative change 
from RFC 7159.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 13 Mar 2017 20:57:00 +0100

On 13 Mar 2017, at 20:51, Julian Reschke <[email-masked]> wrote:
> 
> Well, if we allow UTF-16 and UTF-32 - even if we discourage their use - we should say how to detect those...

Consistency is the hobgoblin of the little minds.
(Apparently, not having that nonsense in there did not hurt for 7159?
If people want to implement something useless and cannot figure this stuff out, should we help them?)

Putting back any form of apparent support for UTF-16 and UTF-32 is a regression.

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Tue, 14 Mar 2017 09:58:39 -0400

On Mon, Mar 13, 2017 at 3:57 PM, Carsten Bormann <[email-masked]> wrote:

The actual quotation says "foolish consistency".  But not all consistency
is foolish.

> Putting back any form of apparent support for UTF-16 and UTF-32 is a

In my opinion, we should either take all references to UTF-16 or UTF-32
out, or add back a correct detection algorithm. UTF-8 is what is actually
interoperable, and interoperability is what RFCs are supposed to be all
about.

--

---

From: Peter Cordell <[email-masked]>
Date: Thu, 16 Mar 2017 10:28:28 +0000

On 14/03/2017 13:58, John Cowan wrote:
> out, or add back a correct detection algorithm. UTF-8 is what is
> actually interoperable, and interoperability is what RFCs are supposed
> to be all about.

Combining the original (Tim's) & Martin's proposal with my tweak, plus 
John & Carsten's direction, how about:

8.1.  Character Encoding

    JSON text SHOULD be encoded in UTF-8 [UNICODE] (Section 3).  JSON
    texts that are encoded in UTF-8 are interoperable in the sense that
    they will be read successfully by the maximum number of
    implementations.

    There are many implementations that cannot successfully read texts
    in other encodings.  JSON text MAY be encoded in other encodings if
    the generator is sure that the intended parsers can read them.

    Implementations MUST NOT add a byte order mark to the beginning of a
    JSON text.  In the interests of interoperability, implementations
    that parse JSON texts MAY ignore the presence of a byte order mark
    rather than treating it as an error.

Are "generator" and "parser" the correct terms to use in this instance, 
or does that functionality sit above the character encoding layer?

Pete Cordell
Codalogic Ltd
Rules for Describing JSON Content, http://json-content-rules.org

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Mar 2017 11:49:03 +0100

> or does that functionality sit above the character encoding layer?

Not convinced.

a) It's not constrained to UTF-8/16/32, so people might decide to 
support ISO-8859-1, or UTF-7-

b) It doesn't state that the only way to support encodings other than 
UTF-8 is to inspect the leading octets for zeros (or their lack of).

Best regards, Julian

---

From: Peter Cordell <[email-masked]>
Date: Thu, 16 Mar 2017 11:23:47 +0000

> support ISO-8859-1, or UTF-7-

Why is that a problem if the generator knows the parser can read it?  If 
someone wants to use EBCDIC for whatever reason, are they not allowed to 
call it JSON?

> UTF-8 is to inspect the leading octets for zeros (or their lack of).

UTF detection is one way.  It's not the only way.  If you want to go the 
UTF detection way or some other way, rfc7159bis shouldn't prevent it, 
but it doesn't have to tell you how to do it.

Cheers,

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Julian Reschke <[email-masked]>
Date: Thu, 16 Mar 2017 12:35:39 +0100

> someone wants to use EBCDIC for whatever reason, are they not allowed to

For application/json, it would violate a SHOULD-level requirement... 
<https://greenbytes.de/tech/webdav/rfc7159.html#rfc.section.8.1.p.1>:

"JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32. The default 
encoding is UTF-8, and JSON texts that are encoded in UTF-8 are 
interoperable in the sense that they will be read successfully by the 
maximum number of implementations; there are many implementations that 
cannot successfully read texts in other encodings (such as UTF-16 and 
UTF-32)."

So I agree it's technically allowed.

> but it doesn't have to tell you how to do it.

Oh, it's not the only way?

How do we interoperate then?

Best regards, Julian

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 16 Mar 2017 12:38:45 +0000

> On 16 Mar 2017, at 11:35, Julian Reschke <[email-masked]> wrote:
> 
> For application/json, it would violate a SHOULD-level requirement... <https://greenbytes.de/tech/webdav/rfc7159.html#rfc.section.8.1.p.1>:
> 
> "JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32. The default encoding is UTF-8, and JSON texts that are encoded in UTF-8 are interoperable in the sense that they will be read successfully by the maximum number of implementations; there are many implementations that cannot successfully read texts in other encodings (such as UTF-16 and UTF-32)."
> 

As this document is intended to be Internet Standard, it should strive to remove number of choices and generally non interoperable features. So listing the minimal list of allowed encodings in this document would be a good thing.