Subject: [Json] I-D Action: draft-ietf-jsonbis-rfc7159bis-04.txt

From: [email-masked]
Date: Wed, 19 Jul 2017 06:45:11 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Javascript Object Notation Update WG of the IETF.

        Title           : The JavaScript Object Notation (JSON) Data Interchange Format
        Author          : Tim Bray
	Filename        : draft-ietf-jsonbis-rfc7159bis-04.txt
	Pages           : 15
	Date            : 2017-07-18

Abstract:
   JavaScript Object Notation (JSON) is a lightweight, text-based,
   language-independent data interchange format.  It was derived from
   the ECMAScript Programming Language Standard.  JSON defines a small
   set of formatting rules for the portable representation of structured
   data.

   This document removes inconsistencies with other specifications of
   JSON, repairs specification errors, and offers experience-based
   interoperability guidance.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-jsonbis-rfc7159bis/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-jsonbis-rfc7159bis-04
https://datatracker.ietf.org/doc/html/draft-ietf-jsonbis-rfc7159bis-04

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-jsonbis-rfc7159bis-04

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 19 Jul 2017 16:03:01 +0200

As requested by several people, this is the version that includes RFC Editor notes.

> On 19 Jul 2017, at 15:45, [email-masked] wrote:
>

---

From: Barry Leiba <[email-masked]>
Date: Thu, 20 Jul 2017 12:50:18 +0200

I've reviewed this version and I think it's ready to publish.

Barry

On Wed, Jul 19, 2017 at 4:03 PM, Alexey Melnikov
<[email-masked]> wrote:
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: Peter Saint-Andre - Filament <[email-masked]>
Date: Thu, 20 Jul 2017 19:42:49 -0600

I have three comments about this new sentence:

   When transmitting over a network protocol, or as a payload of a
   network protocol intended to be interpreted as part of a protocol,
   JSON text MUST be encoded in UTF-8 (Section 3 of [UNICODE]).

1. Is the term "network protocol" sufficiently clear? It might be
interpreted as a communications protocol at the network layer (e.g.,
IP), not as an application-layer protocol above the network layer -
which is where most or all usage of JSON occurs.

2. What are we trying to say with "as a payload of a network protocol
intended to be interpreted as part of a protocol"?

3. Why do we reference Section 3 of the Unicode Standard for the
definition of UTF-8, not RFC 3629? (Among other things, there's no
guarantee that Section 3 of http://www.unicode.org/versions/latest/ will
continue to talk about UTF-8.)

I suggest:

   When an entity transmits JSON text over a network, e.g. as the
   payload of an application protocol, it MUST encode that text using
   UTF-8 [RFC3629].

Peter

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Jul 2017 06:20:09 +0200

> which is where most or all usage of JSON occurs.

No, it's not totally clear, and that's indeed a problem when combined 
with a MUST.

> continue to talk about UTF-8.)

Right.

>     payload of an application protocol, it MUST encode that text using

This gets us back to the suggestion to apply this rule to the use of the 
application/json media type (and types derived from it).

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Jul 2017 07:14:52 +0200

On Jul 21, 2017, at 06:20, Julian Reschke <[email-masked]> wrote:
> This gets us back to the suggestion to apply this rule to the use of the application/json media type (and types derived from it).

The nit-picking opportunity here is that we don’t want to change NFS in case someone accesses a file with UTF-32 JSON in it over the network protocol NFS.  We could recognize this as nit-picking, or use a phrase such as “intended by the protocol to be JSON” — media types are not the only way such an intention could take shape.

(Re the previous discussion: We do not have to define “network protocol” further.  Sheesh.  But the point is that any protocol that claims to interchange JSON should define the JSON to be used in UTF-8 format, just as we define application/json to be UTF-8 format.)

Grüße, Carsten

---

From: Pete Cordell <[email-masked]>
Date: Fri, 21 Jul 2017 08:42:24 +0100

> The nit-picking opportunity here is that we don’t want to change NFS in case someone accesses a file with UTF-32 JSON in it over the network protocol NFS.  We could recognize this as nit-picking, o…

Clearly one person's nit-picking is another person's attempt to reduce 
confusion (especially of those not party to this discussion)!

I think Allen Wirts-Brock's "IETF-specified protocol" is the clearest 
and simplest 
(https://mailarchive.ietf.org/arch/msg/json/0TVI13NELfsuC7FscgDvXZThenE 
).  e.g.

     IETF-specified protocols MUST encode JSON text using UTF-8
     [RFC3629].

Pete Cordell
Codalogic Ltd
Read & write XML in C++, http://www.xml2cpp.com

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Jul 2017 09:51:27 +0200

On 2017-07-21 09:42, Pete Cordell wrote:
>>> This gets us back to the suggestion to apply this rule to the use of 
>>> the application/json media type (and types derived from it).
>>
>> The nit-picking opportunity here is that we don’t want to change NFS 
>> in case someone accesses a file with UTF-32 JSON in it over the 
>> network protocol NFS.  We could recognize this as nit-picking, or use 
>> a phrase such as “intended by the protocol to be JSON” — media types 
>> are not the only way such an intention could take shape.
> 

+1

> Read & write XML in C++, http://www.xml2cpp.com

-1

So if a specify a protocol outside the IETF, and use application/json as 
HTTP payload, I *can* use UTF-16?

Best regards, Julian

---

From: Pete Cordell <[email-masked]>
Date: Fri, 21 Jul 2017 09:27:07 +0100

On 21/07/2017 08:51, Julian Reschke wrote:
>> (https://mailarchive.ietf.org/arch/msg/json/0TVI13NELfsuC7FscgDvXZThenE ).  
>> e.g.
>>

Good point.  How about:

      IETF-specified protocols and media associated with the
      application/json media type, MUST encode JSON text using UTF-8
      [RFC3629].

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Jul 2017 15:35:04 +0200

>     application/json media type, MUST encode JSON text using UTF-8

This is not about the IETF.

If I buy a Mercedes, I want to get a car built by Daimler-Benz.
If people send me JSON, I want to get JSON (the UTF-8 form that everyone uses).

Now, there are other areas where, say, Mercedes is a woman’s name.  There might even be a restaurant called “Mercedes’ kitchen".  The objective cannot be that women or their restaurants no longer be called Mercedes (that the use of UTF-16 encoded JSON in C# or JavaScript becomes illegal).  It’s just when you sell a car as a Mercedes (send me an object as JSON), I want to get a Mercedes (JSON).

Consult a trade mark lawyer if that wasn’t clear :-)

Grüße, Carsten

---

From: Peter Saint-Andre - Filament <[email-masked]>
Date: Fri, 21 Jul 2017 09:35:20 -0600

On 7/20/17 11:14 PM, Carsten Bormann wrote:
>>> I suggest: When an entity transmits JSON text over a network,
>>> e.g. as the payload of an application protocol, it MUST encode
>>> that text using UTF-8 [RFC3629].
>> 
>> This gets us back to the suggestion to apply this rule to the use
>> of the application/json media type (and types derived from it).
> 
> (Re the previous discussion: We do not have to define “network
> protocol” further.  Sheesh.  But the point is that any protocol that
> claims to interchange JSON should define the JSON to be used in UTF-8
> format, just as we define application/json to be UTF-8 format.)

Hallo Carsten,

Improvements over my suggested text are welcome. Or do you think that
the text in -04 is good enough? I find it somewhat ambiguous.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Jul 2017 18:07:57 +0200

> the text in -04 is good enough? I find it somewhat ambiguous.

Hi Peter,

>>> When an entity transmits JSON text over a network,

… seems close enough for me.

It clearly has a nitpicking opportunity, but other attempts to improve the original text so far have made it worse.
I was mainly trying to point out that the mere fact that one can willfully pretend to misunderstand this (or the previous) text does not mean that it actually is ambiguous to a cooperating reader — the previous version is just a bit hard to read (and shouts out that we had trouble wordsmithing it).  I was then trying to point out what the essence of the statement is: The term JSON when used in a context that implies interchange (as opposed to representing the concept of JSON within a program or single system) also includes its encoding in UTF-8.  But I’m not trying to wordsmith here, I’ll leave that to the native speakers.

Grüße, Carsten

---

From: Peter Saint-Andre - Filament <[email-masked]>
Date: Fri, 21 Jul 2017 10:48:47 -0600

On 7/21/17 10:07 AM, Carsten Bormann wrote:
>> Improvements over my suggested text are welcome. Or do you think
>> that the text in -04 is good enough? I find it somewhat ambiguous.
> 
> Hi Peter,
> 
>>>> payload of an application protocol, it MUST encode that text
>>>> using UTF-8 [RFC3629].
> 

Great! I truly hesitated about posting to the list, but citing the
Unicode spec for the definition of UTF-8 pushed me over the edge. :-)

Peter

---

From: Pete Cordell <[email-masked]>
Date: Fri, 21 Jul 2017 22:43:24 +0100

> Consult a trade mark lawyer if that wasn’t clear :-)

Well I'm familiar with trade mark classes, thank you very much.

So I guess your rambling is trying to say JSON in the IETF class, is not 
the same as JSON in the ETSI class.  ETSI defines JSON at the code point 
level, whereas IETF defines it at the UTF-8 character encoding level.

However, I think your example illustrates that far from having the 
context of UTF-8 implied, we need to be explicit.  While you might be 
expecting a car when you you say "I'll have the Mercedes," if I'm 
starving I might be very disappointed if I say "I'll have the Mercedes" 
and end up with a car instead of a meal.  In the knowledge of such 
potential ambiguity you would be better saying "I'll have the Mercedes 
car," and I'd be better off saying "I'll have the Mercedes meal."

This is especially so as both ETSI JSON and whatever we're calling what 
we're defining here, likely both fall into trade mark Class 9, and hence 
a trade mark lawyer would ban one or the other on the grounds that 
people could get confused by them.

(If we really are saying the context is implicit, then all we need is:

     JSON text MUST be encoded using UTF-8 [RFC3629].
)

Pete Cordell
Codalogic Ltd

---

From: Pete Cordell <[email-masked]>
Date: Sat, 22 Jul 2017 08:36:39 +0100

> I was mainly trying to point out that the mere fact that one can willfully pretend to misunderstand this (or the previous) text does not mean that it actually is ambiguous to a cooperating reader —…

You call it nitpicking, but I have struggled more times than I care to 
remember with lazy text in specs and I don't like to knowingly inflict 
it on others.

Your person that is "willfully pretend[ing] to misunderstand" is more 
likely someone who is trying to understand the spec in the first place. 
I find it quite offensive to suggest that anyone that doesn't understand 
the wording the way you do is wilfully trying to misunderstand it.

Here's another attempt, based on the feedback:

     JSON text that is to be machine interpreted in a layer of a protocol
     stack directly related to underlying network activity (e.g.
     JSON text in a REST over HTTP application, as opposed to JSON text
     transported opaquely over a generic data transfer protocol), and
     JSON text identified with the application/json media type, MUST be
     encoded using UTF-8 [RFC3629].

If nothing else, it shouts out better that we had trouble wordsmithing 
it ;-)

Pete Cordell
Codalogic Ltd

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 22 Jul 2017 09:51:36 +0200

> On 21/07/2017 17:07, Carsten Bormann wrote:
<snip>
>       encoded using UTF-8 [RFC3629].

I don't have any text to offer but I believe that UFT-8 encoding is the
only recommendable way for transferring JSON data from one system to
another which includes lots of other use case than HTTP.  For example:
Writing JSON to databases, usage in WebSocket messages, or simply putting
JSON in file.

Yes, it seems to me that "Serialized format"/"External representation" is
really the common denominator rather than networks and/or protocols.

Anders

---

From: Julian Reschke <[email-masked]>
Date: Sat, 22 Jul 2017 09:53:30 +0200

On 2017-07-22 09:36, Pete Cordell wrote:
> ...
> ...

Whatever we say, it should make clear that it includes all derived media 
types (*/*+json) as well.

Best regards, Julian

---

From: Anders Rundgren <[email-masked]>
Date: Sat, 22 Jul 2017 11:18:02 +0200

Since the most basic use case of all obviously doesn't work well if people
use different character encodings (unless you do "byte sniffing" which seems
like a bad thing to *have* to do).

I would consider "external form" as an alternative to limiting the specification
to special cases like networks and or protocols.

Just my two "öres".

Anders

---

From: Pete Cordell <[email-masked]>
Date: Mon, 24 Jul 2017 16:12:29 +0100

> really the common denominator rather than networks and/or protocols.

And:

 > to special cases like networks and or protocols.

And:

On 22/07/2017 08:53, Julian Reschke wrote:
 > Whatever we say, it should make clear that it includes all derived
 > media types (*/*+json) as well.

Thanks Anders and Julian.

Makes me wonder whether the best approach is to start with what is 
currently the second paragraph of 8.1 as a 'softener', and then hit the 
reader with UTF-8 being the only real way to go for interoperability of 
open systems.  So maybe 8.1 becomes something like:

8.1.  Character Encoding

    Previous specifications of JSON have not required the use of UTF-8
    when transmitting JSON text.  However, the vast majority of JSON-
    based software implementations have chosen to use the UTF-8 encoding,
    to the extent that it is the only encoding that achieves
    interoperability.

    Therefore, JSON text exchanged between systems that don't form a
    closed ecosystem MUST be encoded using UTF-8.

    Implementations MUST NOT add a byte order mark (U+FEFF) to the
    beginning of a networked-transmitted JSON text.  In the interests of
    interoperability, implementations that parse JSON texts MAY ignore
    the presence of a byte order mark rather than treating it as an
    error.

Cheers,

Pete.