Subject: [Jsonpath] The draft: ambiguous language

From: Daniel P <[email-masked]>
Date: Mon, 7 Dec 2020 15:38:31 -0500

My general comment is that there is a lot of ambiguous language in the
draft, and ambiguous language is the bane of implementers. Some
examples below:

One. "Data Item: A structure complying to the generic data model of
JSON, i.e., composed of containers such as arrays and maps (JSON
objects), and of atomic data such as null, true, false, numbers, and
text strings."

Why introduce the term "map" as the preferred (unparenthized) term?
RFC 8259 doesn't mention "map", it uses JSON object.  By following
"arrays and maps" with "(JSON objects)", it could be read that "arrays
and maps" will collectively be referred to as "JSON objects". Later in
the draft we read "the root object" ($) and "the current object" (@),
which almost suggest that that is the intended meaning.

My own view is that the terminology should stay consistent with RFC
8259, and that the word "object" should not be used for items that are
not JSON objects in the sense of RFC 8259.

What is the purpose of "such as" in the sentence? Aren't the
itemizations exclusive?

Two. "Since a JSON data item is usually anonymous and doesn't
necessarily have a "root member object", JSONPath used the abstract
name $ to refer to the top level object of the data item."

I realize this sentence is mostly copied from Goessner, but I didn't
understand it there either. Regarding "doesn't necessarily" have a
"root member object", what is that supposed to mean? It seems to me
that the root is _always_ going to be an anonymous JSON value, which,
when Goessner was writing, could be a JSON array or object, and since
RFC 8259, any JSON value.

Three.  "Where a JSONPath processor uses JSONPath expressions as
output paths, these will always be converted to the more general
bracket-notation."

For output paths, Goessner uses the term "normalized path
expressions", which should be unambiguously defined. For uniqueness,
in addition to avoiding dot-notation, there would need to be other
restrictions, including avoiding the descendant operator .., and
filters.

Four. "The symbol @ is used for the current object."

Only if you interpret "object" to mean _any_ JSON value. In the
example above this sentence,

$.store.book[(@.length-1)].title

the symbol @ refers to a JSON array, with a JavaScript like property "length".

That arrays must support a JavaScript like property length is not
explicitly stated in the draft, but is implied in two examples. This
should be clarified. Not all implementations do, for example,
implementations that support functions that can be invoked at the tail
end of a path typically support a function call length() instead.

I also think the @ notation could be better explained, this particular
reader struggled to understand what was meant by "current object" in
original Goessner, and had to figure it out from examples.

Daniel

---

From: Tim Bray <[email-masked]>
Date: Mon, 7 Dec 2020 12:44:59 -0800

All those points granted, I think we should proceed and as a WG adopt the
draft, then you can turn all those issues you just raised into WG issues
and we can proceed to work through them in an orderly way.

On Mon, Dec 7, 2020 at 12:38 PM Daniel P <[email-masked]> wrote:

> --
> Jsonpath mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jsonpath
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 7 Dec 2020 23:37:37 +0100

On 2020-12-07, at 21:38, Daniel P <[email-masked]> wrote:
> draft,

As I said, work is needed.

I don’t plan to wait for adoption to start this work, so I created a PR:

https://github.com/jsonpath-standard/internet-draft/pull/46

> and ambiguous language is the bane of implementers. Some
> examples below:
> 
> RFC 8259 doesn't mention "map", it uses JSON object.  

Sure.  That is probably one of the mistakes that were made in defining the JSON terminology.

(Historically, what happened is that JavaScript has objects and atoms, but arrays are almost, but not entirely unlike objects.  When JSON was defined, it only called the maps “objects”, making the term object be ambiguous between its standard meaning and the “JSON object” meaning.  So to be unambiguous, it is best to have a term that unambiguously refers to “JSON objects”.  Maps naturally is that term, as would be longer terms such as dictionary.  If the WG does not like that term, we can of course revert to the more ambiguous JSON terminology.)

> By following
> and maps" will collectively be referred to as "JSON objects”.

Yep, that is better the other way around.

> Later in

That should be item.

> not JSON objects in the sense of RFC 8259.

I agree with the latter, but the reason for that is that the RFC 8259 terminology is confusing, so we could try to be more unambiguous.

> What is the purpose of "such as" in the sentence? Aren't the

Here, they are, so I replaced “such as” with “namely”.

> name $ to refer to the top level object of the data item."

Yep, item (and no need to open the “member” discussion, see below).

> RFC 8259, any JSON value.

Many people who actually work with JSON don’t think in terms of data items, but in terms of members (entries of a map, i.e. key/value pairs).  Because JSON values are data items and members cannot exist outside a map, this creates a need to wrap those members into maps, and to unwrap them again what talking about them as members.

See https://ietf-wg-asdf.github.io/SDF/sdf.html#name-sdfchoice for an example how that thinking looks like in practice (yes, I wrote that, fully cognizant of the dissonance).

But that prevalent thinking does not have to be reflected here, and I’m proposing a change in the PR.

> output paths, these will always be converted to the more general

Fixed in PR.

> Four. "The symbol @ is used for the current object."

Fixed in PR.

> end of a path typically support a function call length() instead.

Defining .length is part of the outstanding work on the expression language.

> original Goessner, and had to figure it out from examples.

Please send text.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Mon, 7 Dec 2020 14:50:18 -0800

To Carsten's point about what we call things, the number of distinguished
terms per RFC8259 is pretty small: JSON text, value, object, array, number,
string.  Having spent quite a bit of time specifying JSON DSLs, I find that
using just those terms doesn't seem to get in the way or cause problems, so
I'd argue that we should stick to them (and build up to higher-level
constructs as required for JSONPath).

On Mon, Dec 7, 2020 at 2:37 PM Carsten Bormann <[email-masked]> wrote:

> (Historically, what happened is that JavaScript has objects and atoms, but
> arrays are almost, but not entirely unlike objects.  When JSON was defined,
> it only called the maps “objects”, making the term object be ambiguous
> between its standard meaning and the “JSON object” meaning.  So to be
> unambiguous, it is best to have a term that unambiguously refers to “JSON
> objects”.  Maps naturally is that term, as would be longer terms such as
> dictionary.  If the WG does not like that term, we can of course revert to
> the more ambiguous JSON terminology.)
>
> > By following
> I agree with the latter, but the reason for that is that the RFC 8259
> terminology is confusing, so we could try to be more unambiguous.
>
> Many people who actually work with JSON don’t think in terms of data
> items, but in terms of members (entries of a map, i.e. key/value pairs).
> Because JSON values are data items and members cannot exist outside a map,
> this creates a need to wrap those members into maps, and to unwrap them
> again what talking about them as members.
>
> See https://ietf-wg-asdf.github.io/SDF/sdf.html#name-sdfchoice for an
> example how that thinking looks like in practice (yes, I wrote that, fully
> cognizant of the dissonance).
>

---

From: Tim Bray <[email-masked]>
Date: Mon, 7 Dec 2020 14:54:51 -0800

… oh, and I forgot the very useful "member".

On Mon, Dec 7, 2020 at 2:50 PM Tim Bray <[email-masked]> wrote:

> terms per RFC8259 is pretty small: JSON text, value, object, array,
> number, string.  Having spent quite a bit of time specifying JSON DSLs, I
> find that using just those terms doesn't seem to get in the way or cause
> problems, so I'd argue that we should stick to them (and build up to
> higher-level constructs as required for JSONPath).
>
>> (Historically, what happened is that JavaScript has objects and atoms,
>> but arrays are almost, but not entirely unlike objects.  When JSON was
>> defined, it only called the maps “objects”, making the term object be
>> ambiguous between its standard meaning and the “JSON object” meaning.  So
>> to be unambiguous, it is best to have a term that unambiguously refers to
>> “JSON objects”.  Maps naturally is that term, as would be longer terms such
>> as dictionary.  If the WG does not like that term, we can of course revert
>> to the more ambiguous JSON terminology.)
>>
>> > By following

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 8 Dec 2020 00:03:08 +0100

… and “element” (the things in arrays).

The two terms here that create a bit of a problem are JSON object (which, as I mentioned, conflicts with the normal use of that term) and JSON value.

The problem with JSON value is that it also can be quite confusing due to the usual use of that term.  Pointing to a tree and saying “the values inside that tree” is not going to be felt as equivalent to “the set of all subtrees of that tree, including the tree itself”.  But if JSON value is the only term we have, it has to be.  Hence my preference to talk about data items when I mean the items themselves and not their “value”.

Grüße, Carsten

> On 2020-12-07, at 23:54, Tim Bray <[email-masked]> wrote:
> 
> To Carsten's point about what we call things, the number of distinguished terms per RFC8259 is pretty small: JSON text, value, object, array, number, string.  Having spent quite a bit of time specifying JSON DSLs, I find that using just those terms doesn't seem to get in the way or cause problems, so I'd argue that we should stick to them (and build up to higher-level constructs as required for JSONPath).
>

---

From: Daniel P <[email-masked]>
Date: Mon, 7 Dec 2020 20:49:00 -0500

> > original Goessner, and had to figure it out from examples.
Related to that, it would be helpful to determine if JSONPath filters
apply to both JSON objects and arrays, or only to JSON arrays. Our
excellent Burgmer JSONPath comparison has that covered

https://cburgmer.github.io/json-path-comparison/results/filter_expression_on_object.html

and there is no consensus. For comparison, I note that JMESPath only defines
filters for arrays.

For JSON arrays, perhaps text along the following lines:

In a filter expression evaluated against a JSON array, each
item in the array is evaluated in sequence, and the current item
being evaluated is represented by the symbol @.

For JSON objects, it's less clear, but perhaps

In a filter expression evaluated against a JSON object, the object
itself is represented by the symbol @.

For Burgmer's case, that's consistent with the minority of implementations
that returned

 {"key": 42, "another": {"key": 1}}

Daniel

---

From: Glyn Normington <[email-masked]>
Date: Tue, 08 Dec 2020 04:00:18 +0000

> and there is no consensus. For comparison, I note that JMESPath only

I would support restricting filters to arrays, if others agree.

> itself is represented by the symbol @.

Regardless of the current consensus, or lack of one, I find this
confusing since @ does not correspond to the current item (meaning an
individual member of the object). It's also unhelpful in deciding which
items to filter in/out.

---

From: Daniel P <[email-masked]>
Date: Mon, 7 Dec 2020 23:59:56 -0500

> I would support restricting filters to arrays, if others agree.
That would be simplest.
> individual member of the object). It's also unhelpful in deciding which
The "current item" is what is being defined. In my suggested definition,
it is defined as the JSON object itself. With that definition, there can be
only one current object, and it is either selected or not. It is noted
that this definition is asymmetric with a filter expression evaluated
against a JSON array, as with a JSON array, the expression is
evaluated against its elements, and with a JSON object, the
expression is evaluated against the object itself (not its members.)

In Burgmer's case, the JSONPath expression "$[?(@.key)]" is applied to
the JSON instance {"key": 42, "another": {"key": 1}}.

Implementations that interpret the "current item" to be {"key": 42,
"another": {"key": 1}},
i.e. the whole object, find that @.key evaluates as true, and select the
whole object. It looks like that includes 9 implementations.

The other "minority opinion" result is [{"key": 1}]. It looks like that includes
13 implementations. I think that implies a definition:

    In a filter expression evaluated against a JSON object, the value part
    of each name-value pair is evaluated in succession, and the value
    part being evaluated is represented by the symbol @.

Daniel

---

From: Daniel P <[email-masked]>
Date: Thu, 10 Dec 2020 11:58:48 -0500

On Mon, Dec 7, 2020 at 11:00 PM Glyn Normington
<[email-masked]> wrote:
> > and there is no consensus.
>

I've created an issue for this at
https://github.com/jsonpath-standard/internet-draft/issues/47.
> individual member of the object). It's also unhelpful in deciding which
When an undefined term like "current item" occurs in a spec, some
readers (such as this one) may draw a complete blank, while others
may form an intuitive notion of what that means. But while
"an individual member of the object" may seem intuitive,
_none_ of the implementations in JSONPath Comparison support
that notion, in fact that would require notation like @.key and @.value,
which none support.

Daniel

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 10 Dec 2020 18:06:41 +0100

> that notion, in fact that would require notation like @.key and @.value,

The processing model obviously needs to be written up.

I left off https://tools.ietf.org/html/draft-goessner-dispatch-jsonpath-00#section-3 from the merge proposal, maybe I should have left it in as a quarry.

This says:

   Within a script, @ stands for the position under consideration.

(Where “script” would now be “expression language”.)

Positions are on data items (called JSON values in RFC 8259), not on map entries (called members in RFC 8259, which are not JSON values).

   Position:  A JSON data item identical to or nested within the JSON
      data item to which the query is applied to, expressed either by
      the value of that data item or by providing a JSONPath Output
      Path.

(This could have said “argument” instead of "the JSON data item to which the query is applied to”.)

I didn’t want to spend a lot of time on the processing model until we understand what direction the discussion on the handling of duplicate matches is taking.

Grüße, Carsten