Subject: [Jsonpath] About duplicates, and the question thereof

From: Daniel P <[email-masked]>
Date: Tue, 17 Nov 2020 22:39:20 -0500

> From: Glyn Normington <[email-masked]>

> I added “is a union element" to remind the reader that an array slice is just one kind of
> union element. That may not be necessary.

But the term "union element" is used in
https://jsonpath-standard.github.io/internet-draft/ before it is
defined. I think it's better not to use an undefined term when it's
not needed.
>
> I wasn’t thinking of any equivalence between an array slice and a union of array indices, > although that’s an interesting observation, especially as we address the more general
> questions of duplication and ordering.
>
I'd like to understand what you mean by duplicates.

Given a JSON array

[0,1,1]

what would you expect the JSONPath expression

$..[0,1,2]

to yield? Without duplicates, it would give [0,1], but that's not what
the overwhelming majority of implementations yield. Or do you mean
something else?

Daniel

---

From: Marko Mikulicic <[email-masked]>
Date: Wed, 18 Nov 2020 08:23:11 +0100

It was my impression that we were talking about duplicated nodes not
duplicated values:

Given th array [10,20,30]

$..[0,1,0]

Would yield only two results [10, 20]

(Not that I'm advocating for removing duplicates, personally I think we
shouldn't)

On Wed, Nov 18, 2020, 04:39 Daniel P <[email-masked]> wrote:

> > I wasn’t thinking of any equivalence between an array slice and a union
> of array indices, > although that’s an interesting observation, especially
> as we address the more general
> --
> Jsonpath mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jsonpath
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Nov 2020 08:57:28 +0100

> On 2020-11-18, at 08:23, Marko Mikulicic <[email-masked]> wrote:
> 
> It was my impression that we were talking about duplicated nodes not duplicated values:
> 
> (Not that I'm advocating for removing duplicates, personally I think we shouldn’t)

You’re framing this as “removing duplicates”.
Another view is that [10, 20, 10] would be “adding duplicates” (copies of the same node).
Related are ordering issues:

$..[1,0]

➔
[20, 10]
Or
[10, 20]

I would expect the spec will leaves implementations some leeway here, but that should be based on an examination of existing implementations.

Grüße, Carsten

---

From: Tony Finch <[email-masked]>
Date: Wed, 18 Nov 2020 09:11:20 +0000

Carsten Bormann <[email-masked]> wrote:
>
> I would expect the spec will leaves implementations some leeway here,

I think that would be a mistake: if I ask for multiple items and I can't
predict what order they will be returned in, or whether duplicate items
will be present or not, how will I be able to use the result?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Selsey Bill to Lyme Regis: South veering west 5 to 7, occasionally gale 8
later. Moderate, becoming rough later. Rain for a time. Good, occasionally
moderate.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Nov 2020 10:13:48 +0100

> On 2020-11-18, at 10:11, Tony Finch <[email-masked]> wrote:
> 
> I think that would be a mistake:

That is a perfectly valid position, and we need to make up our mind about that.

> if I ask for multiple items and I can't

How do you use JSONpath today?

Grüße, Carsten

---

From: Tony Finch <[email-masked]>
Date: Wed, 18 Nov 2020 10:48:52 +0000

> How do you use JSONpath today?

I don't use JSONpath as such :-) but I do use adjacent things like jq and
the SQL/JSON path language (which have clear semantics for handling
multiple values), and gron (which uses a simplified JSONpath-like syntax).

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rockall: Cyclonic 5 to 7, becoming northwest 7 to severe gale 9. Rough or very
rough, becoming high for a time. Showers. Moderate or poor.

---

From: Glyn Normington <[email-masked]>
Date: Wed, 18 Nov 2020 11:21:08 +0000

(breaking out a separate thread so as not to distract from the main discussion in the previous thread)

> On 18 Nov 2020, at 03:39, Daniel P <[email-masked]> wrote:
> 
>> From: Glyn Normington <[email-masked] <mailto:[email-masked]>>
> 
> https://jsonpath-standard.github.io/internet-draft/ <https://jsonpath-standard.github.io/internet-draft/> before it is

I agree that definition before use is good practice for specs, but ABNF is typically presented "top-down”, so there’s a bit of a tension.

Perhaps a reasonable compromise in this case would be to define “union element” in general terms when it is first referred to, alluding to the alternative types of union element, but then continue to add the alternatives incrementally in later sections. What do you, and others, think?

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Nov 2020 12:29:19 +0100

> Perhaps a reasonable compromise in this case would be to define “union element” in general terms when it is first referred to, alluding to the alternative types of union element, but then continue…

We usually have definitions/terminology sections (Section 2 in most cases), which is where terms can be defined.

Top-down ABNF is fine if the production names have been chosen with that in mind (i.e., don’t call it k93yq but hexdig etc.).

Grüße, Carsten

---

From: Glyn Normington <[email-masked]>
Date: Wed, 18 Nov 2020 11:43:13 +0000

On 18 Nov 2020, at 11:29, Carsten Bormann <[email-masked]> wrote:
> 
> On 2020-11-18, at 12:21, Glyn Normington <[email-masked] <mailto:[email-masked]>> wrote:
>> 

I agree Section 2 is a good place to define general terms which need to be understood before reading the spec, but in this case, I think it’s simpler to define "union elements" in the section on unions. I doubt we’ll need to use the term outside that section. If we defined "union element" in Section 2, we’d need to define “union” there too and I worry that we’ll get into too much detail too early.

> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/jsonpath <https://www.ietf.org/mailman/listinfo/jsonpath>

---

From: Glyn Normington <[email-masked]>
Date: Wed, 18 Nov 2020 15:19:24 +0000

> I would expect the spec will leaves implementations some leeway here, but that should be based on an examination of existing implementations.

The example below is instructive. There is no convincing consensus (* - see the end of this email) among the implementations tested, although one result is far more popular than the others.

Apart from removal or (more typically) not of duplicated nodes the example also raises the spectre of non-deterministic, since JSON objects are unordered. It’s a spectre because all the implementations that manage to produce a result seem to agree on the order in which the object is traversed by .*, which I find disappointing.

https://cburgmer.github.io/json-path-comparison/results/union_with_repeated_matches_after_dot_notation_with_wildcard.html <https://cburgmer.github.io/json-path-comparison/results/union_with_repeated_matches_after_dot_notation_with_wildcard.html>

The example applies the selector $.*[0,:5] to the document:

{
  "a": [
    "string",
    null,
    true
  ],
  "b": [
    false,
    "string",
    5.4
  ]
}
The most popular result, given by 11 of the 40 implementations tested, is:

[
  "string",
  "string",
  null,
  true,
  false,
  false,
  "string",
  5.4
]
Just 4 of the implementations remove duplicate nodes from unions, thus:

[
  "string",
  null,
  true,
  false,
  "string",
  5.4
]
Only one removes duplicate values, as opposed to nodes, thus:

[
  "string",
  null,
  true,
  false,
  5.4
]
* "The consensus is based on the majority across all supported implementations. This is achieved by executing a query against all implementations and finally comparing the results. If a majority of half of all implementations (rounded up) plus one agree on an outcome we speak of a consensus. This guarantees difference of at least 2 votes even in critical cases.” — https://github.com/cburgmer/json-path-comparison/blob/master/FAQ.md

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 18 Nov 2020 21:11:05 +0100

On 2020-11-18, at 16:19, Glyn Normington <[email-masked]> wrote:
> 
> If a majority of half of all implementations (rounded up) plus one agree on an outcome we speak of a consensus. 

That is a rather rough consensus (we don’t vote in the IETF anyway).

But it is indeed strong guidance.

It has the flaw that it isn’t weighted (and that would be very hard to add in an agreeable way), and it is subject to vote stuffing (I can write ten more implementations in a week), so that kind of guidance should always taken with a grain of salt.  But it sure helps!

Grüße, Carsten

---

From: Daniel P <[email-masked]>
Date: Wed, 18 Nov 2020 21:03:18 -0500

> From: Daniel P <[email-masked]>
> From: Michael Kay <[email-masked]>
>
> Assuming a "node set" is a collection of nodes, what do you mean by
> "node"? Is it analogous to the use of "Element Node" in the XQuery and
> XPath Data Model as described in
> https://www.w3.org/TR/xpath-datamodel-31/#ElementNode?
>
> Indeed, I've been struggling with the same question (not just in the context of JsonPath, but also in devising primitives to assist with manipulation of JSON in XPath and XSLT).
>
> I think the key difficulty is whether each (key, value) pair in an object is "a thing" that can be identified and manipulated and potentially returned. (If we're talking analogies, then it's analogous to an attribute node in the XDM model).
>
It's an interesting question. I don't know of any JSON query language,
including JSONPath and JMESPath
(https://jmespath.org/specification.html), that supports the idea of
selecting a sequence of (key,value) pairs.

Daniel