Subject: [Jsonpath] On duplicates, and the consensus therewith

From: Daniel P <[email-masked]>
Date: Wed, 18 Nov 2020 11:35:24 -0500

> From: Glyn Normington <[email-masked]>
>
> The example below is instructive. There is no convincing consensus (* - see the end of this email) among the implementations tested, although one result is far more popular than the others.
>
> Apart from removal or (more typically) not of duplicated nodes the example also raises the spectre of non-deterministic, since JSON objects are unordered. It’s a spectre because all the implementations that manage to produce a result seem to agree on the order in which the object is traversed by .*, which I find disappointing.
>
> https://cburgmer.github.io/json-path-comparison/results/union_with_repeated_matches_after_dot_notation_with_wildcard.html
>
> The example applies the selector $.*[0,:5] to the document:
>
> {
>   "a": [
>     "string",
>     null,
>     true
>   ],
>   "b": [
>     false,
>     "string",
>     5.4
>   ]
> }
>
> The most popular result, given by 11 of the 40 implementations tested, is:
>
> [
>
>   "string",
>   "string",
>   null,
>   true,
>   false,
>   false,
>   "string",
>   5.4
> ]
>

This result (11 of 40) isn't necessarily representative, as many of
the implementations are failing on the slice component ":5".  Maybe we
can get Christoph Burgmer to add some simpler tests that cover
duplicates, if he hasn't already done so.

> Just 4 of the implementations remove duplicate nodes from unions, thus:
>
> [
>   "string",
>   null,
>   true,
>   false,
>   "string",
>   5.4
> ]
>
> Only one removes duplicate values, as opposed to nodes, thus:
>
> [
>   "string",
>   null,
>   true,
>   false,
>   5.4
> ]
>

And the author of that one is starting to think that's a defect.

Daniel

---

From: Glyn Normington <[email-masked]>
Date: Wed, 18 Nov 2020 16:50:03 +0000

> duplicates, if he hasn't already done so.

Christoph is responsive to PRs, so we can add our own examples if we can come up with them. Would [0,0] applied to [“a”] be sufficient for starters? Can you think of other distinct cases?

> And the author of that one is starting to think that's a defect.

Nice!

> 
> Daniel
> 
> -- 
> Jsonpath mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jsonpath

---

From: Daniel P <[email-masked]>
Date: Wed, 18 Nov 2020 12:01:50 -0500

> Christoph is responsive to PRs, so we can add our own examples if we can come up with them. Would [0,0] applied to [“a”] be sufficient for starters? Can you think of other distinct cases?
Perhaps that and "$..['a','a']" applied to {"a" : 1}

Daniel

---

From: Glyn Normington <[email-masked]>
Date: Wed, 18 Nov 2020 19:43:39 +0000

> Perhaps that and "$..['a','a']" applied to {"a" : 1}

Is the “..” essential? Not sure what it adds.

---

From: Glyn Normington <[email-masked]>
Date: Thu, 19 Nov 2020 16:23:16 +0000

> Is the “..” essential? Not sure what it adds.

I ran the two queries through the JSONPath comparison suite, which compares the behaviour of 40 implementations:

$[0,0] applied to ["a”] gave the result ["a", "a”] in 27 cases and ["a”] or “a” In 8 cases, so there is a clear consensus there.

$['a','a’] applied to {"a":1} gave [1, 1] in 9 cases and 3 cases with many implementations giving errors.

See https://gist.github.com/glyn/cfc750d65ee32f5354013cea9ce7475e <https://gist.github.com/glyn/cfc750d65ee32f5354013cea9ce7475e> for the two runs.

I also tried to add these to the comparison matrix, but I’ve failed to get a clean build so far. See https://github.com/cburgmer/json-path-comparison/pull/76 <https://github.com/cburgmer/json-path-comparison/pull/76>.

---

From: Daniel P <[email-masked]>
Date: Thu, 19 Nov 2020 13:11:26 -0500

I checked Goessner online, and the results were

$[0,0] applied to [0,1] yields [0,0]
$['a','a'] applied to {"a":1} yields [1,1]

I'm surprised that that broke so many other implementations
in Burgmer's comparison.

I was interested in what jayway, would do, as it was one
of the first, if not the first, to follow a well defined grammar,
and had a novel interpretation of the meaning of the union
operator applied to an object. A number of commercial
implementations were based on jayway, including
Oracle's JSONPath. From Burgmer's site, it looks like the
Bash JSONPath.sh and the Objective-C SMJJSONPath
mimic jayway behaviour.

In jayway

$[0,0] applied to [0,1] yields [0,0]
$['a','a'] applied to {"a":1} yields {"a":1}

In npm jsonpath (node.js), which describes a departure from
Goessner unions, "unions now yield real unions with no
duplicates rather than concatenated results",

$[0,0] applied to [0,1] yields 0 (without enclosing array)
$['a','a'] applied to {"a":1} yields 1 (without enclosing array)

Daniel

On Thu, Nov 19, 2020 at 11:23 AM Glyn Normington
<[email-masked]> wrote:
> See https://gist.github.com/glyn/cfc750d65ee32f5354013cea9ce7475e for the two runs.
>
> I also tried to add these to the comparison matrix, but I’ve failed to get a clean build so far. See https://github.com/cburgmer/json-path-comparison/pull/76.

---

From: Tim Bray <[email-masked]>
Date: Thu, 19 Nov 2020 10:25:04 -0800

> $['a','a'] applied to {"a":1} yields {"a":1}

I'm trying to build a mental model that explains the ['a','a'] behavior and
entirely failing. Does this make sense to anyone?

---

From: Daniel P <[email-masked]>
Date: Thu, 19 Nov 2020 14:23:36 -0500

That behaviour, and more generally jayway's interpretation of the
union operator applied to
objects, was ultimately accepted as a bug in 2017, see
https://github.com/json-path/JsonPath/issues/395.
It hasn't been fixed, and probably never will be, because of jayway's
wide legacy base, and the fix
would be a breaking change for its very considerable user base. And
judging from Stackoverflow
and other forums, users like that behaviour, not for reasons of
grammatical niceness, but for the
practical reason that it allows them to select from objects
(key-value) pairs, rather than values
disassociated from their keys.

By comparison, JMESPath introduces both a MultiSelect List,
https://jmespath.org/specification.html#multiselect-list,
which is analogous to but more general than a JSONPath union, and a
MultiSelect Hash,
https://jmespath.org/specification.html#multiselect-hash, which
results in a hash of (key-value) pairs,
but with the keys provided by the user.

Daniel

On Thu, Nov 19, 2020 at 1:25 PM Tim Bray <[email-masked]> wrote:
> I'm trying to build a mental model that explains the ['a','a'] behavior and entirely failing. Does this make sense to anyone?
>>
>>

---

From: Glyn Normington <[email-masked]>
Date: Thu, 19 Nov 2020 19:44:17 +0000

On 19 Nov 2020, at 18:25, Tim Bray <[email-masked]> wrote:
> 
> 
> 
> On Thu, Nov 19, 2020 at 10:11 AM Daniel P <[email-masked] <mailto:[email-masked]>> wrote:
> In jayway
> 

I think the term “union” is poor. If we think of it as concatenation of results, then the result is as expected.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 19 Nov 2020 20:50:10 +0100

> I'm trying to build a mental model that explains the ['a','a'] behavior and entirely failing. Does this make sense to anyone?

I sure can hypothesize a mental model from this one data point (select entries, i.e., key/value pairs, not just the values; just output the selected subset).
But of course this model would need to be verified with a few other test cases.
(Or one could read the code :-)

Grüße, Carsten

---

From: Daniel P <[email-masked]>
Date: Thu, 19 Nov 2020 15:40:19 -0500

On Thu, Nov 19, 2020 at 2:44 PM Glyn Normington
<[email-masked]> wrote:
>
> I think the term “union” is poor.

I agree with that comment, but it's partly because I'm used to SQL UNION,
which is different. I prefer the JMESPath term for an analogous construct,
MultiSelect List, https://jmespath.org/specification.html#multiselect-list.

Daniel

---

From: Daniel P <[email-masked]>
Date: Thu, 19 Nov 2020 16:27:25 -0500

> disassociated from their keys.
Here are a couple of examples of folks that really like the jayway
approach,

https://github.com/dchester/jsonpath/issues/86

https://github.com/danielaparker/jsoncons/issues/198

Daniel

---

From: Daniel P <[email-masked]>
Date: Fri, 20 Nov 2020 12:19:35 -0500

> I think the term “union” is poor. If we think of it as concatenation of results, then the result is as expected.

I think you could argue that. But it's inconsistent with the "union"
operator applied to arrays. In "consensus" JSONPath (consistent with
Goessnner and Burgmer's consensus), both the union operator applied
to arrays, and the union operator applied to objects, are lossy. In the former
case, we lose the index position associated with the value, in the latter,
we lose the key associated with the value. jayway's approach is lossy with
the "union" operator applied to arrays, but retains the association
between key and value with the union operator applied to objects.

For reference, see
https://cburgmer.github.io/json-path-comparison/results/union_with_keys.html.

On the "consistency be damned!" side, it does seem like "lost indexes"
are of less concern to real users than "lost keys", judging from the weeping
and gnashing of teeth exhibited by jaway people in the non-jayway universe,
such as in this one,

Daniel

---

From: Glyn Normington <[email-masked]>
Date: Mon, 23 Nov 2020 10:33:24 +0000

On 19 Nov 2020, at 19:44, Glyn Normington <[email-masked]> wrote:
> 
> On 19 Nov 2020, at 18:25, Tim Bray <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 

>> $['a','a'] applied to {"a":1} yields {"a":1}

which was collapsed by my email client.

My mental model doesn’t accommodate this behaviour either, so I’m relieved it’s regarded as a (permanent?) bug.

---

From: Glyn Normington <[email-masked]>
Date: Mon, 23 Nov 2020 11:19:28 +0000

> I also tried to add these to the comparison matrix, but I’ve failed to get a clean build so far. See https://github.com/cburgmer/json-path-comparison/pull/76.

Christoph Burgmer kindly came to my aid and now we have two more rows in the matrix:

Union with duplication from array (consensus): https://cburgmer.github.io/json-path-comparison/results/union_with_duplication_from_array.html

Union with duplication from object (no consensus): https://cburgmer.github.io/json-path-comparison/results/union_with_duplication_from_object.html