Subject: Re: [TOOLS-DEVELOPMENT] xmlDIff-00

From: Sandy Ginoza <[email-masked]>
Date: Mon, 10 Nov 2014 09:35:31 -1000

Hi Robert,

At the last IETF, we talked about having a tool that would allow us to run diffs across formats.  This suggestion came up again at the IESG breakfast when the members were discussing how the review documents while reviewing Internet-Drafts. 

This tool would be extremely helpful to the RPC (and others, it sounds like) in ensuring that diffs appear only where we expect them (e.g., the text file might have “see Figure 1 in the PDF” and the PDF would just have the image as expected).  This would allow us to quickly see that these discrepancies only appear where we expect them.  

Can we add this as a feature to this writeup?  Or is this meant to focus on source file diffs only? 

Thanks!
Sandy

On Jun 26, 2014, at 8:00 AM, Robert Sparks <[email-masked]> wrote:

> Attached is another strawman to give us a place to start discussion from.
> IIRC xml differencing in the general case has some hard problems.
> Is what this strawman starts to describe feasible?
> Does anyone know of programs that do a lot of this already
> (that would have a similar relation that wdiff has to rfcdiff)?
> 
> RjS
> 
> <xmlDiff-00.pdf><xmlDiff-00.docx>_______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Robert Sparks <[email-masked]>
Date: Mon, 10 Nov 2014 09:40:39 -1000

The xmlDiff tool focuses on the source files.

We already have a really good text diff tool, that I expect will work 
with no, or very little, adjustment.
HTML and PDF diffing are different problems. We can look into what it 
would take to provide good differencing there, but that shouldn't be 
part of this xmldiff effort.

RjS

---

From: Sandy Ginoza <[email-masked]>
Date: Mon, 10 Nov 2014 09:47:38 -1000

> The xmlDiff tool focuses on the source files.

Ok.

> We already have a really good text diff tool, that I expect will work with no, or very little, adjustment.
> HTML and PDF diffing are different problems. We can look into what it would take to provide good differencing there, but that shouldn't be part of this xmldiff effort.

I’m not sure I’m explaining myself.  I’m hoping for a tool that will diff across formats; e.g., compares .txt with .html/.pdf output.  There should be no differences between .html/.pdf but there might be differences between .html/.pdf.  I’m hoping for a tool that makes it easy for us to identify the differences in output of the various formats.

Thanks,
Sandy

---

From: Robert Sparks <[email-masked]>
Date: Mon, 10 Nov 2014 09:55:28 -1000

> I’m not sure I’m explaining myself.  I’m hoping for a tool that will diff across formats; e.g., compares .txt with .html/.pdf output.  There should be no differences between .html/.pdf but there mi…
Ok. I hadn't gotten that yet.

I'll think that through, but my initial reaction is that you're asking 
for something very hard (Artificial Intelligence level hard).

Instead, perhaps we should be talking about adding requirements to the 
rendering tools that issue warnings when they know they're doing 
something that will result in what you would consider to be a difference 
between the rendered formats?

---

From: Tony Hansen <[email-masked]>
Date: Mon, 10 Nov 2014 19:14:21 -0500

On 11/10/14, 2:55 PM, Robert Sparks wrote:
>>> We already have a really good text diff tool, that I expect will 
>>> work with no, or very little, adjustment.
>>> HTML and PDF diffing are different problems. We can look into what 
>>> it would take to provide good differencing there, but that shouldn't 
>>> be part of this xmldiff effort.
>> I’m not sure I’m explaining myself.  I’m hoping for a tool that will 
>> diff across formats; e.g., compares .txt with .html/.pdf output.  
>> There should be no differences between .html/.pdf but there might be 
>> differences between .html/.pdf.  I’m hoping for a tool that makes it 
>> easy for us to identify the differences in output of the various 
>> formats.

If we were to do a simple .txt vs .html comparison, there are certainly 
things that would keep showing up as false positives because of the 
inherent differences between the formats. For example, the .txt might 
format links in a way different from the .html; headers and footers are 
in the .txt and not in the .html; the order of data in the document 
header may be totally different; asides may wind up in different order 
within the documents; etc.

With PDF, there is the addiitonal issue that blocks of words have 
coordinates associated with them, and those coordinates need to be 
ordered prior to looking at the underlying text.

Working around those issues is where the difficulty could come in.

> something that will result in what you would consider to be a 
> difference between the rendered formats?

I think this sentence is totally missing Sandy's point. It's not the 
KNOWN and EXPECTED differences that Sandy is worried about, but the 
UNKNOWN differences.

     Tony Hansen

---

From: Robert Sparks <[email-masked]>
Date: Mon, 10 Nov 2014 14:33:01 -1000

On 11/10/14 2:14 PM, Tony Hansen wrote:
>>>> it would take to provide good differencing there, but that 
>>>> shouldn't be part of this xmldiff effort.
>> I'll think that through, but my initial reaction is that you're 
>> asking for something very hard (Artificial Intelligence level hard).
>
> If we were to do a simple .txt vs .html comparison, there are 
> certainly things that would keep showing up as false positives because 
> of the inherent differences between the formats. For example, the .txt 
> might format links in a way different from the .html; headers and 
> footers are in the .txt and not in the .html; the order of data in the 
> document header may be totally different; asides may wind up in 
> different order within the documents; etc.
>
>> Instead, perhaps we should be talking about adding requirements to 
>> the rendering tools that issue warnings when they know they're doing 
Something about unknowns unknowns?

I don't think I've missed her point, actually.

There is one activity where you look at places that are different in the 
renderings on purpose to make sure the differences are what we expect.
I think it will be relatively easy to help see those efficiently by 
pointing to them as we render.
I also think these are the kinds of places (like the difference between 
ascii vs svg art) that Sandy was originally intending to point to.

There is another activity where you are looking to see if there is a bug 
in the rendering code that has produced an unexpected difference (some 
section's numbering is being rendered incorrectly, or a block of text 
was entirely missing). Finding those after the rendering is done will be 
hard, and I'm almost ready to argue that there's no way we can guarantee 
mechanical identification of them. Rather it's a job for a brain to look 
at these things. Now, maybe what we can provide is a good way to look at 
these side-by-side and help control keeping the two views in sync. But 
finding a way to perform a diff, and sensibly display that diff for the 
general case of changes in the presentation formats is not something I 
see happening.  At best, I think it will devolve to what I think you're 
hinting at above - we postprocess each of the formats back into as much 
of a text representation as we can get out of them and running a diff on 
that.

>
>     Tony Hansen
>
> _______________________________________________

---

From: Sandy Ginoza <[email-masked]>
Date: Wed, 12 Nov 2014 13:50:16 -1000

On Nov 10, 2014, at 2:33 PM, Robert Sparks <[email-masked]> wrote:

>>> I'll think that through, but my initial reaction is that you're asking for something very hard (Artificial Intelligence level hard).
>> 
>> If we were to do a simple .txt vs .html comparison, there are certainly things that would keep showing up as false positives because of the inherent differences between the formats. For example, the .txt might format links in a way different from the .html; headers and footers are in the .txt and not in the .html; the order of data in the document header may be totally different; asides may wind up in different order within the documents; etc.
>> 
>> With PDF, there is the addiitonal issue that blocks of words have coordinates associated with them, and those coordinates need to be ordered prior to looking at the underlying text.
>> 
>>> Instead, perhaps we should be talking about adding requirements to the rendering tools that issue warnings when they know they're doing something that will result in what you would consider to be a difference between the rendered formats?
>> 
>> I think this sentence is totally missing Sandy's point. It's not the KNOWN and EXPECTED differences that Sandy is worried about, but the UNKNOWN differences.
> There is one activity where you look at places that are different in the renderings on purpose to make sure the differences are what we expect.
> I think it will be relatively easy to help see those efficiently by pointing to them as we render.
> I also think these are the kinds of places (like the difference between ascii vs svg art) that Sandy was originally intending to point to.
> 
> There is another activity where you are looking to see if there is a bug in the rendering code that has produced an unexpected difference (some section's numbering is being rendered incorrectly, or a block of text was entirely missing). Finding those after the rendering is done will be hard, and I'm almost ready to argue that there's no way we can guarantee mechanical identification of them. Rather it's a job for a brain to look at these things. Now, maybe what we can provide is a good way to look at these side-by-side and help control keeping the two views in sync. But finding a way to perform a diff, and sensibly display that diff for the general case of changes in the presentation formats is not something I see happening.  At best, I think it will devolve to what I think you're hinting at above - we postprocess each of the formats back into as much of a text representation as we can get out of them and running a diff on that.

I am hoping for a tool that will identify differences, expected or otherwise, across formats.  Warnings would work, but they’d be my second choice.   I’m worried about places like when the text file will point to a figure in the PDF/HTML (e.g., does the pointer that is in place of an image that points to the PDF/HTML point appear as expected?), and I’m also worried about things like UTF-8-encoded chars appearing differently in the text (not sure how big of a worry that is, but it is something I think we could easily miss during a visual check).

Thanks,
Sandy