Subject: [dispatch] Zstandard compression

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 1 Nov 2017 14:35:33 -0700

<chair hat clearly off>

In September, I submitted draft-kucherawy-dispatch-zstd, which introduces a
new compression  mechanism and seeks to register media types and content
encodings for it.  The most obvious application for it is HTTP payloads,
but it has other possible applications.

I don't know of IETF venues other than this one and maybe the ART list
where this kind of thing might be discussed, or how we might dispatch it.
Offlist conversation with Mark Nottingham suggested maybe HTTPBIS might
like to at least discuss this, but I haven't looked to see if it fits
within their current charter.

Suggestions for discussion venues or processing options are welcome.

-MSK

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Nov 2017 09:40:10 +1100

To be clear - HTTPbis needs a heads up and might want to have discuss the registration of the content-coding (Section 3.2). The whole spec probably belongs elsewhere.

Cheers,

> On 2 Nov 2017, at 8:35 am, Murray S. Kucherawy <[email-masked]> wrote:
> 
> <chair hat clearly off>
> 
> In September, I submitted draft-kucherawy-dispatch-zstd, which introduces a new compression  mechanism and seeks to register media types and content encodings for it.  The most obvious application for it is HTTP payloads, but it has other possible applications.
> 
> I don't know of IETF venues other than this one and maybe the ART list where this kind of thing might be discussed, or how we might dispatch it.  Offlist conversation with Mark Nottingham suggested maybe HTTPBIS might like to at least discuss this, but I haven't looked to see if it fits within their current charter.
> 
> _______________________________________________
> dispatch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch

--
Mark Nottingham   https://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Thu, 2 Nov 2017 10:32:42 +1100

I think that the right approach here is to approach this similarly
cryptographic algorithms.  It seems like the basic compression
structure is independently useful.  Document the format in one place,
then have short documents that describe content codings and other
things independently.  Then HTTP can take the content-coding work on
at their discretion (or it can progress independently).

I realize that RFC 7932 (brotli) didn't do this, so it would be OK to
just follow that pattern.  It also didn't define a media type. Did you
have a use case in mind for that?

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 2 Nov 2017 10:44:00 +1100

My .02 - I don't know that Brotli is a shining beacon of how things should be done here; I don't see any instrinc problem with registering the appropriate identifiers in the same document -- as long as appropriate consultation takes place.

Cheers,

> On 2 Nov 2017, at 10:32 am, Martin Thomson <[email-masked]> wrote:
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 2 Nov 2017 16:00:26 -0700

> at their discretion (or it can progress independently).

Is there a WG or some other review body that looks at compression where
this should get referred?

> have a use case in mind for that?

Yeah that's basically what I did.  And yes, we intend to use it between our
services and mobile apps for starters.  As it gains more adoption, it'll be
necessary to have these code points anyway; may as well get 'em done.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 2 Nov 2017 16:01:23 -0700

On Wed, Nov 1, 2017 at 3:40 PM, Mark Nottingham <[email-masked]> wrote:

> To be clear - HTTPbis needs a heads up and might want to have discuss the
> registration of the content-coding (Section 3.2). The whole spec probably
> belongs elsewhere.
>

Are you chartered to take it?  Or at a minimum, do you want me to show up
and talk about it?  Or if it would be worthwhile I can try to get the main
protagonists to join remotely.

-MSK

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 3 Nov 2017 10:22:07 +1100

On 3 Nov 2017, at 10:01 am, Murray S. Kucherawy <[email-masked]> wrote:
> Are you chartered to take it?  Or at a minimum, do you want me to show up and talk about it?  Or if it would be worthwhile I can try to get the main protagonists to join remotely.

If "it" is the content-coding registration, yes. If you want to give a five-minute hand-wave, we can probably fit you in, but it may be too early.

I keep on forgetting that IETF Review (the registry policy for content-codings) doesn't require standards track; you just need IESG approval on a document. Brotli did this on an Informational document, and you could do likewise.

So AIUI it really depends on how much review / input you want on the document; having a full WG, going AD sponsored (if you can find one), or just Informational with IESG review are all options. All in the same document and split documents are also options.

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Fri, 3 Nov 2017 10:50:26 +1100

> So AIUI it really depends on how much review / input you want on the document; having a full WG, going AD sponsored (if you can find one), or just Informational with IESG review are all options. Al…

This seems right.  If you want to hand over change control, maybe
change some things and tune up the format some (i.e., do some real
work), then you are probably looking at a working group.  On the other
hand, with Brotli, it was pretty much a case of "Here's a nice thing.
We don't plan to change it, but we'll document it nicely so that
everyone can use it."  That's a posture that leads to Informational
and AD sponsorship works nicely.

I'd be happier if we got some sort of signal that people liked the
format and planned to deploy it.  We just added a new compression
content coding, so this might not be so easy to justify.  That said,
any item in the registry is easy to ignore, so it's no great expense
to have it exist.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 3 Nov 2017 00:23:20 -0700

> any item in the registry is easy to ignore, so it's no great expense

The draft has an Implementation Status section that talks about a few,
though there are no browsers among them at this point.  But the latter's
not a specific requirement for a Content-Encoding, is it?

-MSK

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 8 Nov 2017 17:02:08 -0800

Dispatch seems like a fine place to figure out where to dispatch this. 

It seems to me that most our people with compression expertise tend to be places other that HTTPBIS. Very similar work has been done in the NETVC and codec groups for their entropy encoders. This might be a good candidate for a small mini WG that just did this. 

> On Nov 1, 2017, at 2:35 PM, Murray S. Kucherawy <[email-masked]> wrote:
> 
> <chair hat clearly off>
>

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 13 Nov 2017 00:58:07 +0100

stuff that I couldn't see addressed in the draft or in the thread:

- choice of name - is it still open? (anything with "standard" in the
name probably isn't)
- target application domain (text, video, images, "other")
- performance (I don't think any of these techniques are novel, so
performance should be known)
- IPR

If I were choosing to take this up, pass it along as Info or try to
block it altogether, those would be things I'd want to know.

On 11/01/2017 10:35 PM, Murray S. Kucherawy wrote:
> <chair hat clearly off>
>
> In September, I submitted draft-kucherawy-dispatch-zstd, which
> introduces a new compression  mechanism and seeks to register media
> types and content encodings for it.  The most obvious application for
> it is HTTP payloads, but it has other possible applications.
>
> where this kind of thing might be discussed, or how we might dispatch
> it.  Offlist conversation with Mark Nottingham suggested maybe HTTPBIS
> might like to at least discuss this, but I haven't looked to see if it
> fits within their current charter.
>

-- 
Surveillance is pervasive. Go Dark.

---

From: Sean Leonard <[email-masked]>
Date: Mon, 13 Nov 2017 10:22:48 +0800

Checked out the draft.

To simplify it, it looks like it is most analogous to the RFC 1950 / RFC 1951 documents that describe the algorithm(s) for ZLIB / DEFLATE, for compression of arbitrary data streams.

Those documents do not specify MIME types, and I do not think this document draft-kucherawy-dispatch-zstd should do so either. The media type registration is weak: there is no way to signal inside the format what is inside other than an arbitrary stream of bytes. It is not a file archive format, or another kind of container format that identifies parts.

3.2. Content Encoding is fine.

I request that we get an OID registration for this for use with Compressed Data Content Type for CMS, RFC 3274. That way it could be used with S/MIME or other formats that use CMS or S/MIME (seeing as how CMS / S/MIME are general container formats).

Surveying other container formats and doing the appropriate registrations with them in this draft would be appropriate, if one can articulate a use case in those respective formats.

Sean

> On Nov 2, 2017, at 5:35 AM, Murray S. Kucherawy <[email-masked]> wrote:
> 
> <chair hat clearly off>
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 13 Nov 2017 11:20:24 +0800

Keeping in mind that I'm mostly running process here and not competent in
the underlying technologies:

On Mon, Nov 13, 2017 at 7:58 AM, Harald Alvestrand <[email-masked]>
wrote:

> - choice of name - is it still open? (anything with "standard" in the name
> probably isn't)
>

I can ask what the origin of the name is, but given the number of
implementations that already exist, changing it now might be rather
annoying.

Arbitrary.  The web site where the work is based (http://www.zstd.net)
doesn't identify a specific use case that's targeted.

> performance should be known)

Substantial performance comparison is available at the above URL.

- IPR

In compliance with BCPs 78 and 79.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 13 Nov 2017 11:22:13 +0800

On Mon, Nov 13, 2017 at 10:22 AM, Sean Leonard <[email-masked]> wrote:

> Those documents do not specify MIME types, and I do not think this
> document draft-kucherawy-dispatch-zstd should do so either. The media type
> registration is weak: there is no way to signal inside the format what is
> inside other than an arbitrary stream of bytes. It is not a file archive
> format, or another kind of container format that identifies parts.
>

Why does that imply that a media type is not appropriate?  A module
receiving this content for processing needs to know what format it's in,
and I believe that's what these are for.

I request that we get an OID registration for this for use with Compressed
> Data Content Type for CMS, RFC 3274. That way it could be used with S/MIME
> or other formats that use CMS or S/MIME (seeing as how CMS / S/MIME are
> general container formats).
>

Wilco.

Surveying other container formats and doing the appropriate registrations
> with them in this draft would be appropriate, if one can articulate a use
> case in those respective formats.
>

Can you suggest an example?

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 13 Nov 2017 11:23:58 +0800

>> general container formats).

RFC3274 doesn't describe a process for registering an OID.  Can you point
me in the right direction?

-MSK

---

From: Sean Leonard <[email-masked]>
Date: Mon, 13 Nov 2017 14:01:06 +0800

> On Nov 13, 2017, at 11:22 AM, Murray S. Kucherawy <[email-masked]> wrote:
> 
> On Mon, Nov 13, 2017 at 10:22 AM, Sean Leonard <[email-masked] <mailto:[email-masked]>> wrote:
> Why does that imply that a media type is not appropriate?  A module receiving this content for processing needs to know what format it's in, and I believe that's what these are for.

Working on this. It strikes me as either…odd…or just not really that useful for the purpose. It is a format, but it’s just a format for a stream of arbitrary bytes. There is no way to label it internally beyond the implicit “application/octet-stream” type.

Content-Encoding is therefore very appropriate, but that is for HTTP, not (as of yet) for mail or other messaging formats.

May I ask, what is the reason why you need a media type?

Let me think about it…maybe it would be good to ask the media-types ML.

> Can you suggest an example?

Not offhand, just thought of those two (Content-Encoding, and RFC 3274 CompressedData media type).

Well there’s TLS v1.2 compression methods. Just joking! :-)

Sean

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 13 Nov 2017 14:18:58 +0800

On Mon, Nov 13, 2017 at 2:01 PM, Sean Leonard <[email-masked]> wrote:

> Working on this. It strikes me as either…odd…or just not really that
> useful for the purpose. It is a format, but it’s just a format for a stream
> of arbitrary bytes. There is no way to label it internally beyond the
> implicit “application/octet-stream” type.
>

Don't you need a hint as to how to decode an octet stream that's encoded in
some way?

The kind of signaling Content-Encoding provides is something that's needed
more generally, and this is what a media type would solve.

-MSK

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 13 Nov 2017 18:25:46 +0900

> Don't you need a hint as to how to decode an octet stream that's encoded in

Yes. But I think what Sean is saying is that above and beyond that, 
you'd (in many if not most use cases) also want to know what the decoded 
octet stream is about. Otherwise, this information has to be transmitted 
on a side channel.

Regards,    Martin.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 15 Nov 2017 01:25:43 +0100

On 11/13/2017 10:25 AM, Martin J. Dürst wrote:
> you'd (in many if not most use cases) also want to know what the
> decoded octet stream is about. Otherwise, this information has to be
> transmitted on a side channel.

One (horribly ugly) way of doing this is to have something like a
"contains" parameter:

mime-type: application/zstandard; contains="text/plain; charset=utf-8"

That's one form of side channel for passing the data around. Not sure if
anyone would use it.
But it's such an ugly way of doing things that it should definitely be
suggested on the media-types list, not here.

---

From: Sean Leonard <[email-masked]>
Date: Wed, 15 Nov 2017 12:33:57 +0800

> On Nov 15, 2017, at 8:25 AM, Harald Alvestrand <[email-masked]> wrote:
> 

Yep, talked offline at the conference and that was one possibility that was discussed: add a parameter that records the inner content type. It has advantages and disadvantages. The main thing is, is this a media type where encapsulating things is intended, or not? The main purpose of the document appears to be to describe the Zstd compression algorithm, not to create container formats.

Since it’s an algorithm, it is broadly applicable to lots of different containing formats. For example, it could be used with .ZIP (application/zip) by registering a new “compression method”: see Section 4.4.5 of the ZIP APPNOTE <https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.4.TXT>. Zip already supports BZIP2, LZMA, WavPack, and a handful of others, so it’s just another one to support.

(The same principle applies to CMS CompressedData, which is why I suggested it.)

Another thing to consider is whether it is a candidate for a structured syntax suffix, namely +zstd (which was suggested/discussed offline). Think about it, and ask the media-types@ folks. In this case, I do not find it to be a strong candidate because it’s not a “structured” syntax, in the way that JSON, XML, etc. are “structured” (into arbitrary trees of constituent elements or parts).

Best regards,

Sean