Subject: [video-codec] draft-filippov-netvc-requirements-01

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 14:22:39 +0000

I read this draft after seeing the presentation yesterday. Overall, I dont like the fact that this draft tries to cover a lot of use cases quite narrowly. There are so many omissions here and there.

The charter for this WG is to provide a good codec for interactive video applications. So all the rest of the use cases should be dropped and not discussed at all.

This WG should spend whatever energy it has to focus on the primary use case (interactive video) like RTCweb, conferencing, etc. I dont think we have the energy to create a codec that will satisfy world’s all video demands.

So IPTV, game streaming and video sharing should be removed from the reqs document. I am not so sure about video monitoring, though, as it has quite many similarities to interactive video. 

FWIW, I argue that interlacing should not be considered in this WG (or at all for any video codec) moving forward. 

-acbegen

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 21 Jul 2015 07:28:57 -0700

> The charter for this WG is to provide a good codec for interactive video applications. So all the rest of the use cases should be dropped and not discussed at all.

I think at the very least on-demand Netflix/Youtube/etc. streaming is a 
very important internet use case that we want to satisfy. Interactive is 
called out explicitly by the charter because it imposes a set of severe 
technical constraints on what the codec can do, but I don't think the 
intent was to limit us to that use case (as indicated by the words "but 
not necessarily limited to" in the charter).

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 14:37:06 +0000

-----Original Message-----
From: video-codec on behalf of "Timothy B. Terriberry"
Date: Tuesday, July 21, 2015 at 4:28 PM
To: "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>not necessarily limited to" in the charter).

I understand that part but what I am saying is that developing a codec that will both perform enough well for interactive video and streaming video is a tedious task. Focusing on two different things will increase the risk of failure for this wg. Maybe the wg should consider solving the first (and the more important) problem first and then see whether it can tackle the second problem.

>
>_______________________________________________
>video-codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/video-codec

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 21 Jul 2015 07:57:21 -0700

> I understand that part but what I am saying is that developing a codec that will both perform enough well for interactive video and streaming video is a tedious task. Focusing on two different thin…

Well, I agree we can probably significantly narrow the set of use cases 
we want to consider, but I think narrowing it down to one is too extreme.

 From the perspective of my employer (Mozilla), for example, we need a 
codec that works in the <video> tag and in WebRTC, and we currently see 
a lot more usage of the former than the latter.

Focusing on interactive first may still be smart, but "success" for us 
means solving both problems. I don't think anything we write in our 
requirements draft precludes the WG from deciding to publish a codec 
that only solves one of them, though, if we later decide the other is 
too hard.

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 15:05:35 +0000

-----Original Message-----
From: video-codec on behalf of "Timothy B. Terriberry"
Date: Tuesday, July 21, 2015 at 4:57 PM
To: "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>we want to consider, but I think narrowing it down to one is too extreme.

We disagree, that is fine.

>a lot more usage of the former than the latter.

Sure, the amount of streaming video is far bigger than any webrtc traffic now, and it will be like that maybe forever. What I am saying is that it takes a lot more effort to beat the current de facto codecs that are used for streaming and entertainment video. The path to success is easier for the interactive video. For entertainment video, people pay for quality, they may also pay for webrtc services but it will be much less $.

>that only solves one of them, though, if we later decide the other is 

Fair enough, but then I have to say that I disagree on some of the reqs mentioned in the sections I suggested for removal. Most of them are not even reqs but just a collection of what is in the market today. Obviously if those sections are to stay, they need quite a bit work.

---

From: Thomas Daede <[email-masked]>
Date: Tue, 21 Jul 2015 17:10:08 +0200

> This WG should spend whatever energy it has to focus on the primary use case (interactive video) like RTCweb, conferencing, etc. I dont think we have the energy to create a codec that will satisfy…

I think that this would dangerously narrow the adoption of the codec.
Would Opus have done nearly as well if it was only SILK?

> So IPTV, game streaming and video sharing should be removed from the reqs document. I am not so sure about video monitoring, though, as it has quite many similarities to interactive video. 

I have many issues with the categories in the requirements draft:

- All of the use cases should specify either a high latency or low
latency requirement.
- Error resilience should be moved out of the use cases. It can be
implied by low latency, as can a lot of other things (transport, etc)
- IPTV seems to intend to describe UDP multicast sorts of IPTV, which is
not very common in the US but has some traction in other countries / as
a backend for cable distribution. If this is the case, it really needs
to be made clear.
- The big list of resolutions is overly verbose and not necessary. A
simple range of resolutions and frame rates would be better.
- Game streaming needs to be split into high latency (twitch) and low
latency (steam remote play) applications.

> FWIW, I argue that interlacing should not be considered in this WG (or at all for any video codec) moving forward. 

I think that interlacing tools should not be supported by the video
codec, but it would be good to test on content that was originally
provided in interlaced format and then run through a deinterlacer. How
this is done exactly should be specified.

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 15:39:16 +0000

-----Original Message-----
From: video-codec on behalf of Thomas Daede
Date: Tuesday, July 21, 2015 at 5:10 PM
To: "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>Would Opus have done nearly as well if it was only SILK?

And please tell me where Opus is used in the entertainment world?

>- All of the use cases should specify either a high latency or low

And where is the borderline?

>a backend for cable distribution. If this is the case, it really needs

I still argue iptv should be removed entirely.

>simple range of resolutions and frame rates would be better.

Agreed.

>latency (steam remote play) applications.

Indeed.

>this is done exactly should be specified.

Are you interested in re-encoding of pre-encoded content? If so, why? If not, why bother with this test?

---

From: Thomas Daede <[email-masked]>
Date: Tue, 21 Jul 2015 18:12:12 +0200

> And where is the borderline?

What is a borderline use case? I would much rather keep the number of
configurations as low as possible.

> I still argue iptv should be removed entirely.

I would be fine with this. If it stays, it needs a much clearer use case
definition.

Thomas

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 16:14:56 +0000

-----Original Message-----
From: Thomas Daede
Date: Tuesday, July 21, 2015 at 6:12 PM
To: "Ali C. Begen", "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>configurations as low as possible.

I mean what is low latency vs high latency. And who decides that?

---

From: Thomas Daede <[email-masked]>
Date: Tue, 21 Jul 2015 18:18:25 +0200

> And please tell me where Opus is used in the entertainment world?

Youtube uses Opus in DASH. WebM defines VP9/Opus as one of its standard
profiles.

But not to get sidetracked, I want to clarify that I want NETVC to
produce something that is acceptable for the <video> tag or DASH.
Bluray, non-end-user usage, etc is totally out of scope.

---

From: Thomas Daede <[email-masked]>
Date: Tue, 21 Jul 2015 18:21:02 +0200

> I mean what is low latency vs high latency. And who decides that?

It is defined in draft-daede-netvc-testing-01 [1]. And some of the
definition should probably be moved into the requirements draft, though
individual codec parameters still belong in the testing draft.

https://datatracker.ietf.org/doc/draft-daede-netvc-testing/

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 16:25:20 +0000

-----Original Message-----
From: Thomas Daede
Date: Tuesday, July 21, 2015 at 6:18 PM
To: "Ali C. Begen", "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>Youtube uses Opus in DASH. WebM defines VP9/Opus as one of its standard

Youtube creates a lot of traffic and I suppose a large portion of it is vp9/opus but 99.9% of the youtube video is not something I would consider “entertainment grade”. Dont get me wrong, but when you look at CE devices that consume streaming video, vp9/opus is not on the top of the list.

>produce something that is acceptable for the <video> tag or DASH.

DASH is codec agnostic, so it does not care. Whether someone will use netvc for DASH, I am not holding my breath. 

>Bluray, non-end-user usage, etc is totally out of scope.

Certainly.

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 16:28:30 +0000

-----Original Message-----
From: Thomas Daede
Date: Tuesday, July 21, 2015 at 6:21 PM
To: "Ali C. Begen", "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>https://datatracker.ietf.org/doc/draft-daede-netvc-testing/

I searched for latency and delay, nothing showed up. The only time related number seems to be 300ms in section 5.3.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 21 Jul 2015 09:33:49 -0700

> Youtube creates a lot of traffic and I suppose a large portion of it is vp9/opus but 99.9% of the youtube video is not something I would consider “entertainment grade”. Dont get me wrong, but when…

It's still #3 on the Alexa Top 500 global sites.

Also #7 Wikipedia: 
https://commons.wikimedia.org/wiki/Commons:File_types#Sound

Broadcast:
http://www.tieline.com/products/G5/Merlin
http://www.gatesair.com/products/transport/audio-contribution-distribution/intraplex-ip-link
http://www.comrex.com/products/briclinkII.html
http://www.obe.tv/about-us/obe-blog/item/14-using-opus-audio-in-broadcasting
http://ebu.io/opensource#codecs

Internet radio:
http://lists.xiph.org/pipermail/icecast/2012-July/012237.html

Wireless headphones and mics:
http://www.dialog-semiconductor.com/products/audio-wireless-voice/smartbeat-wireless-audio

Video games:
Opus is included in the Playstation 4, FMOD (used in many games), and 
others.

There are probably lots more. One of the benefits of being completely 
open is that no one has to tell me when they start using Opus.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Tue, 21 Jul 2015 16:36:27 +0000

300 ms (* bitrate) refers to the rate control buffer not coding latency. 

I think the testing draft should define low latency as zero frame delay, i.e. no reordering. The high latency condition can be unrestricted. 

Mo (as individual)

On Jul 21, 2015, at 6:28 PM, Ali C. Begen (abegen) <[email-masked]> wrote:

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Tue, 21 Jul 2015 17:10:12 +0000

Sorry your links do not prove much. When you use a vpX or opus enabled device or browser youtube most likely sends the content with those codecs. But the world has many other devices or browsers  that are not capable of rendering them so...

I did not mean to create a codec discussion but looks like it is to late to revert back. Opus is a great codec and it is widely used but only under certain apps and conditions. But there are lots of other codecs that it will never replace.  I hope netvc will be equally or even more successful.  But again there will be other codecs that it will never replace nor it should. So my point is we should focus on where we have the highest chance to succeed in a reasonable time frame. 

-acbegen 
Sent using iThumbs - DYAC

> On Jul 21, 2015, at 6:34 PM, Timothy B. Terriberry <[email-masked]> wrote:
> 
> Also #7 Wikipedia: https://commons.wikimedia.org/wiki/Commons:File_types#Sound
> 
> Broadcast:
> Opus is included in the Playstation 4, FMOD (used in many games), and others.
> 
> There are probably lots more. One of the benefits of being completely open is that no one has to tell me when they start using Opus.
>

---

From: Thomas Daede <[email-masked]>
Date: Wed, 22 Jul 2015 00:36:18 +0200

> Sorry your links do not prove much. When you use a vpX or opus enabled device or browser youtube most likely sends the content with those codecs. But the world has many other devices or browsers  t…

And your lack of links doesn't prove much either :)

> I did not mean to create a codec discussion but looks like it is to late to revert back. Opus is a great codec and it is widely used but only under certain apps and conditions. But there are lots o…

I am honestly not sure what codecs Opus will "never replace", other than
maybe lossless codecs, codecs that run at bitrates too low to have
practical uses on the Internet, and future codecs that have not been
invented yet.

In addition, I think spending effort on the high latency case is not
nearly as high a toll as you think. Pretty much the only bitstream
feature that is needed for high latency support is bi-predicted frames.
And Thor already has these.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 22 Jul 2015 00:00:42 -0700

> And Thor already has these.

Also, as I have said before, network effects dominate. Having tried this 
royalty-free codec thing several times now, I believe that to the extent 
Opus has been more successful than our prior efforts it is precisely 
because it is better on all fronts across a very wide range of 
applications. You don't build network effects by being a niche codec, 
especially when your competition (as here) is not a niche codec.

I do think focusing on interactive first has some benefits. If you focus 
on streaming/random access first, you wind up with a laundry list of 
features that are not, e.g., robust to packet loss, and thus have to be 
shut off for interactive applications with some large penalty (> 5% 
rate), while alternative features could have been designed that would 
have been nearly as good (within 1-2% rate) and also been robust to 
packet loss. This has already come up several times in Daala and we have 
opted for the latter approach. You can't do that for all features (frame 
re-ordering being one big exception, as you mention), but you can do it 
for a lot of them.

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Wed, 22 Jul 2015 07:48:08 +0000

-----Original Message-----
From: video-codec on behalf of Thomas Daede
Date: Wednesday, July 22, 2015 at 12:36 AM
To: "[email-masked]"
Subject: Re: [video-codec] draft-filippov-netvc-requirements-01

>And your lack of links doesn't prove much either :)

If you google for encoding.com’s global media report 2015 you will see that they report HLS being the market leader with 75%, then smooth streaming comes along (in the streaming domain). Neither of these use vpX/opus to my knowledge. 

You can also google JW player’s trends in online video report, and you will see that webm only has 9% share.

If you stream youtube to your iOS devices today, it is h264 and aac. This also happens with most browsers other than chrome and firefox. Needless to mention, apple tv, Roku, all those streaming players (probably except chromecast) do not use vpX/opus to my knowledge. 

Also do not forget about the connected TVs, all those devices already have to support h264 and most new ones support h265, too, and that is what they use to stream online content. Hybrid settops that are in our houses (from any cable/iptv/sat provider), they support the existing well known codecs. Honestly speaking, there are still boxes out there that only support mpeg2 not even h264 because it probably costs more to the service provider to replace them.

Netflix, the largest streaming company for tv shows, movies, (and has the largest worldwide subscriber number) does not use vpX/opus. Amazon prime uses avc and aac.

>practical uses on the Internet, and future codecs that have not been

As I mentioned situation is different (better) for opus than netvc, but if you are telling me that opus will replace AAC, or other multi-channel technologies (like dolby stuff), I disagree.

>feature that is needed for high latency support is bi-predicted frames.

It is more than that. Even if we have an on par or better codec, it may not get adopted due to the reasons we do not control. Hence, I said, we should focus our energy where it can make the most impact.

---

From: Mohammed Raad <[email-masked]>
Date: Wed, 22 Jul 2015 11:00:42 +0300

But what's the argument here, that if vpx/opus has not dominated yet then
the netvc effort will not produce something that will dominate on the web?

I gather that what is meant by "interactive" is real time communications.
If you limit the netvc effort to only real time communications then you are
ensuring that it will not have a chance to be dominant once it is complete
because the argument for adoption will be weakened by the fact that another
codec will be needed for the high quality material.

Mohammed

On Wed, Jul 22, 2015 at 10:48 AM, Ali C. Begen (abegen) <[email-masked]>
wrote:

> >> Sorry your links do not prove much. When you use a vpX or opus enabled
> device or browser youtube most likely sends the content with those codecs.
> But the world has many other devices or browsers  that are not capable of
> rendering them so...
> >
> If you google for encoding.com’s global media report 2015 you will see
> that they report HLS being the market leader with 75%, then smooth
> streaming comes along (in the streaming domain). Neither of these use
> vpX/opus to my knowledge.
>
> If you stream youtube to your iOS devices today, it is h264 and aac. This
> also happens with most browsers other than chrome and firefox. Needless to
> mention, apple tv, Roku, all those streaming players (probably except
> chromecast) do not use vpX/opus to my knowledge.
>
> Also do not forget about the connected TVs, all those devices already have
> to support h264 and most new ones support h265, too, and that is what they
> use to stream online content. Hybrid settops that are in our houses (from
> any cable/iptv/sat provider), they support the existing well known codecs.
> Honestly speaking, there are still boxes out there that only support mpeg2
> not even h264 because it probably costs more to the service provider to
> replace them.
>
> Netflix, the largest streaming company for tv shows, movies, (and has the
> largest worldwide subscriber number) does not use vpX/opus. Amazon prime
> uses avc and aac.
>
> >
> >> I did not mean to create a codec discussion but looks like it is to
> late to revert back. Opus is a great codec and it is widely used but only
> under certain apps and conditions. But there are lots of other codecs that
> it will never replace.  I hope netvc will be equally or even more
> successful.  But again there will be other codecs that it will never
> replace nor it should. So my point is we should focus on where we have the
> highest chance to succeed in a reasonable time frame.
> >
> As I mentioned situation is different (better) for opus than netvc, but if
> you are telling me that opus will replace AAC, or other multi-channel
> technologies (like dolby stuff), I disagree.
>
> >
> It is more than that. Even if we have an on par or better codec, it may
> not get adopted due to the reasons we do not control. Hence, I said, we
> should focus our energy where it can make the most impact.
>

-- 
Mohammed Raad, PhD.
Partner
RAADTECH CONSULTING
P.O. Box 113
Warrawong
NSW 2502 Australia
Phone: +61 414451478
Email: [email-masked]

---

From: Thomas Daede <[email-masked]>
Date: Wed, 22 Jul 2015 10:03:46 +0200

I have pending updates for a -02 version of the testing draft that
better machines the slides that I presented. I have pasted the revised
selection below. Note that the encoder parameters haven't been updated
yet, and I also plan to incorporate rate-control-less operating modes,
as per consensus in the room.

## Operating Points

Two operating modes are defined. High latency is intended for on demand
streaming, one-to-many live streaming, and stored video. Low latency is
intended for videoconferencing and remote access.

### High Latency

The encoder should be run at the best quality mode available, using the
mode that will provide the best quality per bitrate (VBR or constant
quality mode). Lookahead and/or two-pass are allowed, if supported.
Example configurations follow:

- x264: --crf=x
- x265: --crf=x
- daala: -v=x
- libvpx: --codec=vp9 --end-usage=q --cq-level=x

### Low Latency

Codecs should be run in CBR mode. The maximum allowed bitrate variance
is determined by a buffer model:

- The buffer starts out empty.
- After each frame is encoded, the buffer is filled by the number of
bits spent for the frame.
- The buffer is then emptied by (bitrate * frame duration) bits.
- The buffer fill level is checked. If it is over the limit, the test is
considered a failure.

The buffer size limit is defined by the bitrate target * 0.3 seconds.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Wed, 22 Jul 2015 08:53:18 +0000

I see a much simpler distinction between high/low latency modes.
Low latency is zero frame delay, i.e. no reordering, lookahead, etc.
High latency is unrestricted, i.e. all tools / techniques enabled
to maximize quality or minimize bitrate / file size.

I don't think the definition of these modes should be tied to rate control
modes like VBR or CBR. While I agree interactive applications often have
constraints on both latency and peak bitrate (to remain below available
bandwidth), I think it is confusing to conflate them by defining low
latency in terms of bitrate control.

For example, under the current definition of low latency as CBR with a 300
ms rate control buffer (similar to MPEG VBV model), and no further
constraints, encoders could reorder 18 frames of 1080p60 and still claim
"low latency". I think this violates our goal. The true goal should be
absolute minimum delay, which means zero frame delay restrictions within
the encoder.

Mo (as individual)

On 7/22/15, 4:03 AM, Thomas Daede <[email-masked]> wrote:

## Operating Points

### High Latency

- x264: --crf=x
- x265: --crf=x
- daala: -v=x

### Low Latency

On 07/21/2015 06:36 PM, Mo Zanaty (mzanaty) wrote:
> I think the testing draft should define low latency as zero frame delay,
>i.e. no reordering. The high latency condition can be unrestricted.
> 
> Mo (as individual)
> 
> I searched for latency and delay, nothing showed up. The only time
>related number seems to be 300ms in section 5.3.