Subject: [video-codec] The Can Has Landed

From: Monty Montgomery <[email-masked]>
Date: Thu, 19 Mar 2015 11:33:36 -0400

Hello to everyone interested in Daala, NetVC, and the run up to
the netvc BoF on Tuesday at IETF92 in Dallas!

While I'm mentioning the BoF I'd also like to remind folks that
the Hackathon on Saturday and Sunday includes NetVC and a decent
portion of our own Daala team will be there.  Physical attendance
at the hackathon required registration that's already closed due
to the limits of the room, but the Daala team will be in the
#daala IRC channel on Freenode the whole time, pretty much like
we always are... online participation is always welcome and
encouraged!

And now to the pep talk...

In 2013, Cisco announced the OpenH264 licensing hack as a step
toward ending a long-standing and frustratingly non-technical
roadblock to WebRTC: h.264, though dominant, is unlicenseable
by Free and Open Source entities.   I wrote at the time that although I
supported the practical aspects of the OpenH264 deal, "Licensing
caused this problem, and more licensing is not a solution. [...]
We've merely kicked the can down the road and set a dangerous
precedent for next time around."*

We're now standing where the can takes its first bounce.

NetVC is our collective opportunity to reverse that precedent and
to give the Internet and Open Web the only fundamental Free
technology it lacks: a fully modern video codec with no
'permission required' strings attached.  The elevator pitch here
is very simple: We must make for video what Opus is for audio.
Video will never be a first-class citizen on the Net so long as
it is only available to the 'haves' who are able to license the
required commercial technology.

Judging by the relative calm of the netvc mailing list, I don't
think we (the proponets of netvc) face the same kind of
controversy this time around as we did at the time of 'codec'
BoFs.  That's a bit ironic, actually, as a video codec is a
substantially more challenging technical undertaking.

I do think there likely is some quiet skepticism-- there would
have to be-- and so I'd like to invite the skeptics and Devil's
Advocates to speak up here in the days before the BoF.  I doubt
it will shorten the lines at the microphones, but perhaps the
arguments will be more focused and better honed at that time.

I'm not saying 'speak now or forever hold your peace'.  I'm just
trying to shift the discussions that would normally start in
a cramped uncomfortable room to the mailing list a little early.

Cheers,
Monty
of Xiph Moz Daala

* http://xiphmont.livejournal.com/61927.html

---

From: Keith Winstein <[email-masked]>
Date: Thu, 19 Mar 2015 17:17:08 -0700

Trying to reply in the spirit intended, here's what I would say with a
devil's advocate hat on just for fun:

Daala seems like an extremely important research project to develop a
next-generation coded video representation that will also be free of
practical licensing obstacles.

But what are the merits of an IETF working group performing this kind of
high-risk, high-reward research, versus doing something much more boring
like "writing a specification for VP9 to enable interoperable
implementations, and then iterating on that technology"?

Google said in May 2013 that "a draft bitstream specification is well
underway." For whatever reason, they still haven't published it yet. (There
is also no independent implementation of a decoder written by a
non-Google-employee, afaik, much less of an encoder.)

But the VP9 format is certainly specifiable given effort, and the licensing
situation is probably about as well-understood as Daala's will ever be,
given the MPEG LA license.

So the devil's advocate question would be, if the problem is that "video
will never be a first-class citizen on the Net so long as it is only
available to the 'haves' who are able to license the required commercial
technology," is rallying around VP8/VP9 (perhaps with an actual
high-quality spec) perhaps a better/safer approach?

-Keith
On Mar 19, 2015 8:33 AM, "Monty Montgomery" <[email-masked]> wrote:

> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
>

---

From: Thomas Daede <[email-masked]>
Date: Thu, 19 Mar 2015 17:41:48 -0700

> implementations, and then iterating on that technology"?

If VP9 could be shown to meet our requirements for performance and
licensing, I would be supportive of that path. However, I don't believe
that it currently does.

In the testing draft, while I specified methods of testing codecs, I did
not specify an absolute performance goal. We will need to set one. Opus
achieved state-of-the-art performance, which was highly beneficial to
its adoption. VP9 does not achieve this yet.

In regards to licensing, it is clear that many companies are quite happy
shipping VP9 right now. However, others are not nearly as comfortable
with it, as seen on RTCWEB. These concerns arise from other, non MPEG-LA
rights holders. We need to make sure that we address those concerns in a
working group one way or another. One way to do that is just to stay
well clear of the patents in question.

> non-Google-employee, afaik, much less of an encoder.)

ffmpeg's VP9 decoder was written by a couple of non-Google employees:
https://blogs.gnome.org/rbultje/

---

From: Keith Winstein <[email-masked]>
Date: Thu, 19 Mar 2015 18:30:43 -0700

> its adoption. VP9 does not achieve this yet.

Makes sense.

> https://blogs.gnome.org/rbultje/

No disrespect intended to Ronald Bultje and his prolific work for free
software, but he worked for Google on libvpx for more than two years as
they developed VP9 ([email-masked] has >580 commits between April 2011
and July 2013) and "was one of the people involved in creating" VP9 in the
first place, as he wrote on that blog post. I don't think ffvp9 really
counts for what you want to see in a well-specified format.

-Keith

---

From: Monty Montgomery <[email-masked]>
Date: Sun, 22 Mar 2015 11:22:34 -0400

I know Thomas already responded, but...

> implementations, and then iterating on that technology"?

Personally, I'd like to see Google submit their VPx work as an input
to this process and even more interested in seeing them participate in
development.  Just like with Opus, I don't think the IETF is
interested (and I'm certainly not interested) in rubber-stamping an
existing codec.

...it would also go a long way toward repairing VP9's image in the
marketplace.  Like it or not, the conventional wisdom is 'VP9 is a
proprietary codec with uncertain IPR status'.  IP is obviously a major
concern of ours, and we don't believe a strategy of 'we'll just win in
court' is likely to win an adoption war.

Monty

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sun, 22 Mar 2015 18:19:44 -0700

> development.  Just like with Opus, I don't think the IETF is

Well, I think that's what Keith meant by "iterating on that technology". 
I think if Google wanted to contribute their VPx work we should be very 
happy to see it. VP9 has real-time encoding that shows significant 
performance gains over prior generations and a decoder faster than 
H.264's. Google got usable open source encoder and decoder 
implementations to market before HEVC, and significant hardware support 
agreements with some major chip vendors. If Google codec engineers are 
going to actively participate and bring VPx to the party, that would be 
a real asset.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 23 Mar 2015 13:40:09 +0100

> its adoption. VP9 does not achieve this yet.

Just as a matter of curiosity, which state of the art performance is it
that you think VP9 hasn't achieved yet?

I also wonder a bit about how we should treat evaluation of performance
- a traditional video codec development technique has been to develop a
codec that takes hours-per-second to encode a video, and then seek to
optimize it by a factor of a hundred before shipping it. But sometimes,
there's just limits to how much one can achieve.

How should we treat that aspect in evaluation?

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 23 Mar 2015 13:49:20 +0100

---

From: Mohammed Raad <[email-masked]>
Date: Mon, 23 Mar 2015 14:52:30 +0200

Well, the second part (how one should judge the potential optimization)
typically has been avoided in video standardization and so far there seems
to be little willingness to work on the development of a metric or a set of
metrics that can be used to determine how "optimizable" a codec or tool is.
It would actually be good if people at the IETF can develop such a metric.

Also, keep in mind that the incremental development of a codec means that
prior decisions limit what can be done with tools proposed later in the
process.

Mohammed

-- 
Mohammed Raad, PhD.
Partner
RAADTECH CONSULTING
P.O. Box 113
Warrawong
NSW 2502 Australia
Phone: +61 414451478
Email: [email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 23 Mar 2015 15:09:56 +0100

On 03/23/2015 01:52 PM, Mohammed Raad wrote:
> Well, the second part (how one should judge the potential
> optimization) typically has been avoided in video standardization and
> so far there seems to be little willingness to work on the development
> of a metric or a set of metrics that can be used to determine how
> "optimizable" a codec or tool is. It would actually be good if people
> at the IETF can develop such a metric.

I was thinking more of whether experience with the encoding speed of the
test implementations should be allowed as an argument or not.

Sometimes one can use dataflow analysis and similar tools to show that
certain techniques prevent optimization beyond a certain level (or not)
- but most of the time, I think "this runs at speed x / quality y" is a
more compelling argument than "I think this could be optimized to speed
x / quality y".

>
> Also, keep in mind that the incremental development of a codec means
> that prior decisions limit what can be done with tools proposed later
> in the process.

Yes. Especially when the decision is "we can't use this tool for
nontechnical reasons", it means that we can't accept anything that is a
refinement of this tool either.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 23 Mar 2015 07:13:01 -0700

> that you think VP9 hasn't achieved yet?

I think if we were trying to develop a competitor to HEVC, we could say 
"Ship VP9" and be done. In fact, that is exactly what Firefox did. But I 
am (personally) worried about what comes after HEVC, and I think we have 
an opportunity to get ahead of the curve here like we did with Opus.

> How should we treat that aspect in evaluation?

I have been thinking about this a lot since Mo asked the question a few 
weeks ago, and I am having a hard time coming up with an answer that is 
not "use the normal IETF consensus process".

We've certainly tried to focus on practical algorithms in Daala. 
However, sometimes it is nice to know what the theoretical best we could 
do actually is, so we know how well some of our cheap hacks are performing.

To use a recent example, until the last few weeks, the way we had been 
implementing our lapped transforms, one needed to know the block sizes 
of your neighbors to know what lapping size to apply. That meant we 
couldn't do an exhaustive search of block sizes without making some kind 
of gross simplifications. We restructured things to remove this 
dependency, even though it meant the filtering was theoretically worse. 
That allowed us to write an exhaustive search, which turned out to be 
between 3% and 11% better than the heuristics we had been using to pick 
block sizes. Until we did this, we didn't have a good idea what kind of 
gains were possible, nor how to change our heuristics to produce better 
results. Now we have something we can look at to tell us both.

So having both the heuristic and the exhaustive search is the ideal 
situation, and sometimes asking for it is reasonable. Sometimes it may 
be more work than people are willing to do up-front to make a decision 
about what is a good idea.

But I will also give a second example, that is even more recent. One 
issue Thomas Davies from Cisco pointed out to us was some kind of noise 
buildup along block boundaries in one of their test clips at low rates, 
something we had not seen in our own testing. We were able to track down 
the problem to certain rounding operations in the pre/post filters and 
in converting from the 12-bit data we use in our transforms down to 
8-bit data we store in our reference frames. Storing 12-bit reference 
frames makes the problem go away, but increases the memory bandwidth 
required by the codec by 50%. But we've observed gains between 0.2 dB 
and (on specially constructed examples) 2 dB by doing so. Is that a cost 
people are willing to pay? I don't know the answer to that question, but 
I do know that the easiest way to measure that, x86 cycle counts, is 
also probably the least informative. It also matters what other 
solutions there are (we have at least one more, with a different set of 
unappealing costs), or how hard we want to look for them.

I have a real hard time seeing how to design an up-front process that 
answers questions like this. Maybe others here have better ideas than I do.

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 23 Mar 2015 14:55:44 +0000

Complexity (or more broadly, resource requirements, which encompass
aspects beyond compute cycles like memory and memory bandwidth) should be
a fundamental part of the evaluation methodology. But there are many
pitfalls to avoid here.

Intrinsic algorithmic complexity is difficult to discern with any
precision, since any metric necessarily depends on the level of
implementation optimization (from simple loop unrolling to vector
intrinsics or full assembly).

Optimization can hamper experimentation, so it is usually not a good idea
in the research stage.

Configuration points are also important to consider. Having configuration
points that are impractical for real-world use is still valuable for
steering the research. This should not be the primary evaluation criteria,
which should attempt to focus on real-world use. But such configuration
points should be allowed and analyzed.

Mo

On 3/23/15, 8:40 AM, Harald Alvestrand <[email-masked]> wrote: