Subject: Re: [video-codec] Strategy for an RF video codec

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 16 Jan 2013 09:50:39 -0800

> 1) every separable part of the codec becomes a separate feature and
> both sides negotiate the features they support. I'm imaging 100 or
> more features many of which depend on others or you can't use them.
> The minimal feature might just be uncompressed video. The odds of
> managing for all parts of an advanced video codec to be RF are low
> and this allows us to mitigate the risk by if some part if found to
> not be RF, deployments can disable that feature and the features that
> depend on them. This also makes for a much smother watt to
> incrementally advance the codec over time. Doing this means we are
> not doing a single codec but really a family of codecs .

So, two months ago I wasn't prepared to call this a bad idea. Having 
thought about it a lot more since then, I think I'm prepared to call it 
that now.

1. In reality you often can't "disable" a feature... you need to replace 
it with something. If you "disable" the entropy coder, you have to do 
something in its place. This complicates the design.  Now you have to 
design a bunch of fallbacks you wouldn't otherwise have needed.

2. It also complicates the IPR story. You need to prove the fallbacks 
are just as RF as the original features. Even if you just disable some 
function f, computing g(f(x)) might have been unpatented while computing 
g(x) directly is patented. Since mere combinations of features can be 
patented, more combinations means more risk.

3. I raised the testing issue elsewhere... and while it's easy to 
dismiss this and say we only test a few cases, this poses real security 
issues: how do you know disabling feature X, Y, and Z doesn't expose 
your implementation to a buffer overflow? I know you think codecs are so 
complex they can never be adequately tested anyways, but in my 
experience video moves enough data through the codec that you actually 
can test the vast majority of branches and combinations of things 
relatively easily with automated testing, and the pieces that remain are 
small enough that you can reason about them. The small number of 
security vulnerabilities found in the codecs I've worked on compared to 
the average in projects like, say, FFmpeg bears this out. But when you 
multiply that work by 2^100, then you have a problem.

4. The testing problem means this won't work interoperably. What will 
actually happen is that people will only test the streams that other 
parties send them, and then suddenly one day people will want to disable 
some flag that was in use before, and you'll find that a) the fallback 
is buggy, b) it simply isn't implemented, c) maybe it is implemented, 
but in unoptimized software instead of hardware, and now things are too 
slow to actually be useful, or d) any number of other problems. Even if 
we provide test vectors, and even if (through astounding diligence) 
every implementation actually tests against them, as you point out we 
can only cover a few cases, which likely won't be the ones we wind up 
actually interested in.

5. More likely, what will actually happen is that implementors will use 
the flags to simply avoid implementing whatever random features they 
deem unimportant for whatever reason. We already see this with things 
like H.264 which have a set of well-defined profiles... and then a 
separate set of the things that actually work in real, deployed 
implementations. The sets are not at all the same, and this is what 
allows Google to post VP8 performance benchmarks that beat "the features 
of H.264 that actually have broad enough support that we can enable them 
in our YouTube encodes." I.e., there's a real, measurable cost here, not 
just a theoretical one.

6. People willing to act unscrupulously can easily intimidate others 
into disabling features even when they don't really need to. The reality 
is that most people, confronted with someone telling them that "feature 
X might be unsafe" will simply disable it instead of trying to find out 
if that's really true. For many the analysis itself would be more costly 
than any loss of compression (especially device manufacturers who don't 
even bear the network transmission costs). But those that do bear 
sufficient cost to justify a real analysis have to interoperate with 
those who don't. So you enter a race to the bottom.

7. By definition you don't know which features need flags and fallbacks. 
If you did, you would have just taken that feature out in the first 
place. So we're making decisions about which features justify all the 
costs of a flag (in design time, implementation time, analysis time, 
testing time, attack surface, increased risk of interoperability 
failure) without any knowledge of the benefits.

So in sum, this strategy has significant up-front costs, and a low 
probability of actually solving any problems that arise.

If you're willing to disable a bunch of features without regards to 
compression performance to get something that's RF, a much simpler 
strategy with a much higher probability of success is to just fall back 
to a much older codec. Like, say, Theora. Or if that's not safe enough 
for you, H.261.

> 2) every feature that goes into the codec is specified in a separate
> draft to get separate IRP claims

I mentioned doing something like this at the BoF (which caused Stephan 
Wenger to violently shake his head "no", even though he had been the one 
to suggest that having smaller drafts would have been a good idea for 
Opus, and not that I ever got an explanation what his objection was 
now). I think it's certainly an okay idea at the start... I'm more 
concerned about what happens when we get ready to move things to RFC. 
There's a certain amount of overhead in getting reviewers, tracking 
status, doing write-ups, tracking dependencies, and simply getting 
people's attention associated with every document, and multiplying that 
by some large number of documents seems unappealing. Feel free to tell 
me I'm wrong and there's some easy way to mitigate that.

> 3) every feature includes information about why we think it is RF.
> This might be because it is an implementation of an old algorithm, it
> might be because the IPR has been granted RF, or it might be because
> we are so brilliant no one could have ever thought of this idea
> before.

So I want to do this. We're currently talking at Mozilla about how to do 
this (and I have buy-in from legal that we _can_ do this). But three points:

1. I think many will view most of the above arguments as weak. An old 
algorithm might indeed be RF, but its use in combination with another 
old algorithm might be patentable. There might be IPR on a technique 
that has been granted RF, but the same argument applies here as to old 
algorithms. I don't think I even have to speak to the last one. 
Fundamentally, I don't know of any way to ensure new work is RF other 
than sitting down and reading a lot of patents. I'm sure that's not 
something you want to enshrine into an IETF process.

2. I don't want to put this up as a bar for _contributing_ features. 
That will just discourage people from contributing. Even if we're, as 
Monty puts it, "being thorough in parts", there are significant 
economies of scale to be had by a small group of people researching the 
IPR for all the features. That's what protects you against the 
"combinations" issue I raised in point 1.

3. In another e-mail you claimed we could "easily come up with a much 
better charter"... I don't think figuring out the details here is going 
to be easy. Even if we said today that we're going to have volunteer 
participants fund some consortium to go hire a law firm and some 
engineers to review all the features and publish results to the WG 
(which I think would address my other two points), this isn't something 
we've talked about doing here before, people will have to get buy-in 
from _their_ legal departments, estimate costs, get funding approval... 
this will take weeks and/or months, even though we don't need the 
results until we're prepared to publish an RFC. So why would we block WG 
formation on that?

---

From: John Koleszar <[email-masked]>
Date: Wed, 16 Jan 2013 10:17:35 -0800

On Wed, Jan 16, 2013 at 9:50 AM, Timothy B. Terriberry <[email-masked]>wrote:

> 2. It also complicates the IPR story. You need to prove the fallbacks are
> just as RF as the original features. Even if you just disable some function
> f, computing g(f(x)) might have been unpatented while computing g(x)
> directly is patented. Since mere combinations of features can be patented,
> more combinations means more risk.
>
> 3. I raised the testing issue elsewhere... and while it's easy to dismiss
> this and say we only test a few cases, this poses real security issues: how
> do you know disabling feature X, Y, and Z doesn't expose your
> implementation to a buffer overflow? I know you think codecs are so complex
> they can never be adequately tested anyways, but in my experience video
> moves enough data through the codec that you actually can test the vast
> majority of branches and combinations of things relatively easily with
> automated testing, and the pieces that remain are small enough that you can
> reason about them. The small number of security vulnerabilities found in
> the codecs I've worked on compared to the average in projects like, say,
> FFmpeg bears this out. But when you multiply that work by 2^100, then you
> have a problem.
>
> some flag that was in use before, and you'll find that a) the fallback is
> buggy, b) it simply isn't implemented, c) maybe it is implemented, but in
> unoptimized software instead of hardware, and now things are too slow to
> actually be useful, or d) any number of other problems. Even if we provide
> test vectors, and even if (through astounding diligence) every
> implementation actually tests against them, as you point out we can only
> cover a few cases, which likely won't be the ones we wind up actually
> interested in.
>
> the flags to simply avoid implementing whatever random features they deem
> unimportant for whatever reason. We already see this with things like H.264
> which have a set of well-defined profiles... and then a separate set of the
> things that actually work in real, deployed implementations. The sets are
> not at all the same, and this is what allows Google to post VP8 performance
> benchmarks that beat "the features of H.264 that actually have broad enough
> support that we can enable them in our YouTube encodes." I.e., there's a
> real, measurable cost here, not just a theoretical one.
>
> 6. People willing to act unscrupulously can easily intimidate others into
> disabling features even when they don't really need to. The reality is that
> most people, confronted with someone telling them that "feature X might be
> unsafe" will simply disable it instead of trying to find out if that's
> really true. For many the analysis itself would be more costly than any
> loss of compression (especially device manufacturers who don't even bear
> the network transmission costs). But those that do bear sufficient cost to
> justify a real analysis have to interoperate with those who don't. So you
> enter a race to the bottom.
>
> If you did, you would have just taken that feature out in the first place.
> So we're making decisions about which features justify all the costs of a
> flag (in design time, implementation time, analysis time, testing time,
> attack surface, increased risk of interoperability failure) without any
> knowledge of the benefits.
>
> compression performance to get something that's RF, a much simpler strategy
> with a much higher probability of success is to just fall back to a much
> older codec. Like, say, Theora. Or if that's not safe enough for you, H.261.

I agree with Tim on all of the above.