Subject: [video-codec] Strategy for an RF video codec

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 15 Jan 2013 04:28:35 +0000

I'm going to assert that it will be much more difficult to get an state of the art RF video codec than to get an good RF audio codec. Some of the people working in the CODEC WG had strategies to increate the likely hood of an RF codec but the WG never had any strategies to achieve this. I think the video codec WG should have a strategy to optimize the odds of RF. 

Here's my proposal for a strategy and I think this, or whatever it is we decide is our strategy, should be in the charter:

1) every separable part of the codec becomes a separate feature and both sides negotiate the features they support. I'm imaging 100 or more features many of which depend on others or you can't use them. The minimal feature might just be uncompressed video. The odds of managing for all parts of an advanced video codec to be RF are low and this allows us to mitigate the risk by if some part if found to not be RF, deployments can disable that feature and the features that depend on them. This also makes for a much smother watt to incrementally advance the codec over time. Doing this means we are not doing a single codec but really a family of codecs . 

2) every feature that goes into the codec is specified in a separate draft to get separate IRP claims

3) every feature includes information about why we think it is RF. This might be because it is an implementation of an old algorithm, it might be because the IPR has been granted RF, or it might be because we are so brilliant no one could have ever thought of this idea before. 

Cullen

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 15 Jan 2013 07:03:43 -0800

Cullen,

This is a really interesting idea.

Can you give me some sense of how you envisioned the negotiation happening?
Traditionally, IETF uses one of three types of negotiation:

- One side offers some list of non-categorized features and the other side
picks
a subset. ("extensions")
- We have a bunch of feature categories and one side offers a list of which
ones if supports in each category. The other side picks one from each column
("chinese menu}")
- We have a fixed number of feature combinations and one side offers
a list of the combinations it supports and the other side picks one
("suites")
- We have a list of features and one side lists the combinations it would
accept ("profiles" (?))

What did you have in mind here? I ask because it impacts the relationship
between the features and the required level of engineering
work.

-Ekr

On Mon, Jan 14, 2013 at 8:28 PM, Cullen Jennings (fluffy)
<[email-masked]>wrote:

>
> I'm going to assert that it will be much more difficult to get an state of
> the art RF video codec than to get an good RF audio codec. Some of the
> people working in the CODEC WG had strategies to increate the likely hood
> of an RF codec but the WG never had any strategies to achieve this. I think
> the video codec WG should have a strategy to optimize the odds of RF.
>
> Here's my proposal for a strategy and I think this, or whatever it is we
> decide is our strategy, should be in the charter:
>
> 1) every separable part of the codec becomes a separate feature and both
> sides negotiate the features they support. I'm imaging 100 or more features
> many of which depend on others or you can't use them. The minimal feature
> might just be uncompressed video. The odds of managing for all parts of an
> advanced video codec to be RF are low and this allows us to mitigate the
> risk by if some part if found to not be RF, deployments can disable that
> feature and the features that depend on them. This also makes for a much
> smother watt to incrementally advance the codec over time. Doing this means
> we are not doing a single codec but really a family of codecs .
>
> 3) every feature includes information about why we think it is RF. This
> might be because it is an implementation of an old algorithm, it might be
> because the IPR has been granted RF, or it might be because we are so
> brilliant no one could have ever thought of this idea before.
>
>
> Cullen
>
> _______________________________________________
> video-codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/video-codec
>

---

From: Monty Montgomery <[email-masked]>
Date: Tue, 15 Jan 2013 17:06:48 -0500

> 2) every feature that goes into the codec is specified in a separate draft to get separate IRP claims

Ooo, clever.  Wonder if it'll work.

> 3) every feature includes information about why we think it is RF. This might be because it is an implementation of an old algorithm, it might be because the IPR has been granted RF, or it might be…

I suppose if we're covering all the bases, being thorough in parts is
not substantially more work than being thorough all in one huge draft
statement.

...but I may want to leave that to the people currently owning most of
the actual work...

Monty

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Wed, 16 Jan 2013 15:36:15 +0000

On Jan 15, 2013, at 8:03 AM, Eric Rescorla <[email-masked]> wrote:

> - One side offers some list of non-categorized features and the other side picks
> a subset. ("extensions")
> a list of the combinations it supports and the other side picks one ("suites")

I had not really put too much thought into which style would work best but I was imaging "extensions" for interactive and "suites" for stored video. 

For interactive, the SDP would have the list of features supported by the receiver. The side encoding the video would look at that, and encode appropriately. The actual information of what the encoding was would be in the bitstream so any device could decode just from looking at the bitstream. 

For non interactive media - something more like vimeo, I would imagine that we would define a small number of suites defined in IETF specs and the video is encoded to one or more of the suites and the receiver either can deal with the suite or it can't. More or less how interactive video works today. It does allow new suites to be defined with existing features which would allow for fairly rapid roll out of new suites if a problem was found with existing suite set. 

> 
> -Ekr
> 
> 
> On Mon, Jan 14, 2013 at 8:28 PM, Cullen Jennings (fluffy) <[email-masked]> wrote:
>

---

From: John Koleszar <[email-masked]>
Date: Wed, 16 Jan 2013 08:42:20 -0800

On Wed, Jan 16, 2013 at 7:36 AM, Cullen Jennings (fluffy)<[email-masked]>
 wrote:

> I had not really put too much thought into which style would work best but
> I was imaging "extensions" for interactive and "suites" for stored video.
>
> For interactive, the SDP would have the list of features supported by the
> receiver. The side encoding the video would look at that, and encode
> appropriately. The actual information of what the encoding was would be in
> the bitstream so any device could decode just from looking at the bitstream.
>

This seems like a testing nightmare to me, in addition to the implicit
costs of carrying around fallback code and silicon that never gets used.

Also, with this proposed granularity of features, even if some given
feature level is interoperable, it doesn't mean that the resulting codec as
a whole satisfies all the other requirements. If the minimal feature is
uncompressed video, that doesn't make it a practical choice for the
applications we're trying to cover. If used, features should degrade
quality, not baseline functionality (which includes some lower bound on
quality).

>
> For non interactive media - something more like vimeo, I would imagine
> that we would define a small number of suites defined in IETF specs and the
> video is encoded to one or more of the suites and the receiver either can
> deal with the suite or it can't. More or less how interactive video works
> today. It does allow new suites to be defined with existing features which
> would allow for fairly rapid roll out of new suites if a problem was found
> with existing suite set.
>
>
I fail to see the distinction between codecs and suites -- couldn't each
suite be considered a codec in and of itself?

I think the underlying issue you're addressing here is how do you iterate
on codecs and how quickly can you do so, in response to new research or in
your case as an IPR risk mitigation strategy. I think there are some
straightforward approches to doing this in software, where you can just
download the codec definition at runtime and run it in a sandbox, but doing
so in hardware is much harder. You can take an approach where you define
algorithmic blocks and then the definition of how those blocks connect can
be described in the bitstream as is done with MPEG's RVC effort, but as far
as I know it hasn't been shown to be practical yet.

In other words, even if we only do one codec now, do we do anything
differently because we know there'll be another sometime soon after? I'd
argue that it's outside the scope of a codec definition effort, but
anything that can get the cycle time on codecs reduced so it can be
measured in months, not years, is a worthy goal in my opinion, so maybe
it's worth trying. It's a much harder problem than defining a new codec,
certainly.