Subject: [codec] НА: Re: Codec BOF approved, much work needed

From: "Slava Borilin" <[email-masked]>
Date: Fri, 19 Jun 2009 20:21:35 +0400

Agree with Roni- reqs first. Suggest to change 3 to «requirements fot codec».

Slava Borilin 

----- Исходное сообщение -----
От: Ron Even <[email-masked]>
Отправлено: 19 июня 2009 г. 19:59
Кому: [email-masked] <[email-masked]>
Тема: Re: [codec] Codec BOF approved, much work needed

Cullen,

I looked at the agenda

1. Administrative (Chairs, 5 min)

  - Note takers, Jabber scribes

  - Agenda bashing

2. Introduction and Scoping of BOF (Chairs/ADs, 15 min)

3. Proposed CODECs (TBD, 25 min)

  - Expect to have two or more drafts here

4. Charter Discussion (All, 40 min)

5. Decisions and HUMs (All, 5 min)

6. Conclusions and Next Steps (Chairs/ADs, 5 min)

And it looks to me that it starts with assumption that we will have a new
working group.

I am not sure that that this was agreed and yet I see no time allocated to
discuss the topic. I would expect that item 3 will be rational for doing
codec at the IETF considering that this work is done also by other SDOs that
the IETF is working with.

Now even if there will be such a WG starting with proposed CODECS is like
having a solution before the requirements. In other standard bodies
requirements are not just names but they also have values. So what is the
purpose of proposing codecs, do we need to select at the BOF which one will
be standardize, to me it looks like this should be done later.

Roni Even

> -----Original Message-----

> From: [email-masked] [mailto:[email-masked]] On Behalf

> Of Cullen Jennings

> Sent: Friday, June 12, 2009 8:32 PM

> To: [email-masked]

> Subject: [codec] Codec BOF approved, much work needed

> I have approved the CODEC BOF proposal. However, we need a bunch of

> work to get ready.  Various people on IESG/IAB were not comfortable

> with the current charter and agenda so we need to work on theses

> before the meeting.

> A draft agenda/charter Jason provided are at

> Agenda: http://svn.resiprocate.org/rep/ietf-drafts/codec-bof/agenda.txt

> Charter: http://svn.resiprocate.org/rep/ietf-drafts/codec-

> bof/charter.txt

> After discussion with the IESG/IAB:

> I'd like to remove the "as Proposed Standard" from the charter text

> for both milestones and see how the discussion goes in the BOF before

> deciding which way this should go.

> I'd like to see more consideration around the issues of designing a

> codec that did a good job of mapping a real time stream onto IP

> packets. The way IP deals with packet loss and congestion has

> implications for designing a codec that works well over IP. I know

> some of the discussion I have had off list people talked about how we

> wanted a codec that supported adaptive bit rates that could change on

> the fly to be able to work well on an IP network.

> I will be working the folks to make sure the agenda has time to

> directly address the key topics which include (more may be coming):

> Will we be able to get qualified people to do and review the work?

> Key design goals of a new codec?

> Do we need a new codec or are there already ones defined that meet the

> needs?

> I have asked Jason Fischl  and Jean-Marc Valin to co-chair this BOF.

> Thanks,

> Cullen <RAI AD>

> _______________________________________________

> codec mailing list

> [email-masked]

> https://www.ietf.org/mailman/listinfo/codec

---

From: "Roni Even" <[email-masked]>
Date: Fri, 19 Jun 2009 20:46:50 +0300

Slava,
This was not my suggestion, please read what I wrote, I suggested to discuss the rational for doing codec work at the IETF.

Anyhow codec themselves should not be presented at a BOF or maybe we can hum on the codecs in the BOF and be done with this work

Roni

> From: Slava Borilin [mailto:[email-masked]]
> Sent: Friday, June 19, 2009 7:22 PM
> To: Ron Even; [email-masked]
> Subject: НА: Re: [codec] Codec BOF approved, much work needed
>

---

From: "Roni Even" <[email-masked]>
Date: Fri, 19 Jun 2009 21:41:04 +0300

Hi,
Inline some comments (sections starting with RE) on the charter.

Purpose:

This new working group would be chartered with the purpose of collecting
expertise within the IETF in order to review the design of audio codecs
specifically for use with the Internet. 

Unlike other SDOs, these codecs would be optimized for use on the Internet, 

RE: I would argue that this statement is not correct and I have not seen any data supporting this statement, the fact is that there are numerous codecs coming from other SDOs being used on the Internet. I would also say that optimize for the Internet is not an application description since it is to wide, eventually this will be reflected if you start to discuss requirements when you will need to describe the applications that will use this codec.

and as much as possible choose technology that does not require paying patent royalties.

RE: Is this a selection criteria for selecting a codec.

The Internet Low Bit Rate Codec (iLBC)  work was done in AVT but it was felt
that subsequent work should not be done in the AVT working group. 

RE: to say done is overstating what was done in AVT comparing to other bodies. Furthermore it is not the same to publish a codec done by some company as an Informational and specifying requirements and selecting one codec from multiple proposal based on quality criteria

This new working group will have as its primary purpose the standardization of a
multi-purpose audio codec that can be used in various situations on the
Internet. Some of the proposed Internet-specific requirements include:
* scalable and adaptive bit rate;
* various sampling rate profiles from narrow-band to super-wideband;

RE: Just to clarify do you mean 3.5Khz, 7Khz and 14 Khz?

* scalable complexity;
* low latency; and 
* resilience to packet loss.

RE: I think that the first stage is to describe the use scenarios before giving requirements. For the current text I can add multiple other requirements for example:
Support for hands free operation (open mic), speech and music, multiple languages support, fax, stereo, 5+1, layered codec .........

There are a number of wide-band capable codecs defined by other SDOs. For
instance, G.722 is seeing adoption in Enterprise applications since it is
relatively simple and low-cost to deploy. However, it has a high, fixed
bitrate and is not appropriate for mobile applications where spectrum
efficiency is important or in consumer applications where available
bandwidth is fluctuating or limited. G.722.2 (AMR-wideband) has been adopted
by the 3GPP as a wideband standard for mobile applications. G.722.2 is
relatively high cost due to patent royalties and is seeing minimal
deployments outside of mobile handsets making it challenging to create
wideband experiences on Internet-capable mobile devices when extending
beyond the mobile network. In other cases, proprietary codecs are being
adopted which further create islands with no interoperability unless
widespread transcoding is performed. Transcoding leads to higher costs and
lower quality.

RE: Basing your discussion on G.722 and G.722.2 is not correct (G.722 is 7Khz) the latest codecs technology will be AMR-WB+, MPEG4 AAC-LD or scalable, eAAC+, G.729 with the new enhancement layers for wide band, G.718. Note that the ITU is working on embedded variable bit rates codec technology.

The goal of this working group is to define a single codec with multiple
profiles which can be made available on a wide variety of Internet-capable
devices including low-power, mobile devices as well as devices capable of
utilizing high quality, high bitrate audio.

RE: What is multiple profiles is it embedded variable rate (layered codec) or is it different modes (like G.722.1 and G.722.1C) using the same codec technology.

Proposed Deliverables:

1) Requirements for wideband, Internet audio codec(s).
2) Algorithm description for wideband, Internet audio codec(s) as Proposed
Standard. 
3) Specification of payload format(s) for defined codecs as Proposed
Standard

RE: Other deliverables - testing procedure and metrics. Codec selection criteria - which requirements are more important. I am also not sure what is algorithm description means, I would except a fixed point source reference if we want to have bit exactness between multiple implementations. 
The deliverables should include quality test results from independent test labs to enable selection.
I also thought from the above text that we are talking about super wideband and not wide band codec.

Thanks
Roni Even

---

From: Koen Vos <[email-masked]>
Date: Fri, 19 Jun 2009 13:19:37 -0700

Quoting Roni Even:
> For the current text I can add multiple other requirements for example:
> Support for hands free operation (open mic),

You mean built-in echo cancellation and noise suppression? Or are you  
just specifying a test requirement?

> speech and music, stereo

People on this list seem to think
- efficient speech codec at low bitrates
- high quality for music/stereo at higher rates

> multiple languages support,

Most codec standards are in C. Or are you again talking test specs?

> fax,

Fax has problems with any VoIP codec. Why not stick with T.38?

> 5+1

Not sure how important 5+1 is for real-time communications.

> layered codec

The use case is limited to certain multi-party calls, but it's a valid  
point to discuss. The alternative is transcoding.

best,
koen.

Quoting Roni Even:
> RE: I would argue that this statement is not correct and I have not  
> seen any data supporting this statement, the fact is that there are  
> numerous codecs coming from other SDOs being used on the Internet. I  
> would also say that optimize for the Internet is not an application  
> description since it is to wide, eventually this will be reflected  
> if you start to discuss requirements when you will need to describe  
> the applications that will use this codec.
>
>
> RE: to say done is overstating what was done in AVT comparing to  
> other bodies. Furthermore it is not the same to publish a codec done  
> by some company as an Informational and specifying requirements and  
> selecting one codec from multiple proposal based on quality criteria
>
> RE: I think that the first stage is to describe the use scenarios  
> before giving requirements. For the current text I can add multiple  
> other requirements for example:
> Support for hands free operation (open mic), speech and music,  
> multiple languages support, fax, stereo, 5+1, layered codec .........
>
>
> RE: Basing your discussion on G.722 and G.722.2 is not correct  
> (G.722 is 7Khz) the latest codecs technology will be AMR-WB+, MPEG4  
> AAC-LD or scalable, eAAC+, G.729 with the new enhancement layers for  
> wide band, G.718. Note that the ITU is working on embedded variable  
> bit rates codec technology.
>
>
> RE: What is multiple profiles is it embedded variable rate (layered  
> codec) or is it different modes (like G.722.1 and G.722.1C) using  
> the same codec technology.
>
>
> Proposed Deliverables:
>
> RE: Other deliverables - testing procedure and metrics. Codec  
> selection criteria - which requirements are more important. I am  
> also not sure what is algorithm description means, I would except a  
> fixed point source reference if we want to have bit exactness  
> between multiple implementations.
> The deliverables should include quality test results from  
> independent test labs to enable selection.

---

From: "Roni Even" <[email-masked]>
Date: Fri, 19 Jun 2009 23:38:28 +0300

Inline

> Of Koen Vos
> Sent: Friday, June 19, 2009 11:20 PM
> To: [email-masked]
> Subject: Re: [codec] Codec BOF approved, comments on charter
> 
> Quoting Roni Even:

This are codec requirements. Solution and tests are the next step

> - high quality for music/stereo at higher rates

This is a requirement that need numbers not words

> Most codec standards are in C. Or are you again talking test specs?

Requirements, speech quality not only for English language, that will be
reflected in the test requirements

> Fax has problems with any VoIP codec. Why not stick with T.38?

Need to specify the requirement, I can live without

> Not sure how important 5+1 is for real-time communications.

Music!!!!!
> point to discuss. The alternative is transcoding.

This is not multiparty it will be layering to support the multi rate at
different quality. Narrow band lower layer with enhancement layers for wide
band and super wideband like G.729 and G.718

---

From: "Slava Borilin" <[email-masked]>
Date: Sat, 20 Jun 2009 00:42:59 +0400

I beleive by "languages" Roni meant english, russian spanish etc.
5.1 is not inportant (yet), but stereo (2) is also quite interesting for conferencing.

________________________________

От: [email-masked] от имени Koen Vos
Отправлено: Сб, 20.06.2009 0:19
Кому: [email-masked]
Тема: Re: [codec] Codec BOF approved, comments on charter

best,
koen.

---

From: "Slava Borilin" <[email-masked]>
Date: Sat, 20 Jun 2009 00:52:36 +0400

This is not multiparty it will be layering to support the multi rate at
different quality. Narrow band lower layer with enhancement layers for wide
band and super wideband like G.729 and G.718
'
it is not necessarily to be multy-layer from the bitstreampoint of view  - it can be multiple encoding runs to cover several parts of spectrum, but still can be sent out to network as "solid" stream..
scalably really means ability to "strip out" automatically 'extra layers', which, first of all, make sense in the conferencing enviroment  (as such stripping should be done at the server, after encoding). 
it is similar to H.264SVC.
 
example - SPIRIT developed scalable voice codec many yeasr ago (it was done even before G.729.1 was started).
so the most benefits are in the conferencing space - comparing non-scalable codec @MGW handling 400 chalnnels, and with scalable (=transcoding free) codec it goes to 8000 channels on the same hardware.
 
for p2p PLC you do not need scalability much- as most codecs now are VBR, its ehough already to adapt codec (at the source) to the bandwidth.
but without scalability, you can't change the bitrate later (on GW or conferencing server).
 
Slava Borilin

От: [email-masked] от имени Roni Even
Отправлено: Сб, 20.06.2009 0:38
Кому: 'Koen Vos'; [email-masked]
Тема: Re: [codec] Codec BOF approved, comments on charter

Inline

Music!!!!!

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 19 Jun 2009 17:00:04 -0400

Hi,

My thoughts on the requirements.

> RE: Just to clarify do you mean 3.5Khz, 7Khz and 14 Khz?

Obviously 8 kHz (sampling rate) can be useful for PSTN compatibility. Otherwise,
we probably want from 16 kHz, all the way up to 48 kHz or so.

> Support for hands free operation (open mic),

I assume that by that you mean "robustness to background noise and
reverberation". I agree that this is important, although not too hard to
achieve once we target higher bit-rates anyway (it's mainly tough on <8 kbit/s
codecs).

> speech and music

Definitely important -- at least for the higher sampling rates and bit-rates.

> multiple languages support

That's important, although in practice I've never seen a codec that would work
on English, but not on German. Obviously, at low bit-rate you want to be
careful to preserve the pitch in tonal languages, but I don't think we'll have
much problem with languages.

> fax

T.38

> stereo, 5+1

Stereo is definitely useful. 5.1 is worth supporting, but that would most likely
be through multiple independent channels/pairs, so that's probably not that
much of an issue.

> layered codec .........

That one we can debate, but my initial thought is that I'm not sure it's worth
the trouble in this case. I.e. is it worth sacrificing efficiency for the
ability to cut bits in the middle of the path (rather than just tell the
encoder to reduce its rate).

Cheers,

   Jean-Marc

---

From: Koen Vos <[email-masked]>
Date: Fri, 19 Jun 2009 17:30:27 -0700

>> Not sure how important 5+1 is for real-time communications.

Most music is streamed and has no hard real-time requirements, and  
codecs like AAC or Vorbis are good choices for this. I'm fine with  
having my music-on-hold in mere stereo ;)

> band and super wideband like G.729 and G.718

In a one-to-one call, an encoder should not transmit higher-quality  
layers if these layers are then stripped off in the channel.

The codec and (initial) bitrate are negotiated during call setup, and  
we've already established the requirement of a control mechanism to  
set the bitrate in case of network congestion. So the encoder will  
just send whatever the other side wants to, and can, receive.

koen.

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 19 Jun 2009 21:30:27 -0400

> having my music-on-hold in mere stereo ;)

Actually, there *are* music applications with real-time requirements.
Some examples are:
- Live network music performances (e.g. http://www.soundjack.eu/)
- Wi-fi audio equipment (you want sound in sync with your TV)
- Network sound servers (e.g. netjack http://www.jackaudio.org/)

Also, it turns out that once you when you want really high quality, the
speech coding techniques (based on LPC with pitch) tend to saturate and
you end up with transform codecs that perform well on music. So at this
point, you get music for free.

> we've already established the requirement of a control mechanism to set
> the bitrate in case of network congestion. So the encoder will just send
> whatever the other side wants to, and can, receive.

I totally agree here.

	Jean-Marc

---

From: Randell Jesup <[email-masked]>
Date: Fri, 19 Jun 2009 22:36:04 -0400

>- Network sound servers (e.g. netjack http://www.jackaudio.org/)

Two of those do NOT have hard-real-time requirements.  

WiFi audio may not be hard-real-time either, though the example you give
might be the only hard-real-time case here, since it has to try to sync
to an external device.  This is not "internet" audio though, it's LAN
audio.  Might be addressable or happen to be covered, but I wouldn't
make this a primary goal.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 19 Jun 2009 23:18:36 -0400

> Two of those do NOT have hard-real-time requirements.  

Then I think you misunderstood what I wrote:

- Live network music performances: I'm in Montreal playing guitar and
you're in New York singing. We can synchronise each other as long as the
one-way delay is less than about 25 ms. If you remove the network delay,
that leaves only a few ms delay for the codec. But it *is* possible (and
it's been done with CELT already).

- Network sound servers: The idea is to log on to a remote machine and
run multimedia applications. For this to work well, the sound has to
arrive with as little delay as possible, otherwise there is a perceived lag.

Cheers,

	Jean-Marc

---

From: Randell Jesup <[email-masked]>
Date: Sat, 20 Jun 2009 02:43:10 -0400

>arrive with as little delay as possible, otherwise there is a perceived lag.

You're correct, I mis-understood.  I thought you meant "broadcast" of
live performances, and a stored audio streaming server.

Perhaps "live musical collaberation", and "Remote PC access with
multimedia".  That last one, though, isn't as delay-sensitive (though it
should be low delay) - the issue is more one of audio-video sync.

---

From: Christopher Montgomery <[email-masked]>
Date: Sat, 20 Jun 2009 11:51:50 -0400

> RE: I would argue that this statement is not correct and I have not seen any data supporting this statement

OK, 'unlike _many_ SDOs'.  We are interested in switched-packet
networks and variable transmission rates, not fixed-bandwidth
broadcast channels.  Perhaps it should be stated in the positive (what
we are) instead of the negative (unlike what we are not).  But let's
not bikeshed this too thoroughly.

> The fact is that there are numerous codecs coming from other SDOs being used on the Internet.

Because they are well suited, or because they're what was available at
the moment someone saw an immediate short-term business opportunity?
We need to think past ad-hoc deployment.

>  I would also say that optimize for the Internet is not an application description since it is to wide

"Variable-bandwidth and switched-packet networks"  and "non-linear /
interactive use" might be a better starting point. 'Optimized for the
internet' is not itself meaningless, merely imprecise.

> RE: Is this a selection criteria for selecting a codec.

yes.  YES.

Please get us away from nickel-and-dime IPR attached to every bit
transmitted on the Internet.  The physical transport itself costs
enough. Please end the need to regularly re-up permissions from
multiple licensing bodies to use even 'free' (as in beer) codecs
before being allowed to ship a single line of code. This 'permission
to breathe' model prevents entire swaths of possible applications and
tries to milk 'value' out of something that, at the beginning, has
none.

Codecs are not rocket science.  They are well understood. The math and
engineering are both commodities.

I'm tired of smothering every baby in the cradle with royalties as a
way of offsetting the costs of infant formula.  There will never be a
single clear winner in any codec space that mandates licensing costs
or 'permission' to use.  The only reason mp3 ever got as far as it was
was the incorrect perception for many years that MPEG was not
enforcing patent claims.

> RE: Just to clarify do you mean 3.5Khz, 7Khz and 14 Khz?

No, I think that is overly fixating on voice-only.  Even voice codecs
don't limit to 14kHz today.

> RE: Basing your discussion on G.722 and G.722.2 is not correct (G.722 is 7Khz) the latest codecs technology
> will be AMR-WB+, MPEG4 AAC-LD or scalable, eAAC+, G.729 with the new enhancement layers for wide band,
> G.718. Note that the ITU is working on embedded variable bit rates codec technology.

I realize you're responding to a slightly different point, but it's
worth reiterating:

Anything that requires a license, even a free license, or patent
royalties is a non-starter, full-stop.  If a ten year old student
programmer in Indonesia has to read a legal document or consult a
lawyer to figure out if he's allowed to use a codec, then it's failed
the first and most important test. We need not even bring these codecs
up or list them.

> RE: What is multiple profiles is it embedded variable rate (layered codec) or is it different modes
> (like G.722.1 and G.722.1C) using the same codec technology.

Either.  Both. Whichever works.  Both fit the current usage of the
word.  We have a next generation of codec technology just now gelling,
just now at the stage of benefitting greatly from the eyes and minds
of experience.

> RE: Other deliverables - testing procedure and metrics

Tournament development with extensive listening tests.  May the best ideas win.

> Codec selection criteria - which requirements are more important.

> I am also not sure what is algorithm description means

Plain english* description of spec.  Like a text that explains how a
mathemetician arrived at a proof.

Monty

(* turn of phrase, not literally mandating English)

---

From: Henry Sinnreich <[email-masked]>
Date: Sat, 20 Jun 2009 09:18:21 -0700

Roni,
(Topic changed for focus)

Roni,

Just to make it clear, royalty free was one of the top motivations and should probably be the main requirement for the Internet codec.

Interoperability with existing codecs is nice, but IMHO a secondary order requirement, given that most interactive communications are already on the Internet and will be so even more in the future. Vendors and operators are always free to include some other codecs and let SIP do the codec negotiation - that's what it is for in SIP.

Henry

On 6/19/09 1:41 PM, "Roni Even" <[email-masked]> wrote:

Hi,

Purpose:

* scalable complexity;
* low latency; and

Proposed Deliverables:

Thanks
Roni Even

---

From: "Roni Even" <[email-masked]>
Date: Sat, 20 Jun 2009 19:46:55 +0300

Henry,

I fully agree that royalty free is a good requirement. I just expect that
people who ask for it will come in good faith and provide interoperable
royalty free technology and solutions. Having two products that run the same
"royalty free" codec on the Internet but who cannot connect to each other
and do not offer any such option to other developers to provide it is  bad.

>From knowing your views, I would expect that you will share my view that
having to install on my computer multiple communication clients is not what
we all want as end users. I would prefer to have one client that will be
able to talk with other clients. Some of the requirements for this work  so
far is coming from people who want to continue to provide non-interoperable
products using royalty free technology done by others.  

Telecommunication does not work if there is no interoperability and if the
signaling interoperability will continue to be through PSTN gateways there
is no need for wideband codecs.

Roni

From: Henry Sinnreich [mailto:[email-masked]] 
Sent: Saturday, June 20, 2009 7:18 PM
To: Roni Even; [email-masked]
Subject: Royalty free is top motivation

Roni,

Roni,

Just to make it clear, royalty free was one of the top motivations and
should probably be the main requirement for the Internet codec.

Interoperability with existing codecs is nice, but IMHO a secondary order
requirement, given that most interactive communications are already on the
Internet and will be so even more in the future. Vendors and operators are
always free to include some other codecs and let SIP do the codec
negotiation - that's what it is for in SIP.

Henry

Hi,

Purpose:

RE: I would argue that this statement is not correct and I have not seen any
data supporting this statement, the fact is that there are numerous codecs
coming from other SDOs being used on the Internet. I would also say that
optimize for the Internet is not an application description since it is to
wide, eventually this will be reflected if you start to discuss requirements
when you will need to describe the applications that will use this codec.

and as much as possible choose technology that does not require paying
patent royalties.

RE: to say done is overstating what was done in AVT comparing to other
bodies. Furthermore it is not the same to publish a codec done by some
company as an Informational and specifying requirements and selecting one
codec from multiple proposal based on quality criteria

This new working group will have as its primary purpose the standardization
of a

* scalable complexity;
* low latency; and

RE: I think that the first stage is to describe the use scenarios before
giving requirements. For the current text I can add multiple other
requirements for example:
Support for hands free operation (open mic), speech and music, multiple
languages support, fax, stereo, 5+1, layered codec .........

RE: Basing your discussion on G.722 and G.722.2 is not correct (G.722 is
7Khz) the latest codecs technology will be AMR-WB+, MPEG4 AAC-LD or
scalable, eAAC+, G.729 with the new enhancement layers for wide band, G.718.
Note that the ITU is working on embedded variable bit rates codec
technology.

RE: What is multiple profiles is it embedded variable rate (layered codec)
or is it different modes (like G.722.1 and G.722.1C) using the same codec
technology.

Proposed Deliverables:

RE: Other deliverables - testing procedure and metrics. Codec selection
criteria - which requirements are more important. I am also not sure what is
algorithm description means, I would except a fixed point source reference
if we want to have bit exactness between multiple implementations.
The deliverables should include quality test results from independent test
labs to enable selection.
I also thought from the above text that we are talking about super wideband
and not wide band codec.

Thanks
Roni Even

---

From: "Sandy (Alexander) MacInnis" <[email-masked]>
Date: Sat, 20 Jun 2009 09:52:15 -0700

That sounds great.

I'd like to point a few things that may or may not be obvious to everyone here. I'm not a lawyer; this is an engineer's description (mine), so don't take this a legal advice. I'm pretty familiar with patents, however.

A potentially major problem with trying to make something royalty-free with certainty, is the possibility of existing patents that may be infringed by the "free" technology. There are lots of patents out there.

Even if someone sat all by him/herself and created a codec from scratch, with or without referring to any textbooks, journal articles, or openly available source code, and the publishes the codec and declares that it's free to the world, that does not mean that it does not infringe on anyone's patents. This is true whether or not the creator of the codec files any patent applications on the work.

If such a codec is published and used, a holder of a patent may wait until there is widespread use of the codec, and then sue whoever they want. A deep-pockets company may make an attractive target. That could be a significant problem.

For that reason, something that is thought to be royalty-free may turn out not to be royalty free after all. It could go from seemingly free to quite expensive rather quickly, or it could even be dropped by major companies, or even opposed by them in advance.

It's difficult, but not quite impossible, to avoid this problem.

Some standards groups try to avoid this problem by asking contributors to declare what they know about any existing or planned patent coverage that may cover their submission, and ask contributors who have their own patents to indicate on what terms they will license the patents. That may help a lot, but it may not solve the problem. For one thing, contributors may not know about patents that their method might infringe - often they don't know. There might also be some legal wrangling over the final terms for royalty-free licensing of patents held by contributors.

Another way is to study very carefully all the patents that could possibly be infringed, for each one, either work around it or see that the patent has already expired. This is very hard to do, but it can be done.

This does not mean that everyone wanting to implement a codec that's said to be royalty free has to read contracts. It would help however if someone had done some diligence on sorting this out before the codec is published. With suitable public licenses, it should be possible for all the world's users to be confident that a codec is royalty-free, while also being high quality, interoperable, open source, and a published standard.

--Sandy

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Henry Sinnreich
Sent: Saturday, June 20, 2009 12:18 PM
To: Roni Even; [email-masked]
Subject: [codec] Royalty free is top motivation

Roni,

Roni,

Henry
[... remainder of thread deleted for brevity...]

---

From: Henry Sinnreich <[email-masked]>
Date: Sat, 20 Jun 2009 10:03:10 -0700

Roni,

>From knowing your views, I would expect that you will share my view that having to install on my computer multiple communication clients is not what we all want as end users.

I fully agree. As far as SIP goes however, all SIP phones and PC UAs have multiple codecs already.
I understand Skype had more then one codec as well, but am not sure.
Maybe our colleagues from Skype can share with us.

Henry

On 6/20/09 11:46 AM, "Roni Even" <[email-masked]> wrote:

Henry,
I fully agree that royalty free is a good requirement. I just expect that people who ask for it will come in good faith and provide interoperable royalty free technology and solutions. Having two products that run the same "royalty free" codec on the Internet but who cannot connect to each other and do not offer any such option to other developers to provide it is  bad.

>From knowing your views, I would expect that you will share my view that having to install on my computer multiple communication clients is not what we all want as end users. I would prefer to have one client that will be able to talk with other clients. Some of the requirements for this work  so far is coming from people who want to continue to provide non-interoperable products using royalty free technology done by others.

Telecommunication does not work if there is no interoperability and if the signaling interoperability will continue to be through PSTN gateways there is no need for wideband codecs.

Roni

Roni,

Roni,

Henry

Purpose:

* scalable complexity;
* low latency; and

Proposed Deliverables:

Thanks
Roni Even

---

From: Henry Sinnreich <[email-masked]>
Date: Sat, 20 Jun 2009 10:10:57 -0700

This are excellent points.
For what it's worth, going by the IETF policy is netter than nothing.
All we have to do is to disregard in the proposed IETF Codec WG existing codecs known to have royalty claims.
[Timber !!!  :-)  ]

Henry

On 6/20/09 11:52 AM, "Sandy (Alexander) MacInnis" <[email-masked]> wrote:

Some standards groups try to avoid this problem by asking contributors to declare what they know about any existing or planned patent coverage that may cover their submission, and ask contributors who have their own patents to indicate on what terms they will license the patents. That may help a lot, but it may not solve the problem.

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 20 Jun 2009 13:38:38 -0400

Sandy (Alexander) MacInnis a écrit :
> A potentially major problem with trying to make something royalty-free
> with certainty, is the possibility of existing patents that may be
> infringed by the "free" technology. There are lots of patents out there.

Yes, I'm aware of that and it has been a consideration throughout the
development of Speex and (now) CELT.

> If such a codec is published and used, a holder of a patent may wait
> until there is widespread use of the codec, and then sue whoever they
> want. A deep-pockets company may make an attractive target. That could
> be a significant problem.
>
> For that reason, something that is thought to be royalty-free may turn
> out not to be royalty free after all. It could go from seemingly free to
> quite expensive rather quickly, or it could even be dropped by major
> companies, or even opposed by them in advance.

Unfortunately, this is the sad state of the current patent system, which
we cannot change in the short term. In fact, this is not only a problem
for codecs, but for all software (and possibly other areas, but I won't
go into that). As it is, the only way to be absolutely sure not to
infringe on a patent is not to create anything. Obviously, that is not a
very worthwhile option.

Some people make it sound like only free software is affected, though
but that's not the case. You could have a company claiming that any of
the ITU or MPEG codecs infringe on their patent.

Fortunately, many patents in the audio coding world are written
specifically for a certain codec. The claims of these patents are
usually very strong (hard to defeat by prior art), but at the same time
very narrow so that accidental infringement is not too likely. It is the
broader patents that are more dangerous, but they are usually better
known, so easier to protect against.

> Another way is to study very carefully all the patents that could
> possibly be infringed, for each one, either work around it or see that
> the patent has already expired. This is very hard to do, but it can be done.

Yes, it can be done. And it is best done in a collaborative way, with as
many people as possible using their knowledge to examine possible patent
issues. I believe the IETF is a good place to do that.

Cheers,

	Jean-Marc

---

From: "Dr. Christian Hoene" <[email-masked]>
Date: Sat, 20 Jun 2009 20:41:59 +0200

> Just to make it clear, royalty free was one of the top motivations  
> and should probably be the main requirement for the Internet codec.

Isn't "royalty free" just a means to an end? Are Jean-Marc (and  
hopefully many others) not motivated to provide all users telephony at  
superb quality no mather whether the users can afford to pay for IPRs  
or for a lot of bandwidth?

Or do I get it wrong?

Cheers,
  Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 23 Jun 2009 13:08:03 +0200

Dear Sandy,

what do you think? How large is probability, that CELT is covered by
patents, what we are not aware of?

I just remember the patent struggle with the GIF file format
(http://en.wikipedia.org/wiki/Graphics_Interchange_Format), for which Unisys
requested license fees after the format was spread widely (refer also to
http://en.wikipedia.org/wiki/Submarine_patent)

Should a procedure be considered and described in the standard, on how to
act if the assumed to be royalty free codec is not free anymore? 

I am just thinking of some kind of updating the codec software that MUST be
supported.

Regards,

 Christian

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Sandy (Alexander) MacInnis
Sent: Saturday, June 20, 2009 6:52 PM
To: [email-masked]
Subject: Re: [codec] Royalty free is top motivation

That sounds great.

I'd like to point a few things that may or may not be obvious to everyone
here. I'm not a lawyer; this is an engineer's description (mine), so don't
take this a legal advice. I'm pretty familiar with patents, however.

A potentially major problem with trying to make something royalty-free with
certainty, is the possibility of existing patents that may be infringed by
the "free" technology. There are lots of patents out there.

Even if someone sat all by him/herself and created a codec from scratch,
with or without referring to any textbooks, journal articles, or openly
available source code, and the publishes the codec and declares that it's
free to the world, that does not mean that it does not infringe on anyone's
patents. This is true whether or not the creator of the codec files any
patent applications on the work.

If such a codec is published and used, a holder of a patent may wait until
there is widespread use of the codec, and then sue whoever they want. A
deep-pockets company may make an attractive target. That could be a
significant problem.

For that reason, something that is thought to be royalty-free may turn out
not to be royalty free after all. It could go from seemingly free to quite
expensive rather quickly, or it could even be dropped by major companies, or
even opposed by them in advance.

Some standards groups try to avoid this problem by asking contributors to
declare what they know about any existing or planned patent coverage that
may cover their submission, and ask contributors who have their own patents
to indicate on what terms they will license the patents. That may help a
lot, but it may not solve the problem. For one thing, contributors may not
know about patents that their method might infringe - often they don't know.
There might also be some legal wrangling over the final terms for
royalty-free licensing of patents held by contributors.

Another way is to study very carefully all the patents that could possibly
be infringed, for each one, either work around it or see that the patent has
already expired. This is very hard to do, but it can be done.

This does not mean that everyone wanting to implement a codec that's said to
be royalty free has to read contracts. It would help however if someone had
done some diligence on sorting this out before the codec is published. With
suitable public licenses, it should be possible for all the world's users to
be confident that a codec is royalty-free, while also being high quality,
interoperable, open source, and a published standard.

--Sandy

  _____  

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Henry Sinnreich
Sent: Saturday, June 20, 2009 12:18 PM
To: Roni Even; [email-masked]
Subject: [codec] Royalty free is top motivation

Roni,

Roni,

Henry

---

From: "Sandy (Alexander) MacInnis" <[email-masked]>
Date: Tue, 23 Jun 2009 13:53:25 -0700

Sorry, I'm not familiar with CELT and patents that might perhaps cover it. Without knowing, I'd assume there is some risk.

Interesting question about what to do if a standard is created and one later decides that it's not actually royalty free.  I don't know. It seems to me that no matter what the group says in advance, if this happens, it happens, and there's not much a standards group can do about it. I hope I'm wrong. The choices depend, presumably, on whether the (hypothetical) patent holder participated in the standards activity.

--Sandy

________________________________
From: Christian Hoene [mailto:[email-masked]]
Sent: Tuesday, June 23, 2009 7:08 AM
To: Sandy (Alexander) MacInnis; [email-masked]
Subject: Procedure on how to deal with "submarine" patents?

Dear Sandy,

what do you think? How large is probability, that CELT is covered by patents, what we are not aware of?

I just remember the patent struggle with the GIF file format (http://en.wikipedia.org/wiki/Graphics_Interchange_Format), for which Unisys requested license fees after the format was spread widely (refer also to http://en.wikipedia.org/wiki/Submarine_patent)

Should a procedure be considered and described in the standard, on how to act if the assumed to be royalty free codec is not free anymore?

I am just thinking of some kind of updating the codec software that MUST be supported...

Regards,

 Christian

From: [email-masked] [mailto:[email-masked]] On Behalf Of Sandy (Alexander) MacInnis
Sent: Saturday, June 20, 2009 6:52 PM
To: [email-masked]
Subject: Re: [codec] Royalty free is top motivation

That sounds great.

--Sandy

Roni,

Henry

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 23 Jun 2009 17:16:15 -0400

Hi Sandy,

On 6/23/09 4:53 PM, "Sandy (Alexander) MacInnis" <[email-masked]>
wrote:

> Sorry, I'm not familiar with CELT and patents that might perhaps cover it.
> Without knowing, I'd assume there is some risk.
> 
I share this feeling, but, due to the legal situation, I cannot be more
precise.
>  
> Interesting question about what to do if a standard is created and one later
> decides that it's not actually royalty free.  I don't know. It seems to me
> that no matter what the group says in advance, if this happens, it happens,
> and there's not much a standards group can do about it. I hope I'm wrong. The
> choices depend, presumably, on whether the (hypothetical) patent holder
> participated in the standards activity.
> 
There are SDOs which have a goal to create only RF standards.  These
organizations generally have some form of a conflict resolution mechanism
that kicks in when non-RF IPR is  discovered---often regardless of the
participation of the patent holder.  One typical example is W3C, with its
PAG concept.  Sometimes these mechanisms even work.

The IETF is not one of these organizations.  In fact, the IETF does not even
have a requirement that essential patents must be licensable under RAND
terms (as most RAND organizations, including ITU and ISO/IEC, have).
Arguably, the risk of using a hypothetical IETF codec, advertised as RF or
not, is higher than an ITU codec, at least for those entities which are
willing to dish out RAND licensing money.  That is unless there is very
broad participation in the IETF work in question, but that seems unlikely in
the current situation.

I know that this does not help the open source people, though.  I have no
solution for them.  Codecs are IMHO playing in a RAND licensing ecosystem,
and I do not see a way to change that.

Regards,
Stephan

> Should a procedure be considered and described in the standard, on how to act
> if the assumed to be royalty free codec is not free anymore?
>  
> supportedŠ
> certainty, is the possibility of existing patents that may be infringed by the
> "free" technology. There are lots of patents out there.
>  
> Even if someone sat all by him/herself and created a codec from scratch, with
> or without referring to any textbooks, journal articles, or openly available
> source code, and the publishes the codec and declares that it's free to the
> world, that does not mean that it does not infringe on anyone's patents. This
> is true whether or not the creator of the codec files any patent applications
> on the work.
>  
> For that reason, something that is thought to be royalty-free may turn out not
> to be royalty free after all. It could go from seemingly free to quite
> declare what they know about any existing or planned patent coverage that may
> cover their submission, and ask contributors who have their own patents to
> indicate on what terms they will license the patents. That may help a lot, but
> it may not solve the problem. For one thing, contributors may not know about
> patents that their method might infringe - often they don't know. There might
> also be some legal wrangling over the final terms for royalty-free licensing
> of patents held by contributors.
> 
>  
> 
> Another way is to study very carefully all the patents that could possibly be
> infringed, for each one, either work around it or see that the patent has
> Just to make it clear, royalty free was one of the top motivations and should
> probably be the main requirement for the Internet codec.
> 
> always free to include some other codecs and let SIP do the codec negotiation
>  that¹s what it is for in SIP.
> 
> Henry
>