Subject: [codec] Next Steps for WG

From: Cullen Jennings <[email-masked]>
Date: Thu, 13 Jan 2011 15:29:19 -0700

The OPUS draft authors believe the bitstream for OPUS will be ready to be "frozen" by the end of the month. At that point we plan to spend the following month testing and, baring any surprises, will likely go to WGLC after that. 

There has been some questions about what "frozen" means in IETF context. The bitstream could change any time up to when the IESG approves it which is after Working Group Last call and after IETF Last Call so there is no guarantee that nothing will change but ...  by "frozen" we mean that we believe no changes are needed and don't plan to make changes unless some significant problem is found.

Thanks,
Cullen, Jonathan, and Mike

---

From: Roman Shpount <[email-masked]>
Date: Thu, 13 Jan 2011 19:34:58 -0500

Just out of curiosity, have the OPUS codec been review against the list of
requirements that we put together?

In particular, I raised the requirement that multiple encoded streams form
different sources can be efficiently combined. Is it possible with OPUS? If
it is, then how?
_____________________________
Roman Shpount - www.telurix.com

On Thu, Jan 13, 2011 at 5:29 PM, Cullen Jennings <[email-masked]> wrote:

>
> The OPUS draft authors believe the bitstream for OPUS will be ready to be
> "frozen" by the end of the month. At that point we plan to spend the
> following month testing and, baring any surprises, will likely go to WGLC
> after that.
>
> There has been some questions about what "frozen" means in IETF context.
> The bitstream could change any time up to when the IESG approves it which is
> after Working Group Last call and after IETF Last Call so there is no
> guarantee that nothing will change but ...  by "frozen" we mean that we
> believe no changes are needed and don't plan to make changes unless some
> significant problem is found.
>
> Thanks,
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 13 Jan 2011 21:50:55 -0500

Hi Roman,

We believe that we meet all requirements that were identified. As for 
mixing multiple encoded streams, that was marked as desirable in the 
draft because it's very hard to achieve without sacrificing efficiency. 
To some extent, if you constrain the Opus encoder to a subset of the 
available options (CELT only without short blocks or post-filter), then 
you can create streams that can be mixed at roughly half the normal 
complexity. I think that's about as much as we can do here.

Cheers,

	Jean-Marc

On 11-01-13 07:34 PM, Roman Shpount wrote:
> Just out of curiosity, have the OPUS codec been review against the list
> of requirements that we put together?
>
> In particular, I raised the requirement that multiple encoded streams
> form different sources can be efficiently combined. Is it possible with
> OPUS? If it is, then how?
> Roman Shpount - www.telurix.com <http://www.telurix.com>
>
>
> On Thu, Jan 13, 2011 at 5:29 PM, Cullen Jennings <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>     The OPUS draft authors believe the bitstream for OPUS will be ready
>     to be "frozen" by the end of the month. At that point we plan to
>     spend the following month testing and, baring any surprises, will
>     likely go to WGLC after that.
>
>     There has been some questions about what "frozen" means in IETF
>     context. The bitstream could change any time up to when the IESG
>     approves it which is after Working Group Last call and after IETF
>     Last Call so there is no guarantee that nothing will change but ...
>       by "frozen" we mean that we believe no changes are needed and
>     don't plan to make changes unless some significant problem is found.
>
>     Thanks,
>     [email-masked] <mailto:[email-masked]>

---

From: "Christian Hoene" <[email-masked]>
Date: Fri, 14 Jan 2011 09:27:02 +0100

The easiest way to mix the multiple streams is to take advantage of the DTX
feature, which is available in the Silk and hybrid mode. However, I am not
aware whether DTX or VAD is supported in the CELT mode.

Christian

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Jean-Marc Valin
> Sent: Friday, January 14, 2011 3:51 AM
> To: Roman Shpount
> Cc: [email-masked]
> Subject: Re: [codec] Next Steps for WG
> 
> Hi Roman,
> 
mixing
> multiple encoded streams, that was marked as desirable in the draft
because
> it's very hard to achieve without sacrificing efficiency.
you can
> create streams that can be mixed at roughly half the normal complexity. I
> think that's about as much as we can do here.
> 
> Cheers,
> 
> 	Jean-Marc
> 
> > Just out of curiosity, have the OPUS codec been review against the
> > list of requirements that we put together?
> >
> > form different sources can be efficiently combined. Is it possible
> > with OPUS? If it is, then how?

---

From: Koen Vos <[email-masked]>
Date: Fri, 14 Jan 2011 22:41:10 +0100 (CET)

Even without DTX, the Voice modes will expose a Voice Activity flag in the decoder API. This will reveal without any decoding whether a payload contains active speech.

And all decoder modes have low CPU demands, so decoding all streams to determine which ones are most active is relatively cheap.

koen.

----- Original Message -----
From: "Christian Hoene" <[email-masked]>
To: "Jean-Marc Valin" <[email-masked]>, "Roman Shpount" <[email-masked]>
Cc: [email-masked]
Sent: Friday, January 14, 2011 12:27:02 AM
Subject: Re: [codec] Next Steps for WG

Christian

mixing
because
you can

---

From: Roman Shpount <[email-masked]>
Date: Sat, 15 Jan 2011 00:27:07 -0500

My question was not as much about the VAD, as about multiple stream
combining. What I want is to implement is switching between multiple talker
based on VAD (computation of which is done by some external methods). In
this case we need to indicate to the remote party that this is a new stream
and ideally get the remote decoder in such a state that it can immediately
start correctly decoding the new audio without a glitch. So, the minimum
requirement would be to have a flag or a packet that indicates the decoder
that it needs to reset its state. Ideally, we need an algorithm to get the
proper decoder state based on some audio history stored on the conferencing
server, and to send this decoder state to the remote party so that it can be
synchronized. I don't think this should affect the codec performance. This
is just something that needs to be accommodated in the bitstream by adding a
packet type which either resets the decoder state or sets it to some
specified value.

I would also prefer if this was working across all the codec modes and did
not require the conferencing server from restricting what kinds of encoded
audio it can accept.
_____________
Roman Shpount

On Fri, Jan 14, 2011 at 4:41 PM, Koen Vos <[email-masked]> wrote:

> Even without DTX, the Voice modes will expose a Voice Activity flag in the
> decoder API. This will reveal without any decoding whether a payload
> contains active speech.
>
> And all decoder modes have low CPU demands, so decoding all streams to
> determine which ones are most active is relatively cheap.
>
> koen.
>
>

---

From: "Kevin P. Fleming" <[email-masked]>
Date: Sat, 15 Jan 2011 07:12:22 -0600

On 01/14/2011 11:27 PM, Roman Shpount wrote:
> combining. What I want is to implement is switching between multiple
> talker based on VAD (computation of which is done by some external
> methods). In this case we need to indicate to the remote party that this
> is a new stream and ideally get the remote decoder in such a state that
> it can immediately start correctly decoding the new audio without a
> glitch. So, the minimum requirement would be to have a flag or a packet
> that indicates the decoder that it needs to reset its state. Ideally, we
> need an algorithm to get the proper decoder state based on some audio
> history stored on the conferencing server, and to send this decoder
> state to the remote party so that it can be synchronized. I don't think
> this should affect the codec performance. This is just something that
> needs to be accommodated in the bitstream by adding a packet type which
> either resets the decoder state or sets it to some specified value.

This sort of mechanism already exists when using RTP as the transport 
mechanism, by setting the marker bit and changing the SSRC to indicate 
that the payload in the packet is from a different source than the 
previous packet. In my opinion there's no need for the codec bitstream 
to have any provisions for such an indication.

-- 
Kevin P. Fleming
Digium, Inc. | Director of Software Technologies
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
skype: kpfleming | jabber: [email-masked]
Check us out at www.digium.com & www.asterisk.org

---

From: Roman Shpount <[email-masked]>
Date: Sat, 15 Jan 2011 17:07:28 -0500

First of all, CODEC definition should be independent from the transport
protocol, and we might need this functionality when RTP SSRC are not
available.

Furthermore, in case of RTP, there are two problems with your suggestion:

1. Quite a few clients do not allow remote party to change SSRC without the
re-Invite. SIP clients note SSRC of the first received RTP packet and ignore
RTP packets with different SSRC

2. Even when the client allows switching of SSRC on the fly, a few initial
RTP packets are typically discarded due to RTP probation. After this,
clients typically need to pre-fill the jitter buffer, and only after this
start audio playback. This produces from 40ms to 100ms gap in audio. This is
very audible and highly undesirable.

Finally, what I was looking for was a bit more then just decoder reset.
Ideally I wanted to set decoder state to a known value to start decoding
audio packets that I will send to it.

P.S. As a side note, Intel Performance Primitives CODECs implement a packet
type in its wave file format that resets the decoder. This packet is used to
simplify implementation of performance and regression tests. I think they
are using standard sized packet with all bits set to zero for this purpose.
We can at least do something similar.

_____________
Roman Shpount

On Sat, Jan 15, 2011 at 8:12 AM, Kevin P. Fleming <[email-masked]>wrote:

> mechanism, by setting the marker bit and changing the SSRC to indicate that
> the payload in the packet is from a different source than the previous
> packet. In my opinion there's no need for the codec bitstream to have any
> provisions for such an indication.
>
> --
> Kevin P. Fleming

---

From: Colin Perkins <[email-masked]>
Date: Sat, 15 Jan 2011 22:36:01 +0000

On 15 Jan 2011, at 22:07, Roman Shpount wrote:
> First of all, CODEC definition should be independent from the transport protocol, and we might need this functionality when RTP SSRC are not available.
> 
> 1. Quite a few clients do not allow remote party to change SSRC without the re-Invite. SIP clients note SSRC of the first received RTP packet and ignore RTP packets with different SSRC

If they do, then they're broken. RFC 3550 is very clear that the RTP SSRC can change at any time, due to SSRC collisions, changes in the transport address of the source, or source restarts.

> 2. Even when the client allows switching of SSRC on the fly, a few initial RTP packets are typically discarded due to RTP probation. After this, clients typically need to pre-fill the jitter buffer, and only after this start audio playback. This produces from 40ms to 100ms gap in audio. This is very audible and highly undesirable.

The two initial packets needed to establish an RTP stream as valid don't have to be discarded. They can be used to pre-fill the jitter buffer, and decoded once the stream is established as being valid. Only if the jitter buffer is smaller than two packets (the MIN_SEQUENTIAL parameter used to establish a source as valid) should a gap in the playout occur due to an SSRC change, and even then that gap should be possible to conceal. 

Colin

> Finally, what I was looking for was a bit more then just decoder reset. Ideally I wanted to set decoder state to a known value to start decoding audio packets that I will send to it.
> 
> P.S. As a side note, Intel Performance Primitives CODECs implement a packet type in its wave file format that resets the decoder. This packet is used to simplify implementation of performance and regression tests. I think they are using standard sized packet with all bits set to zero for this purpose. We can at least do something similar.
> 
> _____________
> Roman Shpount
> 
> 
> On Sat, Jan 15, 2011 at 8:12 AM, Kevin P. Fleming <[email-masked]> wrote:
> This sort of mechanism already exists when using RTP as the transport mechanism, by setting the marker bit and changing the SSRC to indicate that the payload in the packet is from a different source than the previous packet. In my opinion there's no need for the codec bitstream to have any provisions for such an indication.
> 
> -- 
> Kevin P. Fleming

-- 
Colin Perkins
http://csperkins.org/

---

From: Koen Vos <[email-masked]>
Date: Tue, 18 Jan 2011 23:35:20 +0100 (CET)

Roman: 

Jean-Marc and I discussed this a bit, and this is how we see it: 

Resetting the decoder state would lead to a discontinuity in the output signal and may create an objectionable "click" sound. Moreover, the incoming stream that's enabled in the switching node at the moment of the decoder reset may depend on the last frame before the switch. So the first few frames would not be decoded correctly. 

Coding and transmitting the decoder state doesn't overcome the discontinuity problem. And it would add a lot of complications and code to Opus. 

A better method seems to be: 
1. Determine if the first frame after switching has significant dependencies on the previous frame (i.e., long-term prediction in the case of SILK, or energy prediction in the case of CELT) 
2a: If yes: decode the first frame and re-encode it independently. After that, switch to the new stream. 
2b. If no: simply switch to the new stream immediately. 
This ensures a smooth transition and fast convergence to the correct output signal for the new stream. 

best, 
koen. 

----- Original Message -----
From: "Roman Shpount" <[email-masked]> 
To: "Kevin P. Fleming" <[email-masked]> 
Cc: [email-masked] 
Sent: Saturday, January 15, 2011 2:07:28 PM 
Subject: Re: [codec] Next Steps for WG 

_____________ 
Roman Shpount 

On Sat, Jan 15, 2011 at 8:12 AM, Kevin P. Fleming < [email-masked] > wrote:

---

From: Roman Shpount <[email-masked]>
Date: Tue, 18 Jan 2011 17:55:09 -0500

Thank you, this is very helpful.

What I was considering was something similar: Typically you have one speaker
in the conference with short periods of double talk. I was thinking that
decoding, mixing and encoding using independent frames the double talk
period with the few initial frames of the new speaker, and then switching to
a new speaker stream would sound the most natural and will only require
re-encoding for short periods of time.

Thank you for your great work,
_____________________________
Roman Shpount - www.telurix.com

On Tue, Jan 18, 2011 at 5:35 PM, Koen Vos <[email-masked]> wrote:

> Resetting the decoder state would lead to a discontinuity in the output
> signal and may create an objectionable "click" sound.  Moreover, the
> incoming stream that's enabled in the switching node at the moment of the
> decoder reset may depend on the last frame before the switch. So the first
> few frames would not be decoded correctly.
>
> Coding and transmitting the decoder state doesn't overcome the
> discontinuity problem.  And it would add a lot of complications and code to
> Opus.
>
> 1. Determine if the first frame after switching has significant
> dependencies on the previous frame (i.e., long-term prediction in the case
> of SILK, or energy prediction in the case of CELT)
> ------------------------------
> *From: *"Roman Shpount" <[email-masked]>
> *To: *"Kevin P. Fleming" <[email-masked]>
> *Cc: [email-masked]
> *Sent: *Saturday, January 15, 2011 2:07:28 PM
>
> *Subject: *Re: [codec] Next Steps for WG
>

---

From: Koen Vos <[email-masked]>
Date: Wed, 19 Jan 2011 00:13:50 +0100 (CET)

That should work. 
Make sure to get the time alignment right by adjusting for look-ahead etc. 
And only the first frame of the double talk period has to be coded independently; for the others you could let the encoder decide. In general, whenever you're adding or removing a stream from the mix you want an independent frame. 

best, 
koen. 

----- Original Message -----
From: "Roman Shpount" <[email-masked]> 
To: "Koen Vos" <[email-masked]> 
Cc: [email-masked] 
Sent: Tuesday, January 18, 2011 2:55:09 PM 
Subject: Re: [codec] Next Steps for WG 

What I was considering was something similar: Typically you have one speaker in the conference with short periods of double talk. I was thinking that decoding, mixing and encoding using independent frames the double talk period with the few initial frames of the new speaker, and then switching to a new speaker stream would sound the most natural and will only require re-encoding for short periods of time. 

On Tue, Jan 18, 2011 at 5:35 PM, Koen Vos < [email-masked] > wrote: 

Roman: 

best, 
koen. 

From: "Roman Shpount" < [email-masked] > 
To: "Kevin P. Fleming" < [email-masked] > 
Cc: [email-masked] 
Sent: Saturday, January 15, 2011 2:07:28 PM 

_____________ 
Roman Shpount