Subject: [video-codec] Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

From: Roman Danyliw via Datatracker <[email-masked]>
Date: Tue, 11 Jun 2019 18:24:45 -0700

Roman Danyliw has entered the following ballot position for
draft-ietf-netvc-requirements-09: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-netvc-requirements/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

(1) I worry that the level of detail in the in the Compression Performance
Evaluation (Section 4.1) is insufficient for implementation.  Specifically:

(a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or
VP9), a set of 10 QP quantization parameter) values should be specified (in a
separate document on Internet video codec testing) and corresponding quality
values should be calculated.”

-- How should a set of QPs be specified?

--How should the quality values be calculated?

-- What does the text “(in a separate document on Internet video codec
testing)” mean?

(b) Per “A list of video sequences that should be used for testing as well as
the 10 QP values for the reference codec are defined in a separate document ",
what document is that?  Is it draft-ietf-netvc-testing?

(2) Per the Security Considerations Section (Section 5)

-- What does “codec implementation (for both an encoder and a decoder) should
cover the worst case of computational complexity, memory bandwidth, and
physical memory size” mean?

-- Please add additional language that codec should be written in a defensive
style as they will be processing untrusted input.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

(1) There is something odd about the document formatting – the title and the
first author last name in the footer is wrapped in “< … >”

(2) It would be helpful to forward reference that acronyms are explained in
Appendix A.

(3) This draft uses the words should and must to prescribe action.  Why wasn’t
RFC2119 cited to explain these words?

(4) Section 2.0.  A reference to explain “YCbCr 4:2:0” would be helpful

(5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I
initiate read that as a qualitative measure.  However, the text says “up to 10x
and more” so that implies some quantitative measure.  What is that?

(6) Section 2.1.  Expand QP values on first use

(7) Section 2.x.  The language around content doesn’t appear to be consist. 
For example:

-- Section 2.1, Internet Video Streaming says “movies, TV-series and shows, and
animation.”

-- Section 2.2, IPTV says “television content”

-- Section 2.5, Screen casting says “business presentations …, animation
(cartoons), gaming content, data visualization, …, virtual desktop
infrastructure (VDI), screen/desktop sharing and collaboration, supervisory
control and data acquisition (SCADA) display, automotive/navigation display,
cloud gaming, factory automation display, wireless display, display wall,
digital operating room (DiOR), etc.

What the difference between Section 2.1’s animation and Section 2.5’s cartoons?

What’s the different between Section 2.1’s “movies, TV series …” and Section
2.2’s “television content”?

(8) Section 2.5.  The sentence “Currently, …” is very challenging to parse as
it includes inline “i.e.,” and “etc”.

(9) Section 2.5.  Per “powerpoint, word documents”, these are specific
Microsoft products.  I recommend using more generic names.

(10) Section 4.  I found it confusing that an evaluation methodology was in a
requirements document.  I would have expected it in the draft-ietf-netvc-testing

(11) Section 4.1.  VP9 needs a reference.

(12) Section 6.  I don’t think this entire section is necessary.

(13) Editorial Nits:
-- Section 3.  Style nit.  s/chapter/section/

-- Section 4.1. Typo.  s/computged/computed

---

From: Adam Roach <[email-masked]>
Date: Wed, 12 Jun 2019 21:55:57 -0500

> physical memory size” mean?

Would this be clearer if it were rephrased as "...should take into 
consideration the worst-case..." ?

/a

---

From: Filippov Alexey <[email-masked]>
Date: Thu, 13 Jun 2019 17:14:56 +0000

Dear Roman,

Thank you a lot for your comments and raising the important questions.

> How should a set of QPs be specified?
As you probably know, a set of available QP values can be codec specific. Usually, a QP set is selected to cover medium bit-rate range that is considered to be the most complex for compressing. We proposed to set up the assessment process not only for this range but also for low and high QP ranges to cover a wider range of applications. According to my understanding, concrete QP values should be specified for candidate codecs in draft-ietf-netvc-testing.

> How should the quality values be calculated?
It is explained in Section 4.1, namely: “To assess the quality of output (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are separately computed. In the case of the YCbCr color format, PSNR should be calculated for each color plane whereas MS-SSIM is calculated for luma channel only. In the case of the RGB color format, both metrics are computed for R, G and B channels. Thus, for each sequence, 30 RD-points for PSNR (i.e. three RD-curves, one for each channel) and 10 RD-points for MS-SSIM (i.e. one RD-curve, for luma channel only) should be calculated in the case of YCbCr. If content is encoded as RGB, 60 RD-points (30 for PSNR and 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for each channel) are computed for PSNR as well as three RD-curves (one for each channel) for MS-SSIM.” In references [3, 11], these 2 quality assessment metrics and the ways of how to calculate them are described in detail.

>-- What does the text “(in a separate document on Internet video codec testing)” mean?
> Per “A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document ", what document is that?  Is it draft-ietf-netvc-testing?
According to my understanding, it is draft-ietf-netvc-testing.

>What does “codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size” mean?
I’d like to thank Adam for his comment. I also think that his formulation ("...should take into consideration the worst-case...") is clearer. 
A more detailed explanation on this Section can be found in my response to the secdir reviewer: https://mailarchive.ietf.org/arch/msg/secdir/QYx-hl07aec5XhgqkldqgWFG3Is 

>Please add additional language that codec should be written in a defensive style as they will be processing untrusted input.
Good point. Thank you. I’ll add it.

--
Best regards,
Alexey Filippov

-----Original Message-----
From: Roman Danyliw via Datatracker [mailto:[email-masked]] 
Sent: Wednesday, June 12, 2019 4:25 AM
To: The IESG <[email-masked]>
Cc: [email-masked]; Mo Zanaty <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09: (with DISCUSS and COMMENT)

When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.)

(1) I worry that the level of detail in the in the Compression Performance Evaluation (Section 4.1) is insufficient for implementation.  Specifically:

(a) Per “Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP quantization parameter) values should be specified (in a separate document on Internet video codec testing) and corresponding quality values should be calculated.”

-- What does the text “(in a separate document on Internet video codec testing)” mean?

(b) Per “A list of video sequences that should be used for testing as well as the 10 QP values for the reference codec are defined in a separate document ", what document is that?  Is it draft-ietf-netvc-testing?

-- What does “codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size” mean?

-- Please add additional language that codec should be written in a defensive style as they will be processing untrusted input.

(1) There is something odd about the document formatting – the title and the first author last name in the footer is wrapped in “< … >”

(2) It would be helpful to forward reference that acronyms are explained in Appendix A.

(5) Section 2.1. Per “high encoder complexity” and “decoding complexity”, I initiate read that as a qualitative measure.  However, the text says “up to 10x and more” so that implies some quantitative measure.  What is that?

-- Section 2.1, Internet Video Streaming says “movies, TV-series and shows, and animation.”

-- Section 2.5, Screen casting says “business presentations …, animation (cartoons), gaming content, data visualization, …, virtual desktop infrastructure (VDI), screen/desktop sharing and collaboration, supervisory control and data acquisition (SCADA) display, automotive/navigation display, cloud gaming, factory automation display, wireless display, display wall, digital operating room (DiOR), etc.

What’s the different between Section 2.1’s “movies, TV series …” and Section 2.2’s “television content”?

(8) Section 2.5.  The sentence “Currently, …” is very challenging to parse as it includes inline “i.e.,” and “etc”.

(9) Section 2.5.  Per “powerpoint, word documents”, these are specific Microsoft products.  I recommend using more generic names.

(10) Section 4.  I found it confusing that an evaluation methodology was in a requirements document.  I would have expected it in the draft-ietf-netvc-testing

(13) Editorial Nits:

---

From: Roman Danyliw <[email-masked]>
Date: Fri, 21 Jun 2019 14:08:38 +0000

Hi!

> From: iesg [mailto:[email-masked]] On Behalf Of Filippov Alexey
> Sent: Thursday, June 13, 2019 1:15 PM
> To: Roman Danyliw <[email-masked]>; Adam Roach <[email-masked]>;
> The IESG <[email-masked]>
> Cc: [email-masked]; [email-masked]; draft-ietf-netvc-
> [email-masked]; Mo Zanaty <[email-masked]>
> Subject: RE: Roman Danyliw's Discuss on draft-ietf-netvc-requirements-09:
> (with DISCUSS and COMMENT)
> 
> Dear Roman,
> 
> As you probably know, a set of available QP values can be codec specific.
> Usually, a QP set is selected to cover medium bit-rate range that is
> considered to be the most complex for compressing. We proposed to set up
> the assessment process not only for this range but also for low and high QP
> ranges to cover a wider range of applications. According to my
> understanding, concrete QP values should be specified for candidate codecs
> in draft-ietf-netvc-testing

See below.  Can you help with a specific pointer into draft-ietf-netvc-testing which lists candidate codecs+QPs.

> It is explained in Section 4.1, namely: “To assess the quality of output
> (decoded) sequences, two indexes, PSNR [3] and MS-SSIM [3,11] are
> separately computed. In the case of the YCbCr color format, PSNR should be
> calculated for each color plane whereas MS-SSIM is calculated for luma
> channel only. In the case of the RGB color format, both metrics are
> computed for R, G and B channels. Thus, for each sequence, 30 RD-points for
> PSNR (i.e. three RD-curves, one for each channel) and 10 RD-points for MS-
> SSIM (i.e. one RD-curve, for luma channel only) should be calculated in the
> case of YCbCr. If content is encoded as RGB, 60 RD-points (30 for PSNR and
> 30 for MS-SSIM) should be calculated, i.e. three RD-curves (one for each
> channel) are computed for PSNR as well as three RD-curves (one for each
> channel) for MS-SSIM.” In references [3, 11], these 2 quality assessment
> metrics and the ways of how to calculate them are described in detail.

Got it.  Thanks.

> > Per “A list of video sequences that should be used for testing as well as the
> 10 QP values for the reference codec are defined in a separate document ",

Can you please help me with the specific section references in draft-ietf-netvc-testing as I'm not seeing it -- where is the explicit reference codec being named and its associate 10 QP values?  For what it's worth I see Section 4.1 of this draft saying that:

   As
   the reference for evaluation, state-of-the-art video codecs such as
   HEVC/H.265 [4,5] or VP9  must be used. The reference source code of
   the HEVC/H.265 codec can be found at [6]. The HEVC/H.265 codec must
   be configured according to [13] and Table 9.

I'm looking for something that either unambiguously points to the reference codec+QPs; or clearer language that says that all of this is out of scope in this and/or the -testing draft, but is clear on what input into the processes of this draft is required.

> >What does “codec implementation (for both an encoder and a decoder)
> should cover the worst case of computational complexity, memory
> bandwidth, and physical memory size” mean?
> I’d like to thank Adam for his comment. I also think that his formulation
> ("...should take into consideration the worst-case...") is clearer.
> A more detailed explanation on this Section can be found in my response to
> the secdir reviewer: https://mailarchive.ietf.org/arch/msg/secdir/QYx-
> hl07aec5XhgqkldqgWFG3Is

I'd like to offer a further refinement of what Adam proposed:

Original:
However, it is worth noting that a codec implementation (for both an encoder and a decoder) should cover the worst case of computational complexity, memory bandwidth, and physical memory size (e.g., for  decoded pictures used as references).

Adam:
However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size (e.g., for decoded pictures used as references).

Roman+Adam:
However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size needed to processes the input (e.g., the decoded pictures used as references).

> >Please add additional language that codec should be written in a defensive

Regards,
Roman

> Cc: [email-masked]; Mo Zanaty
> <[email-masked]>; [email-masked]; [email-masked]; video-
> [email-masked]
> When responding, please keep the subject line intact and reply to all email
> addresses included in the To and CC lines. (Feel free to cut this introductory
> paragraph, however.)
> 
> 
> separate document on Internet video codec testing) and corresponding
> quality values should be calculated.”
> 
> the 10 QP values for the reference codec are defined in a separate document
> ", what document is that?  Is it draft-ietf-netvc-testing?
> 
> -- What does “codec implementation (for both an encoder and a decoder)
> should cover the worst case of computational complexity, memory
> bandwidth, and physical memory size” mean?
> 
> -- Please add additional language that codec should be written in a
> defensive style as they will be processing untrusted input.
> 
> 
> (1) There is something odd about the document formatting – the title and
> the first author last name in the footer is wrapped in “< … >”
> 
> initiate read that as a qualitative measure.  However, the text says “up to
> 10x and more” so that implies some quantitative measure.  What is that?
> 
> control and data acquisition (SCADA) display, automotive/navigation
> display, cloud gaming, factory automation display, wireless display, display
> wall, digital operating room (DiOR), etc.
>