[asterisk-dev] OPUS horrible quality with packet loss

[Yury Tsaregorodtsev]

Examples of OPUS with 30% of LOSS with FEC on http://opus-codec.org/examples/ <http://opus-codec.org/examples/>
Sounds too perfect. Why I can't achieve even similar quality on asterisk with built codec by digium?
To simulate 30% packet loss I use:
# tc qdisc add dev eth0 root netem delay 100ms loss 30%
But even with 15-20% of loss it's almost impossible to talk.
Tried different bitrates, result is always same.
If I do PESQ predicted MOS of degraded audio (recorded with packet loss) in compare to original: 1.025
Horrible results, horrible sound.

Anyone get better experience using opus with huge packet losses ?

Yury

[Alexander Traud]

With Opus Codec, speech frames interdepend. When one RTP packet is lost, several frames are corrupted. Furthermore, when the RTP packet is not lost but just late, Asterisk simply forwards it down to the transcoding modules. Consequently, a late packet corrupts even more frames.

The Opus Codec is able to cope with all this. It is called native packet-loss concealment (Native-PLC). However, Asterisk has to tell the Opus Codec library that there was loss. In RTP, its sequence number can be used for this. More details in <http://issues.asterisk.org/jira/browse/ASTERISK-25629>.

At <http://github.com/traud/asterisk-opus>, you find a module which uses the PLC of Opus Codec. To enable Asterisk to signal loss/late, you have to apply the <enable_native_plc.patch> there. Please, give it a try and report whether it improves the situation for you.

When that approach is better, someone (you?) has to drive Native-PLC in Asterisk. Or you sponsor a bug bounty so someone drives Native-PLC into Asterisk on your behalf.

[Yury Tsaregorodtsev]

Hi Alexander,
I did try your opus version including native PLC patch (on asterisk 13.14.0, everything compiled without issues but anyway maybe version is too high ?)
I wouldn't see much improvements on quality..... From my personal point of view it's sounds little bit better, from point of view PESQ - predicted MOS its same as without native patch, around 1.025.
If you have time I can setup for you 2x ESXI instances with prepared environment of 2x asterisk with simple configuration and ITU-T PESQ utility, so you can hear everything yourself.

P.S. stupid question, but anyway, will it be a solution to use RTP on TCP if we have high loses/delay spikes ?

Post by Alexander Traud
With Opus Codec, speech frames interdepend. When one RTP packet is lost, several frames are corrupted. Furthermore, when the RTP packet is not lost but just late, Asterisk simply forwards it down to the transcoding modules. Consequently, a late packet corrupts even more frames.
The Opus Codec is able to cope with all this. It is called native packet-loss concealment (Native-PLC). However, Asterisk has to tell the Opus Codec library that there was loss. In RTP, its sequence number can be used for this. More details in <http://issues.asterisk.org/jira/browse/ASTERISK-25629>.
At <http://github.com/traud/asterisk-opus>, you find a module which uses the PLC of Opus Codec. To enable Asterisk to signal loss/late, you have to apply the <enable_native_plc.patch> there. Please, give it a try and report whether it improves the situation for you.
When that approach is better, someone (you?) has to drive Native-PLC in Asterisk. Or you sponsor a bug bounty so someone drives Native-PLC into Asterisk on your behalf.
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev

[Alexander Traud]

maybe version is too high [for that Native-PLC patch]

You can answer that question yourself by adding debug output into codecs/codec_opus_open_source.c. For example, put one ast_log(…) for each "/* Case x". Case 5 and 6 are the perfect case without loss and therefore without activating Native-PLC. When that patch works, you should see (all) other cases.

That is the benefit of the open-source variant. The black box turns into a white box, where you can add debugging wherever you like.

not see much improvements on quality

Then, it is time to go over to the Opus mailing list <http://opus-codec.org/contact/> and ask them for help. For example, how to debug the Opus library states in more detail. Whether everything is right when it comes to FEC. And so on.

I was not able to find much documentation about this area. Therefore and because I do not have quality-debugger tools here, I cannot even go that path. The open-source community would more than happy, if you could go that path. Perhaps you are even able to create a best-practice paper, how to use/access the API of Opus Codec in case of a VoIP/SIP/RTP application. I have not found something like that, yet.

Alternatively, have a look at FreeSWITCH and whether it works there.

will it be a solution to use RTP on TCP if we have high loses/delay spikes ?

No. The idea of UDP is to drop fast and early. If the link is instable, you face much more issues with TCP, like its windows and maintaining its connection at all. Here, the question is, why a Opus Codec in Asterisk is affected by that loses so much.

[Kevin Harwell]

Post by Yury Tsaregorodtsev
Examples of OPUS with 30% of LOSS with FEC on http://opus-codec.org/
examples/
Sounds too perfect. Why I can't achieve even similar quality on asterisk
with built codec by digium?
# tc qdisc add dev eth0 root netem delay 100ms loss 30%
But even with 15-20% of loss it's almost impossible to talk.
Tried different bitrates, result is always same.
If I do PESQ predicted MOS of degraded audio (recorded with packet loss)
in compare to original: 1.025
Horrible results, horrible sound.
Anyone get better experience using opus with huge packet losses ?

Have you enabled FEC and made sure both sides negotiated it? If not you can
enable it within "codecs.conf" by setting "fec=yes".

When a packet has been lost and the decoder receives a frame with FEC data
(and fec is enabled) it will attempt to rebuild the lost packet (current
packet minus one) from the given FEC information.


Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org

[Yury Tsaregorodtsev]

after fixing attr->fec in open source edition of OPUS and applying Alexanders patch: ASTERISK-25629 (Native Packet-Loss Concealment)
I have following results:

I setup 2 hosts with asterisk 13.14.0 and made dozen calls in OPUS between them.
On 2nd host I've simulated packet delay and loss:
tc qdisc add dev eth0 root netem loss 20% delay 100ms 20ms distribution normal

On 1st host (call origination side) all media been recorded using App_Monitor.
On 2nd host it was simple Playback extension with demo-instruct.sln16

Config for digium codec_opus on both was:
[opus]
type=opus
packet_loss=40
max_playback_rate=48000
bitrate=24000
cbr=0
fec=1

Config for open source edition of OPUS was same, but replaced values in include/asterisk/opus.h
(and as well I've added to codecs/codec_opus_open_source.c status = opus_encoder_ctl(opvt->opus, OPUS_SET_PACKET_LOSS_PERC(packet_loss));)

All recorded degraded media files were compared to original (recorded with AppMonitor as well, but without any delays, losses) accordingly to ITU-T P.862 (Perceptual evaluation of speech quality (PESQ)):

Reading degraded file opus_opensource-maxbitrate_24000-cbr_0-fec_1-ID1.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.840 2.589 8000 nb
Reading degraded file opus_opensource-maxbitrate_24000-cbr_0-fec_1-ID2.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.507 2.143 8000 nb
Reading degraded file opus_opensource-maxbitrate_24000-cbr_0-fec_1-ID3.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.379 1.994 8000 nb
Reading degraded file opus_opensource-maxbitrate_24000-cbr_0-fec_1-ID4.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.472 2.102 8000 nb
Reading degraded file opus_opensource-maxbitrate_24000-cbr_0-fec_1-ID5.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.413 2.032 8000 nb

Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID1.wav P.862 Prediction (Raw MOS, MOS-LQO): = 1.332 1.258 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID2.wav P.862 Prediction (Raw MOS, MOS-LQO): = 0.738 1.110 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID3.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.426 2.047 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID4.wav P.862 Prediction (Raw MOS, MOS-LQO): = 0.920 1.143 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID5.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.680 2.366 8000 nb

MOS on calls using open source opus higher almost twice.
Subjective opinion regarding audio quality: using open source codec quality almost same as in example on http://opus-codec.org/examples/ <http://opus-codec.org/examples/> with 30% loss and FEC, acceptable for ears, but
using digium opus quality is not acceptable, a lot of spikes, interruptions.

I also double checked the fact before applying ASTERISK-25629 patch asterisk don't drop lately arrived RTP.
I've recorded UDP dump and replayed it using "bittwist" utility, without patch all RTP packets been played again, with patch all of them been ignored.


Thx, Alexander :)

Post by Yury Tsaregorodtsev
Examples of OPUS with 30% of LOSS with FEC on http://opus-codec.org/examples/ <http://opus-codec.org/examples/>
Sounds too perfect. Why I can't achieve even similar quality on asterisk with built codec by digium?
# tc qdisc add dev eth0 root netem delay 100ms loss 30%
But even with 15-20% of loss it's almost impossible to talk.
Tried different bitrates, result is always same.
If I do PESQ predicted MOS of degraded audio (recorded with packet loss) in compare to original: 1.025
Horrible results, horrible sound.
Anyone get better experience using opus with huge packet losses ?
Have you enabled FEC and made sure both sides negotiated it? If not you can enable it within "codecs.conf" by setting "fec=yes".
When a packet has been lost and the decoder receives a frame with FEC data (and fec is enabled) it will attempt to rebuild the lost packet (current packet minus one) from the given FEC information.
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com <http://digium.com/> & http://asterisk.org <http://asterisk.org/>--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com <http://www.api-digital.com/> --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev <http://lists.digium.com/mailman/listinfo/asterisk-dev>

[Kevin Harwell]

<snip>

MOS on calls using open source opus higher almost twice.

Subjective opinion regarding audio quality: using open source codec
quality almost same as in example on http://opus-codec.org/examples/ with
30% loss and FEC, acceptable for ears, but
using digium opus quality is not acceptable, a lot of spikes,
interruptions.
I also double checked the fact before applying ASTERISK-25629 patch
asterisk don't drop lately arrived RTP.

Dropped packets and late arriving packets are two separate issue and are
handled as such in Asterisk. The Digium Opus codec can handle dropped
packets by enabling FEC. If the problem is late arriving packets than
applying a jitter buffer to the audio stream exhibiting the problem should
help alleviate that.

[Yury Tsaregorodtsev]

Even forced enabled jitter doesn't make asterisk to ignore late arrived packets.
During my tests jb was always enabled (forced).
I made also tests without delay, only with drops - quality of Digium Opus not acceptable for voice conversation anyway.
The fact is open source opus handles better dropped packets.
You can't disagree with quality after all,
I can send you recorded samples, you can compare.

Yury

<snip>
MOS on calls using open source opus higher almost twice.
Subjective opinion regarding audio quality: using open source codec quality almost same as in example on http://opus-codec.org/examples/ <http://opus-codec.org/examples/> with 30% loss and FEC, acceptable for ears, but
using digium opus quality is not acceptable, a lot of spikes, interruptions.
I also double checked the fact before applying ASTERISK-25629 patch asterisk don't drop lately arrived RTP.
Dropped packets and late arriving packets are two separate issue and are handled as such in Asterisk. The Digium Opus codec can handle dropped packets by enabling FEC. If the problem is late arriving packets than applying a jitter buffer to the audio stream exhibiting the problem should help alleviate that.
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com <http://www.api-digital.com/> --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev <http://lists.digium.com/mailman/listinfo/asterisk-dev>

[Matt Fredrickson]

Hey Yury,

Thanks for letting us know about the challenges you're having with the
Digium sanctioned version of codec_opus. We're going to try to lab up
the scenario you have found and see if there is a bug in our
encoder/decoder processing that's making it perform worse under your
scenarios.

Sorry about the trouble - as you know, with software, it tends to
improve as it goes forward but sometimes it takes time. Hopefully we
can figure out what's different in the Digium codec quickly and
release a new version of it with the fix.

Best wishes,
Matthew Fredrickson

On Mon, Apr 3, 2017 at 3:38 PM, Yury Tsaregorodtsev

Post by Yury Tsaregorodtsev
Even forced enabled jitter doesn't make asterisk to ignore late arrived packets.
During my tests jb was always enabled (forced).
I made also tests without delay, only with drops - quality of Digium Opus
not acceptable for voice conversation anyway.
The fact is open source opus handles better dropped packets.
You can't disagree with quality after all,
I can send you recorded samples, you can compare.
Yury

<snip>
MOS on calls using open source opus higher almost twice.
Subjective opinion regarding audio quality: using open source codec
quality almost same as in example on http://opus-codec.org/examples/ with
30% loss and FEC, acceptable for ears, but
using digium opus quality is not acceptable, a lot of spikes,
interruptions.
I also double checked the fact before applying ASTERISK-25629 patch
asterisk don't drop lately arrived RTP.

Dropped packets and late arriving packets are two separate issue and are
handled as such in Asterisk. The Digium Opus codec can handle dropped
packets by enabling FEC. If the problem is late arriving packets than
applying a jitter buffer to the audio stream exhibiting the problem should
help alleviate that.
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev

--
Matthew Fredrickson
Digium, Inc. | Engineering Manager
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA

[Kevin Harwell]

Post by Yury Tsaregorodtsev
Even forced enabled jitter doesn't make asterisk to ignore late arrived packets.
During my tests jb was always enabled (forced).

Hrm I'd think the jitter buffer should ignore or drop the late packets.
That is packets arriving with timestamps marked as before packets that have
already left the jitter buffer and have been processed by the audio
core/translated.

Just to double check what channel (inbound or outbound) are you attaching
the jitter buffer too and is it the one that would receive the late
packets? For instance if it is needed on the outbound channel then you'd
need to use a pre-dial handler:

exten => outbound_jitterbuf,1,NoOp()
same => n,Set(JITTERBUFFER(adaptive)=default)
same => n,Return()

exten => _1XX,1,Noop()
same => n,Dial(${TECH}/${EXTEN},20,b(default^outbound_jitterbuf^1))
same => n,Hangup()

Thanks,

Kevin

[Kevin Harwell]

Post by Yury Tsaregorodtsev
after fixing attr->fec in open source edition of OPUS and applying
Alexanders patch: ASTERISK-25629 (Native Packet-Loss Concealment)
I setup 2 hosts with asterisk 13.14.0 and made dozen calls in OPUS between them.
tc qdisc add dev eth0 root netem loss 20% delay 100ms 20ms distribution normal
On 1st host (call origination side) all media been recorded using App_Monitor.
On 2nd host it was simple Playback extension with demo-instruct.sln16
[opus]
type=opus
packet_loss=40
max_playback_rate=48000
bitrate=24000
cbr=0
fec=1
Config for open source edition of OPUS was same, but replaced values in
include/asterisk/opus.h
(and as well I've added to codecs/codec_opus_open_source.c status =
opus_encoder_ctl(opvt->opus, OPUS_SET_PACKET_LOSS_PERC(packet_loss));)
All recorded degraded media files were compared to original (recorded with
AppMonitor as well, but without any delays, losses) accordingly to ITU-T P.862
Reading degraded file opus_opensource-maxbitrate_
24000-cbr_0-fec_1-ID1.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.840
2.589 8000 nb
Reading degraded file opus_opensource-maxbitrate_
24000-cbr_0-fec_1-ID2.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.507
2.143 8000 nb
Reading degraded file opus_opensource-maxbitrate_
24000-cbr_0-fec_1-ID3.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.379
1.994 8000 nb
Reading degraded file opus_opensource-maxbitrate_
24000-cbr_0-fec_1-ID4.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.472
2.102 8000 nb
Reading degraded file opus_opensource-maxbitrate_
24000-cbr_0-fec_1-ID5.wav P.862 Prediction (Raw MOS, MOS-LQO): = 2.413
2.032 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID1.wav P.862
Prediction (Raw MOS, MOS-LQO): = 1.332 1.258 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID2.wav P.862
Prediction (Raw MOS, MOS-LQO): = 0.738 1.110 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID3.wav P.862
Prediction (Raw MOS, MOS-LQO): = 2.426 2.047 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID4.wav P.862
Prediction (Raw MOS, MOS-LQO): = 0.920 1.143 8000 nb
Reading degraded file opus_digium-maxbitrate_24000-cbr_0-fec_1-ID5.wav P.862
Prediction (Raw MOS, MOS-LQO): = 2.680 2.366 8000 nb
MOS on calls using open source opus higher almost twice.
Subjective opinion regarding audio quality: using open source codec
quality almost same as in example on http://opus-codec.org/examples/ with
30% loss and FEC, acceptable for ears, but
using digium opus quality is not acceptable, a lot of spikes,
interruptions.
I also double checked the fact before applying ASTERISK-25629 patch
asterisk don't drop lately arrived RTP.
I've recorded UDP dump and replayed it using "bittwist" utility, without
patch all RTP packets been played again, with patch all of them been
ignored.

Hi Yury,

We for sure want to look more into this. If you would open an issue on the
Asterisk issue tracker [1] it would be much appreciated. Be sure to include
all your findings.

When possible list the exact version numbers you are using for the various
tools and libraries being used (Asterisk - output of running "asterisk -V",
PESQ, Opus library for when you used the Asterisk open source codec, etc).

Also if possible include information about your Asterisk setup. Dialplan
used (extensions.conf), endpoint configuration (pjsip.conf or sip.conf),
etc...

Also include the exact commands you used when running tests. All of this
will greatly help in duplicating results.

[1] https://issues.asterisk.org

Thanks!

Kevin