Discussion:
[Icecast-dev] Icecast relay crash.
Sourav Saha
2016-11-05 11:08:05 UTC
Permalink
Hi All,

I'm facing a problem in icecast relay machines. I'm seeing frequently that
some of the relay machines are crashing and generates core dump but the
reason for the crash is not clear from the trace of the core. Here is how
the gdb trace looks like:

(gdb) bt

#0 0xffffe435 in __kernel_vsyscall ()

#1 0xf742c607 in raise () from /lib/i386-linux-gnu/libc.so.6

#2 0xf742fa33 in abort () from /lib/i386-linux-gnu/libc.so.6

#3 0xf7466e53 in ?? () from /lib/i386-linux-gnu/libc.so.6

#4 0xf747133a in ?? () from /lib/i386-linux-gnu/libc.so.6

#5 0xf7472ee9 in ?? () from /lib/i386-linux-gnu/libc.so.6

*#6 0xf7474708 in malloc () from /lib/i386-linux-gnu/libc.so.6*

*#7 0x080b66b7 in _accept_connection () at connection.c:598*

*#8 connection_accept_loop () at connection.c:826*

*#9 0x080a9c05 in _server_proc () at main.c:384*

*#10 main (argc=3, argv=0xffbbc554) at main.c:633*


As you can see at frame #6 malloc() is failing but I don't know the reason
behind this. Does anybody has any idea how to fix this issue? Note: this is
happening only in the relay machines but not in edges.


-- Sourav
Philipp Schafft
2016-11-05 13:38:02 UTC
Permalink
Good morning,
Post by Sourav Saha
Hi All,
I'm facing a problem in icecast relay machines. I'm seeing frequently
that some of the relay machines are crashing and generates core dump
but the reason for the crash is not clear from the trace of the core.
(gdb) bt
#0 0xffffe435 in __kernel_vsyscall ()
#1 0xf742c607 in raise () from /lib/i386-linux-gnu/libc.so.6
#2 0xf742fa33 in abort () from /lib/i386-linux-gnu/libc.so.6
#3 0xf7466e53 in ?? () from /lib/i386-linux-gnu/libc.so.6
#4 0xf747133a in ?? () from /lib/i386-linux-gnu/libc.so.6
#5 0xf7472ee9 in ?? () from /lib/i386-linux-gnu/libc.so.6
#6 0xf7474708 in malloc () from /lib/i386-linux-gnu/libc.so.6
#7 0x080b66b7 in _accept_connection () at connection.c:598
#8 connection_accept_loop () at connection.c:826
#9 0x080a9c05 in _server_proc () at main.c:384
#10 main (argc=3, argv=0xffbbc554) at main.c:633
As you can see at frame #6 malloc() is failing but I don't know the
reason behind this. Does anybody has any idea how to fix this issue?
Note: this is happening only in the relay machines but not in edges.
Thank you for your report.
What is the _exact_ version you run?


You can also reach us on IRC (on Freenode, #icecast) for some live
debugging (when the sun is over central Europe).


with best regards,
--
Philipp Schafft (CEO/GeschÀftsfÌhrer)
Telephon: +49.3535 490 17 92

Löwenfelsen UG (haftungsbeschrÀnkt) Registration number:
Bickinger Straße 21 HRB 12308 CB
04916 Herzberg (Elster) VATIN/USt-ID:
Germany DE305133015
Sourav Saha
2016-11-05 14:03:42 UTC
Permalink
Hi,

I am using icecast 2.3.2 and I'm not in position to upgrade the version
right away since this version is already in use. Please let me know if you
need any additional details for this issue. Nevertheless if there is any
potential fix has gone towards this part of code in later versions, I can
just use that patch in my current code base.

Also I'll try to catch up on Freenode but can't be very sure due to time
difference, so I'm betting on the replies on this thread.

Thanks/

On Sat, Nov 5, 2016 at 7:08 PM, Philipp Schafft <
Post by Philipp Schafft
Good morning,
Post by Sourav Saha
Hi All,
I'm facing a problem in icecast relay machines. I'm seeing frequently
that some of the relay machines are crashing and generates core dump
but the reason for the crash is not clear from the trace of the core.
(gdb) bt
#0 0xffffe435 in __kernel_vsyscall ()
#1 0xf742c607 in raise () from /lib/i386-linux-gnu/libc.so.6
#2 0xf742fa33 in abort () from /lib/i386-linux-gnu/libc.so.6
#3 0xf7466e53 in ?? () from /lib/i386-linux-gnu/libc.so.6
#4 0xf747133a in ?? () from /lib/i386-linux-gnu/libc.so.6
#5 0xf7472ee9 in ?? () from /lib/i386-linux-gnu/libc.so.6
#6 0xf7474708 in malloc () from /lib/i386-linux-gnu/libc.so.6
#7 0x080b66b7 in _accept_connection () at connection.c:598
#8 connection_accept_loop () at connection.c:826
#9 0x080a9c05 in _server_proc () at main.c:384
#10 main (argc=3, argv=0xffbbc554) at main.c:633
As you can see at frame #6 malloc() is failing but I don't know the
reason behind this. Does anybody has any idea how to fix this issue?
Note: this is happening only in the relay machines but not in edges.
Thank you for your report.
What is the _exact_ version you run?
You can also reach us on IRC (on Freenode, #icecast) for some live
debugging (when the sun is over central Europe).
with best regards,
--
Philipp Schafft (CEO/GeschÀftsfÌhrer)
Telephon: +49.3535 490 17 92
Bickinger Straße 21 HRB 12308 CB
Germany DE305133015
Philipp Schafft
2016-11-05 17:35:11 UTC
Permalink
Good evening,
Post by Sourav Saha
Hi,
I am using icecast 2.3.2 and I'm not in position to upgrade the
version right away since this version is already in use. Please let me
know if you need any additional details for this issue. Nevertheless
if there is any potential fix has gone towards this part of code in
later versions, I can just use that patch in my current code base.
Version 2.3.2 is not just old but stone age. It has a range of known
bugs, security vulnerabilities both local and remote.

The problem you see is very likely because of some random buffer overrun
bug. Maybe one of those vulnerabilities.
Post by Sourav Saha
Also I'll try to catch up on Freenode but can't be very sure due to
time difference, so I'm betting on the replies on this thread.
with best regards,
Post by Sourav Saha
Thanks/
On Sat, Nov 5, 2016 at 7:08 PM, Philipp Schafft
[...]
Post by Sourav Saha
I'm facing a problem in icecast relay machines. I'm seeing
frequently
Post by Sourav Saha
that some of the relay machines are crashing and generates
core dump
Post by Sourav Saha
but the reason for the crash is not clear from the trace of
the core.
Post by Sourav Saha
(gdb) bt
[...]
#6 0xf7474708 in malloc ()
from /lib/i386-linux-gnu/libc.so.6
Post by Sourav Saha
#7 0x080b66b7 in _accept_connection () at connection.c:598
#8 connection_accept_loop () at connection.c:826
#9 0x080a9c05 in _server_proc () at main.c:384
#10 main (argc=3, argv=0xffbbc554) at main.c:633
As you can see at frame #6 malloc() is failing but I don't
know the
Post by Sourav Saha
reason behind this. Does anybody has any idea how to fix
this issue?
Post by Sourav Saha
Note: this is happening only in the relay machines but not
in edges.
Thank you for your report.
What is the _exact_ version you run?
You can also reach us on IRC (on Freenode, #icecast) for some live
debugging (when the sun is over central Europe).
--
Philipp Schafft (CEO/GeschÀftsfÌhrer)
Telephon: +49.3535 490 17 92

Löwenfelsen UG (haftungsbeschrÀnkt) Registration number:
Bickinger Straße 21 HRB 12308 CB
04916 Herzberg (Elster) VATIN/USt-ID:
Germany DE305133015
Sourav Saha
2016-11-07 13:42:41 UTC
Permalink
Thanks Philipp.

I agree with you, this looks like some buffer overrun issue. I know I'm
using a very old version of icecast software but still I am asking is there
any kind of workaround to this problem apart from upgrading the icecast
version (which is the last option).
Running Valgrind may help to capture the root cause, any thoughts?

BTW what's the latest stable version of icecast?


-- Sourav

On Sat, Nov 5, 2016 at 11:05 PM, Philipp Schafft <
Post by Philipp Schafft
Good evening,
Post by Sourav Saha
Hi,
I am using icecast 2.3.2 and I'm not in position to upgrade the
version right away since this version is already in use. Please let me
know if you need any additional details for this issue. Nevertheless
if there is any potential fix has gone towards this part of code in
later versions, I can just use that patch in my current code base.
Version 2.3.2 is not just old but stone age. It has a range of known
bugs, security vulnerabilities both local and remote.
The problem you see is very likely because of some random buffer overrun
bug. Maybe one of those vulnerabilities.
Post by Sourav Saha
Also I'll try to catch up on Freenode but can't be very sure due to
time difference, so I'm betting on the replies on this thread.
with best regards,
Post by Sourav Saha
Thanks/
On Sat, Nov 5, 2016 at 7:08 PM, Philipp Schafft
[...]
Post by Sourav Saha
I'm facing a problem in icecast relay machines. I'm seeing
frequently
Post by Sourav Saha
that some of the relay machines are crashing and generates
core dump
Post by Sourav Saha
but the reason for the crash is not clear from the trace of
the core.
Post by Sourav Saha
(gdb) bt
[...]
#6 0xf7474708 in malloc ()
from /lib/i386-linux-gnu/libc.so.6
Post by Sourav Saha
#7 0x080b66b7 in _accept_connection () at connection.c:598
#8 connection_accept_loop () at connection.c:826
#9 0x080a9c05 in _server_proc () at main.c:384
#10 main (argc=3, argv=0xffbbc554) at main.c:633
As you can see at frame #6 malloc() is failing but I don't
know the
Post by Sourav Saha
reason behind this. Does anybody has any idea how to fix
this issue?
Post by Sourav Saha
Note: this is happening only in the relay machines but not
in edges.
Thank you for your report.
What is the _exact_ version you run?
You can also reach us on IRC (on Freenode, #icecast) for some live
debugging (when the sun is over central Europe).
--
Philipp Schafft (CEO/GeschÀftsfÌhrer)
Telephon: +49.3535 490 17 92
Bickinger Straße 21 HRB 12308 CB
Germany DE305133015
Philipp Schafft
2016-11-09 08:44:30 UTC
Permalink
Good morning,
Post by Sourav Saha
Thanks Philipp.
I agree with you, this looks like some buffer overrun issue. I know
I'm using a very old version of icecast software but still I am
asking is there any kind of workaround to this problem apart from
upgrading the icecast version (which is the last option).
Running Valgrind may help to capture the root cause, any thoughts?
I just feel like that even if we would find that with a big amount of
work to dig into it you will likely just end up with the next problem.
And the next one may be something even more hidden.

There are about 41k lines of changes since 2.3.2. A version which were
released at a time with a team which has no overlapping with the current
team.

Unless you really want to get into the next problem that will be ahead
and putting some budget into it I can just strongly recommend you to
upgrade.
Post by Sourav Saha
BTW what's the latest stable version of icecast?
Current stable version is Icecast2 2.4.3. We're working on a 2.4.4 at
this point.


with best regards,
Post by Sourav Saha
-- Sourav
On Sat, Nov 5, 2016 at 11:05 PM, Philipp Schafft
Good evening,
Post by Sourav Saha
Hi,
I am using icecast 2.3.2 and I'm not in position to upgrade
the
Post by Sourav Saha
version right away since this version is already in use.
Please let me
Post by Sourav Saha
know if you need any additional details for this issue.
Nevertheless
Post by Sourav Saha
if there is any potential fix has gone towards this part of
code in
Post by Sourav Saha
later versions, I can just use that patch in my current code
base.
Version 2.3.2 is not just old but stone age. It has a range of known
bugs, security vulnerabilities both local and remote.
The problem you see is very likely because of some random buffer overrun
bug. Maybe one of those vulnerabilities.
Post by Sourav Saha
Also I'll try to catch up on Freenode but can't be very sure
due to
Post by Sourav Saha
time difference, so I'm betting on the replies on this
thread.
with best regards,
Post by Sourav Saha
Thanks/
On Sat, Nov 5, 2016 at 7:08 PM, Philipp Schafft
[...]
On Sat, 2016-11-05 at 16:38 +0530, Sourav Saha
Post by Sourav Saha
I'm facing a problem in icecast relay machines.
I'm seeing
Post by Sourav Saha
frequently
Post by Sourav Saha
that some of the relay machines are crashing and
generates
Post by Sourav Saha
core dump
Post by Sourav Saha
but the reason for the crash is not clear from the
trace of
Post by Sourav Saha
the core.
Post by Sourav Saha
(gdb) bt
[...]
#6 0xf7474708 in malloc ()
from /lib/i386-linux-gnu/libc.so.6
Post by Sourav Saha
#7 0x080b66b7 in _accept_connection () at
connection.c:598
Post by Sourav Saha
Post by Sourav Saha
#8 connection_accept_loop () at connection.c:826
#9 0x080a9c05 in _server_proc () at main.c:384
#10 main (argc=3, argv=0xffbbc554) at main.c:633
As you can see at frame #6 malloc() is failing but
I don't
Post by Sourav Saha
know the
Post by Sourav Saha
reason behind this. Does anybody has any idea how
to fix
Post by Sourav Saha
this issue?
Post by Sourav Saha
Note: this is happening only in the relay machines
but not
Post by Sourav Saha
in edges.
Thank you for your report.
What is the _exact_ version you run?
You can also reach us on IRC (on Freenode, #icecast)
for some
Post by Sourav Saha
live
debugging (when the sun is over central Europe).
--
Philipp Schafft (CEO/GeschÀftsfÌhrer)
Telephon: +49.3535 490 17 92

Löwenfelsen UG (haftungsbeschrÀnkt) Registration number:
Bickinger Straße 21 HRB 12308 CB
04916 Herzberg (Elster) VATIN/USt-ID:
Germany DE305133015
Loading...