[muscle] Re: Fwd: Updating from 3.30 to 5.22

  • From: Raymond Dahlberg <rd@xxxxxxxxxxx>
  • To: muscle@xxxxxxxxxxxxx
  • Date: Fri, 9 Apr 2010 18:11:05 +0200

Hi Jeremy,

Thank you very much for your help, much appreciated.

Part of the motivation for updating our code this time were to make it multi
platform again, it has been tied to Windows for some time. We have been
using some Windows code for thread handling, but are planning to use Qt for
that from now on, which should make building on Linux possible (we are using
Qt extensively on other parts of the code already).

So I will try to track down this issue when it builds on Linux.

/Raymond

On Fri, Apr 9, 2010 at 5:39 PM, Jeremy Friesner <jeremyf@xxxxxxxxxxxxxx>wrote:

> Hi Raymond,
>
>
> On Apr 9, 2010, at 6:16 AM, "Raymond Dahlberg" <rd@xxxxxxxxxxx> wrote:
>
> Does this mean we are using code that are incompatible with the new slab
> allocation scheme, or is it just that we have leaks we do not know about
> when using the normal new/delete allocation?
>
>
> What it means is that when the ObjectPool destructors are called (generally
> after main() returns, since the ObjectPools are typically statically
> allocated objects), one of them still has at least one of its objects still
> in use, which means that deleting its object-array could cause strange
> errors, because the code holding the object reference would then be holding
> a dangling pointer.
>
> Typically this would be caused by imperfect cleanup... Either a static data
> structure is holding a reference to a pooled object, or an object that is
> holding a reference was leaked, and thus it never un-referenced the pooled
> objects it is holding.
>
> When I changed ObjectPool to use slab allocation, I had to track down
> several similar problems in my code too.  The slab ObjectPool makes the
> previously invisible problem apparent.
>
> It's your choice... If you don't want to spend time tracking down the
> 'missing' de-allocations, you can just disable the object pools and things
> will run as before.  Or you can go the perfectionist route and figure out
> where the refs are still being held, and fix the code to make sure
> everything is torn down reliably.  That way you can be pretty well assured
> you aren't leaking memory.
>
> If you can compile under Linux, valgrind --leak-check=full might give you
> some good info on what (if anything) is being leaked.
>
> Jeremy
>
>
>
> As I said in the first mail, this code has run unchanged for quite some
> time, and the only changes done now are to make it compile with the newest
> Muscle...
>
> /Raymond
>
> On Thu, Apr 8, 2010 at 4:17 PM, Jeremy Friesner < <jeremyf@xxxxxxxxxxxxxx>
> jeremyf@xxxxxxxxxxxxxx> wrote:
>
>> Hi Raymond,
>>
>> Look in ObjectPool.h, the #ifdef should be present there.  (I'd look it up
>> myself but I'm away from my computer this week and can't look at code from
>> my cell phone)
>>
>> Hunting down the non-released object is a good idea in any case, since it
>> may indicate a memory leak.  You might be able to make the assertion failure
>> more helpful (e.g. By having it print out the type of the ObjectPool, which
>> would include the type of the unreleased object as its template argument).
>>  Unfortunately it's not possible to print out the address of the unreleased
>> object, only the address of the slab whose reference count is non-zero. You
>> could print out the slab's reference count though; then you'd know how many
>> objects are unaccounted for.
>>
>> Jeremy
>>
>>
>>
>>
>> On Apr 8, 2010, at 3:39 AM, "Raymond Dahlberg" < <rd@xxxxxxxxxxx>
>> rd@xxxxxxxxxxx> wrote:
>>
>> The closest define I could find was MUSCLE_DISABLE_MESSAGE_FIELD_POOLS in
>> the BUILDOPTIONS.txt file. I have now tested to build with this flag
>> defined. But the problem is still there, and the error message is the same:
>>
>> [C 04/08 11:35:02] ~ObjectPool 005E2560:  slab 007A9798 is still in use
>> when we destroy it!
>> [C 04/08 11:35:02] ASSERTION FAILED: (\muscle\util\objectpool.h:125)
>> ObjectPool destroyed while its objects were still in use (Is a
>> CompleteSetupSystem object declared at the top of main()?)
>>
>> I guess I have to try to find out which object that is not released.
>>
>> Any hints welcome :-)
>>
>> Raymond
>>
>> On Wed, Apr 7, 2010 at 4:59 PM, Jeremy Friesner 
>> <<jeremyf@xxxxxxxxxxxxxx><jeremyf@xxxxxxxxxxxxxx>
>> jeremyf@xxxxxxxxxxxxxx> wrote:
>>
>>>
>>>
>>>
>>>
>>> Begin forwarded message:
>>>
>>> *From:* "Jeremy Friesner" < <jeremyf@xxxxxxxxxxxxxx><jeremyf@xxxxxxxxxxxxxx>
>>> jeremyf@xxxxxxxxxxxxxx>
>>> *Date:* April 7, 2010 7:22:51 AM PDT
>>> *To:* < <muscle@xxxxxxxxxxxxx> <muscle@xxxxxxxxxxxxx>
>>> muscle@xxxxxxxxxxxxx>
>>> *Subject:* *Re: [muscle] Updating from 3.30 to 5.22*
>>>
>>>
>>> Hi Raymond,
>>>
>>> I think what you are seeing is a result of ObjectPool switching to a slab
>>> allocator mechanism.  Older versions of ObjectPool used individually
>>> allocated objects, but with the move to ConstSocketRefs for socket tracking,
>>> I switched to slab allocation so that a whole array of objects is allocated
>>> at once instead.
>>>
>>> As a side effect of that, the entire array of objects also needs to be
>>> freed at once.  When the ObjectPool object itself is destroyed (e.g. at
>>> process exit time) it needs to delete its allocated object arrays, but if
>>> any of the objects are still in use (I.e. hasn't been released back to the
>>> pool), doing so would cause s dangling pointer, so an assertion failure is
>>> triggered to help track down the 'leak'.
>>>
>>> The best fix is to figure out what object has been allocated from a pool
>>> and never returned to the pool, and make sure it gets returned to the pool
>>> before the process exits.  The quick fix is to add
>>> -DMUSCLE_DISABLE_OBJECT_POOLS (iirc) to your Makefile to disable object
>>> pools (and thus the slab allocations).  With that flag object pools just
>>> become a pass through to new/delete.
>>>
>>> Jeremy
>>>
>>>
>>> On Apr 7, 2010, at 4:30 AM, "Raymond Dahlberg" < 
>>> <rd@xxxxxxxxxxx><rd@xxxxxxxxxxx><rd@xxxxxxxxxxx>
>>> rd@xxxxxxxxxxx> wrote:
>>>
>>> Hi Muscle list!
>>>
>>> We have been using Muscle for several years in our own network library.
>>> The code has been pretty stable and unchanged for some time, but now we want
>>> to do some changes, and decided it was time to upgrade Muscle too.
>>>
>>> I have successfully adapted our code to the source incompatible changes,
>>> and everything builds, and the first tests show that basic functionality
>>> works. Clients connect to the server, and the can communicate. So far so
>>> good :)
>>>
>>> But it seems something is changed in the cleanup code, since I always get
>>> a MUSCLE assertion failure when the last muscle application exits. It also
>>> outputs on the command line:
>>>
>>> [C 04/07 13:20:22] ~ObjectPool 004FB268:  slab 00A89CD0 is still in use
>>> when we destroy it!
>>> [C 04/07 13:20:22] ASSERTION FAILED: (\muscle\util/ObjectPool.h:125)
>>> ObjectPool destroyed while its objects were still in use (Is
>>> a CompleteSetupSystem object declared at the top of main()?)
>>>
>>> I understand that one must have a CompleteSetupSystem on top of main, but
>>> this has been there all the time (not changed during muscle upgrade). For
>>> example a little test server app where this is happening has this in main:
>>>
>>> int main( int argc, char** argv )
>>> {
>>>     // Initialize NetBorealis.
>>>     SNbSetupSystem ss;
>>>
>>>
>>> Where SNbSetupSystem is a singleton with this constructor:
>>>
>>>     SNbSetupSystem::SNbSetupSystem( )
>>>         : m_CssPtr( 0 ),
>>>         _serverExitedSignalHandle( 0 )
>>>     {
>>>         if ( m_CssPtr == 0 )
>>>         {
>>>             try {
>>>                 m_CssPtr = new CompleteSetupSystem( );
>>>             } catch ( bad_alloc& ) { fatalError( "FATAL ERROR: Out of
>>> memory (%s:%d).\n", __FILE__, __LINE__ ); }
>>>         }
>>>         _serverExitedSignalHandle = CreateEvent( 0, 0, 0, 0 );
>>>     }
>>>
>>> Any hints on this?
>>>
>>> Regards,
>>> Raymond
>>>
>>>
>>>  NOTICE: This email may contain confidential information. Please see
>>> <http://www.meyersound.com/confidential/><http://www.meyersound.com/confidential/>
>>> http://www.meyersound.com/confidential/ for our complete policy.
>>>
>>
>>
>>  NOTICE: This email may contain confidential information. Please see
>> <http://www.meyersound.com/confidential/>
>> http://www.meyersound.com/confidential/ for our complete policy.
>>
>
>
>  NOTICE: This email may contain confidential information. Please see
> http://www.meyersound.com/confidential/ for our complete policy.
>

Other related posts: