Editor crashes and performance issues

**Esenthel** · 08-28-2013 09:30 AM

Hi,

Thanks for providing the test case. The results are interesting, when changing the path mesh object property, after time it does seem that the memory usage increases (when viewed from the Task Manager), however I've ran the Editor in debug mode with memory leak detection (it detects all calls to malloc/free/new/delete/new[]/delete[]), and after exiting, there were zero memory leaks.
Which makes me wonder about three possibilities:
1) the memory usage in the Task Manager is incorrect
2) this isn't a problem of memory leaking, but just memory fragmentation, because the Editor uses a lot of memory and operations on computing data, it continuously allocates memory for data to be processed which can result in memory being fragmented, which could result in the Task Manager to display a bigger value than it actually is
3) if for some reason there's a memory leak, then it must be in some library that EE uses (as all memory calls in the EE engine and editor are safe), like lzma, recast, PhysX or DirectX/GPU driver

I'll try to implement memory leak detection in recast for now.

candam · 08-28-2013 11:33 AM

(08-28-2013 09:30 AM)Esenthel Wrote: Hi,

Thanks for providing the test case. The results are interesting, when changing the path mesh object property, after time it does seem that the memory usage increases (when viewed from the Task Manager), however I've ran the Editor in debug mode with memory leak detection (it detects all calls to malloc/free/new/delete/new[]/delete[]), and after exiting, there were zero memory leaks.
Which makes me wonder about three possibilities:
1) the memory usage in the Task Manager is incorrect
2) this isn't a problem of memory leaking, but just memory fragmentation, because the Editor uses a lot of memory and operations on computing data, it continuously allocates memory for data to be processed which can result in memory being fragmented, which could result in the Task Manager to display a bigger value than it actually is
3) if for some reason there's a memory leak, then it must be in some library that EE uses (as all memory calls in the EE engine and editor are safe), like lzma, recast, PhysX or DirectX/GPU driver

I'll try to implement memory leak detection in recast for now.

I guess the first possibility can be be confirmed if you try and find an application to get the ram usage

I think the two other possibilities could be the reason

hope you can find that and solve it

MrPi · 08-28-2013 02:21 PM

(08-28-2013 09:30 AM)Esenthel Wrote: Hi,

Thanks for providing the test case. The results are interesting, when changing the path mesh object property, after time it does seem that the memory usage increases (when viewed from the Task Manager), however I've ran the Editor in debug mode with memory leak detection (it detects all calls to malloc/free/new/delete/new[]/delete[]), and after exiting, there were zero memory leaks.
Which makes me wonder about three possibilities:
1) the memory usage in the Task Manager is incorrect
2) this isn't a problem of memory leaking, but just memory fragmentation, because the Editor uses a lot of memory and operations on computing data, it continuously allocates memory for data to be processed which can result in memory being fragmented, which could result in the Task Manager to display a bigger value than it actually is
3) if for some reason there's a memory leak, then it must be in some library that EE uses (as all memory calls in the EE engine and editor are safe), like lzma, recast, PhysX or DirectX/GPU driver

I'll try to implement memory leak detection in recast for now.

Well, what your leak detection measures is the leaks upon shutting down the application, but not semantic leaks. "Leaks" that result in the application allocating more and more memory for things that will be used and are then properly free'd when the application shuts down. This can be a cache functionality gone wrong for example. Technically those are not leaks detected by a check at shutdown, but the application uses more and more memory. I don't think the task manager is showing wrong numbers.

On my original thread topic, the issues still exists after many, many tests. I tried my notebook with an i7 and the problem didn't happen there. On my main machine the constant allocations are happening in both 32 and 64bit versions of the editor.
But it is still a problem for me, one that makes working with Esenthel very difficult for me. Since it is only happening if the mouse is over the window, could it be something related to input handling?
How can I assist you with finding the root of this. Do you want me to run a special version of the editor here with advanced logging and leak detection and send you the results?

Giving more clues, the issue doesn't seem to happen when playing the game (i.e. Esenthel RPG 2), only in the editor.

**Esenthel** · 08-31-2013 01:15 AM

Thanks for providing the test case Zervox, after I've implemented memory leak detection to Recast, I was able to find one leak related to pathfinding, fixed it (available in next release).

However I think that it's not related to the main issue of this thread.

Quote:Well, what your leak detection measures is the leaks upon shutting down the application, but not semantic leaks. "Leaks" that result in the application allocating more and more memory for things that will be used and are then properly free'd when the application shuts down. This can be a cache functionality gone wrong for example. Technically those are not leaks detected by a check at shutdown, but the application uses more and more memory.

I know what you're saying, but that's not the case, the engine is pretty safe when it comes to memory management.
Your test confirms it:

Quote:I tried my notebook with an i7 and the problem didn't happen there.

I'm not using any separate codepaths for cpu's or gpu's when it comes to just processing in the editor. It looks like there's no problems for 99.9% of engine users, but it just happens on your computer.

Quote:Since it is only happening if the mouse is over the window, could it be something related to input handling?

no, that's not related to input, when the mouse is over the viewport, then multiple threads are fired and each checks for possible collision detection between mouse cursor and the terrain and objects.
It makes me wonder if your system has some issue with creating/deleting threads in each frame.

Quote:How can I assist you with finding the root of this. Do you want me to run a special version of the editor here with advanced logging and leak detection and send you the results?

You could try to write a simple application that in each frame calls 'MultiThreadedCall' for example for "Cpu.cores()" number of elements, and observe the memory usage, something like that:
bool Update()
{
MultiThreadedCall(8, ..);
return true;
}
1) Are you sure you're not having installed any debug/perf. measurement tools or anything suspicious?
2) Did you try uninstalling all apps?
3) Did you try uninstalling all anti-virus, firewalls programs?
4) Did you try updating bios, graphics drivers, windows updates?

Are there any other EE users here that are using AMD CPU and could check the memory usage in the Editor over time?

Zervox · 08-31-2013 03:51 AM

I have 4 computers using different AMD cpus and the issue is noticed on all of them.

I don't have any Intel computers in bootable state at the moment so can't test them.

**Esenthel** · 08-31-2013 03:54 AM

And by that you mean just open the 2.0 editor, open world for editing, do nothing but just keep mouse cursor on the viewport, and the memory will keep growing?

Does this happen even if you open a new empty world?
Move the mouse on the grid plane, so that the semi-transparent dot is visible when "Object" mode is enabled.

MrPi · 08-31-2013 10:03 AM

Some more findings:
It's happening even when the world is empty, but it's only happening when the mouse is over the viewport, not when it's over the controls on top for example.

You are creating and deleting threads in each frame? May I ask how specifically you are doing this, what you are calling?

I was able to reproduce the problem with this simple application:

PHP Code:

void InitPre() // init before engine inits
{
   EE_INIT();
}
bool Init() // initialize after engine is ready
{
   return true;
}
void Shut() // shut down at exit
{
   // this is called when the engine is about to exit
}

Flt fTemp = 0.0f;
void ThreadFunction(int i, ptr p)
{
   // do some random calculations
   fTemp += Sqrt(fTemp * 2.456f);   
}
bool Update() // main updating
{
   MultiThreadedCall(8, ThreadFunction);

   if(Kb.bp(KB_ESC))return false;
   return true;
}
void Draw()
{
   D.clear(TURQ); // clear screen to turquoise color
   D.text (0, 0, "Hello to Esenthel Engine 2.0 !");
} 

The memory is constantly increasing when the app has focus.

**Esenthel** · 08-31-2013 10:36 AM

Ok, great, at least now we know that it's related to thread creation/deletion.

Could you try simplified version of the code:

Code:

/******************************************************************************/

void InitPre() // init before engine inits

{

   EE_INIT();

}

bool Init() // initialize after engine is ready

{

   return true;

}

void Shut() // shut down at exit

{

   // this is called when the engine is about to exit

}

Flt fTemp = 0.01f;

bool ThreadFunction(Thread &t)

{

   // do some random calculations

   fTemp += Sqrt(fTemp * 2.456f);   

   return false;

}

bool Update() // main updating

{

   Thread t[8];

   REPAO(t).create(ThreadFunction);

   REPAO(t).wait();

   if(Kb.bp(KB_ESC))return false;

   return true;

}

void Draw()

{

   D.clear(TURQ); // clear screen to turquoise color

   //D.text (0, 0, "Hello to Esenthel Engine 2.0 !");

   D.text(0, 0, S+fTemp);

} 

/******************************************************************************/

If that generates the same results of memory increasing, then I'll provide more details.

MrPi · 08-31-2013 10:43 AM

Yes, same leaking with this.

As a sidenote it's not a very good idea to create/delete threads as often as this. Using a threadpool for this makes more sense.

**Esenthel** · 08-31-2013 10:59 AM

(08-31-2013 10:43 AM)MrPi Wrote: As a sidenote it's not a very good idea to create/delete threads as often as this. Using a threadpool for this makes more sense.

I've tested it, and there was no performance differences, so I just stick with it.

Here's the very simplified code of the Thread class which should give you an idea what system calls do I use:

Code:

static DWORD WINAPI ThreadFunc(LPVOID user)

{

   Thread &thread=*(Thread*)user;

   if(!thread._func(thread))break; // call user provided function

   thread._active=false; // mark that the thread finished processing and is no longer active

   return 0;

}

void Thread::kill()

{

   if(_handle)

   {

      if(active())TerminateThread(_handle, 0); // if still running then kill it

                  CloseHandle    (_handle   );

      _handle=NULL;

      _active=false;

   }

}

Bool Thread::wait(Int time)

{

   if(active())return WaitForSingleObject(_handle, (time>=0) ? time : INFINITE)!=WAIT_TIMEOUT;

   return true;

}

void Thread::del(Int time)

{

   if(_handle)

   {

      if(active())wait(time); // wait until finishes

      kill();

   }

}

Bool Thread::create(Bool func(Thread &thread), Int priority, Int sleep, Int sleep_min, Ptr user, Bool paused)

{

   T._active   =true;

   T._func     =func;

   if(_handle=CreateThread(NULL, 0, ThreadFunc, this, CREATE_SUSPENDED, NULL))

   {

      ResumeThread(_handle);

      return true;

   }

   return false;

}

MAIN: 'create' sets function pointer 'func' and that it's now 'active'
creates the actual thread and gets its handle

THREAD: the thread now runs via ThreadFunc, where it calls the user 'func' function, then marks that it's no longer 'active' and returns 0

MAIN: wait is called with 'WaitForSingleObject'

then the destructor is called of the thread which calls the 'del' method, which then again checks if its still running, waits, and after that if it's still active, then it kills it, then the handle is closed

MrPi · 08-31-2013 11:13 AM

I would use _beginthreadex and _endthreadex instead of CreateThread etc..

I found this:
"A thread in an executable that is linked to the static C run-time
library (CRT) should use _beginthread and _endthread for thread
management rather than CreateThread and ExitThread. Failure to do so
results in small memory leaks when the thread calls ExitThread."

This is how I use it in my thread library:

PHP Code:

#if !defined(_MT)
#error _beginthreadex requires a multithreaded C run-time library.
#endif

class ThreadBase
{
public:

    ThreadBase();
    virtual ~ThreadBase();
    
    void* Start(int iPriority = THREAD_PRIORITY_NORMAL);

    inline void* GetHandle()    { return m_hThread; }
    inline unsigned int GetID()    { return m_uiThreadID; }

    // functions to override in the real thread class
    virtual bool Initialize();
    virtual void Run();

protected:

    void* m_hThread;
    unsigned int m_uiThreadID;
}; 

PHP Code:

unsigned __stdcall __ThreadProc(void* pArguments)
{
    ThreadBase* pThread = reinterpret_cast<ThreadBase*>(pArguments);
    if (!pThread) return 0;
    if (!pThread->GetHandle()) return 0;

    // do initialization steps before the actual thread loop is running
    if (pThread->Initialize())
    {
        // call the thread loop
        pThread->Run();
    }

    // end the thread
    _endthreadex(0);
    return 0;
} 

ThreadBase::ThreadBase()
{
    m_hThread = NULL;
    m_uiThreadID = 0;
}

ThreadBase::~ThreadBase()
{
    if (m_hThread)
        CloseHandle(m_hThread);
}

HANDLE ThreadBase::Start(int iPriority)
{
    // create thread
    m_hThread = (HANDLE)_beginthreadex( NULL, 0, &__ThreadProc, (void*)this, CREATE_SUSPENDED, &m_uiThreadID);
    if (m_hThread)
    {
        // set priority
        SetThreadPriority(m_hThread, iPriority);

        // run the thread now
        ResumeThread(m_hThread);
    }

    return m_hThread;
}

/*virtual*/ bool ThreadBase::Initialize()
{
    // to be implemented in derived class
    return true;
}

/*virtual*/ void ThreadBase::Run()
{
    // to be implemented in derived class
} 

You are welcome to use it. It has been in use in multiple MMO's.

**Esenthel** · 09-02-2013 06:38 AM

Hi!

I've just uploaded new EE 2.0 version with '_beginthreadex' instead of 'CreateThread'.
Please let me know if that fixed the issue.

MrPi · 09-02-2013 10:21 AM

Esenthel has updated, but no, it's still the same.
Did you also make sure to call _endthreadex at the end of the thread loop and closed the HANDLE?

**Esenthel** · 09-02-2013 11:04 AM

Hi, yes, I do both of the things.
Please try using code from this post http://www.esenthel.com/community/showth...3#pid42393
And use instead of EE::Thread -> _beginthreadex and _endthreadex (or alternatively your class ThreadBase)
To: create 8 threads in each frame, wait until they finish executing function at least once, release them (all that in 1 frame)

MrPi · 09-02-2013 01:59 PM

Ok, I have just built a small application with VS2012 outside of Esenthel2.
You can download it and check the code.

.zip

ThreadTest.zip (Size: 4.65 KB / Downloads: 1)

I am not able to produce any leaks with this. I also didn't expect it to.
That's what you wanted to know, right? That the threading code itself is the reason it leaked on AMD systems?

Editor crashes and performance issues
Author	Message