About Store Forum Documentation Contact



Post Reply 
Editor crashes and performance issues
Author Message
Esenthel Offline
Administrator

Post: #16
RE: Editor crashes and performance issues
Hi,

Thanks for providing the test case. The results are interesting, when changing the path mesh object property, after time it does seem that the memory usage increases (when viewed from the Task Manager), however I've ran the Editor in debug mode with memory leak detection (it detects all calls to malloc/free/new/delete/new[]/delete[]), and after exiting, there were zero memory leaks.
Which makes me wonder about three possibilities:
1) the memory usage in the Task Manager is incorrect
2) this isn't a problem of memory leaking, but just memory fragmentation, because the Editor uses a lot of memory and operations on computing data, it continuously allocates memory for data to be processed which can result in memory being fragmented, which could result in the Task Manager to display a bigger value than it actually is
3) if for some reason there's a memory leak, then it must be in some library that EE uses (as all memory calls in the EE engine and editor are safe), like lzma, recast, PhysX or DirectX/GPU driver

I'll try to implement memory leak detection in recast for now.
08-28-2013 09:30 AM
Find all posts by this user Quote this message in a reply
candam Offline
Member

Post: #17
RE: Editor crashes and performance issues
(08-28-2013 09:30 AM)Esenthel Wrote:  Hi,

Thanks for providing the test case. The results are interesting, when changing the path mesh object property, after time it does seem that the memory usage increases (when viewed from the Task Manager), however I've ran the Editor in debug mode with memory leak detection (it detects all calls to malloc/free/new/delete/new[]/delete[]), and after exiting, there were zero memory leaks.
Which makes me wonder about three possibilities:
1) the memory usage in the Task Manager is incorrect
2) this isn't a problem of memory leaking, but just memory fragmentation, because the Editor uses a lot of memory and operations on computing data, it continuously allocates memory for data to be processed which can result in memory being fragmented, which could result in the Task Manager to display a bigger value than it actually is
3) if for some reason there's a memory leak, then it must be in some library that EE uses (as all memory calls in the EE engine and editor are safe), like lzma, recast, PhysX or DirectX/GPU driver

I'll try to implement memory leak detection in recast for now.

I guess the first possibility can be be confirmed if you try and find an application to get the ram usage

I think the two other possibilities could be the reason

hope you can find that and solve it
08-28-2013 11:33 AM
Find all posts by this user Quote this message in a reply
MrPi Offline
Member

Post: #18
RE: Editor crashes and performance issues
(08-28-2013 09:30 AM)Esenthel Wrote:  Hi,

Thanks for providing the test case. The results are interesting, when changing the path mesh object property, after time it does seem that the memory usage increases (when viewed from the Task Manager), however I've ran the Editor in debug mode with memory leak detection (it detects all calls to malloc/free/new/delete/new[]/delete[]), and after exiting, there were zero memory leaks.
Which makes me wonder about three possibilities:
1) the memory usage in the Task Manager is incorrect
2) this isn't a problem of memory leaking, but just memory fragmentation, because the Editor uses a lot of memory and operations on computing data, it continuously allocates memory for data to be processed which can result in memory being fragmented, which could result in the Task Manager to display a bigger value than it actually is
3) if for some reason there's a memory leak, then it must be in some library that EE uses (as all memory calls in the EE engine and editor are safe), like lzma, recast, PhysX or DirectX/GPU driver

I'll try to implement memory leak detection in recast for now.

Well, what your leak detection measures is the leaks upon shutting down the application, but not semantic leaks. "Leaks" that result in the application allocating more and more memory for things that will be used and are then properly free'd when the application shuts down. This can be a cache functionality gone wrong for example. Technically those are not leaks detected by a check at shutdown, but the application uses more and more memory. I don't think the task manager is showing wrong numbers.

On my original thread topic, the issues still exists after many, many tests. I tried my notebook with an i7 and the problem didn't happen there. On my main machine the constant allocations are happening in both 32 and 64bit versions of the editor.
But it is still a problem for me, one that makes working with Esenthel very difficult for me. Since it is only happening if the mouse is over the window, could it be something related to input handling?
How can I assist you with finding the root of this. Do you want me to run a special version of the editor here with advanced logging and leak detection and send you the results?

Giving more clues, the issue doesn't seem to happen when playing the game (i.e. Esenthel RPG 2), only in the editor.
08-28-2013 02:21 PM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #19
RE: Editor crashes and performance issues
Thanks for providing the test case Zervox, after I've implemented memory leak detection to Recast, I was able to find one leak related to pathfinding, fixed it (available in next release).

However I think that it's not related to the main issue of this thread.

Quote:Well, what your leak detection measures is the leaks upon shutting down the application, but not semantic leaks. "Leaks" that result in the application allocating more and more memory for things that will be used and are then properly free'd when the application shuts down. This can be a cache functionality gone wrong for example. Technically those are not leaks detected by a check at shutdown, but the application uses more and more memory.
I know what you're saying, but that's not the case, the engine is pretty safe when it comes to memory management.
Your test confirms it:
Quote:I tried my notebook with an i7 and the problem didn't happen there.
I'm not using any separate codepaths for cpu's or gpu's when it comes to just processing in the editor. It looks like there's no problems for 99.9% of engine users, but it just happens on your computer.

Quote:Since it is only happening if the mouse is over the window, could it be something related to input handling?
no, that's not related to input, when the mouse is over the viewport, then multiple threads are fired and each checks for possible collision detection between mouse cursor and the terrain and objects.
It makes me wonder if your system has some issue with creating/deleting threads in each frame.

Quote:How can I assist you with finding the root of this. Do you want me to run a special version of the editor here with advanced logging and leak detection and send you the results?
You could try to write a simple application that in each frame calls 'MultiThreadedCall' for example for "Cpu.cores()" number of elements, and observe the memory usage, something like that:
bool Update()
{
MultiThreadedCall(8, ..);
return true;
}
1) Are you sure you're not having installed any debug/perf. measurement tools or anything suspicious?
2) Did you try uninstalling all apps?
3) Did you try uninstalling all anti-virus, firewalls programs?
4) Did you try updating bios, graphics drivers, windows updates?

Are there any other EE users here that are using AMD CPU and could check the memory usage in the Editor over time?
08-31-2013 01:15 AM
Find all posts by this user Quote this message in a reply
Zervox Offline
Member

Post: #20
RE: Editor crashes and performance issues
I have 4 computers using different AMD cpus and the issue is noticed on all of them.

I don't have any Intel computers in bootable state at the moment so can't test them.
(This post was last modified: 08-31-2013 03:57 AM by Zervox.)
08-31-2013 03:51 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #21
RE: Editor crashes and performance issues
And by that you mean just open the 2.0 editor, open world for editing, do nothing but just keep mouse cursor on the viewport, and the memory will keep growing?

Does this happen even if you open a new empty world?
Move the mouse on the grid plane, so that the semi-transparent dot is visible when "Object" mode is enabled.
08-31-2013 03:54 AM
Find all posts by this user Quote this message in a reply
MrPi Offline
Member

Post: #22
RE: Editor crashes and performance issues
Some more findings:
It's happening even when the world is empty, but it's only happening when the mouse is over the viewport, not when it's over the controls on top for example.

You are creating and deleting threads in each frame? May I ask how specifically you are doing this, what you are calling?

I was able to reproduce the problem with this simple application:

PHP Code:
void InitPre() // init before engine inits
{
   
EE_INIT();
}
bool Init() // initialize after engine is ready
{
   return 
true;
}
void Shut() // shut down at exit
{
   
// this is called when the engine is about to exit
}

Flt fTemp 0.0f;
void ThreadFunction(int iptr p)
{
   
// do some random calculations
   
fTemp += Sqrt(fTemp 2.456f);   
}
bool Update() // main updating
{
   
MultiThreadedCall(8ThreadFunction);

   if(
Kb.bp(KB_ESC))return false;
   return 
true;
}
void Draw()
{
   
D.clear(TURQ); // clear screen to turquoise color
   
D.text (00"Hello to Esenthel Engine 2.0 !");


The memory is constantly increasing when the app has focus.
(This post was last modified: 08-31-2013 10:14 AM by MrPi.)
08-31-2013 10:03 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #23
RE: Editor crashes and performance issues
Ok, great, at least now we know that it's related to thread creation/deletion.

Could you try simplified version of the code:
Code:
/******************************************************************************/
void InitPre() // init before engine inits
{
   EE_INIT();
}
bool Init() // initialize after engine is ready
{
   return true;
}
void Shut() // shut down at exit
{
   // this is called when the engine is about to exit
}
Flt fTemp = 0.01f;
bool ThreadFunction(Thread &t)
{
   // do some random calculations
   fTemp += Sqrt(fTemp * 2.456f);  
   return false;
}
bool Update() // main updating
{
   Thread t[8];
   REPAO(t).create(ThreadFunction);
   REPAO(t).wait();

   if(Kb.bp(KB_ESC))return false;
   return true;
}
void Draw()
{
   D.clear(TURQ); // clear screen to turquoise color
   //D.text (0, 0, "Hello to Esenthel Engine 2.0 !");
   D.text(0, 0, S+fTemp);
}
/******************************************************************************/

If that generates the same results of memory increasing, then I'll provide more details.
08-31-2013 10:36 AM
Find all posts by this user Quote this message in a reply
MrPi Offline
Member

Post: #24
RE: Editor crashes and performance issues
Yes, same leaking with this.

As a sidenote it's not a very good idea to create/delete threads as often as this. Using a threadpool for this makes more sense.
(This post was last modified: 08-31-2013 10:46 AM by MrPi.)
08-31-2013 10:43 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #25
RE: Editor crashes and performance issues
(08-31-2013 10:43 AM)MrPi Wrote:  As a sidenote it's not a very good idea to create/delete threads as often as this. Using a threadpool for this makes more sense.
I've tested it, and there was no performance differences, so I just stick with it.

Here's the very simplified code of the Thread class which should give you an idea what system calls do I use:
Code:
static DWORD WINAPI ThreadFunc(LPVOID user)
{
   Thread &thread=*(Thread*)user;
   if(!thread._func(thread))break; // call user provided function
   thread._active=false; // mark that the thread finished processing and is no longer active
   return 0;
}
void Thread::kill()
{
   if(_handle)
   {
      if(active())TerminateThread(_handle, 0); // if still running then kill it
                  CloseHandle    (_handle   );
      _handle=NULL;
      _active=false;
   }
}
Bool Thread::wait(Int time)
{
   if(active())return WaitForSingleObject(_handle, (time>=0) ? time : INFINITE)!=WAIT_TIMEOUT;
   return true;
}
void Thread::del(Int time)
{
   if(_handle)
   {
      if(active())wait(time); // wait until finishes
      kill();
   }
}
Bool Thread::create(Bool func(Thread &thread), Int priority, Int sleep, Int sleep_min, Ptr user, Bool paused)
{
   T._active   =true;
   T._func     =func;

   if(_handle=CreateThread(NULL, 0, ThreadFunc, this, CREATE_SUSPENDED, NULL))
   {
      ResumeThread(_handle);
      return true;
   }
   return false;
}
MAIN: 'create' sets function pointer 'func' and that it's now 'active'
creates the actual thread and gets its handle

THREAD: the thread now runs via ThreadFunc, where it calls the user 'func' function, then marks that it's no longer 'active' and returns 0

MAIN: wait is called with 'WaitForSingleObject'

then the destructor is called of the thread which calls the 'del' method, which then again checks if its still running, waits, and after that if it's still active, then it kills it, then the handle is closed
08-31-2013 10:59 AM
Find all posts by this user Quote this message in a reply
MrPi Offline
Member

Post: #26
RE: Editor crashes and performance issues
I would use _beginthreadex and _endthreadex instead of CreateThread etc..

I found this:
"A thread in an executable that is linked to the static C run-time
library (CRT) should use _beginthread and _endthread for thread
management rather than CreateThread and ExitThread. Failure to do so
results in small memory leaks when the thread calls ExitThread."

This is how I use it in my thread library:

PHP Code:
#if !defined(_MT)
#error _beginthreadex requires a multithreaded C run-time library.
#endif

class ThreadBase
{
public:

    
ThreadBase();
    
virtual ~ThreadBase();
    
    
voidStart(int iPriority THREAD_PRIORITY_NORMAL);

    
inline voidGetHandle()    { return m_hThread; }
    
inline unsigned int GetID()    { return m_uiThreadID; }

    
// functions to override in the real thread class
    
virtual bool Initialize();
    
virtual void Run();

protected:

    
voidm_hThread;
    
unsigned int m_uiThreadID;
}; 

PHP Code:
unsigned __stdcall __ThreadProc(voidpArguments)
{
    
ThreadBasepThread reinterpret_cast<ThreadBase*>(pArguments);
    if (!
pThread) return 0;
    if (!
pThread->GetHandle()) return 0;

    
// do initialization steps before the actual thread loop is running
    
if (pThread->Initialize())
    {
        
// call the thread loop
        
pThread->Run();
    }

    
// end the thread
    
_endthreadex(0);
    return 
0;


ThreadBase::ThreadBase()
{
    
m_hThread NULL;
    
m_uiThreadID 0;
}

ThreadBase::~ThreadBase()
{
    if (
m_hThread)
        
CloseHandle(m_hThread);
}

HANDLE ThreadBase::Start(int iPriority)
{
    
// create thread
    
m_hThread = (HANDLE)_beginthreadexNULL0, &__ThreadProc, (void*)thisCREATE_SUSPENDED, &m_uiThreadID);
    if (
m_hThread)
    {
        
// set priority
        
SetThreadPriority(m_hThreadiPriority);

        
// run the thread now
        
ResumeThread(m_hThread);
    }

    return 
m_hThread;
}

/*virtual*/ bool ThreadBase::Initialize()
{
    
// to be implemented in derived class
    
return true;
}

/*virtual*/ void ThreadBase::Run()
{
    
// to be implemented in derived class


You are welcome to use it. It has been in use in multiple MMO's.
(This post was last modified: 08-31-2013 11:14 AM by MrPi.)
08-31-2013 11:13 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #27
RE: Editor crashes and performance issues
Hi!

I've just uploaded new EE 2.0 version with '_beginthreadex' instead of 'CreateThread'.
Please let me know if that fixed the issue.
09-02-2013 06:38 AM
Find all posts by this user Quote this message in a reply
MrPi Offline
Member

Post: #28
RE: Editor crashes and performance issues
Esenthel has updated, but no, it's still the same.
Did you also make sure to call _endthreadex at the end of the thread loop and closed the HANDLE?
09-02-2013 10:21 AM
Find all posts by this user Quote this message in a reply
Esenthel Offline
Administrator

Post: #29
RE: Editor crashes and performance issues
Hi, yes, I do both of the things.
Please try using code from this post http://www.esenthel.com/community/showth...3#pid42393
And use instead of EE::Thread -> _beginthreadex and _endthreadex (or alternatively your class ThreadBase)
To: create 8 threads in each frame, wait until they finish executing function at least once, release them (all that in 1 frame)
09-02-2013 11:04 AM
Find all posts by this user Quote this message in a reply
MrPi Offline
Member

Post: #30
RE: Editor crashes and performance issues
Ok, I have just built a small application with VS2012 outside of Esenthel2.
You can download it and check the code.


.zip  ThreadTest.zip (Size: 4.65 KB / Downloads: 1)

I am not able to produce any leaks with this. I also didn't expect it to.
That's what you wanted to know, right? That the threading code itself is the reason it leaked on AMD systems?
09-02-2013 01:59 PM
Find all posts by this user Quote this message in a reply
Post Reply