T3D 3.7 Crash Heap Corruption

[Mud-H]

I have been porting my TorqueLab project to T3D 3.7 and since that, the game crash everytime I quit after a mission has been loaded. Yesterday, I spent countless hours trying to fix it with no luck... The crash don't happen when using the stock T3D Full template, it only happen when my TorqueLab editor is installed. I tried the same version on my previous code (pre 3.7) and it run fine. The crash occur right after the quit(); call and I couldn't figure what is causing it. I'm familliar with C++ and basic debugging but this time is something I have never experienced which seem to be called Heap Corruption.

I tried to find information about what is causing the crash and the only reference I found is gLightingProgress. I'm not sure how to proceed to debug such crash so if anyone can point me to the good direction it would be really appreciated... What I could do to examine the crash deeper, I tried with the dumpUnflaggedAllocs thing and I got nothing (but I really don't know what I was doing...). Is there a way to get more data from the crash, I'm using VisualStudio 2013. (Tried with 2010 since some research made me think that it could be related to VS 2013).

I have attached 2 screenshots from VS 2010 debugging showing the callstack and some watchs (In the #2 you can see the gLightingProgress references). With the exception of that quit crash, everyting run fine and never crash.

Let me know if there's more usefull information I could share.

EDIT: Forgot to mention that this is with Stock T3D 3.7 code with no changes at all.

http://mud-h.com/miscweb/t3dforum/CrashHeap/GLab_DebugHeapCrash_1.jpg

http://mud-h.com/miscweb/t3dforum/CrashHeap/GLab_DebugHeapCrash_2.jpg

[Mud-H]

I have been reading about how to debug that kind of memory corruption but still haven't figure exactly how to get the cause... I have try with WinDbg and GFlags but not sure how to use it well yet... Anyway here a more detailled report of what happen. The debugger breaks 10 times and then it exit normally. The 8 firsts seem to be related to the gLightingProgress and the 2 last about gTerminateLighting

Here's a custom report describing a bit what those 10 breaks are: (====> = line where it break)

Break #1

	 /*
        * If this ASSERT fails, a bad pointer has been passed in. It may be
        * totally bogus, or it may have been allocated from another heap.
        * The pointer MUST come from the 'local' heap.
        */
 ===>   _ASSERTE(_CrtIsValidHeapPointer(pUserData));
//--------------------------------------------------------------
pUserData	0x111dc3c8 {GameLab_OPTIMIZEDDEBUG DLL.dll!float gLightingProgress}	void *
//--------------------------------------------------------------
Console:		
Program: ...abT3D\My Projects\GameLab\game\GameLab_OPTIMIZEDDEBUG DLL.dll
File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
Line: 1322

Expression: _CrtIsValidHeapPointer(pUserData)

Break #2

 	/* get a pointer to memory block header */
 pHead = pHdr(pUserData);	

/* verify block type */
===>_ASSERTE(_BLOCK_TYPE_IS_VALID(pHead->nBlockUse));

//--------------------------------------------------------------
pUserData	0x111dc3c8 {GameLab_OPTIMIZEDDEBUG DLL.dll!float gLightingProgress}	void *
pHead	GameLab_OPTIMIZEDDEBUG DLL.dll!0x111dc3a8 {pBlockHeaderNext=0x00000000 <NULL> pBlockHeaderPrev=0x00000000 <NULL> ...}	_CrtMemBlockHeader *
nBlockUse	1	int
//--------------------------------------------------------------
Console:
Program: ...abT3D\My Projects\GameLab\game\GameLab_OPTIMIZEDDEBUG DLL.dll
File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
Line: 1328

Expression: _BLOCK_TYPE_IS_VALID(pHead->nBlockUse)

Break #3, #4

		_RPT3(_CRT_ERROR, "HEAP CORRUPTION DETECTED: before %hs block (#%d) at 0x%p.\n"
                       "CRT detected that the application wrote to memory before start of heap buffer.\n",
                       szBlockUseName[_BLOCK_TYPE(pHead->nBlockUse)],
                       pHead->lRequest,
===>                    (BYTE *) pbData(pHead));
//--------------------------------------------------------------						
pHead	GameLab_OPTIMIZEDDEBUG DLL.dll!0x111dc3a8 {pBlockHeaderNext=0x00000000 <NULL> pBlockHeaderPrev=0x00000000 <NULL> ...}	_CrtMemBlockHeader *
//--------------------------------------------------------------						
Program: ...abT3D\My Projects\GameLab\game\GameLab_OPTIMIZEDDEBUG DLL.dll

HEAP CORRUPTION DETECTED: before Free block (#0) at 0x111DC3C8.
CRT detected that the application wrote to memory before start of heap buffer.

Break #5

		/* Error if freeing incorrect memory type */
===>   _ASSERTE(pHead->nBlockUse == nBlockUse);
//--------------------------------------------------------------
pHead	GameLab_OPTIMIZEDDEBUG DLL.dll!0x111dc3a8 {pBlockHeaderNext=0x00000000 <NULL> pBlockHeaderPrev=0x00000000 <NULL> ...}	_CrtMemBlockHeader *
pHead->nBlockUse  	0 	int
nBlockUse			1	int
//--------------------------------------------------------------
Program: ...abT3D\My Projects\GameLab\game\GameLab_OPTIMIZEDDEBUG DLL.dll
File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
Line: 1399

Expression: pHead->nBlockUse == nBlockUse

Break #6

		/* remove from the linked list */
           if (pHead->pBlockHeaderNext)
           {
               pHead->pBlockHeaderNext->pBlockHeaderPrev = pHead->pBlockHeaderPrev;
           }
           else
           {
 ===>         _ASSERTE(_pLastBlock == pHead);
               _pLastBlock = pHead->pBlockHeaderPrev;
           }

//--------------------------------------------------------------
_pLastBlock	0x00616ce8 {pBlockHeaderNext=0x00000000 <NULL> pBlockHeaderPrev=0x0061a188 {pBlockHeaderNext=0x00616ce8 {...} ...} ...}	_CrtMemBlockHeader *
pHead	GameLab_OPTIMIZEDDEBUG DLL.dll!0x111dc3a8 {pBlockHeaderNext=0x00000000 <NULL> pBlockHeaderPrev=0x00000000 <NULL> ...}	_CrtMemBlockHeader *
//--------------------------------------------------------------
Program: ...abT3D\My Projects\GameLab\game\GameLab_OPTIMIZEDDEBUG DLL.dll
File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
Line: 1414

Expression: _pLastBlock == pHead

Break #7

	if (pHead->pBlockHeaderPrev)
           {
               pHead->pBlockHeaderPrev->pBlockHeaderNext = pHead->pBlockHeaderNext;
           }
           else
           {
===>            _ASSERTE(_pFirstBlock == pHead);
               _pFirstBlock = pHead->pBlockHeaderNext;
           }

//--------------------------------------------------------------
_pFirstBlock	0x156d0e38 {pBlockHeaderNext=0x0d824f08 {pBlockHeaderNext=0x0d80fa98 {pBlockHeaderNext=0x0c731c60 {pBlockHeaderNext=...} ...} ...} ...}	_CrtMemBlockHeader *
pHead	GameLab_OPTIMIZEDDEBUG DLL.dll!0x111dc3a8 {pBlockHeaderNext=0x00000000 <NULL> pBlockHeaderPrev=0x00000000 <NULL> ...}	_CrtMemBlockHeader *
//--------------------------------------------------------------
Program: ...abT3D\My Projects\GameLab\game\GameLab_OPTIMIZEDDEBUG DLL.dll
File: f:\dd\vctools\crt\crtw32\misc\dbgheap.c
Line: 1424

Expression: _pFirstBlock == pHead

Break #8

void __cdecl _free_base (void * pBlock)
{

       int retval = 0;


       if (pBlock == NULL)
           return;

       RTCCALLBACK(_RTC_Free_hook, (pBlock, 0));

===>    retval = HeapFree(_crtheap, 0, pBlock);
       if (retval == 0)
       {
           errno = _get_errno_from_oserr(GetLastError());
       }
}

//--------------------------------------------------------------
retval	0	int
_crtheap	0x00600000	void *
pBlock	GameLab_OPTIMIZEDDEBUG DLL.dll!0x111dc3a8	void *
//--------------------------------------------------------------
Critical error detected c0000374
GameLab_OPTIMIZEDDEBUG.exe has triggered a breakpoint.

First-chance exception at 0x77D5EA0B (ntdll.dll) in GameLab_OPTIMIZEDDEBUG.exe: 0xC0000374: Un segment de mémoire a été endommagé (parameters: 0x77D94270).
Unhandled exception at 0x77D5EA0B (ntdll.dll) in GameLab_OPTIMIZEDDEBUG.exe: 0xC0000374: Un segment de mémoire a été endommagé (parameters: 0x77D94270).

Break #9 and #10

After same break #1 and #2 happen but with new pUserData which refer to gTerminateLighting
pUserData	0x111dc3c4 {GameLab_OPTIMIZEDDEBUG DLL.dll!bool gTerminateLighting}	void *

2 breaks are on those line but refer to break #1 and #2 for more info:
_ASSERTE(_CrtIsValidHeapPointer(pUserData));
_ASSERTE(_BLOCK_TYPE_IS_VALID(pHead->nBlockUse));


Then the game exit normally

 

Sorry for bad formatting but should provide more details if someone can help... (Hard to read with those blocks... I have attach a text file showing the full report. Well can't attach text file so here a link to full text: http://mud-h.com/miscweb/t3dforum/CrashHeap/CRASHINFO.txt)

[Mud-H]

I think I figured it out finally. With my last report I discovered that the issue was about gLightingProgress and gTerminateLighting so I reviewed how I used those in script and I think my mistake was that I was storing those variables like prefs... Once I commented the $SceneLighting::* export of my game base the crash was gone.

The 2 script globals are: $sceneLighting::terminateLighting and $sceneLighting::lightingProgress.

I think because $sceneLighting::terminateLighting was store as true and loaded as true, the code never terminate the lighting or something like that... (it seem to do nothing in the code.)

Anyway, I don't really care now that the crash is gone.

[buckmaster]

Oh gosh. Thanks for doing that research! One thing that I really believe is that we should try to make the engine bulletproof from scripts i.e. unless you write an infinite loop in scripts, you shouldn't be able to cause a crash. I know that's an unrealistic goal but we can at least head in that direction, so I'm going to log this as an issue for future reference.

[Mud-H]

Sound like a good idea, there's might be a lot of work to make it bulletproof but we would get there with 1 fix at a time :)

Also thanks for logging it as an issue, it make my time lost not useless...