Troubleshooting random crashes

[andrewmac]

Did you run the debug build? Things don't usually just crash with no debug information. Is it producing an error?

[PaulWeston]

The debug build usually just dumps me back in some DataChunker function with a generic memory error. Nothing that points specifically to any of my additional resources or anything.

What is the weirdest thing is that it worked, I cheered, I played it, then I shut it down and had lunch... Ran it again, and had more fun, showed it off to the family, then broke for dinner. Then sometime later I ran it again and it started crashing. With nothing changed. WTF?

I'm racking my brain for what I did during that time... I did install two pieces of software on that machine - Format Factory and another program to convert video files to OGV format. Because I was preparing to convert my Awesomium stuff to GuiTheoraControls. But I was still able to run the game after that. It was only after another 4 or 5 loads that the crashes started. I was in-game working on the GuiTheoraControl stuff, almost had it working, then on my next load I started having problems. Restored all my scripts and also the materials files (which had been tweaked for the theora stuff), but the crashes stayed. Even with the same files they worked on before.

So odd... Ideas I've thought of so far:

Possibly my machine is too low on disk space? I was down to about 6 gigs at one point during this time, but after crashes started I freed up 30 gigs and rebooted, no difference.

Too many other programs resident, totalling over 2 gigs, so when game loads it adds 2 gigs and I max out my 4 gigs? Hence why problem does not occur on my 8 gig work PC? Well, my machine idles between 800 MB and 1.4 gigs, so there should be plenty of room.

Code was maybe always buggered, but just by sheer BS luck I was able to run the game a dozen times before the problems started manifesting?

The saga continues lol...

[PaulWeston]

OMG would you believe that after fussing with this for 3 days pulling my hair out, on a whim I just set the exe compatiblity mode in Windows to be "Windows 7", and then everything goes back to working fine in 64 bit?

WTF, really Microsoft?

lol

Anyhow, I guess my runtime libraries or redistributable got messed up somehow or something, but forcing the compatibility to Windows 7 seems to fix it. No probs running all over the ship and stuff loading in and AI spawning, all good.

Phew, relief. But ya, how stupid is that?

Glad it's fixed at least, now back to work :) The 64 bit really is awesome.

[PaulWeston]

Teaser showing the whole ship loaded, with hull removed

Lovin' it :)

http://riversidefantasy.ca/screencaps/00-Cutaway.jpg

[buckmaster]

That's super bizarre. Are you in Windows 7, or 8?

[PaulWeston]

Running Windows 7 home premium 64-bit on all machines at home. At work it's windows 7 professional or enterprise or whatever. Only difference really is amount of ram, home machines have only 4 and work has 8. Home machines even all have after market video and rate between 5.9 and 7.1 in the performance index. Other new games are fine.

So ya its weird lol. But at least that fixes it.

As for the low FPS, my machine is actually the slowest in the house, kids got the quad cores and mine is just an old dual core refurb. Good video card though. But i always get low FPS.

Its kind of decieving though, its actually playable at that speed and is not as choppy as one would think.

What do you mean by profiling? I'm curious :) we are not open source but for you guys I could provide a build for r&d if you like.

[buckmaster]

By profiling I mean enabling the profiler to see which parts of the code are taking the most time. See if there are any particular bottlenecks.

[PaulWeston]

That would be great to know... Is that something I can do myself?

Maybe it's just that I have a lot of script stuff going on all the time, and that I am running on dual core with 4 gigs. I will know more tomorrow when I am at my colleague's using a 6-core 8 gb machine what the performance difference is.

Would be happy to provide everything for testing if need be. It would be big though - engine code is not bad of course but game folder is 1.4 gigs zipped, there is a lot of stuff :)

[buckmaster]

If you hold down ctrl F3 for a little while the profiler will dump a sample. It'd be very interesting to see what results you get!

[PaulWeston]

Wow, cool, have never seen this before :)

At a glance, I don't see anything particular that is caught in a loop or anything. But maybe you can tell me a bit more about what it going on.

Some notes:

- I put the zones and portals back in, because now with 64-Bit they no longer seem to add to the crashing problem I was having before... I do still have some issues with the portals, they seem to be revealing more than I think they should, when you look at that direction even if you are looking through walls. But with zones back in I gained a lot of FPS on my dual-core system.

- I also set it up so when you are in the saucer section, it completely removes the lower hull decks, and vice-versa. When you move from one to the other, it plays a short turbolift video animation while it takes the 10 seconds or so to load in all the decks again. So when the turbolift door opens, the deck is loaded there for you. This also gained a lot of FPS.

So now on my dual-core system with only 4 gb ram, I can average 15-25 FPS in most areas (drops to 5-10 when loading stuff, then comes back up). With that, the game becomes viable even on older systems.

I did 20 seconds or so of profiling for this. Have attached the file to this post.

Thanks!

P

profilerDumpToFile127904.zip

[andrewmac]

For those too lazy to download and open the file, here's the top items:

%%NSTime %% Time Invoke # Name

889301.429 1412807.143 2966895 GenericConstBufferLayout_set

556268.571 556508.571 1168668 AdvancedLightManager_GetLightingShaderConstants

523953.810 523953.810 1100303 String_String_constructor

496644.762 503736.190 1057846 ShapeBase_ProcessTick

493198.571 523505.714 1099362 GFXD3D9ShaderBufferLayout_setMatrix

428440.000 428440.000 899724 FeatureSet_hasFeature

374434.286 374434.286 786312 RenderBinManager_setupSGData

247898.571 249967.143 524931 LightManager_Update4LightConsts

224756.667 228688.571 480246 ProcessedShaderMaterial_SetShaderConstants

223342.381 223342.381 469019 ProcessedShaderMaterial_stepInstance

221014.286 228688.571 480246 ProcessedShaderMaterial_SetTextureTransforms

211990.952 211990.952 445181 VectorResize

184622.857 184622.857 387708 SceneCullingState_isOccludedByTerrain

134128.571 134128.571 281670 String_default_constructor

133589.524 133589.524 280538 GFXD3D9ShaderConstBuffer_activate_dirty_check_1

130927.619 236888.571 497466 GFXDevice_updateStates

107886.190 241475.714 507099 GFXD3D9ShaderConstBuffer_activate

106340.476 106340.476 223315 GFXD3D9VertexBuffer_unlock

105960.952 105960.952 222518 GFXD3D9StateBlock_Activate

103763.333 103763.333 217903 GFXD3D9Device_findVBPool

[buckmaster]

There are some very interesting results there. String? isOccludedByTerrain?

[PaulWeston]

IsOccludedByTerrain is in scene/culling/sceneCullingState.cpp, in the function:

U32 SceneCullingState::cullObjects

And the String bit appears to be from core/util/str.cpp:

String::String()

{

PROFILE_SCOPE(String_default_constructor);

_string = StringData::Empty();

}

They are part of the stock engine code I'm pretty sure, I didn't add them, is there a reason we should not be seeing those?

[andrewmac]

The list you're seeing is listing, from the top, the functions that are called the most and/or taking the most time. The point of looking at the profile is to figure out whats taking so long so you can try to reduce that particular item.

Seeing IsOccludedByTerrain seems odd because you don't have any terrain. And String::String() is odd because why is the engine spending so much time allocating strings? It almost seems like you're creating new strings every frame for some reason. Is there anything you can think of that's occuring per-frame that creates a new string? It may also be coming from torquescript.

[PaulWeston]

Hmm, that's interesting for sure...

The scene is full of zones, so maybe part of having zones in there causes it to run through every possible occlusion method, including terrain? Agreed that I do not have terrain in there at all so why would that need to be checked, at least in this mission - I do have a mission with a Cestus III terrain and the Vasquez rocks where you can go fight some Gorn, but that wasn't what I used to do the profiling.

As for the String stuff, I'm not sure... I do a lot of command to client and command to server back-and-forth in my TorqueScript, doesn't that use tagged strings and such flying back and forth? Other than that I don't know what else I'm doing that is out of the ordinary.

[buckmaster]

The ShapeBase::processTick is also interesting. Do you use a lot of ShapeBase objects? Or derived classes like StaticShape?

EDIT: I should mention that my interest is primarily in seeing examples of real games and seeing what takes performance, and what we should maybe think about paying some attention to in optimisation. AFAICT T3D seems to be one of those 'uniformly slow' programs that just runs slower than it should overall, rather than having particular hotspots - though those profile results do show some interesting hotspots like the String constructor.

EDIT EDIT: oh, the indentation seems a little messed up on the tree-based profile dump (the second section in the dump file). It's a shame. There's some interesting-looking pieces but I'm not sure if the indentation is just wrong, haha. Like this:

 

4800.000 -18500.000    10080       GFXD3D9Device_allocVertexBuffer
13700.000 13700.000    28770         String_String_constructor

[PaulWeston]

The ship is made up of mostly TSStatic objects, but yes there is a fair amount of StaticShape datablocked shapes for all the stuff that has to work..

I agree T3D seems 'uniformly slow' lol, you really think it would run faster than it does. But at least it is malleable and we've been able to tweak it enough to make this workable.

Anyhow, I'm happy to share any data that may help with engine development, Not sure why the output file tabs are messed up in the latter part of that file, will try to get another profile dump and post it tomorrow.

Cheers

P

[Johxz]

@PaulWeston Is MIT now https://github.com/StefanLundmark/Torque3D_Awesomium

Check this is you want to see other products releases: http://forums.torque3d.org/viewtopic.php?f=25&t=675