Engine Crash

[Hodo33]

I say engine crash but probably my bad script code so I'll open with that.

Did a release build and getting crashes at various parts of script execution. Got in the engine and altered a couple defines and it's a bit more stable.

#define TORQUE_TOOLS commented out and removed tools directory from release

#define TORQUE_SHIPPING added in

Other than these two are there any other defines or flags that should be set ?

The crash mostly happens on a zone switch. I use triggers to call a zone switch and tried to follow the cyclegame routines.

Stop all schedules, wait for any mid schedule events to complete, stop all client side actions kill the server, endgame etc then load the new zone. The transition seems to be the crash point. Is there a "safe" order to shutdown one zone and load another? Yeah I know big ask in the blind but any suggestions would help... thanks....

[Duion]

Well it is hard to guess what is wrong from that description, you should try to debug yourself first.

First you should install something like Torsion, a Torquescript debugger and see if you can identify the error in the scripts, if that does not help, use a code debugger like Visualstudio for example and see if you can find the error there.

At least that is how I do that.

It seems like you want to remove the tools from the game, I think you can just delete the tools folder and remove it from being executed, but I think it loads automatically from the main.cs in the tools folder or so.

[Hodo33]

I have Torsion

Tried a debug build but it runs at 2 fps

Did a optimized debug build but it wouldn't run

Just looking for any other flags or defines in the engine to alter. I can chase it down but trying to minimize or optimize what should be there on a release build.

I have read through the books/docs but not much of explanation on order of execution for loading and unloading a game zone. Just curious if any of the engine gurus have some insight. I would like to think it's a timing issue like server being shut down before client data, ghosting, materials get cleared. That sort of thing has never been defined.

thanks for help

[Duion]

It should run smooth, you do not need a debug build for debugging with Torsion, Torsion just goes through the scripts.

Try setting a break point at the beginning, then you can see what gets executed in what order, may take a while, but you will be sure later, you should try that at least once anyway, to see how the engine loads stuff.

[Happenstance]

The new template modules might be different but in the old 'Full' template search for onCyclePauseEnd(). That gets called on the server when it's time to cycle to the next mission. From there you can trace the mission loading process - through the loadMission() calls and so on.

Your hunch about it being a timing issue is probably correct but you'll have trouble pinpointing that without running a debug build and seeing the stack trace, etc.

[Bloodknight]

Tried a debug build but it runs at 2 fps

There is something terribly wrong going on somewhere if this is happening.

[Hodo33]

I tracked the problem or closed in on it some. The crash happens here

BATCH OF DATABLOCKS DONE 8 8

ON TO NEXT PHASE

LOADING PROGRESS SET TO 1

*** Phase 2: Download Ghost Objects 8 levels/Meyrud.mis

Happened here more than once so I am thinking it's a ghost issue.

Next question is: I have 300 to 400 objects being loaded ( added a counter in script) could it be the game wants to start running before the ghosting is complete ?

As a side note I have been chasing the "cant find material for this texture" console spam so to hone in on what object was causing it I did this:

Line 373 MaterialList.cpp added in the full path so I know which directory to look in:

Con::errorf( "[MaterialList::mapMaterials] Unable to find material for texture: %s %s",mLookupPath.c_str(), mMaterialNames.c_str() );

Not sure if that spam causes any issues but I don't like red ink...

Thanks for the help...

[Happenstance]

So "phase 2" of the mission loading process happens after the client has downloaded datablocks and asks the server to send over any active ghosts. Typically this would be any objects within the client control object's view range. Crashes at this point are usually caused by a mismatched read/write order in an object's packUpdate()/unpackUpdate() methods (i.e. you're sending some data without reading it on the other end) or possibly a bad file path (maybe a missing texture, material, or shape).

Attempting to track this down without running a debug build is going to be pretty difficult. The console log can narrow things down but there's a lot that happens between "Phase 2" starting and the mission being fully loaded and ready to play.

[Hodo33]

I found the issue and fixed or hacked it here is what happens:

I am running 8 missions cycle. The game crashes downloading ghosts on the 4th or 5th mission.

In the engine netGhost.cpp line 961 we have this

// Set up a buffer for the object send.

U8 iBuffer[4096];

BitStream mStream(iBuffer, 4096);

I thought that array would be cleared with the call to clearGhostInfo but it is not. I calculated the ghost count and size and it seems I overrun the array so I changed it to 8192 and the game runs fine. So perhaps that should be a malloc/free or dmemset on each call ... not sure but changing the size of that array makes the game work without fail now.... sorry to drop bug stuff in this area and thanks for help everyone ....

[Hodo33]

Well I was wrong on that last post. It just pushed the bug out to another location. I have 8 missions and it will always crash going into number 5. If I go to main menu after 4 then re enter the game, no crash. So something is not being cleared on a zone switch. I was careful to put everything into missioncleanup.add(%x) I delete things in order, cancel all schedules and cleanup every thing I can find that might hang out there. It still crashes in the ghosting.

[Jason Campbell]

I feel you. I have BitStream PTSD. I was playing a sick game of wack-a-mole. At one point I was considering disabling the Fatal Interupt...

Anyway, I finally squashed all the bugs with the help of GDB(Gnu Debugger). Torsion might do the same thing. Can you backtrace with Torsion?

I absolutely had to view my backtrace and with GDB, I could go frame by frame of the crash.

Apperently GDB is in MinGW

https://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/gdb/

[Azaezel]

Well I was wrong on that last post. It just pushed the bug out to another location. I have 8 missions and it will always crash going into number 5. If I go to main menu after 4 then re enter the game, no crash. So something is not being cleared on a zone switch. I was careful to put everything into missioncleanup.add(%x) I delete things in order, cancel all schedules and cleanup every thing I can find that might hang out there. It still crashes in the ghosting.

 

that include https://github.com/GarageGames/Torque3D/blob/9ebebc1f5ec93d52b2b1cba6e0625a3d5e0ff327/Templates/BaseGame/game/core/clientServer/scripts/server/server.cs#L266 ? sounds very very much like you're overflowing the max object ID limit

[Hodo33]

Max objects per zone is 400. I see the onFire() function does a missioncleanup.add(BULLET) every time a gun is fired. Yet the projectile has a finite time to exist. I have 10 or so bots all shooting (300 - 500 count) so is every bullet stacking or does it self delete on life timeout? That looks like something that could cause an issue if memory is allocated for data on each bullet.

@Jason so far it never has crashed in debug. The one consistent time it crashes is trying to ghost objects at a zone transition.

[Hodo33]

Quick question I was tracking down this

netGhost.cpp line 973 AssertFatal((mGhostArray[j]->flags & GhostInfo::ScopeAlways) != 0 << game will crash here

I see where most objects have this as part of the constructor first line:

mNetFlags.set(Ghostable | ScopeAlways);

Looking around I don't see that the vehicles have this set in the constructor. I added it in but wont compile as mNetFlags is not part of the wheeledvehicle

What include file do I need to make this vehicle ghostable? and why wouldn't all objects be ghostable by default ?

Thanks

[Bloodknight]

That assert is for ScopeAlways which is a permanent scoping from the server to client, does it still break if you specify just ghostable?

[Hodo33]

@Bloodknight that worked... no crash. I couldn't figure how to do it exactly

AssertFatal((mGhostArray[j]->flags & GhostInfo::ScopeAlways) << GhostInfo does not compile setting this to Ghostable so I looked up the flag for it and just hard coded the number like this

AssertFatal((mGhostArray[j]->flags & 256) != 0

And you dont have to tell me that is a bad way to do things I know. Just can't figure how to just make it work the nice c++ way.

NetObject::NetFlags::Ghostable is not part of the mGhostArray so any hint there would be great.

Thanks for the help...

[Happenstance]

Those asserts suggest some higher level issue going on in your code so be careful about changing them, you might just be hiding the issue until it blows up somewhere else.

Getting an assert @ 973 in netGhost.cpp means you have a ghost that shouldn't be in scope that's somehow ended up in the ScopeAlways set. I'm not sure how that would happen but you might try compiling with TORQUE_DEBUG_NET defined and see what info you get.

And if you're constantly seeing crashes every time you enter a "level change" trigger, you might want to post the trigger code. My hunch is this is happening because some object isn't being cleaned up correctly during the level change.

[Azaezel]

Do keep in mind there's 2 ID lists. server and client. keeps folks form doing things like %yourIDonthierSystem.damage(9001); so does your new area cleanup include a variation on https://github.com/GarageGames/Torque3D/search?q=clientmissioncleanup&unscoped_q=clientmissioncleanup ?

[Hodo33]

I had made one change in hovervehicle.cpp here some time ago

// Todo: ScopeAlways?

mNetFlags.set(Ghostable); << had this set to scopealways, changed it back. I do use the hovervehicle for a helicopter.

Which brings up a question about vehicles in general, they are not scoped ?

In the vehicle creation in my code I was calling onAdd() on each of the vehicles I made, and discovered the vehicle.cs also did an onAdd so each vehicle was hitting an onAdd twice via Parent:;onadd so I changed that.

I wanted to be sure it wasn't some object I was adding to the scene so I added this at line 970:

// Iterate through the scope always objects...

for (j = mGhostZeroUpdateIndex - 1; j >= 0; j--)

{

Con::printf("Ghost %d %s",j,mGhostArray[j]->obj->getClassName());

What this did was allow me to see each object from the mission file get loaded ( but it doesn't show vehicles )

After these changes we got through the game couple times with no crash.

@Happenstance I did add the TORQUE_DEBUG_NET define and now I get this in the log file. Mostly around mount and dismount and on death Game crashed shortly after this line.

PlayerData::doDismount(331, Kublah, 1)

packetDataChecksum disagree!

packetDataChecksum disagree!

@Azaezel yes all that code is there.

I use the same code you would do for a dm cycle

I call cancel all game schedules then endMission() in gamecore.cs then delay 1 to 5 seconds before a call to loadMission(x.mis) and it fails on the load in ghost loading

Edited January 31, 2019 by Hodo33

[Azaezel]

should be getting scoped via parental inheritance chains, yes.

EDIT: hrm... yeah... https://github.com/GarageGames/Torque3D/blob/561f010f2e6411d8253d23f0cfcff794e81f60bf/Templates/Full/game/core/scripts/server/missionLoad.cs#L160-L162 would indeed suggest that that end should be getting garbage-collected... very strange...

EDIT: https://github.com/GarageGames/Torque3D/blob/5b8f316d804e4df2cf2b4748e5a64e6b44353204/Engine/source/T3D/gameBase/gameConnection.cpp#L1478 for the packetdata bit seems like it may be tied to the client to server *keybind sends...*

are you freeing the controlobject before removing the simobjectgroups?

Edited January 31, 2019 by Azaezel

[Bloodknight]

I wonder if this is a pack/unpack update issue somewhere? last checked it was an insidious little error that could and frequently does mask itself in all kinds of errant behaviour.

ghostable makes things scopable, scopeAllways forces the server to always ghost that object to the client regardless, which is an override that should be reserved for special things, if you already know all this then feel free to ignore my rambling.

[Hodo33]

Lot more stable now. I had my own truck::onAdd() and left the parent::onAdd()

changed the truck to not call the one in vehicle.cs

I had changed some files and the new version I got from steam still had the old garbage files so did a complete uninstall, delete the directory and get new.

Found a couple of schedules running, I cancel those before calling endMission() and added some delay between the final part of endMission and first part of loadMission(next) seems one of the testers has a SSD and that fixed his issue.

Thanks to everyone that helped on this, I'm sure I'll crash it again.