Rendering freeze (Radeons, 64-bit Torque)

[Flashback]

Hello. We have a serious issue in our game, Age of Decadence, for quite some time, but not being an expert in rendering/hardware low-level code, I've been unable to resolve it so far. It always happens to Radeon users and affects only 64-bit build of Torque3D. It is also related to lighting/shadow rendering in some way, because the only workaround we've found so far (and recommending our players) is to change Lighting Quality setting in options to Low.

The console message at the end of the logs is repeating:

 

GFXPCD3D9Device::beginScene - Device needs to be reset, waiting on device...
GFXPCD3D9Device::beginScene - Device needs to be reset, waiting on device...
GFXPCD3D9Device::beginScene - Device needs to be reset, waiting on device...

 

This is coming from gfxPCD3D9Device.cpp, GFXPCD3D9Device::beginSceneInternal() function, where Torque apparently loses D3D device, but why this is happening remain a mystery to me.

This affects rather large part of our audience and leads to negative reviews and complaints, so we'd be really thankful for a good advice.

[JeffR]

Sounds like a tricky issue, but I'd be glad to help out!

Since you mention lighting-related, just to clarify, are the users able to get to the main menu and stuff even with advanced lighting on and set to high?

Also, any chance you can list some of the cards you're seeing this on? Or is it a total random smorgasboard of radeon/AMD cards?

[Flashback]

Hi, thanks for your reply.

 

Since you mention lighting-related, just to clarify, are the users able to get to the main menu and stuff even with advanced lighting on and set to high?

Yep, main menu loads up without problems on any settings, and even training mission (not sure, though, if it works for everyone having this problem, or just some people. Training mission is mostly empty terrain, while first "real" mission is a big town, with a lot of light sources and objects). The freeze is happening when players finish character creation (GUI dialog with GuiObjectView to show character model, covering empty level) and do switchMission(). At least one frame gets rendered, so they see the scene, then the screen freezes, while music keeps playing and console keeps spamming the message from the first post).

 

Also, any chance you can list some of the cards you're seeing this on? Or is it a total random smorgasboard of radeon/AMD cards?

I started collecting card models and driver versions at first, but then understood it's a huge variety of radeons that doesn't tell me anything useful. Here:

1)

o Card    : 'AMD Radeon HD 8760'
  o Version : '8.17.0010.1404'
  o VRAM    : 1024 MB

2)

o Card    : 'AMD Radeon HD 7700 Series'
  o Version : '8.17.0010.1429'
  o VRAM    : 1024 MB

3)

"270x ati
  radeon v16.3.1 (20-03-2016)
  ive got the 2048MB version."

4)

"I'm having the same issue... Picked the game up on Steam this weekend & played a bit... then I updated my video drivers today to version 16.3.2, I have an AMD Radeon R9 200 series, and most games purr on this at the highest settings. AoD was fine until the update; now I log in, load up a game - and the screen just freezes."

5)

"Just purchased AoD on steam this morning and I'm very excited to play but I'm getting a freeze on load after character creation.
The screen shows the opening scene with text overlay correctly (perhaps in the middle of a fade-in) but the screens and mouse will not move, and keyboard input is ignored.
No problems, however, with the training option. That loads fine.
I've updated to the latest Amd Radeon Crimson 16.3.2 drivers and software. Is that the issue?"

[JeffR]

Hmm, do note that the 16.3.x drivers are mentioned several times, but that is a wide spread of cards.

I have access to an AMD machine, is there any chance you have a test build you could email me so I can try it on my end and see if I spot anything when I run it?

[Flashback]

There is also a possibility that Radeon Crimson driver is to blame, as one of the players said that "while Omega driver was used, everything was fine". But he's not sure about that.

What kind of test build you'd prefer? Is Steam build with compiled scripts ok, or you'd like to look at development version?

[JeffR]

Either should let me confirm the issue, but if you have a development build that'd let me use a debugger.

[Bloodknight]

A quick track might be able to nail down a potential driver issue is to get release dates of drivers and the dates of player issues and do a quick compare.

Also, AMD will have a bunch of people who take pretty specific issues like this and attack them from their end, its worth making contact with somebody at AMD to see how to get hold of the devs there.

[Flashback]

Either should let me confirm the issue, but if you have a development build that'd let me use a debugger.

 

Sent a download link via PM.

 

A quick track might be able to nail down a potential driver issue is to get release dates of drivers and the dates of player issues and do a quick compare.

 

Thing is, we introduced 64-bit version relatively recently, and "buggy" drivers (if they're to blame) kicked in immediately.

 

Also, AMD will have a bunch of people who take pretty specific issues like this and attack them from their end, its worth making contact with somebody at AMD to see how to get hold of the devs there.

 

I thought of this, but don't really know people to talk to. I guess I can try bothering usual tech support, maybe the will pass my message on to developers.

[JeffR]

Thanks, I'll get 'er set up tonight and check if I can replicate it at all.

Another quick question, do you just use Dx? Or do you support OGL as well? And if you do, any idea if it's behaving the same way for your users?

[Flashback]

DirectX only, so no idea how it would behave on OpenGL.

[JeffR]

'Aight, cool

[JeffR]

'Aight, fired up the final version, bumped up every setting except vertical sync to max, started a new game, created a character, talked to the inn kepper, slept for the night and then was able to go outside and walk around the town, didn't note any issues.

The machine's a R9 200-series, drivers currently at 14.501.1003.0, so it's behind. But so far everything looks alright. I'll try updating the drivers and have another go of it and see if it crops up.

[JeffR]

K, grabbed the latest drivers, went through the same steps, and was able to walk around the town, no hangups no problems, I did skip the tutorial, if we think that'd matter. So I don't think its the driver, at least.

From here, we'd wanna try and narrow things down, such as a particular set of replication steps, so lemme know if you think I missed anything in my little routine above, or if you know of any particular things your users tried and I'll iterate through them.

[JeffR]

Ooh, tried it again, and got it to happen this time. I had put it to fullscreen, 1900x1080 on the resolution with steam running in the background(apparently it wasn't before).

Right, so I'll keep iterating on this, see if I can narrow it down.

[JeffR]

Ok, did a bit more iterating, and it seems like the hangup occurs if I have steam running. When I had it running, I'd make a character, click through the first screen(to open the door) and it'd lock up.

I killed steam, re-did it and it seemed to progress through OK. So something's gotta be going squirrely with the recent drivers and the steam integration.

[Flashback]

...plus it's somehow related to Lighting Quality setting in Torque. I assume if you set that to Low, it's going to work without freezing.

Thanks for investigation, Jeff. I'm going to approach both AMD and Valve about this. But since there is no guarantee they will do something about it or even reply, it would be good to investigate it on Torque3D rendering end. Valve usually replies that "if Steam overlay is causing problems for you, it's most likely an issue in your rendering code".

We do have an old Radeon at work, but it seems to be considered now a "legacy product" by AMD and problematic Crimson drivers do not support it.

[JeffR]

Yeah, it was fine on low lighting + steam, or high lighting + no steam.

What version of T3D are you guys on?

[Flashback]

Updated to T3D 3.8 on Dec 9, 2015. We've accumulated a lot of modifications regarding gameplay objects and such through the years, but since we started with TGE in 2005, I always tried to stay away from rendering code.

[JeffR]

Ah, cool.

Yeah, if it's particular to the lighting, there's a few things I can have a look at, such as manually disabling certain things via pref settings(like shadows) to narrow 'er down further. I'll poke at it some more this evening.

[Flashback]

Emailed AMD support on Friday, hopefully they'll answer something.

Also, one of our player who helps to test the issue says that the game freezes with or without Steam overlay, and the last drivers version working with T3D is 14.12.

[JeffR]

When I triggered it, I merely had Steam running in the background. Didn't seem to trigger if steam was completely closed out.

The missus hijacked the AMD machine to game all day, so I didn't get a chance to test any further today. I'll get back at it tomorrow and let you know if I can narrow 'er down any further.

[JeffR]

So I tried running through that first part 4 times last night and couldn't run across a freeze, and that was on maxed settings with steam running in the background, same as before. So it's a bit of a flakey one.

In the meantime, your dude that's been helping you test, can you ask him if he notes any difference in behavior if he disables JUST the shadows in his preferences( $pref::Shadows::disable = 1 )?

[Flashback]

Strange, for everyone else the issue is very consistent, rendering the game unplayable. But if it's something about Radeon drivers, incorrect handling of 64-bit pointers and such, it could as well depend on current state of the memory.

Sure, asked my test guy to try that pref.

I also noticed earlier it's one of three prefs that change with a quality level, and I even included it in my "Radeon hack" for the latest build that autodetects lighting to Low for Radeon users.

[Flashback]

$pref::Shadows::disable = 1; <-- helps

$pref::Shadows::textureScalar = 0.5; <-- doesn't help

[JeffR]

Alright, that narrows it down quite a bit.

It means that it's not really an issue with advanced(deferred) lighting itself, but something with shadows being rendered - resolution of said shadows being a non-factor.

I'll do some more iterating(pending I can get it to kick in again) today.