JeffR

Hey everyone!

It's my pleasure to be able to announce that version 3.9 is now released! This has been quite a big release, with a whopping 142 merged PR's totaling to 515 commits, which is more than DOUBLE what went into 3.8. A herculean effort on everyone's part for sure!

Here's a list of all the fine people that helped make this release possible:

• Azaezel
  
• rextimmy
  
• John3
  
• Bradan
  
• pacomont
  
• dakk
  
• blackwc
  
• jamesu
  
• ghost
  
• irei1as
  
• JeffProgrammer
  
• OTHGMars
  
• Lopuska
  
• Alex-doc
  
• TheDushan
  
• RoundedIcon
  
• camporter
  
• rasteron
  
• Bloodknight
  
• Ragora
  
• ZOD2015

And still more of you that helped file issues and test. A huge, huge thanks to all of you!

Several big things to note with this release is the full and proper roll-over to deferred rendering, the initial implementation of the entity/component stuffs, switching over to epoxy for OpenGL handling and DirectX 11 integration.

Yeah, yeah, where can I get it?

As always, you can nab it from the wiki's download page, found here.

So, next is 4.0, right

A capital question, question text! Indeed, the next major release will be 4.0. If you've been chumming around on the forums or in chat, you may have gotten hints that 4.0 is slated to be a pretty big deal, and I'm more than happy to intimate why.

The big aspect of 4.0 is that it's going to be the build where we're OK with breaking backwards compatibility in order to get stuff updated. For the released up until now, we've put a fair bit of effort into avoiding huge breaking changes because, well, that sucks to have to rework your project when that kinda stuff hits. However, it also means that large improvements to the understructure, rendering, art pipeline, etc can't be implemented as those will, naturally, impact and probably break existing setups.

So, we've been holding back on quite a number of big things for 4.0, and now that 3.9 is out, we can charge full-steam ahead to get all this stuff in there are really bring T3D back to where it should be.

So, here's a main features list that we're looking at bringing into Torque for 4.0:

• Physically-Based Rendering.
  
  This is, unsurprisingly a pretty big one. Luckily, it's also most of the way done! The only main piece that needs finishing is reflection probes, and the core features are in. From there it would just be a matter of dialing in the supporting art pipeline and tweaking the math to make it look as good as possible.
  
• MacOS support.
  
  This was originally slated for 3.9, if you look at the 3.8 release blog, but a lack of machines to test on, and the age-addled old platform code made it impractical to really tackle initially. The good news is, near the end of 3.9's development, with SDL and epoxy to take a LOT of the workload off, huge strides were made and it's actually almost completed. There's still some issues with rendering on some machines and catching errors and warnings that crop up, but it's looking very practical to have MacOS as an official platform for 4.0! You can see Timmy's screenshot in the 3.9 RC thread showing off T3D running on MacOS with his and JeffH's work now.
  
• SDL as the main platform layer.
  
  Torque3D's supported SDL for quite a while now, and it's naturally the main platform layer for Linux(and MacOS). However, it's been a little bit of a red-headed stepchild to the Win32 platform code. For 4.0, that's changing and SDL will be adopted as the core platform layer with whatever glue needed sitting atop it. This will drastically simplify the platform code and make it easier to maintain. It also offers a good bit of future proofing should any other platforms come into the equation down the line. SDL's stuff is very nearly at parity with the Windows side, it's pretty much just a matter of jumping over the last bits and doing cleanup/bugfixes to make it rock-solid.
  
• New Project Manager
  
  The old project manager was useful, but was hacked together with QT and php, so maintaining it was a little sketchy and fell by the wayside for the more standard CMake gui. However, there's lots of management things CMake can't do, which is why the Project Manager will be making a comeback, it'll just be a graphical frontend sitting upon the more standard cmake project generation. This new PM will let you manage your updates, and hopefully in the future hook into online repositories for easier grabbing and integration of updates as well as fetching content packages without hassle.
  
• New Shadergen
  
  As great a job as the current workhorse does of generating shaders for all the many materials you need to make a game, updating and expanding it has proven to be a pretty annoying thing to deal with due to how it's structured. Excellent peice of tech, but a chore to maintain, just like ye olde German tanks in WW2. As such, we'll be looking into restructuring shadergen to not only make it easier to update and expand upon in the backend-sense, but also easier to build materials/shaders for the end user as well, up to, and probably including a node-based approach for engineering your fancy shaders for your fancy materials.
  
• Graphics API refinements
  
  DirectX 9 has proven to be a monstrous workhorse of the graphical APIs, reliably serving well beyond what anyone though it would. However, while it has proven to be a rock-solid API, the time has come to retire the poor girl and send her off to the glue facto-eeeer, the farm. Yes. As such, DirectX 11 will be taking over as the main Dx rendering API, and will continue to see refinements to bring it up to speed now that Dx9 isn't hampering it or OpenGL. Timmy's been poking at this and has noted improved performance in Dx11 by quite a lot, and some gains in OpenGL as well.
  
• Threads. Threads for days
  
  One major blocker to rendering performance that even dropping Dx9 won't help is that we curretly don't thread or do much to optimize out the render calls themselves. That'll change in 4.0. We've been brainstorming the optimal approach for a few months now and have a pretty solid plan of attack that will see the render calls threaded out, and also proper batching of rendered geometry where appropriate. So fewer, faster drawcalls for the same workload. We'll also be looking at threads for handling resource loading and spawning of objects to cut out all those hitches when the map starts or stuff is created.
  
• Hardware Skinning
  
  Another thing that's been pretty much complete outside a few oddball behaviors we'll be getting in there is Hardware Skinning. When it's behaving, the current implementation already notes a huge improvement in performance for animated meshes, so getting it polished up and in should be a huge, immediate performance boon.
  
• Physics API
  
  T3D's had a physics abstraction layer for a long time now, and it's been pretty useful in letting the end user decide if they wanted PhysX or Bullet. However, Torque's stock physics has still served a useful niche in being network-reliable, or lightweight for basic physics mechanics, and some people find it to be a pretty workable thing. As such, the Torque physics will be converted over into a Physics plugin, so all physics and collisions will go through the Physics API and standardize all the behavior for that across the engine, cleaning up quite a lot and making it easier to maintain.
  
• Entities, Components, and Assets
  
  This is the big one from non-rendering side. 3.9 already has the initial implementation of the entity/component stuff, as well as the Assets/Modules systems, but they're not yet fully utilized. That'll change in 4.0. If you've been checking out my work blog with the recent updates, I've been covering work I've been doing on the improved asset pipeline, as well as continued work with the Entity/Component stuff. For 4.0, this will become the standard, and all existing gameplay classes(Vehicle, Player, Item, Shapebase) will be replaced with GameObjects built of entities and components to do the same work. The idea is to give users a similar base to what is in Torque now in terms of starting objects, but remove all the bulky, hardcoded functionality. It should be a standard point, not an anchor, after all. Tying to that, we also will have:
  
• Assimp support
  
  This was actually mostly done back in the day, just some issues with animation stuffs and a few other minor problems. I'd poked at this with some R&D time a while back, so it shouldn't take much tweaking to get it polished up and hooked into the assets system, which will allow quite a few new 3d model formats, including FBX.
  
• A new base template
  
  This is also basically complete. This will be a much more streamlined starting template intended to easily drop in modules, assets and the like into to build up your game project, as opposed to having to spend time stripping out the stuff you don't need. Numerous ancillary improvements also help load times, easier to read and comprehend code and the like. This will replace the empty and full templates, cutting down on the effort needed to maintain(no need to duplicate changes).
  
• Vive support
  
  This comes from our own Mango's efforts. It's pretty much done, but didn't quite squeak in for 3.9. It will be going into 4.0 for sure.

We're also looking at possible C# support, based on Lukas's fine work.

So, the main takeaway from this is that the major goals for 4.0 are:

A cleaned up codebase.

Improved rendering and performance.

Easier to use and expand - Source code should be for rendering, optimization and fancy stuff, it shouldn't be REQUIRED for the prototyping phase.

As always, if you note any issues, please make an issue on the github page, and if you have anything you'd like to contribute or suggestions of things you'd like to see, by all means hit us up on the forums, we'd love to hear feedback!

Here's the release notes for 3.9

Have a good one, and happy developing!

So I tried running through that first part 4 times last night and couldn't run across a freeze, and that was on maxed settings with steam running in the background, same as before. So it's a bit of a flakey one.

In the meantime, your dude that's been helping you test, can you ask him if he notes any difference in behavior if he disables JUST the shadows in his preferences( $pref::Shadows::disable = 1 )?

When I triggered it, I merely had Steam running in the background. Didn't seem to trigger if steam was completely closed out.

The missus hijacked the AMD machine to game all day, so I didn't get a chance to test any further today. I'll get back at it tomorrow and let you know if I can narrow 'er down any further.

Ah, cool.

Yeah, if it's particular to the lighting, there's a few things I can have a look at, such as manually disabling certain things via pref settings(like shadows) to narrow 'er down further. I'll poke at it some more this evening.

Yeah, it was fine on low lighting + steam, or high lighting + no steam.

What version of T3D are you guys on?

Ok, did a bit more iterating, and it seems like the hangup occurs if I have steam running. When I had it running, I'd make a character, click through the first screen(to open the door) and it'd lock up.

I killed steam, re-did it and it seemed to progress through OK. So something's gotta be going squirrely with the recent drivers and the steam integration.

Ooh, tried it again, and got it to happen this time. I had put it to fullscreen, 1900x1080 on the resolution with steam running in the background(apparently it wasn't before).

Right, so I'll keep iterating on this, see if I can narrow it down.

K, grabbed the latest drivers, went through the same steps, and was able to walk around the town, no hangups no problems, I did skip the tutorial, if we think that'd matter. So I don't think its the driver, at least.

From here, we'd wanna try and narrow things down, such as a particular set of replication steps, so lemme know if you think I missed anything in my little routine above, or if you know of any particular things your users tried and I'll iterate through them.

'Aight, fired up the final version, bumped up every setting except vertical sync to max, started a new game, created a character, talked to the inn kepper, slept for the night and then was able to go outside and walk around the town, didn't note any issues.

The machine's a R9 200-series, drivers currently at 14.501.1003.0, so it's behind. But so far everything looks alright. I'll try updating the drivers and have another go of it and see if it crops up.

'Aight, cool

Thanks, I'll get 'er set up tonight and check if I can replicate it at all.

Another quick question, do you just use Dx? Or do you support OGL as well? And if you do, any idea if it's behaving the same way for your users?

Either should let me confirm the issue, but if you have a development build that'd let me use a debugger.

Hmm, do note that the 16.3.x drivers are mentioned several times, but that is a wide spread of cards.

I have access to an AMD machine, is there any chance you have a test build you could email me so I can try it on my end and see if I spot anything when I run it?

Sounds like a tricky issue, but I'd be glad to help out!

Since you mention lighting-related, just to clarify, are the users able to get to the main menu and stuff even with advanced lighting on and set to high?

Also, any chance you can list some of the cards you're seeing this on? Or is it a total random smorgasboard of radeon/AMD cards?

What @LukasPJ said. There's no need for this to be an Us vs Them debate. We can easily point at a thing and go 'Hey that's neato' without it becoming some big deal.

@andrewmac You mentioned 'rather than discussing area lights', but I don't think anyone's linked that paper TO discuss it. I believe that's part of the problem, here. There's not much discussion to be had as-is. The video looks nice, but there's not much to critique, and no one had posted stuff relating to the techniques used in it's production such as that area light paper, so currently there's almost no discussion to be had.

Edit: Right, thread was locked, so no one'll be able to post the whitepaper link, haha. Will say that, in the future, I think threads like these would be better served if someone offers discussion points or tech speculation. There's only so much that can be said about how generally neat a video is.

So to thread starters and posters in general, a good attitude to approach from is "How did they do X", or "Are there any papers/videos/code for doing Y", and "What would be needed to replicate this look". These are good for discussion, and get to the heart of what we all ACTUALLY want to discuss, rather than sitting around cooing over a video. Leave that to the youtube comments section :P

As Az said, the structure with PBR isn't COMPLETELY different, you won't have to remake your UVs or anything crazy like that.

With PBR, you need the following data in textures:

Surface color(Albedo)

Surface normals

Metal-ness

Roughness/Smoothness

Ambient Occlusion(i believe this is technically optional, but helps a lot in ensuring the nooks and crannies on a surface are shaded right).

While you can pass in the Metal, Rough and AO in as separate grayscale images, because Az rigged up a compositor function, it's going to be easier to work with if you combine all 3 into a single image and just tell the PBR mat to use that. Again, as Az said, Red channel is smoothness, Green is AO and Blue is metal.

Once you have that set up, you just need a cubemap. Level-wide will work, though we're trying to wrap up an IBL(Image-based lighting)-style system that uses area probes to provide regional ambient lighting info. You'd place a probe in the map, scale it's area of influence(if the pixels all inside it, it's cubemap is supplied for the PBR calculations) and then you'd either manually set a cubemap, or let it bake one from the probe's point of view. Idea then being that you sprinkle them around your level so that the lighting on objects is properly consistent to their surroundings.

For making the various maps, and assembling the composite texture, we highly recommend you nab Allegorithmic's Substance tools which you can nab from here. Their Substance Live bundle has 3 tools that are super useful for cranking out PBR materials and they use a rent-to-own system where you pay $20 a month until you pay off the license, and then it's yours forever. So it's a pretty good deal for such good tools. If not, just putting them into the right channels via any image editing software will work fine as well.

We'll also be fleshing out the process of making and converting images and materials over to PBR on the wiki to make all this easier to keep on top of :)

I had started a wiki page for stuff to test soas to be consistent and cover most all bases here: http://wiki.torque3d.org/testing:pre-release-checklist

This would be a good starting point. Obviously it's not complete yet, but it gives a good list of stuff to check-over, and is applicable to all platforms.

On the demo, it, much like other movie demos unity and unreal have put out in the past are definitely pretty neat. Good demonstration of what some obscenely talented artists can do with the platforms. Not as great for showing 'this is what games could be like', because not needing to do physics, AI and other gameplay stuff frees up a lot of processing power to further funnel into graphics.

 

But as I mentioned before this was not the intention of the video. It's not an advertisement for Unity's gaming capabilities. It's their answer to Unreal's Kite video and the other graphical demonstrations. This is a quote from the Unity website for Adam:

 

Adam is a short film created with the Unity game engine and rendered in real time. It’s built to showcase and test out the graphical quality achievable with Unity in 2016.

 

People are faulting it for goals it never had. The real discussion here should be what new tech did they bring to the table that T3D doesn't have, and how could T3D be improved to reach that level of quality. Instead we've just got a bunch of people trying to explain why the video is bad. It's just.. sad :(

Yeah, more of just a broad statement on my end that I'd prefer they put out gameplay demos instead of movies. For what it is, the Adam demo is very well done, just as the Kite demo is.

@andrewmac

They're releasing it soon so everyone can run it on their own machine.

 

I wonder why they have not released it already. It would certainly give them a lot more advertising if they did.

Could it be perhaps that not all is as it seems? :? I guess we will have to wait for that release and see lol

 

I think it has more to do with upcoming SIGGRAPH 2016 than some kind of nefarious reasoning. Theres a huge computers graphics conference called SIGGRAPH at the end of the month and I bet they're holding off to release it as part of the presentation there. Eric Heitz and friends will be presenting the area light stuff, I imagine unity will have some kind of presentation that showcases Adam and follows it with the release. Of course this is all speculation.

It's disappointing this community isn't more supportive of things going on outside of it. It seems anything that isn't pro-torque is immediately met with defensive responses, like its somehow an insult to Torque. It's a shame :( we're supposed to be computer scientists not fanboys.

Hopefully they have more papers and stuff than the area light one they already pushed out. Would be kinda lame if they just re-hashed their presentation.

On the demo, it, much like other movie demos unity and unreal have put out in the past are definitely pretty neat. Good demonstration of what some obscenely talented artists can do with the platforms. Not as great for showing 'this is what games could be like', because not needing to do physics, AI and other gameplay stuff frees up a lot of processing power to further funnel into graphics. Would be nice to see them funnel their art team to a small gameplay demo or something instead for a better game benchmark.

Heh, reminds me with Square Enix's Luminous engine demonstration a few years back that looked incredible, and then people found out it was running on 3 or 4 titan x's to achieve realtime.

I'm sorry to hear that.

Could I make a suggestion for the steering committee?

Instead of pushing the engine even further ahead in 4.0 and going for PBR which is ahead of the curve but maybe a little too far ahead of the curve, why not put all our resources into making the engine cross-platform compatible which will do more than anything else to bring it up to date and increase the user base?

It's been tried several times now and failed, how about putting all our resources into it and putting it on an equal footing with the other game engines and making a Mac, Linux and Android version before we go any further ahead?

I'd be keen on mac support happening, and lots of stuff has had the tangental benefit of making that easier going forward, stuff like epoxy helps a lot for making sure systems with flakey or out of date drivers work(such as osx or some units distros), and SDL shaves off a lot of the work in dealing with the platform layer work.

The big thing holding back mac being a minor drop-in effort is Apple putting very minimal effort into keeping osx working with modern OGL, which isn't something us full-in'ing on can change. We can work around odd osx behaviors, but we can't really fix them.

That said, the last attempt that was made a few months back showed it's pretty close, I recall some odd graphical behavior and some platform oddities with SDL, but it did compile and run. As we continue to refine the OGL gfx layer, and switch over to SDL for the platform core, it should be much easier to double-back around and have a bit of slap-around on mac ;)

Stuff like PBR - which is almost completed already and the entity/component work don't negatively impact osx other than it potentially coming about a bit slower, and stuff like e/c and PBR help a lot with standardizing art and gameplay paths, which do benefit the mac dudes too when we get that end working.

Another limiter that is slowly being resolved is anyone with the code know-how having access to macs which almost no-one did in prior attempts. I've got access to one from my work, and a few other people are getting them for testing and all that jazz. That means that when we look at it again, there's a much higher number of eyes on it so testing and fixing can occur much faster.

So yeah, I can promise you it'll be looked at and we'll make an effort to get mac support added proper - chances are better now than ever before - but I disagree that we need to stall on other fronts to get there, and quite a bit of work, if indirect, has already been pushed to making it workable going forward. :)

Yeah, development branch is the 'work in progress and could have issues', but we try to keep it pretty stable and not break stuff if at all possible, so having a fairly up to date development binary should be MOSTLY safe :P

A minor bit of clarity of it being from the WIP development branch would be a good idea in general though, yeah just so no one confuses themselves.

Glad to hear! :D

I merged in some improvements to the mounting a week or two back, so if Johzx's binaries are up to date, it should include those improvements.

You'll still see some jittery behavior if you have a chain of mounted objects, like a scope mounted onto a gun mounted onto a vehicle, and when the vehicle moves, the scope'll lag behind some, but the improvements helped with at least the first-step mounted objects to keep up better(and allow most classes to do it, even the TSStatics).

So I'd say check with Johzx on if his precompiled binary is up-to-date and give it a whirl with the mounting improvements in there and lemme know if that helps.

How's the 3.9 final release coming along?

Can I put in a request for out of the box Mac support?

Some of our users aren't gamers and they have Macs.

Johxz is correct, shooting for this weekend, and if any snags pop up, it'd be next week. Really wanting to have it out for the weekend though.

As for osx, we'd done some work into that a little while back, and while it worked, it was kinda in a weird place stability-wise. JeffH was looking at beating it all up and getting cocoa in there for a stable layer. He had been doing the work here: https://github.com/JeffProgrammer/Torque3D/tree/macosx_cocoa

If you're able to help or help test, that'd definitely speed up that side of things. Part of the problem is not everyone has a mac to easily develop/test on, so if you can help on that front it'd be quite the boon. I'd suggest trying to hit him up if you can help.

It supports DX9 and 11.

11 support is solid, but not fully optimized(not taking advantage of command buffers and all that fun stuff).

Once 3.9 is released(should be this weekend-ish), we'll be working at dropping DX9 and going full-in on optimizing DX11. Work is already being done on that front and DX11 has seen a good bit of performance gain over 9 already, so for 4.0 it should be solid.