Hey everyone!
So, been a while since the last workblog, but the good news is, I didn’t quite threshold past the one year mark!
So that’s… good.
Anywho. Predictably there’s a frankly preposterous amount of stuff to go over so lets get getting and dig on in!

 

What’s new?
A lot, as said. But for specifics, since the last workblog, we’ve rolled in over 150 pull requests. These range everything from bugfixes, library updates, QOL improvements and all the way up to converting pretty much every class that touches a content file into using the assets system. We even got the site updated, as I’m sure you all had noticed.

Lets cover a few of the specifics and big ones.

 

New Site
So, if you’re reading this, then you’re at least slightly aware that the site underwent some changes relatively recently. We were using a git-based pages system before which while lightweight and anyone could contribute, the turnaround time ON changes was fairly suboptimal, and because the only way to make any changes at all was via PR’s, it disincentivized people from contributing content.
Pushing tutorials or updating documentation on the site had enough of a hurdle that no one really wanted to bother.

So we did a lot of digging and landed on a fully integrated CMS platform. This allowed the forums to be integrated into the main site and allowed us to easily manage the content that was on the site itself.
Other features like broad site search and a few handy additional boons like being able to post micro-communities for games all tie together to make a better centralized point of community engagement.

And as noted prior, it drastically lowers the bar for people pitching in to update the documentation if they have at least the Contributor designation, as someone trusted within the community on helping out and know their business.
As we’re in the home stretch on 4.0(as I’ll get into more below), I’m starting to be able to split my time from that as we wait on rounds of testing to begin filling out the documentation section.
The plan is to consolidate all the disparate docs from the various sites torque’s found itself on, from the old GG docs to the old TDN, to the wiki and so on, and fold it all back here in the docs section.

This way we have a concrete point people KNOW they can find what they’re looking for, instead of having to guess what random site the info they need is on.

Major, MAJOR props to Tony and Lukas for helping get this thing where it is now. It's definitely a big deal!

 

PBR
It’s been a bit since I’ve covered PBR stuffs, but honestly not TOO much has changed since I’d last covered it, so this can be fairly quick.
The main points to note is we spent a lot of time refining and stabilizing the math so you can have pretty high confidence that whatever is exported out from your art tools like blender or substance, if they use the standard Metalness/Roughness PBR formatting, will look Tools Correct. Also, as noted previously, we’d shifted up to utilize the more industry-standard ORM Configuration map, which further emphasizes the Tools-Correct approach.
Of course, you can still use sliders or individual maps to fill in the PBR properties too. That hadn’t gone anywhere.

  

Other things were cleaning up and stabilizing probes and IBL and improving performance with them by adding some nice convenience functionality with controlling active probe counts and the like.
All in all, PBR should give stable, quick results that look quite nice.
 

Assets
Alright, now we’re getting into the meat of things. So, assets. I’ve talked about them a good bit before, so if you’ve been following my workblogs, you’ve got at least a general notion of them.
Suffice it to say, they’ve gotten a LOT of work done in the past months.
So, as of this moment, outside of single-digit numbering exceptions, every class that ingests a normal “content” file type - that is, Images, Shapes, Materials - now utilizes assets.
While a lot of work to get right, we did everything we could to ensure that the process is stable and doesn’t impact the functionality of the classes or objects themselves.
Part of that is we made some… liberal use of macros to fill in a lot of duplicated and common code pertaining to these content file types.
Whereas before each class would have to implement the entire file ingest, load and bind logic each time - which as with all duplicate and repeated code, leads to points of possible failure - they now use a common set of macros which have been aggressively and thoroughly tested, making everything a bit leaner and less prone to issues sneaking in.
In this shot, we see an example of defining our asset macro in the class header file:

And here's how you use the asset macros to set up the init persist fields to expose them to script:

There's even macros for the packing and unpacking of the network data:

And finally, an important bit, is that it doesn't just generate function info or anything, but actually configures the internal variables and the like off the parameters you pass in. This allows everything to be VERY consistent and predictable, making usage in the classes once you actually want to utilize the data set much more comprehensible, like so:

 
This means that pretty much every class has a very similar content-file flow, which allows us to standardize object fields more(less guessing if it was shapeName, or shapeFile, or shape, etc). This also standardizes accessor functions.
Want the gui’s bitmap?

%bitmap = %gui.getBitmap();

 

And you can be sure it’ll get it, no questions asked.
Part of these macros is retaining the legacy fields too, so if you load up a project and some stuff doesn’t get processed successfully via the Project Importer(will get to that here shortly), then the old fields are held onto so you can retry, or debug it.
So those accessor functions are smart enough to figure out where your bitmap data is and return it back.
Using an asset? Returns the asset info. It still using the legacy bitmap filename field? Returns that instead.
This should alleviate a good chunk of pain for existing projects that are looking to port up into 4.0 without utterly snapping all your stuff over its knee.

Ultimately, this all ends out with a standardized, consistent content flow without having to juggle different means, forms and fields depending on the object and class, which is nice.

 

Asset Browser
So, obviously, if we’re gunna be mainlining assets now, the asset workflow needs to be up to snuff as well. First up, we’ve got the Asset Browser. I’ve shown and talked about this thing before, but now it’s the primary access point for all content in your game. 
Previous, I’d gone over the improvements to searches in the AB with stuff like complex searches and collection sets. This is obviously all still in, but I’ve added some further tweaks to improve reliability here. A search isn’t much help if it doesn’t find what you need it to.
Additionally, I’d covered how you could do stuff like drag-n-drop datablocks to spawn the vehicle or the like. This too has been improved and expanded on, so that pretty much anything that is generally spawnable via the Library tab can be spawned via the Asset Browser. The plan here is to ultimately replace the Library tab with the AB as the centralized one-stop-shop for all your content needs.
To that end, most of the ‘Creator’ entries have been brought over to the AB as well, so spawning lights, probes, precipitation, skies and so on can be done via the AB too.

(Of course, the current class icons are a bit low res and need some sprucing)

Beyond that, improvements to the tooltips for when you hover over an asset to present where the asset is located in the file structure and other meta info should help tracking down where those assets you knew you had but forgot where you put. We’ll also be looking into some ‘Go To Asset’ navigation option/actions as well to further facilitate tracking down these lost sheep, er, assets.

Now, if anyone threw some big images, or a lot of shapes at the AB previously(Catographer, for example, uses like a dozen 4k images for material atlases), you’d find there’d be a load hitch every time it’d try and populate the asset previews in the AB.
Obviously, this is a minor annoyance that, when repeated enough times(which it would, now that the AB is the primary content nexus) it would become grating fast, and hurt workflow.
So I added a system where it’ll take preview images and create a scaled down version, ensuring that no preview is so large as to cause hitching, allowing fast loading of directories even if they have a crazy number of assets visible.

Whenever the AB loads, it’ll generate a preview image based off the source content - a shape or image - and generate a scaled down preview image to a given size, such as a 512x512 preview.
This keeps them reasonably sized, which keeps loading in the AB fast, but doesn’t hurt the quick-glance recognition of what you’re looking for either.
Additionally, I added logic to compare the file modified timestamps whenever the AB is opened, so if an asset’s file saw changes newer than the preview image, it’ll regenerate it. This ensure that if you recently re-exported an image or a model, the previews will always show the correct version.

 

Asset Importing
So with everything actually for-realsies using assets now, we want to ensure the importing process is as sexy-smooth as possible, so this saw improvements as well. Various sanity checks, safeguards and issue-resolution methods were added to the import configs.
For example, a new Issue Resolution step “FolderPrefix” was added. In the event of an importing asset happening to have the same name as an existing asset, like say, both have “grass” for the AssetName.
In this case, with the FolderPrefix resolution, it’ll then find the folder the asset’s file is in, and then just prefix it.
So if our conflicting importing file is in the ‘Foliage’ folder, it’d resolve the conflict via turning the asset name into Foliage_grass. It’s also smart enough to try and find non-conflicting folders in case THAT was already taken too.
Additionally, options for force-adding type suffixes onto the asset name help prevent collisions too. Like say we had a ‘Player’ model, that has a Player material and uses the Player image for the diffuse.
These 3 would normally conflict, being all named player.
With the options on to add the type suffixes, though, they become Player_shape, Player_mat and Player_image.
This prevents collisions AND becomes easier to search for/spot at a glance which-is-which.
So several birds with one stone there!

And remember, this doesn’t affect the actual file the asset uses, just the assetName the engine builds for the AssetId, so you don’t have to worry about it doing spooky rename voodoo on your png file so you can’t find it again later.

 

Asset Loading/Error Tracking
A big addition that Az pushed for was better tracking of the explicit load status of an asset. Before, you effectively had two states for an asset. It either loaded, or it didn’t. Obviously, depending on what’s going on, this could be wildly unhelpful. For example, if the asset definition loaded ok, but something was wrong with the associated file(like lets say the model file was corrupted), we can report the error while still continuing to load the asset, instead presenting a fallback shape.

This avoids the old situation where if a model failed to load on a TSStatic, the TSStatic just wholesale doesn’t load, disappearing from your level and some point later down the line you realize a bunch of your objects are missing.
Beyond that, naturally having more specific loading error codes lets you pin down exactly what’s wrong. From bad file references, to invalid formats, being able to note the problem is the first step to fixing it, which can save on development time without the engine just throwing its hands up and failing to load everything.

 

Modules
Also important to assets are the modules as well. Per previous updates, we’ve been refining how modules handle loading of scripts that may overlap or override via the new queueExec() function.
We’ve been shoring up problems that’d come up and dialing it in more, but overall modules haven’t seen any big functionality shifts, which is good, as that means even with people wrenching on it, it’s proven to be rather stable.

 

Legacy Project Importer
So this has been a big one. When we were implementing the conversion process of existing projects to work with 4.0(since obviously there’s a number of people with existing projects that would be keen just update update to latest to get the sweet new benefits) we originally tried a sort of “in line” import process.
As part of the above mentioned asset macros, we would integrate in the original filename fields, that when a file is fed into, would run the Asset Importer in-place, get the results, and update the field definitions.
And in a lot of cases, this worked!
The problem was there was also a lot of cases where we started seeing some really weird and funky behavior, timing issues, order of operations shenanigans, and needing to close and open things multiple times to process everything.
The sexy, seamless system unfortunately didn’t play out quite as smoothly as the theory indicated.
So, I opted to pivot on it and build it as a dedicated importer process. There were several advantages to this approach. Sure, we lost the ‘pure automagic’ of the inline approach, but you could start up a new 4.0 project, open the importer, point at your old game, and it’d just import everything in and process it into a valid module for you.
So maybe not PURE automagic, but like, 56.5% automagic.
So I did the initial pass of it and with Az helping crunch a lot of the logical refinements, we got it to a spot where we could get others throwing their projects or old content kits at it. While there’s still some bugs to be fully chased down yet, the results are looking really good, and should make it MUCH less painful for anyone and everyone with an existing torque game(in theory all the way back to TGE games, though that’s not a dedicated focus at the moment) can get rolled over to 4.0 in an afternoon.

Nice, How’s It Work?
As mentioned, the process involves you pointing at your old torque project, and getting a nice, 4.0 compliant module in your new 4.0 project as a result.
But lets get into the specifics of it.
This:

Is the Project Importer. It’s formatted as a classical wizard utility that walks one through the process in pretty plain terms.
It’s pretty straightforward, but we can do a quick overview of how it works here. If you continue past the welcome page, you select what version you're importing from(though currently everything is pre-4.0, this gives us options to have by-version import rules for people upgrading from 4.0 to 4.1 and so on later) it goes on to prompt you about what content you’re going to process in with the importer, as well as it’s location, as seen here:

Depending on which you pick dictates some supplemental behavior. As you can see your options are:

• A folder that is inside the current project already - useful for re-importing something if required
• A folder that is outside the current project. This is the normal option and would be what you pick to select your old project’s main directory
• The Core and Tools folder. This is primarily for getting the BaseGame template up to speed, but if you’ve got some custom tools or the like you’ve copied into place already, you could run that.

Normally, this is where you would select the Outside folder option. Then you click the locate button, and navigate to your project you want to import. Once you’ve done that, you can continue.

It’ll next prompt what the new module being created will be called. This is defaulted to whatever folder you pointed to’s name, but it can be named anything like ‘MyCoolGame’. 

Continuing  on will see the files be copied into the new module, then processed. It’ll run all content files(shapes, images, etc) through the Asset Importer, and then it’ll process all editable/code files looking for legacy fields.
If it finds one, it’ll then convert that field into the associated asset field, and find the new asset associated to the filepath it had originally. So you go from something like this:

Bitmap = “./background.png”;

To:

bitmapAsset = “MyCoolGame:background_image”;

 

It’ll only process lines and objects that actually can successfully convert, so if a file doesn’t import and generate an asset, the legacy line will stay as-is. You can re-run the importer again on the existing module if needbe, but tests have so far proven it’s generally a one-and-done process.

Once the import process has completed, the only real thing left is to integrate in the scripts to the module. You’d crack open the module script file, and then slap in your exec calls for the client, server, etc in the appropriate place and there ya go. Your project’s pretty well converted up into a 4.0 module!
In example, this we can see the FPSGameplay content imported in as a module now here:

Naturally, we’ll be fully fleshing out the process and details of this in the documentation, but that’s the broadstrokes. Currently it’s part of the tools suite, but the main plan is it being part of the new project manager, so you can create a project, launch the LPI, and convert in an old project into the 4.0 one all in rapid succession in the PM.

 

Terrain Improvements
Up next, going to do a quick go-over on the terrain updates. No major overhauls here, but we did bring in Lukas’ improvements to the terrain material blending, as well as toggles for if it should utilize the height data in the terrain materials to blend materials or not, as seen here:

This naturally allows not only nicer looking terrain materials blending, but the toggle gives more control if it fits your style.

Additionally, Lukas did some nice wizardry to streamline the rendering of the terrain a lot, utilizing texture arrays to remove a LOT of the drawcalls and render overhead that came with the older way we rendered terrain. So looks better and runs better. Not too shabby

 

TScript Conversion
This one has been discussed for quite a while, but we went ahead and pulled the trigger on it a little while back. Namely, shifting of the default torquescript extension to ‘tscript’ from ‘cs’. A number of reasons for this, but the big ones being that a lot of tools treat cs as a C# extension, which can have some weird results when dealing with code IDEs.
Another reason tying into that was the expansion of the cinterface and support for more script languages Lukas spearheaded, namely, C#.
With that soon to become a fully viable possible route for people’s T3D games, dealing with confusing extension mix-n-matches - especially for project generation and asset/module installs not being able to tell a torquescript and a c# version apart - all lead to the lock-in on swapping the default extension.

That said, we know people liked having that cs extension, so while the default may have changed, we added a new project setting for CMAKE as well as a global var that sets the extension used. So if you wanted to keep with cs, you can just change that in your project settings and presto!

And to better facilitate the extension swapping, we went in and adjusted a number of filename handlings to check for both the default and the ‘other options’. Meaning that doing an exec() call can just take the path and name, leaving off the file extension and the engine’ll figure it out. Additionally, we added the isScriptFile() console function to specifically check valid script extensions if you're trying to test if a filename is a valid script file, rather than trying to bruteforce your way through with isFile().
Further, If you wanted to go ahead and use the tscript extension and were importing in an old project, fear not, because the Project Importer will check what the extension global variable is, and update any scripts to utilize that extension instead(including filename references in files, in cases where it doesn't just drop the extension as unneeded entirely);

So while it is a change and I know some of you may not be fond of it, we’ve done everything possible to make it painless to swap, or just keep to the classical if you so desire.

 

TorqueScript Interpreter Update
While not merged in juuuuust yet, we’re in the final testing pass for some big updates to the TS Interpreter(lovingly dubbed as ‘TSNeo’). Hutch has been doing some solid work not only simplifying the interpreter, making it easier to maintain and expand, but also some biiiig performance gains. How big? Lets have a gander.
He drafted up a number of tests to check perf of various things, from crunching math right in TS, to handling objects and variables, to invoking up into the engine via console methods. All metrics are for his i9-9900K @ 4.8ghz. All numbers are in milliseconds.

So here’s stock:

Quote

Local Variable Benchmark in milliseconds: 61, 64, 61, 61, 62
Function Call Overhead Benchmark in milliseconds: 398, 398, 397, 398, 403
Function Call Overhead 3 Parameter Benchmark in milliseconds: 1489, 1487, 1492, 1492, 1489
Method Call Overhead blah Benchmark in milliseconds: 967, 965, 963, 964, 965
Method Call Overhead Parameter blah2 Benchmark in milliseconds: 2829, 2244, 2636, 2662, 2669
Looping over Large Collection of Objects Benchmark in milliseconds: 88, 85, 85, 84, 84
String Pressure Benchmark in milliseconds: 226, 213, 213, 213, 214
Collatz Conjecture Benchmark in milliseconds: 829, 824, 827, 824, 822
Fibonacci Benchmark in milliseconds: 1245, 1248, 1245, 1244, 1245

As you can see, some stuff isn’t too bad, but function calls in particular are really painful.
Now let’s look at the TSNeo benchmarks:

Quote

Local Variable Benchmark in milliseconds: 20, 19, 22, 19, 20
Function Call Overhead Benchmark in milliseconds: 48, 48, 48, 47, 48
Function Call Overhead 3 Parameter Benchmark in milliseconds: 77, 79, 75, 75, 75
Method Call Overhead blah Benchmark in milliseconds: 92, 92, 92, 93, 91
Method Call Overhead Parameter blah2 Benchmark in milliseconds: 422, 517, 526, 525, 522
Looping over Large Collection of Objects Benchmark in milliseconds: 25, 24, 22, 21, 22
String Pressure Benchmark in milliseconds: 258, 257, 255, 256, 257
Collatz Conjecture Benchmark in milliseconds: 389, 390, 389, 391, 387
Fibonacci Benchmark in milliseconds: 299, 296, 297, 295, 296

As we can see, biiiiiiig gains in basically everything except the string pressure tests. Some benchmarks in fact have an order of magnitude improvement. And even though the string pressure tests show a dip in performance, it’s pretty slight.
So suffice it to say, big, big gains here.

Beyond that, he fixed several bugs that were lurking around, in particular with the telnet debugger(which was causing some weird behaviors in Torsion), as well as added a pretty large list of unit tests. So any future changes to the interpreter can come easier because we can validate the crap out of it with standardized testing.
Mind, part of the improvements do come with the consequence of some relatively minor changes to how the interpreter handles things.
For example, you can’t have local variables be in the global scope anymore. Aka, if you have a %localVar it HAS to be within a function or namespace. 
The good news is, if something is unsupported full stop by the interpreter, it’ll error and provide a specific message including the offending file, so correcting these problems is pretty simple.

I’m also planning to try and make the Project Importer catch as many of these as possible, so *in theory* it should be a seamless transition unless you got REAL freaky with your torquescript. And in the end, I think the performance gains are so good it’s worth it.

 

Project Manager
I’ve been working on this bit by bit for a while now, so most of you are aware that there’ll be a new PM for 4.0 to work with. I’ll be doing a follow-up workblog soon with more deets about it, but I had tossed it out to those on the discord for an initial peek and slap around and got a lot of excellent feedback on its shortcomings.
So with all the real big changes out of the way and shifting into the polish-up phase for the 4.0 release, I can get some grind time on this very soon and get the first real Release Candidate build out to work in conjunction with the upcoming 4.0 RC. 

 

And Now for Some Other Bits
So that’s it for the big standalone update bits, but there’s definitely a lot of other smaller things to note still, so lets go over some of those, shall we?

Base UI Updates
A number of small fixes went in, but an important one was further improvements and fixes to handling of the window state, like with borderless mode. These options should be quite robust and stable to work with.

Multiplatform Fixes
Ragora, Az, HiGuy, Hutch and TRON all got some really good grid time recently to fix a number of lingering issues for our non-windows platforms. Between a litany of file handling fixes, compiler shenanigans fixes(GCC, as ever, being very strict) and crash fixes, the multiplat situation is coming along nicely.
Hutch even confirmed that 4.0’ll run on Apple’s new M1 chips, so we needn’t worry about people upgrading their machines causing problems on the mac side of things, which is exciting

Library Updates
As part of the chasedown for multiplat shenanigans, we’ve made sure that OpenAL, SDL and TinyXML are up to date as well.

Zip Loading
Mars helped a lot in ensuring that the zip handling was ironed out and brought back up to working. This means you can package your modules or games into zips again and it won’t get all freaky-deaky when trying to load or save stuff.
This also opens up some options for the future as well, such as collapsing assets(and their associated files) into singular archive *.asset files or the like, keeping file counts down and potentially reduced disk footprints, which is pretty cool.

So there we are!
With all the big stuff settled, or in final testing for roll-in, the plan for the next month or so is bug chasing, polishing and shoring up the new Project Manager, and just generally refining 4.0 to make it locked and prepped.
So keep your eyes out and ears to the ground for the RC builds so we can make sure 4.0 is good to go sooner rather than later, because we’ve all definitely been waiting long enough

Until next time!
-JeffR