The pinnacle raw Torque power, Torque3D has been used for everything from driving simulators to MMOs!
Check Out Torque3D
All the power of Torque with less of those pesky dimensions to tie you down!
Check Out Torque2D
Looking for answers, wanna see some cool projects or talk about new / upcoming features?
Read the Forums
Work Blog Updates
By JeffR in Torque 3DMid-October workblog time! (Which should've been last month, but chasing down bugs like memleaks and straight up crashes before I wanted to post caused delays, so...whoops!)
So, how's it going everyone? Time for fanciful development news.
First, lets go over what all work has happened thus far since the last workblog:
76 pull requests merged in, which had over 164 changes which ranged from bugfixes, to improvements to additions.
Notable examples include:
Updating SDL to latest Steve passing along a fix to correct render issues for the ribbon particles Preference settings(which will get integration into the options menu soon) for forcing static object fadeout at a distance, as well as limiting the number of local lights renderiable at a time, and if they fade out over distance as well. These can potentially help a lot in very object-dense scenes with lots of small clutter stuff that doesn't need to render at a distance Some better default values for local lights, and cleaning unneeded fields Fixing of gamepad inputs Various shadergen cleanups A whole metric butt-ton of fixes, improvements and QoL changes for the asset workflow Ability to better control script queue ordering between modules A crossplat option for 'open in browser', which could see a lot of use in jumping to documentation Improvements to baking of meshes Adds populating of a level's utilized assets into it's dependencies list to ensure everything preloads as expected instead of trying to do it at the last second, which could cause scripts executing during object creation and lead to variable corruption Settings files are now properly sorted(A small change, but it keeps the layout of the settings.xml and asset import config files consistent making them easier to catch up on changes) Re-implementing SIS files for the importer so there can be special-case overrides for any given file type Fixes the resource change detection for TSStatics, so if a shape files is changed, it auto-updates the objects in the scene Fixed several potential memleaks and one confirmed one that could balloon the mem usage pretty substantially over time Misc improvements to asset import pipeline stuff, such as suffix parsing improvements Shuffled some gui profiles into core to better standardize them Created a number of macros to wrapper around defining and setup/usage of assets(image asset for now, but others later). This is so you don't have to define a bunch of supporting stuff in a given class over and over. Basically, convenience and code templating thing. GL and GCC compilation fixes Integrated the old MeshRoad Profile editor, so you can have more control over the shape of the meshroad Added guiRenderTargetViz control, which lets you specify any given render target and display it to a GUI control. Minimal direct use currently, but useful in debug operations, and in the future could drastically simplify doing stuff like picture-in-picture displays or multi view GUIs and the like. Fixed a pretty gnarly memleak, so we're memory stable again. Lukas got his C# project caught up to current BaseGame so we can better test the cinterface changes(and soon people can play around with all that too) For some of the bigger changes worth going into more detail:
First, Mars contributed some very important improvements to window and resolution handling.
This adds in Borderless window mode, as well as the ability to set in the options what display(if more than one is detected) the game window should be on. Additionally better handling for what screen resolution should be in play based on window mode(ie, Borderless is always the desktop resolution) as well as ensuring display options apply properly.
I added handling to disable options fields if they are non-applicable to avoid confusion
Secondly, an update to both PostFX organization, Integration, and Editing behavior
All stock PostFXs and their shaders are now safely tucked into the core/PostFX module. Easier to find, easier to edit, and the shaders aren't in a completely different module.
The loader logic was tweaked slightly as well, so that the levelAsset has a field which indicates what the postFX file is. This requires less manual logic for 'lemme go dig around for a posteffectpreset file' and should be a bit more reliable.
Additionally, the PostFX editor saw some fairly big updates, both in how its accessed, and how it works.
You can now either edit the default postFX preset, or edit the current Scene's preset, as seen here:
The editor now better integrates into the PostFXs themselves, as well as auto-applying changes as they happen which is much, much, much better for dialing in how they impact a scene.
Importantly, you'll note that the list of PostFX's displayed in the editor there looks kinda...lean.
This is because it now only displays active PostFXs rather than the entire registered library, which should help cut down on confusion about what PostFXs are actually active and impacting the scene.
Adding is as simple as pressing the green plus button at the top and then picking which to add from the list that auto-populates with all registered postFXs
And then selecting from the list on the left to edit a given postFX:
Removal is as easy as selecting the PostFX in question in the list and pressing the red minus button.
You may also note a bit of change in what PostFXs are there. Most are the same as always, but a few tweaks happened to make things a little more consistent and interop-friendly, such as moving the LUT color correction logic out of HDR and into it's own. I also added a simple Sharpen PostFX, and integrated a Chromatic Aberration PostFX.
We also recently shifted over to utilize 'ORM' formatting for the PBR image maps.
In order to better integrate with tools, and because GLTF assumes ORM, and it's the closest thing to an industry standard for the PBR map, we're shuffling it around internally to work as ORM as well. What's ORM? It stands for (Ambient) Occlusion, Roughness, Metalness. It's basically the PBR image maps arranged in a specific channel order. GLTF and a handful of engines assume for it in that order, making it the closest thing to a standard, so to keep it simple, we're doing the reorg.
Likewise, we were operating with Smoothness being the front-facing value instead of Roughness. This is - similar to the PBR channel order - sorta a 'eh whatever works', but Roughness is a bit more standard, so we're going ahead and making that the front-face standard as well.
Internally the math assumed roughness anyways, so it isn't a huge change principally.
Touching to the above, we'd noted some oddball behavior when adjusting roughness and metalness values, so with Timmy's fantastical eye for that stuff, he was able to spot some problem areas with the logic and our BRDF image. The changes he passed result in much, much better behavior across the full roughness/metalness ranges and thus looks much more correct on any given material. Huge props there.
I had also mentioned some crashes and stuff. There was a bug that snuck in with the probes where baking them could cause crashes. It hit some hardware more consistently than others, and was a right pain to track down. In the end, though, I put up a PR with a big update to the probe handling. Before, you could have up to 50 active probes, and they would render all in one big go. It worked, and it guaranteed consistent performance, but a lot of scenes don't utilize anywhere near than many probes in your line of sight.
So I shifted the logic to where you have registered probes, and active probes.
You can have up to 250 registered probes in the scene now, which is quite a lot for anything other than a big open world deal(and you can always unregister them selectively as needed), and an adjustable amount of active probes. The default is 8 but it can technically go all the way up the whole 250(though that's not recommended for performance reasons).
One of the big advantages outside of the basic performance consideration of not needing to actively render as many probes at once, is we can now lean on culling to ignore probes you can't even see, which just compounds the performance gains, and be smarter about which ones to bother with. It calculates and picks the best probes based on the camera's position, and I'll be adding an 'influence' value for more artistic control so certain probes can be noted as more important than others.
All of this together means it selects the best probes and renders only those up to the set per-frame limit(which again, is adjustable for a given game's needs) yielding the same results in terms of blending between probes, but much smarter and targeted selection of which ones to render, yielding improvements in performance.
We also fixed it finally so that surfaces that are metallic will correctly render in the baked cubemaps for probes instead of the flat black they were before, which should yield more accurate results in the reflections.
While the above probe reorg and crash chase-down sorta dominated the last 2 weeks(crashes are a pretty big deal, so it was important to get out of the way), we can get back on the track of getting the asset integration with the game classes sorted out. I mentioned the utility macros added for image assets, which are most recently utilized by the materials class now.
This has streamlined quite a bit of code and makes everything Just Work, and between it and the asset usage in TSStatic, we're feeling pretty confident in finally moving forward and adding asset integration for the remaining classes. This is probably the last big obstacle for 4.0, especially as the general asset pipeline is proving to be fairly stable at this point, and PBR is getting a good workout in several projects and what minor issues crop up are getting plugged quickly. The issues list I've got keeps creeping downward, and things are looking quite nice 🙂
For a parting bit, one thing we also added in recently was a Prototyping module. It has a number of color-coded materials designed to be used for certain surface/object types and be visually distinctive so as to help understand the map/design space without needing to worry about the fine texture detail. Additionally, a number of primitive objects like a cube, cylinder, etc are in there.
However, we also needed a size reference stand-in, and I thought, who better for that than our very own Kork-chan?
Definitely won't be as long for the next workblog, and I imagine we'll have a number of modules to try, art to ogle, and a new test build up quite soon, so stay posted!
By JeffR in Torque 3DHey everyone!
So, been a while since the last workblog, but the good news is, I didn’t quite threshold past the one year mark!
So that’s… good.
Anywho. Predictably there’s a frankly preposterous amount of stuff to go over so lets get getting and dig on in!
A lot, as said. But for specifics, since the last workblog, we’ve rolled in over 150 pull requests. These range everything from bugfixes, library updates, QOL improvements and all the way up to converting pretty much every class that touches a content file into using the assets system. We even got the site updated, as I’m sure you all had noticed.
Lets cover a few of the specifics and big ones.
So, if you’re reading this, then you’re at least slightly aware that the site underwent some changes relatively recently. We were using a git-based pages system before which while lightweight and anyone could contribute, the turnaround time ON changes was fairly suboptimal, and because the only way to make any changes at all was via PR’s, it disincentivized people from contributing content.
Pushing tutorials or updating documentation on the site had enough of a hurdle that no one really wanted to bother.
So we did a lot of digging and landed on a fully integrated CMS platform. This allowed the forums to be integrated into the main site and allowed us to easily manage the content that was on the site itself.
Other features like broad site search and a few handy additional boons like being able to post micro-communities for games all tie together to make a better centralized point of community engagement.
And as noted prior, it drastically lowers the bar for people pitching in to update the documentation if they have at least the Contributor designation, as someone trusted within the community on helping out and know their business.
As we’re in the home stretch on 4.0(as I’ll get into more below), I’m starting to be able to split my time from that as we wait on rounds of testing to begin filling out the documentation section.
The plan is to consolidate all the disparate docs from the various sites torque’s found itself on, from the old GG docs to the old TDN, to the wiki and so on, and fold it all back here in the docs section.
This way we have a concrete point people KNOW they can find what they’re looking for, instead of having to guess what random site the info they need is on.
Major, MAJOR props to Tony and Lukas for helping get this thing where it is now. It's definitely a big deal!
It’s been a bit since I’ve covered PBR stuffs, but honestly not TOO much has changed since I’d last covered it, so this can be fairly quick.
The main points to note is we spent a lot of time refining and stabilizing the math so you can have pretty high confidence that whatever is exported out from your art tools like blender or substance, if they use the standard Metalness/Roughness PBR formatting, will look Tools Correct. Also, as noted previously, we’d shifted up to utilize the more industry-standard ORM Configuration map, which further emphasizes the Tools-Correct approach.
Of course, you can still use sliders or individual maps to fill in the PBR properties too. That hadn’t gone anywhere.
Other things were cleaning up and stabilizing probes and IBL and improving performance with them by adding some nice convenience functionality with controlling active probe counts and the like.
All in all, PBR should give stable, quick results that look quite nice.
Alright, now we’re getting into the meat of things. So, assets. I’ve talked about them a good bit before, so if you’ve been following my workblogs, you’ve got at least a general notion of them.
Suffice it to say, they’ve gotten a LOT of work done in the past months.
So, as of this moment, outside of single-digit numbering exceptions, every class that ingests a normal “content” file type - that is, Images, Shapes, Materials - now utilizes assets.
While a lot of work to get right, we did everything we could to ensure that the process is stable and doesn’t impact the functionality of the classes or objects themselves.
Part of that is we made some… liberal use of macros to fill in a lot of duplicated and common code pertaining to these content file types.
Whereas before each class would have to implement the entire file ingest, load and bind logic each time - which as with all duplicate and repeated code, leads to points of possible failure - they now use a common set of macros which have been aggressively and thoroughly tested, making everything a bit leaner and less prone to issues sneaking in.
In this shot, we see an example of defining our asset macro in the class header file:
And here's how you use the asset macros to set up the init persist fields to expose them to script:
There's even macros for the packing and unpacking of the network data:
And finally, an important bit, is that it doesn't just generate function info or anything, but actually configures the internal variables and the like off the parameters you pass in. This allows everything to be VERY consistent and predictable, making usage in the classes once you actually want to utilize the data set much more comprehensible, like so:
This means that pretty much every class has a very similar content-file flow, which allows us to standardize object fields more(less guessing if it was shapeName, or shapeFile, or shape, etc). This also standardizes accessor functions.
Want the gui’s bitmap?
%bitmap = %gui.getBitmap();
And you can be sure it’ll get it, no questions asked.
Part of these macros is retaining the legacy fields too, so if you load up a project and some stuff doesn’t get processed successfully via the Project Importer(will get to that here shortly), then the old fields are held onto so you can retry, or debug it.
So those accessor functions are smart enough to figure out where your bitmap data is and return it back.
Using an asset? Returns the asset info. It still using the legacy bitmap filename field? Returns that instead.
This should alleviate a good chunk of pain for existing projects that are looking to port up into 4.0 without utterly snapping all your stuff over its knee.
Ultimately, this all ends out with a standardized, consistent content flow without having to juggle different means, forms and fields depending on the object and class, which is nice.
So, obviously, if we’re gunna be mainlining assets now, the asset workflow needs to be up to snuff as well. First up, we’ve got the Asset Browser. I’ve shown and talked about this thing before, but now it’s the primary access point for all content in your game.
Previous, I’d gone over the improvements to searches in the AB with stuff like complex searches and collection sets. This is obviously all still in, but I’ve added some further tweaks to improve reliability here. A search isn’t much help if it doesn’t find what you need it to.
Additionally, I’d covered how you could do stuff like drag-n-drop datablocks to spawn the vehicle or the like. This too has been improved and expanded on, so that pretty much anything that is generally spawnable via the Library tab can be spawned via the Asset Browser. The plan here is to ultimately replace the Library tab with the AB as the centralized one-stop-shop for all your content needs.
To that end, most of the ‘Creator’ entries have been brought over to the AB as well, so spawning lights, probes, precipitation, skies and so on can be done via the AB too.
(Of course, the current class icons are a bit low res and need some sprucing)
Beyond that, improvements to the tooltips for when you hover over an asset to present where the asset is located in the file structure and other meta info should help tracking down where those assets you knew you had but forgot where you put. We’ll also be looking into some ‘Go To Asset’ navigation option/actions as well to further facilitate tracking down these lost sheep, er, assets.
Now, if anyone threw some big images, or a lot of shapes at the AB previously(Catographer, for example, uses like a dozen 4k images for material atlases), you’d find there’d be a load hitch every time it’d try and populate the asset previews in the AB.
Obviously, this is a minor annoyance that, when repeated enough times(which it would, now that the AB is the primary content nexus) it would become grating fast, and hurt workflow.
So I added a system where it’ll take preview images and create a scaled down version, ensuring that no preview is so large as to cause hitching, allowing fast loading of directories even if they have a crazy number of assets visible.
Whenever the AB loads, it’ll generate a preview image based off the source content - a shape or image - and generate a scaled down preview image to a given size, such as a 512x512 preview.
This keeps them reasonably sized, which keeps loading in the AB fast, but doesn’t hurt the quick-glance recognition of what you’re looking for either.
Additionally, I added logic to compare the file modified timestamps whenever the AB is opened, so if an asset’s file saw changes newer than the preview image, it’ll regenerate it. This ensure that if you recently re-exported an image or a model, the previews will always show the correct version.
So with everything actually for-realsies using assets now, we want to ensure the importing process is as sexy-smooth as possible, so this saw improvements as well. Various sanity checks, safeguards and issue-resolution methods were added to the import configs.
For example, a new Issue Resolution step “FolderPrefix” was added. In the event of an importing asset happening to have the same name as an existing asset, like say, both have “grass” for the AssetName.
In this case, with the FolderPrefix resolution, it’ll then find the folder the asset’s file is in, and then just prefix it.
So if our conflicting importing file is in the ‘Foliage’ folder, it’d resolve the conflict via turning the asset name into Foliage_grass. It’s also smart enough to try and find non-conflicting folders in case THAT was already taken too.
Additionally, options for force-adding type suffixes onto the asset name help prevent collisions too. Like say we had a ‘Player’ model, that has a Player material and uses the Player image for the diffuse.
These 3 would normally conflict, being all named player.
With the options on to add the type suffixes, though, they become Player_shape, Player_mat and Player_image.
This prevents collisions AND becomes easier to search for/spot at a glance which-is-which.
So several birds with one stone there!
And remember, this doesn’t affect the actual file the asset uses, just the assetName the engine builds for the AssetId, so you don’t have to worry about it doing spooky rename voodoo on your png file so you can’t find it again later.
Asset Loading/Error Tracking
A big addition that Az pushed for was better tracking of the explicit load status of an asset. Before, you effectively had two states for an asset. It either loaded, or it didn’t. Obviously, depending on what’s going on, this could be wildly unhelpful. For example, if the asset definition loaded ok, but something was wrong with the associated file(like lets say the model file was corrupted), we can report the error while still continuing to load the asset, instead presenting a fallback shape.
This avoids the old situation where if a model failed to load on a TSStatic, the TSStatic just wholesale doesn’t load, disappearing from your level and some point later down the line you realize a bunch of your objects are missing.
Beyond that, naturally having more specific loading error codes lets you pin down exactly what’s wrong. From bad file references, to invalid formats, being able to note the problem is the first step to fixing it, which can save on development time without the engine just throwing its hands up and failing to load everything.
Also important to assets are the modules as well. Per previous updates, we’ve been refining how modules handle loading of scripts that may overlap or override via the new queueExec() function.
We’ve been shoring up problems that’d come up and dialing it in more, but overall modules haven’t seen any big functionality shifts, which is good, as that means even with people wrenching on it, it’s proven to be rather stable.
Legacy Project Importer
So this has been a big one. When we were implementing the conversion process of existing projects to work with 4.0(since obviously there’s a number of people with existing projects that would be keen just update update to latest to get the sweet new benefits) we originally tried a sort of “in line” import process.
As part of the above mentioned asset macros, we would integrate in the original filename fields, that when a file is fed into, would run the Asset Importer in-place, get the results, and update the field definitions.
And in a lot of cases, this worked!
The problem was there was also a lot of cases where we started seeing some really weird and funky behavior, timing issues, order of operations shenanigans, and needing to close and open things multiple times to process everything.
The sexy, seamless system unfortunately didn’t play out quite as smoothly as the theory indicated.
So, I opted to pivot on it and build it as a dedicated importer process. There were several advantages to this approach. Sure, we lost the ‘pure automagic’ of the inline approach, but you could start up a new 4.0 project, open the importer, point at your old game, and it’d just import everything in and process it into a valid module for you.
So maybe not PURE automagic, but like, 56.5% automagic.
So I did the initial pass of it and with Az helping crunch a lot of the logical refinements, we got it to a spot where we could get others throwing their projects or old content kits at it. While there’s still some bugs to be fully chased down yet, the results are looking really good, and should make it MUCH less painful for anyone and everyone with an existing torque game(in theory all the way back to TGE games, though that’s not a dedicated focus at the moment) can get rolled over to 4.0 in an afternoon.
Nice, How’s It Work?
As mentioned, the process involves you pointing at your old torque project, and getting a nice, 4.0 compliant module in your new 4.0 project as a result.
But lets get into the specifics of it.
Is the Project Importer. It’s formatted as a classical wizard utility that walks one through the process in pretty plain terms.
It’s pretty straightforward, but we can do a quick overview of how it works here. If you continue past the welcome page, you select what version you're importing from(though currently everything is pre-4.0, this gives us options to have by-version import rules for people upgrading from 4.0 to 4.1 and so on later) it goes on to prompt you about what content you’re going to process in with the importer, as well as it’s location, as seen here:
Depending on which you pick dictates some supplemental behavior. As you can see your options are:
A folder that is inside the current project already - useful for re-importing something if required A folder that is outside the current project. This is the normal option and would be what you pick to select your old project’s main directory The Core and Tools folder. This is primarily for getting the BaseGame template up to speed, but if you’ve got some custom tools or the like you’ve copied into place already, you could run that. Normally, this is where you would select the Outside folder option. Then you click the locate button, and navigate to your project you want to import. Once you’ve done that, you can continue.
It’ll next prompt what the new module being created will be called. This is defaulted to whatever folder you pointed to’s name, but it can be named anything like ‘MyCoolGame’.
Continuing on will see the files be copied into the new module, then processed. It’ll run all content files(shapes, images, etc) through the Asset Importer, and then it’ll process all editable/code files looking for legacy fields.
If it finds one, it’ll then convert that field into the associated asset field, and find the new asset associated to the filepath it had originally. So you go from something like this:
Bitmap = “./background.png”;
bitmapAsset = “MyCoolGame:background_image”;
It’ll only process lines and objects that actually can successfully convert, so if a file doesn’t import and generate an asset, the legacy line will stay as-is. You can re-run the importer again on the existing module if needbe, but tests have so far proven it’s generally a one-and-done process.
Once the import process has completed, the only real thing left is to integrate in the scripts to the module. You’d crack open the module script file, and then slap in your exec calls for the client, server, etc in the appropriate place and there ya go. Your project’s pretty well converted up into a 4.0 module!
In example, this we can see the FPSGameplay content imported in as a module now here:
Naturally, we’ll be fully fleshing out the process and details of this in the documentation, but that’s the broadstrokes. Currently it’s part of the tools suite, but the main plan is it being part of the new project manager, so you can create a project, launch the LPI, and convert in an old project into the 4.0 one all in rapid succession in the PM.
Up next, going to do a quick go-over on the terrain updates. No major overhauls here, but we did bring in Lukas’ improvements to the terrain material blending, as well as toggles for if it should utilize the height data in the terrain materials to blend materials or not, as seen here:
This naturally allows not only nicer looking terrain materials blending, but the toggle gives more control if it fits your style.
Additionally, Lukas did some nice wizardry to streamline the rendering of the terrain a lot, utilizing texture arrays to remove a LOT of the drawcalls and render overhead that came with the older way we rendered terrain. So looks better and runs better. Not too shabby 😉
This one has been discussed for quite a while, but we went ahead and pulled the trigger on it a little while back. Namely, shifting of the default torquescript extension to ‘tscript’ from ‘cs’. A number of reasons for this, but the big ones being that a lot of tools treat cs as a C# extension, which can have some weird results when dealing with code IDEs.
Another reason tying into that was the expansion of the cinterface and support for more script languages Lukas spearheaded, namely, C#.
With that soon to become a fully viable possible route for people’s T3D games, dealing with confusing extension mix-n-matches - especially for project generation and asset/module installs not being able to tell a torquescript and a c# version apart - all lead to the lock-in on swapping the default extension.
That said, we know people liked having that cs extension, so while the default may have changed, we added a new project setting for CMAKE as well as a global var that sets the extension used. So if you wanted to keep with cs, you can just change that in your project settings and presto!
And to better facilitate the extension swapping, we went in and adjusted a number of filename handlings to check for both the default and the ‘other options’. Meaning that doing an exec() call can just take the path and name, leaving off the file extension and the engine’ll figure it out. Additionally, we added the isScriptFile() console function to specifically check valid script extensions if you're trying to test if a filename is a valid script file, rather than trying to bruteforce your way through with isFile().
Further, If you wanted to go ahead and use the tscript extension and were importing in an old project, fear not, because the Project Importer will check what the extension global variable is, and update any scripts to utilize that extension instead(including filename references in files, in cases where it doesn't just drop the extension as unneeded entirely);
So while it is a change and I know some of you may not be fond of it, we’ve done everything possible to make it painless to swap, or just keep to the classical if you so desire. 🙂
TorqueScript Interpreter Update
While not merged in juuuuust yet, we’re in the final testing pass for some big updates to the TS Interpreter(lovingly dubbed as ‘TSNeo’). Hutch has been doing some solid work not only simplifying the interpreter, making it easier to maintain and expand, but also some biiiig performance gains. How big? Lets have a gander.
He drafted up a number of tests to check perf of various things, from crunching math right in TS, to handling objects and variables, to invoking up into the engine via console methods. All metrics are for his i9-9900K @ 4.8ghz. All numbers are in milliseconds.
So here’s stock:
As you can see, some stuff isn’t too bad, but function calls in particular are really painful.
Now let’s look at the TSNeo benchmarks:
As we can see, biiiiiiig gains in basically everything except the string pressure tests. Some benchmarks in fact have an order of magnitude improvement. And even though the string pressure tests show a dip in performance, it’s pretty slight.
So suffice it to say, big, big gains here.
Beyond that, he fixed several bugs that were lurking around, in particular with the telnet debugger(which was causing some weird behaviors in Torsion), as well as added a pretty large list of unit tests. So any future changes to the interpreter can come easier because we can validate the crap out of it with standardized testing.
Mind, part of the improvements do come with the consequence of some relatively minor changes to how the interpreter handles things.
For example, you can’t have local variables be in the global scope anymore. Aka, if you have a %localVar it HAS to be within a function or namespace.
The good news is, if something is unsupported full stop by the interpreter, it’ll error and provide a specific message including the offending file, so correcting these problems is pretty simple.
I’m also planning to try and make the Project Importer catch as many of these as possible, so *in theory* it should be a seamless transition unless you got REAL freaky with your torquescript. And in the end, I think the performance gains are so good it’s worth it.
I’ve been working on this bit by bit for a while now, so most of you are aware that there’ll be a new PM for 4.0 to work with. I’ll be doing a follow-up workblog soon with more deets about it, but I had tossed it out to those on the discord for an initial peek and slap around and got a lot of excellent feedback on its shortcomings.
So with all the real big changes out of the way and shifting into the polish-up phase for the 4.0 release, I can get some grind time on this very soon and get the first real Release Candidate build out to work in conjunction with the upcoming 4.0 RC.
And Now for Some Other Bits
So that’s it for the big standalone update bits, but there’s definitely a lot of other smaller things to note still, so lets go over some of those, shall we?
Base UI Updates
A number of small fixes went in, but an important one was further improvements and fixes to handling of the window state, like with borderless mode. These options should be quite robust and stable to work with.
Ragora, Az, HiGuy, Hutch and TRON all got some really good grid time recently to fix a number of lingering issues for our non-windows platforms. Between a litany of file handling fixes, compiler shenanigans fixes(GCC, as ever, being very strict) and crash fixes, the multiplat situation is coming along nicely.
Hutch even confirmed that 4.0’ll run on Apple’s new M1 chips, so we needn’t worry about people upgrading their machines causing problems on the mac side of things, which is exciting 🙂
As part of the chasedown for multiplat shenanigans, we’ve made sure that OpenAL, SDL and TinyXML are up to date as well.
Mars helped a lot in ensuring that the zip handling was ironed out and brought back up to working. This means you can package your modules or games into zips again and it won’t get all freaky-deaky when trying to load or save stuff.
This also opens up some options for the future as well, such as collapsing assets(and their associated files) into singular archive *.asset files or the like, keeping file counts down and potentially reduced disk footprints, which is pretty cool.
So there we are!
With all the big stuff settled, or in final testing for roll-in, the plan for the next month or so is bug chasing, polishing and shoring up the new Project Manager, and just generally refining 4.0 to make it locked and prepped.
So keep your eyes out and ears to the ground for the RC builds so we can make sure 4.0 is good to go sooner rather than later, because we’ve all definitely been waiting long enough 😛
Until next time!
By JeffR in Torque 3DMinor delay because I was feeling under the weather.
Anyways! Part twooooo!
Predictably, a bunch of various improvement and fixes have gone in on the engine side of things. Various crash fixes, out of bounds fixes, etc.
Beyond that, though some other nice stuff has happened. A big one was moving the asset importing logic up from the tools scripts into the engine. This was done for a few reasons.
One was processing speed. The native C++ is obviously going to run faster than TS, so in general, the processing time for import has improved. But bigger than that is the ability to do in-place auto-importing.
Similar to how, say, a TSStatic would reference a DAE file. If no cached.dts file was found, it would automatically run the import of the DAE file, and generate the cached.dts file. The assets system can now operate in the same way. In the event a non-asset loose file is utilized, we can try a multi-step process to handle that.
We would first try and find any asset that utilizes that particular file, and if found, we just utilize that asset and update the TSStatic's fields to reflect it. If that fails, we then try and run an in-place auto-import. If the tools folder exists we even try and look up if a default import config is set in the editor preferences, otherwise we just use the default import settings. If the import is successful, we like above return the new assetId and that is updated in the TSStatic's fields. If all that fails, we can return back a placeholder assetId(in TSStatics' case, it's a noShape mesh) to indicate 'hey something went wrong with loading' to the level editor, while still retaining the original fields so debugging can be done to find out what went wrong without contaminating the setup.
We've been beating the system up on TSStatics and a few remaining issues are there, but it's proven to be rather fast and robust. And because it can defer to an importConfig, it's customizable to the project's needs, as well as utilizing Assimp for multiple other file formats like FBX.
Once the last bits are plugged, we can then move on to implementing it for all the 'regular' types, shapes, images, materials, etc.
This means that legacy projects being imported in will auto-process a ton of content into assets without needing to do any manual work, which is a huge timesaver, and non-legacy projects have it as a handy fallback for workflow efficiency.
There was also various improvements to the PBR and IBL stuff to make the math more correct. It'll continue to get refined as we find assets and scenes that throw odd results, but it's pretty robust at this point, both for new, PBR-friendly assets:
And even non-PBR legacy assets can end out looking pretty solid by just being tossed in with no additional work(with some bonus footage of the asset import process doing it's thang)
Beyond that, some other various improvements/expansions on the engine side in no specific order:
Shifted glow over to a full separate RBG image map, allowing a lot more artistic control Added the ability to in-place insert inspector fields and groups, allowing arbitrary editor hook ins(an example of this would be if ECS was a module, it could be dropped in and just inject the inspector group for components into an entity in the inspector without having to hard-code the fields. This allows flexible and dynamic inspector and editor behavior and opens up a LOT of cool options for the future Added ability for text edit fields to have placeholder text rather than needing to do that janky voodoo of setting it's text to some placeholder value and hoping nothing goes funky when typing Fixed CPU detection and reporting on 64bit windows(courtesy of Hutch) Functionality expansions to groundcover spawning Corrected some erroneous behavior with terrainAssets and terrainMaterialAssets so everything works together more predictably Added a guiVariableInspector field type 'Range' for slider support in inspectors Fixed OGL handling of probe baking - very important for linux and mac for obvious reasons Whole lotta cleanup of uninitialized vars which should help with stability in the engine by preventing unexpected default values Fixed an issue with Assimp loading where shape scales were off(courtesy of Mars) Added ability for shadows to work with translucent materials utilizing alpha test
And a sprinkling of other stuff like crashfixes, as mentioned prior with several contributors helping out on that end like Chaigler, Bloodknight, Tony, Hutch and Mars.
There's a number of other stuff cooking at the moment getting worked on, but a few highlight bits that I'll actually be making PRs for over the weekend are performance oriented.
Namely, the ability to, via preference variables(as sometimes you don't want it, or want to leave it up to the end user) to force on static object fading, as well as fading of local light(point and spot) and limiting how many local lights can be rendered at a time.
These are pretty handy for lower-end machines because whatever standard tricks to help performance in rendering - reducing shadow resolution, not rendering shadows at all, LODs, etc - ultimately the fastest possible thing to render is absolutely nothing at all 😉
So in action, that all looks like this:
In the first part you can see that in the current setting, I have a maximum of 3 local lights allowed, so the others in the grid are culled, based on score which is a calculation of the lights' size, brightness and distance from the camera.
Then as the camera pulls out you can see that the lights smoothly fade out based on the light fade setting. There's a start and end fade distance. So, say, fading doesn't start until the light is 50m away, and will fully fade out by 75.
Then I flip on the object fade setting and we can see that now the static objects behave similarly. Based on their size, they can begin fading out sooner, so small clutter objects can be quickly culled from rendering without negatively impacting the fidelity of the scene up close, while larger objects are slower to fade out.
Additionally, there's a setting that allows the defining of an object size considered large enough it shouldn't ever be culled like this, which is useful for landmark objects and the like.
The idea being that these can be adjustable for the end user via some options menu settings, so people with weaker computers can have this stuff kick in fairly aggressively(in combination with other options to lower render cost) giving more control and options for making the game run better without negatively impacting those with powerful enough rigs to render everything at full blast.
When I get these added in, I can then add options to the graphical options menu to facilitate this. The dev can set the desired baseline behavior, and the preference would permute it to be more aggressive as needed(or disabled entirely for an 'ultra' setting.
Next, tools stuff!
The big one would be updates to the Asset Browser.
There's a LOT to unpack here, so lets get into it.
From the notations in the first image, we can see...well, a lot of stuff, so lets walk through it.
First, you can note a modified layout from what was there before, which is, hopefully, overall clearer and easier to understand most of it at a glance. Refinements will continue, of course, but this should(especially the utility buttons at the top) be much more immediately legible.
So, per the notations:
Slap this to create a new asset in the current path, via a popup listing of all acceptable types. Can also just RMB in the main browser window.
Opens a 'open file' dialog to go rooting around for valid file types to import in as new assets into the current path. Can also do the same via Drag-n-Drop.
A popup that lets you easily toggle various filter options, like filtering by asset type.
Auto-Import in Current Directory
This one is a cool convenience feature. It'll run through the current directory and run an in-place auto-import on any loose files(files that don't have an associated asset that owns it). Another handy feature for importing of legacy projects. This button doesn't display if there are no non-asset loose files in the directory.
Ok, now we get into the FUN stuff.
Before, search was pretty basic, conceptually. You typed something in, like say 'wood' and any asset with a matching name would be filtered for display in the main view.
Now, however, we have complex search phrasing logic.
Take a search string like this:
What this would do is search in the current directory(and any child folders, of course) for any asset that is:
A materialAsset Has a tag of 'wood' Has 'old' in it's name So the end result would be any asset that is, effectively, a material asset of a wood type with old in the name, like a hypothetical 'Old_Ship_Wood_Material' asset.
This feeds back into the filter options above, so flipping on/off the type filters automatically updates the search phrase. Poking any of the 'tag' section entries from the second Asset Browser image likewise toggles in/out tags.
It also can support multiple words per 'term', via comma separation.
So if we did:
It'd find any materialAsset with a tag of wood that has old Or new in the name.
Pretty powerful, to be sure! But it gets niftier. Because as neat as complex searches like that are, having to flip the various tag and filter values on, and then type a relevent name to always look up, say a common set of materials you use is a pain every time.
If only there was some way to save them...
Save As Collection Set
Oh wait, we totally can. If you slap that little button there, you get a name prompt, and then it saves the current search phase into a collection set of that name, which then shows up in the Collections section in the Asset Browser.
You can then hit that item and it automatically fills out the search term for you with that saved one. So just like that we click a single item and we are back to seeing only materialAssets with the tag wood and old or new in the name.
This can be pretty huge if you have collections of associated assets and can save a lot of time from having to constantly dig around to find them again.
Basically, the general idea is it doesn't really matter where or how you ultimately organized your assets(or maybe you didn't organize them yourself at all like if you got an art pack), you should be able to find what you're looking for quickly and easily.
Asset Preview Size Slider
Not quite as cool as that, but still important, is the slider for the Asset Preview size. It...uh, adjusts the size of the Asset Previews. In fact, if you crank it all the way down to the smallest, it turns into a list for maximum browsability!
Given that the plan is for the asset browser to replace the old Creator, Meshes and Library tabs in the Scene Window(with the ability to make the AB dock into that window so it doesn't always take up a big ol' chunk of your screen) this is very useful for being able to actually see what the heck you're working with in the current directory.
Topically, but in the earlier updated AB walkthrough image, you can see not only the Tags section, but the Creator section. The ability to create new objects like how you do in the creator tab will all end up over there, so the Asset Browser will be a one-stop-shop for all your content browsing needs. You can even browse and spawn 'scripted' objects via their datablocks!
So spawning behavior is consistent, and it'll work with dang-bloody-near everything. So yeah, pretty slick.
Also, you may have noticed some colored outlining in the screenshots around the previews. The idea there is that different asset types would get different outline colors so even in list mode you can at a glance determine if it's a material asset, or shape asset or whatnot. Minimize confusion, maximize quick parsing of what content you're working with, you know.
Topical to looks, there's some hard-to-read text and other funkiness in those shots. The editor theme will be cleaned up and more or less finalized based on what was learned with the BaseGame UI's theme, so it'll all be much more legible and visually consistent when we're done.
For one last shot, A quick gander at the current Asset Import Window:
(Remember, the hard to read text'll be fixed 😉 )
Under normal circumstances, you won't likely even see this window unless you intentionally disable the 'Auto Import with Default Import Config' setting in the editor preferences, but just to highlight some cool logic the importer can do:
I dragged and dropped just the shape file into the Asset Browser window. It then parsed the materials from the mesh, and added it to the importing list of assets, namely the material GothicCabinet_01, and automatically childed to the shape. Then, it noted that the material had an image assigned to it in the shape file, so it grabbed that, and childed it to the material. The image asset is that GothicCabinet_01_8-bit_Diffuse (sweet naming scheme, am I right?). It recognized this as an albedo image via the 'diffuse' suffix on the end, so when everything is imported, it'll automatically be slotted into the materials' albedo image slot. It then proceeded to scan the original file's directory for potentially related images based on naming convention. It looked for GothicCabinet_01_8-bit with a bunch of different, common suffxes on the end based on various types. It found a file that matched the name but had _NORMAL on the end, so it also brought that into the import and childed it to the material as well. Like our diffuse, it recognized the suffix so it knows it's a normal map, and will ultimately fill in that image into the materials' normalMap slot.
All of that is driven by the settings of the import configuration, and ultimately meant that when I said 'import this model' it automagically found and processed in a bunch of associated assets saving me from having to manually import in the images and assign them to the material myself, which over the course of development can save quite a lot of busywork time.
I've shown off the general thrust of that before, but The UI is in a better space now to convey how things associate, and I thought it was worth re-iterating how 'smart' the import tries to be.
In fact, you may note from the image that it considered the Shape file a 'duplicate asset'. This is because the naming convention of the shape and material collide as they both have the same name, which could cause reference problems. I can either let it auto-resolve the problem by, say, renaming the offending asset, or having it auto-append _Mat to the material name, etc based on import config options.
And the entire import processing is logged now, so if anything happens that is unexpected, you can go back through the logs and see exactly what happened, where, and why.
No system is infallible, but I'm definitely shooting for as robust as feasibly possible at least 😉
Ok, wow! This part actually ended up being longer than I anticipated, so we'll be doing a part three where I cover the remainder bits, and then a whats-what of whats left!
See you tomorrow!
- 36 replies
- 24942 views
- 79 replies
- 26942 views
- 26 replies
- 1644 views
- 3 replies
- 1217 views
I made all these assets and sounds. Its made in Torque3d but using the version in BeamNG World Editor.
- 2 replies
- 1315 views
Built with Torque
Open Community · 1 fan
Open Community · 1 fan
Open Community · 3 fans
Open Community · 5 fans
Open Community · 6 fans
Open Community · 1 fan
Open Community · 2 fans