Jump to content
  • Torque3D

    Torque3D has been around for a while, and it's been used for everything from driving simulators to MMOs.


    Check Out Torque3D

    Torque2D

    Torque2D is built to be a 2D focused game engine..


    Check Out Torque2D

    Forums

     

    Looking for answers, wanna see some cool projects or talk about new / upcoming features?


    Read the Forums
  • Topics

  • Work Blog Updates

    • By JeffR in Torque 3D
         0
      Mid-October workblog time! (Which should've been last month, but chasing down bugs like memleaks and straight up crashes before I wanted to post caused delays, so...whoops!)
      So, how's it going everyone? Time for fanciful development news.
      First, lets go over what all work has happened thus far since the last workblog:
      76 pull requests merged in, which had over 164 changes which ranged from bugfixes, to improvements to additions.
      Notable examples include:
      Updating SDL to latest Steve passing along a fix to correct render issues for the ribbon particles Preference settings(which will get integration into the options menu soon) for forcing static object fadeout at a distance, as well as limiting the number of local lights renderiable at a time, and if they fade out over distance as well. These can potentially help a lot in very object-dense scenes with lots of small clutter stuff that doesn't need to render at a distance Some better default values for local lights, and cleaning unneeded fields Fixing of gamepad inputs Various shadergen cleanups A whole metric butt-ton of fixes, improvements and QoL changes for the asset workflow Ability to better control script queue ordering between modules A crossplat option for 'open in browser', which could see a lot of use in jumping to documentation Improvements to baking of meshes Adds populating of a level's utilized assets into it's dependencies list to ensure everything preloads as expected instead of trying to do it at the last second, which could cause scripts executing during object creation and lead to variable corruption Settings files are now properly sorted(A small change, but it keeps the layout of the settings.xml and asset import config files consistent making them easier to catch up on changes) Re-implementing SIS files for the importer so there can be special-case overrides for any given file type Fixes the resource change detection for TSStatics, so if a shape files is changed, it auto-updates the objects in the scene Fixed several potential memleaks and one confirmed one that could balloon the mem usage pretty substantially over time Misc improvements to asset import pipeline stuff, such as suffix parsing improvements Shuffled some gui profiles into core to better standardize them Created a number of macros to wrapper around defining and setup/usage of assets(image asset for now, but others later). This is so you don't have to define a bunch of supporting stuff in a given class over and over. Basically, convenience and code templating thing. GL and GCC compilation fixes Integrated the old MeshRoad Profile editor, so you can have more control over the shape of the meshroad Added guiRenderTargetViz control, which lets you specify any given render target and display it to a GUI control. Minimal direct use currently, but useful in debug operations, and in the future could drastically simplify doing stuff like picture-in-picture displays or multi view GUIs and the like. Fixed a pretty gnarly memleak, so we're memory stable again. Lukas got his C# project caught up to current BaseGame so we can better test the cinterface changes(and soon people can play around with all that too) For some of the bigger changes worth going into more detail:
      First, Mars contributed some very important improvements to window and resolution handling.
      This adds in Borderless window mode, as well as the ability to set in the options what display(if more than one is detected) the game window should be on. Additionally better handling for what screen resolution should be in play based on window mode(ie, Borderless is always the desktop resolution) as well as ensuring display options apply properly.
      I added handling to disable options fields if they are non-applicable to avoid confusion
      Secondly, an update to both PostFX organization, Integration, and Editing behavior
      All stock PostFXs and their shaders are now safely tucked into the core/PostFX module. Easier to find, easier to edit, and the shaders aren't in a completely different module.
      The loader logic was tweaked slightly as well, so that the levelAsset has a field which indicates what the postFX file is. This requires less manual logic for 'lemme go dig around for a posteffectpreset file' and should be a bit more reliable.
      Additionally, the PostFX editor saw some fairly big updates, both in how its accessed, and how it works.
      You can now either edit the default postFX preset, or edit the current Scene's preset, as seen here:

      The editor now better integrates into the PostFXs themselves, as well as auto-applying changes as they happen which is much, much, much better for dialing in how they impact a scene.

      Importantly, you'll note that the list of PostFX's displayed in the editor there looks kinda...lean.
      This is because it now only displays active PostFXs rather than the entire registered library, which should help cut down on confusion about what PostFXs are actually active and impacting the scene.
      Adding is as simple as pressing the green plus button at the top and then picking which to add from the list that auto-populates with all registered postFXs

      And then selecting from the list on the left to edit a given postFX:

      Removal is as easy as selecting the PostFX in question in the list and pressing the red minus button.
      You may also note a bit of change in what PostFXs are there. Most are the same as always, but a few tweaks happened to make things a little more consistent and interop-friendly, such as moving the LUT color correction logic out of HDR and into it's own. I also added a simple Sharpen PostFX, and integrated a Chromatic Aberration PostFX.
      We also recently shifted over to utilize 'ORM' formatting for the PBR image maps.
      In order to better integrate with tools, and because GLTF assumes ORM, and it's the closest thing to an industry standard for the PBR map, we're shuffling it around internally to work as ORM as well. What's ORM? It stands for (Ambient) Occlusion, Roughness, Metalness. It's basically the PBR image maps arranged in a specific channel order. GLTF and a handful of engines assume for it in that order, making it the closest thing to a standard, so to keep it simple, we're doing the reorg.
      Likewise, we were operating with Smoothness being the front-facing value instead of Roughness. This is - similar to the PBR channel order - sorta a 'eh whatever works', but Roughness is a bit more standard, so we're going ahead and making that the front-face standard as well.
      Internally the math assumed roughness anyways, so it isn't a huge change principally.
      Touching to the above, we'd noted some oddball behavior when adjusting roughness and metalness values, so with Timmy's fantastical eye for that stuff, he was able to spot some problem areas with the logic and our BRDF image. The changes he passed result in much, much better behavior across the full roughness/metalness ranges and thus looks much more correct on any given material. Huge props there.
      I had also mentioned some crashes and stuff. There was a bug that snuck in with the probes where baking them could cause crashes. It hit some hardware more consistently than others, and was a right pain to track down. In the end, though, I put up a PR with a big update to the probe handling. Before, you could have up to 50 active probes, and they would render all in one big go. It worked, and it guaranteed consistent performance, but a lot of scenes don't utilize anywhere near than many probes in your line of sight.
      So I shifted the logic to where you have registered probes, and active probes.
      You can have up to 250 registered probes in the scene now, which is quite a lot for anything other than a big open world deal(and you can always unregister them selectively as needed), and an adjustable amount of active probes. The default is 8 but it can technically go all the way up the whole 250(though that's not recommended for performance reasons).
      One of the big advantages outside of the basic performance consideration of not needing to actively render as many probes at once, is we can now lean on culling to ignore probes you can't even see, which just compounds the performance gains, and be smarter about which ones to bother with. It calculates and picks the best probes based on the camera's position, and I'll be adding an 'influence' value for more artistic control so certain probes can be noted as more important than others.
      All of this together means it selects the best probes and renders only those up to the set per-frame limit(which again, is adjustable for a given game's needs) yielding the same results in terms of blending between probes, but much smarter and targeted selection of which ones to render, yielding improvements in performance.
      We also fixed it finally so that surfaces that are metallic will correctly render in the baked cubemaps for probes instead of the flat black they were before, which should yield more accurate results in the reflections.
      While the above probe reorg and crash chase-down sorta dominated the last 2 weeks(crashes are a pretty big deal, so it was important to get out of the way), we can get back on the track of getting the asset integration with the game classes sorted out. I mentioned the utility macros added for image assets, which are most recently utilized by the materials class now.
      This has streamlined quite a bit of code and makes everything Just Work, and between it and the asset usage in TSStatic, we're feeling pretty confident in finally moving forward and adding asset integration for the remaining classes. This is probably the last big obstacle for 4.0, especially as the general asset pipeline is proving to be fairly stable at this point, and PBR is getting a good workout in several projects and what minor issues crop up are getting plugged quickly. The issues list I've got keeps creeping downward, and things are looking quite nice 🙂 
      For a parting bit, one thing we also added in recently was a Prototyping module. It has a number of color-coded materials designed to be used for certain surface/object types and be visually distinctive so as to help understand the map/design space without needing to worry about the fine texture detail. Additionally, a number of primitive objects like a cube, cylinder, etc are in there.
      However, we also needed a size reference stand-in, and I thought, who better for that than our very own Kork-chan?

      Definitely won't be as long for the next workblog, and I imagine we'll have a number of modules to try, art to ogle, and a new test build up quite soon, so stay posted!
      -JeffR
    • By JeffR in Torque 3D
         0
      Hey boys and girls!
      After an... unintended hiatus from workblogs, I got off my butt and kicked things back off. Now, as you can see, with a separated forum for easier organization and allowing discussion to not over-shadow literally everything else.
       
      So, what's been happening since the last workblog? Whole bunches of stuff, naturally.
      I just took a peruse through the repo(which you may recall from the community refresharoo thread, is now over at https://github.com/TorqueGameEngines/Torque3D and counted it up and we've had 124 commits since the last workblog.
       
      What all has that covered? Well, lets have a look, shall we?
       
      BaseGame
      The BaseGame template is cruising along quite nicely. With the help and feedback of peeps in the discord, it's got a much more legible, consistent and stable look and feel. Here's a few pictures to show how it's looking now:





       
      As we can see, a nice, clean look with a modern, unoffensive dark theme. A great launch-off point for customizing into your project's special UI.
      It has a fleshed out options menu, a litany of default preference options, as well as the keybind menus for both keyboard and mouse, as well as gamepad.
      Even the level select page is module-friendly. It automatically scans for any LevelAssets and auto-populates into the listing.
       
      Also from a usability standpoint, it not only supports button presses via the MenuInputButtons at the bottom, but they also work with gamepad inputs, detect the last input type and refresh the UI to show the input-specific images on said buttons(supports keyboard + mouse, xbox, switch and ps4 controller button prompts)
       
      The logic for the menuInputButtons will also later be expanded to be able to be used on other UIs like the playGUI so you can show the correct input prompt during gameplay too.
       
      Beyond the looks, overall behavior is much more stable and cleaner as a general example.
      Not only does the general behavior of modules lend itself to extendability, but there's also logic with being able to better control queue and execution order of scripts, allowing modules to override script files. So if you don't want to use the BaseUI's MainMenuGUI, you can have your own module have a mainMenu file that overrides the BaseUI and creates a clean swapout replacement.
       
      Specifically pertaining to it's looks the dark theme used was used as a general visibility/design testbed which can then be fed back into the dark theme for the editor, and the new Project Manager.
      So lots of good stuff from that.
       
      Oh, and another bit of usability that just recently went in from Tony was a slider to control the opacity of the console window. Because depending on what it's backdropped against, sometimes legibility can get pretty bad:
       


       
      Plus, a lot of cleanup has been going on to de-tangle various guiProfiles so they make more sense, and there's less unused/unneeded/duplicate ones dangling around cluttering the place up.
       
      Now, you may ask why put any effort into the BaseGame template if it's likely to see a lot of replacement of elements for any real project. Which is a good question, of course. I've mentioned it before, but I see the BaseGame template not only a launch-off point for projects, but also a general guide.
      It provides a lot of common functionality, which lets people get on with actually working on their games, and it has a decent amount of comments in the code already(and will get more as we go) explaining how stuff works. This lets newer users see a working example without having to download 'ActuallyWorkingGUI' pack or the like.
      Beyond that, establishing good habits for newer designers is a win for everyone. If the default options menu comes fully loaded with a bunch of performance preferences they don't have to fuss with, then that means nearly all games will have that then. Much better than most other engines' defaults which generally are nothing.
      If the dev is lazy or not skilled enough to implement those kinds of things, that can lead to a bad general impression. A lack of performance options means that players have no recourse in improving how the game runs, which leads to a bad time. Inconsistent UI behavior, or gamepads support that doesn't work in the menus also leave a bad taste in the mouth.
      All of which can come back and negatively impact on T3D.
      So if the starting point new developers jump off from is concrete and more fleshed out, then that encourages a development direction towards that end-user experience, which ultimately reflects positively on the engine as well.
      Plus, developers being able to know that if they slap 'New Project', they're getting a solid baseline project and UI they can just work on what makes their game their game, it saves a lot of time, and they can have assurance that it's solid(if needing customizing), which isn't a guarantee with a random pre-made UI pack on an asset store.
       
      There's still some bits to polish and refine, of course. Ensuring the options menu buttons better indicate what page you're on, finishing filling out the tooltips/descriptions of options and keybinds, and documenting the crap out of the code for it all. But it's pretty solid at this point and the remaining kinks are getting hammered out quite nicely.
       
      Tomorrow, I'll get into what's been up on the engine and tools side, and a look into the new Project Manager! Exciting!
      Laters!
      -JeffR
    • By JeffR in Torque 3D
         0
      Minor delay because I was feeling under the weather.
      Anyways! Part twooooo!
      Engine Work
      Predictably, a bunch of various improvement and fixes have gone in on the engine side of things. Various crash fixes, out of bounds fixes, etc.
      Beyond that, though some other nice stuff has happened. A big one was moving the asset importing logic up from the tools scripts into the engine. This was done for a few reasons.
      One was processing speed. The native C++ is obviously going to run faster than TS, so in general, the processing time for import has improved. But bigger than that is the ability to do in-place auto-importing.
      Similar to how, say, a TSStatic would reference a DAE file. If no cached.dts file was found, it would automatically run the import of the DAE file, and generate the cached.dts file. The assets system can now operate in the same way. In the event a non-asset loose file is utilized, we can try a multi-step process to handle that.
       
      We would first try and find any asset that utilizes that particular file, and if found, we just utilize that asset and update the TSStatic's fields to reflect it. If that fails, we then try and run an in-place auto-import. If the tools folder exists we even try and look up if a default import config is set in the editor preferences, otherwise we just use the default import settings. If the import is successful, we like above return the new assetId and that is updated in the TSStatic's fields. If all that fails, we can return back a placeholder assetId(in TSStatics' case, it's a noShape mesh) to indicate 'hey something went wrong with loading' to the level editor, while still retaining the original fields so debugging can be done to find out what went wrong without contaminating the setup.  
      We've been beating the system up on TSStatics and a few remaining issues are there, but it's proven to be rather fast and robust. And because it can defer to an importConfig, it's customizable to the project's needs, as well as utilizing Assimp for multiple other file formats like FBX.
      Once the last bits are plugged, we can then move on to implementing it for all the 'regular' types, shapes, images, materials, etc.
      This means that legacy projects being imported in will auto-process a ton of content into assets without needing to do any manual work, which is a huge timesaver, and non-legacy projects have it as a handy fallback for workflow efficiency.
      There was also various improvements to the PBR and IBL stuff to make the math more correct. It'll continue to get refined as we find assets and scenes that throw odd results, but it's pretty robust at this point, both for new, PBR-friendly assets:

       
      And even non-PBR legacy assets can end out looking pretty solid by just being tossed in with no additional work(with some bonus footage of the asset import process doing it's thang)
       
      Beyond that, some other various improvements/expansions on the engine side in no specific order:
       
      Shifted glow over to a full separate RBG image map, allowing a lot more artistic control Added the ability to in-place insert inspector fields and groups, allowing arbitrary editor hook ins(an example of this would be if ECS was a module, it could be dropped in and just inject the inspector group for components into an entity in the inspector without having to hard-code the fields. This allows flexible and dynamic inspector and editor behavior and opens up a LOT of cool options for the future Added ability for text edit fields to have placeholder text rather than needing to do that janky voodoo of setting it's text to some placeholder value and hoping nothing goes funky when typing Fixed CPU detection and reporting on 64bit windows(courtesy of Hutch) Functionality expansions to groundcover spawning Corrected some erroneous behavior with terrainAssets and terrainMaterialAssets so everything works together more predictably Added a guiVariableInspector field type 'Range' for slider support in inspectors Fixed OGL handling of probe baking - very important for linux and mac for obvious reasons Whole lotta cleanup of uninitialized vars which should help with stability in the engine by preventing unexpected default values Fixed an issue with Assimp loading where shape scales were off(courtesy of Mars) Added ability for shadows to work with translucent materials utilizing alpha test  
      And a sprinkling of other stuff like crashfixes, as mentioned prior with several contributors helping out on that end like Chaigler, Bloodknight, Tony, Hutch and Mars.
       
      There's a number of other stuff cooking at the moment getting worked on, but a few highlight bits that I'll actually be making PRs for over the weekend are performance oriented.
      Namely, the ability to, via preference variables(as sometimes you don't want it, or want to leave it up to the end user) to force on static object fading, as well as fading of local light(point and spot) and limiting how many local lights can be rendered at a time.
      These are pretty handy for lower-end machines because whatever standard tricks to help performance in rendering - reducing shadow resolution, not rendering shadows at all, LODs, etc - ultimately the fastest possible thing to render is absolutely nothing at all 😉
       
      So in action, that all looks like this:
      In the first part you can see that in the current setting, I have a maximum of 3 local lights allowed, so the others in the grid are culled, based on score which is a calculation of the lights' size, brightness and distance from the camera.
      Then as the camera pulls out you can see that the lights smoothly fade out based on the light fade setting. There's a start and end fade distance. So, say, fading doesn't start until the light is 50m away, and will fully fade out by 75.
      Then I flip on the object fade setting and we can see that now the static objects behave similarly. Based on their size, they can begin fading out sooner, so small clutter objects can be quickly culled from rendering without negatively impacting the fidelity of the scene up close, while larger objects are slower to fade out.
      Additionally, there's a setting that allows the defining of an object size considered large enough it shouldn't ever be culled like this, which is useful for landmark objects and the like.
      The idea being that these can be adjustable for the end user via some options menu settings, so people with weaker computers can have this stuff kick in fairly aggressively(in combination with other options to lower render cost) giving more control and options for making the game run better without negatively impacting those with powerful enough rigs to render everything at full blast.
      When I get these added in, I can then add options to the graphical options menu to facilitate this. The dev can set the desired baseline behavior, and the preference would permute it to be more aggressive as needed(or disabled entirely for an 'ultra' setting.
       
      Next, tools stuff!
       
      Tools Stuff
      The big one would be updates to the Asset Browser.


       
      There's a LOT to unpack here, so lets get into it.
      From the notations in the first image, we can see...well, a lot of stuff, so lets walk through it.
      First, you can note a modified layout from what was there before, which is, hopefully, overall clearer and easier to understand most of it at a glance. Refinements will continue, of course, but this should(especially the utility buttons at the top) be much more immediately legible.
      So, per the notations:
      New Asset
      Slap this to create a new asset in the current path, via a popup listing of all acceptable types. Can also just RMB in the main browser window.

       
      Import File
      Opens a 'open file' dialog to go rooting around for valid file types to import in as new assets into the current path. Can also do the same via Drag-n-Drop.
      Filter Options
      A popup that lets you easily toggle various filter options, like filtering by asset type.

       
      Auto-Import in Current Directory
      This one is a cool convenience feature. It'll run through the current directory and run an in-place auto-import on any loose files(files that don't have an associated asset that owns it). Another handy feature for importing of legacy projects. This button doesn't display if there are no non-asset loose files in the directory.
      Complex Search
      Ok, now we get into the FUN stuff.
      Before, search was pretty basic, conceptually. You typed something in, like say 'wood' and any asset with a matching name would be filtered for display in the main view.
      Now, however, we have complex search phrasing logic.
      Take a search string like this:
      What this would do is search in the current directory(and any child folders, of course) for any asset that is:
      A materialAsset Has a tag of 'wood' Has 'old' in it's name So the end result would be any asset that is, effectively, a material asset of a wood type with old in the name, like a hypothetical 'Old_Ship_Wood_Material' asset.
      Neat, right?
      This feeds back into the filter options above, so flipping on/off the type filters automatically updates the search phrase. Poking any of the 'tag' section entries from the second Asset Browser image likewise toggles in/out tags.
      It also can support multiple words per 'term', via comma separation.
      So if we did:
      It'd find any materialAsset with a tag of wood that has old Or new in the name.
      Pretty powerful, to be sure! But it gets niftier. Because as neat as complex searches like that are, having to flip the various tag and filter values on, and then type a relevent name to always look up, say a common set of materials you use is a pain every time.
      If only there was some way to save them...
       
      Save As Collection Set
      Oh wait, we totally can. If you slap that little button there, you get a name prompt, and then it saves the current search phase into a collection set of that name, which then shows up in the Collections section in the Asset Browser.
      You can then hit that item and it automatically fills out the search term for you with that saved one. So just like that we click a single item and we are back to seeing only materialAssets with the tag wood and old or new in the name.
      This can be pretty huge if you have collections of associated assets and can save a lot of time from having to constantly dig around to find them again.
      Basically, the general idea is it doesn't really matter where or how you ultimately organized your assets(or maybe you didn't organize them yourself at all like if you got an art pack), you should be able to find what you're looking for quickly and easily.
       
      Asset Preview Size Slider
      Not quite as cool as that, but still important, is the slider for the Asset Preview size. It...uh, adjusts the size of the Asset Previews. In fact, if you crank it all the way down to the smallest, it turns into a list for maximum browsability!

       
      Given that the plan is for the asset browser to replace the old Creator, Meshes and Library tabs in the Scene Window(with the ability to make the AB dock into that window so it doesn't always take up a big ol' chunk of your screen) this is very useful for being able to actually see what the heck you're working with in the current directory.
      Topically, but in the earlier updated AB walkthrough image, you can see not only the Tags section, but the Creator section. The ability to create new objects like how you do in the creator tab will all end up over there, so the Asset Browser will be a one-stop-shop for all your content browsing needs. You can even browse and spawn 'scripted' objects via their datablocks!
      So spawning behavior is consistent, and it'll work with dang-bloody-near everything. So yeah, pretty slick.
      Also, you may have noticed some colored outlining in the screenshots around the previews. The idea there is that different asset types would get different outline colors so even in list mode you can at a glance determine if it's a material asset, or shape asset or whatnot. Minimize confusion, maximize quick parsing of what content you're working with, you know.
      Topical to looks, there's some hard-to-read text and other funkiness in those shots. The editor theme will be cleaned up and more or less finalized based on what was learned with the BaseGame UI's theme, so it'll all be much more legible and visually consistent when we're done.
      For one last shot, A quick gander at the current Asset Import Window:

      (Remember, the hard to read text'll be fixed 😉 )
       
      Under normal circumstances, you won't likely even see this window unless you intentionally disable the 'Auto Import with Default Import Config' setting in the editor preferences, but just to highlight some cool logic the importer can do:
      I dragged and dropped just the shape file into the Asset Browser window. It then parsed the materials from the mesh, and added it to the importing list of assets, namely the material GothicCabinet_01, and automatically childed to the shape. Then, it noted that the material had an image assigned to it in the shape file, so it grabbed that, and childed it to the material. The image asset is that GothicCabinet_01_8-bit_Diffuse (sweet naming scheme, am I right?). It recognized this as an albedo image via the 'diffuse' suffix on the end, so when everything is imported, it'll automatically be slotted into the materials' albedo image slot. It then proceeded to scan the original file's directory for potentially related images based on naming convention. It looked for GothicCabinet_01_8-bit with a bunch of different, common suffxes on the end based on various types. It found a file that matched the name but had _NORMAL on the end, so it also brought that into the import and childed it to the material as well. Like our diffuse, it recognized the suffix so it knows it's a normal map, and will ultimately fill in that image into the materials' normalMap slot.  
      All of that is driven by the settings of the import configuration, and ultimately meant that when I said 'import this model' it automagically found and processed in a bunch of associated assets saving me from having to manually import in the images and assign them to the material myself, which over the course of development can save quite a lot of busywork time.
      I've shown off the general thrust of that before, but The UI is in a better space now to convey how things associate, and I thought it was worth re-iterating how 'smart' the import tries to be.
      In fact, you may note from the image that it considered the Shape file a 'duplicate asset'. This is because the naming convention of the shape and material collide as they both have the same name, which could cause reference problems. I can either let it auto-resolve the problem by, say, renaming the offending asset, or having it auto-append _Mat to the material name, etc based on import config options.
      And the entire import processing is logged now, so if anything happens that is unexpected, you can go back through the logs and see exactly what happened, where, and why.
      No system is infallible, but I'm definitely shooting for as robust as feasibly possible at least 😉
      Ok, wow! This part actually ended up being longer than I anticipated, so we'll be doing a part three where I cover the remainder bits, and then a whats-what of whats left!
      Exciting!
      See you tomorrow!
  • Show-Off

  • Built with Torque

×
×
  • Create New...