End of March-ish Workblog 2023
Given that the rapid-fire releases have waned and we’re back to working on the primary feature release of 4.1, it’s good to get back onto regular work blogs.
And so, to that end of things, here. We. goooooo.
So how is 4.1 going? Getting there. As you know, we shifted the target for development of the engine for 4.1 from the ECS, to Editor/UI work.
So how’s it going? Not bad, overall. Lots of foundational work’s been chipped at previously with work done going into 4.0, but since it wasn’t “ready” it wasn’t frontline for the featurestuffs.
But, now that that IS the point, the actual work of “updating the editor suite to not be all old spaghetti” is chugging along.
So, lets get into talking about the work done so far, and what’s on the TODO!
Az and I had recently put up the 1.0 of the Objectives Module onto the Torque3DResources repository(you can get it here.)
Specifically, it’s a module for managing and utilizing Objectives in your game. Objectives being a more general-purpose name, but it can be used for any task a player must complete. Quests, Mission Objectives, yada yada.
We worked on it because we needed it for quests tracking in Catographer anyways, but obviously opening that up for beatups and further refinement by everyone else was an easy win.
Well that doesn’t sound like it has much to do with Editor or UI updates
Right, right. Back on track. One element that was very useful as a tester of the paradigm, but also just useful for our needs, was a custom drop-in editor tool.
Specifically, the ObjectivesModule within it contains a tools folder for the ObjectivesEditor, as seen below:
This works out quite nicely as a demonstrator for module-tools generally, of course. Being able to drop a module into the data directory and immediately get whatever tooling for the modules’ needs right then and there without needing additional copies is an obvious quality of life gain.
But beyond that, there’s the actual tooling utility itself.
One thing we’ve had in the engine and used lightly in various places previously was the “VariableInspector”.
What is it? It’s basically the existing GuiInspector used anytime you went to inspect an object, with all the display of the various fields and properties of the object.
Except for the VariableInspector, it was originally fashioned as a simple inspector for variables, per it’s namesake.
For 4.0, I’d expanded the functionality to inspecting objects, arbitrary fields, global variables, callback actions, and support for completely arbitrary field types.
It was certainly useful, and is used in stuff like the Editor Settings editor and a few other places.
But ultimately it was a duplicate Inspector class, so the next step - both in the interests of simplifying how many GUI controls we have, as well as simplifying usage of inspecting things generally - was to fold the ability to do arbitrary fields and field types into the GuiInspector directly.
And so now, you have what we’re doing in the ObjectiveEditor. Below is a code snippet showing off the onInspect callback function:
As you can see, we have the onInspect callback. This is called when any object is inspected(natch).
This is an important notion, though, because normally the GuiInspector doesn’t do that at all. It all handles the field logic internally with the statically-defined fields in the engine side for the class.
Now, though, we inspect the object, we get the onInspect callback, and then can go on the manage or create groups as well as add fields to said groups entirely in script.
There is no source class defining the logic or behavior of the ObjectiveInfo. In the module, that’s just a scriptObject with the namespace ObjectiveInfo defined. Once it’s inspected, we call through, and do the prompted logic.
Further, we can see pretty common types on the fields. Strings, bools, etc. But there’s an interesting one. “ObjectiveList”
Now, you may have surmised that that’s not a normal type in T3D, and you’d be correct. That type doesn’t exist. On the engine side.
Instead, what happens is when the GuiInspector is processing the fields, if it hits a type it is unfamiliar with, it will attempt to call down into the script for a build function for that type, allowing 100% custom field types to be implemented.
So, we do this function here:
And when that builds the field for us as part of the field population logic, we end out with a nice custom field type that is a dropdown that populates a list of Objectives the engine currently has loaded:
Pretty slick, right?
So that’s why this is important to the Editor and UI update work. A litmus test and showcase of more free-form handling and boosting the script-only viability of editor tools.
This will become increasingly important as we overhaul field-heavy editor tools like the Material or Particle Editors.
They currently use custom GUIs with lots of backend logic for all the binding behavior, but we’ll be updating them to utilize GuiInspectors with the script hooks as needbe for special field types for 4.1.
This will give us less special snowflake code, more standardized layouts, easier to expand or update tooling, and just generally be cleaner and more consistent an experience overall.
Editor Core work
Next on the discuss-a-roo for what’s currently baking in the oven at the moment, is the EditorCore module.
What is that?
Well, calm down a second and I’ll tell ya!
The EditorCore module is designed to rectify the whole old-spaghetti issue that currently plagues the whole of the tools folder.
If you’ve ever gone digging into the tools stuffs, you’ll realize it’s a big blob of auto-executing scripts, seemingly completely arbitrary execution orders of scripts and guis, multiple redundant re-executions of stuff like GuiControlProfiles, and then everything is duct taped together by just adding every single GUI and EditorTool to the EditorGUi control, smooshing everything together and making it almost impossible to edit or detangle without a ton of manual work.
So the EditorCore is a “clean slate” after a fashion, while also skipping the need to actually start the entire editor suite over, because that would be really, really dumb.
Instead, we’ll do a progressive update structure with a common core.
Specifically, the EditorCore will do the following:
- It acts as the central loading point for the entire tools suite. It always executes first, and all the common-use stuff like icons, common images, GuiControlProfiles, etc are handled in it. Since it is always executed first, you can rely on the fact that anything within it will be ready and loaded before any subsequent tolls are loaded. This will simplify the loading behavior considerably
- It will provide several standardized utility functions for common behaviors. Stuff like layout management, Menubar/Context Menu building, etc. This will cut out a lot of the mess we currently do with the menubars and RMB popup menus being defined inconsistently, and also all over the place.
- Help standardize theming, by letting us massively reduce how many random, floating GuiControlProfiles we have everywhere. The EditorCore profiles can be relied on to be there if the tools are loaded, and will be rebuilt from the ground up to be consistent. We’ll also look at theming support with them so tweaking colors and the like for better user experience is less of a nightmare.
Once the EditorCore is in place, we can then begin porting the existing tools to actual modules, simplify and standardize the hook-ins and execution code, and then doing any other thematic/format updates to the tools(like the aforementioned updating of the Material/Particle editors to use the Inspectors)
In this way, we can keep the vast majority of the existing functionality, but we can ensure all the tools get some TLC and brought up to a modern standard.
Nil’s Theme and style work
Topical to theming and style, Nils had done some work on a side-branch of the engine in his repo that does some very lovely touch-ups. It standardizes some of the colors and stylistic notions, and updates quite a number of the icons or images utilized throughout the editor.
Additionally, it has some nice expansions like the ability to snap certain windows to a layout. While the final implementation in devhead won’t be identical to this, you can expect a similar ability to lock certain windows and panes to a layout(and have it keep that layout next time the editor loads)
Work on updated AB
Tying into editor form and style updates, we’ve gotten a bunch of feedback with the Asset Browser, which is very good!
It seems like by and large, the layout and operation of the AB is pretty widely liked, but there’s room for improvement on clarity and functionality.
So I’ve been working on updates to the layout and behavior. Here’s an image of the WIP:
One thing that gets ‘lost’ in terms of immediate user experience affordances is how spawning/creating objects for the scene works.
While the drag-and-drop action is intuitive to a lot of people, it isn’t intuitive to everyone - or it could be intuitive if it was more apparent what was a spawnable item listing versus non-spawnable.
For example, one can drag a datablock listed in the AB into the scene and it will spawn the class associated to that DB. Drag a ItemDatablock, spawns an item with that datablock.
But an image is not spawnable and dragging and dropping does nothing.
So, while the functionality generally won’t change for drag-and-drop, We’ll be having a separate window/tab for “Creator” that is JUST spawnable assets and classes. If you want to spawn an item, waterblock, or light. You can go into the Creator tab and only spawnable items will be listed there.
Beyond that, once customizable layouts are working, it’ll be easy for you guys to put the Creator tab up by the Scene Tree like the old layout if that’s your fancy, or you can keep the Browser and Creator docked in a panel at the bottom of the screen, or whatever else makes the mose sense for you.
Additionally, per the image, you can see we’re going with icon buttons that also have text for the main action buttons. So for newer users not used to the iconography can still very easily find what actions they need to get their work done(this will be a common design thread throughout the editor going forward, incidentally).
And you can see some tweaks to the display of the ‘cards’ for the items listed in the browser, with a whole sub-line indicating the type for it. In conjunction with the border colorization we already had, there should be multiple ways a user can quickly identify an image from a material, or a material from a shape.
Work on updated importer
Another one that’s gotten a lot of excellent feedback is the Asset Import system. In it’s current form, it’s all processed in the engine, and since very few people have really fiddled with it, it’s a sort of ‘black box’ in terms of how it behaves.
While there’s a lot of customization settings in the Editor Settings:
It can be a pretty daunting wall of things you “have” to tweak. Plus some settings impact other settings and it’s hard to convey that well in this format.
So, in the interest of boosting maintainability, making it so other people can more readily fiddle with the importer behavior, and making it easier to customize AND understand, the Asset Import toolstuffs will be getting an overhaul as well.
Specifically, I’ve been experimenting with something like this:
What’s all this, then?
Well, it’s a WIP of the updated UI for the importer editor/designer.
Rather than a monolithic blob-class of executable logic, it’ll be shifted over to a “tasks” system. By which I mean, when you go to import a given file, it figures out the type of file, and then that type will have a list of “tasks” assigned to it.
Each task has a discrete set of settings that tell the task how to do it’s work.
For example, a RenameAssetTask simply establishes rules to rename the asset from the originating file name, since file names can use characters that asset names don’t support.
Once it knows the type and has the tasks for that type, it then just walks through them in-order, performing the informed task based on the settings.
This gives complete control over the person setting up the import pipeline without having to manually edit any code for sequencing.
Once the tasks for that item are done, the item is marked as processed, and - presuming all items are processed - final asset creation is performed.
At which point, the asset is loaded, registered and ready for use immediately.
Now, ideally, we have a good config out of the box so most people won’t need to fiddle with this at all, but the idea is that if you do - whether it’s because your artists have… interesting naming conventions, or there’s an entirely new type of file and asset you want to import - it’s easy to modify and expand.
Tasks will be script-defined objects with standard callback functions for them all, so adding new tasks or processing new asset types will be quite easy.
Now, topical to “new types of assets”, one thing that was confirmed to work(but ultimately not a good fit for the ObjectivesModule) was expanded behavior for ScriptAssets.
Originally a ScriptAsset was an asset for a script file(natch).
However, this is kinda ‘meh’ for utility, since most of the time scripts are either associated to other stuff(GUIs, shape constructor files, etc) or they’re standalone and need to be told specifically when to execute as part of the module/gameplay setup logic.
So, not especially useful most of the time
But then it occurred: why not just…let ScriptAssets be SCRIPTED assets?
In the same way we have “ScriptObject”, aka an object who’s properties and behaviors are defined entirely in script, you have the ability for ScriptAssets to do similar.
If you need management and tracking of a special asset type for something, but it doesn’t need special handling code drafted on the engine side, then you can define it as a new ScriptAsset type.
This can hook against the prior mentioned onInspect behavior to be able to easily draft in custom editor logic.
Additionally, it has standard callbacks for when the asset is initialized, reloaded or unloaded.
AND you can keep the pre-existing scriptFile association if needbe.
So you can have a full micro-ecosystem of a special type of “thing” via ScriptAsset, with full editor, resource management and execution tracking.
While still being able to lean on the Asset Dependencies system, easily finding stuff regardless of location with AssetId’s and being able to find by type with the AssetType.
They’ll even show up in the Asset Browser, and have callbacks for working with that.
Want a special data type asset for a maze generator ruleset, then be able to drag the asset in and spawn an object that uses that ruleset to spawn a maze on map load?
All pretty easy to set up by using the callbacks and interop provided.
There’s definitely a lot of room for cool stuff with that, and it also makes the drop-in-and-go-ness of modules with custom types and editors even more expandable, since it means you can often avoid needing to add a new asset type on the engine side.
Just drop in the module and bam. It’s ready to go now.
Marauder’s work with graphical updates
While not a focal point for 4.1(since the feature target for it is the UI/Editor updates) it’s worth observing that ongoing work is being done to keep updating the rendering side of thing.
Marauder had been doing work into updating some stuff, like expanding the global illumination behavior from just the IBL with probes, to an Screen-Space GI implementation. While there’s always more work to do, the work done thus far with it is certainly very promising:
It's subtle, but there's color bleeding happening with the bounced light from the curtains and the like. The next step is ensuring rays properly contribute to the AO for occluded areas like back behind said curtains and the technique'll be pretty well top-notch AND fully dynamic.
He’d also done some work to do more physically correct camera behavior, specifically a higher quality Depth of Field effect. While not used during gameplay too often, it can be very useful for framing certain shots or cutscenes:
And also technique update for shadows, to make them higher quality, for less resources and fewer irregularities:
You can see it even does penumbra and loss of sharpness the further the shadow is from the caster. This helps better ground objects and makes the shadows more realistic.
Additionally, there's updates to the physical bloom, so even when over-exposed and bright, it doesn't completely blow out detailing:
Now, as said, it’s not the focus, but if any of these techniques get finished out by release, then they’ll probably go in. But even if they miss the release target for 4.1, you can expect further fidelity refinements and new techniques to go in render-wise.
Now, to change gears, another module that’s juuuuust about ready to go in, with a few bits of refinement is another one that we’d made for Catographer.
Specifically, it’s a dialogue system that utilizes the ‘yarnspinner’ format.
If you’re unfamiliar, yarn, or yarnspinner, is a dialogue library/format standard that has been used in a buuuuunch of indie games.
It’s pretty well beat up and proven, and we needed a dialogue system for cats. So, the implementation to work with the *.yarn files was done.
Internally, it converts all logic to torquescript, so you can easily invoke functions, access global variables and do other evaluatable logic as part of the normal flow of it.
Conversations can have any number of speakers, and the dialogue bubbles will ‘find’ the speaker as long as the AI or Player has an NPCName field to match the speaker from the dialogue file.
It also supports ‘barks’, or non-conversational dialogue pop-ups, like if a shopkeep tries to call the player over as they walk past.
As said, a few bits to polish up, but you can expect that in the Torque3DResources in the neat future as well.
For future expansions, I’d like to have it more easily support different styles of UI(as is, it’s designed around ‘word bubble over head’) like being able to have the dialogue show across the bottom of the screen, have character portraits, etc, etc.
With a fairly consistent file format like yarn, it’s mostly just a question of ensuring flexibility of the UI/UX stuff(and possibly a fully integrated dialogue editor in the future).
Quick overview of TTR gameplan
Speaking of future prospects, if you’ve been in the discord, especially in the past months, you’ll probably have heard me mention ‘TTR’. Originally a UT99 mod, The Third Reich was a WW2 total conversion mod that emphasized “fun realism”. It had a pretty good sized community back in the day, and a good swath of content.
It was actually what got me into the gamedev scene when I was onboarded and as the original work from the project waned and faded off, I was put in charge of.
Unfortunately, being poopy baby-faced new developer that I was, I didn’t have the skills or know-how to meaningfully port a full-fat complete project to Torque back in the day.
So it’s just sorta been simmering in the backburner waiting for an opportunity to be ported and done justice.
Ok, why is this relevant? Aren’t you working on Catographer?
Excellent inquiry, reader-stand-in!
Indeed, far as commercial prospects go, Catographer is the main game project I’m working on. Very much the primary-focus of my time if it isn’t for T3D itself.
Plus, in the interests of maintaining the legacy of the original mod that was fun and free, I don’t wanna sell TTR for a price tag anyways.
So why would I bring this up at all?
Because what TTR can do for the T3D community is quite a lot, actually!
One thing I’d noted some months back in the discord as the discussion at the time covered modding, was that some of the best introduction to game development was the modding scene.
And while modding may not be quite as gigantic as it’s heyday because people can just grab a game engine and jump straight into their dream project, I think there’s a misjudgement on if that’s the best route for a complete newbie developer.
You see, what modding did, was it gave a new developer a fully functional, fully playable game.
No duh, right?
But this matters, because it means that what the developer themselves has to do is very, very small.
If you wanna learn how to map? You have gigabytes of assets already and maps to look at and reference from.
You don’t have to make or buy trees, and cars, and rock assets to put into your map just so you can learn the ropes.
Same with modelling, or texturing, or materials, or sounds.
You can focus on that tiny sliver of game development, drop it in, and immediately play with it in the context of a full project.
This minimizes how overwhelming starting out brand new to game development can be.
If you’ve ever watched tutorials for getting started - with any engine, really - the problem is it’s basically a DEEPEST LORE DUMP of the engine you’re trying to use.
You wanna make maps? Well you gotta have assets first. And code a game mode. And make sure you have a player to spawn. And that player has to have physics and movement. And input/keybinds. And, and and.
So even simple introduction tutorials tend to have quite a spread of obligatory knowledge, and for someone that has NO experience, it can be paralyzing.
So, What if you can remove a lot of that initial burden? Then tutorials can be purely focused on a specific thing, because there’s already a full game’s content there.
It’s also a good morale booster, because getting a gun model in, or new texture, can be part of a full, playable experience immediately.
It’s a small victory, but for someone just starting out, it can be a big validator that they’re going the right direction.
So, having a fully functional game that’s 100% designed for actual play, not just a random demo scene is a good foundation for newbies to build off of, rather than jumping and drowning in the ocean of gamedev.
So it’s just a moddable project?
Well, not JUST that.
As noted, it being a fully featured game, with a lot of existing content that’s just fun to play is the first step.
Once it’s there, then it can be made moddable.
This lets us have “first contact” tutorials emphasize modding TTR first as a ‘dip your toes’ deal. And once some foundational knowledge is learned, they can go on and start doing their own projects.
But beyond that, having a project that’s a sort of “The Engine’s Game” is good for marketing, drawing attention and a bunch of other stuff.
Well, in no specific order of priority, I see TTR as being able to provide the following for the engine and the community:
Newbie Developers ‘first contact’ tutorials
- Super small, focused tutorials for learning all the baby-step foundational knowledge of gamedev without needing other foundational knowledge to even get to the point of displaying the results helps keep new people “in”
A game people can play that is upfront about what engine it uses, bringing attention
- Similar to Unreal’s Unreal Torunament, or most any game from Valve for Source, having a game that’s up front in it’s association to the engine, and linkbacks and directed flow back to the engine and community is a good funnel. If people like playing the game, they may want to mod it. And if they mod it, they may just use the engine full-time for their personal projects
A Benchmark testbed.
- We can put camera markers and flight paths, and run canned benchmarks in the various levels which lets us find performance hotspots, or see how new tech fares in different environments
Demonstrates/Tests basically every feature of the engine
- The game had a bunch of different maps, and fairly in-depth gameplay. Indoor, outdoor. Forests, water. Vehicles, infantry combat. AI, teams. Times of day and weather and more. So if there’s a thing the engine can do, the game can show it off. And that means it can also do….
Feature Regression tests
- One thing that’s tricky to deal with is regression testing. While we have a few test maps like Outpost, it’s hard to be truely comprehensive for any given feature for the engine. Especially as we look into doing stuff like Components in the future.
- Having all the different features the engine can do in the various maps and gameplay allow much more comprehensive spread of cases and easier to see when something goes wrong. Performance regressions can also be detected with the aforementioned benchmarking
- As a multiplayer game, it’s a good tester for one of the most fundamentally core features of the engine, the networking.
New Feature Testing/Demonstrations
- As new features go in, like SSGI or Components or whatever, they can be tossed into a build for TTR as a good all-rounder way to test them in different scenarios, to ensure they work well with them all before being rolled in. This should help ensure stability of new roll-ins and minimize how much hotfixing happens.
Not bad. So it’ll be ready soon then?
Oh. Ha. Haha, aahhh. No. Not exactly.
To be specific, most of the OG art and content is basically ready and waiting for being rolled in and turned back into TTR, but on T3D.
But that takes time, and I have to ensure all the bits are formatted right, conversion from the old archive files or extracted content isn’t hideously mangled, and so on.
I also gotta finish up the Design Document and probably a roadmap for work on it so if people want to pitch in, they can and know what needs doing.
Plus, as noted, outside of engine work itself, my main priority is Catographer, at the moment.
Having said that, the utility TTR can provide to the engine and community is, in my opinion, quite apparent, so I’m not going to just let it sit in a dusty box for several more years.
My plan is to over the next months, slowly get all the pieces pre-arranged and positioned, with documentation for design and intent on it. Sorta like laying out all the bits before assembling your IKEA furniture.
Once that’s ready, and Catographer is punted out(fingers crossed, this summer!) I can spend some intermediate time after that goes out ripping through the work and getting TTR assembled in T3D and put out there.
Once it’s live, it’ll be easier for people to poke and prod everything to fill in any missing bits or make their own mods for it to expand upon it.
So while it’s not right around the corner, I wouldn’t say it’s far off, either. Just a matter of managing the timing and priorities.
So, uh, yeah. Wow, that was quite a bit of yapping, huh?
Buuuut, it’s been a hot minute since I did a workblog, and the release updates for 4.0 and the hotfixes aren’t exactly real replacements for these then either.
But now you’ve got the jist of the big bits that are cooking. Hopefully that’s got the ol’ game development appetite going
And if you want to help facilitate all this cool stuff and the engine as a whole, I DO have(even though I never remember/bother to mention it) a Patreon.
I’ll be working on updating it this week and providing more details, linking back against roadmaps, and do more to detail out what exactly the whole shindig with it and how it helps the engine.
But it does help. First and foremost helps cover all the hosting and service costs we accrue. But also so I can help feed Az so he doesn’t return to his true shoggoth form and bring about the end of at least 1 small country.
Quite important indeed
So, yeah. Every bit helps. But I also acknowledge that just throwing money with absolutely no return isn’t always appealing either. Budgets are tight for everyone these days, and even a token return is better than nothing.
I’ll be looking into Discord integration for patrons to get a nifty special role indicating that you’re donating, and when TTR comes out, me and Az have talked about some bits for that too. Like if you are a patron, you can get your name in for one of the AI Bot names.
Small, but fun little things like that.
Anywho. Lots of work to do, as you can probably guess from the above, so I’d better get back to it!
Thanks everyone, and happy dev’ing!