Jump to content

TAML for T3D


Recommended Posts

TAML is a persistence layer (and markup language) in use by T2D for saving resources like levels and animations. This page from the T2D wiki has a really good overview. I want to make it extra clear that TAML refers to both the persistence system itself, as well as a particular backend that looks like XML (Torque Application Markup Language, AFAIR). But the TAML persistence system is apparently able to persist via JSON and binary backends as well.


@LukasPJ has already implemented TAML in T3D in some form, which I'm not too familiar with but I'm sure he can elaborate on. @Hutch recently expressed some opinions on the use of TAML, which prompted me to create this thread. We should have a bit of a discussion about how we want to carry forward in this direction.


I'll write in more detail later, but I want to kick off the discussion by saying I'm very much in favour of separating code and data. Which means, TAML for data, and TorqueScript for code. At the moment, our .gui and .mis files, for example, are actually code, which is executed like code. I think separating them would allow for better editing and tooling. I'll finish for now by quoting Rene Damm, from some private correspondence we had a while ago which I've asked for his permission to share:

 

Torque’s serialization/loading system again is basically absent. Melv did some work here for T2D based on the work he did for T4, but in good old T3D that odd idea of serialization by generating script code still is the de-facto persistence mechanism (if that hasn’t changed in the MIT version). This leads to so many problems. Object references need to be by name and can only be resolved after loading. The script VM is dead slow so loading is extremely slow. And so on.


...


This, however, leads me to another issue. I always liked how “alive” things were in the Torque editor – your game is really running while you edit it. It’s fun. However, in the end, it’s not a good thing. It makes much more sense to separate play testing from editing and while Unity has some issues of its own here, it does this much better.


...


Finally, there’s the build pipeline – another concept that is sorely missing in Torque. In Torque, neither importing nor building is well-defined. Importing is sort of a per-asset-type kind of thing without any architectural backbone. And building basically equates to you somehow packaging your executable with the scripts and data. In Unity, building is essentially a final compile step with its own pipeline and processors. This allows targeting a metric ton of platforms from the same project data and again makes for a fully automated, deterministic, and repeatable process.

Link to post
Share on other sites

I love TAML, I think almost every aspect of TAML is great, and T3D already has something similar with the "TorqueScript"-serializer.. If someone wanted to, we could port that serializer to TAML and make TAML output TorqueScript instead for some reason that does not make sense!


But from what I can gather T3D community hates TAML :( That's just how the cookie crumbles!


Edit: I should mention that the PR in question has both XML, JSON and binary support. So at the very least, it's very good for sending SimObjects to e.g. webserver and back.


Edit: Should also mention that this does not affect current code at all... There's no disabled or changed features outside of the TAML stuff, so you can keep scripting as always without worrying about it.

Link to post
Share on other sites

Maybe if we renamed it, they wouldn't notice! Seriously though, I do think a rename is a good idea, so as not to confuse the persistence system with the file format. And, really, .taml should probably just be .xml. Maybe the system should be TASL (Torque Application Serialisation Layer) or TAS (Torque Asset Serialisation)?

Link to post
Share on other sites

So, I was originally quite on board with the idea of doing stuff like storing levels in TAML because of the separation - you could open a TAML file, read it into memory, view it, edit it, operate on it, without creating any actual objects. This seems like it would be a win for editors - but on second thoughts, wouldn't it make it difficult to write editors? Say you open a 3D scene in the editor, which is represented as a TAML file that describes a list of (nested?) objects. What do you do now? You've got to create graphical representations of all those objects, of course, and lay them out in 3D space in the way they're represented in the file, the same way they'll be created when you play the level. But how do those representations get created, without actually instantiating objects anyway? Do you have to write separate rendering logic to render a TAML scene, before actually creating the real scene? isn't that a ton of code duplication?


The obvious answer, of course, is that you load up a TAML file, create all the objects it describes, then edit them like we do currently, and same them all back out to TAML afterwards. But then what benefit does TAML have?

Link to post
Share on other sites

Okay here's the thing.. Torque3D already has a persistence layer, it can be found here. So what does TAML do differently? It generalizes that persistence layer. As described in the thread you linked initially, using e.g. XML is MUCH faster than executing a script file. Furthermore, you can now use serialize from/to any file format, which makes it easy to work with 3rd party apps (the main motivation behind TAML for T2D).


Lastly, having the missions in e.g. XML makes it easier for some world designers to understand the scene structure without having to learn TorqueScript, and you can validate an XML file without having to run the engine.


TAML and the existing persistence layer in essence works the same way. Save a SimObject to a file, read a SimObject from a file.

Link to post
Share on other sites
The obvious answer, of course, is that you load up a TAML file, create all the objects it describes, then edit them like we do currently, and same them all back out to TAML afterwards. But then what benefit does TAML have?

The not-so-obvious answer is something like this: You have an object that is spread across several levels. With TAML you can create an XML visitor to walk the level files and update parameters after changing a single instance. Did it in Three Step Studio. Was glorious.

Link to post
Share on other sites
The not-so-obvious answer is something like this: You have an object that is spread across several levels. With TAML you can create an XML visitor to walk the level files and update parameters after changing a single instance. Did it in Three Step Studio. Was glorious.

Could you elaborate on that? I don't quite understand what happened but it sounds intriguing :P

Link to post
Share on other sites
The not-so-obvious answer is something like this: You have an object that is spread across several levels. With TAML you can create an XML visitor to walk the level files and update parameters after changing a single instance. Did it in Three Step Studio. Was glorious.

Could you elaborate on that? I don't quite understand what happened but it sounds intriguing :P

 

Sounds like they had it where if you had an object like, say, a car for the player to drive, and you tweaked the parameters for it in one scene, you could have it go through the other levels and update it similarly there without having to edit each level yourself.


On a similar line, TAML/TASL would be useful for Prefabs. I'm not a fan of how they are currently(a glorified script file that contains a simgroup). It's an inefficient way to store stuff and requires you to generate additional simObjects, namely the simgroup, so that you can move stuff over to the prefab after load.

Having it in a TASL file means the prefab points to that file, and can just parse and create straight into it's internal list rather than needing to juggle around simgroups, and makes other convenience features around that easier.


I was looking at testing that out in the near future, as I feel it'd be a large step up for prefabs.

Link to post
Share on other sites

The "physics launcher" template had many physics-enabled objects in the scene like enemies and obstacles. If you wanted all obstacles of type "crate01" to have a lower damage threshold you would have to update every instance of that object in each scene file - this is best done with a "visitor" pattern object that can iterate over and modify entities by their type.

Link to post
Share on other sites

So, one of the reasons people want to keep the old persistence layer, is because they want to embed scripts in their .gui files... But some people have pointed out that this is a horrible malpractice that should be stopped with fire and sword.. What is the general opinion on this? I can make it possible to embed scripts in .Taml files but.. Ew..

Link to post
Share on other sites

Option good, mandatory bad. Least when it comes to folks in mid-stride. (Suppose if we're talking 4.0 there's a bit more give there.)


Would say for least disturbance from an intermediary standpoint... .gui and .mis files remain executed, with a .taml load being an (optional) part of the execution. Of course, at that point, it'd actually underscore gui specific script goes in gui files, mission specific script in .mis ect ect. Not really a bad thing if folks are coming at it from the perspective of segregated out drop-ins....

Link to post
Share on other sites

No. No script in TAML files. That seems like an awful idea that we will regret later. I can't see any valid use case for it unless a script is actually part of your data - for example, the way that spawn spheres store a little script fragment to be executed when they spawn an object.

Link to post
Share on other sites

I suspect we're talking past one another in that reguard dan. Talking replacing entries in .mis or .gui files that fall within:

//--- OBJECT WRITE BEGIN ---
//--- OBJECT WRITE END ---

with

TamlRead(relevantguidata.taml);

 

Not flat out replacing what a .gui or .mis file means at a fundamental level. That? That would flat out put me in the 'hate camp'.

Link to post
Share on other sites

Oh sorry, I wasn't responding directly to you. Yeah, that's basically how I see it at this point, though it still begs the question of why we'd want to use TAML at all if all it does is exactly the same as TS serialisation.


Actually, a good example of why TAML might be a good idea is the way we handle mis files currently. We have to do manual text parsing of the TS syntax to extract the mission info for the main level, which is kind of disgusting. If we could load up a TAML file as an abstract entity (an XML document tree or whatever), without creating a bunch of SceneObjects like you do when you exec a mis file, then we could traverse down into the mission info object and get the information in a nicer manner.


As for replacing what a mis or gui file means fundamentally, I'm not sure I follow you. What do they mean fundamentally? Nothing, really. They're just script files that create objects.

Link to post
Share on other sites

As for replacing what a mis or gui file means fundamentally, I'm not sure I follow you. What do they mean fundamentally? Nothing, really. They're just script files that create objects.

 

Well, that's just it, the perception that a .gui or .mis file is for asset creation is correct, but incomplete. There's several examples of files that both define structure and behavior. While my own default mode is to split such things apart for my own use, that doesn't mean I'd get behind breaking things if they aren't required to be.

 

it still begs the question of why we'd want to use TAML at all if all it does is exactly the same as TS serialisation.

 

Honestly? It puts us that much closer to eating T2D, and the way that's been going lately, folks from there might very well end up needing to look for a new home themselves...

Link to post
Share on other sites

I'd like to point out that TAML isn't a hypothetical concept - it is complete and working. If you're not sure how it would fit in, just fire up T2D and have a look. It should work the same way here. There is no need to serialize an object or object collection in the same file with its objects' "code-behind" (for lack of a better term). It is pretty clean and it makes sense. I personally get really tired of thinking to myself "hey, where the hell is this callback defined?" only to finally locate it in some .gui file somewhere because there is both a PlayGUI.gui and PlayGUI.cs file - and I expect the "functional" stuff to be in the .cs file.

Link to post
Share on other sites
  • 2 months later...

Hi,


Few days ago I downloaded the engine and started looking through TorqueScript code to more idea how the engine works. I'm still very fresh but I saw this discussion on TAML and thought this might be relevant:


From what I found out in my investigation of scripts, is that there is also tons of stuff in core/ that could be implemented as TAML. This might result in faster engine initialization and definitely better learning curve as there would be less code to look at.


Particularly this files:

core/scripts/client/postFx/* (most stuff)
core/scripts/client/lighting/* (most stuff)
core/scripts/client/audioAmbiences.cs
core/scripts/client/audioDescriptions.cs
core/scripts/client/audioEnvironments.cs
core/scripts/client/commonMaterialData.cs
core/scripts/client/materials.cs
core/scripts/client/renderManager.cs
(^ perhaps DiffuseRenderManager could be implemented in TAML)
core/scripts/client/scatterSky.cs
core/scripts/client/shaders.cs
core/scripts/client/terrainBlock.cs
core/scripts/client/water.cs

My concern is that this would require to split every object to its own TAML file, causing shitload files to load and potential IO performance problems. Plus all that objects would still be global. But maybe tweaking some of this files on C++ side to be SimGroups or adding composition syntax to TAML (property = new Object) would solve the problem.

Link to post
Share on other sites
Hi,


Few days ago I downloaded the engine and started looking through TorqueScript code to more idea how the engine works. I'm still very fresh but I saw this discussion on TAML and thought this might be relevant:


From what I found out in my investigation of scripts, is that there is also tons of stuff in core/ that could be implemented as TAML. This might result in faster engine initialization and definitely better learning curve as there would be less code to look at.


Particularly this files:

core/scripts/client/postFx/* (most stuff)
core/scripts/client/lighting/* (most stuff)
core/scripts/client/audioAmbiences.cs
core/scripts/client/audioDescriptions.cs
core/scripts/client/audioEnvironments.cs
core/scripts/client/commonMaterialData.cs
core/scripts/client/materials.cs
core/scripts/client/renderManager.cs
(^ perhaps DiffuseRenderManager could be implemented in TAML)
core/scripts/client/scatterSky.cs
core/scripts/client/shaders.cs
core/scripts/client/terrainBlock.cs
core/scripts/client/water.cs

My concern is that this would require to split every object to its own TAML file, causing shitload files to load and potential IO performance problems. Plus all that objects would still be global. But maybe tweaking some of this files on C++ side to be SimGroups or adding composition syntax to TAML (property = new Object) would solve the problem.

 

You'd prolly just throw it in a SimGroup as you suggested. Like "WaterGroup" "TerrainBlockGroup" and then it could be thrown into a single TAML file. And yes TAML is significantly faster than TorqueScript.

Link to post
Share on other sites
Hi,


Few days ago I downloaded the engine and started looking through TorqueScript code to more idea how the engine works. I'm still very fresh but I saw this discussion on TAML and thought this might be relevant:


From what I found out in my investigation of scripts, is that there is also tons of stuff in core/ that could be implemented as TAML. This might result in faster engine initialization and definitely better learning curve as there would be less code to look at.


Particularly this files:

core/scripts/client/postFx/* (most stuff)
core/scripts/client/lighting/* (most stuff)
core/scripts/client/audioAmbiences.cs
core/scripts/client/audioDescriptions.cs
core/scripts/client/audioEnvironments.cs
core/scripts/client/commonMaterialData.cs
core/scripts/client/materials.cs
core/scripts/client/renderManager.cs
(^ perhaps DiffuseRenderManager could be implemented in TAML)
core/scripts/client/scatterSky.cs
core/scripts/client/shaders.cs
core/scripts/client/terrainBlock.cs
core/scripts/client/water.cs

My concern is that this would require to split every object to its own TAML file, causing shitload files to load and potential IO performance problems. Plus all that objects would still be global. But maybe tweaking some of this files on C++ side to be SimGroups or adding composition syntax to TAML (property = new Object) would solve the problem.

 

You'd prolly just throw it in a SimGroup as you suggested. Like "WaterGroup" "TerrainBlockGroup" and then it could be thrown into a single TAML file. And yes TAML is significantly faster than TorqueScript.

Yes. "Level" .taml files are just like .mis files, really - just more greater-than/less-than signs and fewer semi-colons. Again, look at a T2D GUI .taml file and a T3D .gui file and you'll see that they are effectively just decorated differently. And a "level" is a simgroup. So you could wrap all of those object initialization data files in a single "environment.taml" file and run - as long as there is no associated object scripting defined. If there is, you'll still have those .cs files but they'll only hold the script code and not the object state information. Object instantiation is still time-consuming, as we discovered in the Tower Defense Template for 3SS - particles, units, and projectiles were best kept in pools so they didn't have to be created and destroyed in game-time where the object creation coupled with creation of all of its behavior instances could bog things down when many objects had to be cranked out quickly. So TAML really addresses some speed issues with processing the files but still doesn't change the way that objects are created.


And to be honest I have trouble seeing why the .taml file format is being confused with the TAML serialization system. Do people have this same confusion surrounding XML, XAML, or HTML?

Link to post
Share on other sites

XML, and HTML don't also have a serialisation system named the same thing. Also, TAML stands for 'Torque app markup language', which doesn't sound like a serialisation system to me. It sounds like a file format. Which is why I'd push to rename TAML in T3D.

Link to post
Share on other sites

Still not sure what's "confusing" about it. It's just a file format.


What is the serializer for JSON called? The one I use is just called JSONSerializer....


What is the XAML serializer called?


If it needs a cute name, call it Torquey the TAML Serializer. I guess I'm just not up for spending the effort....

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...