Replacing TorqueScript

[saindd]

How decoupled is TorqueScript from the engine itself? Is it just communicating via a common API, like a normal language binding, or is it more complicated? I was interested in dumping TorqueScript in favor of Lua.

[LukasPJ]

It's bound into the engine like a web of wtf's and glue.

Ripping it out is pretty hard and no one has done it so far. Adding another language on top of TorqueScript is what people usually do.

[MangoFusion]

Certainly possible, but keep in mind the whole SimObject system is closely coupled with the console system so you'll have to rip out or refactor a lot of that stuff.

As Lukas says you'll probably end up with still needing to leave TS in, though I think .net torque managed to rip most of it out.

[Rene Damm]

We (I) ripped it out for Torque 4 (a.k.a. "The Torque That Never Happened v2"). It wasn't a huge pain. We extended the engineAPI stuff and used it to interface to a .NET-based high-level part of the engine.

[Azaezel]

Happen to recall the general direction taken, given the discussion keeps coming up?

[saindd]

LukasPJ, what do you mean, adding another language on top of it? Is there some kind of "Adapter" for languages or multiple interpreters?

[LukasPJ]

@saindd, well an example could be the C# bridge that I'm working on.

This is an example of header and source written in C++/CLI.

With that kind of wrappers I can get C# scripts that look like this.

The engine I'm working with (Torque6) doesn't have an adapter so I made one for Engine->CLR calls : here

Torque3D has this which you should be able to bind with, alas I have no experience with it.

[Rene Damm]

Happen to recall the general direction taken, given the discussion keeps coming up?

 

engineAPI basically already has all the bits for interfacing Torque to whatever control layer you want. The current implementation making use of the console system is easily replaced with something else.

What we did initially was to just purge all the scripting wrappers that engineAPI generates internally and leave only straight DLL export functions for the various DefineEngineMethods etc. We then wrote a small tool in C# that took the XML output generated by engineAPI (there's a public C function that gives you a string in XML format that describes the entire supported API; forgot what it's called) and generated a C# wrapper for it. It simply loaded the DLL, called the function, and then parsed the output.

This we hooked into our build chain. I.e. we'd compile the engine DLL, then invoke the tool to generate/update the C# code and then built that into an assembly.

However, we later changed to a model where we no longer auto-generated the C# wrapper but rather hand-maintained it. This was because we had an additional abstraction layer in there that allowed to use both Torque's old C++ engine as well as a new XNA-based engine as a backend to the high-level C# code -- the latter being the part that the game developer was actually using (a component-based engine in pure C#).

[MangoFusion]

Interesting Rene... I always wondered how the engineAPI info was meant to be used. Before I couldn't really find a concrete explanation for the merits of such a system besides "better docs" or "better parameter type checking".

I investigated making use of the information myself for embedding but didn't find it useful enough since certain things seemed to be missing (I forget which, it was a while ago), memory management seemed a bit weird, and the whole process seemed a bit backwards if you weren't intending to use it to generate a C#, Java, or a basic module. Also no varargs :(

As a veteran torque dev, IMO one of the strengths of the older console system is I can quickly make scriptable functions or objects anywhere and rebuild the exe without having to worry about external tools in the pipeline. That seemed to be missing a bit in engineAPI. Thinking about it though I guess one could probably achieve something similar with engineAPI if they made use of a scripting API which relies on an FFI-like interface (like LuaJIT or Angelscript), though then again its probably not going to be good enough unless the DLL-with-a-processing-tool approach is used.

[buckmaster]

Whoah, it's @Rene Damm!

For reference, here are some previous discussions that are relevant. Should really stick all this info in a wiki page. EDIT: this, too. And my rant about TS.

[Rene Damm]

Whoah, it's @Rene Damm!

 

Hey Daniel :) Been coming round here occasionally to see what's going on with good old Torque. Thought I may as well contribute something, even if it's just random comments.

 

Interesting Rene... I always wondered how the engineAPI info was meant to be used. Before I couldn't really find a concrete explanation for the merits of such a system besides "better docs" or "better parameter type checking".

I investigated making use of the information myself for embedding but didn't find it useful enough since certain things seemed to be missing (I forget which, it was a while ago), memory management seemed a bit weird, and the whole process seemed a bit backwards if you weren't intending to use it to generate a C#, Java, or a basic module. Also no varargs :(

As a veteran torque dev, IMO one of the strengths of the older console system is I can quickly make scriptable functions or objects anywhere and rebuild the exe without having to worry about external tools in the pipeline. That seemed to be missing a bit in engineAPI. Thinking about it though I guess one could probably achieve something similar with engineAPI if they made use of a scripting API which relies on an FFI-like interface (like LuaJIT or Angelscript), though then again its probably not going to be good enough unless the DLL-with-a-processing-tool approach is used.

 

Aside from the benefits you mentioned, engineAPI was mainly meant to hide all the internals of how the interop works (such as the manual string parsing that the old console methods were doing) such that the interop can be done in whatever way you want without changing any of the script methods.

The "final" version also supported varargs. What's now in T3D is sort of engineAPI v0.8.

We eventually used it from C# without any generator tool. It requires manually adding some interop stuff but .NET makes that very, very easy. Still, once it was set up, the interop generator actually made for a really nice experience. Add something in C++, hit build, and continue writing in C# with VS/ReSharper seeing all the new definitions (and even documentation; we converted the docs added in C++ to XML docs in C#).

[JeffR]

Whoah, it's @Rene Damm!

 

Hey Daniel :) Been coming round here occasionally to see what's going on with good old Torque. Thought I may as well contribute something, even if it's just random comments.

 

Interesting Rene... I always wondered how the engineAPI info was meant to be used. Before I couldn't really find a concrete explanation for the merits of such a system besides "better docs" or "better parameter type checking".

I investigated making use of the information myself for embedding but didn't find it useful enough since certain things seemed to be missing (I forget which, it was a while ago), memory management seemed a bit weird, and the whole process seemed a bit backwards if you weren't intending to use it to generate a C#, Java, or a basic module. Also no varargs :(

As a veteran torque dev, IMO one of the strengths of the older console system is I can quickly make scriptable functions or objects anywhere and rebuild the exe without having to worry about external tools in the pipeline. That seemed to be missing a bit in engineAPI. Thinking about it though I guess one could probably achieve something similar with engineAPI if they made use of a scripting API which relies on an FFI-like interface (like LuaJIT or Angelscript), though then again its probably not going to be good enough unless the DLL-with-a-processing-tool approach is used.

 

Aside from the benefits you mentioned, engineAPI was mainly meant to hide all the internals of how the interop works (such as the manual string parsing that the old console methods were doing) such that the interop can be done in whatever way you want without changing any of the script methods.

The "final" version also supported varargs. What's now in T3D is sort of engineAPI v0.8.

We eventually used it from C# without any generator tool. It requires manually adding some interop stuff but .NET makes that very, very easy. Still, once it was set up, the interop generator actually made for a really nice experience. Add something in C++, hit build, and continue writing in C# with VS/ReSharper seeing all the new definitions (and even documentation; we converted the docs added in C++ to XML docs in C#).

 

Man, I hadn't even noticed that the EngineAPI stuff got that much attention in Torque 4. I'm gunna have to dig into that at some point.

[buckmaster]

Behold, a wiki page!

[chriscalef]

Hell of a nice wiki page!

Buckmaster +1

[rlranft]

Are you really going to get people using Lua instead of TorqueScript?

Well, I would, but it's not worth the effort to me to put it in there. TorqueScript works fine, warts and all, so why spend any time at all pulling it out just to plug something else in?

Oh, I'd probably enjoy working in Lua considerably more and there would overall probably be fewer WTFs per second. But personally I'm fine with using what's there.