Replacing TorqueScript with Lua

[jamesonj]

I just started this small project: replacing ugly TS with a more versatile and easy to use script language. I have started with Lua, because i have experience with it, but i plan to implement AngelScript in the future.

Is this something that i should work on the side, or prepare a PR?

[Duion]

If you can do it properly, why not, but I guess it will not be that easy.

[JeffR]

Well, we'd only really integrate a PR if it was more or less "production ready".

So if you're still doing dev work on it, I'd say hold off on the PR. but if you ever get it completed, I'd say by all means, PR it. We'd have to decide if we wanted to actually gut TS or not, but I certainly feel it'd be worth you putting it forward for consideration!

[buckmaster]

For reference.

[eugene2k]

As it happens I also wanted to try and make Lua an option for torque. So here I am today, looking at all the *.cs files and wondering: wtf? :)

I'm not familiar with torque as an engine, but it seems to me kind of backwards to have all the subsystems initialized in the scripts. It makes no sense to me why Torque does that. Why is all this stuff done in the script as opposed to doing it in the engine internally? If you want to disable sound or physics in the engine, isn't it much easier to do so at compile time?

[rlranft]

Okay, you disable sound at compile time. Now you want to turn sound back on. Recompile? What game does this? In almost any game you can just go to a menu somewhere and disable sound - chances are, this setting is saved to a configuration file somewhere (declarative script at minimum - i.e. a .ini file). This one feature is literally the most common thing to do in script.

The larger question is this: Why would you leap to replace a scripting language that you don't know for one that you do when you have never even seen the engine code before? In any game engine this would intuitively be perceived as a non-trivial task - multiplied in complexity by not being familiar with the target codebase. You're not the first person to give this a try - and the scripting language is still TorqueScript. The effort is much greater than the payoff here.

Don't get me wrong - I prefer Lua to TorqueScript. However, TorqueScript is pretty C-like and Lua is pretty C-like, so learning TorqueScript seems the path of least resistance here.... If you want to know just how deep the "change the language" rabbit-hole is, ask LukasPJ - he's in the process of adding C# as an optional scripting language and he's been at it for a long time now.

[eugene2k]

"Disable the sound subsystem" vs "mute the sound". There's a serious difference between the two. In fact I don't think there is a game out there that disables the sound subsystem altogether at runtime (hell, I don't think there's a game out there that disables the sound subsystem period)

"The larger question is this" - not really. It's a learning experience for me. I'm interested in knowing how the scripting subsystem works, I know there's a number of people who would like to have Lua as an alternative, myself included, so the most straightforward thing to do would be to set this as a goal and try to analyze the code involved. After all it's unlikely that I'll find out anything about the internals of the scripting subsystem if I try making a game in Torque or if I add a new shader.

[Azaezel]

"Disable the sound subsystem" vs "mute the sound". There's a serious difference between the two. In fact I don't think there is a game out there that disables the sound subsystem altogether at runtime (hell, I don't think there's a game out there that disables the sound subsystem period)

 

You disable the sound at runtime in the process of swapping sound subsystems, like openAL to Fmod, for instance. Physics aren't swappable, and graphics are only checked with a fallback. Think of the engine as a dll. (it's pretty much always been a glorified self-executing dll even *before* the project manager came along and made it explicit.)

That tangent out of the way, study:

Pretty much all of the ones https://github.com/GarageGames/Torque3D/pulls?q=is%3Apr+author%3Ajamesu+is%3Aclosed that talk about the console for deep guts alterations,

https://github.com/GarageGames/Torque3D/pull/887 for the two methods for binding script methods to source method execution *

https://github.com/GarageGames/Torque3D/blob/development/Engine/source/T3D/projectile.cpp#L107 for source to script callbacks *

https://github.com/GarageGames/Torque3D/blob/development/Engine/source/T3D/projectile.cpp#L197 for source to script and Inspector GUI bindings *

*things you run across normally in the process of tweaking the engine for your personal game needs.

[rlranft]

After all it's unlikely that I'll find out anything about the internals of the scripting subsystem if I try making a game in Torque or if I add a new shader.

 

This is true - but my point is, it's better to poke around a bit before trying to rip the guts out of almost every system in the engine and replace it with something else. And I agree - Lua is a "better" language, I'm not discounting that in any way. I just believe you are gravely underestimating the magnitude of the task you have chosen. I've been fiddling with this hideous beast for about 11 years now, and I can tell you Torquescript is deeply embedded in almost everything. Setting small, attainable goals and achieving them is preferable to deciding to climb Mt. Everest in your gym shorts....

And the purpose to having the subsystem initialization requested at load time by script is so that you load the ones you want. If you don't want it initialized just remove it from the startup script - problem solved. Now you don't have to have a version of the engine built just for every possible system combination....

[eugene2k]

And the purpose to having the subsystem initialization requested at load time by script is so that you load the ones you want. If you don't want it initialized just remove it from the startup script - problem solved. Now you don't have to have a version of the engine built just for every possible system combination....

Well, that seems to be the only logical reason to have it there. Only I doubt that's very efficient. How hard is it to reconfigure the project, really? In CMake all you need to do is set some variables and then execute two commands. And that's it, you're set: you can now do what you need to do in the script while also having an engine that is custom-fitted to your needs. No redundant code sitting in the memory and never being used, faster loading times. I see only benefits in this approach.

 

You disable the sound at runtime in the process of swapping sound subsystems, like openAL to Fmod, for instance. Physics aren't swappable, and graphics are only checked with a fallback.

And yet both physics and graphics need to be initialized in the script. As for swapping the sound engines: I agree that you need to reinitialize sound after doing that, however, personally I would have put a function in the script that specifically does this, instead of having something like this:

/* from Templates/Empty/game/scripts/main.cs */
  // Init the physics plugin.
  physicsInit();

  // Start up the audio system.
  sfxStartup();

  // Server gets loaded for all sessions, since clients
  // can host in-game servers.
  initServer();

  // Start up in either client, or dedicated server mode
  if ($Server::Dedicated)
     initDedicated();
  else
     initClient();

That tangent out of the way, study

Thx for the links. My brain shut down when I first looked at the DefineConsole/Engine macros. Buffer overflow :D

[rlranft]

Well, that seems to be the only logical reason to have it there. Only I doubt that's very efficient. How hard is it to reconfigure the project, really? In CMake all you need to do is set some variables and then execute two commands. And that's it, you're set: you can now do what you need to do in the script while also having an engine that is custom-fitted to your needs. No redundant code sitting in the memory and never being used, faster loading times. I see only benefits in this approach.

I think you missed my point again. If you initialize the subsystems you want at run-time via script you only load the subsystems you want and so there is no "redundant code sitting in memory never being used." It just doesn't load the stuff you don't ask for. How will you allow your end-user to select the subsystems they want? Provide a special version of your game executable for each variant? Like Az said - T3D supports DirectSound and OpenAL for sound subsystems, so you provide the user the ability to select between them (or the null device, which initializes nothing). The same for renderers (though at the moment this is still mostly a moot point in T3D - Windows just uses Direct3D and Linux just uses OpenGL). 3D Game Engine Programming (while not the greatest book out there) shows a perfectly viable (and not uncommon) approach to creating your subsystems as .dlls that can be loaded, swapped, and unloaded at runtime. T3D doesn't use this particular approach, but the internal subsystem constructed in memory is the one you ask for and no other is created.

[eugene2k]

Why would the end-user need to select a different subsystem, though? The days when OpenGL handled stuff differently than Direct3D are pretty much dead and gone. Both APIs nowadays support essentially the same features. I think it's the same with sound APIs. And further, I said it in the previous post already, but if you want to offer the user a way of switching between two subsystems, it can be done with an in-engine function that is geared specifically for that, instead of a collection of functions.

[rlranft]

Yes, OpenGL and Direct3D are achieving feature parity - that does not mean you can just throw the same data structures at both and get the same results. Likewise, DirectSound and OpenAL have wildly different APIs to achieve the same results.

Look man, I'm not making this up - go check out UnrealEngine, or CryEngine. This is industry standard practice. I'm not trying to justify it - you asked why, I was just trying to answer the question.

But do it however you like. You obviously have something in mind, so I'll stand aside.

Keep us apprised of your progress - I would love to use Lua in T3D, so I eagerly await your success.

[eugene2k]

Ehh... Not saying I will be able to get Lua integrated. I set my goals high, so that I could achieve more by the time I give up on the goal :)

And thanks for responding. I'm not trying to argue either, just trying to figure out if I'm missing something in my line of thinking.

How should I say this... I'm primarily studying Torque's architecture, since there's no design manual and no comments in the code as to why something was done one way and not the other. Just by looking at the code of a shipped commercial product one can't tell whether a certain part of the code was a conscious design decision or just someone being lazy or if this was a design decision born out of some concrete need instead of an imaginary one.

[JeffR]

My recommendation would probably be to try and implement lua alongside TS as a starter.

The DefineEngineMethod macros and the like are designed to act as the interop between whatever language you want, they're just currently geared towards TS. In theory you could modify them to work with any language you want.

Rather than stripping out what's there and does the same job just with a different language, I'd probably suggest trying to copy them and just make a permutation that works with lua alongside the existing one.

This gives you the learning benefit of implementing stuff yourself, so you learn it as you go, but you don't lose your frame of reference - in this case a working script interop. So if you hit a part you can't get working, you can just walk through the TS side of things to see how it works there and compare why yours isn't.

[LukasPJ]

I've (almost) implemented C# into Torque6. Torque6 =/= Torque3D, but the base architecture is very alike.

I haven't read most of this discussion, but as JeffR said, I implemented it alongside TorqueScript, prioritizing C# so they'd be faster.

The branch is kind of big: https://github.com/lukaspj/Torque6/tree/C%23-Bridge but the general idea was to simply make a C-Interface for outgoing calls, and then for ingoing (i.e. C#->C++) I wrote C functions for these methods and simply DLL-imported them in C#. It'd be a lot different in Lua, but perhaps you could use some of it (such as the call C# or TS code).

I think these two commits might be of interest, they are where I make the engine call C# instead of TorqueScript:

https://github.com/lukaspj/Torque6/commit/aea65470b4fa53612421db862538a9ff4bcd27fd

https://github.com/lukaspj/Torque6/commit/679f765429d14c349499b8e3043693a22ab8db6b

[rlranft]

With Lua you could use Luna to make the task easier - it's a short template-based header chunk that is pretty easy to use:

/* 
 * File:   luna.h
 *
 * Created on November 15, 2013, 9:58 AM
 */
#pragma once
 
#include "lua/lua.hpp"
#include  // For strlen
 
template < class T > class Luna {
public:
 
	struct PropertyType {
		const char     *name;
		int             (T::*getter) (lua_State *);
		int             (T::*setter) (lua_State *);
	};
 
	struct FunctionType {
		const char     *name;
		int             (T::*func) (lua_State *);
	};
 
	/*
	@ check
	Arguments:
	* L - Lua State
	* narg - Position to check
 
	Description:
	Retrieves a wrapped class from the arguments passed to the func, specified by narg (position).
	This func will raise an exception if the argument is not of the correct type.
	*/
	static T* check(lua_State * L, int narg)
	{
		T** obj = static_cast (luaL_checkudata(L, narg, T::className));
		if (!obj)
			return NULL; // lightcheck returns NULL if not found.
		return *obj;		// pointer to T object
	}
 
	/*
	@ lightcheck
	Arguments:
	* L - Lua State
	* narg - Position to check
 
	Description:
	Retrieves a wrapped class from the arguments passed to the func, specified by narg (position).
	This func will return NULL if the argument is not of the correct type.  Useful for supporting
	multiple types of arguments passed to the func
	*/
	static T* lightcheck(lua_State * L, int narg) {
		T** obj = static_cast (luaL_testudata(L, narg, T::className));
		if (!obj)
			return NULL; // lightcheck returns NULL if not found.
		return *obj;		// pointer to T object
	}
 
	/*
	@ Register
	Arguments:
	* L - Lua State
	* namespac - Namespace to load into
 
	Description:
	Registers your class with Lua.  Leave namespac "" if you want to load it into the global space.
	*/
	// REGISTER CLASS AS A GLOBAL TABLE 
	static void Register(lua_State * L, const char *namespac = NULL) {
 
		if (namespac && strlen(namespac))
		{
			lua_getglobal(L, namespac);
			if (lua_isnil(L, -1)) // Create namespace if not present
			{
				lua_newtable(L);
				lua_pushvalue(L, -1); // Duplicate table pointer since setglobal pops the value
				lua_setglobal(L, namespac);
			}
			lua_pushcfunction(L, &Luna < T >::constructor);
			lua_setfield(L, -2, T::className);
			lua_pop(L, 1);
		}
		else {
			lua_pushcfunction(L, &Luna < T >::constructor);
			lua_setglobal(L, T::className);
		}
 
		luaL_newmetatable(L, T::className);
		int             metatable = lua_gettop(L);
 
		lua_pushstring(L, "__gc");
		lua_pushcfunction(L, &Luna < T >::gc_obj);
		lua_settable(L, metatable);
 
		lua_pushstring(L, "__tostring");
		lua_pushcfunction(L, &Luna < T >::to_string);
		lua_settable(L, metatable);
 
		lua_pushstring(L, "__eq");		// To be able to compare two Luna objects (not natively possible with full userdata)
		lua_pushcfunction(L, &Luna < T >::equals);
		lua_settable(L, metatable);
 
		lua_pushstring(L, "__index");
		lua_pushcfunction(L, &Luna < T >::property_getter);
		lua_settable(L, metatable);
 
		lua_pushstring(L, "__newindex");
		lua_pushcfunction(L, &Luna < T >::property_setter);
		lua_settable(L, metatable);
 
		for (int i = 0; T::properties.name; i++) { 				// Register some properties in it
			lua_pushstring(L, T::properties.name);				// Having some string associated with them
			lua_pushnumber(L, i); 									// And a number indexing which property it is
			lua_settable(L, metatable);
		}
 
		for (int i = 0; T::methods.name; i++) {
			lua_pushstring(L, T::methods.name); 					// Register some functions in it
			lua_pushnumber(L, i | (1 << 8));						// Add a number indexing which func it is
			lua_settable(L, metatable);								//
		}
	}
 
	/*
	@ constructor (internal)
	Arguments:
	* L - Lua State
	*/
	static int constructor(lua_State * L)
	{
		T*  ap = new T(L);
		T** a = static_cast(lua_newuserdata(L, sizeof(T *))); // Push value = userdata
		*a = ap;
 
		luaL_getmetatable(L, T::className); 		// Fetch global metatable T::classname
		lua_setmetatable(L, -2);
		return 1;
	}
 
	/*
	@ createNew
	Arguments:
	* L - Lua State
	T*	- Instance to push
 
	Description:
	Loads an instance of the class into the Lua stack, and provides you a pointer so you can modify it.
	*/
	static void push(lua_State * L, T* instance)
	{
		T **a = (T **)lua_newuserdata(L, sizeof(T *)); // Create userdata
		*a = instance;
 
		luaL_getmetatable(L, T::className);
 
		lua_setmetatable(L, -2);
	}
 
	/*
	@ property_getter (internal)
	Arguments:
	* L - Lua State
	*/
	static int property_getter(lua_State * L)
	{
		lua_getmetatable(L, 1); // Look up the index of a name
		lua_pushvalue(L, 2);	// Push the name
		lua_rawget(L, -2);		// Get the index
 
		if (lua_isnumber(L, -1)) { // Check if we got a valid index
 
			int _index = (int)lua_tonumber(L, -1);
 
			T** obj = static_cast(lua_touserdata(L, 1));
 
			lua_pushvalue(L, 3);
 
			if (_index & (1 << 8)) // A func
			{
				lua_pushnumber(L, _index ^ (1 << 8)); // Push the right func index
				lua_pushlightuserdata(L, obj);
				lua_pushcclosure(L, &Luna < T >::function_dispatch, 2);
				return 1; // Return a func
			}
 
			lua_pop(L, 2);    // Pop metatable and _index
			lua_remove(L, 1); // Remove userdata
			lua_remove(L, 1); // Remove [key]
 
			return ((*obj)->*(T::properties[_index].getter)) (L);
		}
 
		return 1;
	}
 
	/*
	@ property_setter (internal)
	Arguments:
	* L - Lua State
	*/
	static int property_setter(lua_State * L)
	{
 
		lua_getmetatable(L, 1); // Look up the index from name
		lua_pushvalue(L, 2);	//
		lua_rawget(L, -2);		//
 
		if (lua_isnumber(L, -1)) // Check if we got a valid index
		{
 
			int _index = (int)lua_tonumber(L, -1);
 
			T** obj = static_cast(lua_touserdata(L, 1));
 
			if (!obj || !*obj)
			{
				luaL_error(L, "Internal error, no object given!");
				return 0;
			}
 
			if (_index >> 8) // Try to set a func
			{
				char c[128];
				sprintf_s(c, "Trying to set the method [%s] of class [%s]", (*obj)->T::methods[_index ^ (1 << 8)].name, T::className);
				luaL_error(L, c);
				return 0;
			}
 
			lua_pop(L, 2);    // Pop metatable and _index
			lua_remove(L, 1); // Remove userdata
			lua_remove(L, 1); // Remove [key]
 
			return ((*obj)->*(T::properties[_index].setter)) (L);
		}
 
		return 0;
	}
 
	/*
	@ function_dispatch (internal)
	Arguments:
	* L - Lua State
	*/
	static int function_dispatch(lua_State * L)
	{
		int i = (int)lua_tonumber(L, lua_upvalueindex(1));
		T** obj = static_cast < T ** >(lua_touserdata(L, lua_upvalueindex(2)));
 
		return ((*obj)->*(T::methods.func)) (L);
	}
 
	/*
	@ gc_obj (internal)
	Arguments:
	* L - Lua State
	*/
	static int gc_obj(lua_State * L)
	{
		T** obj = static_cast < T ** >(lua_touserdata(L, -1));
		T* instance = *obj;
		if ((*obj)->isPrecious)
			return 0;
 
		if (obj && *obj)
			delete(*obj);
 
		return 0;
	}
 
	static int to_string(lua_State* L)
	{
		T** obj = static_cast(lua_touserdata(L, -1));
 
		if (obj)
			lua_pushfstring(L, "%s (%p)", T::className, (void*)*obj);
		else
			lua_pushstring(L, "Empty object");
 
		return 1;
	}
 
	/*
	* Method which compares two Luna objects.
	* The full userdatas (as opposed to light userdata) can't be natively compared one to other, we have to had this to do it.
	*/
	static int equals(lua_State* L)
	{
		T** obj1 = static_cast(lua_touserdata(L, -1));
		T** obj2 = static_cast(lua_touserdata(L, 1));
 
		lua_pushboolean(L, *obj1 == *obj2);
 
		return 1;
	}
};

I did not write Luna - it's available from http://www.lua.org somewhere in all of the binding discussion links...

For examples of it's usage, see https://github.com/RichardRanft/LuaVMTool

[buckmaster]

Since we were talking about engine startup I'm going to mention t3d-bones, which contains a very minimal example of starting up the engine. My 'minimal' I mean it still has 1MB of scripts in the sys/ directory, but at least it's a bit more approachable. Also, since credit is due: Michael Hall is responsible for pretty much all of sys/; I figured out most of the stuff outside it.

 

No redundant code sitting in the memory and never being used, faster loading times.

@eugene2k while I tend to agree with making the engine more configurable at compile time, this strikes me as wishful thinking. T3D is pretty tiny at ~15MB, and unused code will be sitting in RAM, not in your processor's cache; compared to other game assets I think the code is probably the smallest memory worry. As for faster startup times, I'm very skeptical that you'll gain much. Bigger gains are to be had optimising the code that does run rather than removing code that doesn't.

 

The days when OpenGL handled stuff differently than Direct3D are pretty much dead and gone. Both APIs nowadays support essentially the same features.

That may be true, but driver support still isn't ubiquitous. Though if we're talking about D3D maybe it's a moot point; are there any known cases where OpenGL works better on Windows?

[eugene2k]

and unused code will be sitting in RAM, not in your processor's cache; compared to other game assets I think the code is probably the smallest memory worry. As for faster startup times, I'm very skeptical that you'll gain much. Bigger gains are to be had optimising the code that does run rather than removing code that doesn't.

I agree with all that. I suppose I was a bit unclear, though. There's a lot of stuff done in TS that should be implemented inside the engine. And while it would make the engine a tiny fraction more efficient, the biggest gain would be that that particular bit of code will be easier to maintain than the whole TS + Needlessly exported functions mess. Case in point: adding a new scripting language would've been a lot easier if a bare-bones torque app didn't have 1MB worth of scripts :) Thanks for the t3d-bones branch, by the way. Now I don't have to go the long way trying to figure out what is actually needed in a bare-bones Torque app.

[buckmaster]

Yeah, I definitely agree that having more stuff in the engine would make it easier to ditch TS - and something we were chatting idly about within the SC around the start of the year was starting to port the editor suite into C++ for just that reason. But the effort involved would be monumental.