Benchmarking Scripts

[Timmy]

Hi Guys,

I wanted a way to be able to reliably benchmark a scene and record raw raw frame rates etc. So i wrote a few scripts to help with this. It can all be found here https://github.com/rextimmy/Torque3D/tree/benchmarking . No c++ changes, all script. It uses a Path Camera and basically goes along the path and records frame stats. It can produce two different reports 1)CSV raw dump showing fps,mspf,draw calls, polycount and camera position for each frame. Useful to import into a spreadsheet and or database. 2) A summary text file that shows average/min/max fps,mspf,draw calls and some basic details about the scene.

To use it you must:

[*] Create a SimGroup called Benchmark under your MissionGroup

[*] Create a Path called BenchmarkPathN (where N is 1 through to however many different paths you want).

[*] Add markers to the path, these can be named whatever you like, just make sure the sequence is correctly number 1-N. Just a word of warning, the field msToNext on the marker, with a path camera this actually means Meters per second to the next marker not Milliseconds to the next marker as the documentation for it says. A sensible starting value is something around 10 not the default 1000. Needs adjusting depending on distance between each marker though.

To launch the benchmark there is a function called

 

function Benchmark(%id,%numRuns,%writeSummary,%writeReport)

 

so id is the N from BenchmarkPathN above. numRuns is the number of runs you want to do. writeSummary & writeReport (true/false variables) are both explained above. If you pass no arguments to this function it will default to using id=1 | numRuns = 1 | writeSummary = true | writeReport = true

When benchmarking, i highly recommending doing multiple runs and completely discarding the first runs results, i'm sure everyone reading has suffered from the dreaded Torque3D loading hiccups as you run around a level, by discarding the first run you allow everything to get loaded in the first run and all subsequent runs after that should be nice and smooth.

Now some user configurable stuff is here . They are already commented in the code so i will leave it at that.

Anyway if you find any bugs or have any suggestions than fire away. Originally i was going to create a gui dialog to actually launch the benchmark and thus relax the BenchmarkPathN naming convention but kinda got busy with other stuff ;)

Few pics of the setup:

http://i.imgur.com/GYErGk4.png

http://i.imgur.com/9JGr7qc.png

[Johxz]

nice one timmy! sorry I can't finish this script on time :( I will play with this and give some feedback... thank you!

[Bloodknight]

This looks fun, i recall a conversation with Az a week or few ago about how useful this would be, ive been heads down in Assignment handin mode the past 2 weeks, i'll see if i have some time this weekend to test and break this (breaking, its what i'm best at :p )

[JeffR]

Haha, yeah.

I'm doing up a GUI interface to select the benchmark path, that'll go in today, but otherwise yeah, feel free to hammer the crap out of it and suggest things the benchmarker should record. It's perfectly extensible.