April 22, 2021

Extending BF2 Python API Part 1

Game engine logic could be split into low-level and high-level. low-level includes the simulation of the game world, things like handling movement, collision and physics of objects and keeping track of object states. high-level logic includes what/where/when to create objects, controlling the score system, deciding on how to put players into teams, or checking if win conditions have been achieved.

Low-level game logic usually involves large amount of processing power doing relatively obvious (yet not always simple) tasks. For instance, when a vehicle drives into another vehicle, it should collide with it. There is relatively less argument regarding what the expected behaviour should be, and this kind of code doesn’t need to be written in iterations and recompiled too often. High-level logic however, may be less obvious but requires practically no processing power. For example, when should a player be allowed to use a specific tool? Should capturing an object score 2 points or 3 points? What if we want to award the player 2 points if he captures the object normally, but 3 points if he defeats an enemy doing it? These game rules and balance decision are not clear and constantly change during development. Writing and recompiling unsafe c++ code for every change is a lot of wasted time, and while implementing a settings file to configure these helps, it won’t let you implement complex scoring algorithms and game rules.

Most modern games go with integrating a scripting language. In Battlefield 2, its Python. Python does not need to be compiled, and is easy to write without worrying about unsafe code. High-level logic doesn’t involve any complex physics simulations, so Python being slow is not an issue. Battlefield 2 can load Python files, and offers a built-in host module that can be imported. This module provides an API for controling high-level game logic. You can register to events (such as PlayerConnect, PlayerKilled, onChat), create timers, or triggers (which call your code if a player enters a given radius around an object). You can perform operations on a player such as setting his score, his respawn time, getting his current vehicle or remove him from the server. You can spawn and modify objects, setting their position or health, or getting the list of players that are currently using them. You can decide that the game is over and a specific team has won. With this, there’s no need to recompile anything when working on the high-level game logic.

BF2 Object reference

Example - Control point capture with extra rules.

We have an area that we want players to fight over. We also want the side that is attacking to have at least 2 times the player count in radius to start changing the point’s ownership. Also, we want players who are “pilots” to not count in the calculation for the attackers. (For simplicity sake, lets assume that pilots cannot suddenly become non-pilots while in radius for now).

  • First we set up a trigger for the area which calls our code when a player enters/leaves the radius, and initialize two player counters to 0.
    host.trig_create( controlPointObject, callback, objName, radius, isHemiSphere=1, data=None )
  • In the callback, we check the control point’s current ownership and if the player is a pilot. If he is and he is attacking, we return.
    if controlPointObject.cp_getParam('team') != player.getTeam() and "pilot" in player.getKit().templateName:...
  • We increment/decrement our counters depending if this was a enter/leave event. Then we check if the attacking team has enough players to perform a capture now. If it does, we tell the game at what speed this control point is changing hands.
    controlPointObject.cp_setParam('takeOverChangePerSecond', value)

Python Objects and C Objects

There are plenty of object types in the BF2 engine, all of which inherit from the same base. Vehicles, Weapons, Projectiles, Soldiers (“human” vehicles), capturable control points, RotationalBundles which rotate based on player Input. All of them can be represented with a Python wrapper object.
When the engine wants to pass an object to Python, the game creates a Python instance of the wrapper for the object. The C object caches the created Python object in its struct, and subsequent retrivals of the same object will return the same Python object. This is done only on demand, so a weapon firing 5000 bullets will not be creating 5000 Python objects, unless something in Python specifically asks for these projectiles.
The Python object contains a token field of type PyCObject, which stores a pointer. This is used by the c++ code to retrieve the real C Object when API functions are called, and is not accessible to Python by any normal means. One of the important API calls is object.isValid(), which returns 0 if this internal pointer is not longer valid, telling us that the real C object has already been freed and any attempted operation on the object will raise an exception.
Despite Python being just a couple of steps away from the real C Object, the given API is somewhat limited. Over the first decade since the release of the game, pretty much everything that could have been done with it as already been done. But this can be greatly improved with just a bit of hacking.

Getting the real object from the Python wrapper, without leaving Python.

We want to improve the API, and write functions in pure python that modify the real C objects, without hooks, without c++ and without binary code. For that, we used the ctypes library which gives us the simplest thing a programmer needs, to directly read/write to any place in the process memory from Python. First of all, before strating to “hack” an object, we are going to make sure the C Object hasn’t been freed by verifying that isValid() is 1, so we’re not accidentally messing with junk.

def _verifyObjectValidity(obj):
    if not hasattr(obj, 'token'):
        raise Exception('Token missing from object! Are you sure this is a PhysicalObject?')
    if not obj.isValid():
        raise Exception('Memory functions called on already invalid object! template %s' % obj.templateName)

Then we are going to play with the token, we want to find the pointer it contains in it. As this data isn’t accesible from Python, we are going to have to hack it out.
The built-in Python function id returns a unique integer represnting the Python object, however its actual implementation in the common CPython versions is just the Python object’s address in memory. On 32bit Python (when compiled for release) all objects have a 8 byte header: 4 bytes refcount + 4 bytes type pointer. The next 4 bytes would contain our internal pointer. Following this pointer with a memory viewer, and comparing in IDA what the host module does with it (we find the function in IDA by searching for the exception string that is returned when an object is no longer valid):

We can see that it points to an object, which at offset 0 points to another object, which then at offset 4 points to the real object. A bit later I found out that these somethings are just a WeakPtr and its SharedPtr, and that isValid just checks the validity of the weakptr.

from ctypes import *
offsets_tokenIDToObject = [0x08, 0x00, 0x04]
def _pointerWalk(start, offsets):
    ptr = start
    for offset in offsets:
        _verifyAddress(ptr) # null check

        ptr = c_void_p.from_address(ptr + offset).value

    return ptr
def _getObjectPtr(obj):
    _verifyObjectValidity(obj)

    if not hasattr(obj, "_CStruct"):
        tokenPythonAddress = id(obj.token)
        ptr = _pointerWalk(tokenPythonAddress, offsets_tokenIDToObject)

        _verifyAddress(ptr)
        obj._CStruct = ptr

    return obj._CStruct

This gives us a pointer to the real object. In the next part, we can start reading or (extremely carefully) modifying it.

Written on April 22, 2021