Hooking Lua Part 1

13-05-2017 - 4 minutes, 35 seconds -
reverse engineering lua

Reproduced from https://sites.google.com/site/sbobovyc/writing/reverse-engineering/hooking-lua-part-1


Lua is a widely used scripting language in games. When it comes to modding/hacking games, getting access to the scripting system of a game is a huge win. I decided to learn Lua and set up a few experiments in order to learn how to hook into it. The following is a write up of my adventures in Lua.

Compiling, linking, hooking

There are a few different ways to use Lua in a game. Most games are written in C++, and the Lua interpreter is called from inside native code. A game can also register functions with Lua, and after registration these native functions can be used from inside Lua scripts. My little toy application registers two native functions and executes some Lua scripts. The injectable DLL calls a Lua script that dumps the Lua global variables to a file. The globals will contain all the functions that can be called, and include non-standard Lua function that the game registered.

I compiled Lua 5.2.3 with mingw g++ 4.8.1:

$ make mingw

This generated src/liblua.a and src/lua52.dll.

My application was linked to the static Lua library. The application registered two internal functions that simply take a Lua state as a parameter and return some integer values. The application also prints out the address of the Lua state, various Lua C API functions, and the two registered functions:

This is the address of lua state: 0x009123B8
This is the address of luaL_newstate: 0x00405C40
This is the address of luaL_loadstring: 0x00405650
This is the address of luaL_loadfilex: 0x004053E0
This is the address of luaL_loadbufferx: 0x00405600
This is the address of lua_pcallk: 0x00403000
[Exe] This is a dostring print
[Lua] internal_function_1: function: 004016C0
[Lua] internal_function_2: function: 00401790
[Exe] Called internal_function_1
[Lua] got 10
[Exe] Called internal_function_2

After the application is started, the DLL is injected and its DllMain function is called. The DLL uses the DllMain to start a new thread and hook the Lua state that is created in the main application. Most of the Lua C API function require a pointer to a Lua state, so the DLL needs to know its address in memory. No special memory allocator was used, so Lua uses realloc is used to allocate the state on the heap and the address of the state is not known ahead of time. To make things easier, I had the application write the address to a file and then the DLL reads the address from the file. Since the DLL is statically linked to Lua, it will use it's own Lua functions instead of the application's functions. If the application and the DLL were dynamically linked to Lua, when the DLL tried to access the Lua state it would cause a "multiple Lua VMs detected" error. The DLL executes a Lua script that dumps _G, the global Lua variables to a log file named state_dump.txt. This variable is a dictionary, and it contains all the functions that could be called from Lua. The addresses of the two internal functions can be seen in the dump:

internal_function_2 = function: 00401790
internal_function_1 = function: 004016C0

A game would contain many registered functions, and their names and addresses could be used to reverse engineer some interesting parts of the game.

Debugging symbols where included which makes finding Lua function in the application easy. This is the location of luaL_newstate. Notice that the function luaLcheckversion is right below it:

00405C40 >/$ 53             PUSH EBX
00405C41  |. 83EC 18        SUB ESP,18
00405C44  |. C74424 04 0000>MOV DWORD PTR SS:[ESP+4],0
00405C4C  |. C70424 D042400>MOV DWORD PTR SS:[ESP],lua_simp.004042D0
00405C53  |. E8 C8DDFFFF    CALL lua_simp.lua_newstate
00405C58  |. 85C0           TEST EAX,EAX
00405C5A  |. 89C3           MOV EBX,EAX
00405C5C  |. 74 10          JE SHORT lua_simp.00405C6E
00405C5E  |. C74424 04 8042>MOV DWORD PTR SS:[ESP+4],lua_simp.00404280
00405C66  |. 890424         MOV DWORD PTR SS:[ESP],EAX
00405C69  |. E8 52BFFFFF    CALL lua_simp.lua_atpanic
00405C6E  |> 83C4 18        ADD ESP,18
00405C71  |. 89D8           MOV EAX,EBX
00405C73  |. 5B             POP EBX
00405C74  \. C3             RETN
00405C75     8D7426 00      LEA ESI,DWORD PTR DS:[ESI]
00405C79     8D             DB 8D
00405C7A     BC             DB BC
00405C7B     27             DB 27                                                  ;  CHAR '''
00405C7C     00             DB 00
00405C7D     00             DB 00
00405C7E     00             DB 00
00405C7F     00             DB 00
00405C80 > $ 56             PUSH ESI
00405C81   . 53             PUSH EBX
00405C82   . 83EC 24        SUB ESP,24
00405C85   . 8B5C24 30      MOV EBX,DWORD PTR SS:[ESP+30]
00405C89   . DD4424 34      FLD QWORD PTR SS:[ESP+34]
00405C8D   . DD5C24 18      FSTP QWORD PTR SS:[ESP+18]
00405C91   . 891C24         MOV DWORD PTR SS:[ESP],EBX
00405C94   . E8 47BFFFFF    CALL lua_simp.lua_version
00405C99   . C70424 0000000>MOV DWORD PTR SS:[ESP],0
00405CA0   . 89C6           MOV ESI,EAX
00405CA2   . E8 39BFFFFF    CALL lua_simp.lua_version
00405CA7   . 39F0           CMP EAX,ESI
00405CA9   . 74 75          JE SHORT lua_simp.00405D20
00405CAB   . C74424 04 3636>MOV DWORD PTR SS:[ESP+4],lua_simp.00423636             ;  ASCII "multiple Lua VMs detected"
00405CB3   . 891C24         MOV DWORD PTR SS:[ESP],EBX
00405CB6   . E8 75EAFFFF    CALL lua_simp.luaL_error

This is the source code of the two functions:

LUALIB_API lua_State *luaL_newstate (void) {
  lua_State *L = lua_newstate(l_alloc, NULL);
  if (L) lua_atpanic(L, &panic);
  return L;

LUALIB_API void luaL_checkversion_ (lua_State *L, lua_Number ver) {
  const lua_Number *v = lua_version(L);
  if (v != lua_version(NULL))
    luaL_error(L, "multiple Lua VMs detected");
  else if (*v != ver)
    luaL_error(L, "version mismatch: app. needs %f, Lua core provides %f",
                  ver, *v);

Depending on how Lua is compiled into a game, this could be used to find and hook the luaL_newstate function and get the pointer to any Lua states that are created.

Source for these experiments can be found here https://github.com/sbobovyc/WinHookInject/tree/master/lua_experiments