VisionAppster Blog

If you have ever shipped large software that depends on 3rd party libraries, you may know the situation. There is a bug in one of the dependencies, but you cannot fix it there. You may not have the source code, it may be a system-wide library or you just want to avoid the hassle of patching it.

So, what can you do? On Linux, LD_PRELOAD is a typical answer. I may come back to that some day, but what if the bug affects many platforms? I don’t even know how to do the same on Windows, and platform-specific DLL injection hacks aren’t something I’d like to put into production anyway.

A few years back, we were in this situation. The library in question was written in C++. It provided a base class with a virtual function whose implementation in a derived class failed in some special cases. About like this:

class Base
{
public:
  virtual ~Base();
  virtual void buggyFunction();
};

class Derived : public Base
{
public:
  // This function fails sometimes.
  void buggyFunction() override;
};

To complicate things a bit, the function was quite complex and did the right thing for most instances of Derived. Thus, completely replacing it with a fixed version didn’t look like a good option. The specific corner case where it failed could luckily be detected by inspecting other members of the class.

The C++ standard does not specify how virtual functions should be implemented. In practice, however, compilers generate a virtual function table and place a pointer to it as the first member of a class. Multiple inheritance complicates things, but let’s not jump into that rabbit hole for now. Instead, consider the following C++ class:

class A
{
public:
  virtual ~A();
  virtual int func(int);
  int member;
};

Since the structure of the virtual function table is not standardized, compilers may use different layouts. The layout used by the GCC compiler is this:

struct A_vtable
{
  void* type_info; // Pointer to type information
  void (*destructor)(A*); // Function pointer
  int (*func)(A*,int);    // Function pointer
};

Virtual function pointers are laid out in declaration order. The A_vtable structure reveals another C++ secret: member functions are just obfuscated versions of normal functions that get this as the first argument. That’s it. If you know Python, D or Nim, this won’t surprise you a bit.

The fact that C++ has a dedicated syntax for member functions leads to numerous problems which I’m not going to detail here. Fixes have been proposed since at least 2004, to no avail. The std::function class however had to do it the right way, so we ended up with a language that has one syntax as a language feature and another as a standard library feature.

OK, enough whining. Back to our original track.

// You write
int A::func(int param)
{
  return param;
}

// The compiler actually generates
int A_func(A* this, int param)
{
  return param;
}

Generated member function names are actually much more complicated due to name mangling.

There is one virtual table that is shared by all instances of class A, and the compiler will statically initialize it for you by placing the addresses of A‘s member functions and the generated type info into it. About like this:

const A_vtable A_vtable_instance
{
  &A_type_info,  // We'll skip this now
  &A_destructor, // Pointer to an ordinary function
  &A_func        // Ditto
};

Now, whenever you create an instance of class A, you actually instantiate a structure like this:

struct class_A
{
  A_vtable* vtable;
  int member;
};

The compiler will initialize the vtable pointer for you.

// You write
A a;

// What actually happens
class_A a;
a.vtable = &A_vtable_instance;

If you derive B from A, this is what happens (conceptually):

// You say
class B : public A
{
public:
  int func(int) override;
  virtual void func2() const; // A new virtual function
};

int B::func(int) {}

void B::func2() const {}

// The compiler generates

struct B_vtable
{
  void* type_info;
  void (*destructor)(B*);
  int (*func)(A*,int);
  void (*func2)(const B*);
};

// Auto-generated default destructor.
void B_destructor(B* b)
{
  A_destructor((A*)b);
}

int B_func(B*, int) {}

// Member function qualifiers are actually qualifiers
// of the this pointer.
void B_func2(const B*) {}

// Note that the first three members have the same
// layout as A_vtable, which means we can safely access
// a B_vtable pointer as an A_vtable pointer.
const B_vtable B_vtable_instance
{
  &B_type_info,
  &B_destructor,
  &B_func,
  &B_func2
};

struct class_B
{
  B_vtable* vtable;
  int member;
};

Whenever your code calls a virtual function in A, the compiler looks up the actual function to call through the virtual function table:

// You say
B b;
A* a = &b;
a->func(1);

// You get
class_B b;
b.vtable = &B_vtable_instance;
class_A* a = &b;
a->vtable->func(a, 1);

Now that we know how C++ actually dispatches virtual functions the solution becomes straightforward: detect which class instances need a fix and change the pointer to the buggy function in the virtual function table. Like so:

// hack.cc
#include <iostream>

// This is the published interface we work with.
class Base
{
public:
  virtual ~Base()
  {
    std::cout << "Base::~Base()\n";
  }

  virtual void buggyFunction()
  {
    std::cout << "Base::buggyFunction()\n";
  }
};

// This class is implemented in a closed library.
class Derived : public Base
{
public:
  ~Derived()
  {
    std::cout << "Derived::~Derived()\n";
  }

  void buggyFunction() override
  {
    // This sometimes fails.
    std::cout << "Derived::buggyFunction()\n";
    Base::buggyFunction();
  }
};

static void fixedFunction(Base* thisPointer)
{
  std::cout << "fixedFunction()\n";
  // Bypass virtual function resolution.
  thisPointer->Base::buggyFunction();
}

// This is GCC's vtable layout for Base.
struct FixedVTable
{
  void* typeInfo = nullptr;
  void (*destructor)(Base*) = nullptr;
  // The fix.
  void (*buggyFunction)(Base*) = &fixedFunction;
};

FixedVTable fixedVTableInstance;

static void overwrite(Base* b)
{
  // b actually points to a vtable pointer.
  auto pVTable = reinterpret_cast<FixedVTable**>(b);
  // Copy retained entries from actual vtable.
  fixedVTableInstance.typeInfo = (*pVTable)->typeInfo;
  fixedVTableInstance.destructor = (*pVTable)->destructor;
  // Replace vtable pointer in this instance.
  *pVTable = &fixedVTableInstance;
}

int main()
{
  Derived d;
  Base* b = &d;
  b->buggyFunction();
  overwrite(b);
  b->buggyFunction();
}

Compile and run:

g++ -o hack hack.cc
./hack

The result:

Derived::buggyFunction()
Base::buggyFunction()
fixedFunction()
Base::buggyFunction()
Derived::~Derived()
Base::~Base()

As you can see, fixedFunction gets called instead of Derived::buggyFunction. The destructor outputs are there just to make sure we didn’t mess up the virtual function table.

This hack never went to production as we found another, less dangerous solution. But I thought it was worth sharing nevertheless.