No more inline ASM in VC++ on x64

I’m working on a project to port a 32-bit application for the x64 platform. The first errors that came up when building for x64 were related to inline ASM code, which is no longer supported in VC++ for x64. VC++ has an inline assembler built within the compiler, so you could write assembly code directly in C++ without the need of an external assembler (such as MASM). However, this works only on x86. For x64 and IA64 this is no longer possible. There are several workaround for this:

• put the assembly code into a separate file and use MASM for x64
• use compiler intrinsics, which are functions that basically wrap assembly instructions or sequence of intructions
• rewrite in C++, use Windows APIs, etc.

One error that I had was related to this assembly code:

   __asm {
      int 3
   }

This was used to generate a breakpoint. This can be easily replaced with compiler intrinsic __debugbreak(), which has an identical effect, except that is available on all platforms.

Other errors I had were due to assembly code used to retrieve the value of the EIP and EBP registers. They were used for walking the stack.

__declspec(naked) DWORD_PTR* GetEip()
{
   __asm {
      pop   eax
      push  eax
      ret
   }
}

__declspec(naked) DWORD_PTR* GetEbp()
{
   __asm {
      mov   eax, ebp
      ret
   }
}

The naked specifier is another thing that is not supported on x64. One way to retrieve the value of these registers that works both with x86 and x64 is using RtlCaptureContext, except that this won’t work on operating system previous to Windows XP. In you don’t care about those operating systems, you could write something like this:

      CONTEXT context;
      RtlCaptureContext(&context);

#ifdef _WIN64
      DWORD_PTR* ebp = (DWORD_PTR*)context.Rsp;
      DWORD_PTR* eip = (DWORD_PTR*)context.Rip;
#else
      DWORD_PTR* ebp = (DWORD_PTR*)context.Ebp;
      DWORD_PTR* eip = (DWORD_PTR*)context.Eip;
#endif

Attention, on x64, register EBP (actually RBP) is no longer used. You should use RSP for getting the stack frame.

However, if you want to build a portable stack walking, use StackWalk64. Despite the 64 suffix, this function works on all platforms (x86, x64, IA64). Here is an article that shows how to walk the stack using StackWalk64.