How to Debug Applications You Didn't Compile - Source Indexing and Symbol Stores

Introduction

Have you ever tried to track down a bug in an application that you didn’t compile on your development machine? (If you use a build server then the answer to this question is almost certainly, “Yes.”) If you have, then you will appreciate what a time sink it can be.

If you are very lucky, you will be able reproduce the bug on your development machine with the version of the source code that is already in your workspace, or after getting the most recent source code from your version control system. Failing that, if you are moderately lucky you may be able to check out a previous version of the source code, compile it, and reproduce the bug while running it with your debugger. But what if you aren’t so fortunate? What if it is a version of the application that was released two years ago? What if it is an application you never developed before but you now have to maintain? What if you are in the middle of working on a new feature and it would be really inconvenient to stop everything you are doing to reproduce the bug with a different version of the source code? In any of these cases, and many more, reproducing the bug and identifying its root cause will be a huge pain in the butt.

Wouldn’t it be great if you could just attach your debugger to the executable that is exhibiting the bug and debug it directly? Well, it turns out that you can! In Visual Studio, you can press Ctrl+Alt+P and attach the VS Debugger to any process that is running on your computer.

If you just tried to attach your debugger to a random process to see what would happen, then I applaud your intellectual curiosity. You probably didn’t see very much though. Depending on your version of Visual Studio, you probably saw either an empty window or a page informing you that no source is available. If you press the Pause button, you can view the disassembly for the current execution point and the call stack will do its best to show the file name and byte offset for each stack frame.

 

Debug Applications

Visual Studio screenshot

 

While this information might be interesting, it isn’t terribly useful for debugging an application. What we would like is to be able to see the relevant source code and step through it line by line. A call stack, that showed us method names and line numbers, would be nice as well.

There are two reasons why the debugger can’t show us this information. First, the debugger doesn’t have access to any symbol files (.pdb files). The symbol files tell the debugger what source files correspond to the instructions in a library/executable file. Second, even if the debugger had access to the necessary symbol files, it still wouldn’t know where to look to get the relevant source code files. The source file paths in a symbol file will be relative to the locations of the source files when the library/executable was compiled. Unless you have the right versions of the source files in the exact same path on your development machine’s file system, then the debugger will fail to load the source correctly.

Of course, if you had the symbol files and all of the necessary source files on your computer, then you would have everything you needed and the debugger would work as expected. (Incidentally, that is why the debugger works when you have compiled the application yourself – the compiler automatically generates the symbol files and all of the necessary source files are in your solution path.) It follows logically then that you could debug any executable, even one you didn’t compile yourself, if you had access to the symbol files and the necessary source files.

That is exactly what symbol stores and source indexing let you do. A symbol store gives you a place to centrally store the symbol files that are generated by your automated builds. Source indexing modifies the symbol files so that they reference file paths in your version control system instead of paths in your build server’s file system. With a symbol store and source indexing enabled, your debugger will search the symbol store for the symbol files it needs and download them automatically. The debugger will also automatically retrieve the correct versions of the necessary source files from your version control system, based on the information in the symbol files.

Setting up a symbol store and source indexing is a straightforward process. At a high level, these are the steps:

  • Symbol Store
    • Create a file share for storing symbol files.
    • Configure your automated build process to deploy symbol files to the file share.
    • Configure the VS Debugger to look for symbol files in the file share.
  • Source Indexing
    • Configure your automated build process to index sources.
    • Configure the VS Debugger to enable source server support.
Setting Up a Symbol Store

In this section, I will go over the steps to set up a symbol store, configure your automated build process to write symbol files to the symbol store, and configure the VS Debugger to use the symbol store to look up symbols when debugging.

 
Creating a Symbol Store

A symbol store is just a folder on a server where you will store symbol files that were generated by your automated builds. Pick a location on a file server that can be accessed by all of the developers in your organization and your build server. Then create a new folder for the symbol files (ex. \\fileserv1\Symbols).

 

Configuring the Automated Build Process

If you use TFS 2010 or higher to run your automated builds, then Microsoft has made this step easy for you. All you need to do is edit your build definition, choose the Process tab, expand the Source and Symbol Server Settings section, and then enter the path to the folder you created as the value for the Path to Publish Symbols setting.

 

Configure Automated Build Process

Configuring the Automated Build Process

 

If you do not use TFS, then you will need to add a step to your automated build process to run symstore.exe to deploy the symbol files. The symstore.exe program comes with the Debugging Tools for Windows. You will need to download and install the debugging tools on your build server. This article links to the installer for the Debugging Tools and also has some good examples of symstore commands. Also, this is the MSDN article for symstore.

 
Verifying the Symbol Store

The easiest way to verify that your build process has deployed the symbol files correctly is to inspect the contents of the symbol store folder after the build completes. You should have a sub-folder for each PDB file that your build generated. Within each PDB sub-folder, you should see one or more sub-folders that correspond to the builds of the library/executable associated with the PDB file.

 

Configure Automated Build Process

Verifying the Symbol Store

 

You may be wondering about the naming convention for the sub-folders. Each “version” of the symbol file is in a sub-folder whose name is a unique 33-character identifier. The first 32 characters of this identifier correspond to a GUID that is also embedded in the library/executable file. The last character of the identifier corresponds to the “PDB age.” Every time you compile a library/executable it will get a different identifier. The VS Debugger will use this information to look up the version of the symbol file that it needs in the symbol store.

If you want to see this for yourself, you can use the dumpbin utility to view the headers of the binary file. If you run this command:

dumpbin [YourFile].exe /HEADERS

In the output that results, you should see a section that looks something like this:

Debug Directories

Time     Type   Size     RVA      Pointer
-------- ------ -------- -------- --------
5316856E cv     11C      00002FD8 11D8    
Format: RSDS, {6984417A-A02D-4B8C-AE4F-1FD0D10446FD}, 1, [ServerBuildPath]\[YourFile].pdb

The parts of the output highlighted in yellow above, form the unique identifier for the PDB file that corresponds to this version of the executable.  So in our example, if the build process has deployed the symbol files correctly, then the following file would exist in the symbol store:

\\fileserv1\Symbols\[YourFile].pdb\6984417AA02D4B8CAE4F1FD0D10446FD1\[YourFile].pdb

 

Configuring the VS Debugger

Once you have set up your automated build process to correctly deploy the symbol files to the symbol store, the next step is to configure your debugger so that it can make use of the symbol store. In Visual Studio, open the Tools menu and select Options to open the Options dialogue box. In the left-side panel, expand the Debugging section and select Symbols to view the Symbols pane. Then add a new entry for your symbol store in the “Symbol file (.pdb) locations” list.

 

Configure VS Debugger

Configuring the VS Debugger

 

I also recommend that you create a local symbol cache. This will give VS a place to store symbol files locally after it downloads them. By caching the symbol files, VS will not have to re-download them from the symbol store for subsequent debugging sessions. To do this, just enter the path to a local folder (ex. C:\Users\user1\SymbolCache) into the “Cache symbols in this directory” textbox.

You may also want to consider checking the checkbox next to the Microsoft Symbol Servers option. This will instruct the debugger to load to the symbol files that Microsoft has published for its own libraries. This will allow you to get information in your call stack for MS system libraries when you are debugging. This can be very useful when you encounter exceptions that are raised by system libraries that are referenced by your code. However, loading all of these additional symbols will make your debugger load much slower. So you may want to leave this option de-selected most of the time and only select it when you need the extra information.

 
Setting Up Source Indexing

Now that the symbol store is set up, the next step is to set up source indexing. First, you will need to configure your automated build process to generate the source indexing information. Then you will need to configure the VS Debugger to look for the source indexing information in the symbol files.

 

Indexing the Source Files

As with the symbol store, if you are using TFS 2010 or higher for your automated builds then Microsoft has made this very easy for you. Just edit your build definition, choose the Process tab, expand the Source and Symbol Server Settings section, and then change the Index Sources setting’s value to True.

 

Indexing Source Files

Indexing the Source Files

 

If you use TFS, you can skip down to the section on verifying source indexing.

If you do not use TFS to run your builds, then you will need to add a step to your build process to call the source server indexing command. To do this, you will need to download and install the Debugging Tools for Windows on your build server, if you didn’t do so already when you set up your symbol store. If you navigate to the installation directory for the debugging tools, all of the commands and documentation for source indexing will be in the srcsrv sub-folder.

 

SRCSRV Subfolder

srcsrv sub-folder

 

Srcsrv comes with support for the following version control systems: TFS, CVS, SVN, Perforce, and Visual SourceSafe. There is a command file in the srcsrv sub-folder for each of these version control systems: tfsindex.cmd, cvsindex.cmd, svnindex.cmd, p4index.cmd, and vssindex.cmd, respectively. An extension has been written to support Git as well.

The srcsrv.doc file in the srcsrv sub-folder contains useful documentation for the source server. Also, this article, by Bruce Dawson, is very helpful. Jelle Druyts’ instructions for setting up a source server also have a lot of good information.

 
Verifying Source Indexing

Once you have source indexing set up, the easiest way to verify that your build process has indexed the source files correctly is to inspect one of the generated symbol files. You can do this with the pdbstr utility that comes with the Debugging Tools for Windows (it is in the srcsrv sub-folder). If you run this command:

pdbstr.exe –r –p:[YourFile].pdb -ssrcsrv

You should get output that looks something like this:

SRCSRV: ini ------------------------------------------------
VERSION=3
INDEXVERSION=2
VERCTRL=Team Foundation Server
DATETIME=Thu Feb 20 14:32:06 2014
INDEXER=TFSTB
SRCSRV: variables ------------------------------------------
TFS_EXTRACT_CMD=tf.exe view ...
TFS_EXTRACT_TARGET=%targ ...
SRCSRVVERCTRL=tfs
SRCSRVERRDESC=access
SRCSRVERRVAR=var2
VSTFSSERVER=http://[yourtfsserver]/tfs/[yourproject]
SRCSRVTRG=%TFS_extract_target%
SRCSRVCMD=%TFS_extract_cmd%
SRCSRV: source files ---------------------------------------
[ServerFilePath]\[YourApp]\File1.cs*VSTFSSERVER*/[YourApp]/File1.cs*5*File1.cs
[ServerFilePath]\[YourApp]\File2.cs*VSTFSSERVER*/[YourApp]/File2.cs*7*File2.cs
...

The value of the VSTFSSERVER variable should be set to the URL of your version control server. Also, each entry in the “source files” list should contain a path into your version control repository, along with a version number (see the highlighted lines above). This is how the debugger knows which source files (and which versions of those source files) it needs to retrieve from your version control system when you step through the code in the debugger.

 
Configuring the VS Debugger

Once your build process is configured to generate symbol files with source indexing, the next step is to configure the VS Debugger to enable source server support. In Visual Studio, open the Tools menu and select Options to open the Options dialogue box. In the left-side panel, expand the Debugging section and select General to display the General Debugging Settings pane.  Then select the checkbox labeled “Enable source server support.”

 

Enable Source Server Support

Configuring the VS Debugger

 

I also recommend that you check the checkbox that is labeled “Print source server diagnostic messages to the Output window.” Enabling this option is not required, but the output from source server can be very useful when you are setting all of this up for the first time.

 
Conclusion

That’s all there is to it. You should now be able to run a version of your application that was built by your build server, attach the VS debugger to the process, and step through the code just as if you had built the application locally. The debugger will automatically load the necessary symbol files from your symbol store and it will automatically download the right version of each source file as you step through the code.

 
Notes

This article assumes that the application you want to debug is running on your local computer. However, these techniques will also work for debugging processes running on remote servers, such as web applications and services. In an upcoming article, I will show you how to set up Remote Debugging and use it in conjunction with your symbol store and source indexing to debug processes that are not running on your computer.

If you use TFS to run your builds then you should bear in mind that, unless you change your build retention policy, TFS will delete the symbol files from your symbol store when it deletes a build. You need to make sure this doesn’t happen for builds that you actually release. Otherwise, when you go to debug a version of your application, you may find that the necessary symbols are no longer in your symbol store. See the article linked in the next section by Ed Blankenship for details on how to edit your build retention policy in order to avoid this.

If your deployment process currently deploys your symbol files along with your application, you should change it to stop doing that once you have your symbol store set up. The symbol files are quite large and removing them from your deployment package will make it much smaller. Also, distributing your symbol files can be a security risk. So make sure you keep your symbol files in your symbol store and out of your deployment.

 
Additional Links

In this article, I’ve stayed at a fairly high level. For more detail about symbol stores and source indexing, I recommend this great article by Ed Blankenship.

This article by John Robbins about symbol files is also very useful.

In the section above on indexing source files, I mentioned Bruce Dawson’s article on source indexing. Bruce also gave a great talk about debugging on Linux at Valve’s Steam Dev Days conference last month. If your application currently runs on a Microsoft platform and you are interested in supporting Linux as well, then this talk is a great starting point.