Saturday, 31 January 2015

CA Certificates - The tale of the invisible certificate

I've been through a memory upgrade on my 5-year old PC. My goal is to set up a few VMs running simultaneously, because I need to widen my scope for experimentation. I found out my BIOS has an incompatibility with the memory DIMMs currently available, but fortunately a friend lent me 8GB, so I can start working now, while I try to sort out this mess.

As I set up each VM, I'm importing my bookmarks, so that I have my net environment available "everywhere". And I've come across a curious situation, regarding certificates.

One of the URLs I have on my bookmarks is The first time I accessed it on Firefox, I got an error message:
Peer's certificate has an invalid signature. (Error code: sec_error_bad_signature)

Using openssl s_client, I checked that sends only its own certificate, not the chain, so I looked up the chain in IE, and checked the intermediate CA on Firefox's certificate store. It was there, but it was a different certificate - different signature, different validity (both valid, because the validities on both certificates overlapped), different issuer, only the subject was the same.

I exported that CA certificate from IE, and ran openssl verify using each CA certificate; using the one from Firefox certificate store, I got an error; using the site's CA certificate, the validation succeeded.

So, I imported the site's CA certificate to Firefox, accessed the site, and all was well again.

Then, I checked Firefox's certificate store. And I only found the exact same certificate that was there already, and which wasn't previously validating's certificate. Except that now it was.

And much scratching of head ensued.

Until yesterday, when discussing this at lunch with a friend, he told me the obvious: "Well, if you imported it, and the site's certificate is now correctly validated, then it must be there, even if you can't see it". And that gave me a memory jolt, to an issue I had a little more a year ago, with Sun One's web server certificate store, where we had two certificates for the same CA, but only one was visible on the web console. In order to correctly see both, I had to use certutil on the command line.

And in this case, the solutions was the same:

certutil -L -d sql:path to Firefox profile directory -n certificate subject

Which promptly listed the two certificates.

And another Mystery of the Universe was solved during a meal.

I don't understand why the GUI shows just one certificate. I'm not going to say it's stupid because it may be a reasonable decision, based on knowledge I don't have. But to completely hide the fact that a CA has two simultaneously valid certificates on the store is terribly misleading, it's definitely not what I'd call a good solution.

In the end, it was command line to the rescue... as usual.

Saturday, 10 January 2015

Visual Studio - Getting to your debugging symbols

I've recently been reformulating my lib environment. Basically, it consists of:
  • A loading zone, where I install the library sources, and run the build process. It has a folder for each lib, with each version in its own sub-folder.
  • A library root folder, with sub-roots for mingw and MSVC, where I store the files resulting from the builds. Again, each lib folder has a sub-folder for each installed version.

There's a bit more to it, but it's not important for today's post.

This reformulation has gone through several iterations, and it's still a work in progress. This time, I had the following goals:
  • G1 Getting some further progress on automating the process of building libraries from source, both release and debug versions. Actually, it was this requirement for debug versions of all the libs that led to my patch to OpenSSL with a debug configuration for mingw32.
  • G2 Correcting some bad decisions regarding the lib folders' names, especially when it comes to version numbers. The goal is to use the same number format each lib uses for its own files, where applicable.
  • G3 Clearing the landing zone after building the libs. Now, when I finish building, I run make (actually, mingw32-make or nmake) clean.

Point G3 led me on another learning experience.

The most common debug option when using MSVC seems to be /Zi, which stores the debug symbols in PDB files. As such, I haven't looked into the other options, which store debugging information in the object files themselves; I won't discuss them in this post, but if I had to hazard a guess, I'd say the behaviour is the same as the one described below.

During a debug build, the compiler/linker stores the absolute path to the PDB file on the executable files (EXE or DLL). When you fire up the debugger, as it loads these executable files, it looks up the PDB using this path, to load the symbols. If it can't find the PDB, it then goes through other steps.

On the open-source projects I've seen, I've come across two default behaviours - either the PDB files are not copied to the library folder at all (e.g., Boost); or they're copied to the folder containing the static libraries/import libraries, but not the DLLs themselves (e.g., ICU). I call these default behaviours, because there may be options for gettting a different behaviour; I've looked for these, but found none.

This, combined with point G3 above, is not-so-good news, because make clean deletes the PDB files. So, when we fire up the debugger, it cannot find the symbols for these DLLs. It never happened before because my procedure has always been:
  • build release.
  • clean up.
  • build debug.
  • don't clean up.

So, the PDBs were always available at the landing zone, i.e., the path stored on the executables. And even though some libs copied them to the lib folders, these weren't actually used when the DLLs were loaded by the debugger.

There are several options for dealing with this, including these three:
  • O1 Setting up a local symbol server, which, as I understand it, is more of a local cache to store symbol files.
  • O2 Adding each individual folder where PDB files are installed to the _NT_SYMBOL_PATH env variable, which MS debuggers use to locate symbol files.
  • O3 Manually copying the PDB files to the DLLs folders.

I believe option O1 is the best, but I'll leave it for another iteration. For now, I'll go with option O3. Like I said, this is a work in progress, and I don't feel a particular pressure to go for the optimal solution (which I'll have to test before I adopt it), I prefer getting to a working solution faster, in order to keep my self-imposed deadline (which will end this weekend).

Of course, once you have all this worked out, you still need your debugger to load the correct DLLs. You may have other versions of those DLLs on your path; with popular libraries, like OpenSSL, this is more common than you may think.

On Qt Creator, assuring that you load the correct DLLs is very simple. On Projects mode (Ctrl + 5), you switch to the Run configuration and edit the PATH on the Run Environment. Since I don't have these libraries on the PATH, I always need to do this; if you usually have your debug libraries on the PATH, you don't need to edit anything.

Visual Studio is a different sort of creature. A larger sort of creature, where everything usually takes a bit more work to find.

I knew it was on Project Properties (Alt + F7). At first, I thought it was on Configuration Properties -> VC++ Directories -> Executable Directories. When I hit help, it took me here, where we can read: "Directories in which to search for executable files. Corresponds to the PATH environment variable". Fine, that's just what we need. Somewhat later, and after a moderate amount of gnashing of the dental (not mental, mind you) persuasion, I noticed that the description on the project Property Pages was a wee-bit more complete, namely "Directories in which to search for executable files while building a VC++ project". Ah... right... building the project... as in, "not running the executable".

Then, I turned to the next obvious choice, Configuration Properties -> Debugging -> Environment. This time, I read the description before going for the help button. It is quite helpful, it says "Specifies the environment for the debugee, or variables to merge with existing environment". After reading this, I knew exactly what to do. Which was hitting the help button, and hoping the help page for this option was more useful.

Fortunately, it was. We control the PATH here, using something like this: PATH=E:\Dev\lib\msvc\openssl\openssl1_0_1j\debug\bin;E:\Dev\lib\msvc\icu\icu54_1\debug\bin;%PATH%.

And, finally, after some gnashing of the mental persuasion, I can say that Visual Studio's debugger and me are finally getting along just fine.

As they say, until next time... when I turn this into reusable project configurations, instead of having to specify these manually on every project.

Thursday, 8 January 2015

OpenSSL - Debug build configuration for mingw32

I've just submitted a patch to OpenSSL, to include a debug build configuration for mingw32, as it currently lacks one.

Not only have I done the obvious - removed optimizations and add the -g compiler option, but I've also used debug-Cygwin as a starting point to include a set of debug #defines for OpenSSL. You can find the patch here.

At first, I sent it to the openssl-dev mailing list. Later, I find out there was already an issue open for this on OpenSSL's RT.

Anyway, the patch makes the changes below.


This is the script called from the command line for configuration, which will generate the makefile.

The patch adds MSYS* to the script's platform guessing mechanism, allowing us to build on MSYS2. It wasn't really necessary for the debug configuration, but I like to have the ability to build on MSYS2.


Configure is a perl script that is called by config.

The patch adds the mingw32 debug configuration (debug-mingw), and changes the part of the script that defines the executable extension, to look for /mingw/ instead of /^mingw/, which then makes it work correctly with both mingw configurations.

As a side note, I think it would have been better to name the debug configurations as <platform-debug>, rather than <debug-platform>, but I don't know the reasons behind this, so I might be wrong.

This file is a base for the final makefile.

The patch changes platform checks from mingw and mingw* to .*mingw and *mingw*. Curious, now that I'm writing this, it's the first time I've noticed the incoherence. I'll probably go back to the testing board and change that to *ming* on both.


This file is used when we're building shared libraries (DLLs on mingw).

This is where I was more bold with the changes. The original file was always passing -Wl,-s to the linker, meaning it would clear the symbols from the object files. However, on a debug build we want to keep the symbols, that's the main reason for wanting a debug build in the first place.

So, the patch not only adds a check for .*mingw instead of mingw, it also changes the linker arguments for the debug configuration, excluding the -s.

And that's it. Not that much work, but a lot of experimenting. And, since shell script and make are not the most user-friendly debugging experiences, it has required a lot of creative experimenting.

Hope someone finds it useful.