Never Give Up: Redesigning APM Insight .NET agent

Prologue

I am part of the APM Insight team a.k.a. application scientists (that's how we're referred within the team, and we pride over it) focusing on the .NET side~of the agent. For the one's who are new here, APM Insight is the application performance monitoring tool from Site24x7 that provides end-to-end visibility into the way web transactions behave. We support monitoring of Java, .NET, PHP and Ruby web transactions.

Is this blog for everyone? My answer would be NO? It's~a technical blog and I have taken the liberty to write~it under the assumption that you are aware of what a DLL~is, a bit of C++, a little bit of VC++ and to a certain extent what APM Insight is and does.

The story

It all started sometime back when we decided to do away with our old Visual Studio 2010 for compilation and migrate to the newer 2013 version. We also wanted to become Azure ready. A migration which we thought would~be straightforward turned into an interesting set of problems. This blog explains the problems we faced and how we finally solved them. Yay!

The challenge

Our challenge started with our C++ profiler. We started by separating our core DLLs~which included our Profiler DLL~(a COM C++ DLL) and our core .NET DLL, we put them for testing hoping everything would just work (duh..!), and it didn't. :(

The investigation

We started investigating, we tried abstracting many things, nothing seemed to work. We tried it in our test setup, Azure setups; but we were always getting the profiler-not -loaded error (Not a good sign, because our whole instrumentation depends on the profiler, and when this fails our product is practically useless).

Digging deeper into the profiler with dependency walker

Dependency walker is a great analysis tool to analyze what a native DLL~is dependent on. We started analyzing a few other DLLs with depends.exe. For each module found, it lists all the functions that are exported by that module, and which of those functions are actually being called by other modules. Using this, we found out that there were too many dependent DLLs in our profiler which may not work in an Azure environment. We were using Visual Studio 2010 for our development.

We do not use any GDIs, COMCTLs libraries in our profiler. Thus started our refactoring tasks in the profiler code.

Beware, its MFC the ghost

MFC is a great library that makes~C++ development much easier. But, it also comes with its negatives.

When I started writing the profiler few~years ago, I used MFC classes, particularly CString, CMap, CCriticalSections and the likes. But now, the dependencies seemed to use a lot of MFC, so we decided to remove all dependencies of MFC from our C++ code.

We hoped that when this is done, the dependencies will reduce and in turn help us with other scenarios. But to our surprise the dependency did not go away. Our DLL~was still using GDIs and the COMCTLs, frustration grew; All steps taken seemed like a waste of time. But we didn't give up and continued to look at workarounds.

Then on a knowledgeable instinct, we tried a dummy profiler, but compiled it this time in Visual Studio 2013. This time when we put it to the test in depends, all the unnecessary DLLs~vanished like magic.. :) (VS 2013 was improved to put to trash unwanted DLLs). This excited us, and just like that from about 17, our dependent DLLs reduced to just 5. That's three~times dependency on unwanted DLLs reduced.

Now, when we tried with our new profiler DLL, it worked. As expected, once the dependencies were removed the profiler could easily be hooked onto the .NET process.

Moral of the story.

• Use less of the good ol' MFC as much as possible. Use STL~libraries or just plain C++
• Try to understand the root of API you are using. In our case a simple CString made us depend on the whole MFC
• Update to a more modern IDE if they are available. In our case VS 2013 compilation itself got rid of unwanted dependencies
• Last and probably the most important lesson, NEVER GIVE UP.