Background

Our CMS utilises ASP.NET to render the live site that the end users see. The CMS basically publishes an ASP.NET site as if you hand wrote it yourself. The management classes and ASPX pages behind this are pre-compiled and the pre-compiled configuration file is set to allow updates i.e.:

<precompiledApp version="2" updatable="true"/>

This allows us to have a nice strong pre-compiled set of management classes etc whilst allowing the publication system to send new ASPX files to the live site at any time.

This means that there are heaps of ASPX files on the site possibly needing to be compiled at any given time (on some of our larger client sites there could be 20,000 ASPX files).

Problem

Normally you would expect that a new page would be compiled on the first visit and then run fast and efficient for subsequent hits… however a few months ago (June maybe) Microsoft released a patch which caused a few headaches to a large non-pre-compiled site.

Our clients started ringing up complaining about slow downs after publication (i.e. a page had to be re-compiled by ASP.NET). Some of the slow downs were 15 or more minutes! The CPU and memory would max out even quite high spec machines – and the site would become totally unresponsive.

After some head scratching and remote access to the client networks (we couldn’t replicate the problem in our environment) and lots of file monitoring, run-time monitoring etc. I finally realised that ASP.NET was re-compiling every page in the site (including a special cache we keep for system purposes). Naturally on a large site this was taking forever. Further research on our part tracked the problem back to the KB article http://go.microsoft.com/fwlink/?LinkId=91233. Being a security update, some of our clients installed it straight away – against our general advice (i.e install to staging environment first ).

The solution is however quite simple:

<configuration>
    <system.web>
        <compilation debug="false" batch="false" numRecompilesBeforeAppRestart="50"></compilation>
    </system.web>
 </configuration>

Update your web.config file and add in the attribute batch=”false” to the compilation line. The problem was that the framework patch had turned on batch compilation mode by default – causing obvious compilation delays (well this is what we assume has happened – information was a little scarce about the problem, most sites are not as dynamic as an Objectify live site). With this setting, affected machines now compile only one page at once. Click here for more information.

You will also notice another setting here – numRecompilesBeforeAppRestart. Normally ASP.NET will cycle the worker process after 15 dynamic compiles. You can extend this out depending on your needs.

One final note… if you do use a dynamic site, remember that InProc state management is bad! Your sessions will be lost after 15 compiles!

NB: We couldn’t absolutely prove it was the KB article i listed here that caused the problem… it was our investigation by comparing patches and applying patches in our environment that lead us to this (i.e. we ended up replicating the issue – thank god).

Edit

2007-11-28

I just remembered that the day(s) that I was investigating this issue as a matter of top priority was the same day that my entire company went to Calder Park Raceway to drive V8 Supercars! I missed out…oh boy that was a tough day.