Tuning IIS 7 for static content

Note: The below works for both IIS 7 and IIS 7.5. If you don’t want the story behind this, you can jump straight to the tuning instructions here.

At work we recently discovered some performance issues with our media server. This is a somewhat powerful webserver, which only serves images uploaded by our users.

The server

The server has a Core 2 Quad (Xeon actually) CPU, 20 GB of RAM and a RAID 1+0 consisting of 15k rpm SCSI disks. The RAID controller has 192 MB of cache (not that much, I know :-()

It runs Windows 2008 R2, and thus an IIS 7.5.

With that in mind we figured it should be plenty for serving out some puny images – after all they’re only about ~100 KB each. However some of our users started complaining the site was slow during prime time. Also they said images would occasionally load very slowly, one at a time.

It didn’t take long to assert this, however the CPU was hardly doing anything – running at roughly 10% usage. And the server still had 12GB free memory, so the only reasonable explanation had to be I/O.
A perfmon later we asserted this as well.

Finding the bottleneck

When checking perfmon for I/O bottlenecks, the preferred counter is found under “PhysicalDisk” and called “% Idle Time“, which, as a rule of thumb, should stay above 20%. Ours was flatlining at 0…
Clearly the disk was very busy all the time, and requests were queuing up.

Removing the bottleneck

Now we had two options for solving this:

  1. Upgrade the servers’ hardware with a new RAID controller (with a larger cache) and perhaps some fasters disk.
  2. Try to cache more of the files in memory

Whilst our system administrator checked prices etc. for upgrading the hardware, I started toying around with the IIS 7.

First I made sure we had expiration headers enabled (so the images would be cached locally in the user’s browser)

I also checked that output caching was indeed enabled in the Internet Information Services (IIS) Manager, and the maximum file size was reasonable (it defaults to 256KB, which is fine for our usage)

All this was fine.

From here the Internet Information Services (IIS) Manager is useless, you have to start looking into configuration files instead.

There are 3 settings which have a major impact on caching in the IIS:

  1. frequentHitThreshold: How many times should the same file be requested, before it’s cached ? Defaults to 2.
  2. frequentHitTimePeriod: Time interval in which the same file should be requested {frequentHitThreshold} times, in order to be cached. Defaults to 10 seconds.
  3. ObjectCacheTTL: How long the cache lives, before being flushed for unused files. Defaults to 30 seconds

With the default settings, I checked perfmon for some caching counters (all found under “Web Service Cache”):

  • Current File Cache Memory Usage
  • Current Files Cached
  • File Cache Hits %

These should be pretty self-explanatory.

At default the server used around 8MB memory and cached 600 files. Also the File Cache Hits % was around 1%.

Clearly some horrible numbers…

First I changed frequentHitThreshold to 1, meaning every file is instantly cached. This also means frequentHitTimePeriod is superfluous as it’s never checked.
This alone changed our total counters to this:

  • Current File Cache Memory Usage: 86MB
  • Current Files Cached: 6.000
  • % Idle Time: 85%

Now the server caches 10 times as many files, and the disk is hardly doing anything! (Especially the % Idle Time is important here, as it’s a pretty direct measurement of the end-user performance)

However we still weren’t using that much memory, as many of our images are actually thumbnails at around 10 KB or less.

Not content with that, I went ahead and doubled ObjectCacheTTL to 60 seconds. However this was today, and our primetime is during the evening, so my only measurements of this, are made with quite few users online.

With the default setting of 30 seconds (but frequentHitThreadhold set to 1) we we’re caching around 1.600 files on average.
After changing the setting to 60 seconds, we’re caching around 6.000 files! That’s almost a factor 4 improvement on top of the existing factor 10 improvement.

This evening will tell, whether we get a factor 40 improvement during prime time or not, but so far it’s looking very promising.

Tuning Instructions

Now the last piece missing in the puzzle: Where do you change these settings ? Well here goes:

frequentHitThreshold and frequentHitTimePeriod both reside in a file called “applicationHost.config“, which is found in “%windir%\system32\inetsrv\config

This file resembles web.config and machine.config quite a bit, so you should be familiar with the syntax.

The element these attributes are set on, is called “serverRuntime” and should exist already, albeit being empty.
e.g. it looks like this:
<serverRuntime />

The syntax for setting the attributes is as follows:
<serverRuntime frequentHitTimePeriod="00:00:10" frequentHitThreshold="1" />

so frequentHitTimePeriod is a TimeSpan, and frequentHitThreshold is an unsigned integer (should be 1 or more)
Note that setting frequentHitTimePeriod is in fact superfluous when frequentHitThreshold is set to 1.

These changes take effect as soon as you save the file, and the result can be observed in perfmon immediately.

The last setting, ObjectCacheTTL resides in the registry database, and doesn’t exist per default either. It’s location is “HKEY_LOCAL_MACHINE\System\CurrentControlSet\services\InetInfo\Parameters

Create a new DWORD (32-bit) value called “ObjectCacheTTL” and set its decimal value to the number of seconds, which should pass between the IIS flushing its cache for unused files. (It only removes files which weren’t requested after initially being added to the cache)

There is however a minor quirk to this, as there is in fact *another* registry key, which does pretty much the same – only it doesn’t… Confused ? That’s understandable.

The thing is, IIS caches files either in user-mode or kernel-mode, and depending on which mode it caches in, it *may* use a different registry key for the flush interval.
So to be certain it actually does what you want, create another DWORD (32-bit) value called “OutputCacheTTL” and give it the same value as ObjectCacheTTL.

The reason I say *may* use, is because I haven’t been able to find any official documentation stating exactly when each of the keys are used. So setting both is merely a “better safe than sorry” approach.

After setting these, you’ll need to reboot Windows for them to take effect – so don’t do this during your prime time πŸ˜‰

Finally a word of caution: Whilst tuning the server, make sure you keep an eye on all bottlenecks of the system. If done wrong, you might end up using all your memory, and slowing down the system due to paging. We had no such problems, as we had plenty of memory, and our files are very small – your scenario might be different.

If you have limited memory, your best bet is to modify only frequentHitTimePeriod as it has the least impact of the 3, and as such can be used for more subtle tuning, with little risk of consuming too much memory.

As always, feel free to drop a comment with your own experiences πŸ™‚

Update

Today I got the results from the evening perfmon, and whilst we didn’t archive a factor 40 improvement, we still got around 20 times as many files cached (approx. 12.000), and used 185 MB for cache (compared to the puny 8 MB it initially used)

The disk has almost no activity, and as such we’ll tune it no further for now, even though we could.

Second update

Whilst the server was indeed running very well, we anticipate a somewhat large increment in simultaneous online users, which in return means a higher load on the server. So I set out to increase the caching yet again.

Since I could only increment the ObjectCacheTTL / OutputCacheTTL, that’s what I did. I increased it from 60 seconds to a whopping 180 seconds, and the result was… As expected πŸ™‚

In primetime we’re now caching 26.000 files on average (maxing out at 52.000) and the IIS consumes 400 MB RAM for caching, this is excellent!

I’m looking forward to the increase in users, to see how it copes – so far the disks are doing almost no work.

Third update

As pointed out in the comments by Rovastar, I actually had another issue with the file caching.
Every so the IIS would completely empty the cache (about every 2-3 minutes) for no apparent reason.

After a lot of debugging, and asking the question at Serverfault, I finally figured it out:
In our web.config of the site serving this static content, we had this section:

<caching enableKernelCache="true">

After removing it, the odd complete flushes disappeared. The IIS still makes large flushes at times, however now it keeps around 50.000 files in the cache, when doing so.
At the same time our performance counter for “File cache hit %” went up from 5% to 33% – A huge increase.

If you’re interested, the question at Serverfault can be reached here: http://serverfault.com/questions/448942/why-is-iis-7-5-flushing-file-cache-very-often

28 thoughts on “Tuning IIS 7 for static content

  1. Just amazing! Kudos on being so complete in your descriptions. Question: does IIS 7 have a max size of files cached? We have a LOT of large images 1MB+ will those be cached by default?

    thanks

  2. AFAIR there’s no max size to the files, but it’s been a while since I looked into this, so my memory may be off.

  3. I know “netsh http show cachestate” shows list of files in kernelcache.

    Do you know the CMD command or folder location to view list of files in “File cache”?

    trying to tune file cache hit %

    Please email me notice if you reply.

    thanks

  4. Sorry but no I don’t know such a command πŸ™

    However if you only want to monitor/tune the hit %, the performance counters should provide enough information to help you.

  5. Ok. I have lots of misses in perfmon so wanted to see what files. I’m now tracing them with logging tool.

  6. Oh and found the max file setting:
    HKLM\System\CurrentControlSet\Services\InetInfo\Parameters\MaxCachedFileSize

  7. Very nice, that might come in handy for me as well.

    Thanks for posting it πŸ™‚

  8. your welcome.

    Also I found out why I have all those misses it drags my hit rate % down to under 40%. πŸ™

    I’m using wordpress/php and thus delivering a lot of dynimic pages and not just static files/images. If I add “.php” to IIS 7’s Output cache it WOULD increase hit rate %. However, I can’t/won’t because I’m using a wordpress cache plugin. It’s a lot faster than IIS Output caching and more compatible:
    http://ruslany.net/2008/12/speed-up-wordpress-on-iis-70/

  9. Ah yes that explains quite a lot, and I agree you shouldn’t rely on the IIS for caching PHP pages.

    Actually I’ll admit I don’t understand why you’re using IIS in the first place – I’d host PHP sites on a Linux / Apache combination any day πŸ˜€

  10. Hi!

    Thanks very much for putting up this article. I recently moved from IIS 5 (Win2K) to IIS 7.5 on ’08 R2. I was very surprised to see that virtualy nothing was being cached πŸ™

    This was after I had configured my IIS5 machine to just cache _everything_ on a permanent basis (unless the file changed).

    I’m looking to get the same functionality from IIS 7.5 and to that end I implemented your;

    entry but found that (as above) it still flushes the @&#^$ cache.

    Is there any way to just force it to keep the item in cache indefinately? It would be great if I could change it to operate in the same way as IIS5 – cache permanently unless the file itself changes.

    Any ideas would be greatly appreciated!

  11. Btw, I found that creating the registry setting doesn’t require a server re-start. You can (instead) just go into the IIS management interface and stop/start the web service πŸ™‚

  12. In the IIS 7.5 management interface you can specify expiration type to “file changes”, however this doesn’t affect static content (much like the rest of the settings in the interface)

    Still it indicates that the IIS *is* capable of flushing based on local file changes, so it should be possible I think. Unfortunately I haven’t looked into how to do this.

    A dirty easy hack would be to just set the object time to live extremely high – this would prevent most files from being flushed.

    The IIS doesn’t flush files which are being requested often.

    That is, it checks whether a file was requested since last flush, and if it was, it won’t be flushed.

    That being said, I’ve noticed some “strange” behavior regarding flushing: Our server completely wipes the cache every so often (approximately every 15 minutes). It seems object time to live has no effect on this, so the aforementioned hack might not solve your problem.

    I’ve been unable to find any documentation, as to why it wipes the cache πŸ™

    Sorry I couldn’t be of more help, but hopefully you’ll find a solution πŸ™‚

    Oh and thanks for the tip with just restarting the web service – that’s a lot faster than restarting the entire server.

  13. I have to say … KUDOS! This saved us! 100,000 online users getting images served from our static server … we lost countless hours for finding a solution!

  14. Bravo. You save us.
    Do you have plan to work in Microsoft? Microsoft user need you πŸ™‚

  15. Haven’t any plans for that at the moment no, but thanks for the compliment πŸ™‚

  16. FURTHER IMPROVEMENTS:

    This is an excellent post.

    For further improvements you can also disable IIS from updating the Last Access Timestamp. Every time a file is accesses this value is updated which can slow down the web server significantly.

    Please note that this is now disabled by default in Windows 2008 , but not in earlier versions.

    Using regedit, navigate to the following subkey:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

    If it does not already exist, add a new DWORD value: NtfsDisableLastAccessUpdate

    Give the new key a value of: 1

    This prevents the file system updating the files Last Access Timestamp every time IIS requests the file!

    Restart the server to see changes.

  17. Excellent tip – Unfortunately I cannot implement it as all our servers are already running Windows 2008.

    However for those “stuck” on 2003 this should easily provide a decent performance increase.

  18. Just so all the information is in one place. This is the most complete reference I have seen for understanding and tweaking disc cache for static files in IIS7 and I often link to it when answering cache queries on forums.iis.net

    Steffen found a solution to the regular (15 minutes) cache removal here:
    http://serverfault.com/questions/448942/why-is-iis-7-5-flushing-file-cache-very-often

    “Finally found the answer to the problem.

    We had this in our web.config for the site (despite not running any managed code on the site)

    After changing this to false the IIS stopped doing these full flushes. It still flushes some files regurlarly, but it’s no longer a complete flush (around 50k files stay in cache)

    Our file cache hit % has increased from 5% to 33%, so this definately helps a lot.

    Not sure why kernel caching causes this behaviour, but I can live with the slight overhead of caching in usermode.

    Thought I’d add this as an answer to help others with similar issues.

    Steffen”

    I haven’t checked but as a guess if this was linked to when a user was logged in only I am thinking this might be to do with some sort of login/(windows) authentication credentials cache which likely linked to the kernel cache.

    But don’t quote me I haven’t researched it and that is just a speculative guess. πŸ™‚

  19. Also as I keep forgetting and to keep stuff in one place. I also meant to add, if you need to see what is in your server cache.

    Use this command:

    netsh http show cachestate

  20. Actually the part from serverfault, was my own question. I should probably had added it to the blog entry as well.

    Thanks for pointing that out πŸ™‚

  21. Great post. My server faster 20 times now.

    Thanks.

    Pingdom.com test results excellent. My website is faster than 91% of all tested websites, 1 MB page size, 88 request, 61 images πŸ™‚

  22. This is a great article – thank you for making is as I’ve been looking everywhere for this info and the folks on the IIS site are really useless.

    I’m having an issue though, I’ve configured everything exactly as mentioned, except I set the OutputCacheTTL/ObjectCacheTTL to one WEEK (I have lots of memory) I then re-started the VM and started monitoring the cache while moving through the site and looking at pictures and .pdf files etc.

    What did I find? Current files cached is sitting at 25 and it’s not increasing πŸ™

    I hope you’re still here and checking this post from time to time as this is something I’d really like to get sorted. I have a low volume site and I really want everything running from cache.

  23. Seems very odd – if you do an IISReset (to flush the cache), will it then climb back up to 25 ?

    Also have you checked whether your files are to large to cache ? (check both in the IIS management UI as well as in registry here: HKLM\System\CurrentControlSet\Services\InetInfo\Parameters\MaxCachedFileSize)

  24. Thanks for testing this on IIS 8.5, I haven’t had the chance to do so myself yet πŸ™‚

  25. I made the following script to see how much is actively cached at this moment.
    Usuable if you use git for windows of cygwin

    netsh http show cachestate | grep ‘Content length’ | awk ‘{ SUM += $3;TOTAL+=1} END { printf “Total kilobytecount %.1fkb(%.2fmb) in %i files in active cache”,SUM/1024,SUM/1024/1024,TOTAL }’

Leave a Reply

Your email address will not be published. Required fields are marked *