What makes some Citrix XenApp workloads perform better on Citrix XenServer?

hey folks,

So.. ever since Citrix bought XenServer and had their way with it to optimize the virtualization platform for XenApp i have wondered what it is exactly that makes XenApp run better on XenServer.

I have talked to anyone i could think of, poked and prodded everyone i have come in contact with, and the fact is that everyone comes back to Shadow Page Tables. this is the technology that makes Citrix XenApp perform better on XenServer than other Hypervisors. And more than that, the memory allocated  on the host to build these Shadow Page Tables.

So how to describe the Shadow Page Tables memory, George Dunlap did a good explanation in Anil Madhavapeddy’s blog post:

"The hypervisor allocates a certain amount of memory for each VM to use for shadow page tables; this is called shadow page table memory. As new page tables are created and old ones aren’t used anymore, the hypervisor cycles through this shadow page table memory. When it needs a new page and there isn’t enough, it will ‘unshadow’ the guest page tables that haven’t been used for the longest time to reclaim shadow memory, so that it can use more."

"We don’t know ahead of time how much shadow memory a given workload will use, but we can estimate based on the amount of memory that the VM has. We allocate enough shadow memory for each page to be mapped once, more or less, then add an extra 50% to have some slack. For all the workloads we’ve tested, that’s been enough – except XenApp."

"XenApp is the one workload we’ve found that requires more shadow page table memory than our standard default. Because XenApp generally starts hundreds of copies of the same process, the same memory ends up mapped in hundreds of different processes. What happens when all of those processes are active is that XenServer is continually unshadowing one process’ page tables in order to shadow another process’ pagetables; only to have to re-shadow the original ones a second or two later when it runs again! This is called thrashing, when there’s not enough of a limited resource."

So.. as this was located to be the issue, Citrix did a nice fix for this problem. In XenServer 4.1 they introduced what we all know as the "Optimized for XenApp" template in XenServer.. I for one always wondered what the f… that Shadow Multiplier number was good for. So.. what i have learned, maybe you all knew, is that the Shadow Multiplier is the amount of memory dedicated to allow Shadow Page Tables for a Virtual Server.

So, in conclusion, the Shadow Multiplier available on XenServer templates could actually be used for tuning of other workloads to provide better performance. And naturally, Terminal Servers should benefit almost as much as XenApp servers.. i am assuming Citrix has done other tweaks as well..

Anil Madhavapeddy’s blog was my first lead and after that i have been trying to dig deeper as i would love to be able to better explain the reasons for why XenApp runs better on XenServer. I have not been able to find any technical descriptions or get in contact with people who could explain further, but i will continue to dig deeper, as i personally have had enough of the Vendor-Specific whitepapers which more than anything else, promotes that specific vendor, not the strength or weaknesses in different areas. So .. Georg Dunlap or similar, if you are out there, give me a shout

If you want to read more in depth about Page Tables and the way Memory in virtualized machines work have a look at Anil’s Blog post.

/Rene Vester