The whole problem started with the customer reporting performance difference between dom0 and guest for IO. Every process which initiates an IO in gest starts to slow down things. Dom0 and guest is identically configured with 1024MB memory and four vcpus in both and etc.

Both Dom0 and guest is idle at this time and no processes or applications are running as they are all stopped while doing the dd performance test.

Finally we had to identify that this problem is only visible if the backend is hardware raid 5 (for me) raid 1 (for customer).

So the numbers are as below:

In my test:

With Hardware *RAID1*:

Host: (default: cfq) 

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Fri Apr 24 01:16:44 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 1.9104 seconds, 274 MB/s

real	0m2.013s  ----------------------------------- time taken to write 500 1MB blocks
user	0m0.000s
sys	0m1.344s
Fri Apr 24 01:16:46 IST 2009

real	0m8.010s ------------------------------------ time taken to sync it
user	0m0.000s
sys	0m0.096s
Fri Apr 24 01:16:54 IST 2009

Guest: (default: noop) while host is CFQ
========================================

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Fri Apr 24 01:16:26 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 2.24071 seconds, 234 MB/s

real	0m2.246s
user	0m0.012s
sys	0m1.236s
Fri Apr 24 01:16:28 IST 2009

real	0m9.065s
user	0m0.000s
sys	0m0.024s
Fri Apr 24 01:16:37 IST 2009

The same is the number with CFQ in the guest.

Now the numbers with RAID5 (hardware):
======================================

Host (cfq):

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Thu Apr 23 07:51:10 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 3.12719 seconds, 168 MB/s

real	0m3.131s
user	0m0.000s
sys	0m1.264s
Thu Apr 23 07:51:13 IST 2009

real	0m12.801s
user	0m0.000s
sys	0m0.044s
Thu Apr 23 07:51:26 IST 2009

Guest (noop) while host is CFQ:

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Thu Apr 23 07:51:44 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 10.1823 seconds, 51.5 MB/s

real	0m10.190s
user	0m0.000s
sys	0m1.216s
Thu Apr 23 07:51:54 IST 2009

real	0m43.516s
user	0m0.000s
sys	0m0.004s
Thu Apr 23 07:52:38 IST 2009

Guest (cfq)

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Thu Apr 23 07:53:29 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 6.59633 seconds, 79.5 MB/s

real	0m6.601s
user	0m0.020s
sys	0m1.184s
Thu Apr 23 07:53:35 IST 2009

real	0m27.300s
user	0m0.000s
sys	0m0.004s
Thu Apr 23 07:54:03 IST 2009

Host (deadline):
================

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Thu Apr 23 08:00:14 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 3.2883 seconds, 159 MB/s

real	0m3.292s
user	0m0.000s
sys	0m1.228s
Thu Apr 23 08:00:17 IST 2009

real	0m12.434s
user	0m0.000s
sys	0m0.032s
Thu Apr 23 08:00:29 IST 2009

Guest (noop) while host is deadline:
====================================

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Thu Apr 23 08:02:41 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 10.8122 seconds, 48.5 MB/s   ---------------------------------- Why is it just 48?

real	0m10.817s
user	0m0.000s
sys	0m1.196s
Thu Apr 23 08:02:52 IST 2009

real	0m46.823s
user	0m0.000s
sys	0m0.000s
Thu Apr 23 08:03:39 IST 2009

Guest(cfq) while host is deadline:
==================================

# sync ; date ; time dd if=/dev/zero of=/tmp/test bs=1M count=500 ; date ; time sync ; date
Thu Apr 23 08:04:24 IST 2009
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 6.56803 seconds, 79.8 MB/s ------------------------------------Why is two times to noop?

real	0m6.572s
user	0m0.000s
sys	0m1.220s
Thu Apr 23 08:04:31 IST 2009

real	0m30.750s
user	0m0.000s
sys	0m0.012s
Thu Apr 23 08:05:01 IST 2009

These tests are done on the same system. I configured first RAID1 in the controller BIOS. Then broke that and configured that in RAID5 after breaking the raid1.

So the customer problem is that he is seeing the same difference between dom0 and guest that I am seeing in my test with RAID5. With raid1, I don't see any difference at all. May be this is not related to the the techlist thread!!! So there are two questions?

1 - Why are those huge differences between dom0 and guest only when the backend is hardware raid5 and not with raid1(in my testing) or raid less disks? (In customers' testing he is seeing this differece with raid1 as well). One refence with discusses this is

http://lists.xensource.com/archives/html/xen-users/2008-01/msg00813.html
http://lists.xensource.com/archives/html/xen-users/2008-01/msg00865.html

2 - Other question here is that if the backend is raid5, cfq gives 2x performance to noop and we preferred to hardcore noop!!!