Merge lp:~nttdata/nova/block-migration into lp:~hudson-openstack/nova/trunk

[Kei Masumoto]

Adding kvm-block-migration feature.

I wrote some description the below URL. I hope it may help for reviewing.
<http://etherpad.openstack.org/kvm-block-migration>

[Matt Dietz (cerberus)]

I can't vet this branch fully for functionality, but syntactically I found a few things:

some test failures:

======================================================================
ERROR: test_live_migration_works_correctly_no_volume (nova.tests.test_compute.ComputeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cerberus/code/python/nova/block-migration/nova/tests/test_compute.py", line 627, in test_live_migration_works_correctly_no_volume
    False)
  File "/Users/cerberus/code/python/nova/block-migration/.nova-venv/lib/python2.6/site-packages/mox.py", line 765, in __call__
    return mock_method(*params, **named_params)
  File "/Users/cerberus/code/python/nova/block-migration/.nova-venv/lib/python2.6/site-packages/mox.py", line 998, in __call__
    self._checker.Check(params, named_params)
  File "/Users/cerberus/code/python/nova/block-migration/.nova-venv/lib/python2.6/site-packages/mox.py", line 913, in Check
    'arguments' % (self._method.__name__, i))
AttributeError: live_migration does not take 5 or more positional arguments
-------------------- >> begin captured logging << --------------------
2011-06-17 13:06:36,744 AUDIT nova.auth.manager [-] Created user fake (admin: False)
2011-06-17 13:06:36,745 DEBUG nova.ldapdriver [-] Local cache hit for __project_to_dn by key pid_dn-fake from (pid=8116) inner /Users/cerberus/code/python/nova/block-migration/nova/auth/ldapdriver.py:153
2011-06-17 13:06:36,745 DEBUG nova.ldapdriver [-] Local cache hit for __dn_to_uid by key dn_uid-uid=fake,ou=Users,dc=example,dc=com from (pid=8116) inner /Users/cerberus/code/python/nova/block-migration/nova/auth/ldapdriver.py:153
2011-06-17 13:06:36,745 AUDIT nova.auth.manager [-] Created project fake with manager fake
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_live_migration_works_correctly_no_volume (nova.tests.test_compute.ComputeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cerberus/code/python/nova/block-migration/nova/tests/test_compute.py", line 81, in tearDown
    super(ComputeTestCase, self).tearDown()
  File "/Users/cerberus/code/python/nova/block-migration/nova/test.py", line 94, in tearDown
    self.mox.VerifyAll()
  File "/Users/cerberus/code/python/nova/block-migration/.nova-venv/lib/python2.6/site-packages/mox.py", line 286, in VerifyAll
    mock_obj._Verify()
  File "/Users/cerberus/code/python/nova/block-migration/.nova-venv/lib/python2.6/site-packages/mox.py", line 506, in _Verify
    raise ExpectedMethodCallsError(self._expected_calls_queue)
ExpectedMethodCallsError: Verify: Expected methods never called:
  0. nova.db.instance_get(<nova.context.RequestContext object at 0x10db02ed0>, 1) -> <nova.db.sqlalchemy.models.Instance object at 0x10db021d0>
  1. nova.db.queue_get_for(<nova.context.RequestContext object at 0x10db02ed0>, 'compute', 'dummy') -> 'compute.dummy'
-------------------- >> begin captured logging << --------------------
2011-06-17 13:06:36,744 AUDIT ...

review: Needs Fixing

[Kei Masumoto (masumotok)]

Hello, thanks for reviewing this!

>
> 81 + block_migration=False, **kwargs):
> 99 + kwargs.get('disk'))
>
> I'd prefer if you just passed in disk as another parameter, here. If there
> was a whole suite of option args, then kwargs would be justifiable, but
> probably not for just one key
Agreed. I've already fixed it. Please check it.

>
> 192 + # TODO:
>
> Mind attaching a name to do the TODO so that person (presumably you) knows
> to follow up on it?
>
Agreed. I've already fixed it. (I added some error-recover operation, and removed "TODO")

> 403 + if block_migration:
> 404 + raise
>
> This raise should provide an exception type and message. Additionally,
> is this the exception you hoped to catch in the following except block?
This is our mistake. I've fixed it.

> 1070 + #disk_type = driver_nodes[cnt].get_properties().getContent()
>
> Looks like you meant to remove this line?
>
I rewirte comment, please check it.

P.S. Regarding to test failure, I checked all test pass after above code-fixing. If you got any testcase failure, please let me know.

[Matt Dietz (cerberus)]

I've got several mocked tests failing on my end, but I suspect it's me, as it's not the first branch I've seen this in.

I approve of this syntactically. I'd still prefer several other reviews that can properly vet it for functionality.

review: Approve

[Kei Masumoto (masumotok)]

Matt, thanks for approval!

[Dan Prince (dan-prince)]

Marking 'work in progress' to fix conflicts:

Text conflict in nova/scheduler/driver.py
Text conflict in nova/tests/test_libvirt.py

[Brian Waldon (bcwaldon)]

I can't functionally test this as well as I want to, so I could only review the code and run the unit tests. It looks good to me after these few very minor things are addressed:

390, 1190: misspelled "Destination"

514: You might as well i18n ("_(...)") this string

665-667: This example looks a bit off: missing a comma and duplicate keys memory_mb, local_gb

I'm also having the same test failures as cerberus, even in a clean virtual environment. Can you look into this?

review: Needs Fixing

[Kei Masumoto (masumotok)]

Hello brian, thanks for reviewing our branch!

> I'm also having the same test failures as cerberus, even in a clean virtual
> environment. Can you look into this?
Now I can reproduce this error. I always execute "run_test.sh -N", then no error occurs.
"run_test.sh" raises error that you and cerberus pointed out to me.

Fix it soon..

[Brian Waldon (bcwaldon)]

Thanks, Kei, all the tests pass now. Once the comment on line 666 is fixed (repeated keys), this is good to go.

[Brian Waldon (bcwaldon)]

It looks good to me, but I want somebody to deploy and run this. I added Vish as a reviewer hoping he can do that for us.

review: Approve

[Kei Masumoto (masumotok)]

> Review: Approve
> It looks good to me, but I want somebody to deploy and run this. I added Vish
> as a reviewer hoping he can do that for us.

Thank you, Brian! I am looking forward he is going to come for review.

[Vish Ishaya (vishvananda)]

I ran into a few errors testing this branch:

Trying to migrate from cloudbuilders03 to cloudbuilders02:

First error (is this expected?):

2011-07-18 17:41:51,673 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/root/nova/nova/exception.py", line 97, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/root/nova/nova/compute/manager.py", line 1143, in check_shared_storage_test_file
(nova.exception): TRACE: raise exception.FileNotFound(file_path=tmp_file)
(nova.exception): TRACE: FileNotFound: File /root/nova/nova/..//instances/tmperxzdb could not be found.
(nova.exception): TRACE:
2011-07-18 17:41:51,675 ERROR nova [-] Exception during message handling
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/root/nova/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/root/nova/nova/exception.py", line 123, in wrapped
(nova): TRACE: raise Error(str(e))
(nova): TRACE: Error: File /root/nova/nova/..//instances/tmperxzdb could not be found.
(nova): TRACE:

Second error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.6/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/root/nova/nova/virt/libvirt/connection.py", line 1563, in _live_migration
    FLAGS.live_migration_bandwidth)
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 581, in migrateToURI
    if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://cloudbuilder02/system
Removing descriptor: 10

Does cloudbuilders03 have to be able to hit qemu on the cloudbuilders02? Is there any way to factor this out? If not, is there any way to determine the ip address without dns lookup?

finally, the rollback failed:

(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/root/nova/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: TypeError: rollback_live_migration_at_destination() got an unexpected keyword argument 'context'
(nova): TRACE:

looks like this line:

262 + def rollback_live_migration_at_destination(self, ctxt, instance_id):

needs to use context instead of ctxt, as rpc calls via kwarg so the name can't be changed.

review: Needs Fixing

[Kei Masumoto (masumotok)]

Thanks for reviewing, Vish!
Regarding to 2nd comments, I need some advice from you. Please see below..

> First error (is this expected?):
(snip)
>
> 2011-07-18 17:41:51,673 ERROR nova.exception [-] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/root/nova/nova/exception.py", line 97, in wrapped
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/root/nova/nova/compute/manager.py", line 1143, in check_shared_storage_test_file
> (nova.exception): TRACE: raise exception.FileNotFound(file_path=tmp_file)
> (nova.exception): TRACE: FileNotFound: File /root/nova/nova/..//instances/tmperxzdb could not be found.
> (nova.exception): TRACE:

In case of block migration, the above is expected error. This checks any shared FLAGS.instances_dir is not mounted by any shared storage. But this message causes misunderstanding - I'm going to add some additional messages, such as, " this can be ignored".

> Second error:
>
> (snip)
> Does cloudbuilders03 have to be able to hit qemu on the cloudbuilders02? Is there any way to factor this out? If not, is there any way to determine the ip address without dns lookup?

Hmm.. libvirtd running at cloudbuilders03 has to be network-reachable to libvirtd running on cloudbuilders02, AFAIK. Also, if dns lookup is unprefered, I propose to write some entry to /etc/hosts.

Users can start live migration such as, "nova-manage vm live_migration(block_migration) i-00000001 hostname". Using hostname, nova can check given hostname is correct. Because nova-compute registers its hostname to Service table. Otherwise, if user specifiy wrong hostname, nova-compute understands wrong hostname(or ipaddress) when it try to start live migration(i.e. source libvirtd connects to destination libvirtd and it has to wait timeout). What do you think? Do we have to add a new FLAGS that can pass hostname checking?

> finally, the rollback failed:
(snip)
>
> 262 + def rollback_live_migration_at_destination(self, ctxt, instance_id):
>
> needs to use context instead of ctxt, as rpc calls via kwarg so the name can't be changed.

That is our mistake. I'll fix it soon..

[Dave Walker (davewalker)]

On 19/07/11 05:54, Kei Masumoto wrote:

<SNIP>
>> 2011-07-18 17:41:51,673 ERROR nova.exception [-] Uncaught exception
>> (nova.exception): TRACE: Traceback (most recent call last):
>> (nova.exception): TRACE: File "/root/nova/nova/exception.py", line 97, in wrapped
>> (nova.exception): TRACE: return f(*args, **kw)
>> (nova.exception): TRACE: File "/root/nova/nova/compute/manager.py", line 1143, in check_shared_storage_test_file
>> (nova.exception): TRACE: raise exception.FileNotFound(file_path=tmp_file)
>> (nova.exception): TRACE: FileNotFound: File /root/nova/nova/..//instances/tmperxzdb could not be found.
>> (nova.exception): TRACE:
> In case of block migration, the above is expected error. This checks any shared FLAGS.instances_dir is not mounted by any shared storage. But this message causes misunderstanding - I'm going to add some additional messages, such as, " this can be ignored".
>
<SNIP>

I understand this is to check that nova is attempting a live migration
without shared storage, when the instance is infact on shared storage?

If so, surely this should just be caught, and turned into a LOG.debug..
rather than a "this can be ignored"?

Thanks.

[Kei Masumoto (masumotok)]

> I understand this is to check that nova is attempting a live migration without shared storage, when the instance is infact on shared storage?
>
> If so, surely this should just be caught, and turned into a LOG.debug..
> rather than a "this can be ignored"?

There are 2 patterns. One is that normal live migration. Then nova expects FLAGS.instances_path is on shared storage. Another is block migration. then nova expects FLAGS.instances_path is *NOT* on shared storage. The exception is caught, but for exceptions raised at compute node(not scheduler), exception message cannot be suppressed. That is what I am thinking about.

[Vish Ishaya (vishvananda)]

On Jul 19, 2011, at 3:37 AM, Kei Masumoto wrote:
>
> There are 2 patterns. One is that normal live migration. Then nova expects FLAGS.instances_path is on shared storage. Another is block migration. then nova expects FLAGS.instances_path is *NOT* on shared storage. The exception is caught, but for exceptions raised at compute node(not scheduler), exception message cannot be suppressed. That is what I am thinking about.

Why not modify the method to return different values based on whether the exception is hit, rather than passing the exception back to the caller?

[Kei Masumoto (masumotok)]

Hi,

>> There are 2 patterns. One is that normal live migration. Then nova expects FLAGS.instances_path is on shared storage. Another is block migration. then nova expects FLAGS.instances_path is *NOT* on shared storage. The exception is caught, but for exceptions raised at compute node(not scheduler), exception message cannot be suppressed. That is what I am thinking about.
>
> Why not modify the method to return different values based on whether the exception is hit, rather than passing the exception back to the caller?

Please check updated version(updated 12 hours before). I've already fixed as you said on this matter. I should have said it before changing status "work in progress" -> "needs review".

[Vish Ishaya (vishvananda)]

Still having some trouble with this. Setup was a little tough, I had to
1) add /etc/hosts entries for the other hosts
2) modify /etc/libvirt/libvirt.conf to turn off tls and turn on tcp and turn off authentication
3) modify /etc/init/libvirt-bin.conf to add the -l parameter
4) stop and start libvirt-bin (restart doesn't add the extra param

Now I'm getting a new error:

2011-07-20 15:37:57,062 DEBUG nova.rpc [-] Making asynchronous cast on compute.cloudbuilder03... from (pid=20518) cast /root/nova/nova/rpc.py:554
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.6/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/root/nova/nova/virt/libvirt/connection.py", line 1563, in _live_migration
    FLAGS.live_migration_bandwidth)
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 581, in migrateToURI
    if ret == -1: raise libvirtError ('virDomainMigrateToURI() failed', dom=self)
libvirtError: Unknown failure
Removing descriptor: 12

Any way to find out what might have failed? Do I need a more up-to-date version of kvm?

Vish

[Vish Ishaya (vishvananda)]

on a positive note, it recovered from the failure ok, the vm just didn't migrate.

[Vish Ishaya (vishvananda)]

libvirt logs from cloudbuilder02:

16:02:49.825: 21622: info : virDomainLoadAllConfigs:8190 : Scanning for configs in /etc/libvirt/uml
16:03:16.783: 21629: error : qemuMonitorTextMigrate:1260 : operation failed: migration to 'tcp:cloudbuilder03.Marriot.com:49152' failed: migration failed

16:03:17.101: 21629: error : remoteIO:10609 : Unknown failure

and from cloudbuilder03:

16:02:18.197: 19115: info : umlStartup:427 : Adding inotify watch on /var/run/libvirt/uml-guest
16:02:18.197: 19115: info : virDomainLoadAllConfigs:8190 : Scanning for configs in /etc/libvirt/uml
16:03:15.632: 19118: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/disk' to '103:110'
16:03:15.632: 19118: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/console.log' to '103:110'
16:03:15.632: 19118: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/kernel' to '103:110'
16:03:15.632: 19118: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/ramdisk' to '103:110'
16:03:16.986: 19121: info : virSecurityDACRestoreSecurityFileLabel:139 : Restoring DAC user and group on '/root/nova/nova/..//instances/instance-00000001/disk'
16:03:16.986: 19121: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/disk' to '0:0'
16:03:16.986: 19121: info : virSecurityDACRestoreSecurityFileLabel:139 : Restoring DAC user and group on '/root/nova/nova/..//instances/instance-00000001/console.log'
16:03:16.986: 19121: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/console.log' to '0:0'
16:03:16.986: 19121: info : virSecurityDACRestoreSecurityFileLabel:139 : Restoring DAC user and group on '/root/nova/nova/..//instances/instance-00000001/kernel'
16:03:16.987: 19121: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/kernel' to '0:0'
16:03:16.987: 19121: info : virSecurityDACRestoreSecurityFileLabel:139 : Restoring DAC user and group on '/root/nova/nova/..//instances/instance-00000001/ramdisk'
16:03:16.987: 19121: info : virSecurityDACSetOwnership:99 : Setting DAC user and group on '/root/nova/nova/..//instances/instance-00000001/ramdisk' to '0:0'

Don't know if that helps at all

[Vish Ishaya (vishvananda)]

I think I found the issue by turning on debug:

16:34:40.921: 20692: debug : virNWFilterLookupByName:11602 : conn=0xf51870, name=nova-instance-instance-00000001-secgroup
16:34:40.921: 20691: debug : virEventMakePollFDs:361 : Prepare n=5 w=6, f=9 e=25
16:34:40.921: 20691: debug : virEventMakePollFDs:361 : Prepare n=6 w=7, f=8 e=25
16:34:40.921: 20691: debug : virEventMakePollFDs:361 : Prepare n=7 w=8, f=7 e=25
16:34:40.921: 20691: debug : virEventMakePollFDs:361 : Prepare n=8 w=9, f=15 e=1
16:34:40.921: 20691: debug : virEventCalculateTimeout:302 : Calculate expiry of 2 timers
16:34:40.921: 20691: debug : virEventCalculateTimeout:332 : Timeout at 0 due in -1 ms
16:34:40.921: 20692: debug : nwfilterLookupByName:257 : Network filter not found: no nwfilter with matching name 'nova-instance-instance-00000001-secgroup'
16:34:40.921: 20691: debug : virEventRunOnce:583 : Poll on 9 handles 0x7f7334081440 timeout -1
16:34:40.921: 20692: debug : remoteSerializeError:146 : prog=536903814 ver=1 proc=175 type=1 serial=15, msg=Network filter not found: no nwfilter with matching name 'nova-instance-instance-00000001-secgroup'

Looks like security groups aren't made on the new host before migration...

[Kei Masumoto (masumotok)]

Thanks for trying, Vish! Please give us some time to reproduce, since this is first time to look at it.
In our expectation, secgroup is prepared before migrating at any condition. But the log you sent shows setting up secgroup is failed. Let me check it..

[Vish Ishaya (vishvananda)]

Just FYI. I migrated once and it failed to connect. Then I fixed the libvirt
listen and tried to migrate again. If you can't reproduce I will try again
tomorrow with a new instance.
On Jul 20, 2011 5:16 PM, "Kei Masumoto" <email address hidden> wrote:
> Thanks for trying, Vish! Please give us some time to reproduce, since this
is first time to look at it.
> In our expectation, secgroup is prepared before migrating at any
condition. But the log you sent shows setting up secgroup is failed. Let me
check it..
>
>
>
> --
> https://code.launchpad.net/~nttdata/nova/block-migration/+merge/64662
> You are reviewing the proposed merge of lp:~nttdata/nova/block-migration
into lp:nova.

[Kei Masumoto (masumotok)]

Thank you for your reply! Actually in our development environment, if migrating is failed over and over again for some reason(we make many try-and-errors in coding/testing, don’t we?), my colleague and I sometimes encountered the situation that restarting libvirtd/OS is necessary. But once we completed coding/testing, it is not happened. Anyway, my colleague and I are going to try reproducing, and report to you again.

[Kei Masumoto (masumotok)]

Hi Vish,

We are still trying to reproduce... let me make sure about your environment. From your log you sent to me, I suppose
1) --firewall_driver=nova.virt.libvirt.firewall.NWFilterFirewall
2) ami/ari/aki is devided.
3) ami is qcow2
Is that correct? Above 3 points may be important maybe not. I just want to make sure anything to reproduce.

Thanks!

[Vish Ishaya (vishvananda)]

Below,

On Jul 21, 2011, at 6:46 AM, Kei Masumoto wrote:

> Hi Vish,
>
> We are still trying to reproduce... let me make sure about your environment. From your log you sent to me, I suppose
> 1) --firewall_driver=nova.virt.libvirt.firewall.NWFilterFirewall

No, using the default Iptables driver
> 2) ami/ari/aki is devided.

correct

> 3) ami is qcow2

the ami is actually raw, but i have --use_cow_images on (which is the default). That means that the image is actually a copy on write snapshot that is using the raw base image in instances/_base

> Is that correct? Above 3 points may be important maybe not. I just want to make sure anything to reproduce.

Also, I tried with a new instance just in case and i'm seeing the same issue.

Vish

>
> Thanks!
>
> --
> https://code.launchpad.net/~nttdata/nova/block-migration/+merge/64662
> You are reviewing the proposed merge of lp:~nttdata/nova/block-migration into lp:nova.

[Kei Masumoto (masumotok)]

Hi, thanks for your reply! Today I got successful to reproduce, but I am not quite sure. Please see below.

In our today's trial, we use same image(ami/ari/aki is divided, raw image and use_cow_image=True). Then block migration finishes successfully.
Also, I gave much thought to logs you sent to me. Here is the analysis.

>16:34:40.921: 20692: debug : remoteSerializeError:146 : prog=536903814 ver=1 proc=175 type=1 serial=15,
> msg=Network filter not found: no nwfilter with matching name 'nova-instance-instance-00000001-secgroup'

The reason for this log is unfilter_instance() in iptables driver is called twice. When live migration finishes, at source compute node, unfilter_instance() should be called to erase filtering rules. In addition, in case of block migration, erasing vm image at source compute node is necessary. Since I don’t like to define new method only for this purpose, I use destroy() in libvirt driver, and destroy() calls unfilter_instance() again. Then above logs popped up.
Although I may have to look for better handling, it is not critical issue for block migration failure.

One thing I would like to mention is I can reproduce block migration failure. At that time, at source compute node, I used different libvirt-bin package which is same version but is rebuilt by me. Then I found (probably) exactly same failure log that you sent. Once I tried "apt-get install libvirt-bin --reinstall", everything was fine. I ignored some test failures on rebuilding packages, that may be the reason :). So, from this experience, I am wondering that you use your own custom libvirt-bin package or not. Any comment on this? For example, below is md5sum in my environment.

> root@compute01:/opt/openstack/dev/nova/block-migration-rev1172 # md5sum /usr/sbin/libvirtd
> c0cc0abbb9e93c6152fef33bcee17255 /usr/sbin/libvirtd

BTW, I would highly appreciate if you show me nova.conf at cloudbuilder02 and cloudbuilder03. It is very useful for further analysis.

Thanks in advance,
Kei

[Vish Ishaya (vishvananda)]

Hope this helps:

root@cloudbuilder02:~# cat nova/bin/nova.conf
--verbose
--nodaemon
--dhcpbridge_flagfile=/root/nova/bin/nova.conf
--network_manager=nova.network.manager.FlatDHCPManager
--my_ip=192.168.1.12
--public_interface=eth0
--vlan_interface=eth0
--sql_connection=mysql://root:nova@192.168.1.12/nova
--auth_driver=nova.auth.dbdriver.DbDriver
--libvirt_type=qemu
--fixed_range=10.0.0.0/24
--glance_api_servers=192.168.1.12:9292
--rabbit_host=192.168.1.12
--flat_interface=eth0
root@cloudbuilder02:~# dpkg -l | grep libvirt
ii libvirt-bin 0.8.8-1ubuntu3~ppamaverick1 the programs for the libvirt library
ii libvirt0 0.8.8-1ubuntu3~ppamaverick1 library for interfacing with different virtualization systems
ii python-libvirt 0.8.8-1ubuntu3~ppamaverick1 libvirt Python bindings

[Vish Ishaya (vishvananda)]

different version of libvirt for sure:

root@cloudbuilder02:~# md5sum /usr/sbin/libvirtd
69dfa442a2b978b493b555d7dfc88ae0 /usr/sbin/libvirtd

[Kei Masumoto (masumotok)]

Thanks. Now I've got extra hints. I have different environment from yours. Here is ours. I'll report to you soon..

> root@compute01:/root# dpkg -l | grep libvirt
> ii libvirt-bin 0.8.8-1ubuntu6.2 the programs for the libvirt library
> ii libvirt0 0.8.8-1ubuntu6.2 library for interfacing with different virtualization systems
> ii python-libvirt 0.8.8-1ubuntu6 libvirt Python bindings

> root@compute01:/root# uname -a
> Linux cb-blsv4b1 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
(meaning Lucid, not merverick)

> root@compute01:/root# dpkg -l | grep kvm
> ii kvm 1:84+dfsg-0ubuntu16+0.14.0+noroms+0ubuntu4 dummy transitional package from kvm to qemu-kvm
> ii kvm-pxe 5.4.4-7ubuntu2 PXE ROM's for KVM
> ii qemu-kvm 0.14.0+noroms-0ubuntu4 Full virtualization on i386 and amd64 hardware

[Kei Masumoto (masumotok)]

Hi Vish, let me ask one additional question - are you using KVM on VMWare (or anyway, KVM on some other hypervisor)?
Today, I have lots of our network problem, so futher analysis has not been done yet(even installing marverick took lots of time..). I'm trying to check in marverick environment, but if you are using above situation, I think it is also better to be checked.

Thanks in advance,
Kei

[Vish Ishaya (vishvananda)]

No, kvm is running on bare metal. I have it set to use software (qemu) mode though. I have also tried with kvm mode on and i get the same error.

[Kei Masumoto (masumotok)]

Hello Vish,

I installed maverick and tried to reproduce today. Unfortunately, I still cannot reproduce the error. But this error obviously looks like KVM/libvirt related error. In this assumption, once again, I would like to share our libvirt.conf to check any additional config exists. Also, I attached version info just for reference.

--- [ libvirtd.conf ]---
(p.s. comments are removed.)

listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none"

--- [ versions ] ---
ii libvirt-bin 0.8.8-1ubuntu3~ppamaverick1 the programs for the libvirt library
ii libvirt0 0.8.8-1ubuntu3~ppamaverick1 library for interfacing with different virtualization systems
ii python-libvirt 0.8.8-1ubuntu3~ppamaverick1 libvirt Python bindings
ii kvm 1:84+dfsg-0ubuntu16+0.14.0~rc1+noroms+0ubuntu4~ppamaverick1 dummy transitional pacakge from kvm to qemu-kvm
ii qemu-common 0.14.0~rc1+noroms-0ubuntu4~ppamaverick1 qemu common functionality (bios, documentation, etc)
ii qemu-kvm 0.14.0~rc1+noroms-0ubuntu4~ppamaverick1 Full virtualization on i386 and amd64 hardware

Thanks in advance!
Kei

[Vish Ishaya (vishvananda)]

So I'm getting a few other errors when trying to reproduce with your branch. I think I might have isolated the difference. It looks like you are using vlan mode and I am using flat dhcp.

There are some issues with using flatdhcp in your branch that have been fixed in trunk. Can you merge with trunk so I can test again?

Thanks,

Vish

[Kei Masumoto (masumotok)]

Hi!,
I tried "bzr merge lp:nova" and doing block migration again. block migration itself is ok, but I got below error. I'll let you know once I fix it and try again.

2011-07-27 17:11:10,773 ERROR nova [-] Exception during message handling
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/compute/manager.py", line 1414, in post_live_migration_at_destination
(nova): TRACE: block_migration)
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/virt/libvirt/connection.py", line 1643, in post_live_migration_at_destination
(nova): TRACE: xml = self.to_xml(instance_ref)
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/virt/libvirt/connection.py", line 1040, in to_xml
(nova): TRACE: block_device_mapping)
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/virt/libvirt/connection.py", line 984, in _prepare_xml_info
(nova): TRACE: nics.append(self.vif_driver.plug(instance, network, mapping))
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/virt/libvirt/vif.py", line 90, in plug
(nova): TRACE: return self._get_configurations(network, mapping)
(nova): TRACE: File "/opt/openstack/dev/block-migration-rev1173-merge/nova/virt/libvirt/vif.py", line 65, in _get_configurations
(nova): TRACE: 'dhcp_server': mapping['dhcp_server'],
(nova): TRACE: KeyError: 'dhcp_server'
(nova): TRACE:
2011-07-27 17:11:10,774 ERROR nova.rpc [-] Returning exception 'dhcp_server' to caller

Thanks,
Kei

[Kei Masumoto (masumotok)]

Hi!
I merged recent trunk, and confirm no error occurs. Could you please check it?

one additional note - the below is possible reason that block migration fails.
1) libvirt/qemu configuration
source/destionation host has to have exactly same libvirt.conf and qemu.conf.
For example, at source host, "user=root" and "group=root" but there is no such configuration at destination host. in this case, block migration fails.

2) nane resolution
source and destination host can resolve their name (ex. writing /etc/hosts ).
For example, if destination host refer DNS and dns entry is wrong, or DNS entry - hostname is different, block migration fails.
Apart from nova, if we go "virsh migrate --live --copy-storage-inc instance_id dest",
and dest is specified by ip address, such as "qemu+tcp://192.168.0.1/system", libvirt exchange their host name.

3) install dir
nova has to be installed exactly same dir at source and destination host.

4) Permission
libvirt can access to FLAGS.instance_dire ?
/etc/init.d/apparmor teardown may be help.

Thanks,
Kei

[Vish Ishaya (vishvananda)]

Still no luck with new code. Are you using vlanmode? I'm still getting migrate to uri failed

[Kei Masumoto (masumotok)]

Thanks for checking our branch. I should have said this, my colleagues and me already have tried FlatDHCPManager and could not reproduce. However, since I may have some mistakes, I'll send you how-to-doc that explains our environment. Could you please check it?

[Vish Ishaya (vishvananda)]

Kei came by and got this working. Turns out there was an issue with images smaller than one GB that he fixed.

review: Approve

[Kei Masumoto (masumotok)]

Thanks for review, Vish!

[madhavi (prasanna-desaraju)]

Hi kei,

Iam facing some issue while doing block migration.Can you plz help me out on this.

Instances files are being destroyed.

plz check the below log

ed': False} from (pid=15050) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-11-24 21:38:09,155 INFO nova.compute.manager [-] #################### in pre_live_migration
2011-11-24 21:38:09,252 INFO nova.compute.manager [-] instance-00000002 has no volume.
2011-11-24 21:38:09,253 INFO nova.compute.manager [-] @@@@@@@@@@@@@@@@@before driver.pre_live_migration instance_id 2
2011-11-24 21:38:09,253 INFO nova.compute.manager [-] @@@@@@@@@@@@@@@@@ after driver.pre_live_migration instance_id 2
2011-11-24 21:38:09,254 DEBUG nova.rpc [-] Making asynchronous call on network ... from (pid=15050) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:721
2011-11-24 21:38:09,254 DEBUG nova.rpc [-] MSG_ID is fba4149739ec484682ccece37cc0bcbb from (pid=15050) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:724
2011-11-24 21:38:09,527 INFO nova.compute.manager [-] @@@@@@@@@@@@@@@@@ after nstance nw_info 2 nwinfo [[{u'bridge': u'br100', u'multi_host': True, u'bridge_interface': u'eth0', u'vlan': 100, u'id': 1, u'injected': False, u'cidr': u'10.0.0.0/24', u'cidr_v6': None}, {u'should_create_bridge': True, u'dns': [], u'vif_uuid': u'93f0e47c-d3b5-4614-a760-34145e67317c', u'label': u'private', u'broadcast': u'10.0.0.255', u'ips': [{u'ip': u'10.0.0.5', u'netmask': u'255.255.255.0', u'enabled': u'1'}], u'mac': u'02:16:3e:00:70:4a', u'rxtx_cap': 0, u'should_create_vlan': False, u'dhcp_server': u'10.0.0.8', u'gateway': u'10.0.0.1'}]]
2011-11-24 21:38:09,528 INFO nova.virt.libvirt_conn [-] ######### endure_filtering_rules_for_instance
2011-11-24 21:38:09,528 INFO nova [-] called setup_basic_filtering in nwfilter
2011-11-24 21:38:09,529 INFO nova [-] ensuring static filters
2011-11-24 21:38:10,059 DEBUG nova.virt.libvirt.firewall [-] iptables firewall: Setup Basic Filtering from (pid=15050) setup_basic_filtering /usr/lib/python2.7/dist-packages/nova/virt/libvirt/firewall.py:525
2011-11-24 21:38:10,060 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=15050) inner /usr/lib/python2.7/dist-packages/nova/utils.py:721
2011-11-24 21:38:10,060 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=15050) inner /usr/lib/python2.7/dist-packages/nova/utils.py:726
2011-11-24 21:38:10,067 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=15050) inner /usr/lib/python2.7/dist-packages/nova/utils.py:721
2011-11-24 21:38:10,067 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=15050) inner /usr/lib/python2.7/dist-packages/nova/utils.py:726
2011-11-24 21:38:10,068 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=15050) execute /usr/lib/python2.7/dist-packages/nova/utils.py:169
2011-11-24 21:38:10,090 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=15050) execute /usr/lib/python2.7/dist-packa...

[Kei Masumoto (masumotok)]

I am expecting there must be some exception(stack trace). I recommend you to find it. In my memory, any exception is logged into console/logfile.
I don’t know which revision you are using, but unintentionally suppressing sometimes happens in trunk nova. Kei

[madhavi (prasanna-desaraju)]

Hi kei,

Thanks a lot for your reply.

This is our environment which we are using.

root@Compute104:/usr/share/pyshared/nova# dpkg -l | grep libvirt
ii libvirt-bin 0.8.8-1ubuntu6.5 the programs for the libvirt library
ii libvirt0 0.8.8-1ubuntu6.5 library for interfacing with different virtualization systems
ii python-libvirt 0.8.8-1ubuntu6.5 libvirt Python bindings

root@Compute104:/usr/share/pyshared/nova# uname -a
Linux Compute104 2.6.38-12-server #51-Ubuntu SMP Wed Sep 28 16:07:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

root@Compute104:/usr/share/pyshared/nova# dpkg -l | grep kvm
ii nova-compute-kvm 2012.1~e2~20111116.11495-0ubuntu0ppa1~natty1 OpenStack Compute - compute node (KVM)
ii qemu-kvm 0.14.0+noroms-0ubuntu4.4 Full virtualization on i386 and amd64 hardware

Apart from this on what other stuff do i need to concentrate ? I have followed all the changes you have mentioned before but facing the same issue still

when i checked the destination compute log

It says instance files deleted successfully.

Please help me out on this.

Thanks in advance

[madhavi (prasanna-desaraju)]

Actually we are working on a multinode setup with one controller node and two compute nodes pointing to same controller.The nova.conf files at all ends is as follows

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--use_deprecated_auth
--verbose
--flagfile=/etc/nova/nova-compute.conf
--sql_connection=mysql://novadbuser:novaDBsekret@10.233.52.102/nova
--network_manager=nova.network.manager.FlatDHCPManager
--auth_driver=nova.auth.dbdriver.DbDriver
--libvirt_type=qemu
--flat_network_bridge=br100
--flat_interface=eth0
--vlan_interface=eth0
--public_interface=eth0
--vncproxy_url=http://10.233.52.102:6080
--daemonize=1
--rabbit_host=10.233.52.102
--cc_host=10.233.52.102
--osapi_host=10.233.52.102
--ec2_host=10.233.52.102
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.233.52.102:9292
--use_syslog

[madhavi (prasanna-desaraju)]

hi kei,

Please tell me the configuration details needed for block migration.Please give me the details of

1. nova.conf at compute and controller ends
2. libvirt.conf
3. qemu.conf

[madhavi (prasanna-desaraju)]

hi vish,

Can you please provide me the configuration details for block migration.I have mentioned my configuration details above .iam not sure where iam going wrong.

Can you plz help me out on this?

Log shows some error in adding sec group for instance.i have got the same error which u got

nwfilter(nova-instance-instance-00000002-secgroup) for instance-00000002 is not found. from (pid=676) unfilter_instance /usr/lib/python2.7/dist-packages/nova/virt/libvirt/firewall.py:320
2011-11-23 19:45:00,695 INFO nova.virt.libvirt_conn [-] instance instance-00000002: deleting instance files /var/lib/nova/instances/instance-00000002

vish can you please help me on this?

[Vish Ishaya (vishvananda)]

That security group error always happens on delete, it isn't an issue. The migration is failing for some other reason. You should verify that you can connect to libvirt remotely from each host to the other.

On Nov 28, 2011, at 3:40 AM, madhavi wrote:

> hi vish,
>
> Can you please provide me the configuration details for block migration.I have mentioned my configuration details above .iam not sure where iam going wrong.
>
> Can you plz help me out on this?
>
> Log shows some error in adding sec group for instance.i have got the same error which u got
>
> nwfilter(nova-instance-instance-00000002-secgroup) for instance-00000002 is not found. from (pid=676) unfilter_instance /usr/lib/python2.7/dist-packages/nova/virt/libvirt/firewall.py:320
> 2011-11-23 19:45:00,695 INFO nova.virt.libvirt_conn [-] instance instance-00000002: deleting instance files /var/lib/nova/instances/instance-00000002
>
> vish can you please help me on this?
> --
> https://code.launchpad.net/~nttdata/nova/block-migration/+merge/64662
> You are reviewing the proposed merge of lp:~nttdata/nova/block-migration into lp:~hudson-openstack/nova/trunk.

[Kei Masumoto (masumotok)]

> Please tell me the configuration details
<http://docs.openstack.org/diablo/openstack-compute/admin/content/configuring-live-migrations.html>
Libvirt config is same as normal live migration. also, make sure "user=root" in qemu.conf.

[madhavi (prasanna-desaraju)]

root@Compute104:/usr/share/pyshared/nova# dpkg -l | grep libvirt
ii libvirt-bin 0.8.8-1ubuntu6.5 the programs for the libvirt library
ii libvirt0 0.8.8-1ubuntu6.5 library for interfacing with different virtualization systems
ii python-libvirt 0.8.8-1ubuntu6.5 libvirt Python bindings

root@Compute104:/usr/share/pyshared/nova# uname -a
Linux Compute104 2.6.38-12-server #51-Ubuntu SMP Wed Sep 28 16:07:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

root@Compute104:/usr/share/pyshared/nova# dpkg -l | grep kvm
ii nova-compute-kvm 2012.1~e2~20111116.11495-0ubuntu0ppa1~natty1 OpenStack Compute - compute node (KVM)
ii qemu-kvm 0.14.0+noroms-0ubuntu4.4 Full virtualization on i386 and amd64 hardware

[madhavi (prasanna-desaraju)]

Actually we are working on a multinode setup with one controller node and two compute nodes pointing to same controller.The nova.conf files at all ends is as follows

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--use_deprecated_auth
--verbose
--flagfile=/etc/nova/nova-compute.conf
--sql_connection=mysql://novadbuser:novaDBsekret@10.233.52.102/nova
--network_manager=nova.network.manager.FlatDHCPManager
--auth_driver=nova.auth.dbdriver.DbDriver
--libvirt_type=qemu
--flat_network_bridge=br100
--flat_interface=eth0
--vlan_interface=eth0
--public_interface=eth0
--vncproxy_url=http://10.233.52.102:6080
--daemonize=1
--rabbit_host=10.233.52.102
--cc_host=10.233.52.102
--osapi_host=10.233.52.102
--ec2_host=10.233.52.102
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.233.52.102:9292
--use_syslog

Do i need to add additional flags?

[madhavi (prasanna-desaraju)]

hi kei/vish,

These are the configuration details at my end.Please help me out.I need this to happen urgently

special thanks to both of you for responding immediately.Thanks a lot for the reply

libvirt.conf
=============

listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none

qemu.conf
========

vnc_listen = "0.0.0.0"
vnc_auto_unix_socket = 1
vnc_tls = 1
vnc_tls_x509_cert_dir = "/etc/pki/libvirt-vnc"
vnc_tls_x509_verify = 1
vnc_password = "XYZ12345"
vnc_sasl = 1
spice_listen = "0.0.0.0"
spice_tls = 1
spice_password = "XYZ12345"
spice_tls_x509_cert_dir = "/etc/pki/libvirt-spice"
security_driver = "selinux"
user = "root"

[Kei Masumoto (masumotok)]

> security_driver = "selinux"
can't say anything except error log messages, but one thing I found is above. I always comment it out.

[madhavi (prasanna-desaraju)]

hi kei

qemu.conf at host and destination nodes
===============================
user = root
group=root in qemu.conf

libvirt.conf at host and destination nodes
===============================
listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none

and nova.conf at source and destination nodes
=============================================

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
#--force_dhcp_release=True
--use_deprecated_auth
--verbose
--sql_connection=mysql://novadbuser:novaDBsekret@10.233.52.103/nova
#--network_manager=nova.network.manager.FlatDHCPManager
--network_manager=nova.network.manager.VlanManager
--vlan_interface=eth0
--auth_driver=nova.auth.dbdriver.DbDriver
#--flat_network_dhcp_start=10.0.0.2
--flat_network_bridge=br100
#--flat_injected=False
#--routing_source_ip=10.233.52.3
--flat_interface=eth0
--libv
--connection_type=libvirt
--public_interface=eth0
--daemonize=1
--rabbit_host=10.233.52.103
--cc_host=10.233.52.103
--s3_host=10.233.52.103
--osapi_host=10.233.52.103
--ec2_url=http://10.233.52.103:8773/services/Cloud
--ec2_host=10.233.52.103
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.233.52.103:9292
--vncproxy_url=http://10.233.52.3:6080
--use_syslog
--allow_admin_api=True
--fixed_range=10.0.0.0/24
--sql_min_pool_size=1

Iam working on kvm which is on bare metal but still the same issue.Can you plz help me a little on this.

Below is the compute-log at destination host side
================================================

11-12-01 23:56:22,781 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2011-12-01T18:26:41.555857', 'auth_token': None, 'msg_id': u'ff651bedf5db4821a5f2760a4eff6282', 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'18760f6d-e82f-404f-8fae-c42849112555', 'project_id': None, 'read_deleted': False} from (pid=4836) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-12-01 23:56:22,937 INFO nova.compute.manager [-] server-39 has no volume.
2011-12-01 23:56:22,938 DEBUG nova.rpc [-] Making asynchronous call on network ... from (pid=4836) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:721
2011-12-01 23:56:22,939 DEBUG nova.rpc [-] MSG_ID is 807151e24ed94860bc8e576a95333382 from (pid=4836) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:724
2011-12-01 23:56:23,324 INFO nova [-] called setup_basic_filtering in nwfilter
2011-12-01 23:56:23,324 INFO nova [-] ensuring static filters
2011-12-01 23:56:24,176 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=4836) inner /usr/lib/python2.7/dist-packages/nova/utils.py:672
2011-12-01 23:56:24,177 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=4836) inner /usr/lib/python2.7/dist-packages/nova/utils.py:677
2011-12-01 23:56:24,178 DEBUG nova.utils [-] Running cmd (subprocess): sudo ...

[Kei Masumoto (masumotok)]

Usually, libvirt related error is shown at source compute node log. I recommend you to check src/dest compute, scheduler and network. Destination compute log looks no problem.

[madhavi (prasanna-desaraju)]

hi kei,

Can you please tell me what mite be the reasons for the rollback happening at destination node?

Will block migration work if i using qemu? or do i need to use kvm on bare metal?

[madhavi (prasanna-desaraju)]

hi kei,

If you dont mind,Can you please give the necessary configuration details needed for block migration.I have been trying since a month and i still get the same error instances files deleted successfully as stated in above comments.Iam not understanding the root cause for this.
can you please share your configuration details and even the environment details.

At my end i have given a try on qemu and even tried with KVM on bare metal but the issue remains the same.

Plz plz help me out on this.I need this urgently.

my set up is as follows:
========================

1.)I have a single node setup in one node
2.)I have only compute in other node.The controller and remaining components are common for both the nodes.
3.)both are with kvm on bare metal.Both are physical boxes with underlying hypervisor as KVM and i installed openstack on top of it .

libvirt.conf at both nodes
=========================
listen_tls = 0
listen_tcp = 1
unix_sock_group = "libvirtd"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tcp = "none

qemu.conf at both nodes
========================
user = root
group=root in qemu.conf

nova.conf
==========

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--flagfile=/etc/nova/nova-compute.conf
--use_deprecated_auth
--verbose
--sql_connection=mysql://novadbuser:novaDBsekret@10.233.52.103/nova
--network_manager=nova.network.manager.FlatDHCPManager
--vlan_interface=eth0
--auth_driver=nova.auth.dbdriver.DbDriver
--flat_network_bridge=br100
--routing_source_ip=10.233.52.103
--flat_interface=eth0
--libvirt_type=kvm
--connection_type=libvirt
--public_interface=eth0
--daemonize=1
--rabbit_host=10.233.52.103
--cc_host=10.233.52.103
--s3_host=10.233.52.103
--osapi_host=10.233.52.103
--ec2_url=http://10.233.52.103:8773/services/Cloud
--ec2_host=10.233.52.103
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.233.52.103:9292
--vncproxy_url=http://10.233.52.103:6080
--use_syslog
--allow_admin_api=True
--fixed_range=10.0.0.0/24
--sql_min_pool_size=1

Source Log shows no error

Below is the compute-log at destination host side
================================================

11-12-01 23:56:22,781 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2011-12-01T18:26:41.555857', 'auth_token': None, 'msg_id': u'ff651bedf5db4821a5f2760a4eff6282', 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'18760f6d-e82f-404f-8fae-c42849112555', 'project_id': None, 'read_deleted': False} from (pid=4836) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646
2011-12-01 23:56:22,937 INFO nova.compute.manager [-] server-39 has no volume.
2011-12-01 23:56:22,938 DEBUG nova.rpc [-] Making asynchronous call on network ... from (pid=4836) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:721
2011-12-01 23:56:22,939 DEBUG nova.rpc [-] MSG_ID is 807151e24ed94860bc8e576a95333382 from (pid=4836) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:724
2011-12-01 23:56:23,324 INFO nova [-] cal...

[madhavi (prasanna-desaraju)]

hi kei,

The before log was on qemu.

The below log is on kvm on baremetal.Configurations all remain same.Please see the below log at destination compute.

    <features name="ss" />
    <features name="acpi" />
    <features name="ds" />
    <features name="vme" />
</cpu>

2011-12-09 11:24:44,321 DEBUG nova.rpc [-] skipping a message reply due to a value of None returned by the func: <bound method ComputeManager.compare_cpu of <nova.compute.manager.ComputeManager object at 0x237f950>> from (pid=2338) _process_data /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:656
2011-12-09 11:24:46,371 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_msg_id': u'5683b393e463469f8cce9a0d154c58fe', u'_context_read_deleted': False, u'_context_request_id': u'62c8b42e-399b-402e-9889-44ea5a9ddb07', u'args': {u'instance_id': 16639, u'block_migration': True, u'disk': u'[{"path": "/var/lib/nova/instances/instance-000040ff/disk", "local_gb": "4G", "type": "qcow2"}, {"path": "/var/lib/nova/instances/instance-000040ff/disk.local", "local_gb": "40G", "type": "qcow2"}]'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2011-12-09T11:24:41.886399', u'_context_user_id': None, u'method': u'pre_live_migration', u'_context_remote_address': None} from (pid=2338) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:621
2011-12-09 11:24:46,372 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2011-12-09T11:24:41.886399', 'auth_token': None, 'msg_id': u'5683b393e463469f8cce9a0d154c58fe', 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'62c8b42e-399b-402e-9889-44ea5a9ddb07', 'project_id': None, 'read_deleted': False} from (pid=2338) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:676
2011-12-09 11:24:46,559 INFO nova.compute.manager [-] server-16639 has no volume.
2011-12-09 11:24:46,560 DEBUG nova.rpc [-] Making asynchronous call on network ... from (pid=2338) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:751
2011-12-09 11:24:46,560 DEBUG nova.rpc [-] MSG_ID is 22f0c8be2d5047d98e1ec88ba201a8fe from (pid=2338) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:754
2011-12-09 11:24:46,784 DEBUG nova.virt.libvirt.vif [-] Ensuring vlan 2688 and bridge br2688 from (pid=2338) plug /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py:82
2011-12-09 11:24:46,785 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_vlan" for method "ensure_vlan"... from (pid=2338) inner /usr/lib/python2.7/dist-packages/nova/utils.py:719
2011-12-09 11:24:46,785 DEBUG nova.utils [-] Attempting to grab file lock "ensure_vlan" for method "ensure_vlan"... from (pid=2338) inner /usr/lib/python2.7/dist-packages/nova/utils.py:724
2011-12-09 11:24:46,785 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev vlan2688 from (pid=2338) execute /usr/lib/python2.7/dist-packages/nova/utils.py:165
2011-12-09 11:24:46,826 DEBUG nova.utils [-] Result was 255 from (pid=2338) execute /usr/lib/python2.7/dist-packages/nova/utils.py:180
2011-12-09 11:24:46,826 DEBU...

[Kei Masumoto (masumotok)]

> Can you please tell me what mite be the reasons for the rollback happening at destination node?
Because some errors occurred. "rollback" here also means cleanup method for pre-block-migration. nothing suspicious.
> Will block migration work if i using qemu? or do i need to use kvm on bare metal?
I recommend you to use kvm.

[madhavi (prasanna-desaraju)]

hi kei,

I tried using kvm and it says same error instance files destroyed successfully.

INFO nova.virt.libvirt_conn [-] Instance instance-000040ff destroyed successfully.

INFO nova.virt.libvirt_conn [-] instance instance-000040ff: deleting instance files /var/lib/nova/instances/instance-000040ff
Destination log shows these stuff but it doesn't showcase any errors.I have pasted the log above.

How can i trace the problem?

[madhavi (prasanna-desaraju)]

do we have any pre-requisites for doing block migration? iam made sure everything is fine but still the same issue.plz guide me a little on this

[madhavi (prasanna-desaraju)]

hi kei,

How do i check if the name resolution is working fine from one host to other.I mean how do i ensure if one host is libvirtd reachable to other?

=== modified file 'Authors'
--- Authors	2011-08-09 21:17:56 +0000
+++ Authors	2011-08-12 19:02:28 +0000
@@ -58,6 +58,7 @@
 Justin Santa Barbara <[e-mail]>
 Justin Shepherd <[e-mail]>
 Kei Masumoto <[e-mail]>
+masumoto<[e-mail]>
 Ken Pepple <[e-mail]>
 Kevin Bringard <[e-mail]>
 Kevin L. Mitchell <[e-mail]>

=== modified file 'bin/nova-manage'
--- bin/nova-manage	2011-08-09 15:41:55 +0000
+++ bin/nova-manage	2011-08-12 19:02:28 +0000
@@ -834,11 +834,13 @@
                 instance['availability_zone'],
                 instance['launch_index'])
 
-    @args('--ec2_id', dest='ec2_id', metavar='<ec2 id>', help='EC2 ID')
-    @args('--dest', dest='dest', metavar='<Destanation>',
-            help='destanation node')
-    def live_migration(self, ec2_id, dest):
-        """Migrates a running instance to a new machine."""
+    def _migration(self, ec2_id, dest, block_migration=False):
+        """Migrates a running instance to a new machine.
+         :param ec2_id: instance id which comes from euca-describe-instance.
+         :param dest: destination host name.
+         :param block_migration: if True, do block_migration.
+
+        """
 
         ctxt = context.get_admin_context()
         instance_id = ec2utils.ec2_id_to_id(ec2_id)
@@ -859,11 +861,28 @@
                  {"method": "live_migration",
                   "args": {"instance_id": instance_id,
                            "dest": dest,
-                           "topic": FLAGS.compute_topic}})
+                           "topic": FLAGS.compute_topic,
+                           "block_migration": block_migration}})
 
         print _('Migration of %s initiated.'
                'Check its progress using euca-describe-instances.') % ec2_id
 
+    @args('--ec2_id', dest='ec2_id', metavar='<ec2 id>', help='EC2 ID')
+    @args('--dest', dest='dest', metavar='<Destanation>',
+            help='destanation node')
+    def live_migration(self, ec2_id, dest):
+        """Migrates a running instance to a new machine."""
+
+        self._migration(ec2_id, dest)
+
+    @args('--ec2_id', dest='ec2_id', metavar='<ec2 id>', help='EC2 ID')
+    @args('--dest', dest='dest', metavar='<Destanation>',
+            help='destanation node')
+    def block_migration(self, ec2_id, dest):
+        """Migrates a running instance to a new machine with storage data."""
+
+        self._migration(ec2_id, dest, True)
+
 
 class ServiceCommands(object):
     """Enable and disable running services"""
@@ -937,9 +956,19 @@
             mem_u = result['resource']['memory_mb_used']
             hdd_u = result['resource']['local_gb_used']
 
+            cpu_sum = 0
+            mem_sum = 0
+            hdd_sum = 0
             print 'HOST\t\t\tPROJECT\t\tcpu\tmem(mb)\tdisk(gb)'
             print '%s(total)\t\t\t%s\t%s\t%s' % (host, cpu, mem, hdd)
-            print '%s(used)\t\t\t%s\t%s\t%s' % (host, cpu_u, mem_u, hdd_u)
+            print '%s(used_now)\t\t\t%s\t%s\t%s' % (host, cpu_u, mem_u, hdd_u)
+            for p_id, val in result['usage'].items():
+                cpu_sum += val['vcpus']
+                mem_sum += val['memory_mb']
+                hdd_sum += val['local_gb']
+            print '%s(used_max)\t\t\t%s\t%s\t%s' % (host, cpu_sum,
+                                                    mem_sum, hdd_sum)
+
             for p_id, val in result['usage'].items():
                 print '%s\t\t%s\t\t%s\t%s\t%s' % (host,
                                                   p_id,

=== modified file 'nova/compute/manager.py'
--- nova/compute/manager.py	2011-08-09 09:54:51 +0000
+++ nova/compute/manager.py	2011-08-12 19:02:28 +0000
@@ -1224,6 +1224,7 @@
     @exception.wrap_exception(notifier=notifier, publisher_id=publisher_id())
     def check_shared_storage_test_file(self, context, filename):
         """Confirms existence of the tmpfile under FLAGS.instances_path.
+           Cannot confirm tmpfile return False.
 
         :param context: security context
         :param filename: confirm existence of FLAGS.instances_path/thisfile
@@ -1231,7 +1232,9 @@
         """
         tmp_file = os.path.join(FLAGS.instances_path, filename)
         if not os.path.exists(tmp_file):
-            raise exception.FileNotFound(file_path=tmp_file)
+            return False
+        else:
+            return True
 
     @exception.wrap_exception(notifier=notifier, publisher_id=publisher_id())
     def cleanup_shared_storage_test_file(self, context, filename):
@@ -1254,11 +1257,13 @@
         """
         return self.driver.update_available_resource(context, self.host)
 
-    def pre_live_migration(self, context, instance_id, time=None):
+    def pre_live_migration(self, context, instance_id, time=None,
+                           block_migration=False, disk=None):
         """Preparations for live migration at dest host.
 
         :param context: security context
         :param instance_id: nova.db.sqlalchemy.models.Instance.Id
+        :param block_migration: if true, prepare for block migration
 
         """
         if not time:
@@ -1310,17 +1315,24 @@
         # onto destination host.
         self.driver.ensure_filtering_rules_for_instance(instance_ref)
 
-    def live_migration(self, context, instance_id, dest):
+        # Preparation for block migration
+        if block_migration:
+            self.driver.pre_block_migration(context,
+                                            instance_ref,
+                                            disk)
+
+    def live_migration(self, context, instance_id,
+                       dest, block_migration=False):
         """Executing live migration.
 
         :param context: security context
         :param instance_id: nova.db.sqlalchemy.models.Instance.Id
         :param dest: destination host
+        :param block_migration: if true, do block migration
 
         """
         # Get instance for error handling.
         instance_ref = self.db.instance_get(context, instance_id)
-        i_name = instance_ref.name
 
         try:
             # Checking volume node is working correctly when any volumes
@@ -1331,16 +1343,25 @@
                           {"method": "check_for_export",
                            "args": {'instance_id': instance_id}})
 
-            # Asking dest host to preparing live migration.
+            if block_migration:
+                disk = self.driver.get_instance_disk_info(context,
+                                                          instance_ref)
+            else:
+                disk = None
+
             rpc.call(context,
                      self.db.queue_get_for(context, FLAGS.compute_topic, dest),
                      {"method": "pre_live_migration",
-                      "args": {'instance_id': instance_id}})
+                      "args": {'instance_id': instance_id,
+                               'block_migration': block_migration,
+                               'disk': disk}})
 
         except Exception:
+            i_name = instance_ref.name
             msg = _("Pre live migration for %(i_name)s failed at %(dest)s")
             LOG.error(msg % locals())
-            self.recover_live_migration(context, instance_ref)
+            self.rollback_live_migration(context, instance_ref,
+                                         dest, block_migration)
             raise
 
         # Executing live migration
@@ -1348,9 +1369,11 @@
         # nothing must be recovered in this version.
         self.driver.live_migration(context, instance_ref, dest,
                                    self.post_live_migration,
-                                   self.recover_live_migration)
+                                   self.rollback_live_migration,
+                                   block_migration)
 
-    def post_live_migration(self, ctxt, instance_ref, dest):
+    def post_live_migration(self, ctxt, instance_ref,
+                            dest, block_migration=False):
         """Post operations for live migration.
 
         This method is called from live_migration
@@ -1359,6 +1382,7 @@
         :param ctxt: security context
         :param instance_id: nova.db.sqlalchemy.models.Instance.Id
         :param dest: destination host
+        :param block_migration: if true, do block migration
 
         """
 
@@ -1401,8 +1425,29 @@
                         "%(i_name)s cannot inherit floating "
                         "ip.\n%(e)s") % (locals()))
 
-        # Restore instance/volume state
-        self.recover_live_migration(ctxt, instance_ref, dest)
+        # Define domain at destination host, without doing it,
+        # pause/suspend/terminate do not work.
+        rpc.call(ctxt,
+                 self.db.queue_get_for(ctxt, FLAGS.compute_topic, dest),
+                     {"method": "post_live_migration_at_destination",
+                      "args": {'instance_id': instance_ref.id,
+                               'block_migration': block_migration}})
+
+        # Restore instance state
+        self.db.instance_update(ctxt,
+                                instance_ref['id'],
+                                {'state_description': 'running',
+                                 'state': power_state.RUNNING,
+                                 'host': dest})
+        # Restore volume state
+        for volume_ref in instance_ref['volumes']:
+            volume_id = volume_ref['id']
+            self.db.volume_update(ctxt, volume_id, {'status': 'in-use'})
+
+        # No instance booting at source host, but instance dir
+        # must be deleted for preparing next block migration
+        if block_migration:
+            self.driver.destroy(instance_ref, network_info)
 
         LOG.info(_('Migrating %(i_name)s to %(dest)s finished successfully.')
                  % locals())
@@ -1410,31 +1455,64 @@
                    "Domain not found: no domain with matching name.\" "
                    "This error can be safely ignored."))
 
-    def recover_live_migration(self, ctxt, instance_ref, host=None, dest=None):
+    def post_live_migration_at_destination(self, context,
+                                instance_id, block_migration=False):
+        """Post operations for live migration .
+
+        :param context: security context
+        :param instance_id: nova.db.sqlalchemy.models.Instance.Id
+        :param block_migration: block_migration
+
+        """
+        instance_ref = self.db.instance_get(context, instance_id)
+        LOG.info(_('Post operation of migraton started for %s .')
+                 % instance_ref.name)
+        network_info = self._get_instance_nw_info(context, instance_ref)
+        self.driver.post_live_migration_at_destination(context,
+                                                       instance_ref,
+                                                       network_info,
+                                                       block_migration)
+
+    def rollback_live_migration(self, context, instance_ref,
+                                dest, block_migration):
         """Recovers Instance/volume state from migrating -> running.
 
-        :param ctxt: security context
+        :param context: security context
         :param instance_id: nova.db.sqlalchemy.models.Instance.Id
-        :param host: DB column value is updated by this hostname.
-                     If none, the host instance currently running is selected.
-
+        :param dest:
+            This method is called from live migration src host.
+            This param specifies destination host.
         """
-        if not host:
-            host = instance_ref['host']
-
-        self.db.instance_update(ctxt,
+        host = instance_ref['host']
+        self.db.instance_update(context,
                                 instance_ref['id'],
                                 {'state_description': 'running',
                                  'state': power_state.RUNNING,
                                  'host': host})
 
-        if dest:
-            volume_api = volume.API()
         for volume_ref in instance_ref['volumes']:
             volume_id = volume_ref['id']
-            self.db.volume_update(ctxt, volume_id, {'status': 'in-use'})
-            if dest:
-                volume_api.remove_from_compute(ctxt, volume_id, dest)
+            self.db.volume_update(context, volume_id, {'status': 'in-use'})
+            volume.API().remove_from_compute(context, volume_id, dest)
+
+        # Block migration needs empty image at destination host
+        # before migration starts, so if any failure occurs,
+        # any empty images has to be deleted.
+        if block_migration:
+            rpc.cast(context,
+                     self.db.queue_get_for(context, FLAGS.compute_topic, dest),
+                     {"method": "rollback_live_migration_at_destination",
+                      "args": {'instance_id': instance_ref['id']}})
+
+    def rollback_live_migration_at_destination(self, context, instance_id):
+        """ Cleaning up image directory that is created pre_live_migration.
+
+        :param context: security context
+        :param instance_id: nova.db.sqlalchemy.models.Instance.Id
+        """
+        instances_ref = self.db.instance_get(context, instance_id)
+        network_info = self._get_instance_nw_info(context, instances_ref)
+        self.driver.destroy(instances_ref, network_info)
 
     def periodic_tasks(self, context=None):
         """Tasks to be run at a periodic interval."""

=== modified file 'nova/db/api.py'
--- nova/db/api.py	2011-08-09 23:59:51 +0000
+++ nova/db/api.py	2011-08-12 19:02:28 +0000
@@ -570,27 +570,6 @@
                                             security_group_id)
 
 
-def instance_get_vcpu_sum_by_host_and_project(context, hostname, proj_id):
-    """Get instances.vcpus by host and project."""
-    return IMPL.instance_get_vcpu_sum_by_host_and_project(context,
-                                                          hostname,
-                                                          proj_id)
-
-
-def instance_get_memory_sum_by_host_and_project(context, hostname, proj_id):
-    """Get amount of memory by host and project."""
-    return IMPL.instance_get_memory_sum_by_host_and_project(context,
-                                                            hostname,
-                                                            proj_id)
-
-
-def instance_get_disk_sum_by_host_and_project(context, hostname, proj_id):
-    """Get total amount of disk by host and project."""
-    return IMPL.instance_get_disk_sum_by_host_and_project(context,
-                                                          hostname,
-                                                          proj_id)
-
-
 def instance_action_create(context, values):
     """Create an instance action from the values dictionary."""
     return IMPL.instance_action_create(context, values)

=== modified file 'nova/db/sqlalchemy/api.py'
--- nova/db/sqlalchemy/api.py	2011-08-12 05:02:21 +0000
+++ nova/db/sqlalchemy/api.py	2011-08-12 19:02:28 +0000
@@ -1483,45 +1483,6 @@
 
 
 @require_context
-def instance_get_vcpu_sum_by_host_and_project(context, hostname, proj_id):
-    session = get_session()
-    result = session.query(models.Instance).\
-                      filter_by(host=hostname).\
-                      filter_by(project_id=proj_id).\
-                      filter_by(deleted=False).\
-                      value(func.sum(models.Instance.vcpus))
-    if not result:
-        return 0
-    return result
-
-
-@require_context
-def instance_get_memory_sum_by_host_and_project(context, hostname, proj_id):
-    session = get_session()
-    result = session.query(models.Instance).\
-                      filter_by(host=hostname).\
-                      filter_by(project_id=proj_id).\
-                      filter_by(deleted=False).\
-                      value(func.sum(models.Instance.memory_mb))
-    if not result:
-        return 0
-    return result
-
-
-@require_context
-def instance_get_disk_sum_by_host_and_project(context, hostname, proj_id):
-    session = get_session()
-    result = session.query(models.Instance).\
-                      filter_by(host=hostname).\
-                      filter_by(project_id=proj_id).\
-                      filter_by(deleted=False).\
-                      value(func.sum(models.Instance.local_gb))
-    if not result:
-        return 0
-    return result
-
-
-@require_context
 def instance_action_create(context, values):
     """Create an instance action from the values dictionary."""
     action_ref = models.InstanceActions()

=== modified file 'nova/db/sqlalchemy/models.py'
--- nova/db/sqlalchemy/models.py	2011-08-04 21:58:42 +0000
+++ nova/db/sqlalchemy/models.py	2011-08-12 19:02:28 +0000
@@ -127,14 +127,14 @@
                                 'ComputeNode.service_id == Service.id,'
                                 'ComputeNode.deleted == False)')
 
-    vcpus = Column(Integer, nullable=True)
-    memory_mb = Column(Integer, nullable=True)
-    local_gb = Column(Integer, nullable=True)
-    vcpus_used = Column(Integer, nullable=True)
-    memory_mb_used = Column(Integer, nullable=True)
-    local_gb_used = Column(Integer, nullable=True)
-    hypervisor_type = Column(Text, nullable=True)
-    hypervisor_version = Column(Integer, nullable=True)
+    vcpus = Column(Integer)
+    memory_mb = Column(Integer)
+    local_gb = Column(Integer)
+    vcpus_used = Column(Integer)
+    memory_mb_used = Column(Integer)
+    local_gb_used = Column(Integer)
+    hypervisor_type = Column(Text)
+    hypervisor_version = Column(Integer)
 
     # Note(masumotok): Expected Strings example:
     #

=== modified file 'nova/exception.py'
--- nova/exception.py	2011-08-09 23:59:51 +0000
+++ nova/exception.py	2011-08-12 19:02:28 +0000
@@ -273,6 +273,11 @@
                 "has been provided.")
 
 
+class DestinationDiskExists(Invalid):
+    message = _("The supplied disk path (%(path)s) already exists, "
+                "it is expected not to exist.")
+
+
 class InvalidDevicePath(Invalid):
     message = _("The supplied device path (%(path)s) is invalid.")
 
@@ -699,6 +704,10 @@
     message = _("Instance %(name)s already exists.")
 
 
+class InvalidSharedStorage(NovaException):
+    message = _("%(path)s is on shared storage: %(reason)s")
+
+
 class MigrationError(NovaException):
     message = _("Migration error") + ": %(reason)s"
 

=== modified file 'nova/scheduler/driver.py'
--- nova/scheduler/driver.py	2011-07-01 14:53:20 +0000
+++ nova/scheduler/driver.py	2011-08-12 19:02:28 +0000
@@ -30,6 +30,7 @@
 from nova import rpc
 from nova import utils
 from nova.compute import power_state
+from nova.api.ec2 import ec2utils
 
 
 FLAGS = flags.FLAGS
@@ -78,7 +79,8 @@
         """Must override at least this method for scheduler to work."""
         raise NotImplementedError(_("Must implement a fallback schedule"))
 
-    def schedule_live_migration(self, context, instance_id, dest):
+    def schedule_live_migration(self, context, instance_id, dest,
+                                block_migration=False):
         """Live migration scheduling method.
 
         :param context:
@@ -87,9 +89,7 @@
         :return:
             The host where instance is running currently.
             Then scheduler send request that host.
-
         """
-
         # Whether instance exists and is running.
         instance_ref = db.instance_get(context, instance_id)
 
@@ -97,10 +97,11 @@
         self._live_migration_src_check(context, instance_ref)
 
         # Checking destination host.
-        self._live_migration_dest_check(context, instance_ref, dest)
-
+        self._live_migration_dest_check(context, instance_ref,
+                                        dest, block_migration)
         # Common checking.
-        self._live_migration_common_check(context, instance_ref, dest)
+        self._live_migration_common_check(context, instance_ref,
+                                          dest, block_migration)
 
         # Changing instance_state.
         db.instance_set_state(context,
@@ -130,7 +131,8 @@
         # Checking instance is running.
         if (power_state.RUNNING != instance_ref['state'] or \
            'running' != instance_ref['state_description']):
-            raise exception.InstanceNotRunning(instance_id=instance_ref['id'])
+            instance_id = ec2utils.id_to_ec2_id(instance_ref['id'])
+            raise exception.InstanceNotRunning(instance_id=instance_id)
 
         # Checing volume node is running when any volumes are mounted
         # to the instance.
@@ -147,7 +149,8 @@
         if not self.service_is_up(services[0]):
             raise exception.ComputeServiceUnavailable(host=src)
 
-    def _live_migration_dest_check(self, context, instance_ref, dest):
+    def _live_migration_dest_check(self, context, instance_ref, dest,
+                                   block_migration):
         """Live migration check routine (for destination host).
 
         :param context: security context
@@ -168,16 +171,18 @@
         # and dest is not same.
         src = instance_ref['host']
         if dest == src:
-            raise exception.UnableToMigrateToSelf(
-                    instance_id=instance_ref['id'],
-                    host=dest)
+            instance_id = ec2utils.id_to_ec2_id(instance_ref['id'])
+            raise exception.UnableToMigrateToSelf(instance_id=instance_id,
+                                                  host=dest)
 
         # Checking dst host still has enough capacities.
         self.assert_compute_node_has_enough_resources(context,
                                                       instance_ref,
-                                                      dest)
+                                                      dest,
+                                                      block_migration)
 
-    def _live_migration_common_check(self, context, instance_ref, dest):
+    def _live_migration_common_check(self, context, instance_ref, dest,
+                                     block_migration):
         """Live migration common check routine.
 
         Below checkings are followed by
@@ -186,11 +191,26 @@
         :param context: security context
         :param instance_ref: nova.db.sqlalchemy.models.Instance object
         :param dest: destination host
+        :param block_migration if True, check for block_migration.
 
         """
 
         # Checking shared storage connectivity
-        self.mounted_on_same_shared_storage(context, instance_ref, dest)
+        # if block migration, instances_paths should not be on shared storage.
+        try:
+            self.mounted_on_same_shared_storage(context, instance_ref, dest)
+            if block_migration:
+                reason = _("Block migration can not be used "
+                           "with shared storage.")
+                raise exception.InvalidSharedStorage(reason=reason, path=dest)
+        except exception.FileNotFound:
+            if not block_migration:
+                src = instance_ref['host']
+                ipath = FLAGS.instances_path
+                logging.error(_("Cannot confirm tmpfile at %(ipath)s is on "
+                                "same shared storage between %(src)s "
+                                "and %(dest)s.") % locals())
+                raise
 
         # Checking dest exists.
         dservice_refs = db.service_get_all_compute_by_host(context, dest)
@@ -229,37 +249,96 @@
                                 "original host %(src)s.") % locals())
             raise
 
-    def assert_compute_node_has_enough_resources(self, context,
-                                                 instance_ref, dest):
+    def assert_compute_node_has_enough_resources(self, context, instance_ref,
+                                                 dest, block_migration):
+
         """Checks if destination host has enough resource for live migration.
 
-        Currently, only memory checking has been done.
-        If storage migration(block migration, meaning live-migration
-        without any shared storage) will be available, local storage
-        checking is also necessary.
-
-        :param context: security context
-        :param instance_ref: nova.db.sqlalchemy.models.Instance object
-        :param dest: destination host
-
-        """
-
-        # Getting instance information
-        hostname = instance_ref['hostname']
-
-        # Getting host information
-        service_refs = db.service_get_all_compute_by_host(context, dest)
-        compute_node_ref = service_refs[0]['compute_node'][0]
-
-        mem_total = int(compute_node_ref['memory_mb'])
-        mem_used = int(compute_node_ref['memory_mb_used'])
-        mem_avail = mem_total - mem_used
+        :param context: security context
+        :param instance_ref: nova.db.sqlalchemy.models.Instance object
+        :param dest: destination host
+        :param block_migration: if True, disk checking has been done
+
+        """
+        self.assert_compute_node_has_enough_memory(context, instance_ref, dest)
+        if not block_migration:
+            return
+        self.assert_compute_node_has_enough_disk(context, instance_ref, dest)
+
+    def assert_compute_node_has_enough_memory(self, context,
+                                              instance_ref, dest):
+        """Checks if destination host has enough memory for live migration.
+
+
+        :param context: security context
+        :param instance_ref: nova.db.sqlalchemy.models.Instance object
+        :param dest: destination host
+
+        """
+
+        # Getting total available memory and disk of host
+        avail = self._get_compute_info(context, dest, 'memory_mb')
+
+        # Getting total used memory and disk of host
+        # It should be sum of memories that are assigned as max value,
+        # because overcommiting is risky.
+        used = 0
+        instance_refs = db.instance_get_all_by_host(context, dest)
+        used_list = [i['memory_mb'] for i in instance_refs]
+        if used_list:
+            used = reduce(lambda x, y: x + y, used_list)
+
         mem_inst = instance_ref['memory_mb']
-        if mem_avail <= mem_inst:
-            reason = _("Unable to migrate %(hostname)s to destination: "
-                       "%(dest)s (host:%(mem_avail)s <= instance:"
-                       "%(mem_inst)s)")
-            raise exception.MigrationError(reason=reason % locals())
+        avail = avail - used
+        if avail <= mem_inst:
+            instance_id = ec2utils.id_to_ec2_id(instance_ref['id'])
+            reason = _("Unable to migrate %(instance_id)s to %(dest)s: "
+                       "Lack of disk(host:%(avail)s <= instance:%(mem_inst)s)")
+            raise exception.MigrationError(reason=reason % locals())
+
+    def assert_compute_node_has_enough_disk(self, context,
+                                            instance_ref, dest):
+        """Checks if destination host has enough disk for block migration.
+
+        :param context: security context
+        :param instance_ref: nova.db.sqlalchemy.models.Instance object
+        :param dest: destination host
+
+        """
+
+        # Getting total available memory and disk of host
+        avail = self._get_compute_info(context, dest, 'local_gb')
+
+        # Getting total used memory and disk of host
+        # It should be sum of disks that are assigned as max value
+        # because overcommiting is risky.
+        used = 0
+        instance_refs = db.instance_get_all_by_host(context, dest)
+        used_list = [i['local_gb'] for i in instance_refs]
+        if used_list:
+            used = reduce(lambda x, y: x + y, used_list)
+
+        disk_inst = instance_ref['local_gb']
+        avail = avail - used
+        if avail <= disk_inst:
+            instance_id = ec2utils.id_to_ec2_id(instance_ref['id'])
+            reason = _("Unable to migrate %(instance_id)s to %(dest)s: "
+                       "Lack of disk(host:%(avail)s "
+                       "<= instance:%(disk_inst)s)")
+            raise exception.MigrationError(reason=reason % locals())
+
+    def _get_compute_info(self, context, host, key):
+        """get compute node's infomation specified by key
+
+        :param context: security context
+        :param host: hostname(must be compute node)
+        :param key: column name of compute_nodes
+        :return: value specified by key
+
+        """
+        compute_node_ref = db.service_get_all_compute_by_host(context, host)
+        compute_node_ref = compute_node_ref[0]['compute_node'][0]
+        return compute_node_ref[key]
 
     def mounted_on_same_shared_storage(self, context, instance_ref, dest):
         """Check if the src and dest host mount same shared storage.
@@ -283,15 +362,13 @@
                                 {"method": 'create_shared_storage_test_file'})
 
             # make sure existence at src host.
-            rpc.call(context, src_t,
-                     {"method": 'check_shared_storage_test_file',
-                      "args": {'filename': filename}})
+            ret = rpc.call(context, src_t,
+                          {"method": 'check_shared_storage_test_file',
+                           "args": {'filename': filename}})
+            if not ret:
+                raise exception.FileNotFound(file_path=filename)
 
-        except rpc.RemoteError:
-            ipath = FLAGS.instances_path
-            logging.error(_("Cannot confirm tmpfile at %(ipath)s is on "
-                            "same shared storage between %(src)s "
-                            "and %(dest)s.") % locals())
+        except exception.FileNotFound:
             raise
 
         finally:

=== modified file 'nova/scheduler/manager.py'
--- nova/scheduler/manager.py	2011-08-04 00:26:37 +0000
+++ nova/scheduler/manager.py	2011-08-12 19:02:28 +0000
@@ -113,7 +113,7 @@
     # NOTE (masumotok) : This method should be moved to nova.api.ec2.admin.
     #                    Based on bexar design summit discussion,
     #                    just put this here for bexar release.
-    def show_host_resources(self, context, host, *args):
+    def show_host_resources(self, context, host):
         """Shows the physical/usage resource given by hosts.
 
         :param context: security context
@@ -121,43 +121,45 @@
         :returns:
             example format is below.
             {'resource':D, 'usage':{proj_id1:D, proj_id2:D}}
-            D: {'vcpus':3, 'memory_mb':2048, 'local_gb':2048}
+            D: {'vcpus': 3, 'memory_mb': 2048, 'local_gb': 2048,
+                'vcpus_used': 12, 'memory_mb_used': 10240,
+                'local_gb_used': 64}
 
         """
 
+        # Getting compute node info and related instances info
         compute_ref = db.service_get_all_compute_by_host(context, host)
         compute_ref = compute_ref[0]
-
-        # Getting physical resource information
-        compute_node_ref = compute_ref['compute_node'][0]
-        resource = {'vcpus': compute_node_ref['vcpus'],
-                    'memory_mb': compute_node_ref['memory_mb'],
-                    'local_gb': compute_node_ref['local_gb'],
-                    'vcpus_used': compute_node_ref['vcpus_used'],
-                    'memory_mb_used': compute_node_ref['memory_mb_used'],
-                    'local_gb_used': compute_node_ref['local_gb_used']}
-
-        # Getting usage resource information
-        usage = {}
         instance_refs = db.instance_get_all_by_host(context,
                                                     compute_ref['host'])
+
+        # Getting total available/used resource
+        compute_ref = compute_ref['compute_node'][0]
+        resource = {'vcpus': compute_ref['vcpus'],
+                    'memory_mb': compute_ref['memory_mb'],
+                    'local_gb': compute_ref['local_gb'],
+                    'vcpus_used': compute_ref['vcpus_used'],
+                    'memory_mb_used': compute_ref['memory_mb_used'],
+                    'local_gb_used': compute_ref['local_gb_used']}
+        usage = dict()
         if not instance_refs:
             return {'resource': resource, 'usage': usage}
 
+        # Getting usage resource per project
         project_ids = [i['project_id'] for i in instance_refs]
         project_ids = list(set(project_ids))
         for project_id in project_ids:
-            vcpus = db.instance_get_vcpu_sum_by_host_and_project(context,
-                                                                 host,
-                                                                 project_id)
-            mem = db.instance_get_memory_sum_by_host_and_project(context,
-                                                                 host,
-                                                                 project_id)
-            hdd = db.instance_get_disk_sum_by_host_and_project(context,
-                                                               host,
-                                                               project_id)
-            usage[project_id] = {'vcpus': int(vcpus),
-                                 'memory_mb': int(mem),
-                                 'local_gb': int(hdd)}
+            vcpus = [i['vcpus'] for i in instance_refs \
+                if i['project_id'] == project_id]
+
+            mem = [i['memory_mb']  for i in instance_refs \
+                if i['project_id'] == project_id]
+
+            disk = [i['local_gb']  for i in instance_refs \
+                if i['project_id'] == project_id]
+
+            usage[project_id] = {'vcpus': reduce(lambda x, y: x + y, vcpus),
+                                 'memory_mb': reduce(lambda x, y: x + y, mem),
+                                 'local_gb': reduce(lambda x, y: x + y, disk)}
 
         return {'resource': resource, 'usage': usage}

=== modified file 'nova/tests/api/openstack/contrib/test_keypairs.py'
=== modified file 'nova/tests/api/openstack/test_extensions.py'
--- nova/tests/api/openstack/test_extensions.py	2011-08-10 21:16:13 +0000
+++ nova/tests/api/openstack/test_extensions.py	2011-08-12 19:02:28 +0000
@@ -127,9 +127,7 @@
                 "updated": "2011-01-22T13:25:27-06:00",
                 "description": "The Fox In Socks Extension",
                 "alias": "FOXNSOX",
-                "links": [],
-            },
-        )
+                "links": []})
 
     def test_list_extensions_xml(self):
         app = openstack.APIRouterV11()
@@ -336,27 +334,18 @@
 
     def test_serialize_extenstion(self):
         serializer = extensions.ExtensionsXMLSerializer()
-        data = {
-            'extension': {
-                'name': 'ext1',
-                'namespace': 'http://docs.rack.com/servers/api/ext/pie/v1.0',
-                'alias': 'RS-PIE',
-                'updated': '2011-01-22T13:25:27-06:00',
-                'description': 'Adds the capability to share an image.',
-                'links': [
-                    {
-                        'rel': 'describedby',
-                        'type': 'application/pdf',
-                        'href': 'http://docs.rack.com/servers/api/ext/cs.pdf',
-                    },
-                    {
-                        'rel': 'describedby',
-                        'type': 'application/vnd.sun.wadl+xml',
-                        'href': 'http://docs.rack.com/servers/api/ext/cs.wadl',
-                    },
-                ],
-            },
-        }
+        data = {'extension': {
+          'name': 'ext1',
+          'namespace': 'http://docs.rack.com/servers/api/ext/pie/v1.0',
+          'alias': 'RS-PIE',
+          'updated': '2011-01-22T13:25:27-06:00',
+          'description': 'Adds the capability to share an image.',
+          'links': [{'rel': 'describedby',
+                     'type': 'application/pdf',
+                     'href': 'http://docs.rack.com/servers/api/ext/cs.pdf'},
+                    {'rel': 'describedby',
+                     'type': 'application/vnd.sun.wadl+xml',
+                     'href': 'http://docs.rack.com/servers/api/ext/cs.wadl'}]}}
 
         xml = serializer.serialize(data, 'show')
         print xml
@@ -378,48 +367,30 @@
 
     def test_serialize_extensions(self):
         serializer = extensions.ExtensionsXMLSerializer()
-        data = {
-            "extensions": [
-                {
-                    "name": "Public Image Extension",
-                    "namespace": "http://foo.com/api/ext/pie/v1.0",
-                    "alias": "RS-PIE",
-                    "updated": "2011-01-22T13:25:27-06:00",
-                    "description": "Adds the capability to share an image.",
-                    "links": [
-                        {
-                            "rel": "describedby",
-                            "type": "application/pdf",
-                            "href": "http://foo.com/api/ext/cs-pie.pdf",
-                        },
-                        {
-                            "rel": "describedby",
-                            "type": "application/vnd.sun.wadl+xml",
-                            "href": "http://foo.com/api/ext/cs-pie.wadl",
-                        },
-                    ],
-                },
-                {
-                    "name": "Cloud Block Storage",
-                    "namespace": "http://foo.com/api/ext/cbs/v1.0",
-                    "alias": "RS-CBS",
-                    "updated": "2011-01-12T11:22:33-06:00",
-                    "description": "Allows mounting cloud block storage.",
-                    "links": [
-                        {
-                            "rel": "describedby",
-                            "type": "application/pdf",
-                            "href": "http://foo.com/api/ext/cs-cbs.pdf",
-                        },
-                        {
-                            "rel": "describedby",
-                            "type": "application/vnd.sun.wadl+xml",
-                            "href": "http://foo.com/api/ext/cs-cbs.wadl",
-                        },
-                    ],
-                },
-            ],
-        }
+        data = {"extensions": [{
+                "name": "Public Image Extension",
+                "namespace": "http://foo.com/api/ext/pie/v1.0",
+                "alias": "RS-PIE",
+                "updated": "2011-01-22T13:25:27-06:00",
+                "description": "Adds the capability to share an image.",
+                "links": [{"rel": "describedby",
+                            "type": "application/pdf",
+                            "type": "application/vnd.sun.wadl+xml",
+                            "href": "http://foo.com/api/ext/cs-pie.pdf"},
+                           {"rel": "describedby",
+                            "type": "application/vnd.sun.wadl+xml",
+                            "href": "http://foo.com/api/ext/cs-pie.wadl"}]},
+                {"name": "Cloud Block Storage",
+                 "namespace": "http://foo.com/api/ext/cbs/v1.0",
+                 "alias": "RS-CBS",
+                 "updated": "2011-01-12T11:22:33-06:00",
+                 "description": "Allows mounting cloud block storage.",
+                 "links": [{"rel": "describedby",
+                             "type": "application/pdf",
+                             "href": "http://foo.com/api/ext/cs-cbs.pdf"},
+                            {"rel": "describedby",
+                             "type": "application/vnd.sun.wadl+xml",
+                             "href": "http://foo.com/api/ext/cs-cbs.wadl"}]}]}
 
         xml = serializer.serialize(data, 'index')
         print xml

=== modified file 'nova/tests/api/openstack/test_limits.py'
--- nova/tests/api/openstack/test_limits.py	2011-08-04 00:26:37 +0000
+++ nova/tests/api/openstack/test_limits.py	2011-08-12 19:02:28 +0000
@@ -915,86 +915,56 @@
 
     def setUp(self):
         self.view_builder = views.limits.ViewBuilderV11()
-        self.rate_limits = [
-            {
-                "URI": "*",
-                "regex": ".*",
-                "value": 10,
-                "verb": "POST",
-                "remaining": 2,
-                "unit": "MINUTE",
-                "resetTime": 1311272226,
-            },
-            {
-                "URI": "*/servers",
-                "regex": "^/servers",
-                "value": 50,
-                "verb": "POST",
-                "remaining": 10,
-                "unit": "DAY",
-                "resetTime": 1311272226,
-            },
-        ]
-        self.absolute_limits = {
-            "metadata_items": 1,
-            "injected_files": 5,
-            "injected_file_content_bytes": 5,
-        }
+        self.rate_limits = [{"URI": "*",
+                             "regex": ".*",
+                             "value": 10,
+                             "verb": "POST",
+                             "remaining": 2,
+                             "unit": "MINUTE",
+                             "resetTime": 1311272226},
+                            {"URI": "*/servers",
+                             "regex": "^/servers",
+                             "value": 50,
+                             "verb": "POST",
+                             "remaining": 10,
+                             "unit": "DAY",
+                             "resetTime": 1311272226}]
+        self.absolute_limits = {"metadata_items": 1,
+                                "injected_files": 5,
+                                "injected_file_content_bytes": 5}
 
     def tearDown(self):
         pass
 
     def test_build_limits(self):
-        expected_limits = {
-            "limits": {
-                "rate": [
-                    {
-                        "uri": "*",
-                        "regex": ".*",
-                        "limit": [
-                            {
-                                "value": 10,
-                                "verb": "POST",
-                                "remaining": 2,
-                                "unit": "MINUTE",
-                                "next-available": "2011-07-21T18:17:06Z",
-                            },
-                        ]
-                    },
-                    {
-                        "uri": "*/servers",
-                        "regex": "^/servers",
-                        "limit": [
-                            {
-                                "value": 50,
-                                "verb": "POST",
-                                "remaining": 10,
-                                "unit": "DAY",
-                                "next-available": "2011-07-21T18:17:06Z",
-                            },
-                        ]
-                    },
-                ],
-                "absolute": {
-                    "maxServerMeta": 1,
-                    "maxImageMeta": 1,
-                    "maxPersonality": 5,
-                    "maxPersonalitySize": 5
-                }
-            }
-        }
+        expected_limits = {"limits": {
+                "rate": [{
+                      "uri": "*",
+                      "regex": ".*",
+                      "limit": [{"value": 10,
+                                 "verb": "POST",
+                                 "remaining": 2,
+                                 "unit": "MINUTE",
+                                 "next-available": "2011-07-21T18:17:06Z"}]},
+                   {"uri": "*/servers",
+                    "regex": "^/servers",
+                    "limit": [{"value": 50,
+                               "verb": "POST",
+                               "remaining": 10,
+                               "unit": "DAY",
+                               "next-available": "2011-07-21T18:17:06Z"}]}],
+                "absolute": {"maxServerMeta": 1,
+                             "maxImageMeta": 1,
+                             "maxPersonality": 5,
+                             "maxPersonalitySize": 5}}}
 
         output = self.view_builder.build(self.rate_limits,
                                          self.absolute_limits)
         self.assertDictMatch(output, expected_limits)
 
     def test_build_limits_empty_limits(self):
-        expected_limits = {
-            "limits": {
-                "rate": [],
-                "absolute": {},
-            }
-        }
+        expected_limits = {"limits": {"rate": [],
+                           "absolute": {}}}
 
         abs_limits = {}
         rate_limits = []
@@ -1012,45 +982,28 @@
 
     def test_index(self):
         serializer = limits.LimitsXMLSerializer()
-
-        fixture = {
-            "limits": {
-                "rate": [
-                    {
-                        "uri": "*",
-                        "regex": ".*",
-                        "limit": [
-                            {
-                                "value": 10,
-                                "verb": "POST",
-                                "remaining": 2,
-                                "unit": "MINUTE",
-                                "next-available": "2011-12-15T22:42:45Z",
-                            },
-                        ]
-                    },
-                    {
-                        "uri": "*/servers",
-                        "regex": "^/servers",
-                        "limit": [
-                            {
-                                "value": 50,
-                                "verb": "POST",
-                                "remaining": 10,
-                                "unit": "DAY",
-                                "next-available": "2011-12-15T22:42:45Z"
-                            },
-                        ]
-                    },
-                ],
-                "absolute": {
-                    "maxServerMeta": 1,
-                    "maxImageMeta": 1,
-                    "maxPersonality": 5,
-                    "maxPersonalitySize": 10240
-                }
-            }
-        }
+        fixture = {"limits": {
+                   "rate": [{
+                         "uri": "*",
+                         "regex": ".*",
+                         "limit": [{
+                              "value": 10,
+                              "verb": "POST",
+                              "remaining": 2,
+                              "unit": "MINUTE",
+                              "next-available": "2011-12-15T22:42:45Z"}]},
+                          {"uri": "*/servers",
+                           "regex": "^/servers",
+                           "limit": [{
+                              "value": 50,
+                              "verb": "POST",
+                              "remaining": 10,
+                              "unit": "DAY",
+                              "next-available": "2011-12-15T22:42:45Z"}]}],
+                    "absolute": {"maxServerMeta": 1,
+                                 "maxImageMeta": 1,
+                                 "maxPersonality": 5,
+                                 "maxPersonalitySize": 10240}}}
 
         output = serializer.serialize(fixture, 'index')
         actual = minidom.parseString(output.replace("  ", ""))
@@ -1083,12 +1036,9 @@
     def test_index_no_limits(self):
         serializer = limits.LimitsXMLSerializer()
 
-        fixture = {
-            "limits": {
-                "rate": [],
-                "absolute": {},
-            }
-        }
+        fixture = {"limits": {
+                   "rate": [],
+                   "absolute": {}}}
 
         output = serializer.serialize(fixture, 'index')
         actual = minidom.parseString(output.replace("  ", ""))

=== modified file 'nova/tests/api/openstack/test_servers.py'
--- nova/tests/api/openstack/test_servers.py	2011-08-11 19:30:43 +0000
+++ nova/tests/api/openstack/test_servers.py	2011-08-12 19:02:28 +0000
@@ -3049,8 +3049,7 @@
             address_builder,
             flavor_builder,
             image_builder,
-            base_url,
-            )
+            base_url)
         return view_builder
 
     def test_build_server(self):

=== modified file 'nova/tests/scheduler/test_scheduler.py'
--- nova/tests/scheduler/test_scheduler.py	2011-08-09 22:46:57 +0000
+++ nova/tests/scheduler/test_scheduler.py	2011-08-12 19:02:28 +0000
@@ -643,10 +643,13 @@
         self.mox.StubOutWithMock(driver_i, '_live_migration_dest_check')
         self.mox.StubOutWithMock(driver_i, '_live_migration_common_check')
         driver_i._live_migration_src_check(nocare, nocare)
-        driver_i._live_migration_dest_check(nocare, nocare, i_ref['host'])
-        driver_i._live_migration_common_check(nocare, nocare, i_ref['host'])
+        driver_i._live_migration_dest_check(nocare, nocare,
+                                            i_ref['host'], False)
+        driver_i._live_migration_common_check(nocare, nocare,
+                                              i_ref['host'], False)
         self.mox.StubOutWithMock(rpc, 'cast', use_mock_anything=True)
-        kwargs = {'instance_id': instance_id, 'dest': i_ref['host']}
+        kwargs = {'instance_id': instance_id, 'dest': i_ref['host'],
+                  'block_migration': False}
         rpc.cast(self.context,
                  db.queue_get_for(nocare, FLAGS.compute_topic, i_ref['host']),
                  {"method": 'live_migration', "args": kwargs})
@@ -654,7 +657,8 @@
         self.mox.ReplayAll()
         self.scheduler.live_migration(self.context, FLAGS.compute_topic,
                                       instance_id=instance_id,
-                                      dest=i_ref['host'])
+                                      dest=i_ref['host'],
+                                      block_migration=False)
 
         i_ref = db.instance_get(self.context, instance_id)
         self.assertTrue(i_ref['state_description'] == 'migrating')
@@ -735,7 +739,7 @@
 
         self.assertRaises(exception.ComputeServiceUnavailable,
                           self.scheduler.driver._live_migration_dest_check,
-                          self.context, i_ref, i_ref['host'])
+                          self.context, i_ref, i_ref['host'], False)
 
         db.instance_destroy(self.context, instance_id)
         db.service_destroy(self.context, s_ref['id'])
@@ -748,7 +752,7 @@
 
         self.assertRaises(exception.UnableToMigrateToSelf,
                           self.scheduler.driver._live_migration_dest_check,
-                          self.context, i_ref, i_ref['host'])
+                          self.context, i_ref, i_ref['host'], False)
 
         db.instance_destroy(self.context, instance_id)
         db.service_destroy(self.context, s_ref['id'])
@@ -756,15 +760,33 @@
     def test_live_migration_dest_check_service_lack_memory(self):
         """Confirms exception raises when dest doesn't have enough memory."""
         instance_id = self._create_instance()
-        i_ref = db.instance_get(self.context, instance_id)
-        s_ref = self._create_compute_service(host='somewhere',
-                                             memory_mb_used=12)
-
-        self.assertRaises(exception.MigrationError,
-                          self.scheduler.driver._live_migration_dest_check,
-                          self.context, i_ref, 'somewhere')
-
-        db.instance_destroy(self.context, instance_id)
+        instance_id2 = self._create_instance(host='somewhere',
+                                             memory_mb=12)
+        i_ref = db.instance_get(self.context, instance_id)
+        s_ref = self._create_compute_service(host='somewhere')
+
+        self.assertRaises(exception.MigrationError,
+                          self.scheduler.driver._live_migration_dest_check,
+                          self.context, i_ref, 'somewhere', False)
+
+        db.instance_destroy(self.context, instance_id)
+        db.instance_destroy(self.context, instance_id2)
+        db.service_destroy(self.context, s_ref['id'])
+
+    def test_block_migration_dest_check_service_lack_disk(self):
+        """Confirms exception raises when dest doesn't have enough disk."""
+        instance_id = self._create_instance()
+        instance_id2 = self._create_instance(host='somewhere',
+                                             local_gb=70)
+        i_ref = db.instance_get(self.context, instance_id)
+        s_ref = self._create_compute_service(host='somewhere')
+
+        self.assertRaises(exception.MigrationError,
+                          self.scheduler.driver._live_migration_dest_check,
+                          self.context, i_ref, 'somewhere', True)
+
+        db.instance_destroy(self.context, instance_id)
+        db.instance_destroy(self.context, instance_id2)
         db.service_destroy(self.context, s_ref['id'])
 
     def test_live_migration_dest_check_service_works_correctly(self):
@@ -776,7 +798,8 @@
 
         ret = self.scheduler.driver._live_migration_dest_check(self.context,
                                                              i_ref,
-                                                             'somewhere')
+                                                             'somewhere',
+                                                             False)
         self.assertTrue(ret is None)
         db.instance_destroy(self.context, instance_id)
         db.service_destroy(self.context, s_ref['id'])
@@ -809,9 +832,10 @@
              "args": {'filename': fpath}})
 
         self.mox.ReplayAll()
-        self.assertRaises(exception.SourceHostUnavailable,
+        #self.assertRaises(exception.SourceHostUnavailable,
+        self.assertRaises(exception.FileNotFound,
                           self.scheduler.driver._live_migration_common_check,
-                          self.context, i_ref, dest)
+                          self.context, i_ref, dest, False)
 
         db.instance_destroy(self.context, instance_id)
         db.service_destroy(self.context, s_ref['id'])
@@ -835,7 +859,7 @@
         self.mox.ReplayAll()
         self.assertRaises(exception.InvalidHypervisorType,
                           self.scheduler.driver._live_migration_common_check,
-                          self.context, i_ref, dest)
+                          self.context, i_ref, dest, False)
 
         db.instance_destroy(self.context, instance_id)
         db.service_destroy(self.context, s_ref['id'])
@@ -861,7 +885,7 @@
         self.mox.ReplayAll()
         self.assertRaises(exception.DestinationHypervisorTooOld,
                           self.scheduler.driver._live_migration_common_check,
-                          self.context, i_ref, dest)
+                          self.context, i_ref, dest, False)
 
         db.instance_destroy(self.context, instance_id)
         db.service_destroy(self.context, s_ref['id'])
@@ -893,7 +917,8 @@
         try:
             self.scheduler.driver._live_migration_common_check(self.context,
                                                                i_ref,
-                                                               dest)
+                                                               dest,
+                                                               False)
         except rpc.RemoteError, e:
             c = (e.message.find(_("doesn't have compatibility to")) >= 0)
 

=== modified file 'nova/tests/test_compute.py'
--- nova/tests/test_compute.py	2011-08-09 22:46:57 +0000
+++ nova/tests/test_compute.py	2011-08-12 19:02:28 +0000
@@ -714,11 +714,15 @@
         dbmock.queue_get_for(c, FLAGS.compute_topic, i_ref['host']).\
                              AndReturn(topic)
         rpc.call(c, topic, {"method": "pre_live_migration",
-                            "args": {'instance_id': i_ref['id']}})
+                            "args": {'instance_id': i_ref['id'],
+                                     'block_migration': False,
+                                     'disk': None}})
+
         self.mox.StubOutWithMock(self.compute.driver, 'live_migration')
         self.compute.driver.live_migration(c, i_ref, i_ref['host'],
                                   self.compute.post_live_migration,
-                                  self.compute.recover_live_migration)
+                                  self.compute.rollback_live_migration,
+                                  False)
 
         self.compute.db = dbmock
         self.mox.ReplayAll()
@@ -739,13 +743,18 @@
         dbmock.queue_get_for(c, FLAGS.compute_topic, i_ref['host']).\
                              AndReturn(topic)
         rpc.call(c, topic, {"method": "pre_live_migration",
-                            "args": {'instance_id': i_ref['id']}}).\
+                            "args": {'instance_id': i_ref['id'],
+                                     'block_migration': False,
+                                     'disk': None}}).\
                             AndRaise(rpc.RemoteError('', '', ''))
         dbmock.instance_update(c, i_ref['id'], {'state_description': 'running',
                                                 'state': power_state.RUNNING,
                                                 'host': i_ref['host']})
         for v in i_ref['volumes']:
             dbmock.volume_update(c, v['id'], {'status': 'in-use'})
+            # mock for volume_api.remove_from_compute
+            rpc.call(c, topic, {"method": "remove_volume",
+                                "args": {'volume_id': v['id']}})
 
         self.compute.db = dbmock
         self.mox.ReplayAll()
@@ -766,7 +775,9 @@
                              AndReturn(topic)
         self.mox.StubOutWithMock(rpc, 'call')
         rpc.call(c, topic, {"method": "pre_live_migration",
-                            "args": {'instance_id': i_ref['id']}}).\
+                            "args": {'instance_id': i_ref['id'],
+                                     'block_migration': False,
+                                     'disk': None}}).\
                             AndRaise(rpc.RemoteError('', '', ''))
         dbmock.instance_update(c, i_ref['id'], {'state_description': 'running',
                                                 'state': power_state.RUNNING,
@@ -791,11 +802,14 @@
         dbmock.queue_get_for(c, FLAGS.compute_topic, i_ref['host']).\
                              AndReturn(topic)
         rpc.call(c, topic, {"method": "pre_live_migration",
-                            "args": {'instance_id': i_ref['id']}})
+                            "args": {'instance_id': i_ref['id'],
+                                     'block_migration': False,
+                                     'disk': None}})
         self.mox.StubOutWithMock(self.compute.driver, 'live_migration')
         self.compute.driver.live_migration(c, i_ref, i_ref['host'],
                                   self.compute.post_live_migration,
-                                  self.compute.recover_live_migration)
+                                  self.compute.rollback_live_migration,
+                                  False)
 
         self.compute.db = dbmock
         self.mox.ReplayAll()
@@ -829,6 +843,10 @@
             self.compute.volume_manager.remove_compute_volume(c, v['id'])
         self.mox.StubOutWithMock(self.compute.driver, 'unfilter_instance')
         self.compute.driver.unfilter_instance(i_ref, [])
+        self.mox.StubOutWithMock(rpc, 'call')
+        rpc.call(c, db.queue_get_for(c, FLAGS.compute_topic, dest),
+            {"method": "post_live_migration_at_destination",
+             "args": {'instance_id': i_ref['id'], 'block_migration': False}})
 
         # executing
         self.mox.ReplayAll()

=== modified file 'nova/tests/test_libvirt.py'
--- nova/tests/test_libvirt.py	2011-08-11 12:34:04 +0000
+++ nova/tests/test_libvirt.py	2011-08-12 19:02:28 +0000
@@ -21,6 +21,7 @@
 import re
 import shutil
 import sys
+import tempfile
 
 from xml.etree.ElementTree import fromstring as xml_to_tree
 from xml.dom.minidom import parseString as xml_to_dom
@@ -690,17 +691,20 @@
                 return vdmock
 
         self.create_fake_libvirt_mock(lookupByName=fake_lookup)
-        self.mox.StubOutWithMock(self.compute, "recover_live_migration")
-        self.compute.recover_live_migration(self.context, instance_ref,
-                                            dest='dest')
+#        self.mox.StubOutWithMock(self.compute, "recover_live_migration")
+        self.mox.StubOutWithMock(self.compute, "rollback_live_migration")
+#        self.compute.recover_live_migration(self.context, instance_ref,
+#                                             dest='dest')
+        self.compute.rollback_live_migration(self.context, instance_ref,
+                                            'dest', False)
 
-        # Start test
+        #start test
         self.mox.ReplayAll()
         conn = connection.LibvirtConnection(False)
         self.assertRaises(libvirt.libvirtError,
                       conn._live_migration,
-                      self.context, instance_ref, 'dest', '',
-                      self.compute.recover_live_migration)
+                      self.context, instance_ref, 'dest', False,
+                      self.compute.rollback_live_migration)
 
         instance_ref = db.instance_get(self.context, instance_ref['id'])
         self.assertTrue(instance_ref['state_description'] == 'running')
@@ -711,6 +715,95 @@
         db.volume_destroy(self.context, volume_ref['id'])
         db.instance_destroy(self.context, instance_ref['id'])
 
+    def test_pre_block_migration_works_correctly(self):
+        """Confirms pre_block_migration works correctly."""
+
+        # Skip if non-libvirt environment
+        if not self.lazy_load_library_exists():
+            return
+
+        # Replace instances_path since this testcase creates tmpfile
+        tmpdir = tempfile.mkdtemp()
+        store = FLAGS.instances_path
+        FLAGS.instances_path = tmpdir
+
+        # Test data
+        instance_ref = db.instance_create(self.context, self.test_instance)
+        dummyjson = '[{"path": "%s/disk", "local_gb": "10G", "type": "raw"}]'
+
+        # Preparing mocks
+        # qemu-img should be mockd since test environment might not have
+        # large disk space.
+        self.mox.StubOutWithMock(utils, "execute")
+        utils.execute('sudo', 'qemu-img', 'create', '-f', 'raw',
+                      '%s/%s/disk' % (tmpdir, instance_ref.name), '10G')
+
+        self.mox.ReplayAll()
+        conn = connection.LibvirtConnection(False)
+        conn.pre_block_migration(self.context, instance_ref,
+                                 dummyjson % tmpdir)
+
+        self.assertTrue(os.path.exists('%s/%s/' %
+                                       (tmpdir, instance_ref.name)))
+
+        shutil.rmtree(tmpdir)
+        db.instance_destroy(self.context, instance_ref['id'])
+        # Restore FLAGS.instances_path
+        FLAGS.instances_path = store
+
+    def test_get_instance_disk_info_works_correctly(self):
+        """Confirms pre_block_migration works correctly."""
+        # Skip if non-libvirt environment
+        if not self.lazy_load_library_exists():
+            return
+
+        # Test data
+        instance_ref = db.instance_create(self.context, self.test_instance)
+        dummyxml = ("<domain type='kvm'><name>instance-0000000a</name>"
+                    "<devices>"
+                    "<disk type='file'><driver name='qemu' type='raw'/>"
+                    "<source file='/test/disk'/>"
+                    "<target dev='vda' bus='virtio'/></disk>"
+                    "<disk type='file'><driver name='qemu' type='qcow2'/>"
+                    "<source file='/test/disk.local'/>"
+                    "<target dev='vdb' bus='virtio'/></disk>"
+                    "</devices></domain>")
+
+        ret = ("image: /test/disk\nfile format: raw\n"
+               "virtual size: 20G (21474836480 bytes)\ndisk size: 3.1G\n")
+
+        # Preparing mocks
+        vdmock = self.mox.CreateMock(libvirt.virDomain)
+        self.mox.StubOutWithMock(vdmock, "XMLDesc")
+        vdmock.XMLDesc(0).AndReturn(dummyxml)
+
+        def fake_lookup(instance_name):
+            if instance_name == instance_ref.name:
+                return vdmock
+        self.create_fake_libvirt_mock(lookupByName=fake_lookup)
+
+        self.mox.StubOutWithMock(os.path, "getsize")
+        # based on above testdata, one is raw image, so getsize is mocked.
+        os.path.getsize("/test/disk").AndReturn(10 * 1024 * 1024 * 1024)
+        # another is qcow image, so qemu-img should be mocked.
+        self.mox.StubOutWithMock(utils, "execute")
+        utils.execute('sudo', 'qemu-img', 'info', '/test/disk.local').\
+            AndReturn((ret, ''))
+
+        self.mox.ReplayAll()
+        conn = connection.LibvirtConnection(False)
+        info = conn.get_instance_disk_info(self.context, instance_ref)
+        info = utils.loads(info)
+
+        self.assertTrue(info[0]['type'] == 'raw' and
+                        info[1]['type'] == 'qcow2' and
+                        info[0]['path'] == '/test/disk' and
+                        info[1]['path'] == '/test/disk.local' and
+                        info[0]['local_gb'] == '10G' and
+                        info[1]['local_gb'] == '20G')
+
+        db.instance_destroy(self.context, instance_ref['id'])
+
     def test_spawn_with_network_info(self):
         # Skip if non-libvirt environment
         if not self.lazy_load_library_exists():

=== modified file 'nova/virt/fake.py'
--- nova/virt/fake.py	2011-08-09 22:46:57 +0000
+++ nova/virt/fake.py	2011-08-12 19:02:28 +0000
@@ -492,7 +492,7 @@
         raise NotImplementedError('This method is supported only by libvirt.')
 
     def live_migration(self, context, instance_ref, dest,
-                       post_method, recover_method):
+                       post_method, recover_method, block_migration=False):
         """This method is supported only by libvirt."""
         return
 

=== modified file 'nova/virt/libvirt/connection.py'
--- nova/virt/libvirt/connection.py	2011-08-11 12:34:04 +0000
+++ nova/virt/libvirt/connection.py	2011-08-12 19:02:28 +0000
@@ -117,6 +117,10 @@
 flags.DEFINE_string('live_migration_flag',
                     "VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER",
                     'Define live migration behavior.')
+flags.DEFINE_string('block_migration_flag',
+                    "VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, "
+                    "VIR_MIGRATE_NON_SHARED_INC",
+                    'Define block migration behavior.')
 flags.DEFINE_integer('live_migration_bandwidth', 0,
                     'Define live migration behavior')
 flags.DEFINE_string('qemu_img', 'qemu-img',
@@ -727,6 +731,7 @@
 
         If cow is True, it will make a CoW image instead of a copy.
         """
+
         if not os.path.exists(target):
             base_dir = os.path.join(FLAGS.instances_path, '_base')
             if not os.path.exists(base_dir):
@@ -1552,7 +1557,7 @@
             time.sleep(1)
 
     def live_migration(self, ctxt, instance_ref, dest,
-                       post_method, recover_method):
+                       post_method, recover_method, block_migration=False):
         """Spawning live_migration operation for distributing high-load.
 
         :params ctxt: security context
@@ -1560,20 +1565,22 @@
             nova.db.sqlalchemy.models.Instance object
             instance object that is migrated.
         :params dest: destination host
+        :params block_migration: destination host
         :params post_method:
             post operation method.
             expected nova.compute.manager.post_live_migration.
         :params recover_method:
             recovery method when any exception occurs.
             expected nova.compute.manager.recover_live_migration.
+        :params block_migration: if true, do block migration.
 
         """
 
         greenthread.spawn(self._live_migration, ctxt, instance_ref, dest,
-                          post_method, recover_method)
+                          post_method, recover_method, block_migration)
 
-    def _live_migration(self, ctxt, instance_ref, dest,
-                        post_method, recover_method):
+    def _live_migration(self, ctxt, instance_ref, dest, post_method,
+                        recover_method, block_migration=False):
         """Do live migration.
 
         :params ctxt: security context
@@ -1592,27 +1599,21 @@
 
         # Do live migration.
         try:
-            flaglist = FLAGS.live_migration_flag.split(',')
+            if block_migration:
+                flaglist = FLAGS.block_migration_flag.split(',')
+            else:
+                flaglist = FLAGS.live_migration_flag.split(',')
             flagvals = [getattr(libvirt, x.strip()) for x in flaglist]
             logical_sum = reduce(lambda x, y: x | y, flagvals)
 
-            if self.read_only:
-                tmpconn = self._connect(self.libvirt_uri, False)
-                dom = tmpconn.lookupByName(instance_ref.name)
-                dom.migrateToURI(FLAGS.live_migration_uri % dest,
-                                 logical_sum,
-                                 None,
-                                 FLAGS.live_migration_bandwidth)
-                tmpconn.close()
-            else:
-                dom = self._conn.lookupByName(instance_ref.name)
-                dom.migrateToURI(FLAGS.live_migration_uri % dest,
-                                 logical_sum,
-                                 None,
-                                 FLAGS.live_migration_bandwidth)
+            dom = self._conn.lookupByName(instance_ref.name)
+            dom.migrateToURI(FLAGS.live_migration_uri % dest,
+                             logical_sum,
+                             None,
+                             FLAGS.live_migration_bandwidth)
 
         except Exception:
-            recover_method(ctxt, instance_ref, dest=dest)
+            recover_method(ctxt, instance_ref, dest, block_migration)
             raise
 
         # Waiting for completion of live_migration.
@@ -1624,11 +1625,150 @@
                 self.get_info(instance_ref.name)['state']
             except exception.NotFound:
                 timer.stop()
-                post_method(ctxt, instance_ref, dest)
+                post_method(ctxt, instance_ref, dest, block_migration)
 
         timer.f = wait_for_live_migration
         timer.start(interval=0.5, now=True)
 
+    def pre_block_migration(self, ctxt, instance_ref, disk_info_json):
+        """Preparation block migration.
+
+        :params ctxt: security context
+        :params instance_ref:
+            nova.db.sqlalchemy.models.Instance object
+            instance object that is migrated.
+        :params disk_info_json:
+            json strings specified in get_instance_disk_info
+
+        """
+        disk_info = utils.loads(disk_info_json)
+
+        # make instance directory
+        instance_dir = os.path.join(FLAGS.instances_path, instance_ref['name'])
+        if os.path.exists(instance_dir):
+            raise exception.DestinationDiskExists(path=instance_dir)
+        os.mkdir(instance_dir)
+
+        for info in disk_info:
+            base = os.path.basename(info['path'])
+            # Get image type and create empty disk image.
+            instance_disk = os.path.join(instance_dir, base)
+            utils.execute('sudo', 'qemu-img', 'create', '-f', info['type'],
+                          instance_disk, info['local_gb'])
+
+        # if image has kernel and ramdisk, just download
+        # following normal way.
+        if instance_ref['kernel_id']:
+            user = manager.AuthManager().get_user(instance_ref['user_id'])
+            project = manager.AuthManager().get_project(
+                instance_ref['project_id'])
+            self._fetch_image(nova_context.get_admin_context(),
+                              os.path.join(instance_dir, 'kernel'),
+                              instance_ref['kernel_id'],
+                              user,
+                              project)
+            if instance_ref['ramdisk_id']:
+                self._fetch_image(nova_context.get_admin_context(),
+                                  os.path.join(instance_dir, 'ramdisk'),
+                                  instance_ref['ramdisk_id'],
+                                  user,
+                                  project)
+
+    def post_live_migration_at_destination(self, ctxt,
+                                           instance_ref,
+                                           network_info,
+                                           block_migration):
+        """Post operation of live migration at destination host.
+
+        :params ctxt: security context
+        :params instance_ref:
+            nova.db.sqlalchemy.models.Instance object
+            instance object that is migrated.
+        :params network_info: instance network infomation
+        :params : block_migration: if true, post operation of block_migraiton.
+        """
+        # Define migrated instance, otherwise, suspend/destroy does not work.
+        dom_list = self._conn.listDefinedDomains()
+        if instance_ref.name not in dom_list:
+            instance_dir = os.path.join(FLAGS.instances_path,
+                                        instance_ref.name)
+            xml_path = os.path.join(instance_dir, 'libvirt.xml')
+            # In case of block migration, destination does not have
+            # libvirt.xml
+            if not os.path.isfile(xml_path):
+                xml = self.to_xml(instance_ref, network_info=network_info)
+                f = open(os.path.join(instance_dir, 'libvirt.xml'), 'w+')
+                f.write(xml)
+                f.close()
+            # libvirt.xml should be made by to_xml(), but libvirt
+            # does not accept to_xml() result, since uuid is not
+            # included in to_xml() result.
+            dom = self._lookup_by_name(instance_ref.name)
+            self._conn.defineXML(dom.XMLDesc(0))
+
+    def get_instance_disk_info(self, ctxt, instance_ref):
+        """Preparation block migration.
+
+        :params ctxt: security context
+        :params instance_ref:
+            nova.db.sqlalchemy.models.Instance object
+            instance object that is migrated.
+        :return:
+            json strings with below format.
+           "[{'path':'disk', 'type':'raw', 'local_gb':'10G'},...]"
+
+        """
+        disk_info = []
+
+        virt_dom = self._lookup_by_name(instance_ref.name)
+        xml = virt_dom.XMLDesc(0)
+        doc = libxml2.parseDoc(xml)
+        disk_nodes = doc.xpathEval('//devices/disk')
+        path_nodes = doc.xpathEval('//devices/disk/source')
+        driver_nodes = doc.xpathEval('//devices/disk/driver')
+
+        for cnt, path_node in enumerate(path_nodes):
+            disk_type = disk_nodes[cnt].get_properties().getContent()
+            path = path_node.get_properties().getContent()
+
+            if disk_type != 'file':
+                LOG.debug(_('skipping %(path)s since it looks like volume') %
+                          locals())
+                continue
+
+            # In case of libvirt.xml, disk type can be obtained
+            # by the below statement.
+            # -> disk_type = driver_nodes[cnt].get_properties().getContent()
+            # but this xml is generated by kvm, format is slightly different.
+            disk_type = \
+                driver_nodes[cnt].get_properties().get_next().getContent()
+            if disk_type == 'raw':
+                size = int(os.path.getsize(path))
+            else:
+                out, err = utils.execute('sudo', 'qemu-img', 'info', path)
+                size = [i.split('(')[1].split()[0] for i in out.split('\n')
+                    if i.strip().find('virtual size') >= 0]
+                size = int(size[0])
+
+            # block migration needs same/larger size of empty image on the
+            # destination host. since qemu-img creates bit smaller size image
+            # depending on original image size, fixed value is necessary.
+            for unit, divisor in [('G', 1024 ** 3), ('M', 1024 ** 2),
+                                  ('K', 1024), ('', 1)]:
+                if size / divisor == 0:
+                    continue
+                if size % divisor != 0:
+                    size = size / divisor + 1
+                else:
+                    size = size / divisor
+                size = str(size) + unit
+                break
+
+            disk_info.append({'type': disk_type, 'path': path,
+                              'local_gb': size})
+
+        return utils.dumps(disk_info)
+
     def unfilter_instance(self, instance_ref, network_info):
         """See comments of same method in firewall_driver."""
         self.firewall_driver.unfilter_instance(instance_ref,

=== modified file 'nova/virt/xenapi_conn.py'
--- nova/virt/xenapi_conn.py	2011-08-09 22:46:57 +0000
+++ nova/virt/xenapi_conn.py	2011-08-12 19:02:28 +0000
@@ -314,7 +314,7 @@
         return
 
     def live_migration(self, context, instance_ref, dest,
-                       post_method, recover_method):
+                       post_method, recover_method, block_migration=False):
         """This method is supported only by libvirt."""
         return