apache/cloudstack
1
0
Fork 0
mirror of https://github.com/apache/cloudstack.git synced 2025-10-26 08:42:29 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1796 from sudhansu7/CLOUDSTACK-9626

Browse Source 
CLOUDSTACK-9626: Instance fails to start after unsuccesful computeISSUE
============
Instance fails to start after unsuccesful compute offering upgrade.

TROUBLESHOOTING
==================
We observed VM instance get compute values "cpuNumber","cpuSpeed","memory" removed from table "user_vm_details", which cause instance fail to startup next time on XenServer

`mysql> select * from user_vm_details where vm_id=10;
--------------------------------------------------------------------------------------------------
id 	vm_id 	name 	value 	display

--------------------------------------------------------------------------------------------------
218 	10 	platform 	viridian:true;acpi:1;apic:true;pae:true;nx:true 	1
219 	10 	hypervisortoolsversion 	xenserver56 	1
220 	10 	Message.ReservedCapacityFreed.Flag 	true 	1

--------------------------------------------------------------------------------------------------
3 rows in set (0.00 sec)`

`2016-11-29 06:49:03,667 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-49c25b1d job-125) (logid:114a2f1b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin
java.lang.NullPointerException
	at com.cloud.vm.UserVmManagerImpl.upgradeRunningVirtualMachine(UserVmManagerImpl.java:1664)
	at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1631)
	at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1561)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy197.upgradeVirtualMachine(Unknown Source)
	at org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin.execute(ScaleVMCmdByAdmin.java:48)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)`

REPRO STEPS
==================
1. Set global setting enable.dynamic.scale.vm to true
2. Create a custom Compute Offerings A
3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M
4. Create another custom Compute Offerings B
5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M (ensure 4 times over previous memory size), then you will encounter scaling failed
6. Stop VM instance , you will never startup again

EXPECTED BEHAVIOR
==================
Succeed Startup VM instance

ACTUAL BEHAVIOR
==================
Fail to start instance

RCA:
The ROLLBACK does not take care of restoring old service offering details. In case failure we are removing the new service offering details but restoring old service offering details is missing.

Before Fix:
`user_vm_details before upgrade.
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id  | vm_id | name                               | value                                           | display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 118 |     9 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true |       1 |
| 119 |     9 | hypervisortoolsversion             | xenserver56                                     |       1 |
| 120 |     9 | Message.ReservedCapacityFreed.Flag | false                                           |       1 |
| 121 |     9 | cpuNumber                          | 1                                               |       1 |
| 122 |     9 | cpuSpeed                           | 1000                                            |       1 |
| 123 |     9 | memory                             | 256                                             |       1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
6 rows in set (0.00 sec)

user_vm_details after unsuccessful upgrade.
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id  | vm_id | name                               | value                                           | display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 133 |     9 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true |       1 |
| 134 |     9 | hypervisortoolsversion             | xenserver56                                     |       1 |
| 135 |     9 | Message.ReservedCapacityFreed.Flag | false                                           |       1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
3 rows in set (0.00 sec)`

After fix:
`
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id  | vm_id | name                               | value                                           | display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 166 |     9 | cpuNumber                          | 1                                               |       1 |
| 167 |     9 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true |       1 |
| 168 |     9 | cpuSpeed                           | 1000                                            |       1 |
| 169 |     9 | Message.ReservedCapacityFreed.Flag | false                                           |       1 |
| 170 |     9 | memory                             | 256                                             |       1 |
| 171 |     9 | hypervisortoolsversion             | xenserver56                                     |       1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
6 rows in set (0.00 sec)
`

* pr/1796:
  CLOUDSTACK-9626: Instance fails to start after unsuccesful compute offering  upgrade.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This commit is contained in:
Rohit Yadav 2016-12-09 23:52:47 +05:30
commit c89198f4c6
1 changed files with 5 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
server/src/com/cloud/vm/UserVmManagerImpl.java
View File

@ -1759,13 +1759,15 @@ public class UserVmManagerImpl extends ManagerBase implements UserVmManager, Vir
} finally {
if (!success) {
_itMgr.upgradeVmDb(vmId, currentServiceOffering.getId()); // rollback
if (newServiceOffering.isDynamic()) {
removeCustomOfferingDetails(vmId);
}
// Decrement CPU and Memory count accordingly.
if (newCpu > currentCpu) {
_resourceLimitMgr.decrementResourceCount(caller.getAccountId(), ResourceType.cpu, new Long(newCpu - currentCpu));
}
//restoring old service offering will take care of removing new SO.
if(currentServiceOffering.isDynamic()){
saveCustomOfferingDetails(vmId, currentServiceOffering);
}
if (memoryDiff > 0) {
_resourceLimitMgr.decrementResourceCount(caller.getAccountId(), ResourceType.memory, new Long(memoryDiff));