mirror of
https://github.com/apache/cloudstack.git
synced 2025-11-02 20:02:29 +01:00
c89198f4c6
CLOUDSTACK-9626: Instance fails to start after unsuccesful computeISSUE ============ Instance fails to start after unsuccesful compute offering upgrade. TROUBLESHOOTING ================== We observed VM instance get compute values "cpuNumber","cpuSpeed","memory" removed from table "user_vm_details", which cause instance fail to startup next time on XenServer `mysql> select * from user_vm_details where vm_id=10; -------------------------------------------------------------------------------------------------- id vm_id name value display -------------------------------------------------------------------------------------------------- 218 10 platform viridian:true;acpi:1;apic:true;pae:true;nx:true 1 219 10 hypervisortoolsversion xenserver56 1 220 10 Message.ReservedCapacityFreed.Flag true 1 -------------------------------------------------------------------------------------------------- 3 rows in set (0.00 sec)` `2016-11-29 06:49:03,667 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-49c25b1d job-125) (logid:114a2f1b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin java.lang.NullPointerException at com.cloud.vm.UserVmManagerImpl.upgradeRunningVirtualMachine(UserVmManagerImpl.java:1664) at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1631) at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1561) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy197.upgradeVirtualMachine(Unknown Source) at org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin.execute(ScaleVMCmdByAdmin.java:48) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)` REPRO STEPS ================== 1. Set global setting enable.dynamic.scale.vm to true 2. Create a custom Compute Offerings A 3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M 4. Create another custom Compute Offerings B 5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M (ensure 4 times over previous memory size), then you will encounter scaling failed 6. Stop VM instance , you will never startup again EXPECTED BEHAVIOR ================== Succeed Startup VM instance ACTUAL BEHAVIOR ================== Fail to start instance RCA: The ROLLBACK does not take care of restoring old service offering details. In case failure we are removing the new service offering details but restoring old service offering details is missing. Before Fix: `user_vm_details before upgrade. mysql> select * from user_vm_details where vm_id =9; +-----+-------+------------------------------------+-------------------------------------------------+---------+ | id | vm_id | name | value | display | +-----+-------+------------------------------------+-------------------------------------------------+---------+ | 118 | 9 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | | 119 | 9 | hypervisortoolsversion | xenserver56 | 1 | | 120 | 9 | Message.ReservedCapacityFreed.Flag | false | 1 | | 121 | 9 | cpuNumber | 1 | 1 | | 122 | 9 | cpuSpeed | 1000 | 1 | | 123 | 9 | memory | 256 | 1 | +-----+-------+------------------------------------+-------------------------------------------------+---------+ 6 rows in set (0.00 sec) user_vm_details after unsuccessful upgrade. mysql> select * from user_vm_details where vm_id =9; +-----+-------+------------------------------------+-------------------------------------------------+---------+ | id | vm_id | name | value | display | +-----+-------+------------------------------------+-------------------------------------------------+---------+ | 133 | 9 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | | 134 | 9 | hypervisortoolsversion | xenserver56 | 1 | | 135 | 9 | Message.ReservedCapacityFreed.Flag | false | 1 | +-----+-------+------------------------------------+-------------------------------------------------+---------+ 3 rows in set (0.00 sec)` After fix: ` mysql> select * from user_vm_details where vm_id =9; +-----+-------+------------------------------------+-------------------------------------------------+---------+ | id | vm_id | name | value | display | +-----+-------+------------------------------------+-------------------------------------------------+---------+ | 166 | 9 | cpuNumber | 1 | 1 | | 167 | 9 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | | 168 | 9 | cpuSpeed | 1000 | 1 | | 169 | 9 | Message.ReservedCapacityFreed.Flag | false | 1 | | 170 | 9 | memory | 256 | 1 | | 171 | 9 | hypervisortoolsversion | xenserver56 | 1 | +-----+-------+------------------------------------+-------------------------------------------------+---------+ 6 rows in set (0.00 sec) ` * pr/1796: CLOUDSTACK-9626: Instance fails to start after unsuccesful compute offering upgrade. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>