Adding support for cross-cluster storage migration for managed storage when using XenServerThis PR adds support for cross-cluster storage migration of VMs that make use of managed storage with XenServer.
Managed storage is when you have a 1:1 mapping between a virtual disk and a volume on a SAN (in the case of XenServer, an SR is placed on this SAN volume and a single virtual disk placed in the SR).
Managed storage allows features such as storage QoS and SAN-side snapshots to work (sort of analogous to VMware VVols).
This PR focuses on enabling VMs that are using managed storage to be migrated across XenServer clusters.
I have successfully run the following tests on this branch:
TestVolumes.py
TestSnapshots.py
TestVMSnapshots.py
TestAddRemoveHosts.py
TestVMMigrationWithStorage.py (which is a new test that is being added with this PR)
* pr/1671:
Adding support for cross-cluster storage migration for managed storage when using XenServer
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Added checks to prevent netwrok update when router state is unknown or when
the new offering removes a service that is in use.
Added a new param forced to the updateNetwork API. The network will
undergo a forced update when this param is set to true.
CLOUDSTACK-8751 Clean network config like firewall rules etc, when network services are removed during network update.
CLOUDSTACK-9428: Fix for CLOUDSTACK-9211 - Improve performance of 3D GPU support in cloud-plugin-hypervisor-vmwareJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9428
### Introduction
On #1310 passing vRAM size to support 3D GPU problem was addressed on VMware. It was found out that it could be improved to increase performance by reducing extra API calls, as we'll describe later
### Improvement
On WMware, `VmwareResource` manages execution of `StartCommand.` Before sending power on command to ESXi hypervisor, vm is configured by calling `reconfigVMTask` web method on vSphere's client `VimPortType` web service.
It was found out that we were using this method 2 times when passing vRAM size, as it implied creating a new vm config spec only editing video card specs and making an extra call to `reconfigVMTask.`
We propose reducing the extra web service call by adjusting vm's config spec. This way video card gets properly configured (when passing vRAM size) in the same configure call, increasing performance.
### Use case (passing vRAM size)
* Deploy a new VM, let its id be X
* Stop VM
* Execute SQL, where X is vm's id and Z is vRAM size (in kB):
````
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'mks.enable3d', 'true');
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'mks.use3dRenderer', 'automatic');
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'svga.autodetect', 'false');
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'svga.vramSize', Z);
````
* Start VM
* pr/1605:
CLOUDSTACK-9428: Add marvin test
CLOUDSTACK-9428: Fix for CLOUDSTACK-9211 - Improve performance
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
CLOUDSTACK-9238: Fix URL length to 2048 for all url fields in VOI will update the PR to add max field length in the API commands too
* pr/1567:
API: update url field max length
not needed on host table
Fix URL length to 2048 for all url fields in VO
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.7:
CLOUDSTACK-9376: Restrict listTemplates API with filter=all for root admin
CLOUDSTACK-9369: Restrict default login to ldap/native users
Add lsb-release dependency to mgmt server and agent on Debian/Ubuntu.
Emit template UUID and class type over event bus when deleting templates.
- Restricts default login auth handler to ldap and native-cloudstack users
- Refactors and create re-usable method to find domain by id/path
- Adds unit test for refactored method in DomainManagerImpl
- Adds smoke test for login handler
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts## Description
JIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9368
This pull request address a problem introduced in #1361 in which NFS version couldn't be changed after hosts resources were configured on startup (for hosts using `VmwareResource`), and as host parameters didn't include `nfs.version` key, it was set `null`.
## Proposed solution
In this proposed solution `nfsVersion` would be passed in `NfsTO` through `CopyCommand` to `VmwareResource`, who will check if NFS version is still configured or not. If not, it will use the one sent in the command and will set it to its storage processor and storage handler. After those setups, it will proceed executing command.
* pr/1518:
CLOUDSTACK-9368: Fix for Support configurable NFS version for Secondary Storage mounts
Signed-off-by: Will Stevens <williamstevens@gmail.com>
cherry-picked from a exoscale internal fix:
Implement security group move on updateVM API call
Prevent security group update while instance is running
Conflicts:
api/src/org/apache/cloudstack/api/command/user/vm/UpdateVMCmd.java
server/src/com/cloud/vm/UserVmManager.java
server/src/com/cloud/vm/UserVmManagerImpl.java
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMThis PR introduces the changes proposed in PR #780 with some work to make the code null safe.
During this PR, I have also removed some unused code.
* pr/1444:
Removed unnecessary check when creating the “userVmResponse” object.
Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Remove classes with no referencesI used UCDetector (http://www.ucdetector.org/) as a plugin for Eclipse. With this tool, I discovered a lot of code without any reference (variables, methods and classes).
Following the work that was done at [https://github.com/apache/cloudstack/pull/1448]; this pull request had the goal of removing some of these classes. To check if I wasn't missing anything I searched for any file that could reference some of those classes. As I haven't found any way of these classes being used, they were removed. Note that some of them I found other references, but references such as commented lines or tests, nothing that could indicate their use (as XML files configuring beans or another class instantiating an object with "new").
Waiting for tests. Please tell me if I am missing something.
Removed Classes:
- org.apache.cloudstack.framework.jobs.JobCancellationException (**Note:** removed
variable JobCancellationException in com.cloud.utils.SerialVersionUID)
- org.apache.cloudstack.ldap.NoSuchLdapUserException (**Note:** removed test file
/cloud-plugin-user-authenticator-ldap/test/groovy/org/apache/cloudstack/ldap/NoSuchLdapUserExceptionSpec.groovy)
- com.cloud.agent.api.storage.CreateVolumeOVAAnswer
- com.cloud.exception.MissingParameterValueException
- org.apache.cloudstack.api.response.StatusResponse
- org.apache.cloudstack.api.response.VolumeDetailResponse
- org.apache.cloudstack.api.response.UpgradeVmResponse
- org.apache.cloudstack.api.response.AddIpToVmNicResponse
- org.apache.cloudstack.api.response.TemplateZoneResponse (**Note:** at
org.apache.cloudstack.api.response.TemplateResponse, there is this
comment "To avoid breaking backwards compatibility, we still treat a
template at different zones as different templates, so not embedding
template_zone information in this TemplateZoneResponse set. `private
Set<TemplateZoneResponse> zones;`" but right now it is not used)
- org.apache.cloudstack.api.response.NicDetailResponse
* pr/1453:
Removed classes with no reference
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.
Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.
This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host
For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This feature allows root administrators to define new roles and associate API
permissions to them.
A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
Admin, Domain Admin and User) which maintains this association by requiring
all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
removal of roles and/or modifications of permissions, without the need
of restarting management server(s)
Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
deployments will continue to use the older static role based api access checker
with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
to add existing set of permissions to the default roles. cloud.account
will have a new role_id column which will be populated based on default roles
as well
Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Validate API arguments based on annotations. Introduces:
- NotNullOrEmpty: for doing null and empty string checks
- PositiveNumber: number > 0 (natural number)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9088: Update the description for migrateVirtualMachineWithVolume apihttps://issues.apache.org/jira/browse/CLOUDSTACK-9088
* pr/1126:
CLOUDSTACK-9088: Update the description for migrateVirtualMachineWithVolume api.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- Improve disk chain usage while attaching, migrating disks
- Gets root disk controller based diskDeviceBusName from volume's chain info
- Refactor and move VirtualMachineDiskInfo to cloud-utils
- Allows mixing of scsi controller types
- Fixes a NPE case with map passed as null, for example in case of detach volume
command
- Use a osdefault translator that allow use of recent os types added (enums of
which) are not available in the sdk
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9251: Fix issue in scale VM to dynamic service offeringThis reverts commit 9c4162ac7f451fc3e2155418dcfff224c8c08a4a and 16baa1289b7de383e98d0070717b3f1873fa2db3
Before change: exception when change compute offering (to dynamic service offering) on UI
After change: succeed
* pr/1363:
Fix issue in scale VM to dynamic service offering
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Removed unused code from com.cloud.api.ApiServer**Removed \_ from variables names**: private variables with \_ at the beginning is common in C++ but not in Java.
**Removed unused code from ApiServer:**
- com.cloud.api.ApiServer.getPluggableServices(): unused method;
- com.cloud.api.ApiServer.getApiAccessCheckers(): unused method;
**Methods and variables access level reviewed:**
- com.cloud.api.ApiServer.handleAsyncJobPublishEvent(String, String ,Object): this method was private but the annotation @MessageHandler requests public methods, as can be seen in org.apache.cloudstack.framework.messagebus.MessageDispatcher.buildHandlerMethodCache(Class\<?\>), which searches methods with the @MessageHandler annotation and changes
it to be accessible (setAccessible(true)). Thus, there is no reason for handleAsyncJobPublishEvent be a private method and lead some other dev to wrong conclusions about the use of the method;
- Global variables and methods called just by this class (ApiServer) were changed to private.
**Changed variables and methods from static to non-static (if possible):** as some variables/methods are used just by one object of this class, instantiated by Spring, they were changed to non-static.
With that, calls from com.cloud.api.ApiServlet.ApiServlet() that used static methods from ApiServer, were changed from ApiServer.\<staticMethodName\> to \_apiServer.\<methodName\> that refers to the org.apache.cloudstack.api.ApiServerService interface. Thus, methods com.cloud.api.ApiServer.getJSONContentType() and com.cloud.api.ApiServer.isSecureSessionCookieEnabled() had to be added in the interface (org.apache.cloudstack.api.ApiServerService, interface implemented by class ApiServer).
* pr/1263:
The goal of this PR is to review com.cloud.api.ApiServer class, with the following actions:
Signed-off-by: Will Stevens <williamstevens@gmail.com>
following actions:
Removed “_” in beginning of global variables names:
Variables was changed from “_<variablename>” to “<variablename>”, as
this convension (private veriables with “_”) is common in C++ but not in
Java.
Removed unused code from ApiServer:
- com.cloud.api.ApiServer.getPluggableServices():
Unused method.
- com.cloud.api.ApiServer.getApiAccessCheckers():
Unused method.
Methods and variables access level reviewed:
- com.cloud.api.ApiServer.handleAsyncJobPublishEvent(String, String,
Object):
This method was private but the annotation @MessageHandler requests
public methods, as can be seen in
org.apache.cloudstack.framework.messagebus.MessageDispatcher.buildHandlerMethodCache(Class<?>),
which searches methods with the @MessageHandler annotation and changes
it to accessible (“setAccessible(true)”). Thus, there is no reason for
handleAsyncJobPublishEvent be a private method.
- Global variables and methods called just by this class (ApiServer)
were changed to private.
Changed variables and methods from static to non static (if possible):
As some variables/methods are used just by one object of this class
(instantiated by springer), they were changed to non static.
With that, calls from com.cloud.api.ApiServlet.ApiServlet() that used
static methods from ApiServer, was changed from
ApiServer.<staticMethodName> to _apiServer.<methodName> that refers to
the org.apache.cloudstack.api.ApiServerService interface. Thus, methods
com.cloud.api.ApiServer.getJSONContentType() and
com.cloud.api.ApiServer.isSecureSessionCookieEnabled() had to be
included in the interface (org.apache.cloudstack.api.ApiServerService,
interface implemented by class ApiServer).
However, com.cloud.api.ApiServer.isEncodeApiResponse() was keept static,
as its call hierarchy would have to be changed (more than planed for
this PR).
- org.apache.cloudstack.framework.jobs.JobCancellationException (removed
variable JobCancellationException in com.cloud.utils.SerialVersionUID)
- org.apache.cloudstack.ldap.NoSuchLdapUserException (removed test file
/cloud-plugin-user-authenticator-ldap/test/groovy/org/apache/cloudstack/ldap/NoSuchLdapUserExceptionSpec.groovy)
- com.cloud.agent.api.storage.CreateVolumeOVAAnswer
- com.cloud.exception.MissingParameterValueException
- org.apache.cloudstack.api.response.StatusResponse
- org.apache.cloudstack.api.response.VolumeDetailResponse
- org.apache.cloudstack.api.response.UpgradeVmResponse
- org.apache.cloudstack.api.response.AddIpToVmNicResponse
- org.apache.cloudstack.api.response.TemplateZoneResponse (at
org.apache.cloudstack.api.response.TemplateResponse, there is this
comment "To avoid breaking backwards compatibility, we still treat a
template at different zones as different templates, so not embedding
template_zone information in this TemplateZoneResponse set. `private
Set<TemplateZoneResponse> zones;`" but right now it is not used)
- org.apache.cloudstack.api.response.NicDetailResponse
CLOUDSTACK-9298: Improve performance of resource retrieval that have tags associated and target volumes, VMs and templatesJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9298
## Description of the problem
When retrieving a large number of resources which have tags associated with, retrieval methods took too long. Our goal is to improve performance of this methods avoiding query the database for each tag, managing that information in memory.
API methods to improve: <code>listTemplates</code>, <code>listVolumes</code>, <code>listVirtualMachines</code>
To achive it, it's necessary to include new columns in <code>template_view</code>, <code>volume_view</code> and <code>user_vm_view</code>:
* tag_account_name
* tag_domain_name
* tag_domain_uuid
* pr/1425:
CLOUDSTACK-9298: Remove user definer from view creations
CLOUDSTACK-9298: Add @MappedSuperClass support for persistence inheritance
CLOUDSTACK-9298: Improve ListTemplatesCmd, ListVolumesCmd and ListVMsCmd performance
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMfor xenserver,kvm and for vmware.
* pr/780:
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.
Signed-off-by: Kishan Kavala <kishan@apache.org>
Change variable "ROOK_DISK_CONTROLLER" to "ROOT_DISK_CONTROLLER"Change com.cloud.vm.VmDetailConstants variable name from "ROOK_DISK_CONTROLLER" to "ROOT_DISK_CONTROLLER". The ROOK_DISK_CONTROLLER variable stores the "rootDiskController" String; thus, this PR changes the name to "ROOT_DISK_CONTROLLER".
* pr/1434:
VmDetailConstants "ROOK_DISK_CONTROLLER" to "ROOT_DISK_CONTROLLER"
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
CLOUDSTACK-8850: revertSnapshot command does not workUpdating the documentation before the api is actually supported
* pr/831:
Updating the Description as only supported with KVM
CLOUDSTACK-8850: revertSnapshot command does not work Updating the documentation before the api is actually supported
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* 4.7:
CLOUDSTACK-9254: Make longer names display pretty
CLOUDSTACK-9245 - Deletes ACL items when destroying the VPC or deleting the ACL itself
CLOUDSTACK-9245 - Formatting NetworkACLServiceImpl class
CLOUDSTACK-9245 - Formatting VpcManagerImpl class
CLOUDSTACK-9245 - Formatting NetworkACLManagerImpl class
More VR performance!
