CLOUDSTACK-9535: [API] listVMSnapshots improvement### Improvements
- Include missing fields in response: `account`, `domain`, `domainid`, `zoneid`
- Display total count of snapshots, not depending on page size
### Example
After creating 2 vm snapshots for a given vm, and making this API call: `command=listVMSnapshot&listAll=true&virtualmachineid=c8531ef8-8502-4b42-b1c5-c52ace0e7801&_=1475516598524&pagesize=1&page=1` we get this response:
```
<listvmsnapshotresponse cloud-stack-version="4.9.1.0-SNAPSHOT">
<count>2</count>
<vmSnapshot>
<id>88f7416a-8799-4245-99c6-c707cfbe6f47</id>
<name>i-2-10482-VM_VS_20161003174340</name>
<state>Ready</state>
<description>2</description>
<displayname>testsnap2</displayname>
<zoneid>0d074f25-ed31-482f-8bc5-44c9314fc417</zoneid>
<virtualmachineid>c8531ef8-8502-4b42-b1c5-c52ace0e7801</virtualmachineid>
<parent>24e44fe5-5f2e-4d35-a8f8-109b644a04e0</parent>
<parentName>testsnap</parentName>
<current>true</current>
<type>Disk</type>
<created>2016-10-03T10:43:40-0700</created>
<account>admin</account>
<domainid>5a7ffa07-3fca-11e5-9c45-005056ad45b7</domainid>
<domain>ROOT</domain>
</vmSnapshot>
</listvmsnapshotresponse>
```
**NOTES:** As in `listVirtualMachines`, despite `pagesize=1`, `count` field shows total snapshots count for given vm. Also, `account`, `domain`, `domainid`, `zoneid` fields are listed
* pr/1702:
CLOUDSTACK-9535: [API] listVMSnapshots improvement
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9525 add support for windows 10 guest os in xenserver 6.5added guest_os and guest_os_hypervisor mappings for windows10 and xenserver 6.5.0
* pr/1698:
CLOUDSTACK-9525 add support for windows 10 guest os in KVM
CLOUDSTACK-9525 add support for windows 10 guest os in vmware ESX 6.0
CLOUDSTACK-9525 add support for windows 10 guest os in xenserver 6.5.0
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9513 Migrate transifex workflow and format to jsonMove the localization resource files from Java Properties format to JSON Key-Value format
Change the Transifex sync script to handle JSON resource files instead of Properties files
Update the README
Remove old version from the Transifex configuration file
Remove unused gen-l10n.py script and update the ui/pom.xml to remove the execution of this script
cc @rhtyd @jburwell
* pr/1699:
CLOUDSTACK-9513 Migrate transifex workflow and format to json
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Move the localization resource files from Java Properties format to JSON Key-Value format
Change the Transifex sync script to handle JSON resource files instead of Properties files
Update the README
Remove old version from the Transifex configuration file
Remove unused gen-l10n.py script and update the ui/pom.xml to remove the execution of this script
[4.9/lts] CLOUDSTACK-9364: Add Ubuntu 16.04 as a guest OSAdds Ubuntu 16.04 as a supported guest os. This allows users to select
the OS when creating a template etc.
SQL queries tested and adapted from https://issues.apache.org/jira/browse/CLOUDSTACK-9364
* pr/1696:
CLOUDSTACK-9364: Add Ubuntu 16.04 as a guest OS
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Adds Ubuntu 16.04 as a supported guest os. This allows users to select
the OS when creating a template etc.
Note: As XenServer 6.5 does not have 16.04 in its list of known Ubuntu releases,
as a workaround 16.04 guest os refers to 14.04 for XenServer 6.5.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Update L10N resource files with 4.10 strings from Transifex (20160925)
Please merge the #1619 before.
And please merge this current PR before 4.10.0.0 release.
cc @jburwell
* pr/1690:
Update L10N resource files with 4.10 strings from Transifex (20160925)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Add the Transifex config for next version of CS (4.10)
* pr/1619:
Add the Transifex config for next version of CS (4.10)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Make CloudStack JSP-freeCLOUDSTACK-9842: Make UI JSP Free
We use some JSP files mainly for translation of strings in the UI. This is
achievable purely in JavaScript. This removes those JSPs, simplifies
translation usage and workflow (purely JS based). The l10n js (dictionary)
files are generated from existing messages.properties files during client-ui
code generation phase.
/cc @jburwell @karuturi @abhinandanprateek @murali-reddy @wido @NuxRo @milamberspace @koushik-das and others
@milamberspace can our translation workflow/infra be changed to translate the JS files instead of the messages.properties file?
* pr/1669:
CLOUDSTACK-9842: Make UI JSP Free
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9505: Fix test_deploy_vgpu_enabled tests cleanupJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9505
## Description
Cleanup resources after running `test_deploy_vgpu_enabled.py`. Although test passes, resources are left and need to be cleaned up
* pr/1693:
CLOUDSTACK-9505: Fix test_deploy_vgpu_enabled tests cleanup
Signed-off-by: John Burwell <meaux@cockamamy.net>
Switched to the official SolidFire SDK for PythonSolidFire has recently released an official SDK for Python.
I have converted all of the SolidFire integration tests over to making use of this new SDK.
For testing, I re-ran each test and observed success.
https://pypi.python.org/pypi/solidfire-sdk-python/1.1.0.92
* pr/1689:
Switched to the official SolidFire SDK for Python
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
We use some JSP file just for translation of strings in the UI. This is
achievable purely in JavaScript. This removes those JSPs, simplifies
translation usage and workflow (purely JS based). The l10n js (dictionary)
files are generated from existing messages.properties files during client-ui
code generation phase.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9422: Granular 'vmware.create.full.clone' as Primary Storage setting### Introduction
For VMware, It is possible to decide creating VMs as full clones on ESX HV, adjusting `vmware.create.full.clone` global setting. We would like to introduce this property as a primary storage detail, and use its value instead of global setting's value.
We propose introducing `fullCloneFlag` on `PrimaryDataStoreTO` sent on `CopyCommand`. This way we can reconfigure `VmwareStorageProcessor` and `VmwareStorageSubsystemCommandHandler` similar as it was done for `nfsVersion` but refactoring it to be more general.
* pr/1602:
CLOUDSTACK-9422: Granular VMware vms creation as full clones on HV
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9480, CLOUDSTACK-9495 fix egress rule incorrect behaviorWhen 'default egress policy' is set to 'allow' in the network offering, any egress rule that is added will 'deny' the traffic overriding the default behaviour.
Conversely, when 'default egress policy' is set to 'deny' in the network offering, any egress rule that is added will 'allow' the traffic overriding the default behaviour.
While this works for 'tcp', 'udp' as expected, for 'icmp' protocol its always set to ALLOW. This patch keeps all protocols behaviour consistent.
Results of running test/integration/component/test_egress_fw_rules.py. With out the patch test_02_egress_fr2 test was failing. This patch fixes the test_02_egress_fr2 scenario.
-----------------------------------------------------------------------------------------------------
Test By-default the communication from guest n/w to public n/w is NOT allowed. ... === TestName: test_01_1_egress_fr1 | Status : SUCCESS ===
ok
Test By-default the communication from guest n/w to public n/w is allowed. ... === TestName: test_01_egress_fr1 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_1_egress_fr2 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_egress_fr2 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_1_egress_fr3 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_egress_fr3 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_1_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule without CIDR ... === TestName: test_06_1_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without CIDR ... === TestName: test_06_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without End Port ... === TestName: test_07_1_egress_fr7 | Status : EXCEPTION ===
ERROR
Test Create Egress rule without End Port ... === TestName: test_07_egress_fr7 | Status : SUCCESS ===
ok
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Delete Egress rule ... === TestName: test_09_1_egress_fr9 | Status : SUCCESS ===
ok
Test Delete Egress rule ... === TestName: test_09_egress_fr9 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_1_egress_fr10 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_egress_fr10 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_1_egress_fr11 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_egress_fr11 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_1_egress_fr12 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_egress_fr12 | Status : EXCEPTION ===
ERROR
Test Redundant Router : Master failover ... === TestName: test_13_1_egress_fr13 | Status : SUCCESS ===
ok
Test Redundant Router : Master failover ... === TestName: test_13_egress_fr13 | Status : SUCCESS ===
ok
-----------------------------------------------------------------------------------------------------
* pr/1666:
fix egress rule incorrect behavior
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9480: Egress Firewall: Incorrect use of Allow/Deny for ICMP
fix ensures, ICMP, TCP, UDP are handled similalry w.r.t egress rule action
CLOUDSTACK-9495: Egress rules functionalty broken when protocol=all specified
when protocol=all specified, CIDR was ignored. Fix ensures if CIDR is specified
its always used in configuring iptable rules
2 new test cased to test /32 CIDR
On snapshot backup, this converts the rbd raw format on disk to qcow2 for compression.
* pr/1645:
CLOUDSTACK-9461
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Export UUID for zone creation event completion.Action events have support for exporting their "first class entities" over the event bus, if it's turned on. This works mostly for BaseAsyncCmds and any synchronous command that operates on already-existing entities. They are populated by the ApiServer and found by ActionEventUtils.
Where it tends to falter is when dealing with synchronous creation commands. This commit makes createZone export the zone UUID over the event bus once the zone has been created.
* pr/1661:
Export UUID for zone creation event completion.
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Adding support for cross-cluster storage migration for managed storage when using XenServerThis PR adds support for cross-cluster storage migration of VMs that make use of managed storage with XenServer.
Managed storage is when you have a 1:1 mapping between a virtual disk and a volume on a SAN (in the case of XenServer, an SR is placed on this SAN volume and a single virtual disk placed in the SR).
Managed storage allows features such as storage QoS and SAN-side snapshots to work (sort of analogous to VMware VVols).
This PR focuses on enabling VMs that are using managed storage to be migrated across XenServer clusters.
I have successfully run the following tests on this branch:
TestVolumes.py
TestSnapshots.py
TestVMSnapshots.py
TestAddRemoveHosts.py
TestVMMigrationWithStorage.py (which is a new test that is being added with this PR)
* pr/1671:
Adding support for cross-cluster storage migration for managed storage when using XenServer
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9386: DS template copies dont get deleted in VMware ESXi with multiple clusters and zone wide storageJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9386
### Introduction
In some production environments with multiple clusters it was noticed that unused templates were consuming too much storage. It was discovered that template cleanup was not deleting marked templates on ESXi.
### Description of the problem
Suppose we have multiple clusters `(c1, c2,...,cN)` on a data center and template `T` from which we deploy vms on `c1.`
Suppose now that we expunge those vms, and there's no other vm instance from template `T,` so this was the actual workflow:
1. CloudStack marks template for cleanup after `storage.cleanup.interval` seconds, by setting `marked_for_gc = 1` on `template_spool_ref` table, for that template.
2. After another `storage.cleanup.interval` seconds a `DestroyCommand` will be sent, to delete template from primary storage
3. On `VmwareResource`, command is processed, and it first picks up a random cluster, say `ci != c1` to look for vm template (using volume's path) and destroy it. But, as template was on `c1` it cannot be found, so it won't be deleted. Entry on `template_spool_ref` is deleted but not the actual template on hypervisor side.
### Proposed solution
We propose a way to attack problem shown in point 3, by not picking up a random cluster to look for vm but using vSphere data center. This way we make sure vm template will be deleted in every case, and not depending on random cluster selection
* pr/1560:
CLOUDSTACK-9386: Find vm on datacenter instead of randomly choosing a cluster
Signed-off-by: Koushik Das <koushik@apache.org>
Added an additional JSON diff output to the ApiXmlDocReaderThe original TXT diff format is very hard to work with if you are trying to programmatically use the output to generate documentation. Previously, @pdion891 spent a lot of time trying to manually reformat this output in order for it to be represented in the Release Notes.
This pull request adds JSON as an output format as well as the existing TXT to simplify the ability for parsers and generators to use the output.
This new JSON format was used to generate the 4.9.0 release notes (that code will be contributed separately).
Old manually formatted output:
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.7.0/api-changes.html
New auto-formatted output using this new JSON output:
http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.9.0/api-changes.html
I have included the output of the `diff.txt` as well as the `diff.json` to this PR so you can see the resulting output of both file formats (diff between 4.8 and 4.9).
[diff.txt.zip](https://github.com/apache/cloudstack/files/432983/diff.txt.zip)
[diff.json.zip](https://github.com/apache/cloudstack/files/432982/diff.json.zip)
* pr/1658:
Added JSON diff output to the ApiXmlDocReader in additon TXT for parsability
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8751 minimise downtime of network when network update is in progress.more info in the bug link below.
https://issues.apache.org/jira/browse/CLOUDSTACK-8751
* pr/866:
Added license headders
Added a test to test update router in sequence
Do not update network if one of the router's state is unknown
CLOUDSTACK-8751 Added tests
CLOUDSTACK-8751
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
Added checks to prevent netwrok update when router state is unknown or when
the new offering removes a service that is in use.
Added a new param forced to the updateNetwork API. The network will
undergo a forced update when this param is set to true.
CLOUDSTACK-8751 Clean network config like firewall rules etc, when network services are removed during network update.
In comment, Add missing packages for Docker Ubuntu builds
Please note: No need to wait the results of Jenkins or Travis, this fix only add the missing packages in the docker command line to build CloudStack with Docker (for Ubuntu 14.04 and 16.04)
cc @wido
* pr/1672:
In comment, Add missing packages for Docker Ubuntu builds
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9428: Fix for CLOUDSTACK-9211 - Improve performance of 3D GPU support in cloud-plugin-hypervisor-vmwareJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9428
### Introduction
On #1310 passing vRAM size to support 3D GPU problem was addressed on VMware. It was found out that it could be improved to increase performance by reducing extra API calls, as we'll describe later
### Improvement
On WMware, `VmwareResource` manages execution of `StartCommand.` Before sending power on command to ESXi hypervisor, vm is configured by calling `reconfigVMTask` web method on vSphere's client `VimPortType` web service.
It was found out that we were using this method 2 times when passing vRAM size, as it implied creating a new vm config spec only editing video card specs and making an extra call to `reconfigVMTask.`
We propose reducing the extra web service call by adjusting vm's config spec. This way video card gets properly configured (when passing vRAM size) in the same configure call, increasing performance.
### Use case (passing vRAM size)
* Deploy a new VM, let its id be X
* Stop VM
* Execute SQL, where X is vm's id and Z is vRAM size (in kB):
````
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'mks.enable3d', 'true');
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'mks.use3dRenderer', 'automatic');
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'svga.autodetect', 'false');
INSERT INTO cloud.user_vm_details (vm_id, name, value) VALUES (X, 'svga.vramSize', Z);
````
* Start VM
* pr/1605:
CLOUDSTACK-9428: Add marvin test
CLOUDSTACK-9428: Fix for CLOUDSTACK-9211 - Improve performance
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
* 4.9:
CLOUDSTACK-9470: Fix for SshHelper - test_network_acl was failing on Vmware due to a bug in sshExecute, in which value returned was null and there was still stdout to consume. This fix addresses this problem, consuming stdout peoperly to return expected value in sshExecute
Conflicts:
engine/schema/src/com/cloud/upgrade/DatabaseUpgradeChecker.java
engine/schema/test/com/cloud/upgrade/DatabaseUpgradeCheckerTest.java
tools/marvin/setup.py
This fixes class names to make things consistent as per the 4.9 PR on master.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Changes database upgrade script names to be consistent for the 4.9.1.0 release * Changes the names of the schema-490to491* scripts to
schema-490to4910*
* Changes the name of the Upgrade490to491 class to Upgrade490to4910
* Modifies the Marvin setup.py script to use version 4.9.1.0-SNAPSHOT
/cc @rhtyd @karuturi
* pr/1665:
Renames of 4.9.0->4.9.1.0 upgrade scripts to match the four position version scheme
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Renames schema-490to491*.sql to schema490to4910*.sql
* Renames the Upgrade490to491 class to Upgrade490to4910
* Removes the unused s_logger contant from Upgrade490to4910
* Updates the version in tools/marvin/setup to 4.9.1.0-SNAPSHOT
