32040 Commits

Author SHA1 Message Date
Anurag Awasthi
9e6ee7c63e server: Add support for new heuristics based VM Deployement for admins (#3454)
Currently an admin can choose which host a VM is to be started on.
They should be able to 'override' the allocation algorthm to a greater
or lesser extent at will, and be able to choose the pod, cluster or host
that they wish a new VM to be deployed in.

DeployVirtualMachine API has been extended with additional, optional
parameters podid and clusterid that will be passed to and used in the
deployment planner, when selecting a viable host. If the user supplies
a pod, a suitable host in the given pod will be selected. If the user
supplies a cluster, a suitable host in the given cluster will be selected.

Based on the parameter supplied and on passing validation, the VM will
then be deployed on the selected host, cluster or pod.
2019-07-13 12:52:48 +05:30
Rohit Yadav
97df52956c
master: travis and trillian smoketests fixes and stabilisation (#3476)
Fix failing test, add more component tests, optimise and refactor test jobs.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-12 17:27:49 +05:30
Abhishek Kumar
b40fdbb2f7 api: fix account deletion event description (#3483)
Fix account deletion event description.
Fixes #3109

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2019-07-12 13:10:30 +05:30
Sid Kattoju
1bf4dd39ca kvm: use IDE as the bus type for root disks and VIRTIO for data disks on platforms without support for para virtualization when using managed storage (#3319)
This change addresses #3089. There was an issue when disks were being added with bus type IDE when creating windows VMs from ISOs. It is not possible to select bus type when creating a VM with an ISO. The bus type is inferred based on the platform emulator string provided to the KVM agent. Currently when creating a VM with managed storage (ex: Solidfire) and OS type string Windows*, all disks are added as IDE. Qemu currently does not support multiple IDE controllers and this configuration results in VMs that cannot be started. This issue does not occur when using NFS as the storage provider due to logic in that KVM agent that makes all data volumes (non root) use a virtio controller for file based disk. Similar logic was added for raw physical disks so that managed storage has the same behavior as NFS. In addition specific versions were removed from the code that guesses the disk controller to be used based on the platform emulator string since most modern operating systems support virtio.

Fixes #3089
2019-07-12 12:35:23 +05:30
Abhishek Kumar
5df11b5c43 ui: fix custom offerings selection in upload volume form (#3479)
Removed duplicate code for diskOffering field.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2019-07-11 15:14:53 +05:30
Wei Zhou
b7988a3e5f vr: Fix vpc router in UNKNOWN state (#3465)
If there are more than 10 vpc tiers or public ip subnets in a VPC, eth1X will be added in vpc router.
The redundant state is UNKNOWN in this case.
2019-07-11 01:48:37 +05:30
Spaceman1984
b509e086f9 api: Allowing template owner to download template (#3475)
Removed the download icon when a template is not extractable.

Modified the api to allow a user from the same account as the template, to change the extractable attribute on the template.

Fixes #3400
2019-07-09 12:42:41 +05:30
Anurag Awasthi
c88540d5a5 framework/db: Fix bug in counting items for search query (#3457)
The implementation of GenericBaseDao#searchAndCount() can result in
incorrect count. This happens because the implementation of the
GenericBaseDao#getCount method excludes the groupBy components of the
search queries. This means the count returned will always be larger than
the actual count when not considering pagination.

The change was brought in b0ce8fd which also fails to explain the rationale.

Further investigation of the getCount usage reveal they are always
accompanied by search queries which include groupBy components via
GenericBaseDao#searchIncludingRemoved method.

```
Current code diff between search and count methods is as follows -
diff --git a/framework/db/src/main/java/com/cloud/uddtils/db/GenericDaoBase.java b/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java
@@ -371,7 +371,7 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone
             clause = null;
         }

-        final StringBuilder str = createPartialSelectSql(sc, clause != null, enableQueryCache);
+        final StringBuilder str = createCountSelect(sc, clause != null);
@@ -384,19 +384,12 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone
             }
         }

-        List<Object> groupByValues = addGroupBy(str, sc);
-        addFilter(str, filter);
-
+        // we have to disable group by in getting count, since count for groupBy clause will be different.
+        //List<Object> groupByValues = addGroupBy(str, sc);
         final TransactionLegacy txn = TransactionLegacy.currentTxn();
-        if (lock != null) {
-            assert (txn.dbTxnStarted() == true) : "As nice as I can here now....how do you lock when there's no DB transaction?  Review your db 101 course from college.";
-            str.append(lock ? FOR_UPDATE_CLAUSE : SHARE_MODE_CLAUSE);
-        }
-
@@ -410,20 +403,19 @@ public abstract class GenericDaoBase<T, ID extends Serializable> extends Compone

+            /*
             if (groupByValues != null) {
                 for (Object value : groupByValues) {
                     pstmt.setObject(i++, value);
                 }
             }
+             */

-            if (s_logger.isDebugEnabled() && lock != null) {
-                txn.registerLock(pstmt.toString());
-            }
             final ResultSet rs = pstmt.executeQuery();
             while (rs.next()) {
-                result.add(toEntityBean(rs, cache));
+                return rs.getInt(1);
             }
-            return result;
+            return 0;
--
2.17.1
```

The fix is to update the way we setup query for counting with group by
params.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-08 17:05:47 +05:30
Richard Lawley
d70f574a7e plugins: fix removing SRX port forwarding rules, improve add/remove logic (#3393)
This PR partially fixes the logic around port forwarding rules on the Juniper SRX plugin. The code in the plugin is based on JunOS 10, which is very old. The changes here should not break compatibility, but should enable the plugin to be used on newer devices. Note that an additional change to a script file is required to be able to add port forwarding rules, but as this PR was targetted for 4.11.3, I thought it best not to include this change as it might break compatibility for anyone still using JunOS 10.

I've made the logic better and consistent for adding/removing static nat and port forwarding rules - these were multi-step processes which did not check each individual step. This would aid in manually fixing rules in case of further problems.

I've also improved the logging for communication with the SRX by stripping out the Apache header before sending it, and indicating the name of the template filename in use.

To be able to add port forwarding rules, the <dst-port> tags in dest-nat-rule-add.xml must be changed to <low>.

Fixes: #3379
2019-07-08 15:46:12 +05:30
Rohit Yadav
e4ddee6fb9
network: allow icmp code 16 in firewall rules (#3468)
This allows for icmp code range 0-16.

Type 9 Router advertisement reference:
https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml#icmp-parameters-codes-9/#table-icmp-parameters-ext-classes

Fixes #3349

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-05 23:10:02 +05:30
Anurag Awasthi
f1614aa7c2 api: allow copy tags from template/iso image to VM via deployVM API (#3297)
Support copy tags from template/iso image to VM from deploy vm command. Allow creation of tags from the source template/iso image to vm when deploy vm command creates virtual machine.

Fixes: #3048

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-05 17:08:25 +05:30
Rohit Yadav
c93630f125
travis: use explicit change directory and use -pl to build rat check (#3472)
This tries to fix build failures seen in job 1 of Travis. Also fixes a pylint issue.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-05 15:47:44 +05:30
Rohit Yadav
7ed3dc33b3
schema: add support for XenServer 7.1.2 (LTS) (#3467)
Fixes #3461

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-05 13:41:17 +05:30
HuaDream
7d90652dfa quota: fix issue of QuotaType name (#3463)
Fix issue of QuotaType name.
This bug causes Quota - Summary > Statement to show the incorrect name of VM_DISK_BYTES_WRITE.
2019-07-05 13:40:17 +05:30
Wei Zhou
6780930ecc server: Add option 'details' to listProjects and listAccounts (#3331)
If there are many projects and accounts, listing projects/accounts will take long time getting the resource limitation and resource count in the process. However resource count/limitation are not needed sometimes.

Add an option 'details' to listProjects and listAccounts. If you do not need the resource count/limitation, please add details=min to api call. The api execution time will be reduced significantly.
2019-07-04 16:26:04 +05:30
Anurag Awasthi
a0097d83da engine/schema: count Starting along with Running VMs for user dispersing planner (#3462)
Consider running and starting VMs when considering load ona host for VM deployement for more accurate dispersion.

Fixes: #3442
2019-07-04 16:21:16 +05:30
Rohit Yadav
92773ea9fd Merge remote-tracking branch 'origin/4.12' 2019-07-04 15:32:53 +05:30
Rohit Yadav
47f5c656ac Merge remote-tracking branch 'origin/4.11' into 4.12
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-04 15:32:01 +05:30
Rohit Yadav
35b914570d
travis: use openjdk8 and xenial (ubuntu 16.04) (#3466)
This fixes Travis to use openjdk8 and Xenial (Ubuntu 16.04) as base
distro.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-04 15:30:18 +05:30
EK
9f18b5b620 ui: allow for the VM Hostname to be edited when VM is switched off (#3412)
Allow for VM name (hostname) to be edited when VM is switched off.
2019-07-04 14:49:32 +05:30
smlshn
f3aa147806 ui: sort list of templates, serviceOfferings, diskOfferings etc in the deploy VM wizard (#3336)
Adds functionality to sort the data that is available on each in the deploy VM wizard's step by their suitable fields:

affinityGroups by name
sshkeyPairs by name
vpcs by name

Fixes: #3050
2019-07-04 14:29:17 +05:30
Rohit Yadav
75e4882f32 Merge remote-tracking branch 'origin/4.12' 2019-07-04 14:01:26 +05:30
Rohit Yadav
f0d7942162 Merge remote-tracking branch 'origin/4.11' into 4.12
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-04 13:56:24 +05:30
Paul Angus
b72aadb65d Updating pom.xml version numbers for release 4.11.4.0-SNAPSHOT
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>
2019-07-03 09:28:09 +01:00
Paul Angus
f855e6d3b9 Merge branch '4.11.3.0-RC20190625T2045' into 4.11 2019-07-03 09:10:24 +01:00
Wei Zhou
452b48ea0c server: reduce execution time while listing project if projects have many resource tags (#3306)
If projects have many resource tags, it will take a long time to list projects.
Remove resource tags information from project_view will fix it the issue.

Fixes #3178
2019-07-03 11:41:53 +05:30
Nicolas Vazquez
4c28a2bcaa ui: Fix wrap text for accounts on project view (#3451)
Fix wrap text for account names on the project view, as it cannot be fully displayed.
2019-07-02 17:28:28 +05:30
Wei Zhou
047003315b utils: reverse ip addresses of a nic returned by java to get the first ip address (#3449)
Java methods getInterfaceAddresses() returns ip addresses in reverse order as "ip addr show"
If there are multiple IPs assigned to a management interface, the last ip will be used as management ip in cloudstack. We need to reverse the ip addresses to get the first ip that makes more sense.

Fixes #3311
2019-07-02 13:16:18 +05:30
Anurag Awasthi
4185452366 ui: Fix sorting bug in UI Code (#3445)
Current master has sorting broken and the order reverses as opposed to
what is set if sortkey.algorithm is set to true. The problem lies in the
way the update APIs were being called by the UI. The code was agnostic
to a global config that backend uses to set the order of the entities
in the corresponding list APIs.

We need to make UI aware of the global config and instead of changing
sign of sort key pass proper numbers so that DB isn't confusing when
some tables have positive sortkey and some have negative.

The fix is in 2 steps-
1) Make use of sortkey name in place where it's relevant. Mere row index
is not sufficient.
2) Reverse the sortkey if we are sorting by descending (when global
config is false)
2019-07-02 12:53:10 +05:30
Anurag Awasthi
14bf2e98f3 ui: Don't pass account name when in project mode (#3435)
When users try to list instances in project mode from Account > SSHKeyPairs > Key > View Instances, we don't need to include sshkeypairs account name into the API Param. It's redundant and causes error.

Fixes #2895
2019-07-02 12:51:44 +05:30
Rohit Yadav
2ecd5ec804
systemvm: don't fork to curl to save password (#3437)
This fixes to avoid forking curl to save password but instead call
a HTTP POST url directly within Python code. This may reduce bottleneck
during high VM launches that require passwords.

Fixes #3182

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-01 23:53:44 +05:30
Rohit Yadav
4449556aba Merge remote-tracking branch 'origin/4.12' 2019-07-01 14:38:30 +05:30
Rohit Yadav
0649821a21 Merge remote-tracking branch 'origin/4.11' into 4.12
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-07-01 14:37:40 +05:30
Wei Zhou
9ac32aae38 server: VPC redundant vrs run on same hypervisor (#3421)
For VPC supports redundant VRs, when start the second VR, the pod/cluster/host of first VR should be added to avoid list. This provides higher availability.

The network VRs have the same process already.
2019-06-29 12:09:36 +05:30
Rohit Yadav
58c18fc09f
debian: install file as cloudstack-management dependency (#3436)
This install `file` package that provides the `file` tool used by
cloud-install-sys-tmplt script, as part of cloudstack-management
package installation on Ubuntu distros.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-29 00:24:10 +05:30
Rohit Yadav
c160e7532a
ui: fix LB protocol bug (#3438)
Fixes #3252

This fixes labels for missing LB protocol(s).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-29 00:23:36 +05:30
Anurag Awasthi
541d280e3b ui: Bug fix for distinguishing between string and map type tags in forms (#3441)
Some APIs consume 'tags' param as string and some consume as maps.
Since each API can have at most one 'tags' param the extraction of
map based tags should only happen when strings based tags are not
extracted from the form serialization.
2019-06-28 17:05:24 +05:30
Philipp Bankonier
044e4e4b21 ui: re-add custom css (#3431)
This readds the custom.css file to overwrite the default cloudstack styles.
Currently this is only possible by adding them to the custom.scss in the repository and compile them into the cloudstack3.css.
Now there is an empty custom.css created when compiling the scss source. This custom.css can be overwritten with the custom styles afterwards.

This functionality was broken by #3328.
2019-06-27 19:02:40 +05:30
Andrija Panic
4ce0562cff
Merge pull request #3432 from shapeblue/config_another_cleanup
Remove additional line from config as storagepool isn't available for users
2019-06-27 13:28:08 +02:00
Anurag Awasthi
d967993a0a Remove additional line from config as storagepool isn't available for users 2019-06-27 16:53:37 +05:30
Rohit Yadav
23a8b74e4f
engine/schema: add guest-os support and mappings for XenServer 7.6 (#3427)
This adds hypervisor capabilities and mappings for XenServer 7.6

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-27 16:13:21 +05:30
Anurag Awasthi
95a509a1ed ui: Update config file with tables that users can see (#3429)
This is a purely documentation change of config file.
The cleanup was needed post merge of PR #3258
2019-06-27 12:51:29 +05:30
Abhishek Kumar
58474530f6 api: snapshot, snapshotpolicy tag support (#3228)
Problem: Currently tags cannot be applied to snapshot when it is being created but through separate “create tags” API calls. For snapshot policies tags cannot be set either at creation or through “create tags” API.

Root Cause: The “create snapshots” API does not support adding tags during creation and it can only be done through “create tags” API. Snapshot policy as a resource does not support tags and no tags can be set for them through any API.

Solution: Tag support for snapshot policy has been added. Snapshot policy with tags when executed will produce snapshots containing the same tags from snapshot policy.

Following APIs have been updated:

Both “create snapshotpolicy” and “create snapshot” now accepts “tags” as a new parameter. The expected format for “tags” parameter is similar to parameter “tags” in “create tags“ API.
Deletion support for tags associated with snapshots policy has been added to “delete snapshotpolicies” API.
Tags set for snapshot policies are added to the Response of “list snapshotpolicies“ API.
UI support for setting tags to snapshots and snapshot policy is provided through the corresponding menus with a new section in each form to set tags.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2019-06-27 09:21:08 +05:30
Rohit Yadav
14bff7bd03
server: export granular volume bytes and iops metrics (#3259)
Problem: The VM metrics has aggregated volume bytes read/write and iops metrics but not on per volume basis.
Root Cause: The volume stats sub-system is not used to export the metrics, the support is not available for VMware.
Solution: Use the volume stats sub-system and DB table to export the metrics via the listVolumes and listVolumeMetrics API, and implement support for VMware and fix issue with network and disk metrics in the VM metrics view.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-27 09:18:10 +05:30
Rohit Yadav
9f4f2c5348
api: instance and template details are free text (#3240)
Problem: Users don't know what keys/values to enter for template and VM details.
Root Cause: The feature does not exist that can list possible details and options.
Solution: Based on the possible VM and template details handled by the
codebase, those details were refactored and a list API is introduced
that can return users those details along with possible values. When
users add details now, they will be presented with a list of key details
and their possible options if any.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-27 09:14:47 +05:30
EK
f9998e418c server: warn on migration of volumes within the same storage pool (#3424)
Added an if statement that catches the destination poolid as an invalid parameter if it is similer to the current poolid. 

Fixes #3291
2019-06-27 09:06:27 +05:30
Rohit Yadav
2c3c88e209
console-proxy: fix potential NPE condition (#3419)
When checking if the console proxy URL domain starts with *, the code
does not check if the provided string is null. When domain is not
configured the IP address should be used.

Fixes #3164

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-06-26 23:18:20 +05:30
Rohit Yadav
6784cc516b Merge remote-tracking branch 'origin/4.12' 2019-06-26 16:29:49 +05:30
Rohit Yadav
a2323e1425 Merge remote-tracking branch 'origin/4.11' into 4.12 2019-06-26 16:27:29 +05:30
ustcweizhou
2408783355 ssvm: use secstorage.ssl.cert.domain as hostname if it does not start with '*' when upload a template or volume from local (#3420)
If secstorage.ssl.cert.domain does not start with '*', we should use it as host name in url when upload template/volume from local

Fixes #3305
2019-06-26 16:27:07 +05:30