31711 Commits

Author SHA1 Message Date
Rafael Weingärtner
077e692dbb Forward merge Fix DirectNetworkGuru canHandle for lowercase isolation methods
forward merge of #3010 -- Fix DirectNetworkGuru canHandle checks for lowercase isolation methods
2018-11-07 09:56:05 -02:00
Nicolas Vazquez
af0c1e48cf Fix DirectNetworkGuru canHandle checks for lowercase isolation methods (#3010) 2018-11-07 09:53:01 -02:00
Rafael Weingärtner
e175e93281
Fix set affinity for user overlay problem (Fixes #2986) (#2987)
This PR fixes #2986
2018-11-01 10:30:53 -03:00
Yoan Blanc
17c164d59a api: signature v3 to accept more formats (#2893)
It does it by reusing the DateUtil helpers. DateUtil uses java.time.* as that one knows how to deal
with timezones correctly.

The format expected by signatureVersion=3&expires=.... is quite limited.

It should accept the following formats that are containing a timezone and/or milliseconds.

2018-10-01T08:12:14Z
2018-10-01T08:12:14+01:00
2018-10-01T08:12:14+0100
2018-10-01T08:12:14.000Z
2018-10-01T08:12:14.000+01:00
2018-10-01T08:12:14.000+0100
afaik only 2018-10-01T08:12:14+0100 is accepted by the current codebase.

This PR echoes other pull requests I made earlier this year. #2392 and #2867

Signed-off-by: Yoan Blanc <yoan.blanc@exoscale.ch>
2018-10-31 16:57:48 +05:30
Rohit Yadav
7479e2877f Merge remote-tracking branch 'origin/4.11' 2018-10-30 15:15:35 +05:30
Rohit Yadav
c6e53f6cc6
kvm: reset KVM host on heartbeat failure (#2984)
On actual testing, I could see that kvmheartbeat.sh script fails on NFS
server failure and stops the agent only. Any HA VMs could be launched
in different hosts, and recovery of NFS server could lead to a state
where a HA enabled VM runs on two hosts and can potentially cause
disk corruptions. In most cases, VM disk corruption will be worse than
VM downtime. I've kept the sleep interval between check/rounds but
reduced it to 10s. The change in behaviour was introduced in #2722.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-30 15:13:59 +05:30
Marc-Aurèle Brothier
3101612ad1 docker: add compose file for database (#2898)
A docker-compose file to setup quickly the database for CS. This helps during local development and remove the need to install a local MariaDB/MySQL.
2018-10-29 21:25:48 +05:30
Gabriel Beims Bräscher
cdc6e6e50a CLOUDSTACK-9315: Removed unused Classes (#1448)
* Remove some unused Classes

These classes were deleted because they have no references in our code base. They are not in Spring execution flow nor instantiated with "new":
- com.cloud.agent.api.CheckStateAnswer
- com.cloud.agent.api.StartupVMMAgentCommand
- com.cloud.agent.api.routing.UserDataCommand
	- remove from description at
com.cloud.configuration.Config.ExecuteInSequenceNetworkElementCommands
enum
- com.cloud.agent.api.storage.UpgradeDiskCommand
- com.cloud.agent.api.storage.CreatePrivateTemplateCommand
- com.cloud.agent.api.storage.DestroyAnswer
	- Note: "FIXME: Should have an DestroyAnswer" at
com.cloud.storage.resource.StoragePoolResource
- com.cloud.agent.api.storage.UpgradeDiskAnswer
- com.cloud.agent.api.storage.ManageVolumeAvailabilityAnswer
- com.cloud.agent.api.storage.ManageVolumeAvailabilityCommand
- com.cloud.exception.UsageServerException
- com.cloud.info.SecStorageVmLoadInfo
- com.cloud.serializer.SerializerHelper

* PR#1448 update description of 'execute.in.sequence.network.element.commands' param

Update description of 'execute.in.sequence.network.element.commands'parameter to reflect an unused command that has been removed. The removed class command is 'UserDataCommand'.

* Add cloud schema to update SQL
2018-10-29 08:27:00 -03:00
Rohit Yadav
323d381767 Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:27:08 +05:30
Rene Diepstraten
f7bc5807a3 vr: defer was broken in VR because of json name change
Committed at f0491d5c72c3161777ca49ae809606a6704df5ff (#2979).

After upgrade from CS 4.10 to CS 4.11, multiple VRs did not start through.
It did not properly defer the finalize config in update_config.py.
Apparently, the json files are now called differently: where it used to
be vm_dhcp_entry.json it now has a uuid added, for example
vm_metadata.json.4d727b6e-2b48-49df-81c3-b8532f3d6745.
The if statement that checks if the finalize can be safely deferred
therefore no longer matches. This PR contains a fix so finalize is
defered again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:19:33 +05:30
Nicolas Vazquez
dffb430975 kvm: Fix migrating VM from ISO failures (#2928)
Prevents errors while migrating VM from ISO:

Test 1: Deploy VM from ISO -> Live migrate VM to another host -> ERROR
Test 2: Register ISO using Direct Download on KVM -> Deploy VM from ISO -> Live migrate VM to another host -> ERROR

- Prevent NullPointerException migrating VM from ISO
- Prevent mount secondary storage on ISO direct downloads on KVM
2018-10-29 16:14:20 +05:30
Rohit Yadav
f0491d5c72
vr: defer was broken in VR because of json name change (#2979)
After upgrade from CS 4.10 to CS 4.11, multiple VRs did not start through.
It did not properly defer the finalize config in update_config.py.
Apparently, the json files are now called differently: where it used to
be vm_dhcp_entry.json it now has a uuid added, for example
vm_metadata.json.4d727b6e-2b48-49df-81c3-b8532f3d6745.
The if statement that checks if the finalize can be safely deferred
therefore no longer matches. This PR contains a fix so finalize is
defered again.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:11:43 +05:30
Rohit Yadav
8738ca75b1
kvm: use libvirtd as service name for all distributions (#2925)
Since we support only Ubuntu 16.04+ on master/4.12+, we can now use
the libvirt service name `libvirtd` for all distributions. This also
fixes an optional package name for libvirtd installation on Debian 9+.
Fixes #2909

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 11:55:28 +05:30
Rohit Yadav
e2ba934c19
server: fix unwanted txn commit warning messages (#2927)
This fixes unwanted transaction commit warning messages such:

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 02:49:54 +05:30
Gabriel Beims Bräscher
bfc326384d saml: redirect saml2 failed login message to a configurable URL (#2185)
When a user fails to authenticate with SAML2, it returns an error page
showing the content of the attached image.

To make it more user-friendly and customizable, one could configure a
desirable URL to redirect when such authentication failure happens.

This ticket proposes a global settings variable
(saml2.failed.login.redirect.url). If null, the SAML2 authentication
flow does not change from the current; however, if the user configures
an URL then ACS redirects to that URL.
2018-10-28 00:55:06 +05:30
alexanderbazhenoff
a87acf93d8 kvm: improved performance on creating VM (#2923)
Improved performance on creating VM for KVM virtualization.

On a huge hosts every "ifconfig | grep" takes a lot of time (about 2.5-3 minutes on hosts with 500 machines). For example: ip link show dev $vlanDev > /dev/null is faster than ifconfig |grep -w $vlanDev > /dev/null. But using ip command is much better. Using this patch you can create 500s machine in 10 seconds. You don't need slow ifconfig prints anymore.
2018-10-25 16:28:13 +05:30
Rohit Yadav
9cf57d2568
network: on rolling restart force stop old routers (#2926)
This force stops old VRs when performing rolling restart with
cleanup=true. This will ensure that VRs are powered off quickly than
wait longer for the normal ACPI shutdown. During testing, it was found
on VMware where VM stops are slow compared to XenServer and KVM.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-25 09:20:39 +05:30
Rohit Yadav
b8ed159f47 Merge remote-tracking branch 'origin/4.11' 2018-10-25 08:14:49 +05:30
Simon Weller
c4b621a418 kvm: HyperV Enlightment for Improved Windows Server 2008+ Performance (#2870)
Windows has support for several paravirt features that it will use when running on Hyper-V, Microsoft's hypervisor. These features are called enlightenments. Many of the features are similar to paravirt functionality that exists with Linux on KVM (virtio, kvmclock, PV EOI, etc.)

Nowadays QEMU/KVM can also enable support for several Hyper-V enlightenments. When enabled, Windows VMs running on KVM will use many of the same paravirt optimizations they would use when running on Hyper-V.

A number of years ago, a PR was introduced that added a good portion of the code to enable this feature set, but it was never completed. This PR enables the existing features. The previous patch set detailed in #1013 also included the tests.

By selecting Windows PV, the enlightenment additions will be applied to the libvirt configuration. This is support on Windows Server 2008 and beyond, so all currently supported versions of Windows Server.

In our testing, we've seen benchmark improvements of around 20-25% running on Centos 7 hosts and it is also supported on Centos/RHEL 6.5 and later. Testing on Ubuntu would be appreciated.
2018-10-25 06:54:13 +05:30
Rohit Yadav
9b35b64b3c
packaging: install plugins at /usr/share/cloudstack-management/lib (#2915)
Install any additional plugin jars in the lib directory to be picked up
by the classpath builder, otherwise one has to manually add the jar
to /etc/default/cloudstack-management after installation. This fixes
the issue for `mysql-ha` plugin.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-24 18:21:03 +05:30
Abhishek
6f44a8d7ba template: Return requireshvm key as part of list template response (#2921)
Added requireshvm key in the response of list template API.
Fixes #2821

Signed-off-by: Abhishek Kumar <abhishek.mrt@gmail.com>
2018-10-24 16:20:27 +05:30
Rohit Yadav
8d31024a60 Merge remote-tracking branch 'origin/4.11' 2018-10-24 11:08:00 +05:30
Abhishek
c8ca9e2fa0 template: create/updateTemplate should allow to set/change sshKeyEnabled (#2922)
Added sshKeyEnabled key in createTemplate and updateTemplate API.
Fixes #2822.

Signed-off-by: Abhishek Kumar <abhishek.mrt@gmail.com>
2018-10-24 11:07:03 +05:30
Rohit Yadav
e092529c98
systemvm: Ensure cloud service reboots after failure (#2916)
This fixes an issue for systemvms (CPVM and SSVM) on VMware, as eth0
is not programmed (link-local) the networking.service fails to start
which is a dependency for cloud-postinit service. When cloud-postinit
service fails to start/run, it fails to start the agent (cloud) process.
This fixes the smoketest failures we saw in case of VMware 6.5 with
4.11.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-23 23:33:08 +05:30
Rafael Weingärtner
71e5a88fc1 Forward merge client: mgmt server listen default to 0.0.0.0 (#2907) 2018-10-22 16:32:06 -03:00
Abhishek
d0fbd00e9b tools: Added '-B' flag for mvn current version cmd (#2910) (#2914)
maven command for finding current version might need to download packages and without batch mode it will end in variable. Added '-B' to enable batch mode.
2018-10-22 22:04:42 +05:30
Rohit Yadav
47c9c1cb58
client: mgmt server listen default to 0.0.0.0 (#2907)
This makes the management server listen on all interfaces by default.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-22 20:00:51 +05:30
Abhishek
f5d52421b9 kvm: Remove libvirt-bin check for Debian (#2909) (#2912)
Debian does not have libvirt-bin package. Therefore, only for Ubuntu host distro process name libvirt-bin is used, otherwise libvirtd will be used
2018-10-22 17:48:11 +05:30
Nicolas Vazquez
5cf163d888 server: Unify templates/ISOs checksum API output (#2911)
Unify checksum API output for templates and ISOs: not list the checksum algorithm on:
KVM direct downloads

On in progress normal template downloads. The algorithm is shown on the listtemplates API, but after it is downloaded it is not shown anymore.
2018-10-21 22:33:04 +05:30
Rohit Yadav
233f46c94b Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-17 20:29:58 +05:30
Rohit Yadav
5ce14df31f
network: Allow ability to disable rolling restart feature (#2900)
This adds a global setting for admins who may not want the rolling
restart of routers or are seeing any issues around it. In future, this
setting may be removed.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-17 20:27:08 +05:30
Rohit Yadav
e871638dc8 Merge remote-tracking branch 'origin/4.11' 2018-10-17 13:25:06 +05:30
Rohit Yadav
1904a70512
agent: on shutdown don't allow server reconnection (#2904)
When agent is stopped, don't allow reconnection. Previously this would
send a shutdown command to the management server which would put the
host state to Disconnected but then agent's reconnection logic may kick
in sometimes which would connect the agent to the management server
but then the agent process would terminate causing the host to be
put in Alert state (due to ping timeout or it waiting too long).

This fixes the issue by ensuring that when the agent is stopped, it
does not reconnect to the management server.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-17 06:31:13 +05:30
Gabriel Beims Bräscher
143fe5375c Allow enable the Java remote Debug for CloudStack Agent (#2897)
The idea is to create a property in /etc/defaults/cloudstack-agent that
can be enabled/disabled (uncommented/commented) to control the enabling
of Java remote debug in CloudStack's JVM.
2018-10-16 15:57:13 -03:00
Rafael Weingärtner
92f8cda64a Merge forward #2902 (Add checksum sanity validation on template registration) 2018-10-16 10:25:50 -03:00
Nicolas Vazquez
9003c7bfdc Add checksum sanity validation on template registration (#2902)
* Add checksum sanity validation on template registration

* Refactor

* Rename checksum sanity method
2018-10-16 10:21:20 -03:00
Pierre-Yves Ritschard
58b4e71b0e
params: allow signatureversion and expires without logging
This patch considers the new expires and signatureversion parameters
valid. Without this, all calls log when using the V3 signature scheme.
2018-10-16 09:01:46 +02:00
Rohit Yadav
84994c841f Merge remote-tracking branch 'origin/4.11' 2018-10-16 10:54:39 +05:30
Nicolas Vazquez
11d83fab43 agent: set log level to INFO as default for http wire (#2903)
Avoid logging bytes on direct download on KVM.
2018-10-16 10:32:03 +05:30
Rohit Yadav
933ee23104
vr: memory and swap optimizations (#2892)
This tries to provide a threshold based fix for #2873 where swappinness of VR is not used until last resort. By limiting swappiness unless actually needed, the VR system degradation can be avoided for most cases. The other change is around not starting baremetal-vr by default on all VRs, according to the spec https://cwiki.apache.org/confluence/display/CLOUDSTACK/Baremetal+Advanced+Networking+Support only vmware VRs need to run it and that too only as the last step of the setup/completion, so we don't need to run it all the time.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-16 10:29:48 +05:30
Rafael Weingärtner
272167c3b3 merge forward #2876 -- PULL_REQUEST_TEMPLATE simplification 2018-10-15 15:33:13 -03:00
Rohit Yadav
63f4d852d5 PULL_REQUEST_TEMPLATE: simplify and remove unpopular sections (#2876)
This removes the section from the pull request template that is not very
popular or filled by the PR author.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-15 15:21:13 -03:00
Rafael Weingärtner
ebc6b2f706 Remove push force test file 2018-10-13 09:40:23 -03:00
Rafael Weingärtner
1758a75a54 test force push 2018-10-12 10:58:41 -03:00
Rohit Yadav
bd9880003f Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-10 16:10:26 +05:30
Rohit Yadav
ea771cfda4
router: Fixes #2719 program VR nics by device id order for VPC (#2888)
This fixes #2719 where private gateway IP might be incorrectly
programmed on a guest network nic. The VR would now check ipassoc
requests by mac addresses than provided nic/device id in case they are
wrong.

The root cause is that the device id information is lost when aggregated
commands are created upon starting of a new VPC VR, without the correct
device id in ip_associations json it mis-programs the VR.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-10 15:20:36 +05:30
Frank Maximus
a6196b0a60 Fixes: #2881 Improve Exception message (#2889)
Network.Service and Network.Provider were missing a toString() method.
Added this so appending (a list of) them will be understandable.
2018-10-09 15:43:48 +05:30
Paul Angus
e73fb765d2 ui: Display i18n language name in the same language (#2883)
This update changes how the languages are shown on the UI's login page. Previously all languages were shown in the current language of the login page. This meant for instance, that a Japanese speaker would have to know English to find their language. Likewise if the language had been changed, a German speaker might need to speak Korean in order to find their language.

Therefore the UI has been altered to always show the name of each language in the appropriate language.
2018-10-09 05:57:10 +05:30
Rohit Yadav
80debcb409 Merge remote-tracking branch 'origin/4.11' 2018-10-09 05:44:10 +05:30
Paul Angus
37ecfe2d28
Merge pull request #2884 from shapeblue/usage-server-timstamp
add date to usage server logs

Merged based on 2x LGTM and checking errors in smoke tests - none in any way related to the logging output change.
2018-10-08 15:20:51 -04:00