31849 Commits

Author SHA1 Message Date
Abhishek Kumar
f86a7a058b utils: removed port check for url validation (#2802) (#3254)
This allows urls with ports other thatn 80 and 443

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2019-04-08 14:16:14 +05:30
Rohit Yadav
cc24801c87 Merge remote-tracking branch 'origin/4.12' 2019-04-08 13:52:57 +05:30
Rohit Yadav
9171942565 Merge remote-tracking branch 'origin/4.11' into 4.12 2019-04-08 13:52:14 +05:30
Rohit Yadav
55efaf14d9
packaging: don't skip unit tests while building packages (#3266)
This may slow down CI and release, but ensures that unit tests always
run as part of the packaging build process.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-04-08 13:51:30 +05:30
Daan Hoogland
0237ee7cd5 Merge release branch 4.12 to master
* 4.12:
  fix bugs that may cause program crash, change mkdir to mkdirs (#3249)
2019-04-05 11:32:57 +02:00
bd2019us
09045bd3e5 fix bugs that may cause program crash, change mkdir to mkdirs (#3249)
Fixes: [#CLOUDSTACK-10406](https://issues.apache.org/jira/browse/CLOUDSTACK-10406)
2019-04-05 11:14:04 +02:00
skattoju3
beb0422dd5 fix incorrect iscsi path stat for managed storage (#3181)
## Description
The issue was that an Incorrect iscsi path was being passed for managed storage pools when collecting volume stats. Storage pools normally have a UUID based path while managed storage pools require an IQN based path
2019-03-29 10:25:01 +01:00
Rohit Yadav
4985e57f30
ui: use executable template filter for users (#3214)
Problem: When reinstalling a VM, the UI only shows featured templates and not all possible list of allowed templates.
Root Cause: The list of templates for reinstall used featured as filter in ‘listTemplates’ API which did not include all available options.
Solution: The issue is fixed by using the executable template filter of the ‘listTemplates’ API to list all
possible templates that a user is allowed to use to deploy a VM.
2019-03-29 12:13:26 +05:30
Gabriel Beims Bräscher
6c841bf69b apidocs: Make the API documentation version not *hardcoded* to v4.9.0 (#3247)
This commit makes the API documentation automatically updated to the
current project version. If one wants to set another version, it is
possible by just editing the ACS_RELEASE variable in 'build-apidoc.sh'.
2019-03-29 11:55:45 +05:30
Rohit Yadav
fd541efe80 Merge remote-tracking branch 'origin/4.12' 2019-03-29 11:50:04 +05:30
Boris
c2bd32fa41 Merge remote-tracking branch 'upstream/4.11' into 4.12 2019-03-27 12:04:15 +02:00
Boris Stoyanov - a.k.a Bobby
c4b06ffe19
Merge pull request #3238 from shapeblue/tls-vmware-issue-fix
client: don't disable TLSv1, TLSv1.1 by default that breaks VMware env
2019-03-27 11:39:54 +02:00
Rohit Yadav
671a70a9a5
schema: add empty DB upgrade path from 4.12.0.0 to 4.13.0.0 (#3236)
This adds empty empty upgrade path from 4.12.0.0 to 4.13.0.0.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-27 00:00:47 +05:30
Rohit Yadav
6e51bde228 client: don't disable TLSv1, TLSv1.1 by default that breaks VMware env
This fixes the issue that TLSv1 and TLSv1.1 are still used by CloudStack
management server to communicate with VMware vCenter server. With the
current defaults, the setup/deployment on VMware fails. Users/admins
can however setup the security file according to their env needs to
disable TLSv1 and TLSv1.1 for server sockets (8250/agent service for
example).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-26 18:15:20 +05:30
Rohit Yadav
5ee7a9c530 Merge remote-tracking branch 'origin/4.12' 2019-03-26 15:35:17 +05:30
Rohit Yadav
12c0d88e31 Merge remote-tracking branch 'origin/4.11' into 4.12 2019-03-26 15:34:41 +05:30
Paul Angus
8b25fdf520 tests: fix some Marvin smoketests (#2869)
Fixes intermittent failures:
- Add a pause to avoid  tests restarting before VRs recovering from HA have fully booted.
- Add some pauses to allow services to restart and hosts to reconnect before continuing tests.
- Adding a loop around this method because sometimes VRs are overloaded and just respond with exception, so it'll catch it and try 5 times with 30sec cooldown. If that fails as well it'll fail the test.
- wait until host is up using explicit check
2019-03-26 15:33:40 +05:30
Philipp Bankonier
2e2b6700a5 ui: Fix behavior of multiselect in list view (#3161)
Hide multiselectaction buttons when selection is resetted after a action is performed
Use checked prop instead of attr to show/hide triggered selection when box was clicked by a user before.

How to reproduce:

Select one entry in a multiSelect listView (e.g. instance list) and then deselect it again. Now click on the "select all" checkbox. The entries which were clicked previously are not displayed as selected.

Select at least one entry. Use the multiSelectAction buttons to trigger an action. Wait for completion. The list refreshes and no entry is selected but the multiSelectAction buttons are displayed.

Fix:

Use "checked" property instead of the attribute to select all entries. This is necessary because when you checked an entry by clicking on it the property gets set and the browser is preferably using this. So setting the attribute by clicking on the "select all" checkbox had no impact anymore on the checkBox of the previously clicked entries.

Hide the multiSelectAction buttons when the list is refreshed after an action is performed and no element is selected anymore.
2019-03-26 13:11:41 +05:30
Philipp Bankonier
b81171af7d ui: add start button for multiple selected instances in list view (#3160)
Adds start button for multiple instances in list view.
Change button order for instances listview multiselect actions.
2019-03-26 13:06:06 +05:30
Rohit Yadav
b9cc6c8c73
debian: cleanup commons-daemon no longer needed by agent (#3165)
With 4.13+ for debian, only systemd enabled distros are supported. This removes an unnecessary dependency.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-26 13:02:40 +05:30
Rohit Yadav
9d3009f2b9
ui: remove CA certificate button from UI (#3211)
This removes CA certificate button from UI as this is not used and
confuses both admin and users how to use it. This may be re-introduced
in future when a use-case can be justified.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-25 14:57:55 +05:30
dahn
f88de7b863 kvm: NotImplemented, with message to signify what is not implemented, using commons-lang version (#3170)
NotImplementedException from sun.reflect.generics.reflectiveObjects is not standard/guarateed by the jvm so a custom implementation is in order.
2019-03-25 13:58:08 +05:30
Rohit Yadav
462a37d1b4
server: make snapshotting on KVM non-blocking (#3209)
* server: make snapshotting on KVM non-blocking

This references and uses an already fixed solution from
https://github.com/MissionCriticalCloud/cosmic/pull/68 to make
snapshotting on KVM non-blocking.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* move StorageSubSystemCommand instanceof check above as CopyCommand is a type of StorageSubSystemCommand

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-25 13:57:21 +05:30
Rohit Yadav
a3938b330a Merge remote-tracking branch 'origin/4.12' 2019-03-25 13:27:28 +05:30
Rohit Yadav
bbf5807688 Merge remote-tracking branch 'origin/4.11' into 4.12
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-25 13:24:46 +05:30
Dingane Hlaluku
55fb1c4eb6 server: Allow users to create L2 network types (#3158)
Allow users of all types to create L2 guest networks.

Fixes #3081
2019-03-25 13:12:19 +05:30
Dingane Hlaluku
0d21f0c1e7 api: rename ListUsageRecords file name to ListUsageRecordsCmd (#3151)
* Rename ListUsageRecords API command file name to ListUsageRecordsCmd

* Refactor to use APINAME variable and remove unused s_logger field

* Remove unused import
2019-03-25 13:10:53 +05:30
GabrielBrascher
8d3feb100a Updating pom.xml version numbers for release 4.13.0.0-SNAPSHOT
Signed-off-by: GabrielBrascher <gabriel@pcextreme.nl>
2019-03-20 18:47:35 -03:00
GabrielBrascher
a137398bf1 Updating pom.xml version numbers for release 4.12.0.0
Signed-off-by: GabrielBrascher <gabriel@pcextreme.nl>
4.12.0.0
2019-03-14 10:11:46 -03:00
Rohit Yadav
a6f4d6d312 Merge remote-tracking branch 'origin/4.11' 2019-03-14 17:55:33 +05:30
Rohit Yadav
f7327c7457 systemd: Fix -Dpid arg passing to systemd usage service (#3210)
* systemd: Fix -Dpid arg passing to systemd usage service

This fixes regression introduced by refactoring PR #3163 where `-Dpid`
was incorrectly passed string `$$` instead of parent PID integer.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* fix systemd limitation, exec using /bin/sh instead and wrap in ${} syntax

https://www.freedesktop.org/software/systemd/man/systemd.service.html#Command%20lines

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* usage: don't hide exception from Gabriel's https://github.com/apache/cloudstack/pull/3207/files#diff-062fcf5ae32de59dfd6cd4f780e1d7cd

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-03-14 09:19:12 -03:00
Gabriel Beims Bräscher
34030be393
Fix XenServer Security Groups 'vmops' script (#3197)
* Fix XenServer Security Groups 'vmops' script

- fix tokens = line.split(':') to tokens = line.split(';')
- fix expected tokens size from 5 to 4
- enhance logs
- remove unused vmops script. The XCP patch points to the vmops script
on the parent folder [1]. Thus, all XenServer versions are considering
the vmops script located at [2].
- fix UI ipv4/ipv6 cidr validator to allow a list of cidirs.

Fixing issue: #3192 Security Group rules not applied at all for
XenServer 6.5 / Advanced Zone
https://github.com/apache/cloudstack/issues/3192

* Update security group rules after VM migration

Add security group rules on target host
Cause: vmops script expected secondary IPs as "0;" but received "0:"

Remove security group network rules on source host.
Cause: destroy_network_rules_for_vm function on vmops script was not
called when migrating VM

* Add unit tests and address reviewers
2019-03-06 09:13:40 -03:00
Gabriel Beims Bräscher
709845f4a3
Keep iotune section in the VM's XML after live migration (#3171)
* Keep iotune section in the VM's XML after live migration

When live migrating a KVM VM among local storages, the VM loses the
<iotune> section on its XML, therefore, having no IO limitations.

This commit removes the piece of code that deletes the <iotune> section
in the XML.

* Add test for replaceStorage in LibvirtMigrateCommandWrapper

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* Fix Javadoc for method replaceIpForVNCInDescFile
2019-02-12 22:07:03 -02:00
Gabriel Beims Bräscher
7c5eca9481
Copy template to target KVM host if needed when migrating local <> local storage (#3154)
* Migrate template to target host if needed.

Fix KVM VM local storage live migration by migrating its template to the
target host if needed.

* Address reviewer and add method that updates the DB template reference

* Remove deprecated Config.PrimaryStorageDownloadWait

* Code formating of @Inject to follow checkstyle
2019-02-05 00:18:29 -02:00
GabrielBrascher
3f17671449 Fix conflict and merge forward PR #3163 from 4.11 to master (4.12)
# Conflicts:
#	packaging/debian/init/cloud-management
#	packaging/systemd/cloudstack-agent.default
#	packaging/systemd/cloudstack-agent.service
#	packaging/systemd/cloudstack-management.service
2019-02-04 23:53:19 -02:00
Rohit Yadav
cb3fed0e4e systemd: fix services to allow TLS configurations via java.security.ciphers (#3163)
* systemd: fix services to allow TLS configurations via java.security.ciphers

This fixes the management server and systemd services to allow the
java.security.ciphers file to configure disabled TLS protocols and
algorithms. This also cleans up systemd service files for agent and
usage server.

This fixes #3140

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* configure: fix travis failure due pycodestyle error

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-02-04 19:51:30 -02:00
Nathan Johnson
637cc6ec4e feature: add libvirt / qemu io bursting (#3133)
* feature: add libvirt / qemu io bursting

Adds the ability to set bursting features from libvirt / qemu

This allows you to utilize the iops and bytes temporary "burst" mode
introduced with libvirt 2.4 and improved upon with libvirt 2.6.

https://blogs.igalia.com/berto/2016/05/24/io-bursts-with-qemu-2-6/

* updates per rafael et al
2019-02-04 19:47:44 -02:00
GabrielBrascher
460d3127ec Fix conflict and merge forward PR #3122 from 4.11 to master (4.12) 2019-02-04 19:24:59 -02:00
Nathan Johnson
bf805d1483 Add back ability to disable backup of snapshot to secondary (#3122)
* The snapshot.backup.rightafter configuration variable was removed by:

SHA: 6bb0ca2f854

This adds it back, though named snapshot.backup.to.secondary now instead.

This global parameter, once set, will allow you to prevent automatic backups of
     snapshots to secondary storage, unless they're actually needed.

Fixes #3096

* updates per review
2019-02-04 19:08:42 -02:00
GabrielBrascher
ff07c17a26 Foward merge #3139 from 4.11 to master (4.12) 2019-02-04 19:05:11 -02:00
Gabriel Beims Bräscher
f0a2bf93bf
Fix issue with verification of ipv4/ipv6 address. (#3162) 2019-02-04 15:58:50 -02:00
dahn
58466c8954 api: add command to list management servers (#3150)
* api: add command to list management servers

* api: add number of mangement servers in listInfrastructure command

* ui: add block for mangement servers on infra page

* api name resolution method cleanup
2019-02-01 16:23:39 -02:00
Frank Maximus
00e6d5991d Fix UI bug (#3149)
Upgrade to jquery 3.3.1: $('selector').context was deprecated,
and has been removed.

Fixes: #3148
2019-01-30 15:47:42 -02:00
Wido den Hollander
9c4292cc45 network: Offerings do not have to have Security Grouping enabled (#3112)
Offerings can co-exist where on does provide Security Grouping in the
network, but other guest Networks have no Security Grouping.

In V(X)LAN isolation environments the L2 separation is handled by V(X)LAN
and protection between Instances is handled by Security Grouping.

There are multiple scenarios possible where one network has Security Grouping
enabled because that is required in that network.

In the other network, but in the same zone it could be a choice to have
Security Grouping disabled and allow all traffic to flow.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-28 15:38:08 -02:00
Rohit Yadav
463372bc7e
packaging: management default file cleanup (#3139)
This cleanups management server default file, the `cloud.jks` is no
longer created by the management server but instead created in-memory
by the root CA plugin on management server startup.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2019-01-25 22:19:33 +05:30
dahn
b363fd49f7 Vmware offline migration (#2848)
* - Offline VM and Volume migration on Vmware hypervisor hosts
- Also add VM disk consolidation call on successful VM migrations

* Fix indentation of marvin test file and reformat against PEP8

* * Fix few comment typos
* Refactor debug messages to use String.format() when debug log level is enabled.

* Send list of commands returned by hypervisor Guru instead of explicitly selecting the first one

* Fix unhandled NPE during VM migration

* Revert back to distinct event descriptions for VM to host or storage pool migration

* Reformat test_primary_storage file against PEP-8 and Remove unused imports

* Revert back the deprecation messages in the custom StringUtils class to favour the use of the ApacheUtils
2019-01-25 10:05:13 -02:00
Bitworks LLC
d68712eb7b CLOUDSTACK-3049: Implemented role update for account. (#3058) 2019-01-25 10:02:56 -02:00
Malcolm Taylor
97ddd8dffd Fix some LGTM alerts (#3143)
* Fix some LGTM alerts

* address review comments from Gabriel Brascher and Rafael Weingartner
2019-01-24 20:22:39 -02:00
Wido den Hollander
c496c84c6c kvm: Properly report available memory to Management Server (#2795)
The KVM Agent had two mechanisms for reporting its capabilities
and memory to the Management Server.

On startup it would ask libvirt the amount of Memory the Host has
and subtract and add the reserved and overcommit memory.

When the HostStats were however reported to the Management Server
these two configured values on the Agent were no longer reported
in the statistics thus showing all the available memory in the
Agent/Host to the Management Server.

This commit unifies this by using the same logic on Agent Startup
and during statistics reporting.

  memory=3069636608, reservedMemory=1073741824

This was reported by a 4GB Hypervisor with this setting:

  host.reserved.mem.mb=1024

The GUI (thus API) would then show:

  Memory Total	2.86 GB

This way the Agent properly 'lies' to the Management Server about its
capabilities in terms of Memory.

This is very helpful if you want to overprovision or undercommit machines
for various reasons.

Overcommitting can be done when KSM or ZSwap or a fast SWAP device is
installed in the machine.

Underprovisioning is done when the Host might run other tasks then a KVM
hypervisor, for example when it runs in a hyperconverged setup with Ceph.

In addition internally many values have been changed from a Double to a Long
and also store the amount of bytes instead of Kilobytes.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-24 20:18:04 -02:00
Dingane Hlaluku
323f791efc IP address acquired with associate ip address is marked as source nat (#3125)
* CLOUDSTACK-4045 added a check for network state when determining whether a new IP should be source NAT. this prevents associated IP's to be marked as source NAT when the network is in allocated state, causing disassociateIpAddress to fail later

* Remove mock object that cause other tests to fail

* Remove underscores from variable types and add documentation for the created method

* Improve exception message to include network name

* Include network UUID with the Exception message and fix failing marvin test

* Rebase against latest master and format AssociateIPAddrCmd class
2019-01-23 10:05:16 -02:00