32646 Commits

Author SHA1 Message Date
slavkap
d213a4d956
systemd: Binding listening socket to all address for remote debug (#4345)
Since Java 9 the notation 'address=port' only applies to localhost.
For remote debug you have to explicitly specify that you want to listen
to all IP addresses (e.g. address=*8000)
2020-09-29 12:26:14 +05:30
davidjumani
e0a842cf3a
test: Changing test_pvlan vlan id to prevent conflict with env config (#4340) 2020-09-29 10:58:00 +05:30
Rohit Yadav
86fcb14238 Merge remote-tracking branch 'origin/4.14'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-24 12:20:29 +05:30
Rohit Yadav
1efe6e2df0 Merge remote-tracking branch 'origin/4.13' into 4.14
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-24 12:13:23 +05:30
Gabriel Beims Bräscher
d6152b37ad
server: Broadcast URI not set to vxlan, but vlan (Fix #3040) (#4190)
This PR sets properly Broadcast URI to vxlan://vxlan_id when the physical network is of VXLAN.

Fixes: #3040
2020-09-24 11:09:50 +05:30
Abhishek Kumar
ca1e02fab9
vmware: search unmanaged instances using hypervisor name (#4328)
VMware code keeps a cache of existing VMs on a hypervisor host using cloud.vm.internal.name property of the VM. Searching for unmanaged instances/VMs on a host might not return an expected result when this property differs from the actual name of the VM.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2020-09-24 11:01:33 +05:30
Rohit Yadav
4f8b88baea
vmware: while plugging in nics get existing sorted nic devices (#4336)
In large environments, with VR having multiple nics when plugging in
nic, it must get existing nics by sorted device ID otherwise it may
cause incorrect nic plugging/order.

Fixes #4246

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-24 10:59:39 +05:30
davidjumani
d657ef7d5b
debian: Changing dependency from python3-distutils to python3-distutils-extra (#4305)
* Changing dependency from python3-distutils to python3-distutils-extra

* Update debian/control

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Co-authored-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2020-09-23 15:30:17 +05:30
Wei Zhou
534dd475d8
agent: Compare indirect agent lb algorithm when cloudstack agent connects (#4335)
Compare not only the list of management servers but also the lb algorithm when agent connects.

Fixes: #3895
2020-09-23 12:48:16 +05:30
Rohit Yadav
81c0bec765 Merge remote-tracking branch 'origin/4.14' 2020-09-23 10:50:24 +05:30
Rohit Yadav
766eab8cab Merge remote-tracking branch 'origin/4.13' into 4.14 2020-09-23 10:49:19 +05:30
Lucas Granet
ab02cf7078 router: adding "data-server" dns entry in /etc/hosts (#4319)
The DNS entry "data-server" was not added in /etc/hosts.

Since the VR is now considered as a "dhcpsrvr" (?), we need to apply this commit to add this DNS entry.
/etc/hosts is fully rewritten by this script.

Fixes: #4308
(cherry picked from commit dc65f31f9f3cb47240946c8c1cced44a7ecf9640)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-23 10:48:44 +05:30
Wei Zhou
00ceafe47f
securitygroup: Ubuntu 20.04 fix systemvm cannot start up (#4303)
* security_group.py: fix SyntaxWarning: "is" with a literal.

2020-04-27 09:43:54,172 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-2:null) (logid:c33ba330) /usr/share/cloudstack-common/scripts/vm/network/security_group.py:513: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if rules is None or rules is "":
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:522: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if rules is None or rules is "":
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:823: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if brName is None or brName is "":

* Ubuntu 20.04: Fix systemvm cannot start up

in Ubuntu 16.04:

root@node13:~# bridge -o link show
2: eth0 state UP : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
5: vnet0 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
6: vnet1 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
7: vnet2 state UNKNOWN : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100

root@node13:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1
eth0

root@node13:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 |cut -d ":" -f1
eth0

in Ubuntu 20.04:

root@node62:~# bridge -o link show
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
10: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
12: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100

root@node62:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1
ens3:

root@node62:~# bridge -o link show | awk '/master cloudbr0 / && !/^[0-9]+: vnet/ {print $2}' | head -1 |cut -d ':' -f1
ens3

* security_group.py: use 'if not' instead
2020-09-22 20:56:54 +05:30
davidjumani
ead9a34b3d
Disabling managing firewall - cloudstack-setup-management (#4239)
* Adding message to ensure ports are open

* Removing configuring iptables

* Fixing merge conflict
2020-09-22 17:46:21 +05:30
davidjumani
c06e7ded3c
systemvm: update novnc v1.2.0 (#4323)
Update noVNC v1.2.0, add support for clipboard, explicit button toolbar and resize screensize
2020-09-22 17:42:30 +05:30
Lucas Granet
dc65f31f9f
router: adding "data-server" dns entry in /etc/hosts (#4319)
The DNS entry "data-server" was not added in /etc/hosts.

Since the VR is now considered as a "dhcpsrvr" (?), we need to apply this commit to add this DNS entry.
/etc/hosts is fully rewritten by this script.

Fixes: #4308
2020-09-22 13:07:56 +05:30
Pearl Dsilva
cfbb4ff3dd
schema: change upgrade path to 4.14 (from 4.13) and intensify check (#4331)
* change upgrade path to 4.14 (from 4.13) and intensify check

* extracted check

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-09-22 09:40:51 +05:30
Andrija Panic
238eccc317
packaging: Minor message update (#4333)
adding quotes, to fix the "servers" to "server's"
2020-09-21 14:12:52 +05:30
Pearl Dsilva
90e72b1e40
vmware: Create template from detached data-disks on VMWare (#4294)
Creation of templates from detached data disks results in a Null Pointer Exception on VMWare, as it expects the volume to be attached to a VM.
To fix this behavior and make it consistent with other hypervisors, creation of the template from the volume in case not attached to a VM is facilitated by creating a worker VM, attaching the disk to the worker VM, creating the template from it, and then destroying the VM.

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-09-18 08:25:17 +05:30
Pearl Dsilva
82b6971258
server: Handle listProjects API to list projects with user as members when listAll=true (#4316)
* added defensive checks for avoiding NPE and list projects API fix

* list projects with account name provided to not include users in the account in response

Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-09-17 10:20:34 +05:30
Abhishek Kumar
87e08f8224
cks: fix logging exception (#4309)
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2020-09-17 10:17:07 +05:30
Rohit Yadav
6ee6633e6d
ui: call logout before login to clear old sessionkey cookies (#4326)
This handle edge cases of upgrades and when legacy UI is used along with
Primate or any UI sharing cookies. The specific case it fixes involves
removal of duplicate sessionkey cookies.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-17 10:15:11 +05:30
davidjumani
44bc134162
Adding acl name to several responses (#4315) 2020-09-17 10:13:14 +05:30
Pearl Dsilva
b464fe41c6
server: Secondary Storage Usage Improvements (#4053)
This feature enables the following:
Balanced migration of data objects from source Image store to destination Image store(s)
Complete migration of data
setting an image store to read-only
viewing download progress of templates across all data stores
Related Primate PR: apache/cloudstack-primate#326
2020-09-17 10:12:10 +05:30
Pearl Dsilva
caefb0c9b5
test: Increase wait time before running the ssvm health check script on SSVM reboot (#4312)
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-09-15 13:25:19 +05:30
Pearl Dsilva
f06daa5f8a
Change Global setting type for allow.user.create.projects (#4320)
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-09-15 13:23:54 +05:30
Rakesh
43a25c78f6
Display acl name in listNetworks response (#4317)
* Display acl name in listNetworks response

Display acl name along with its id so that we
dont need to make extra api call to get acl name

* Add since tag
2020-09-11 14:36:20 +02:00
Rohit Yadav
1fb50de9e8 Merge remote-tracking branch 'origin/4.14' 2020-09-09 16:04:18 +05:30
Pearl Dsilva
37c7a2b851
Incorrect md5sums for systemVM templates results in failure to download templates to other image stores (#4297)
Co-authored-by: Pearl Dsilva <pearl.dsilva@shapeblue.com>
2020-09-09 15:57:49 +05:30
Wei Zhou
6c4cdebfd8
Ubuntu 20.04: Fix issue while build package on ubuntu 20.04 (#4306)
error:
    dpkg-checkbuilddeps: error: Unmet build dependencies: python-mysql.connector

root cause: python-mysql.connector is not valid any more in ubuntu 20.04
    root@buildbox-ubuntu20:~# dpkg -l |grep connector
    ii  python3-mysql.connector               8.0.15-2build1                    all          pure Python implementation of MySQL Client/Server protocol (Python3)

solution: use python3-mysql.connector instead
2020-09-09 15:56:16 +05:30
Wei Zhou
f38db8ae65
Ubuntu 20.04: restart libvirtd instead of libvirt-bin (#4301) 2020-09-04 12:57:58 +05:30
Rohit Yadav
c7328652fd Merge remote-tracking branch 'origin/4.14' 2020-09-01 16:02:33 +05:30
Rohit Yadav
578d29e166 Merge remote-tracking branch 'origin/4.13' into 4.14
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-09-01 16:01:52 +05:30
Gabriel Beims Bräscher
5c29d5ba45
influxdb: Avoid out of memory by influxDB (#4291)
After a few hours running with InfluxDB configured, CloudStack hangs due to OutOfMemoryException raised. The exception happens at com.cloud.server.StatsCollector.writeBatches(StatsCollector.java:1510):

2020-08-12 21:19:00,972 ERROR [c.c.s.StatsCollector] (StatsCollector-6:ctx-0a4cfe6a) (logid:03a7ba48) Error trying to retrieve host stats
java.lang.OutOfMemoryError: unable to create new native thread
        ...
        at org.influxdb.impl.BatchProcessor.<init>(BatchProcessor.java:294)
        at org.influxdb.impl.BatchProcessor$Builder.build(BatchProcessor.java:201)
        at org.influxdb.impl.InfluxDBImpl.enableBatch(InfluxDBImpl.java:311)
        at com.cloud.server.StatsCollector.writeBatches(StatsCollector.java:1510)
        at com.cloud.server.StatsCollector$AbstractStatsCollector.sendMetricsToInfluxdb(StatsCollector.java:1351)
        at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:522)
Context on InfluxDB Batch: Enabling batch on InfluxDB is great and speeds writing but it requires caution to avoid Zombie threads.

Solution: This happens because the batching feature creates an internal thread pool that needs to be shut down explicitly; therefore, it is important to add: influxDB.close().
2020-09-01 15:59:43 +05:30
Spaceman1984
cb717741fc
server: Fixed delayed power state update after vm shutdown (#4284)
After a vm is shutdown, the power state isn't updated immediately. This prevents changing the service offering.
This PR updates the power state immediately after the vm is confirmed to be shutdown.

Fixes: #3159
2020-09-01 15:53:52 +05:30
Gabriel Beims Bräscher
d5acabdbf7
server: Avoid Null pointer at DomainChecker and enhance AssignVMCmd (#4279)
When executing request assignVirtualMachine with null domainID and a valid projectID then a NullPointerException happens at DomainChecker.java.

Command example:

assign virtualmachine virtualmachineid=vmID projectid=projectID account=admin
The NullPointerException that is thrown at DomainChecker is handled at AssignVMCmd.java#L142, resulting in the following log message: Failed to move vm null.
2020-09-01 13:58:42 +05:30
Wei Zhou
4746c8c726
server: move UpdateDefaultNic to vm work job queue (#4020)
While remove secondary nic from a Running vm, if update the default nic to the secondary nic before the nic is removed, the vm will not have default nic (and cannot be started) when both operations are completed.

It is because UpdateDefaultNic api is not handled as a vm work job (AddNicToVMCmd and RemoveNicFromVMCmd are), it is processed before nic is removed. The result is that secondary nic becomes default nic and got removed.
2020-09-01 13:54:48 +05:30
Rohit Yadav
749e302e0e Merge remote-tracking branch 'origin/4.14' 2020-08-28 15:00:18 +05:30
Rohit Yadav
14c0d9eaa7 Merge remote-tracking branch 'origin/4.13' into 4.14 2020-08-28 14:59:15 +05:30
Rakesh
3b4a8fcbfd
api: List networks using networkofferingid (#4258)
Add extra parameter for listNetworks command to list
all networks using networkofferingid
2020-08-28 14:55:54 +05:30
Wei Zhou
ba4b04ff37
ui: Hide cpuspeed for custom constrained offering (#3996)
For customer constrained offering, the cpu speed is fixed.
Therefore the 'CpuSpeed' field should be hidden for customer constrained offering when change vm offering on UI.
It is visible only for unconstrained offering.

This is regression issue of #3245
2020-08-28 14:44:51 +05:30
Wei Zhou
8dfc11a57c
router: Save PlaceHolder nic for VR if network does not have source nat (#3902)
This PR aims to fix the issue below

Create a network offering for isolated network, services: Dns/Dhcp/Userdata, and enable it
create a isolated network with the new offering
create a vm
check the guest IP of virtual router,
restart network with cleanup
check the guest IP of new virtual router
The IP in step4 and step6 should be the same, but they are different actually.
2020-08-28 14:44:00 +05:30
Rohit Yadav
9b6983f0e2 Merge remote-tracking branch 'origin/4.14'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-08-28 12:13:22 +05:30
Rohit Yadav
0501575efa
client: explicitly define SslContextFactory::Server for https (#4288)
Fixes #4199

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-08-28 12:09:23 +05:30
Gabriel Beims Bräscher
11e7de7d87
pom: Update Java Rados from v0.5.0 to v0.6.0 (#4287)
This PR updates the Java Rados version v0.6.0. The release artifacts are available at: https://search.maven.org/artifact/com.ceph/rados.

Fixes: #4159
2020-08-27 16:26:30 +05:30
Rohit Yadav
6d6e4025f5 Merge remote-tracking branch 'origin/4.14' 2020-08-25 17:02:19 +05:30
Greg Goodrich
9b72e5dfb8
Adding os type id to the usage record response for virtual machines (#4266) 2020-08-25 16:54:08 +05:30
Spaceman1984
39734afcbc
Changed test failure to warning (#4264)
* Added more time for capacity log

* Changed test to warning instead of fail when a timeout happens

* Update test_human_readable_logs.py

Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-08-25 15:29:59 +05:30
Spaceman1984
c4922c4289
Fixed rolling restart on VPC network (#4272)
This PR fixes the problem where restarting a network with cleanup within a VPC destroys the VR first before creating a new one.

Fixes: #3815
2020-08-25 15:19:51 +05:30
Rohit Yadav
9c7c22eb84
engine: honour bypass VLAN id/range for L2 networks (#4274)
* engine: honour bypass VLAN id/range for L2 networks

Commit e894238d904a9c49c1140371f612a51d251efc1 (#3899) allowed private
gateways to bypass vlan check while refactoring it did not cover the
case for L2 but only shared network. This fix will re-enable honouring
the bypass vlan check option for L2 guest network (in addition to the
Shared networks).

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* Update NetworkOrchestrator.java
2020-08-25 15:19:02 +05:30