32316 Commits

Author SHA1 Message Date
Wei Zhou
fd5bea838b
New feature: Add support to destroy/recover volumes (#3688)
* server: fix resource count of primary storage if some volumes are Expunged but not removed

Steps to reproduce the issue
(1) create a vm and stop it. check resource count of primary storage
(2) download volume. resource count of primary storage is not changed.
(3) expunge the vm, the volume will be Expunged state as there is a volume snapshot on secondary storage. The resource count of primary storage decreased.
(4) update resource count of the account (or domain), the resource count of primary storage is reset to the value in step (2).

* New feature: Add support to destroy/recover volumes

* Add integration test for volume destroy/recover

* marvin: check resource count of more types

* messages translate to JP

* Update messages for CN

* translate message for NL

* fix two issues per Daan's comments

Co-authored-by: Andrija Panic <45762285+andrijapanicsb@users.noreply.github.com>
2020-02-07 11:25:10 +01:00
Rohit Yadav
d8906d3d8b Merge remote-tracking branch 'origin/4.13' 2020-02-07 11:18:58 +05:30
Wei Zhou
cd55674afe packaging: install python-dnspython or python-dns to fix issue with cloudstack-setup-management (#3854)
Fixes #3817 #3841

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-02-07 11:18:00 +05:30
Spaceman1984
f769c66bc1 ui: fix default text missing from network selection on instance wizard (#3865)
This PR fixes the missing "Default" display issue on the instance wizard when selecting a network.

Fixes: #3633
2020-02-07 11:17:38 +05:30
Rohit Yadav
493415d91b
packaging: install python-dnspython or python-dns to fix issue with cloudstack-setup-management (#3854) (#3869)
Fixes #3817 #3841

Co-authored-by: Wei Zhou <ustcweizhou@gmail.com>
2020-02-07 11:13:57 +05:30
Rakesh
c6e67c482c
ui: display drop down list for VPN customer gateway selection (#3805)
Display drop-down list for VPN customer gateway selection in UI
so that it can be searched according to domains
2020-02-07 11:10:56 +05:30
Bitworks LLC
f78cbf4efc
ssvm: wrong SSVM behavior causes redownloading for all the templates (#3844)
As per discussion at #3838 and proposal by @weizhouapache this PR implements the fix.

Fixes #3838
2020-02-07 11:09:47 +05:30
Spaceman1984
a746b29a83
ui: fix default text missing from network selection on instance wizard (#3865)
This PR fixes the missing "Default" display issue on the instance wizard when selecting a network.

Fixes: #3633
2020-02-07 11:04:20 +05:30
Daan Hoogland
10482da136 Merge release branch 4.13 to master
* 4.13:
  vr: add missing rule for port forwarding rule in vpc (#3857)
  vpc: set traffic type of private gateway IP to Public to fix ke… (#3851)
2020-02-06 20:38:07 +01:00
Wei Zhou
d88c614a35
vr: add missing rule for port forwarding rule in vpc (#3857) 2020-02-06 20:25:56 +01:00
Wei Zhou
a9a1737dd9
vpc: set traffic type of private gateway IP to Public to fix ke… (#3851) 2020-02-06 20:22:08 +01:00
Rohit Yadav
33e9a50945 Merge remote-tracking branch 'origin/4.13' 2020-02-06 11:32:09 +05:30
Pearl Dsilva
bfdb914693
usage: publish zone id while uploading template and volume (#3867)
After a local template is uploaded via browser, the generated usage event with type = "TEMPLATE.CREATE" is persisted with the data store ID instead of the zone ID on the zone_id column. The fix will refactor the upload monitor logic, as after the upload completes, it sets the datastore ID on the zone ID column for the created "TEMPLATE.CREATE" usage event. This refactor will query the DB for the data store and will set its associated zone ID in the usage field.
The fix produces the same behaviour as when registering a template from URL.
FIx is also for uploading VOLUME from local/via browser.
2020-02-06 11:31:24 +05:30
Wei Zhou
c22e99c638
test: check more connectivity in test_privategw_acl.py (#3861) 2020-02-05 12:15:22 +01:00
Rakesh
107bba3da1
router: start all Site-to-Site VPN connections when VPC VR starts (#3863)
When we restart the VPC after destroying the master VR, the backup VR
becomes the master and the site to site connections are not in
the connected state. The passive VPN connection will be in the connected
state but the active VPN connection will be in a disconnected state.
2020-02-05 11:56:47 +05:30
Rakesh
70daee9b10 network: set restart_required to 0 after restarting network (#3803)
After restarting the network with or without cleanup option, the restart_required field in networks table should be reset to 0.
2020-02-04 14:42:46 +01:00
Abhishek Kumar
0f5b0e67f8
VM ingestion (#3606)
The VM ingestion feature allows CloudStack to discover, on-board, import existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future.
2020-02-03 15:43:52 +01:00
Rakesh
6d014fa219
network: de-associate IP address if enabling static nat fails (#3836)
Associating static NAT on IP to VM fails even though the IP is not allocated.
When we try enable static NAT on second IP address to the same VM, the operation fails but the IP address is still allocated in the db and it can't be used to enable static NAT on different VM.

Steps to reproduce the issue:

(1) create a vpc (vpc-001) and a vpc tier (vpc-001-001)

(2) create a vm (vm-001-001) in vpc-001-001

(3) acquire a public ip (ip-1) and enable static nat to vm-001-001,
operation succeeds.

(4) acquire a public ip (ip-2) and enable static nat to vm-001-001,
operation fails but the ip is still assigned to vpc tier vpc-001-001.
Note down the ip address and the id of it.

(5) create another vpc tier vpc-001-002, and vm (vm-001-002) in the tier

(6) enabled ip-2 static nat to vm-001-002, operation should succeed
2020-01-31 16:27:20 +05:30
Rakesh
a2a4968f51
server: Allow creating network with duplicate name (#3807)
Add a global setting to disable creating networks with same name in an account

Add a global setting to disable creating network without
mentioning the start and end IPv4 or IPv6 address

By default we can create networks with the same name in the account.
Sometimes we should not create the networks with same name.
This change adds a global setting which prevents creating the network with same name.
The default value is true and set it to false to prevent creating network with same names.

Also its possible to create a shared network without mentioning the
start and the end IPv4 or IPv6 address.
This change adds a global setting which prevents creating a shared
network without specifying the start and the end IPv4 or IPv6 address
2020-01-31 15:52:42 +05:30
Rakesh
1a5b7c362e
engine/orchestration: display numeric value instead of variable name (#3818)
If the disk size of the vm to be created is greater
than the volume size, then the exception message should
display the numeric value instead of variable name
2020-01-31 15:42:06 +05:30
Rohit Yadav
f72f39cd11 Merge remote-tracking branch 'origin/4.13' 2020-01-31 15:28:34 +05:30
Abhishek Kumar
a71874682c
server: fix checking disk offering access for snapshot volume (#3791)
Fixes #3783
As reported in the issue, creating volumes from pure snapshot fails with NPE. This is due to order of calls where disk offering access is checked before checking disk offering value. This PR fixes the same.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
2020-01-31 15:24:25 +05:30
Wei Zhou
75aa6e3659
ui: fix cannot assign vms to internal lb in VPC (#3832)
This fixes the issue that cannot assign VMs in Internal LB in VPC.
listView is overwritten due to commit 2e2b670 (#3161)

Fixes: #3827
2020-01-31 15:23:11 +05:30
Wei Zhou
76fb811e08
kvm: Fix router migration issue when router has control/public nics on other physical network than guest (#3855)
In VM migration on KVM, libvirt qemu hook script will change the bridge name to bridges for guest networks. It works for user vm. However for virtual router, it has nics on control network and public network. If control/public use different physical networks than guest network, virtual router cannot be migrated.

Fixes: #2783
2020-01-31 15:14:35 +05:30
Rohit Yadav
424f10cc77 Merge remote-tracking branch 'origin/4.13' 2020-01-31 14:18:11 +05:30
Abhishek Kumar
9d105b6546
template: copy md5 mismatch (#3383)
Fixes #3191

When a template is registered, code stores md5sum of the downloaded file in the vm_template table. However, this downloaded file could be deleted after template installation if it is not an actual (.qcow2, .ova, etc.) file. When the user copies a template using copyTemplate API, the actual template file will be copied across the image stores. Matching checksum for the copied templated file and the stored value from the vm_template table will result in a mismatch.
Changes will set an empty checksum value for the copied template while passing to download service which allows skipping wrong checksum check for the copied while install.
However, this results in a change in checksum value for concerned template entry in vm_template table post template install.

Co-authored-by: dahn <daan.hoogland@gmail.com>
2020-01-31 14:16:37 +05:30
Daan Hoogland
f5551be549 Merge branch '4.13' 2020-01-31 08:40:30 +01:00
Sid Kattoju
6baa598033
Clean up inactive iscsi sessions when VMs get moved due to crashes (#3819) 2020-01-30 19:52:28 +01:00
Anurag Awasthi
c0abfce8fa
Health check feature for virtual router (#3575) 2020-01-30 12:39:03 +01:00
Xia Li
f1149bcb09
[CLOUDSTACK-10408] Fix String.replaceAll() to replace() for bet… (#3275)
* [CLOUDSTACK-10408] Fix String.replaceAll() to replace() for better performance

* improve with replace char but string

Co-authored-by: Rohit Yadav <rohit@apache.org>
2020-01-30 12:06:15 +01:00
Daan Hoogland
99ec8a825a Merge release branch 4.13 to master
* 4.13:
  Fix Policy Based Routing for private gateway static routes (#3604)
2020-01-30 11:39:36 +01:00
Dennis Konrad
82d94a87c5
Fix Policy Based Routing for private gateway static routes (#3604)
* Fix for routing table issue with NAT interfaces

* Mark only packets with the public ip as destination
2020-01-30 11:31:30 +01:00
Wei Zhou
ac581d1546
New feature: Resource count (CPU/RAM) take only running vms into calculation (#3760)
* marvin: check resource count of more types

* New feature: add flag resource.count.running.vms.only to count resource consumption of only running vms

Stopped VMs do not use CPU/RAM actually.
A new global configuration resource.count.running.vms.only is added to determine whether resource (cpu/memory) of only running vms (including Starting/Stopping) will be taken into calculation of resource consumption.

* Add integration test for resource count of only running vms
2020-01-30 10:36:50 +01:00
Rohit Yadav
518ed5379c Merge remote-tracking branch 'origin/4.13' 2020-01-30 11:13:14 +05:30
Rakesh
920531f42d
network: set restart_required to 0 after restarting network (#3803)
After restarting the network with or without cleanup option, the restart_required field in networks table should be reset to 0.
2020-01-30 11:12:38 +05:30
davidjumani
7a25e40d5a
api: allow listing management server by id and name (#3840)
The List Management Server api returns a list of all the management servers but fails when trying to list by id or name. This ensures that it fetches the details as per the parameters passed.
Fixes: #3833
2020-01-30 10:38:25 +05:30
Pearl Dsilva
1c130a5dd4
api: metrics API response is not super-set of resources response keys (#3834)
The metrics API has few properties missing that are present in the corresponding resource. 

Fixes #3831

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Co-authored-by: Rohit Yadav <rohit@apache.org>
2020-01-30 08:49:45 +05:30
Wei Zhou
521217c852
vr: fix vr in unknown state (more) (#3848)
This fixes similar issue with #3465.

Meanwhile change log level of CsHelper.execute2 from DEBUG to INFO and fix some typo.
2020-01-30 08:43:46 +05:30
Rohit Yadav
a54afa820e Merge remote-tracking branch 'origin/4.13' 2020-01-29 20:51:27 +05:30
Wei Zhou
be112a0220
vrouter: reload haproxy when cfg file is updated (#3726)
since 4.11.3, haproxy is always restarted when add/delete a lb rule.
When haproxy is started, the processes are
```
root@r-854-VM:~# ps aux |grep haproxy
root     22272  0.0  0.2   4036   668 ?        Ss   07:52   0:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  22274  0.0  2.3  38444  5856 ?        S    07:52   0:00 /usr/sbin/haproxy-master
haproxy  22275  0.0  0.3  38444   880 ?        Ss   07:52   0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
```
When haproxy is reload, the processes are
```
root@r-854-VM:~# ps aux |grep haproxy
root     22272  0.0  0.2   4168   632 ?        Ss   07:52   0:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  22283  0.0  2.3  38444  5884 ?        S    07:53   0:00 /usr/sbin/haproxy-master
haproxy  22286  0.0  0.3  38444   880 ?        Ss   07:53   0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 22275
```

We need to change the pid file from /var/run/haproxy.pid to /run/haproxy.pid, so the haproxy will be reloaded instead of restarted.
2020-01-29 16:01:19 +05:30
Rohit Yadav
28fb93f996 Merge remote-tracking branch 'origin/4.13' 2020-01-29 15:23:48 +05:30
Spaceman1984
4ea6f32566
ui: Fixed unable to edit domain settings (#3846)
This pull request adds a scroll bar to the domain settings page that will allow a user to scroll to the right to change settings.

Fixes: #3032
2020-01-29 12:49:20 +05:30
Rohit Yadav
cf2be8a036 Merge remote-tracking branch 'origin/4.13' 2020-01-28 18:51:10 +05:30
Rohit Yadav
b1783fbcf1
travis: use https based maven repo mirror (#3845)
Travis has been failing lately as it requires that repositories use
https:// URLs.

Reference:
https://blog.sonatype.com/central-repository-moving-to-https

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2020-01-28 18:50:37 +05:30
Rohit Yadav
0cb2db6e1d Merge remote-tracking branch 'origin/4.13' 2020-01-28 11:26:40 +05:30
Rakesh
5ef5ef6253 README: Update Docker README file (#3835) 2020-01-28 11:21:56 +05:30
Andrija Panic
0095272a38 upgrade: kvm-local-pool-trailing-slash (#3813)
Stop asking user (in the upgrade documentation) to remove a trailing slash for local KVM pool - do it here in upgrade path - so not needed in DOC for the upgrade to 4.14 and onwards.
2020-01-28 11:18:27 +05:30
Gregor Riepl
8792070f84 Rethrow takeVMSnapshot() exception instead of returning null in VMSnapshotManagerImpl (#3761)
Fixes: #3518
2020-01-28 11:05:15 +05:30
Wei Zhou
a77d74ba0d server: Fix NPE while update displayvm on vm with dynamic service offering (#3758)
Steps to reproduce the issue
(1) create a custom service offering
(2) create a vm with the offering
(3) update vm with displayvm=false, returns an error

(local) > update virtualmachine id=f33fd06a-7643-40d1-833f-272845d9ba09 displayvm=false
Error 530: {"updatevirtualmachineresponse":{"uuidList":[],"errorcode":530,"cserrorcode":9999}}
2020-01-28 11:04:26 +05:30
Wei Zhou
136505b22c server: double check host capacity when start/migrate a vm (#3728)
When start a vm or migrate a vm (away from a host in host maintenance), cloudstack will check capacity of all hosts and choose one. If there are hundreds of hosts on the platform, it will take some seconds. When cloudstack choose a host and start/migrate vm to it, the resource consumption of the host might have been changed. This normally happens when we start/migrate multiple vms.
It would be better to double check the host capacity when start vm on a host.

This PR includes the fix for cpucore capacity when start/migrate a vm.
2020-01-28 10:55:11 +05:30