- Ensure that FW_EGRESS_RULE chain exists after upgrading the router
- Flush allow all egress rule on 0.0.0.0/0, if such a rule exists in the config
it will be added later (CLOUDSTACK-9437)
CLOUDSTACK-9436: vm_network_map table cleanup, release network resources on expunge commandJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9436
This PR replaces #1594
Due to error on `test/integration/smoke/test_vpc_redundant.py` it was found out that `vm_network_map` table should be less aggresive on vm stop
* pr/1613:
CLOUDSTACK-9436: Release network resources on expunge command
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9407: vm_network_map table doesnt get cleaned up properlyJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9407
### Introduction
It was found out that in production environments `vm_network_map` table entries were slowly growing. It was investigated how this entries were cleaned up.
### Behaviour
On vm creation, vm mappings are inserted on `vm_network_map`.
On vm stop, mappings are deleted from `vm_network_map` for vm, as a result of the release of its nics.
### Problem
If created vm is stopped from hypervisor side (at least on vSphere in which we tested it), when CloudStack realizes vm is stopped it doesn't clean up `vm_network_table,` and, as cleanup is made during vm stop, when vm is eventually destroyed and expunged it won't clean up their entries in that table.
### Proposed solution
We propose to move `vm_network_map` table cleanup to expunge command instead of stop command.
* pr/1594:
CLOUDSTACK-9407: Refactor
CLOUDSTACK-9407: Release network resources on expunge command
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Update L10N resource files with 4.9 strings from Transifex (20160607)cc @swill before the 4.9 release. Just only the latest FR translation.
* pr/1583:
Update L10N resource files with 4.9 strings from Transifex (20160709)
Signed-off-by: Will Stevens <williamstevens@gmail.com>
packaging: Add db.X.driver=jdbc:mysql to db.properties on upgradeThis is required afther the upgrade to 4.9.0 and for convience we
add this to the configuration so our users do not have to.
* pr/1610:
packaging: Add db.X.driver=jdbc:mysql to db.properties on upgrade
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9430] Added fix for adding/editing Network ACL rule orderingBUG: https://issues.apache.org/jira/browse/CLOUDSTACK-9430
The issue occurred because all of the ACL rules get inserted before the old ones. Then, the cleanup deletes the duplicate rows, and leaves any new rule in front of the old ones.
Here is an example with a simplified iptables view for ACL
Ex: adding a rule 4
before add:
1,2,3
during add:
1',2',3',4',1,2,3
after add:
4',1,2,3
After fix:
before add:
1,2,3
during add:
1,2,3,1',2',3',4'
after add:
1',2',3',4'
* pr/1609:
Added fix for adding/editing Network ACL rule ordering
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Cleanup RBD contexts after exceptions to prevent potential agent crashWe noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
* pr/1608:
improve logging readability
Cleanup rbd contexts and improve exception logging
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9348: Reduce Nio selector wait timeThis reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
/cc @swill @kiwiflyer guys can you please apply this fix in your env and test if you're still able to produce any Nio related error b/w mgmt server(s) and kvm agent(s) not being able to connect quickly. Please also watch out for any increased CPU usage (there should not be any significant change), in which case we may increase the timeout from 100ms to 200-400ms.
* pr/1601:
CLOUDSTACK-9348: Reduce Nio selector wait time
Signed-off-by: Will Stevens <williamstevens@gmail.com>
ui: show resize volume button to all users
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this bug fix all users would see resize volume button in the UI.
/cc @swill
* pr/1595:
ui: show resize volume button to all users
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVTPlease refer to CLOUDSTACK-9328 for the details.
Test Results:
==========
test_01_vpc_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_01_vpc_privategw_acl | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 108.273s
OK
* pr/1455:
[CLOUDSTACK-9328]: Fix vlan issues from test suite test_privategw_acl.py in BVT Bug-Id:## CLOUDSTACK-9328
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctlyBug in code set PFS to the same value (yes/no) as DPD.
file.addeq(" pfs=%s" % CsHelper.bool_to_yn(obj['dpd']))
* pr/1480:
CLOUDSTACK-9342: Site to Site VPN PFS not being set correctly
Signed-off-by: Will Stevens <williamstevens@gmail.com>
We noticed that when an exception occurs within the cleanup loop inside
the deletePhysicalDisk routine that the previously allocated contexts
are not cleaned up. This seemed to cause an eventual crash of the host
agent after multiple exceptions within the loop.
In addition to ensuring the contexts are always freed we also improved
the logging when exceptions do occur to include the actual return code
from the underlying library in deletePhysicalDisk and deleteSnapshot.
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDsWith object store like Swift as secondary storage, if a compressed VHD is uploaded as a template, the `VHDProcessor` incorrectly calculates the virutal size leading to the template being useless. This fix tries to guess the virtual size by partially decompressing it and falls back to a sensible default which is the size of the file.
Before the fix: template.properties on Swift
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=2894447637315205059
```
After the fix
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=3145728000
```
Look at the `virutalsize` in both cases
* pr/1598:
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
Signed-off-by: Will Stevens <williamstevens@gmail.com>
This reduced the Nio loop selector wait time, this way the selector will
check frequently (as much as 100ms per iteration) and handle any pending
connection/tasks. This would make reconnections very quick at the expense of
some CPU usage.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Fixes for VirtualRouters in Basic Networking, especially with mutliple ranges in VLANsDuring the last few modifications on the SystemVM scripts, it turns out quite a lot of stuff broke in our setups.
This PR fixes a number of things:
* Multiple IP's per VLAN interface are now supported & working again, including DNS, DHCP ranges, password and metadata services
* `useextdns` fixed (I had a small merge conflict with an attempt to fix this at 4.7, but these fixes are more comprehensive)
* CLOUDSTACK-8303
* Apache configs better in line with best-practices and distro-expected locations
* Added a few more helper functions & getters & setters for utility
* some minor cleanup & fixes
* pr/1547:
Remove /etc/apache2/sites-enabled/000-default in cloud-early-config
SysVM various fixes to previous refactorings * make CORS include a regular glob-matched one * fix NameVirtualHost in CsApp.py as well * even moar cleanups
SysVM: Cleanup and removal of old (and dangerous) config files * ports.conf * default & default-ssl sites * SSL config in httpd.conf * deprecated & dead setup_redundant_router in cloud-early-config
SysVM cloud-early-config: Intermediate fix for SecStore & CORS * Take setup from vhost.template rather than default(-ssl) * should move into Python CS code as well * Move CORS setup to separate conf * Modify vhost template to Optionally include the cors file * Add NameVirtualHost to vhost template for feature parity with ports.conf * Take setup from vhost.template rather than default(-ssl)
VR cloud-early-config: Commonize Apache2 common setup
VR cloud-early-config: Fix Apache2 alias cleanup
VR: consistent SSL setup, vhost is not an example, but a template
VR CsConfig: reintroduce old get_dns() behaviour for redundant non-VPC's
VR CsAddress fixes: * cleanup imports, * fix to_str(), * improve & fix service post_config logic * don't arpPing when there's no gateway
VR CsApp: Expose config to classes, move vhost confs to proper location, allow for multiple IP's per intf, sanitize servername, don't open port 53 if no DNS is foreseen
VR CsConfig: Add is_router(), is_dns(), has_dns(), has_metadata(), use_extdns(), fix get_dns() with use_extdns()
VR CsDhcp: allow multiple ranges & finite lease time (fixes CLOUDSTACK-8303)
VR CsGuestNetwork obey useextdns
VR merge.py ipalias fix & dhcpconfig stub notification
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
* 4.7:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storageIn turn this also fixes VM migration with local storage
This PR is created against 4.7 and can be forward merged to future branches also.
* pr/1596:
CLOUDSTACK-9353: [XenServer] Fixed VM migration with storage
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9404 Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted.
Issue: https://issues.apache.org/jira/browse/CLOUDSTACK-9404
In this example, I created rules with the port numbers the same as the rule numbers.
Chain ACL_INBOUND_eth2 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 225.0.0.50
ACCEPT all -- anywhere vrrp.mcast.net
DROP tcp -- anywhere anywhere tcp dpt:netstat
DROP tcp -- anywhere anywhere tcp dpt:10
DROP tcp -- anywhere anywhere tcp dpt:5
DROP tcp -- anywhere anywhere tcp dpt:3
DROP tcp -- anywhere anywhere tcp dpt:2
DROP all -- anywhere anywhere
We can see above that the rules are inverted.
After the fix:
Chain ACL_INBOUND_eth2 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 225.0.0.50
ACCEPT all -- anywhere vrrp.mcast.net
DROP tcp -- anywhere anywhere tcp dpt:2
DROP tcp -- anywhere anywhere tcp dpt:3
DROP tcp -- anywhere anywhere tcp dpt:5
DROP tcp -- anywhere anywhere tcp dpt:10
DROP tcp -- anywhere anywhere tcp dpt:netstat
DROP all -- anywhere anywhere
* pr/1581:
Added ASF license to unit test file
Added unit test to verify ordering
Fixed ordering of network ACL rules being sent to the VR. The comparator was inverted
Signed-off-by: Will Stevens <williamstevens@gmail.com>
The resize volume is support on all major hypervisors (Xen, VMware, KVM).
The hypervisor key is returned by the list volumes response only for admins
but not for users or domain admin users. This removes the check, as the operation
is supported on all major hypervisors that CloudStack supports.
With this changes all users would see resize volume button in the UI.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
in BVT
Bug-Id:## CLOUDSTACK-9328
[CLOUDSTACK-9328]: Made changes as per the review comment from Shwetaag
[CLOUDSTACK-9328]: Made changes based on the CI results
CLOUDSTACK-9399 : NPE during deletion of host when clusterId is nullIn most network plugins, there's a Resource class which will handle the communication with the actual device / underlaying client / ... They're configured as a host, so ACS is able to send commands towards it.
When they're configured as a host, the clusterId is not filled in since it's not relevant. Hence, the NPE while deleting this host because of ```long clusterId = host.getClusterId();```
* pr/1585:
Nuage VSP : Enhancing Marvin test coverage
CLOUDSTACK-9399 : Marvin test coverage for Nuage VSP device CRUD operations
CLOUDSTACK-9399 : NPE during deletion of host when clusterId is null
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[BLOCKER][FIX] CLOUDSTACK-9409: Add role_id to cloud_usage.accountAdds role_id column to cloud_usage.account, fixes UsageDaoImpl to insert
Accounts with role_id from account table. Without the fix, usage server fails to work.
This fixes a *blocker* for 4.9.0 reported by @nvazquez (thanks!)
/cc @swill @nvazquez for review and merge thanks.
* pr/1584:
CLOUDSTACK-9409: Add role_id to cloud_usage.account
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Adds role_id column to cloud_usage.account, fixes UsageDaoImpl to insert
Accounts with role_id from account table.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
* Take setup from vhost.template rather than default(-ssl)
* should move into Python CS code as well
* Move CORS setup to separate conf
* Modify vhost template to Optionally include the cors file
* Add NameVirtualHost to vhost template for feature parity with ports.conf
* Take setup from vhost.template rather than default(-ssl)
