Fix public IPs not being removed from the VR when deprovisionedThis PR replaces #1706. It does not remove the IP from the database, but it does deprovision the IP correctly from the VR when the public IP is removed.
* pr/1907:
Fix public IPs not being removed from the VR when deprovisioned
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9757: Fixed issue in traffic from additional public subnetAcquire ip from additional public subnet and configure nat on that ip.
After this pick any from that network and access additional public subnet from this vm. Traffic is supposed to go via additional public subnet interface in the VR.
* pr/1922:
CLOUDSTACK-9757: Fixed issue in traffic from additional public subnet
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
* 4.9:
CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
CLOUDSTACK-9788: Fix exception listNetworks with pagesize=0
CLOUDSTACK-8663: Fixed various issues to allow VM snapshots and volume snapshots to exist together
Fix HVM VM restart bug in XenServer
[CLOUDSTACK-9793] Faster IP in subnet checkThis change removes the conversion from IPNetwork to list in one of the router scripts. This makes the router faster at processing static NAT rules, which can prevent timeouts when attaching or detaching IPs.
With the `list` conversion, it has to potentially check a list of 65536 IP strings multiple times. We assume that the comparison implemented in the IPNetwork is far more efficient. We have seen speed-up from 218 seconds to enable static NAT with 18 IPs on the router to 2 or 3 seconds by removing this cast. This also fixes a potential bug where adding IPs to a router time out because the scripts are taking too long. 218 seconds, for example, is beyond the timeout on the KVM agent for script execution, and then all enableStaticNat operations will fail.
* pr/1948:
CLOUDSTACK-9793: Faster ip in subnet check
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
* rotate both daily and by size by using maxsize in stead of size
* decrease the max size to 10M for rsyslog files
* remove delaycompress for rsyslog files
* increase rotate to 10 for cloud.log
This change removes an unnecessary conversion from IPNetwork
to list in one of the router scripts. This makes the router
faster at processing static NAT rules, which can prevent
timeouts when attaching or detaching IPs.
Updated StrongSwan VPN ImplementationThis PR is a merge of @jayapalu changes in #872 and the changes I had to make to get the functionality working.
I have done pretty extensive testing of this code so far and we are looking to be in pretty good shape. One thing to note is that a `Diffie-Hellman` group **is required** in order for this feature to work correctly. It is not highlighted in the tests below, but I have shown that the `PFS` is not required for this feature to work. In #872 I have shown a more exhaustive set of tests of this code, but I have limited this set of tests to a recommended `IKE` and `ESP` configuration in order to reduce the noise and test the other areas of functionality.
**Test Results**
I am testing this functionality by creating two VPCs with VMs in each and creating a S2S VPN connection between the two VPCs. Then I SSH into a VM in one VPC and I ping the private IP of a VM in the other VPC. Then I tear it down and try a different configuration.
_Setup_
```
VPC 1 VPC 2
===== =====
VPN Gateway VPN Gateway
VPN Customer Gateway VPN Customer Gateway
VPN Connection <---> VPN Connection
- Passive = True - Passive = False
```
_Legend_
`SKIP` => At least one of the VPN Connections did not come up, so no test was run.
`OK` => The ping test was successful over the S2S VPN connection.
`FAIL` => The ping test failed over the S2S VPN connection.
`Passive` => Specifies if either the `<vpc_1> : <vpc_2>` sides of the VPN Connection is set to passive.
`Conn State` => Specifies the connection status of the `<vpc_1> : <vpc_2>` VPN Connection in the UI.
`Requires Reset` => If the ping test does not result in an `OK`, then a VPN Connection Reset is performed on either `<vpc_1> : <vpc_2>` sides of the VPN Connection based on which side is not showing `Connected`. The results in the `Status` column is the final result after the reset is performed.
_Results_
```
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| Status | IKE & ESP | DPD | Encap | IKE Life | ESP Life | Passive | Conn State | Requires Reset |
+========+======================+=======+=======+==========+==========+===============+=============================+================+
| OK | aes128-sha1;modp1536 | True | False | 86400 | 3600 | True : False | Disconnected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | True | 86400 | 3600 | True : False | Disconnected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | False | | 3600 | True : False | Disconnected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | False | 86400 | | True : False | Disconnected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | False | | | True : False | Disconnected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | False | 86400 | 3600 | False : False | Connected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | False | 86400 | 3600 | True : True | Disconnected : Disconnected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | True | False | 86400 | 3600 | False : True | Connected : Disconnected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | False | False | 86400 | 3600 | False : False | Connected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | False | False | 86400 | 3600 | True : False | Disconnected : Connected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | False | False | 86400 | 3600 | True : True | Disconnected : Disconnected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| OK | aes128-sha1;modp1536 | False | False | 86400 | 3600 | False : True | Connected : Disconnected | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| SKIP | aes128-sha1 | True | False | 86400 | 3600 | True : False | Disconnected : Error | True : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| SKIP | aes128-sha1 | False | False | 86400 | 3600 | True : False | Disconnected : Error | True : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| FAIL | aes128-sha1 | True | False | 86400 | 3600 | True : True | Disconnected : Disconnected | True : True |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
| SKIP | aes128-sha1 | True | False | 86400 | 3600 | False : False | Connected : Error | False : False |
+--------+----------------------+-------+-------+----------+----------+---------------+-----------------------------+----------------+
```
* pr/1741:
complete implementation of the StrongSwan VPN feature
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
The password server in RVRs has wrong parameters as the gateway of guest nics is None.
In this case, we should get the gateway from /var/cache/cloud/cmdline.
As part of the bug 'CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly'
issue of mismatch of traffic type represented by 'nw_type' in config sent by management server in
ip_associations.json and how it is persisted in the ips.json data bag are differnet,
is addressed, however missed the change in final merge.
this bug is to add the functionality in cs_ip.py, to lower the traffic type sent by management server before persisting in the ips.json databag
-when processing static nat rule, add a mangle table rule, to mark the traffic
from the guest vm when it has associated static nat rule so that traffic gets
routed using the route tabe of the device which has public ip associated
-fix the case where nic_device_id is empty when ip is getting disassociated
resulting in empty deviceid in ips.json
-add utility methods in CsRule, and CsRoute to add 'ip rule' and 'ip route' rules respectivley
-ensure traffic from all public interfaces are connection marked with device number, and restored
for the reverse traffic. use the connection marked number to do device specific routing table lookup
fill the device specific routing table with default route
-component tests for testing multiple public interfaces of VR
CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network when guest VM has nic's in more than one guest network set the tag for each host in /etc/dhcphosts.txt, and use the tag to add exception in /etc/dhcpopts.txt to prevent sending default route, dns server in case if the nic is in non-default network
this was the behaviour with edithosts.sh prior to 4.6
* pr/1766:
CLOUDSTACK-9598: wrong defaut gateway for the nic in non-default network
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This commit adds a additional VirtIO channel with the name
'org.qemu.guest_agent.0' to all Instances.
With the Qemu Guest Agent the Hypervisor gains more control over the Instance if
these tools are present inside the Instance, for example:
* Power control
* Flushing filesystems
* Fetching Network information
In the future this should allow safer snapshots on KVM since we can instruct the
Instance to flush the filesystems prior to snapshotting the disk.
More information: http://wiki.qemu.org/Features/QAPI/GuestAgent
Keep in mind that on Ubuntu AppArmor still needs to be disabled since the default
AppArmor profile doesn't allow libvirt to write into /var/lib/libvirt/qemu
This commit does not add any communication methods through API-calls, it merely
adds the channel to the Instances and installs the Guest Agent in the SSVMs.
With the addition of the Qemu Guest Agent channel a second channel appears in /dev
on a SSVM as a VirtIO port.
The order in which the ports are defined in the XML matters for the naming inside
the SSVM VM and by not relying on /dev/vportXX but looking for a static name the
SSVM still boots properly if the order in the XML definition is changed.
A SSVM with both ports attached will have something like this:
root@v-215-VM:~# ls -l /dev/virtio-ports
total 0
lrwxrwxrwx 1 root root 11 May 13 21:41 org.qemu.guest_agent.0 -> ../vport0p2
lrwxrwxrwx 1 root root 11 May 13 21:41 v-215-VM.vport -> ../vport0p1
root@v-215-VM:~# ls -l /dev/vport*
crw------- 1 root root 251, 1 May 13 21:41 /dev/vport0p1
crw------- 1 root root 251, 2 May 13 21:41 /dev/vport0p2
root@v-215-VM:~#
In this case the SSVM port points to /dev/vport0p1, but if the order in the XML
is different it might point to /dev/vport0p2
By looking for a portname with a pre-defined pattern in /dev/virtio-ports we
do not rely on the order in the XML definition.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
CLOUDSTACK-9498: VR CsFile search utility methods fail when search stThere is no real use of python 're' module in CsFile.py utility methods searchString, deleteLine. Regular string search is sufficient. These methods are used only for VPN user add/delete. Since VPN user password can have python 're' module meta characters, it interfere with search functionality.
Replacing re.search() with regular string search instead.
Change is confined to VPN add/delete users. Have run the test/integration/component/test_vpn_users.py
VPN remote access user limit tests ... === TestName: test_01_VPN_user_limit | Status : SUCCESS ===
ok
Test create VPN when L2TP port in use ... === TestName: test_02_use_vpn_port | Status : SUCCESS ===
ok
Test create NAT rule when VPN when L2TP enabled ... === TestName: test_03_enable_vpn_use_port | Status : SUCCESS ===
ok
Test add new users to existing VPN ... === TestName: test_04_add_new_users | Status : SUCCESS ===
ok
Test add duplicate user to existing VPN ... === TestName: test_05_add_duplicate_user | Status : SUCCESS ===
ok
Test as global admin, add a new VPN user to an existing VPN entry ... === TestName: test_06_add_VPN_user_global_admin | Status : SUCCESS ===
ok
Test as domain admin, add a new VPN user to an existing VPN entry ... === TestName: test_07_add_VPN_user_domain_admin | Status : SUCCESS ===
ok
* pr/1680:
CLOUDSTACK-9498: VR CsFile search utility methods fail when search string has 're' meta chars, and causing VPN user add/deelte to fail
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
when guest VM has nic's in more than one guest network set the tag for
each host in /etc/dhcphosts.txt, and use the tag to add exception in
/etc/dhcpopts.txt to prevent sending default route, dns server in case if the nic is in non-default network
this was the behaviour with edithosts.sh prior to 4.6
added new test case test_router_dhcp_opts to test DHCP option file use of cloudstack
The VR executes a ip route flush command as part of configurations. This command performs a
DNS lookup on the VR hostname. Since the VR does not have a DNS entry, the ip command would
wait 5 seconds before timing out and executing the flush operation. This fix adds the VR
hostname to /etc/hosts mapped to 127.0.0.1 to answer the DNS lookup – reducing the
execution time.
CLOUDSTACK-8326: Always fill UDP checksums in DHCP replies in VRIn some cases the UDP checksums in packets from DHCP servers are
incorrect. This is a problem for some DHCP clients that ignore
packets with bad checksums. This patch inserts an iptables rule
to ensure DHCP servers always send packets with correct checksums.
Due to this bug DHCP offers are sometimes not accepted by Instances.
The end-result without this fix is no connectivity for the Instance
due to the lack of a IPv4 address.
Signed-off-by: Wido den Hollander <wido@widodh.nl>
* pr/1743:
CLOUDSTACK-8326: Always fill UDP checksums in DHCP replies in VR
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
In some cases the UDP checksums in packets from DHCP servers are
incorrect. This is a problem for some DHCP clients that ignore
packets with bad checksums. This patch inserts an iptables rule
to ensure DHCP servers always send packets with correct checksums.
Due to this bug DHCP offers are sometimes not accepted by Instances.
The end-result without this fix is no connectivity for the Instance
due to the lack of a IPv4 address.
This is also commited in OpenStack:
- https://github.com/projectcalico/felix/issues/40
- https://review.openstack.org/148718
- https://bugzilla.redhat.com/show_bug.cgi?id=910619
Signed-off-by: Wido den Hollander <wido@widodh.nl>
CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory
* pr/1744:
CLOUDSTACK-9183: bash: /opt/cloud/bin/getRouterAlerts.sh: No such file or directory
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
're' meta chars, and causing VPN user add/deelte to fail
-there is no real use of python 're' in CsFile.py utility methods searchString, deleteLine
Replacing with regular string search instead.
-modifying the smoke test for VPN user add/delete to have all permissable chars
resulting in internal LB vm not come up
parsing cmd_line to create 'ips' data bag, never handled internal lb vm, but still
worked due to another bug. support for internal lb vm is added with this fix
CLOUDSTACK-9480, CLOUDSTACK-9495 fix egress rule incorrect behaviorWhen 'default egress policy' is set to 'allow' in the network offering, any egress rule that is added will 'deny' the traffic overriding the default behaviour.
Conversely, when 'default egress policy' is set to 'deny' in the network offering, any egress rule that is added will 'allow' the traffic overriding the default behaviour.
While this works for 'tcp', 'udp' as expected, for 'icmp' protocol its always set to ALLOW. This patch keeps all protocols behaviour consistent.
Results of running test/integration/component/test_egress_fw_rules.py. With out the patch test_02_egress_fr2 test was failing. This patch fixes the test_02_egress_fr2 scenario.
-----------------------------------------------------------------------------------------------------
Test By-default the communication from guest n/w to public n/w is NOT allowed. ... === TestName: test_01_1_egress_fr1 | Status : SUCCESS ===
ok
Test By-default the communication from guest n/w to public n/w is allowed. ... === TestName: test_01_egress_fr1 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_1_egress_fr2 | Status : SUCCESS ===
ok
Test Allow Communication using Egress rule with CIDR + Port Range + Protocol. ... === TestName: test_02_egress_fr2 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_1_egress_fr3 | Status : SUCCESS ===
ok
Test Communication blocked with network that is other than specified ... === TestName: test_03_egress_fr3 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_1_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the Firewall_Rules DB table ... === TestName: test_04_egress_fr4 | Status : SUCCESS ===
ok
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule and check the IP tables ... SKIP: Skip
Test Create Egress rule without CIDR ... === TestName: test_06_1_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without CIDR ... === TestName: test_06_egress_fr6 | Status : SUCCESS ===
ok
Test Create Egress rule without End Port ... === TestName: test_07_1_egress_fr7 | Status : EXCEPTION ===
ERROR
Test Create Egress rule without End Port ... === TestName: test_07_egress_fr7 | Status : SUCCESS ===
ok
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Port Forwarding and Egress Conflict ... SKIP: Skip
Test Delete Egress rule ... === TestName: test_09_1_egress_fr9 | Status : SUCCESS ===
ok
Test Delete Egress rule ... === TestName: test_09_egress_fr9 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_1_egress_fr10 | Status : SUCCESS ===
ok
Test Invalid CIDR and Invalid Port ranges ... === TestName: test_10_egress_fr10 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_1_egress_fr11 | Status : SUCCESS ===
ok
Test Regression on Firewall + PF + LB + SNAT ... === TestName: test_11_egress_fr11 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_1_egress_fr12 | Status : SUCCESS ===
ok
Test Reboot Router ... === TestName: test_12_egress_fr12 | Status : EXCEPTION ===
ERROR
Test Redundant Router : Master failover ... === TestName: test_13_1_egress_fr13 | Status : SUCCESS ===
ok
Test Redundant Router : Master failover ... === TestName: test_13_egress_fr13 | Status : SUCCESS ===
ok
-----------------------------------------------------------------------------------------------------
* pr/1666:
fix egress rule incorrect behavior
Signed-off-by: Rajani Karuturi <rajani.karuturi@accelerite.com>
CLOUDSTACK-9480: Egress Firewall: Incorrect use of Allow/Deny for ICMP
fix ensures, ICMP, TCP, UDP are handled similalry w.r.t egress rule action
CLOUDSTACK-9495: Egress rules functionalty broken when protocol=all specified
when protocol=all specified, CIDR was ignored. Fix ensures if CIDR is specified
its always used in configuring iptable rules
2 new test cased to test /32 CIDR
