716 Commits

Author SHA1 Message Date
dahn
2a88b11132 CLOUDSTACK-9336 surround the execution of baremetal-vr.py with condition 2016-04-05 13:28:39 +02:00
Boris Schrijver
30741a3309 Split the cidr lists so we won't hit the iptables-resture limits 2016-02-05 21:11:59 +01:00
Boris Schrijver
709be45cf0 Check the existence of 'forceencap' parameter before use 2016-02-05 21:06:53 +01:00
Boris Schrijver
18d5cd2855 Do not load previous firewall rules as we replace everyhing anyway 2016-02-05 21:04:06 +01:00
Boris Schrijver
eb9706b655 Wait for dnsmasq to finish restart 2016-02-05 12:02:58 +01:00
Boris Schrijver
69e9348900 Remove duplicate spaces, and thus duplicate rules. 2016-02-05 12:02:57 +01:00
Boris Schrijver
b857f79fcb Restore iptables at once using iptables-restore instead of calling iptables numerous times 2016-02-05 12:02:57 +01:00
Boris Schrijver
0f75042566 Add iptables copnversion script.
Source: https://raw.githubusercontent.com/sl0/conv/master/iptables_converter.py
2016-02-05 12:02:57 +01:00
Remi Bergsma
7017a829ea Merge release branch 4.8 to master
* 4.8:
  Display hostname the VPC router runs on
  CLOUDSTACK-9266: Make deleting static routes in private gw work
  CLOUDSTACK-9264: Make /32 static routes for private gw work
2016-02-04 09:27:40 +01:00
Remi Bergsma
be89f64731 Merge release branch 4.7 to 4.8
* 4.7:
  Display hostname the VPC router runs on
  CLOUDSTACK-9266: Make deleting static routes in private gw work
  CLOUDSTACK-9264: Make /32 static routes for private gw work
2016-02-04 09:26:53 +01:00
Remi Bergsma
a243339faa Merge pull request #1386 from remibergsma/fix-del-private-gw-route
CLOUDSTACK-9266: Make deleting static routes in private gw workThe to-be-deleted static routes were removed from the json file, instead of putting them there with revoke=true. The script that parses the json now doesn't find it and thus does not delete it.

Example after adding/removing some:
```
root@r-3-VM:/var/cache/cloud# cat /etc/cloudstack/staticroutes.json
{
    "1.2.3.0/24": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.2.3.0/24",
        "revoke": true
    },
    "1.2.3.4/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.2.3.4/32",
        "revoke": true
    },
    "1.2.33.3/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.2.33.3/32",
        "revoke": true
    },
    "1.22.2.2/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "1.22.2.2/32",
        "revoke": true
    },
    "10.1.2.1/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "10.1.2.1/32",
        "revoke": true
    },
    "10.1.200.0/25": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "10.1.200.0/25",
        "revoke": true
    },
    "10.11.12.13/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "10.11.12.13/32",
        "revoke": true
    },
    "172.16.1.3/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "172.16.1.3/32",
        "revoke": true
    },
    "172.16.15.14/32": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "172.16.15.14/32",
        "revoke": false
    },
    "172.16.17.0/25": {
        "gateway": "172.16.0.1",
        "ip_address": "172.16.0.2",
        "network": "172.16.17.0/25",
        "revoke": false
    },
    "id": "staticroutes"
}
```

This results in:
```
root@r-3-VM:/var/cache/cloud# ip route show
default via 192.168.23.1 dev eth1
169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.1.67
172.16.0.0/24 dev eth2  proto kernel  scope link  src 172.16.0.2
172.16.15.14 via 172.16.0.1 dev eth2
172.16.17.0/25 via 172.16.0.1 dev eth2
192.168.23.0/24 dev eth1  proto kernel  scope link  src 192.168.23.4
```

Two static routes left, the rest deleted:
```
172.16.15.14 via 172.16.0.1 dev eth2
172.16.17.0/25 via 172.16.0.1 dev eth2
```

That also matches the UI:

<img width="1327" alt="screen shot 2016-01-30 at 06 34 06" src="https://cloud.githubusercontent.com/assets/1630096/12693933/83e67d80-c71b-11e5-9241-9f478522b7a4.png">

* pr/1386:
  CLOUDSTACK-9266: Make deleting static routes in private gw work

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-02-04 09:13:00 +01:00
Remi Bergsma
a40415604a CLOUDSTACK-9266: Make deleting static routes in private gw work 2016-01-30 06:25:00 +01:00
Remi Bergsma
78c43669e5 CLOUDSTACK-9264: Make /32 static routes for private gw work 2016-01-29 19:25:29 +01:00
Remi Bergsma
e93fd548d6 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9256 add unique key for static routes in json
2016-01-26 17:07:37 +01:00
Remi Bergsma
48672c7e4a Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9256 add unique key for static routes in json
2016-01-26 17:07:04 +01:00
Remi Bergsma
782d5d73f5 Merge pull request #1364 from remibergsma/CLOUDSTACK-9256
CLOUDSTACK-9256 add unique key for static routes in jsonStatic routes that are being set do not show up in the static_routes.json file. The reason for this is that the index that is used, is the gateway address, which is not unique. Hence stuff is overwritten and lost.

Ping @borisroman @wilderrodrigues @DaanHoogland

* pr/1364:
  CLOUDSTACK-9256 add unique key for static routes in json

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 17:06:21 +01:00
Remi Bergsma
43ab98d823 Updating pom.xml version numbers for release 4.9.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 15:12:20 +01:00
Remi Bergsma
32fcc47117 Updating pom.xml version numbers for release 4.8.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 09:39:00 +01:00
Remi Bergsma
a4ac9d3fff Updating pom.xml version numbers for release 4.7.2-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-26 09:33:56 +01:00
Remi Bergsma
8c60ad214f Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9254: Make longer names display pretty
  CLOUDSTACK-9245 - Deletes ACL items when destroying the VPC or deleting the ACL itself
  CLOUDSTACK-9245 - Formatting NetworkACLServiceImpl class
  CLOUDSTACK-9245 - Formatting VpcManagerImpl class
  CLOUDSTACK-9245 - Formatting NetworkACLManagerImpl class
  More VR performance!
2016-01-26 08:39:28 +01:00
Remi Bergsma
62fc6fef97 CLOUDSTACK-9256 add unique key for static routes in json 2016-01-25 17:17:55 +01:00
Remi Bergsma
3d2bb285f0 Merge pull request #1356 from borisroman/4.7-vr-performance-wip
More VR performance!Ping @remibergsma

Integration tests runnning now!

* pr/1356:
  More VR performance!

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-23 19:16:48 +01:00
Boris Schrijver
3eceb60f5c More VR performance! 2016-01-21 11:51:03 +01:00
Remi Bergsma
62f218b7bd Updating pom.xml version numbers for release 4.8.0
Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-20 23:43:35 +01:00
Remi Bergsma
6f9215cf39 Merge release branch 4.7 to master
* 4.7:
  Refactor public ip retrieval into method
  CLOUDSTACK-9244 Fix setting up RFC1918 routes
  CLOUDSTACK-9239 throw exception on deprecated command
  Enhance VR performance by selectively executing tasks instead of brute-forcing
  CLOUDSTACK-9236: Load Balancing Health Check button displayed when non-NetScaler offering is used
2016-01-20 14:18:09 +01:00
Remi Bergsma
ed23ac627a Merge pull request #1346 from borisroman/4.7-vr-performance
Enhance VR performanceWill post integration test results soon!

Ping @wilderrodrigues @remibergsma

* pr/1346:
  Enhance VR performance by selectively executing tasks instead of brute-forcing

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-20 14:17:37 +01:00
Daan Hoogland
91f7fc05b6 Refactor public ip retrieval into method 2016-01-19 23:02:26 +01:00
Remi Bergsma
d601570053 CLOUDSTACK-9244 Fix setting up RFC1918 routes 2016-01-19 18:29:31 +01:00
Boris Schrijver
73c0242df3 Enhance VR performance by selectively executing tasks instead of brute-forcing 2016-01-18 11:40:59 +01:00
Daan Hoogland
addb26455a Merge release branch 4.7 to master
* 4.7:
  CLOUDSTACK-9154 - Sets the pub interface down when all guest nets are gone
  CLOUDSTACK-9187 - Makes code ready for more something like ethXXXX, if we ever get that far
  CLOUDSTACK-9188 -  Reads network GC interval and wait from configDao
  CLOUDSTACK-9187 - Fixes interface allocation to VRRP instances
  CLOUDSTACK-9187 - Adds test to cover multiple nics and nic removal
  CLOUDSTACK-9154 - Adds test to cover nics state after GC
  CLOUDSTACK-9154 - Returns the guest iterface that is marked as added

 Conflicts:
	engine/orchestration/src/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java
2016-01-17 20:04:11 +01:00
Remi Bergsma
ff89587fd1 Merge pull request #1277 from ekholabs/fix/4.7-rvpc-net-gc-CLOUDSTACK-9154
[4.7] Critical VPCVR issues fixed: CLOUDSTACK-9154; CLOUDSTACK-9187; and CLOUDSTACK-9188This PR applies the same fixes as in the PR #1259, but against branch 4.7.

Please refer to PR #1259 for the tests results and all the comments already made there.

Issues fixed are:

* CLOUDSTACK-9154: rVPC doesn't recover from cleaning up of network garbage collector
* CLOUDSTACK-9187: rVPC routers in Master/Master due to concurrency problem when writing the keepalivd.conf
* CLOUDSTACK-9188: NetworkGarbageCollector is not using gc.interval and gc.wait from settings

Those changes have been covered by 2 new tests added to ```smoke/test_vpc_redundant.py```:

* test_04_rvpc_network_garbage_collector_nics
* test_05_rvpc_multi_tiers

The test ```test_04_rvpc_network_garbage_collector_nics``` depends on the global settings for the network.gc.interval and gc.wait. If one wants the test to run quicker, please change the settings (default is 600 seconds for each) and restart the Management Server before running the tests. I would suggest to set it to 60 seconds.

In addition, the NetworkGarbageCollector was redefining the settings above mentioned and not reading their values through ConfigDao. Due to that, the settings were not being applied properly and the test was waiting to long to check the VPC routers.

* pr/1277:
  CLOUDSTACK-9154 - Sets the pub interface down when all guest nets are gone
  CLOUDSTACK-9187 - Makes code ready for more something like ethXXXX, if we ever get that far
  CLOUDSTACK-9188 -  Reads network GC interval and wait from configDao
  CLOUDSTACK-9187 - Fixes interface allocation to VRRP instances
  CLOUDSTACK-9187 - Adds test to cover multiple nics and nic removal
  CLOUDSTACK-9154 - Adds test to cover nics state after GC
  CLOUDSTACK-9154 - Returns the guest iterface that is marked as added

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-17 19:12:43 +01:00
Remi Bergsma
24818af23a Merge release branch 4.7 to master
* 4.7:
  CLOUDSTACK-9222 Prevent cloud.log.1 filling up the disk
  Add integration test for restartVPC with cleanup, and Private Gateway enabled.
  Nullpointer Exception in NicProfileHelperImpl
2016-01-16 20:25:29 +01:00
Remi Bergsma
c13c5540b0 Merge pull request #1329 from remibergsma/delay_compress_fix
CLOUDSTACK-9222 Prevent cloud.log.1 filling up the diskDelay Compress results in more space usage than needed. Since we have copy truncate we don't need it.

* pr/1329:
  CLOUDSTACK-9222 Prevent cloud.log.1 filling up the disk

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 20:13:49 +01:00
Remi Bergsma
75b68c6829 Merge release branch 4.7 to master
* 4.7:
  Fix unable to setup more than one Site2Site VPN Connection
  FIX S2S VPN rVPC: Check only redundant routers in state MASTER
  PEP8 of integration/smoke/test_vpc_vpn
  Add S2S VPN test for Redundant VPC
  Make integration/smoke/test_vpc_vpn Hypervisor independant
  FIX VPN: non-working ipsec commands
  [UI] MADNESS
  [DB] Add force_encap field to s2s_customer_gateway table
  [ROUTER] Add forceencaps field to python router ipsec config method
  [TEST] unittest needs rework
  [MARVIN] Add forceencap field to VpnCustomerGateway class in marvin base
  [CORE] Add Force UDP Encapsulation option to Site2Site VPN
  CLOUDSTACK-9186: Root admin cannot see VPC created by Domain admin user
  CLOUDSTACK-9192: UpdateVpnCustomerGateway is failing
  CLOUDSTACK-6485 prevent ip asignment of private gw iface
  CLOUDSTACK-9204 Do not error when staticroute is already gone
  make both check lines consistent
  CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh
  CLOUDSTACK-9202 Bump ssh timeout
2016-01-16 19:54:41 +01:00
Remi Bergsma
5fdc77e16c Merge pull request #1276 from michaelandersen/fix/site2sitevpn
[4.7] FIX Site2SiteVPN on redundant VPCThis PR:
- fixes the inability to setup more than one Site2Site VPN connection from a VPC
- fixes starting of Site2Site VPN on redundant VPC
- fixes Site2Site VPN state checking on redundant VPC
- improves the vpc_vpn test to allow multple hypervisors
- adds an integration test for Site2Site VPN on redundant VPC

Tested it on 4.7 single Xen server zone:

command:
```
nosetests --with-marvin --marvin-config=/data/shared/marvin/mct-zone1-xen1.cfg -a tags=advanced,required_hardware=true /tmp/test_vpc_vpn.py
```

results:
```
Test Site 2 Site VPN Across redundant VPCs ... === TestName: test_01_redundant_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_01_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test Site 2 Site VPN Across VPCs ... === TestName: test_01_vpc_site2site_vpn | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 3 tests in 1490.076s

OK
```

also performed numerous manual inspections of state of VPN connections and connectivity between VPC's

* pr/1276:
  Fix unable to setup more than one Site2Site VPN Connection
  FIX S2S VPN rVPC: Check only redundant routers in state MASTER
  PEP8 of integration/smoke/test_vpc_vpn
  Add S2S VPN test for Redundant VPC
  Make integration/smoke/test_vpc_vpn Hypervisor independant
  FIX VPN: non-working ipsec commands

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 19:51:46 +01:00
Remi Bergsma
66a933afab Merge pull request #1296 from remibergsma/fix-checkrouter-script47
CLOUDSTACK-9181 Prevent syntax error in checkrouter.shAdded quotes to prevent syntax errors in weird situations.

Error seen in mgt server:
```
2015-12-15 14:30:32,371 DEBUG [c.c.a.m.AgentManagerImpl] (RedundantRouterStatusMonitor-7:ctx-0dd8ef3e) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: UNKNOWN
/opt/cloud/bin/checkrouter.sh: line 28: [: =: unary operator expected
/opt/cloud/bin/checkrouter.sh: line 31: [: =: unary operator expected
```
Cause:
```
root@r-1191-VM:/opt/cloud/bin# ./checkrouter.sh
./checkrouter.sh: line 28: [: =: unary operator expected
./checkrouter.sh: line 31: [: =: unary operator expected
Status: UNKNOWN
```

Somehow a nic was missing.

After fix the script can handle this:

```
root@r-1191-VM:/opt/cloud/bin# ./checkrouter.sh
Status: UNKNOWN
```

The other states are also reported fine:
```
root@r-1191-VM:/opt/cloud/bin# ./checkrouter.sh
Status: MASTER
```

```
root@r-1192-VM:/opt/cloud/bin# ./checkrouter.sh
Status: BACKUP
```

While at it, I also removed the INTERFACES variable/constant as it was only used once and hardcoded the second time. Now both are hardcoded and easier to read.

* pr/1296:
  make both check lines consistent
  CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 19:46:06 +01:00
Remi Bergsma
b4ebfb640e Merge pull request #1298 from remibergsma/staticroute_delete_47
CLOUDSTACK-9204 Do not error when staticroute is already goneWhen deleting a static route fails because it isn't there any more (KeyError), it should succeed instead.

Error seen:
```
[INFO] Processing JSON file static_routes.json.1451560145
Traceback (most recent call last):
File "/opt/cloud/bin/update_config.py", line 140, in <module>
process_file()
File "/opt/cloud/bin/update_config.py", line 52, in process_file
qf.load(None)
File "/opt/cloud/bin/merge.py", line 258, in load
proc = updateDataBag(self)
File "/opt/cloud/bin/merge.py", line 91, in _init_
self.process()
File "/opt/cloud/bin/merge.py", line 131, in process
dbag = self.process_staticroutes(self.db.getDataBag())
File "/opt/cloud/bin/merge.py", line 179, in process_staticroutes
return cs_staticroutes.merge(dbag, self.qFile.data)
File "/opt/cloud/bin/cs_staticroutes.py", line 26, in merge
del dbag[key]
KeyError: u'192.168.0.3'
```

* pr/1298:
  CLOUDSTACK-9204 Do not error when staticroute is already gone

Signed-off-by: Remi Bergsma <github@remi.nl>
2016-01-16 19:43:24 +01:00
Michael Andersen
435a98cd3f Fix unable to setup more than one Site2Site VPN Connection 2016-01-12 11:24:35 +01:00
Michael Andersen
dfa924bdee FIX VPN: non-working ipsec commands 2016-01-12 11:24:34 +01:00
Remi Bergsma
867be4145d CLOUDSTACK-9222 Prevent cloud.log.1 filling up the disk
Delay Compress results in more space usage than needed. Since we have copy truncate we don't need it.
2016-01-11 21:09:08 +01:00
Michael Andersen
21acc95d57 [ROUTER] Add forceencaps field to python router ipsec config method 2016-01-07 19:27:42 +01:00
Remi Bergsma
779f4b36e0 CLOUDSTACK-9204 Do not error when staticroute is already gone
When deleting fails because it isn't there any morei (KeyError), it should succeed instead.
2015-12-31 14:11:06 +01:00
Remi Bergsma
8fb677027d make both check lines consistent
No need to make a variable, use it on one place and hardcode it on another.
2015-12-28 17:06:58 +01:00
Remi Bergsma
525949e027 CLOUDSTACK-9181 Prevent syntax error in checkrouter.sh 2015-12-28 17:06:43 +01:00
Wilder Rodrigues
5ef3144fdf CLOUDSTACK-9154 - Sets the pub interface down when all guest nets are gone
- Refactors the set_backup, set_master and set_fault methods to have better names for the variable
   - Increase the sleep on the test in order to wait for the routers to be ready. It's now 3 times the GC settings
2015-12-22 14:52:08 +01:00
Wilder Rodrigues
f5a6dee8dd CLOUDSTACK-9187 - Makes code ready for more something like ethXXXX, if we ever get that far
- Adds log info to NetworkOrchestrator in order to make the work of the Net-Scavenger more visible.
2015-12-22 14:52:07 +01:00
Wilder Rodrigues
c99d6f18c9 CLOUDSTACK-9187 - Fixes interface allocation to VRRP instances 2015-12-22 14:52:06 +01:00
Wilder Rodrigues
7988f51ac0 CLOUDSTACK-9154 - Returns the guest iterface that is marked as added
- Force a restart of keepalived if conntrackd is not running or configuration has changed
2015-12-22 14:52:05 +01:00
Remi Bergsma
8f5a2920e8 Updating pom.xml version numbers for release 4.8.0-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-21 22:09:31 +01:00
Remi Bergsma
5b112408c7 Updating pom.xml version numbers for release 4.7.1-SNAPSHOT
Signed-off-by: Remi Bergsma <github@remi.nl>
2015-12-21 19:55:04 +01:00