[4.6] CLOUDSTACK-9113: skip vm with inconsistent state when getVmStats/getVmDiskStatson KVM, if there is a vm has inconsistent state between hypervisor and db, the getVmStat will terminate and return null, all vm stats will not be updated.
we should skip the vm which has inconsistent state, and continue on others.
* pr/1182:
CLOUDSTACK-9113: skip vm with inconsistent state when getVmStats/getVmDiskStats
Signed-off-by: Remi Bergsma <github@remi.nl>
Show actual diff in commits after merge with git-pr / git-fwd-mergeThis shows the diff in commits after using `git-pr` and `git-fwd-merge` tools, like this:
```
44e8c92 Merge pull request #1226 from borisroman/CLOUDSTACK-9148
0554610 Removed .pydevproject from plugin kvm hypervisor.
```
That helps in quickly seeing what change is applied and whether it makes sense. It doesn't touch the actual code base. I'm using this patch for some time already.
* pr/1227:
Show actual diff in commits after merge
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9135 - As a Developer I want the test_internal_lb.py to test Redundant VPCsThis PR refactors the existing tests and adds tests to cover internal LB with Redundant VPCs.
In order to make it possible I had to change the base.py, allowing multiple providers per service. This change requires the changes in the Java side which were dealt with PR #1215 .
* pr/1221:
CLOUDSTACK-9135 - Makes test compliant with Redundant VPCs
CLOUDSTACK-9135 - Makes possible to add multiple providers per service
CLOUDSTACK-9135 - Moves test into smoke directory
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-4374 - As a Developer I want to have HA enabled for routers that are part or a redundant network or VPCThis PR fixes the issue we were facing with crashed redundant routers not being picked up by the HA monitor due to an IF statement in the code.
* Redundant Routers are not the same as Haigh Available. Having a router HA is actually saying that the given router will be controlled by the High Availability monitor. Hence fix any problem we might face.
* pr/1222:
CLOUDSTACK-4374 - Adds HA capabilities to redundant routers
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9086: ACS allows to create isolated networks with invalide gateway IP address - Fixed and Test cases added
Problem: There was no check for the network and broadcast IP addresses in the case where we provide the gateway and netmask while creating the isolated network. As a result the provided IP gets assigned to the eth0 interface of the VR.
Note: This is in continuation of PR #1125 which I closed.
* pr/1188:
CLOUDSTACK-9086: ACS allows to create isolated networks with invalide gateway IP address - Fixed and Test cases added
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9122: latest credit entries should be incorporated when thefuture credit entries should be incorporated when the balance is calculated
1. Some code is refactored so that unit tests (TODO in future) are more comprehensive
2. The fix is to incorporate the credit entries on line 228.
https://issues.apache.org/jira/browse/CLOUDSTACK-9122
* pr/1192:
CLOUDSTACK-9122: latest credit entries should be incorporated when the balance is calculated
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9138 - Adds multiple providers back to VPC implementation - It is need and already allowed/used in the current implementation. For example, the Default [redundant] VPC offerings use two LB providers. If we cannot create offerings with 2 LB providers, the whole internal loadbalancer implementation won't work.
* pr/1215:
CLOUDSTACK-9138 - Adds multiple providers back to VPC implementation
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9143 Setup routes for RFC 1918 ip spaceSetup general route for RFC 1918 space, as otherwise it will be sent to the public gateway and likely to be dropped (internet providers do not route ip space that is meant for internal use). More specific routes that may be set have preference over this generic routes so this works even with private ranges used for public ip space (as shown below).
When using an internal DNS server some hosts may resolve to an RFC 1918 ip address. The SSVM has a default gw to public so if it has no route for this ip address space, it will not work. This PR makes generic RFC 1918 (so all internal ip adresses like 10.0.0.10 etc) to the local management gateway. This makes them reachable. Without this fix, it is sent upstream and it is dropped there.
Should there be a more generic route (smaller prefix), this has preference over the generic routes.
Example in my dev environment:
```
root@v-1-VM:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.23.1 0.0.0.0 UG 0 0 0 eth2
10.0.0.0 192.168.22.1 255.0.0.0 UG 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
172.16.0.0 192.168.22.1 255.240.0.0 UG 0 0 0 eth1
192.168.0.0 192.168.22.1 255.255.0.0 UG 0 0 0 eth1
192.168.22.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.23.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
```
Route `192.168.0.0/16` goes via `eth1` but `192.168.23.0/24` is more specific and has preference and goes via `eth2`. It works:
```
root@v-1-VM:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=0 ttl=49 time=7.179 ms
^C--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 7.179/7.179/7.179/0.000 ms
```
This solves a lot of the 'internal resolving' issues we face.
When the public ip address is RFC1918 itself, we do not set the routes.
* pr/1214:
Setup routes for RFC 1918 ip space
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.7] CLOUDSTACK-9129: list vpc routers by keyword in Infrastructure -> Virtual Routers
and two more changes:
(1) add network name/vpc name in the listRouters response
(2) add network name/vpc id, vpc name in the router details page
* pr/1197:
CLOUDSTACK-9129: list vpc routers by keyword in Infrastructure -> Virtual Routers
Signed-off-by: Remi Bergsma <github@remi.nl>
Setup general route for RFC 1918 space, as otherwise it will be sent to
the public gateway and not work. More specific routes that may be set
have preference over this generic routes.
When public network is RFC1918, we do not setup the routes to avoid
problems with internal-only deployments.
- Redundant Routers are not the same as Haigh Available.
Having a router HA is actually saying that the given router will be controlled by the High Availability monitor.
Hence fix any problem we might face.
- It is need and already allowed/used in the current implementation. For example, the Default [redundant] VPC offerings use
two LB providers. If we cannot create offerings with 2 LB providers, the whole internal loadbalancer implementation won't work
Adapted HypervisorUtilsTest to no longer give false positives
* pr/1187:
Adapted HypervisorUtilsTest to no longer give false positives
This closes#1185
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9131: Create a new API to check if the plugin is enabled.
Adding a new command to check if the quota plugin is enabled.
Also changed the command version to 4.7.
* pr/1201:
CLOUDSTACK-9131: Create a new API to check if the plugin is enabled.
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9123 - As a Developer I want the test_internal_lb.py to work with multiple hypervisorsThis PR make the test_internal_lb.py work with other Hypervisors types.
- Get hypervisor information from testClient.getHypervisorInfo()
- Adds XenServer, HyperV and VMware hypervisors
- Makes sure the OS type is "Other PV (64-bit)"
* pr/1204:
CLOUDSTACK-9123 - Make test compliant with multiple hypervisors
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9120 - As a Developer I want the new component tests to be moved into the smoke directoryThis PR only moves the tests I wrote form the ```component``` to the ```smoke``` directory. Given the nature of the tests, the should be moved.
- Those tests are doing a lot more than component tests would do. They require hardware and a lot of resources from real hypervisors.
I'm now running the five test files that were moved and will post results once they are done.
* pr/1195:
CLOUDSTACK-9120 - Moves tests to the smoke directory
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9118 - As a Developer I want the checkrouter.sh script to report the right information about RVR routers stateThis PR fixes the RVR routers state information retrieved by the Management Server via the ```checkrouter.sh``` script.
In order to cover the changes, a new test has been added to the component/test_routers_network_ops.py test suite: test_03_RVR_Network_check_router_state
* pr/1189:
CLOUDSTACK-9118 - Adds integration test to cover routers state.
CLOUDSTACK-9118 - Improves the existing checkrouter.sh script
Signed-off-by: Remi Bergsma <github@remi.nl>
* 4.6:
CLOUDSTACK-9106 - Makes Enum name compliant with Java code conventions.
CLOUDSTACK-9106 - Adds a test to cover the changes in the applyVpnUsers() method
CLOUDSTACK-9106 - Makes the router commands call more consistent.
CLOUDSTACK-9106 - Enables private gateway tests on Redundant VPCs
CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
CLOUDSTACK-9106 - Reduces the amount of iterations through the routers of a VPC
Add support for not (re)starting server after cloud-setup-management.
Closed PRs that will not be considered for merge:
This closes#1158
This closes#1097
CLOUDSTACK-9095 : Hypervisor changes to support UserData for Nuage VSP
* pr/1142:
CLOUDSTACK-9095 : Hypervisor changes to support UserData for Nuage VSP
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9106 - As a Developer I want the Redundant VPC private gateway feature fixedThis PR contains the same fixes from PR #1179, which was created against the master branch.
In addition, the points mentioned by @DaanHoogland were handled in this new PR:
* Made the code more consistent
- result = result && methodCall(), instead of throwing exceptions in some places or not checking 2 consecutive returns - in case of rVPC.
* Added an unit test to cover changes in the VpcRouterElementImpl.applyVpnUsers() method. The method returns an array of String, so I had to make sure it would contain the users from 2 consecutive calls. There are 2 tests to cover negative scenarios.
* pr/1184:
CLOUDSTACK-9106 - Makes Enum name compliant with Java code conventions.
CLOUDSTACK-9106 - Adds a test to cover the changes in the applyVpnUsers() method
CLOUDSTACK-9106 - Makes the router commands call more consistent.
CLOUDSTACK-9106 - Enables private gateway tests on Redundant VPCs
CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
CLOUDSTACK-9106 - Reduces the amount of iterations through the routers of a VPC
Signed-off-by: Remi Bergsma <github@remi.nl>
Add support for not (re)starting server after cloud-setup-management.This adds an option to the cloud-setup-management script to not start the management server after a successful configuration of it.
The primary motivation for this is to avoid circular dependency issues on systems that use systemd. When calling cloud-setup-management from a unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because /usr/bin/service will delegate to systemd, which is waiting for the Before service to start.
Executing the cloud-setup-management script with this new `--no-start` option will simply leave the management server stopped after a successful configuration. systemd can then be bypassed with `export _SYSTEMCTL_SKIP_REDIRECT=1` and using the init.d script.
* pr/1162:
Add support for not (re)starting server after cloud-setup-management.
Signed-off-by: Remi Bergsma <github@remi.nl>
Prevent live-lock in NSX API clientThe NSX api client relies on a sequence of responses to identify the need to authenticate and to follow redirects. In order to avoid live-locks, the NSX API client has a counter that will abort the execution after 5 consecutive requests that do not produce a Success (200) response.
When a NSX controller enters a faulty state it can allow authentication requests but deny any other API call. In such cases the NSX API client will consider the denied request a reason to follow a redirect and will enter into a live-lock (because the actual redirection is not happening).
This PR changes the NSX API client to no reset it's counter on a Success response from an authentication request. That is, the counter is only reset if another type of API call yields a Success response.
In addition, this PR also:
* changes the configuration of the license-maven-plugin to ignore files generated by pmd
* moves the NSX marvin test to a plugins folder
* refactors the NSX marvin test to reduce duplication
* adds an extra test case to the NSX marvin test that checks that NSX tunnels are properly created
* pr/1178:
Test NSX tunnel in guest network
Refactor test cases to reduce duplication
Use logger to print debug messages during test
Move NSX integrationt test to new plugins folder
Ignore pmd generated files during license check
Fix NSX rest client to not reset execution counter after a login
Add test for NSX plugin that simulates a live lock
Signed-off-by: Remi Bergsma <github@remi.nl>
Implement Quota serviceThis pull request is for quota service implementation, divided into following commits:
Quota server side implementation
a. Quota Framework plugin containing Daos
b. Quota database plugin for quota APIs
c. Quota Usage server enhancements
d. Quota UI plugin
e. Tests
* pr/768:
CLOUDSTACK-8592: Implement Quota service
Signed-off-by: Remi Bergsma <github@remi.nl>
Quota service while allowing for scalability will make sure that the cloud is
not exploited by attacks, careless use and program errors. To address this
problem, we propose to employ a quota-enforcement service that allows resource
usage within certain bounds as defined by policies and available quotas for
various entities. Quota service extends the functionality of usage server to
provide a measurement for the resources used by the accounts and domains using a
common unit referred to as cloud currency in this document. It can be configured
to ensure that your usage won’t exceed the budget allocated to accounts/domain
in cloud currency. It will let user know how much of the cloud resources he is
using. It will help the cloud admins, if they want, to ensure that a user does
not go beyond his allocated quota. Per usage cycle if a account is found to be
exceeding its quota then it is locked. Locking an account means that it will not
be able to initiat e a new resource allocation request, whether it is more
storage or an additional ip. Needless to say quota service as well as any action
on the account is configurable.
Changes from Github code review:
- Added marvin test for quota plugin API
- removed unused commented code
- debug messages in debug enabled check
- checks for nulls, fixed access to member variables and feature
- changes based on PR comments
- unit tests for UsageTypes
- unit tests for all Cmd classes
- unit tests for all service and manager impls
- try-catch-finally or try-with-resource in dao impls for failsafe db switching
- remove dead code
- add missing quota calculation case (regression fixed)
- replace tabs with spaces in pom.xmls
- quota: though default value for quota_calculated is 0, the usage server
makes it null while entering usage entries. Flipping the condition so
as to acocunt for that.
- quotatypes: fix NPE in quota type
- quota framework test fixes
- made statement period configurable
- changed default email templates to reflect the fact that exhausted quota may not result in a locked account
- added quotaUpdateCmd that refreshes quota balances and sends alerts and statements
- report quotaSummary command returns quota balance, quota usage and state for all account
- made UI framework changes to allow for text area input in edit views
- process usage entries that have greater than 0 usage
- orocess quota entries only if tariff is non zero
- if there are credit entries but no balance entry create a dummy balance entry
- remove any credit entries that are before the last balance entry
when displaying balance statement
- on a rerun the last balance is now getting added
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-+FS
PR: https://github.com/apache/cloudstack/pull/768
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Changed the NetworkTopologyContext class just to make the private member accessible from the test
- Added a test class to cover the positive scenario of the VpcVirtualRouterElementTest.applyVpnUsers() method.
- Covering when there is either no VPC or no routers.
- Checks the result of a call against the previous result. Either both are true or the method returns false.
- Do not thrown exceptions because some calls are not handling/rethrowing them. It would cause runtime problems.
- When doing a list.addAll(Arrays.asList(String[]{}) will cause problems when trying to cast the list.toArray() into an aray of String
It would only work if instead of calling addAll() I would pass it straight into the constructor:
e.g. List<String> l = new ArrayList(Arrays.asList(new String[]{});
Stirng [] s = (String[]) l.toArray();
But I did not like that implementation because it would require 2 arrays of string and combine them at the end.
- Use the router to retrieve the instance ID
- Check if the VPC is redundant in order to reuse the private gateway address.
- Brings the private gateways interfaces up.
- It was causing problems because Nics were expected to be plugged before they actually exist. Only in rVPC cases.
- Applies ACL items to routers only after the Pvt GW is setup.
