* 4.6:
CLOUDSTACK-9106 - Makes Enum name compliant with Java code conventions.
CLOUDSTACK-9106 - Adds a test to cover the changes in the applyVpnUsers() method
CLOUDSTACK-9106 - Makes the router commands call more consistent.
CLOUDSTACK-9106 - Enables private gateway tests on Redundant VPCs
CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
CLOUDSTACK-9106 - Reduces the amount of iterations through the routers of a VPC
Add support for not (re)starting server after cloud-setup-management.
Closed PRs that will not be considered for merge:
This closes#1158
This closes#1097
CLOUDSTACK-9095 : Hypervisor changes to support UserData for Nuage VSP
* pr/1142:
CLOUDSTACK-9095 : Hypervisor changes to support UserData for Nuage VSP
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9106 - As a Developer I want the Redundant VPC private gateway feature fixedThis PR contains the same fixes from PR #1179, which was created against the master branch.
In addition, the points mentioned by @DaanHoogland were handled in this new PR:
* Made the code more consistent
- result = result && methodCall(), instead of throwing exceptions in some places or not checking 2 consecutive returns - in case of rVPC.
* Added an unit test to cover changes in the VpcRouterElementImpl.applyVpnUsers() method. The method returns an array of String, so I had to make sure it would contain the users from 2 consecutive calls. There are 2 tests to cover negative scenarios.
* pr/1184:
CLOUDSTACK-9106 - Makes Enum name compliant with Java code conventions.
CLOUDSTACK-9106 - Adds a test to cover the changes in the applyVpnUsers() method
CLOUDSTACK-9106 - Makes the router commands call more consistent.
CLOUDSTACK-9106 - Enables private gateway tests on Redundant VPCs
CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
CLOUDSTACK-9106 - Reduces the amount of iterations through the routers of a VPC
Signed-off-by: Remi Bergsma <github@remi.nl>
Add support for not (re)starting server after cloud-setup-management.This adds an option to the cloud-setup-management script to not start the management server after a successful configuration of it.
The primary motivation for this is to avoid circular dependency issues on systems that use systemd. When calling cloud-setup-management from a unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because /usr/bin/service will delegate to systemd, which is waiting for the Before service to start.
Executing the cloud-setup-management script with this new `--no-start` option will simply leave the management server stopped after a successful configuration. systemd can then be bypassed with `export _SYSTEMCTL_SKIP_REDIRECT=1` and using the init.d script.
* pr/1162:
Add support for not (re)starting server after cloud-setup-management.
Signed-off-by: Remi Bergsma <github@remi.nl>
Prevent live-lock in NSX API clientThe NSX api client relies on a sequence of responses to identify the need to authenticate and to follow redirects. In order to avoid live-locks, the NSX API client has a counter that will abort the execution after 5 consecutive requests that do not produce a Success (200) response.
When a NSX controller enters a faulty state it can allow authentication requests but deny any other API call. In such cases the NSX API client will consider the denied request a reason to follow a redirect and will enter into a live-lock (because the actual redirection is not happening).
This PR changes the NSX API client to no reset it's counter on a Success response from an authentication request. That is, the counter is only reset if another type of API call yields a Success response.
In addition, this PR also:
* changes the configuration of the license-maven-plugin to ignore files generated by pmd
* moves the NSX marvin test to a plugins folder
* refactors the NSX marvin test to reduce duplication
* adds an extra test case to the NSX marvin test that checks that NSX tunnels are properly created
* pr/1178:
Test NSX tunnel in guest network
Refactor test cases to reduce duplication
Use logger to print debug messages during test
Move NSX integrationt test to new plugins folder
Ignore pmd generated files during license check
Fix NSX rest client to not reset execution counter after a login
Add test for NSX plugin that simulates a live lock
Signed-off-by: Remi Bergsma <github@remi.nl>
Implement Quota serviceThis pull request is for quota service implementation, divided into following commits:
Quota server side implementation
a. Quota Framework plugin containing Daos
b. Quota database plugin for quota APIs
c. Quota Usage server enhancements
d. Quota UI plugin
e. Tests
* pr/768:
CLOUDSTACK-8592: Implement Quota service
Signed-off-by: Remi Bergsma <github@remi.nl>
Quota service while allowing for scalability will make sure that the cloud is
not exploited by attacks, careless use and program errors. To address this
problem, we propose to employ a quota-enforcement service that allows resource
usage within certain bounds as defined by policies and available quotas for
various entities. Quota service extends the functionality of usage server to
provide a measurement for the resources used by the accounts and domains using a
common unit referred to as cloud currency in this document. It can be configured
to ensure that your usage won’t exceed the budget allocated to accounts/domain
in cloud currency. It will let user know how much of the cloud resources he is
using. It will help the cloud admins, if they want, to ensure that a user does
not go beyond his allocated quota. Per usage cycle if a account is found to be
exceeding its quota then it is locked. Locking an account means that it will not
be able to initiat e a new resource allocation request, whether it is more
storage or an additional ip. Needless to say quota service as well as any action
on the account is configurable.
Changes from Github code review:
- Added marvin test for quota plugin API
- removed unused commented code
- debug messages in debug enabled check
- checks for nulls, fixed access to member variables and feature
- changes based on PR comments
- unit tests for UsageTypes
- unit tests for all Cmd classes
- unit tests for all service and manager impls
- try-catch-finally or try-with-resource in dao impls for failsafe db switching
- remove dead code
- add missing quota calculation case (regression fixed)
- replace tabs with spaces in pom.xmls
- quota: though default value for quota_calculated is 0, the usage server
makes it null while entering usage entries. Flipping the condition so
as to acocunt for that.
- quotatypes: fix NPE in quota type
- quota framework test fixes
- made statement period configurable
- changed default email templates to reflect the fact that exhausted quota may not result in a locked account
- added quotaUpdateCmd that refreshes quota balances and sends alerts and statements
- report quotaSummary command returns quota balance, quota usage and state for all account
- made UI framework changes to allow for text area input in edit views
- process usage entries that have greater than 0 usage
- orocess quota entries only if tariff is non zero
- if there are credit entries but no balance entry create a dummy balance entry
- remove any credit entries that are before the last balance entry
when displaying balance statement
- on a rerun the last balance is now getting added
FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Quota+Service+-+FS
PR: https://github.com/apache/cloudstack/pull/768
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Changed the NetworkTopologyContext class just to make the private member accessible from the test
- Added a test class to cover the positive scenario of the VpcVirtualRouterElementTest.applyVpnUsers() method.
- Covering when there is either no VPC or no routers.
- Checks the result of a call against the previous result. Either both are true or the method returns false.
- Do not thrown exceptions because some calls are not handling/rethrowing them. It would cause runtime problems.
- When doing a list.addAll(Arrays.asList(String[]{}) will cause problems when trying to cast the list.toArray() into an aray of String
It would only work if instead of calling addAll() I would pass it straight into the constructor:
e.g. List<String> l = new ArrayList(Arrays.asList(new String[]{});
Stirng [] s = (String[]) l.toArray();
But I did not like that implementation because it would require 2 arrays of string and combine them at the end.
- Use the router to retrieve the instance ID
- Check if the VPC is redundant in order to reuse the private gateway address.
- Brings the private gateways interfaces up.
- It was causing problems because Nics were expected to be plugged before they actually exist. Only in rVPC cases.
- Applies ACL items to routers only after the Pvt GW is setup.
CLOUDSTACK-9094: Multiple threads are being used to collect the statsCLOUDSTACK-9094: Multiple threads are being used to collect the stats from the same VR
Same thread is being intialised by two managers, VirtualNetworkApplianceManager and VpcVirtualNetworkApplianceManager
* pr/1140:
CLOUDSTACK-9094: Multiple threads are being used to collect the stats from the same VR
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9105: Logging enhancement: Handle/reference to track API calls end to end in the MS logs
Added logid to logging framework, now all API call logs can be tracked with this id end to end
* pr/1167:
CLOUDSTACK-9105: Logging enhancement: Handle/reference to track API calls end to end in the MS logs Added logid to logging framework, now all API call logs can be tracked with this id end to end
Signed-off-by: Daan Hoogland <daan@onecht.net>
[4.6/master] rate-limit: increase JVM memory and enable fork mode for unit testsEnables forkmode for surefire plugin and increases memory opts for JVM
for rate-limit tests. This tries to fix intermittent Jenkins failures
which look like:
multipleClientsCanAccessWithoutBlocking(org.apache.cloudstack.ratelimit.ApiRateLimitTest):
unable to create new native thread
Previous PR (again master by mistake) -- #1171
* pr/1175:
rate-limit: increase JVM memory and enable fork mode for unit tests
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9025: Fixed can't create usable template from snapshot in Xenserver and Vmwarehttps://issues.apache.org/jira/browse/CLOUDSTACK-9025
Fix also reverts below commit as below solution making assumption about hypervisor which are not applicable in case of XenServer and VmWare
Revert "CLOUDSTACK-8964: Can't create template or volume from snapshot"
This reverts commit ccf5d75cfbe769b34c021ab3653ed318cae25933.
Testing:
Able to deploy VM successfully from template created from linked clone snapshot on XenServer.
* pr/1176:
CLOUDSTACK-9025: Fixed can't create usable template from snapshot in Xenserver and Vmware
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9047 rename enumsmake enums adhere to best practice naming conventions
* pr/1049:
CLOUDSTACK-9046 rename enums to adhere to naming conventions
CLOUDSTACK-9046 renamed enums in kvm plugin
CLOUDSTACK-9047 use 'State's only with context there are more types called 'State' (or to be called so but now 'state') So remove imports and prepend their enclosing class/context to them.
Signed-off-by: Daan Hoogland <daan@onecht.net>
Cwe 190coverity warnings of this type adressed. Some where dismissed and maybe with reason but it seemed possible to remove them and hence obligatory ;p
* pr/1057:
move back to original contract of isNetworksOverlap()
Changed the behavior of methods that use NetUtils.cidrToLong(String)
CWE-190 unit test for extremes of long netMaskFromCidr(long)
CWE-190 netmask as long form cidr-size as method
CID-1116482 cidrToLong cleanup of bitshift problem
CID-1116483 cidr to netmask bitshifts guarded with casts
CID-1116484 cast to long and use long as cidrsize type and simpel test
CID-1116485: cast cidr during bit shifting and simple test included
CID-1175714 casts before bit shift
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-8964: Ovm3HypervisorGuru handle only srcData with HypervisorType is Ovm3This PR can only be applied after PR #1176
The CopyCommand on Ovm3 should be handled by Ovm3StorageProcessor, not SSVM.
Hence, I revert two commits on Ovm3HypervisorGuru, and add the hypervisorType check so that only the this guru will only handle Ovm3 (not KVM)
* pr/1177:
CLOUDSTACK-8964: Ovm3HypervisorGuru handle only srcData with HypervisorType is Ovm3
Revert "simple change to prevent failure and keep OVM3 snapshots working"
Revert "CLOUDSTACK-8964 side effect isolation"
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-8845: set isRevertable of snapshot to false if the volume is removedSome users encounter an exception when listSnapshots.
We should set the isRevertable of snapshot to false if the original volume is removed, without checking if the snapshot is stored in primary store (the exception was thowned during the checking).
* pr/1168:
CLOUDSTACK-8845: set isRevertable of snapshot to false if the volume is removed
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9101: fix some issues in resize volume(1) fix issue: volume size is not updated even if the operation succeed
(2) Add ui support for root volume resize
(3) resize on qcow2 type ROOT volume of stopped vm does not really work
see https://issues.apache.org/jira/browse/CLOUDSTACK-9101
* pr/1161:
CLOUDSTACK-9101: resize root volume of stopped vm on KVM
CLOUDSTACK-9101: add UI support for root volume resize
CLOUDSTACK-9101: update volume size after resizevolume
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9051: update nic IP address of stopped vmThis provides the feature to change ip address of NIC on a stopped vm by API and UI.
* pr/1086:
CLOUDSTACK-9051: reprogram network as a part of vm nic ip update
CLOUDSTACK-9051: add unit tests for UpdateVmNicIp
CLOUDSTACK-9051: update nic IP address of stopped vm
Signed-off-by: Daan Hoogland <daan@onecht.net>
Removed the PlannerBase class because it is does not bring contribution to ACS' code.Removed the PlannerBase class because it is does not bring contribution to ACS' code.
We changed com.cloud.deploy.FirstFitPlanner, now it doesnt extends the PlannerBase and implements the com.cloud.deploy.DeploymentPlanner.
We also removed the method com.cloud.deploy.DeploymentPlanner.check(VirtualMachineProfile, DeploymentPlan, DeployDestination, ExcludeList) that was not used anywhere.
Additionally, we removed the _ from FirstFitPlanner's attributes name, in order to have them in a more standard way.
* pr/1108:
Removed PlannerBase empty class
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8852 Database shows that management server is UP when it iDatabase shows that management server is UP when it is actually stopped.
This was happening as the stop method in clusterMnanagerImpl was not getting callled. Added shutdown hooks to all spring sub contexts, this enables spring to call the stop mehtods of the beans when management server is shutting down.
Conflicts:
framework/spring/module/src/main/java/org/apache/cloudstack/spring/module/web/CloudStackContextLoaderListener.java
* pr/840:
CLOUDSTACK-8852 Database shows that management server is UP when it is actually stopped from the CCP GUI
Signed-off-by: Remi Bergsma <github@remi.nl>
This has been happening for the NSX and BigSwitch plugins.
The maven build prints something like:
[WARNING] Unknown file extension: .../plugins/network-elements/nicira-nvp/.pmd
[WARNING] Missing header in: .../plugins/network-elements/nicira-nvp/.pmdruleset.xml
CLOUDSTACK-8976 - Sorting of security groupsSimple change to sort the security groups in alphabetical order within the instance creation wizard.
This makes it much easier to find a security group when the user is presented with a long list.
* pr/937:
Sorting of security groups
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9107: Description of global config agent.load.threshold and the actual behavior are not in sync
Made configuration parameter description and behavior in sync.
* pr/1172:
CLOUDSTACK-9107: Description of global config agent.load.threshold and the actual behavior are not in sync Made configuration parameter description and behavior in sync.
Signed-off-by: Remi Bergsma <github@remi.nl>
4.6.0 upgrade path is to pass 4.6.1 to create the extra view in theretest build done only
* pr/1173:
4.6.0 upgrade path is to pass 4.6.1 to create the extra view in there
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
[4.7] CLOUDSTACK-9004: Add features to HyperVEnlightenmentFeatureDefAdd function to set vapic, spinlock and retries
Add function to get retry value
Modify toString to output appropriate XML for spinlock value if set
* pr/1013:
CLOUDSTACK-9004: Add features to HyperVEnlightenmentFeatureDef
Signed-off-by: Daan Hoogland <daan@onecht.net>
This adds an option to the cloud-setup-management script to not start
the management server after a successful configuration of it. The
primary motivation for this is to avoid circular dependency issues on
systems that use systemd. When calling cloud-setup-management from a
unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because
/usr/bin/service will delegate to systemd, which is waiting for the
Before service to start.
Send arping to the gateway instead of our own addressWe need to send an Unsolicited ARP to the gateway, instead of our own address. We now encounter problems when people deploy/destroy/deploy and get the same public ip.
Packets arrive, but with incorrect / cached mac and are ignored by the routervm kernel.
Run arping manually to update the arp-cache on the gateway and things start to work.
Then we discovered the `arping` is actually done, but sent to its own address. Therefore the gateway doesn't pick it up. We only saw this happening when rapid deploy tools are used, like Terraform that do deploy/destroy/deploy and might get the same ip but on a new router having a new mac.
```
2015-12-03 18:07:25,589 CsHelper.py execute:160 Executing: arping -c 1 -I eth1 -A -U -s 192.168.23.8 192.168.23.1
```
The integration tests seem happy, although the full run is still ongoing:
```
=== TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
```
Thanks @sspans for helping trouble shoot this. Ping @wilderrodrigues can you review please?
* pr/1163:
CLOUDSTACK-9097 Make public ip work immediately
Signed-off-by: Remi Bergsma <github@remi.nl>
Fix also reverts below commit as below solution making assumption about hypervisor which are not applicable
in case of XenServer and VmWare
Revert "CLOUDSTACK-8964: Can't create template or volume from snapshot"
This reverts commit ccf5d75cfbe769b34c021ab3653ed318cae25933.
debian: allow rules to pick ACS_BUILD_OPTS from envOnly now debian builds can be noredist etc.
Old PR: https://github.com/apache/cloudstack/pull/1149
cc @remibergsma
* pr/1169:
debian: allow rules to pick ACS_BUILD_OPTS from env
Signed-off-by: Remi Bergsma <github@remi.nl>
