CLOUDSTACK-9134: set device_id as the first device_id not in use instead of nic count
when we restart vpc tiers, the old nics will be removed, and create a new nic.
however, the device_id was set to the nic count, which may be already used.
this commit get the first device_id not in use as the device_id of new nic.
This issue also happen when we add multiple networks to a vm and remove them.
* pr/1209:
CLOUDSTACK-9134: set device_id as the first device_id not in use instead of nic count
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9155 make sure logrotate is effective for cloud.logMany processes on the VRs log to cloud.log. When log rotate kicks in, the file is rotated but the scripts still write to the old inode (cloud.log.1 after rotate). Tis quickly fills up the tiny log partition.
Using 'copytruncate' is a small tradeoff, there is a slight change of missing a log entry, but in the old situation nothing ended up in cloud.log after rotate (except for stuff that was (re)started) so I think this is the best solution until we properly rewrite the script to either use their own script or syslog.
More details: https://issues.apache.org/jira/browse/CLOUDSTACK-9155
* pr/1235:
CLOUDSTACK-9155 make sure logrotate is effective
Signed-off-by: Remi Bergsma <github@remi.nl>
Many processes on the VRs log to cloud.log. When logrotate
kicks in, the file is rotated but the scripts still write
to the old inode (cloud.log.1 after rotate). Tis quickly
fills up the tiny log partition.
Using 'copytruncate' is a tradeoff, there is a slight
change of missing a log entry, but in the old situation
we were missing all of them after logrotate.
CLOUDSTACK-9151 - As a Developer I want the VRID to be set within the limits of KeepaliveDThis PR fixes a blocker issue!
- Just like with RVRs, use the VRID 51 instead of making it dependent on the VPCID
- Reason: arbitary unique number 0..255 used to differentiate multiple instances of vrrpd running on the same NIC (and hence same socket). virtual_router_id 51
* pr/1231:
CLOUDSTACK-9151 - Removes the replacement of the VRID in the CsRedundant file
Signed-off-by: Remi Bergsma <github@remi.nl>
Updating pom.xml version numbers for release 4.6.2-SNAPSHOTSet next version in 4.6 release branch to version 4.6.2-SNAPSHOT.
Using ` ./tools/build/setnextversion.sh`.
Ping @bhaisaab @DaanHoogland before we merge this, how will we be creating the upgrade paths from 4.6.2 to 4.7? After this PR is merged, we need to manually do a fwd-merge and make sure we keep the pom versions in master/4.7. Much like in #1071.
* pr/1186:
Fixed typo in iam/pom.xml
Updating pom.xml version numbers for release 4.6.2-SNAPSHOT
Signed-off-by: Daan Hoogland <daan@onecht.net>
- Just like with RVRs, use the VRID 51 instead of making it dependent on the VPCID
- Reason: arbitary unique number 0..255 used to differentiate multiple instances of vrrpd running on the same NIC (and hence same socket). virtual_router_id 51
[UI] bug fix: Delete added ACL lists is not available for Domain Admin and normal usersOnly the owner or domain admin/admin can access the page (vpc->Network ACL Lists), they are also able to remove the network ACL lists, which is not allowed on UI.
* pr/1211:
[UI] bug fix: Delete added ACL lists is not available for Domain Admin and normal users
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.6] CLOUDSTACK-4787 - vmware diskcontrollersSame as #1131 (see this for screenshots etc)
* pr/1132:
CLOUDSTACK-4787: Allow users to select disk controller for VM/template
CLOUDSTACK-4787 Allow selection of scsi controller type in vSphere
Signed-off-by: Daan Hoogland <daan@onecht.net>
[4.6] CLOUDSTACK-9113: skip vm with inconsistent state when getVmStats/getVmDiskStatson KVM, if there is a vm has inconsistent state between hypervisor and db, the getVmStat will terminate and return null, all vm stats will not be updated.
we should skip the vm which has inconsistent state, and continue on others.
* pr/1182:
CLOUDSTACK-9113: skip vm with inconsistent state when getVmStats/getVmDiskStats
Signed-off-by: Remi Bergsma <github@remi.nl>
Show actual diff in commits after merge with git-pr / git-fwd-mergeThis shows the diff in commits after using `git-pr` and `git-fwd-merge` tools, like this:
```
44e8c92 Merge pull request #1226 from borisroman/CLOUDSTACK-9148
0554610 Removed .pydevproject from plugin kvm hypervisor.
```
That helps in quickly seeing what change is applied and whether it makes sense. It doesn't touch the actual code base. I'm using this patch for some time already.
* pr/1227:
Show actual diff in commits after merge
Signed-off-by: Remi Bergsma <github@remi.nl>
Adapted HypervisorUtilsTest to no longer give false positives
* pr/1187:
Adapted HypervisorUtilsTest to no longer give false positives
This closes#1185
Signed-off-by: Remi Bergsma <github@remi.nl>
when we restart vpc tiers, the old nics will be removed, and create a new nic.
however, the device_id was set to the nic count, which may be already used.
this commit get the first device_id not in use as the device_id of new nic.
This issue also happen when we add multiple networks to a vm and remove them.
CLOUDSTACK-9118 - As a Developer I want the checkrouter.sh script to report the right information about RVR routers stateThis PR fixes the RVR routers state information retrieved by the Management Server via the ```checkrouter.sh``` script.
In order to cover the changes, a new test has been added to the component/test_routers_network_ops.py test suite: test_03_RVR_Network_check_router_state
* pr/1189:
CLOUDSTACK-9118 - Adds integration test to cover routers state.
CLOUDSTACK-9118 - Improves the existing checkrouter.sh script
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9106 - As a Developer I want the Redundant VPC private gateway feature fixedThis PR contains the same fixes from PR #1179, which was created against the master branch.
In addition, the points mentioned by @DaanHoogland were handled in this new PR:
* Made the code more consistent
- result = result && methodCall(), instead of throwing exceptions in some places or not checking 2 consecutive returns - in case of rVPC.
* Added an unit test to cover changes in the VpcRouterElementImpl.applyVpnUsers() method. The method returns an array of String, so I had to make sure it would contain the users from 2 consecutive calls. There are 2 tests to cover negative scenarios.
* pr/1184:
CLOUDSTACK-9106 - Makes Enum name compliant with Java code conventions.
CLOUDSTACK-9106 - Adds a test to cover the changes in the applyVpnUsers() method
CLOUDSTACK-9106 - Makes the router commands call more consistent.
CLOUDSTACK-9106 - Enables private gateway tests on Redundant VPCs
CLOUDSTACK-9106 - Refactor the createPrivateNicProfileForGateway() method
CLOUDSTACK-9106 - Reduces the amount of iterations through the routers of a VPC
Signed-off-by: Remi Bergsma <github@remi.nl>
Add support for not (re)starting server after cloud-setup-management.This adds an option to the cloud-setup-management script to not start the management server after a successful configuration of it.
The primary motivation for this is to avoid circular dependency issues on systems that use systemd. When calling cloud-setup-management from a unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because /usr/bin/service will delegate to systemd, which is waiting for the Before service to start.
Executing the cloud-setup-management script with this new `--no-start` option will simply leave the management server stopped after a successful configuration. systemd can then be bypassed with `export _SYSTEMCTL_SKIP_REDIRECT=1` and using the init.d script.
* pr/1162:
Add support for not (re)starting server after cloud-setup-management.
Signed-off-by: Remi Bergsma <github@remi.nl>
- Changed the NetworkTopologyContext class just to make the private member accessible from the test
- Added a test class to cover the positive scenario of the VpcVirtualRouterElementTest.applyVpnUsers() method.
- Covering when there is either no VPC or no routers.
- Checks the result of a call against the previous result. Either both are true or the method returns false.
- Do not thrown exceptions because some calls are not handling/rethrowing them. It would cause runtime problems.
- When doing a list.addAll(Arrays.asList(String[]{}) will cause problems when trying to cast the list.toArray() into an aray of String
It would only work if instead of calling addAll() I would pass it straight into the constructor:
e.g. List<String> l = new ArrayList(Arrays.asList(new String[]{});
Stirng [] s = (String[]) l.toArray();
But I did not like that implementation because it would require 2 arrays of string and combine them at the end.
- Use the router to retrieve the instance ID
- Check if the VPC is redundant in order to reuse the private gateway address.
- Brings the private gateways interfaces up.
- It was causing problems because Nics were expected to be plugged before they actually exist. Only in rVPC cases.
- Applies ACL items to routers only after the Pvt GW is setup.
[4.6/master] rate-limit: increase JVM memory and enable fork mode for unit testsEnables forkmode for surefire plugin and increases memory opts for JVM
for rate-limit tests. This tries to fix intermittent Jenkins failures
which look like:
multipleClientsCanAccessWithoutBlocking(org.apache.cloudstack.ratelimit.ApiRateLimitTest):
unable to create new native thread
Previous PR (again master by mistake) -- #1171
* pr/1175:
rate-limit: increase JVM memory and enable fork mode for unit tests
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9025: Fixed can't create usable template from snapshot in Xenserver and Vmwarehttps://issues.apache.org/jira/browse/CLOUDSTACK-9025
Fix also reverts below commit as below solution making assumption about hypervisor which are not applicable in case of XenServer and VmWare
Revert "CLOUDSTACK-8964: Can't create template or volume from snapshot"
This reverts commit ccf5d75cfbe769b34c021ab3653ed318cae25933.
Testing:
Able to deploy VM successfully from template created from linked clone snapshot on XenServer.
* pr/1176:
CLOUDSTACK-9025: Fixed can't create usable template from snapshot in Xenserver and Vmware
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-8964: Ovm3HypervisorGuru handle only srcData with HypervisorType is Ovm3This PR can only be applied after PR #1176
The CopyCommand on Ovm3 should be handled by Ovm3StorageProcessor, not SSVM.
Hence, I revert two commits on Ovm3HypervisorGuru, and add the hypervisorType check so that only the this guru will only handle Ovm3 (not KVM)
* pr/1177:
CLOUDSTACK-8964: Ovm3HypervisorGuru handle only srcData with HypervisorType is Ovm3
Revert "simple change to prevent failure and keep OVM3 snapshots working"
Revert "CLOUDSTACK-8964 side effect isolation"
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-8845: set isRevertable of snapshot to false if the volume is removedSome users encounter an exception when listSnapshots.
We should set the isRevertable of snapshot to false if the original volume is removed, without checking if the snapshot is stored in primary store (the exception was thowned during the checking).
* pr/1168:
CLOUDSTACK-8845: set isRevertable of snapshot to false if the volume is removed
Signed-off-by: Daan Hoogland <daan@onecht.net>
CLOUDSTACK-9101: fix some issues in resize volume(1) fix issue: volume size is not updated even if the operation succeed
(2) Add ui support for root volume resize
(3) resize on qcow2 type ROOT volume of stopped vm does not really work
see https://issues.apache.org/jira/browse/CLOUDSTACK-9101
* pr/1161:
CLOUDSTACK-9101: resize root volume of stopped vm on KVM
CLOUDSTACK-9101: add UI support for root volume resize
CLOUDSTACK-9101: update volume size after resizevolume
Signed-off-by: Daan Hoogland <daan@onecht.net>
This adds an option to the cloud-setup-management script to not start
the management server after a successful configuration of it. The
primary motivation for this is to avoid circular dependency issues on
systems that use systemd. When calling cloud-setup-management from a
unit with a Before= directive on a service depending on
cloudstack-management, the process will deadlock because
/usr/bin/service will delegate to systemd, which is waiting for the
Before service to start.
Send arping to the gateway instead of our own addressWe need to send an Unsolicited ARP to the gateway, instead of our own address. We now encounter problems when people deploy/destroy/deploy and get the same public ip.
Packets arrive, but with incorrect / cached mac and are ignored by the routervm kernel.
Run arping manually to update the arp-cache on the gateway and things start to work.
Then we discovered the `arping` is actually done, but sent to its own address. Therefore the gateway doesn't pick it up. We only saw this happening when rapid deploy tools are used, like Terraform that do deploy/destroy/deploy and might get the same ip but on a new router having a new mac.
```
2015-12-03 18:07:25,589 CsHelper.py execute:160 Executing: arping -c 1 -I eth1 -A -U -s 192.168.23.8 192.168.23.1
```
The integration tests seem happy, although the full run is still ongoing:
```
=== TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
```
Thanks @sspans for helping trouble shoot this. Ping @wilderrodrigues can you review please?
* pr/1163:
CLOUDSTACK-9097 Make public ip work immediately
Signed-off-by: Remi Bergsma <github@remi.nl>
Fix also reverts below commit as below solution making assumption about hypervisor which are not applicable
in case of XenServer and VmWare
Revert "CLOUDSTACK-8964: Can't create template or volume from snapshot"
This reverts commit ccf5d75cfbe769b34c021ab3653ed318cae25933.
debian: allow rules to pick ACS_BUILD_OPTS from envOnly now debian builds can be noredist etc.
Old PR: https://github.com/apache/cloudstack/pull/1149
cc @remibergsma
* pr/1169:
debian: allow rules to pick ACS_BUILD_OPTS from env
Signed-off-by: Remi Bergsma <github@remi.nl>
Enables forkmode for surefire plugin and increases memory opts for JVM
for rate-limit tests. This tries to fix intermittent Jenkins failures
which look like:
multipleClientsCanAccessWithoutBlocking(org.apache.cloudstack.ratelimit.ApiRateLimitTest):
unable to create new native thread
(cherry picked from commit 421bc00cd6005d9e8484c6df46a3b84ff466afd2)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-6276 Fixing affinity groups for projectsWith some contributions from @resmo and @ustcweizhou.
This closes https://github.com/apache/cloudstack/pull/508
To test manually (need at least 2 hosts):
Create a project
Create an affinity group in that project
Deploy a vm with that affinity group
Deploy a second vm with that affinity group
They should be on different hosts
Ran old and new tests for affinity groups on the simulator
Test create affinity group as admin in project ... === TestName: test_01_admin_create_aff_grp_for_project | Status : SUCCESS ===
ok
Test create affinity group as domain admin for projects ... === TestName: test_02_doadmin_create_aff_grp_for_project | Status : SUCCESS ===
ok
Test create affinity group as user for projects ... === TestName: test_03_user_create_aff_grp_for_project | Status : SUCCESS ===
ok
Test create affinity group that exists (same name) for projects ... === TestName: test_4_user_create_aff_grp_existing_name_for_project | Status : SUCCESS ===
ok
#Delete Affinity Group by id. ... === TestName: test_01_delete_aff_grp_by_id | Status : SUCCESS ===
ok
#Delete Affinity Group by id should fail for user not in project ... === TestName: test_02_delete_aff_grp_by_id_another_user | Status : SUCCESS ===
ok
test DeployVM in anti-affinity groups ... === TestName: test_01_deploy_vm_anti_affinity_group | Status : SUCCESS ===
ok
test DeployVM in anti-affinity groups with more vms than hosts. ... === TestName: test_02_deploy_vm_anti_affinity_group_fail_on_not_enough_hosts | Status : SUCCESS ===
ok
List affinity group for a vm for projects ... === TestName: test_01_list_aff_grps_for_vm | Status : SUCCESS ===
ok
List multiple affinity groups associated with a vm for projects ... === TestName: test_02_list_multiple_aff_grps_for_vm | Status : SUCCESS ===
ok
List affinity groups by id for projects ... === TestName: test_03_list_aff_grps_by_id | Status : SUCCESS ===
ok
List Affinity Groups by name for projects ... === TestName: test_04_list_aff_grps_by_name | Status : SUCCESS ===
ok
List Affinity Groups by non-existing id for projects ... === TestName: test_05_list_aff_grps_by_non_existing_id | Status : SUCCESS ===
ok
List Affinity Groups by non-existing name for projects ... === TestName: test_06_list_aff_grps_by_non_existing_name | Status : SUCCESS ===
ok
List affinity group should list all for a vms associated with that group for projects ... === TestName: test_07_list_all_vms_in_aff_grp | Status : SUCCESS ===
ok
Update the list of affinityGroups by using affinity groupids ... === TestName: test_01_update_aff_grp_by_ids | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 16 tests in 581.706s
OK
Deploy vm as Admin in Affinity Group belonging to regular user (should fail) ... === TestName: test_01_deploy_vm_another_user | Status : SUCCESS ===
ok
Create Affinity Group as admin for regular user ... === TestName: test_02_create_aff_grp_user | Status : SUCCESS ===
ok
List Affinity Groups as admin for all the users ... === TestName: test_03_list_aff_grp_all_users | Status : SUCCESS ===
ok
List Affinity Groups belonging to admin user ... === TestName: test_04_list_all_admin_aff_grp | Status : SUCCESS ===
ok
List Affinity Groups belonging to regular user passing account id and domain id ... === TestName: test_05_list_all_users_aff_grp | Status : SUCCESS ===
ok
List Affinity Groups belonging to regular user passing group id ... === TestName: test_06_list_all_users_aff_grp_by_id | Status : SUCCESS ===
ok
Delete Affinity Group belonging to regular user ... === TestName: test_07_delete_aff_grp_of_other_user | Status : SUCCESS ===
ok
Test create affinity group as admin ... === TestName: test_01_admin_create_aff_grp | Status : SUCCESS ===
ok
Test create affinity group as domain admin ... === TestName: test_02_doadmin_create_aff_grp | Status : SUCCESS ===
ok
Test create affinity group as user ... === TestName: test_03_user_create_aff_grp | Status : SUCCESS ===
ok
Test create affinity group that exists (same name) ... === TestName: test_04_user_create_aff_grp_existing_name | Status : SUCCESS ===
ok
Test create affinity group with existing name but within different account ... === TestName: test_05_create_aff_grp_same_name_diff_acc | Status : SUCCESS ===
ok
Test create affinity group of non-existing type ... === TestName: test_06_create_aff_grp_nonexisting_type | Status : SUCCESS ===
ok
Delete Affinity Group by name ... === TestName: test_01_delete_aff_grp_by_name | Status : SUCCESS ===
ok
Delete Affinity Group as admin for an account ... === TestName: test_02_delete_aff_grp_for_acc | Status : SUCCESS ===
ok
Delete Affinity Group which has vms in it ... === TestName: test_03_delete_aff_grp_with_vms | Status : SUCCESS ===
ok
Delete Affinity Group with id which does not belong to this user ... === TestName: test_05_delete_aff_grp_id | Status : SUCCESS ===
ok
Delete Affinity Group by name which does not belong to this user ... === TestName: test_06_delete_aff_grp_name | Status : SUCCESS ===
ok
Delete Affinity Group by id. ... === TestName: test_08_delete_aff_grp_by_id | Status : SUCCESS ===
ok
Root admin should be able to delete affinity group of other users ... === TestName: test_09_delete_aff_grp_root_admin | Status : SUCCESS ===
ok
Deploy VM without affinity group ... === TestName: test_01_deploy_vm_without_aff_grp | Status : SUCCESS ===
ok
Deploy VM by aff grp name ... === TestName: test_02_deploy_vm_by_aff_grp_name | Status : SUCCESS ===
ok
Deploy VM by aff grp id ... === TestName: test_03_deploy_vm_by_aff_grp_id | Status : SUCCESS ===
ok
test DeployVM in anti-affinity groups ... === TestName: test_04_deploy_vm_anti_affinity_group | Status : SUCCESS ===
ok
Deploy vms by affinity group id ... === TestName: test_05_deploy_vm_by_id | Status : SUCCESS ===
ok
Deploy vm in affinity group of another user by name ... === TestName: test_06_deploy_vm_aff_grp_of_other_user_by_name | Status : SUCCESS ===
ok
Deploy vm in affinity group of another user by id ... === TestName: test_07_deploy_vm_aff_grp_of_other_user_by_id | Status : SUCCESS ===
ok
Deploy vm in multiple affinity groups ... === TestName: test_08_deploy_vm_multiple_aff_grps | Status : SUCCESS ===
ok
Deploy multiple vms in multiple affinity groups ... === TestName: test_09_deploy_vm_multiple_aff_grps | Status : SUCCESS ===
ok
Deploy VM by aff grp name and id ... === TestName: test_10_deploy_vm_by_aff_grp_name_and_id | Status : SUCCESS ===
ok
List affinity group for a vm ... === TestName: test_01_list_aff_grps_for_vm | Status : SUCCESS ===
ok
List multiple affinity groups associated with a vm ... === TestName: test_02_list_multiple_aff_grps_for_vm | Status : SUCCESS ===
ok
List affinity groups by id ... === TestName: test_03_list_aff_grps_by_id | Status : SUCCESS ===
ok
List Affinity Groups by name ... === TestName: test_04_list_aff_grps_by_name | Status : SUCCESS ===
ok
List Affinity Groups by non-existing id ... === TestName: test_05_list_aff_grps_by_non_existing_id | Status : SUCCESS ===
ok
List Affinity Groups by non-existing name ... === TestName: test_06_list_aff_grps_by_non_existing_name | Status : SUCCESS ===
ok
List affinity group should list all for a vms associated with that group ... === TestName: test_07_list_all_vms_in_aff_grp | Status : SUCCESS ===
ok
Update the list of affinityGroups by using affinity groupids ... === TestName: test_01_update_aff_grp_by_ids | Status : SUCCESS ===
ok
Update the list of affinityGroups by using affinity groupnames ... === TestName: test_02_update_aff_grp_by_names | Status : SUCCESS ===
ok
Update the list of affinityGroups for vm which is not associated ... === TestName: test_03_update_aff_grp_for_vm_with_no_aff_grp | Status : SUCCESS ===
ok
Update the list of Affinity Groups to empty list ... SKIP: Skip - Failing - work in progress
Update the list of Affinity Groups on running vm ... === TestName: test_05_update_aff_grp_on_running_vm | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 42 tests in 976.432s
OK (SKIP=1)
* pr/1134:
CLOUDSTACK-6276 Removing unused parameter in integration test for projects
CLOUDSTACK-6276 Removing unused parameter in integration test
CLOUDSTACK-6276 Fixing affinity groups for projects
Signed-off-by: Remi Bergsma <github@remi.nl>
[4.6] CLOUDSTACK-9075 - As a Developer I want the Private GW feature fixed on single VPCsThis PR fixes the issue we faced with Private Gateways on single VPC when using ACS 4.6.0 and onwards.
The root cause: during the VR refactor, the static routes configuration was left unimplemented.
This PR also improves the existing Replace ACL test and adds a new test, that cover the Private Gateway in a more complete way.
The new test does the following:
1. Create 2 VPCs
2. Create 2 Tiers - 1 per VPC
3. Deploy 2 VMs - 1 per Tier
4. Acquire 2 pub IPs - 1 per VPC
5. Create 2 PF rules - 1 per pub IP
6. Create 2 ACLs + rules - 1 per VPC
7. Assign new ACLs to Tiers
8. Create 2 Private GWs - 1 per VPC
9. Replace the Pvt GWs ACLs
10. Create 2 Static routes - 1 per Pvt GW
11. SSH into VM1 (VPC1) and from there ping VM2 (VPC2)
There is also a test for Private Gateways on Redundant VPCs. But I found out that the feature is broken in when used with rVPCs. It will be addressed in a separate Issue/PR.
I'm running the tests. Will post results as soon as they are ready.
* pr/1151:
CLOUDSTACK-9075 - Uses the same vlan since it should have been already released
CLOUDSTACK-9075 - Adds VPC static routes test
CLOUDSTACK-9075 - Covers Private GW ACL with Redundant VPCs
CLOUDSTACK-9075 - Add method to get list of Physical Networks per zone
Signed-off-by: Remi Bergsma <github@remi.nl>
We need to send an Unsolicited ARP to the gateway, instead of our own address. We now encounter problems when people deploy/destroy/deploy and get the same public ip.
