CLOUDSTACK-8812 / CLOUDSTACK-9010: adjust packaging for centos7
Here are a few adjustments for the packaging in centos7. With these changes I was able to start the service. Please review. Thank you.
* pr/1008:
CLOUDSTACK-9010: adjust packaging for centos7
This closes#888
Signed-off-by: Remi Bergsma <github@remi.nl>
* Adjust systemd service to match tomcat7 startup,
and change service type to simple.
* Adjust sysconfig to only have one JAVA_OPTS
due to behaviour change in tomcat7.
* Adjust spec to remove some config files
from WEB-INF since they are placed in /etc.
This is a similar behaviour to the centos6 spec.
Fixed: Network Update from RVR offering to Standalone offering failsProblem: Moving a RVR network offering to standalone makes the status of VR's as UNKNOWN and Redundant Router marked with YES.
Fix: The network's isRedundant was not getting updated.
* pr/818:
CLOUDSTACK-8844: Network Update from RVR offering to Standalone offering fails - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
another typo that appears when monitoring the java process via JMXStatus : in preogress instead of in progress and some other small typos .
* pr/913:
another typo that appears when monitoring the java process via JMX
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9014 Rename xapi plugins for s3 and swift to make them work after renaming the callsMake renaming introduced in a8212d9ef458dd7ac64b021e6fa33fcf64b3cce0 work for S3 and Swift xapi plugins.
This PR is to address comments in PR #970
* pr/982:
Rename xapi plugins for s3 and swift to make them work after renaming the calls
Signed-off-by: Remi Bergsma <github@remi.nl>
Fixed user_vm_view to only display keypairs belonging to the account.The user_vm_view displayes the keypair information by joining vm_details with ssh_keypairs on the key value exclusively.
We found a scenario in which this can cause information leakage. If there are two accounts using the same key, but create a different key name for it, and then a vm is created using one of the keys, the view will list both keypairs as belonging to the vm, which can in turn cause confusion to the users who see a keypair name which they did not create.
The fix simply limits the view to displaying keypairs which belong to vm's account.
I added it to the latest schema migration only; should I also include it in the previous ones?
* pr/1006:
CLOUDSTACK-9011 - Fixed user_vm_view to only display keypairs belonging to the account.
Signed-off-by: Remi Bergsma <github@remi.nl>
[master/4.6] CLOUDSTACK-9000: logrotate cloudstack-agent out and err logsAdds logrotate rules for cloudstack-agent.{err,out}, jsvc err/out log files may fill up disk. This adds a logrotate config in the rpm packages
cc @remibergsma @wido @wilderrodrigues and others
* pr/992:
CLOUDSTACK-9000: logrotate cloudstack-agent out and err logs
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8793 Enable s2s VPN connection for projects
* pr/879:
CLOUDSTACK-8793 Added project id to create vpn customer gateway, and to the impl of list vpn connections and list vpn customer gateways
Signed-off-by: Remi Bergsma <github@remi.nl>
Pass LbProtocol down to the HAProxyConfiguratorThis will let us specify a new load balancer protocol (tcp-proxy) which enables HAProxy's `send-proxy` functionality.
`send-proxy` / [the PROXY protocol][1] will send the real connection origin IP through to the servers behind HAProxy, without requiring any protocol specific changes (such as HTTP header rewriting).
[1]: http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
This is also inline with what [Amazon ELB now supports][2].
[2]: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
* pr/848:
Pass LbProtocol down to the HAProxyConfigurator
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8962: Dedicated cluster is used for virtual routers that belong to non-dedicated account
Earlier the deployment planner was not handling the case of virtual routers.(In Explicit Dedication)
It was only handling for all instance VMs/user VMs.
Added code for checking the case of Virtual Routers.
* pr/945:
CLOUDSTACK-8962: Dedicated cluster is used for virtual routers that belong to non-dedicated account
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9007 - Write test to check that the /etc/dhcphosts.txt doesn't contain duplicate IPsThis PR contains a test that will cover the fix on PR #981
The tests does the following:
* Creates account, service offering, network offering, network
* Deploys two virtual machines
- Each machine with a pre-assigned IP
* Creates two FW and PF rules
* Checks that SSH into the VMs works
* Checks default routes from both VMs
* Checks that the /etc/dhcphosts.txt contains 1 entry per VM IP
* Destroys/Expunges 1 VM
* Creates a new VM with the same IP as the destroyed one
* Checks that the /etc/dhcphosts.txt contains 1 entry per VM IP
* pr/1002:
CLOUDSTACK-9007 - Add test check that /etc/dhcphosts.txt doesn't contain duplicate IPs
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8993: DHCP fails with "no address available" when an IP is reused
Repopulate /etc/dhcphosts.txt to remove old entries with the same IP address.
* pr/981:
CLOUDSTACK-8993: DHCP fails with "no address available" when an IP is reused
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8957 - VR password server feature is brokenThis PR implements the update of the password as it use to be done with the bash file, but using the Python code for it.
A new integration test was added in order to cover the fix. The test can be found at component/test_password_server.py and does the following:
* Creates an Account
* Creates a Service Offering
* Creates a Network Offering
* Creates a Network
* Creates two Virtual Machines
* Creates two FW rules
* Creates two PF rules
* SSH into each VM and checks the default routes (by pinging 8.8.8.8)
* SSH to the host and executes a command in the router to check the password-[gateway] file
- The file is under /var/cache/cloud/passwords-[gateway]
The [gateway] is replaced by the gateway from each virtual machine NIC.
* pr/998:
CLOUDSTACK-8957 - Add integration test that cheks if the password file is present and not empty
CLOUDSTACK-8957 - Implement password server in configure.py
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8991 - IP address is not removed from VR even after disabling static NATThis PR fixes the Public IP removal form the virtual routers. It also improves the existing test_network.py.
* pr/989:
CLOUDSTACK-8991 - Process the IPs that have been removed
CLOUDSTACK-8991 - Remove public IP form interface in case add = false
CLOUDSTACK-8991 - Make sure the public IP is removed form the router before checking
Signed-off-by: Remi Bergsma <github@remi.nl>
[master/4.6] CLOUDSTACK-8999: Don't override resource if provided by agent.propertiesIf a custom resource (kvm/libvirt implementation) is defined in agent.properties
don't override with the default, but check and fallback to the default if
resource property not defined
A simple if-else fix, cc @remibergsma @wido @wilderrodrigues @borisroman and others
* pr/991:
CLOUDSTACK-8999: Don't override resource if provided by agent.properties
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8941: fix NPE when migrate vm to other zone-wide pools the second timeThis is because the pod_id is set to NULL at the first time when I migrate the instance to a zone-wide pool (not cluster-wide).
* pr/918:
CLOUDSTACK-8941: fix NPE when migrate vm to other zone-wide pools the second time
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8990: start a stopped machine on a specific determinable host on UI
* pr/978:
CLOUDSTACK-8990: start a stopped machine on a specific determinable host on UI
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8985: Deleted volume's removed column not updatedI found this issue when a SolidFire integration test yesterday wasn't able to delete primary storage because it claimed there were still volumes using the primary storage in question (this was due to the removed column not being updated appropriately).
I decided to go with a solution where the delete logic would pass in a volume ID to ignore when computing the used space of the primary storage in question.
* pr/968:
CLOUDSTACK-8985: Deleted volume's removed column not updated
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8911: VM start job got stuck in loop looking for suitable host
VM instance creation job get stuck in the loop, when VMs require local storage there are host that reached max guest limit and remain hosts does have storage available. This happens because the hosts that reach the max guest limit were not getting added to the avoid list and hence the cluster.
Verified the fix on my local setup.
Repro Steps:
1. Take an environment with single cluster and 2 hosts.
2. change the max guest limit for the hypervisor such that on one host max guest limit should reach.
3. change thresholds so that other host should not have enough storage. If required create a VM for sufficient bigger disk.
4. Now deploy a VM with local storage.
5. cluster will never be put in the avoid set and job will keep looking for suitable host.
6. once we increase the max guest limit, VM will deploy or will fail if there is a lack of storage.
* pr/895:
CLOUDSTACK-8911: VM start job got stuck in loop looking for suitable host
Signed-off-by: Remi Bergsma <github@remi.nl>
If a custom resource (kvm/libvirt implementation) is defined in agent.properties
don't override with the default, but check and fallback to the default if
resource property not defined
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 536a8b22c8865dc94281bce6267930a63e03ab77)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Cloudstack-8816 some of the events do not have resource uuidsThe key objects in the context map are sometimes String and sometimes object. This causes missing uuids when an entity put in the context map with key entity.toString is queried with key entity
Testing:
manually tested by deploying a vm and checked that the created events in rabbitmq now has uuids.
events before and after the change are update at https://issues.apache.org/jira/browse/CLOUDSTACK-8816?focusedCommentId=14805239
unittests
```
$ mvn -pl :cloud-api test -Dtest=CallContextTest
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.cloudstack.context.CallContextTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.152 sec - in org.apache.cloudstack.context.CallContextTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.445 s
[INFO] Finished at: 2015-09-18T14:58:53+05:30
[INFO] Final Memory: 55M/448M
[INFO] ------------------------------------------------------------------------
```
* pr/849:
CLOUDSTACK-8816 added missing events
CLOUDSTACK-8816: fixed missing resource uuid in delete network cmd
CLOUDSTACK-8816: fixed missing resource uuid in destroy vm event
Cloudstack-8816: Fixed missing resource uuid in delete snapshot events
CLOUDSTACK-8816: some of the events do not have resource uuids
CLOUDSTACK-8816: some of the events do not have resource uuids
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8917 : Instance tab takes long time to load with 12K Vmsmodified sql that is used for retrieving vm count .
In load test environment listVirtualmachine takes 8-11 sec to load. This environment has around 12k active VMs. Total number of rows is 190K.
Performance bottleneck in listVirtualmachine command is fetching the count and distinct vms.
{noformat}
// search vm details by ids
Pair<List<UserVmJoinVO>, Integer> uniqueVmPair = _userVmJoinDao.searchAndCount(sc, searchFilter);
Integer count = uniqueVmPair.second();
{noformat}
This takes 95% of the total time.
To fetch the count and distinct vms we are using below sqls.
Query 1:
{noformat}
SELECT DISTINCT(user_vm_view.id) FROM user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL ORDER BY user_vm_view.id ASC LIMIT 0, 20
{noformat}
Query 2:
select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL
Query 2 is a problematic query.
If we rewrite the query as mentioned below then it will be ~2x faster.
select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
Mysql Test result:
With 134 active Vms (total rows 349)
mysql> select count(*) from vm_instance;
+----------+
| count(*) |
+----------+
| 349 |
+----------+
1 row in set (0.00 sec)
mysql> select count(*) from user_vm_view;
+----------+
| count(*) |
+----------+
| 135 |
+----------+
1 row in set (0.02 sec)
mysql> select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL;
+--------------------+
| count(distinct id) |
+--------------------+
| 134 |
+--------------------+
1 row in set (0.02 sec)
mysql> select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
+----------+
| count(*) |
+----------+
| 134 |
+----------+
1 row in set (0.01 sec)
With 14326 active Vms (total rows 195660)
mysql> select count(*) from vm_instance;
+----------+
| count(*) |
+----------+
| 195660 |
+----------+
1 row in set (0.04 sec)
mysql> select count(*) from user_vm_view;
+----------+
| count(*) |
+----------+
| 41313 |
+----------+
1 row in set (4.55 sec)
mysql> select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL;
+--------------------+
| count(distinct id) |
+--------------------+
| 14326 |
+--------------------+
1 row in set (7.39 sec)
mysql> select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
+----------+
| count(*) |
+----------+
| 14326 |
+----------+
1 row in set (2.08 sec)
UI test Results:
Before:
After
* pr/894:
CLOUDSTACK-8917 : Instance tab takes long time to load with 12K active VM (total vms: 190K)
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
CLOUDSTACK-8889: delete volume doesnt decrement primary store resource countPrimary Storage count for an account does not decrease when a Data Disk
is deleted belonging to the account unless the VM to which volume
belonged is destroyed
The resource counts are updated even before the disk is actually deleted
resulting in the same value.
Moved the resource counts updation to after the expunge operation as
thats when the disk is actually deleted.
Testing:
Earlier, test_create_multiple_volumes in test/integration/component/test_ps_limits.py failed with error AssertionError: Resource count 37 should match with the expected resource count 32
Before
Test create multiple volumes ... === TestName: test_create_multiple_volumes_1_root_domain_admin | Status : FAILED ===
FAIL
Test create multiple volumes ... === TestName: test_create_multiple_volumes_2_child_domain_admin | Status : FAILED ===
FAIL
After the Fix
Test create multiple volumes ... === TestName: test_create_multiple_volumes_1_root_domain_admin | Status : SUCCESS ===
ok
Test create multiple volumes ... === TestName: test_create_multiple_volumes_2_child_domain_admin | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 2 tests in 334.823s
OK
* pr/860:
CLOUDSTACK-8889: delete volume doesnt decrement primary store resource count
Signed-off-by: Remi Bergsma <github@remi.nl>
Primary Storage count for an account does not decrease when a Data Disk
is deleted belonging to the account unless the VM to which volume
belonged is destroyed
The resource counts are updated even before the disk is actually deleted
resulting in the same value.
Moved the resource counts updation to after the expunge operation as
thats when the disk is actually deleted.
all the tests in test/integration/component/test_ps_limits.py now pass
CLOUDSTACK-8964: Can't create template or volume from snapshot on KVM
* pr/954:
CLOUDSTACK-8964: Can't create template or volume from snapshot
Signed-off-by: Remi Bergsma <github@remi.nl>
