CLOUDSTACK-9006 - ListTemplates API returns result in inconsistent order when called concurrentlyThe order of templates returned in the response is based on a field called sortkey and by default value for the field is set to 0.
With more than 1000 templates, we tried listing the templates with different page sizes concurrently, and we noticed the results being inconsistent.
Thus we added a secondary order by clause to list templates call on tempZonePair column to make sure the results are consistent.
The addOrderby method of Filter class was also not appending , if we added more orderby clauses.
* pr/1009:
CLOUDSTACK-9006 - ListTemplates API returns result in inconsistent order when called concurrently
CLOUDSTACK-9006 - ListTemplates API returns result in inconsistent order when called concurrently
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8902 Restart Network fails in EIP/ELB zoneThe restart network was failing when using external loadbalencer. The failure was because of a number format exception. When BroadcastDomainType.getValue(guestConfig.getBroadcastUri() is executed this returns a string untagged. We were trying to parse this as long so there was a number pointer exception.
This happens only when the vlan uri is vlan://untagged. in other cases were there is a number instead of untagged (vlan tag) this used to succeed. Although we were trying to convert the number to long we were not really using it. we were converting the number to long and then back to string when creating the IpAddressTo. so I removed this unnecessary conversion in this case for fixing the issue at hand.
I did a manual restart of the network and checked for this number format exception in a EIP/ELB setup.
* pr/898:
CLOUDSTACK-89027 Restart Network fails in EIP/ELB zone
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-9016 Fail to create VM instance within VPCbug link https://issues.apache.org/jira/browse/CLOUDSTACK-9016.
CS doe not allocate the ip of the form x.x.x.1 to a guest VM. We seem to incorrectly assume that the first ip in the subnet belongs to the gateway.
* pr/1020:
CLOUDSTACK-9016: Deploy vm with gateway ip address in VPC
CLOUDSTACK-9016 Fail to create VM instance within VPC
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACk-9002: VM deployment is successful even when dhcp entry command fails - Fixed
Reason: The return value of the call to accept() function in the applyRules() function of BasicNetworkTopology.java was not checked for success or failure. As a result even if it fails, exception was not thrown and VM deployment went ahead without any errors.
Fix: Added the necessary checks.
* pr/995:
CLOUDSTACk-9002: VM deployment is successful even when dhcp entry command fails - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8940: Wrong value is inserted into nics table netmask field when creating a VM - Fixed
Problem: When creating a VM in shared network with no service, the value of netmask is added in the table in the CIDR format unlike other cases where it is added as normal string in the format xxx.xxx.xxx.xxx. The netmask column in the nics table has a length of 15 chars which gets violated if the CIDR exceeds it(Max CIDR length can be 18).
Fix: Before storing the netmask convert from CIDR to native format.
* pr/916:
CLOUDSTACK-8940: Wrong value is inserted into nics table netmask field when creating a VM - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
Fixed: Network Update from RVR offering to Standalone offering failsProblem: Moving a RVR network offering to standalone makes the status of VR's as UNKNOWN and Redundant Router marked with YES.
Fix: The network's isRedundant was not getting updated.
* pr/818:
CLOUDSTACK-8844: Network Update from RVR offering to Standalone offering fails - Fixed
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8793 Enable s2s VPN connection for projects
* pr/879:
CLOUDSTACK-8793 Added project id to create vpn customer gateway, and to the impl of list vpn connections and list vpn customer gateways
Signed-off-by: Remi Bergsma <github@remi.nl>
Pass LbProtocol down to the HAProxyConfiguratorThis will let us specify a new load balancer protocol (tcp-proxy) which enables HAProxy's `send-proxy` functionality.
`send-proxy` / [the PROXY protocol][1] will send the real connection origin IP through to the servers behind HAProxy, without requiring any protocol specific changes (such as HTTP header rewriting).
[1]: http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
This is also inline with what [Amazon ELB now supports][2].
[2]: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
* pr/848:
Pass LbProtocol down to the HAProxyConfigurator
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8962: Dedicated cluster is used for virtual routers that belong to non-dedicated account
Earlier the deployment planner was not handling the case of virtual routers.(In Explicit Dedication)
It was only handling for all instance VMs/user VMs.
Added code for checking the case of Virtual Routers.
* pr/945:
CLOUDSTACK-8962: Dedicated cluster is used for virtual routers that belong to non-dedicated account
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8911: VM start job got stuck in loop looking for suitable host
VM instance creation job get stuck in the loop, when VMs require local storage there are host that reached max guest limit and remain hosts does have storage available. This happens because the hosts that reach the max guest limit were not getting added to the avoid list and hence the cluster.
Verified the fix on my local setup.
Repro Steps:
1. Take an environment with single cluster and 2 hosts.
2. change the max guest limit for the hypervisor such that on one host max guest limit should reach.
3. change thresholds so that other host should not have enough storage. If required create a VM for sufficient bigger disk.
4. Now deploy a VM with local storage.
5. cluster will never be put in the avoid set and job will keep looking for suitable host.
6. once we increase the max guest limit, VM will deploy or will fail if there is a lack of storage.
* pr/895:
CLOUDSTACK-8911: VM start job got stuck in loop looking for suitable host
Signed-off-by: Remi Bergsma <github@remi.nl>
Cloudstack-8816 some of the events do not have resource uuidsThe key objects in the context map are sometimes String and sometimes object. This causes missing uuids when an entity put in the context map with key entity.toString is queried with key entity
Testing:
manually tested by deploying a vm and checked that the created events in rabbitmq now has uuids.
events before and after the change are update at https://issues.apache.org/jira/browse/CLOUDSTACK-8816?focusedCommentId=14805239
unittests
```
$ mvn -pl :cloud-api test -Dtest=CallContextTest
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.cloudstack.context.CallContextTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.152 sec - in org.apache.cloudstack.context.CallContextTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.445 s
[INFO] Finished at: 2015-09-18T14:58:53+05:30
[INFO] Final Memory: 55M/448M
[INFO] ------------------------------------------------------------------------
```
* pr/849:
CLOUDSTACK-8816 added missing events
CLOUDSTACK-8816: fixed missing resource uuid in delete network cmd
CLOUDSTACK-8816: fixed missing resource uuid in destroy vm event
Cloudstack-8816: Fixed missing resource uuid in delete snapshot events
CLOUDSTACK-8816: some of the events do not have resource uuids
CLOUDSTACK-8816: some of the events do not have resource uuids
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8917 : Instance tab takes long time to load with 12K Vmsmodified sql that is used for retrieving vm count .
In load test environment listVirtualmachine takes 8-11 sec to load. This environment has around 12k active VMs. Total number of rows is 190K.
Performance bottleneck in listVirtualmachine command is fetching the count and distinct vms.
{noformat}
// search vm details by ids
Pair<List<UserVmJoinVO>, Integer> uniqueVmPair = _userVmJoinDao.searchAndCount(sc, searchFilter);
Integer count = uniqueVmPair.second();
{noformat}
This takes 95% of the total time.
To fetch the count and distinct vms we are using below sqls.
Query 1:
{noformat}
SELECT DISTINCT(user_vm_view.id) FROM user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL ORDER BY user_vm_view.id ASC LIMIT 0, 20
{noformat}
Query 2:
select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL
Query 2 is a problematic query.
If we rewrite the query as mentioned below then it will be ~2x faster.
select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
Mysql Test result:
With 134 active Vms (total rows 349)
mysql> select count(*) from vm_instance;
+----------+
| count(*) |
+----------+
| 349 |
+----------+
1 row in set (0.00 sec)
mysql> select count(*) from user_vm_view;
+----------+
| count(*) |
+----------+
| 135 |
+----------+
1 row in set (0.02 sec)
mysql> select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL;
+--------------------+
| count(distinct id) |
+--------------------+
| 134 |
+--------------------+
1 row in set (0.02 sec)
mysql> select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
+----------+
| count(*) |
+----------+
| 134 |
+----------+
1 row in set (0.01 sec)
With 14326 active Vms (total rows 195660)
mysql> select count(*) from vm_instance;
+----------+
| count(*) |
+----------+
| 195660 |
+----------+
1 row in set (0.04 sec)
mysql> select count(*) from user_vm_view;
+----------+
| count(*) |
+----------+
| 41313 |
+----------+
1 row in set (4.55 sec)
mysql> select count(distinct id) from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL;
+--------------------+
| count(distinct id) |
+--------------------+
| 14326 |
+--------------------+
1 row in set (7.39 sec)
mysql> select count(*) from (select distinct id from user_vm_view WHERE user_vm_view.account_type != 5 AND user_vm_view.display_vm = 1 AND user_vm_view.removed IS NULL) as temp;
+----------+
| count(*) |
+----------+
| 14326 |
+----------+
1 row in set (2.08 sec)
UI test Results:
Before:
After
* pr/894:
CLOUDSTACK-8917 : Instance tab takes long time to load with 12K active VM (total vms: 190K)
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
Primary Storage count for an account does not decrease when a Data Disk
is deleted belonging to the account unless the VM to which volume
belonged is destroyed
The resource counts are updated even before the disk is actually deleted
resulting in the same value.
Moved the resource counts updation to after the expunge operation as
thats when the disk is actually deleted.
all the tests in test/integration/component/test_ps_limits.py now pass
CLOUDSTACK-8964: Can't create template or volume from snapshot on KVM
* pr/954:
CLOUDSTACK-8964: Can't create template or volume from snapshot
Signed-off-by: Remi Bergsma <github@remi.nl>
uuid is missing in the first event of VM create as the entity is just
created and never put in the Context.
Added the entity uuid to context on successful creation.
This issue happens on KVM.
Normally the SSVM will process the CopyCommand from snapshot to template.
However, Ovm3HypervisorGuru chooses a KVM hypervisor to process the CopyCommand.
This is obviously wrong.
The labeling was broken. Only labels assigned at zone creation
were used, changing labels was not working. Tested with changing
a label and checking it.
As a bonus fixed the consistency of KVM in Dutch compared to other
traffic labels in dutch and copied in the OVM3 translated label
in other languages.
Earlier the deployment planner was not handling the case of virtual routers.(In Explicit Dedication)
It was only handling for all instance VMs/user VMs.
Added code for checking the case of Virtual Routers.
- The default is Accept and will be changed based on the configuration of the offering.
CLOUDSTACK-8934 - The default egress is set as Deny in the router.
- We had to change it on the Java side in order to make the apply it once the default is defined as allowed on the net offering
CLOUDSTACK-8864: Not able to add TCP port forwarding rule in VPN for specific ports
Setting port forwarding rules for port 500,1701 and 4500 after enabling VPN, gives the error message "The range specified, xxxx, conflicts with rule xxxx which has xxxx." This happens because the rules added for vpn doesn't have a matching condition to allow port forwarding rules.
Added a unit test to verify the detectRulesConflict function of FirewallManagerImpl.
* pr/851:
CLOUDSTACK-8864: Not able to add TCP port forwarding rule in VPN for specific ports
Signed-off-by: Remi Bergsma <github@remi.nl>
CLOUDSTACK-8834: Fixed unable to download Template , when in multi zonesWe were listing image stores by zone id which was resulting in listing of only one image store
If in that image store its download state is not DOWNLOADED then download template is failing
* pr/804:
CLOUDSTACK-8834: Fixed unable to download Template , when in multi zones We were listing image stores by zone id which was resulting in listing of only one image store If in that image store its download state is not DOWNLOADED then download template is failing
Signed-off-by: Wido den Hollander <wido@widodh.nl>
Cloudstack 8816 entityuuid missing in some of the eventsIn some of the events generated, entity uuid was missing making it difficult to find the entity. Fixed the same.
Tested it on rabbitmq instance.
There are the events before after the fix:
Before
--------------------------------------------------------------------------------
routing_key: management-server.ActionEvent.ACCOUNT-DELETE.Account.*
exchange: cloudstack-events
message_count: 2
payload:
{"eventDateTime":"2015-09-04 17:59:24 +0530","status":"Scheduled","description":"deleting User test4 (id: 28) and accountId \u003d 28","event":"ACCOUNT.DELETE","Account":"c09e2e81-8edc-4c27-b072-25005b522b63","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 304
payload_encoding: string
redelivered: False
--------------------------------------------------------------------------------
routing_key: management-server.AsyncJobEvent.complete.Account.*
exchange: cloudstack-events
message_count: 0
payload: {"cmdInfo":"{\"id\":\"9dd3abc2-3f8b-4852-aa60-a74b234acb13\",\"response\":\"json\",\"sessionkey\":\"5ig1ItP2_5v-mgY4cVJbJN5hw_w\",\"ctxDetails\":\"
{\\\"interface com.cloud.user.Account\\\":\\\"9dd3abc2-3f8b-4852-aa60-a74b234acb13\\\"}
\",\"cmdEventType\":\"ACCOUNT.DELETE\",\"expires\":\"2015-09-07T11:11:56+0000\",\"ctxUserId\":\"2\",\"signatureversion\":\"3\",\"httpmethod\":\"GET\",\"uuid\":\"9dd3abc2-3f8b-4852-aa60-a74b234acb13\",\"ctxAccountId\":\"2\",\"ctxStartEventId\":\"447\"}","instanceType":"Account","jobId":"5004989d-0cde-4922-8afa-66bf38b75ea7","status":"SUCCEEDED","processStatus":"0","commandEventType":"ACCOUNT.DELETE","resultCode":"0","command":"org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd","jobResult":"org.apache.cloudstack.api.response.SuccessResponse/null/
{\"success\":true}
","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 914
payload_encoding: string
redelivered: False
--------------------------------------------------------------------------------
After
--------------------------------------------------------------------------------
routing_key: management-server.ActionEvent.ACCOUNT-DELETE.Account.e5e2db91-414d-484c-99d5-c4e265c14ad8
exchange: cloudstack-events
message_count: 13
payload: {"eventDateTime":"2015-09-07 17:32:26 +0530","status":"Completed","description":"Successfully completed deleting account. Account Id: 45","event":"ACCOUNT.DELETE","entityuuid":"e5e2db91-414d-484c-99d5-c4e265c14ad8","entity":"com.cloud.user.Account","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 344
payload_encoding: string
redelivered: True
--------------------------------------------------------------------------------
routing_key: management-server.AsyncJobEvent.complete.Account.e5e2db91-414d-484c-99d5-c4e265c14ad8
exchange: cloudstack-events
message_count: 12
payload: {"cmdInfo":"{\"id\":\"e5e2db91-414d-484c-99d5-c4e265c14ad8\",\"response\":\"json\",\"sessionkey\":\"8AJVbn8HIpg5LZ_VaVfSPs_QN2k\",\"ctxDetails\":\"{\\\"interface com.cloud.user.Account\\\":\\\"e5e2db91-414d-484c-99d5-c4e265c14ad8\\\"}\",\"cmdEventType\":\"ACCOUNT.DELETE\",\"expires\":\"2015-09-07T12:17:42+0000\",\"ctxUserId\":\"2\",\"signatureversion\":\"3\",\"httpmethod\":\"GET\",\"uuid\":\"e5e2db91-414d-484c-99d5-c4e265c14ad8\",\"ctxAccountId\":\"2\",\"ctxStartEventId\":\"465\"}","instanceType":"Account","instanceUuid":"e5e2db91-414d-484c-99d5-c4e265c14ad8","jobId":"0bb08486-6d9f-4e9f-bfef-b7463c42e71b","status":"SUCCEEDED","processStatus":"0","commandEventType":"ACCOUNT.DELETE","resultCode":"0","command":"org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd","jobResult":"org.apache.cloudstack.api.response.SuccessResponse/null/{\"success\":true}","account":"bd73dc2e-35c0-11e5-b094-d4ae52cb9af0","user":"bd7ea748-35c0-11e5-b094-d4ae52cb9af0"}
payload_bytes: 968
payload_encoding: string
redelivered: True
--------------------------------------------------------------------------------
* pr/782:
CLOUDSTACK-8816 Systemvm reboot event doesnt have uuids. Fixed the same
CLOUDSTACK-8816: Project UUID is not showing for some of operations in RabbitMQ.
CLOUDSTACK-8816: entity uuid missing in create network event
CLOUDSTACK-8816: instance uuid is missing in events for delete account
CLOUDSTACK-8816 Fixed entityUuid missing in some cases is events
Signed-off-by: Rajani Karuturi <rajani.karuturi@citrix.com>
unittests to verify empty password is not allowed during account/user create
* pr/807:
unittests to verify empty password is not allowed during account create
Signed-off-by: Wido den Hollander <wido@widodh.nl>
We were listing image stores by zone id which was resulting in listing of only one image store
If in that image store its download state is not DOWNLOADED then download template is failing
