CLOUDSTACK-9656 Preventing autoboxing NPE in Usage by setting a default role when not foundhttps://issues.apache.org/jira/browse/CLOUDSTACK-9656
This is a workaround to avoid NPE when using the usage server with Projects
To reproduce bug:
Create project
Add account to project
Create VM within that project
Run the usage server
Same steps to test resolution
* pr/1820:
CLOUDSTACK-9656: Preventing autoboxing NPE in Usage by setting a default role when not found
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9584: Fix intermittent test failure in `test_volumes`The component/test_volume failures happen when disk offering is random selected to be a custom one. This fixes that.
* pr/1822:
CLOUDSTACK-9584: Fix intermittent test failure in `test_volumes`
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9659: mismatch in traffic type in ip_associations.json and ips.jsonAs part of the bug 'CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly'
issue of mismatch of traffic type represented by 'nw_type' in config sent by management server in
ip_associations.json and how it is persisted in the ips.json data bag are differnet,
is addressed, however missed the change in final merge.
this bug is to add the functionality in cs_ip.py, to lower the traffic type sent by management server before persisting in the ips.json databag
* pr/1821:
CLOUDSTACK-9659: mismatch in traffic type in ip_associations.json and ips.json
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9637: Template create from snapshot does not populate vm_t**ISSUE**
============
Template create from snapshot does not populate vm_template_details
**REPRO STEPS**
==================
1. Register a template A and specify property:
Root disk controller: scsi
NIC adapter type: E1000
Keyboard type: us
2. Create a vm instance from template A
3. Take volume snapshot for vm instance
4. Delete VM instance
5. Switch to "Storage->Snapshots", convert snapshot to a template B
6. Observe template B does not inherit property from template A, the table vm_template_details is empty
**SOLUTION**: Retrieve and add source template details to VMTemplateVO.
Before Fix:
```
mysql> select id,name,source_template_id from vm_template where id=202;
+-----+--------+--------------------+
| id | name | source_template_id |
+-----+--------+--------------------+
| 202 | Debian | NULL |
+-----+--------+--------------------+
1 row in set (0.00 sec)
mysql> select * from vm_template_details where template_id=202;
+----+-------------+--------------------+-------+---------+
| id | template_id | name | value | display |
+----+-------------+--------------------+-------+---------+
| 1 | 202 | keyboard | us | 1 |
| 2 | 202 | nicAdapter | E1000 | 1 |
| 3 | 202 | rootDiskController | scsi | 1 |
+----+-------------+--------------------+-------+---------+
3 rows in set (0.00 sec)
mysql> select id,name,source_template_id from vm_template where source_template_id=202;
+-----+----------------+--------------------+
| id | name | source_template_id |
+-----+----------------+--------------------+
| 203 | derived-debian | 202 |
+-----+----------------+--------------------+
1 row in set (0.00 sec)
mysql> select * from vm_template_details where template_id=203;
Empty set (0.00 sec)
After Fix:
mysql> select id,name,source_template_id from vm_template where source_template_id=202;
+-----+--------------------------+--------------------+
| id | name | source_template_id |
+-----+--------------------------+--------------------+
| 203 | derived-debian | 202 |
| 204 | debian-derived-after-fix | 202 |
+-----+--------------------------+--------------------+
2 rows in set (0.00 sec)
mysql> select * from vm_template_details where template_id=204;
+----+-------------+--------------------+-------+---------+
| id | template_id | name | value | display |
+----+-------------+--------------------+-------+---------+
| 4 | 204 | keyboard | us | 1 |
| 5 | 204 | nicAdapter | E1000 | 1 |
| 6 | 204 | rootDiskController | scsi | 1 |
+----+-------------+--------------------+-------+---------+
3 rows in set (0.00 sec)
```
**Marvin Test :** test_template_from_snapshot_with_template_details.py
**Result:**
```
test_01_create_template_snampshot (integration.component.test_template_from_snapshot_with_template_details.TestCreateTemplate) ... === TestName: test_01_create_template_snampshot | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 864.523s
OK
```
* pr/1805:
CLOUDSTACK-9637: Template create from snapshot does not populate vm_template_details
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
As part of the bug 'CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly'
issue of mismatch of traffic type represented by 'nw_type' in config sent by management server in
ip_associations.json and how it is persisted in the ips.json data bag are differnet,
is addressed, however missed the change in final merge.
this bug is to add the functionality in cs_ip.py, to lower the traffic type sent by management server before persisting in the ips.json databag
CLOUDSTACK-9646: No usage is generated for uploaded templates/volumes from localpublished usage events on successful upload of template or volume.
* pr/1809:
CLOUDSTACK-9646: No usage is generated for uploaded templates/volumes from local
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctlyAs pointed out in CLOUDSTACK-9339, in case of multiple public IP's from different public IP ranges are associated with VR, VR functionality is broken from 4.6. Below are the brief list of problems specific to non-VPC networks addressed in the PR. This PR handles both VPC and non-VPC scenarios.
- reverse traffic for the connections accepted on the eth3 and above public interfaces are getting blocked. Need a rule for e.g "-A FORWARD -i eth3 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT" in the FORWARD chain of filter table to permit reverse path traffic for established connections.
- outbound public traffic from eth0 to eth3 (or for interfaces above like eth4 eth5 etc) needs rule to run through FW_OUTBOUND chain in the filter table
- network stats on public interfaces eth3 are getting gathered
- default gateway is missing in the device specific routing table, resulting in traffic to be looked up in main routing table
- creating a device specific route table is generating "from all lookup Table_eth3" in the
ip rules, resulting in rest of the traffic getting blocked.
Picked few commits from #1519 from dsclose (https://github.com/apache/cloudstack/pull/1519) submitted for 4.7
Marvin tests are added to test below
- Static NAT works on the public interfaces above eth2, in case non-vpc networks
- Portforwarding works on the public interfaces above eth2, in case non-vpc networks
- Route tables are configured as expected for the device specific table for the public interfaces above eth2, in case non-vpc networks
- IP tables rules are as expected for the traffic from and to the public interfaces above eth2, in case non-vpc networks
* pr/1659:
CLOUDSTACK-9339 Virtual Routers don't handle Multiple Public Interfaces correctly
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Issue: Currently many versions of SUSE Linux does not have any hypervisor mapping entry in guest_os_hypervisor table in cloud database for VMware 6.0. Also observed that the guest_os_name field is incorrect for some SUSE Linux variants, which results in deployed instance (with SUSE Linux) set to guest OS type as "Other (64-bit)" on vCenter, which would not represent the guest OS accurately on hypervisor.
Fix: Add the missing hypervisor mappings
-when processing static nat rule, add a mangle table rule, to mark the traffic
from the guest vm when it has associated static nat rule so that traffic gets
routed using the route tabe of the device which has public ip associated
-fix the case where nic_device_id is empty when ip is getting disassociated
resulting in empty deviceid in ips.json
-add utility methods in CsRule, and CsRoute to add 'ip rule' and 'ip route' rules respectivley
-ensure traffic from all public interfaces are connection marked with device number, and restored
for the reverse traffic. use the connection marked number to do device specific routing table lookup
fill the device specific routing table with default route
-component tests for testing multiple public interfaces of VR
CLOUDSTACK-9632: Upgrade bouncy castle to version 1.55- Upgrades Maven dependency version to v1.55
- Fixes bountycastle usages and issues
- Adds timeout to jetty/annotation scanning
- Picks up PR #1510 by Daan
* pr/1799:
CLOUDSTACK-9632: Upgrade bouncy castle to version 1.55
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Upgrades Maven dependency version to v1.55
- Fixes bountycastle usages and issues
- Adds timeout to jetty/annotation scanning
- Fixes servlet issue, uses servlet 3.1.0
- Downgrade javassist used by reflections to fix annotation process errors
- Make console-proxy-rdp bc dependency same as rest of the codebase
- Picks up PR #1510 by Daan
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9564: Fix NPE due to intermittent test assertionThe test assertion on a pool object may return a null object, as objects
can be randomly expired/tombstoned. This will fix a NPE sometimes seen due
to recently merge for the fix for CLOUDSTACK-9564.
(we can merge this if Travis passes)
/cc @abhinandanprateek @murali-reddy
* pr/1816:
CLOUDSTACK-9564: Fix NPE due to intermittent test assertion
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The test assertion on a pool object may return a null object, as objects
can be randomly expired/tombstoned. This will fix a NPE sometimes seen due
to recently merge for the fix for CLOUDSTACK-9564.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary storeThe race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously.
Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
* pr/1765:
Cloudstack 9586: When using local storage with Xenserver prepareTemplate does not work with multiple primary store The race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously. Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9564: Fix memory leaks in VmwareContextPoolIn a recent management server crash, it was found that the largest contributor
to memory leak was in VmwareContextPool where a registry is held (arraylist)
that grows indefinitely. The list itself is not used anywhere or consumed. There
exists a hashmap (pool) that returns a list of contexts for existing poolkey
(address/username) that is used instead.
This fixes the issue by removing the arraylist registry, and limiting the
length of the context list for a given poolkey.
@blueorangutan package
* pr/1729:
CLOUDSTACK-9564: Fix memory leaks in VmwareContextPool
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9635: fix test_privategw_acl.pyensure VLAN used for createPrivateGateway is determined after the guest
networks in the VPC is created, so that we skip VLAN allocated for guest
network for the private network of vpc gateway
* pr/1802:
CLOUDSTACK-9635: fix test_privategw_acl.py
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
ensure VLAN used for createPrivateGateway is determined after the guest
networks in the VPC is created, so that we skip VLAN allocated for guest
network for the private network of vpc gateway
The race condition will happen whenever there are multiple primary storages and the CS tries to mount the secondary store to xenserver host simultaneously.
Due to synchronised block one mount will be successful and other thread will get the already mounted SR. Without the fix the two thread will try to mount it parallely and one will fail on Xenserver.
This fixes build_asf.sh release script to update checkstyle pom.xml with the
provided new version.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9636: The host alerts box should be named as hosts in Alerts.The host Alerts box shows hosts in Alerts. The name host Alerts is misleading,
it should be changed to hosts in alerts.
For rest of the languages, it should be modified accordingly.
As I am not familiar with other languages, contributors familiar with other languages can suggest the change. or Open a new PR.
* pr/1803:
CLOUDSTACK-9636: The host alerts box should be named as hosts in Alerts.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-8854: Apple Mac OS/X VM get created without USB controller in ESXi hypervisorsCLOUDSTACK-8854: Apple Mac OS/X VM get created without USB controller in ESXi hypervisors
Problem Description: CloudStack doesnt add a USB controller to the Apple Mac OS X VMs created in ESXi hypervisors. But, vSphere Client, by default, adds a USB Controller to the Mac OS VMs. Mac OS X machines require USB Controller for USB mouse and keyboard access.
Root Cause: The Guest OS details are specified in the Virtual Machine Configuration Spec for creating the VM (using the SDK API) in the EXSi hypervisor. No USB Controller is added to the Virtual Machine Configuration Spec. As the guest OS Identification details are specified in the VM Configuration Spec, It is assumed that the Create VM SDK API would create the defaults in the VM same as vSphere Client. But, as per the observation, USB Controller is not added to the Guest OS - Mac OS VM created through the SDK API.
Resolution: When the Guest OS is Apple Mac OS, Add the USB Controller (EHCI+UHCI - Mac supported) to the Virtual Machine Configuration Spec before Creating or Starting the VM. For any existing Mac OS VMs, Stop and Start to add the USB Controller. For new VMs with Mac OS, USB Controller is added automatically.
* pr/828:
CLOUDSTACK-8854: Apple Mac OS/X VM get created without USB controller in ESXi hypervisors
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
In a recent management server crash, it was found that the largest contributor
to memory leak was in VmwareContextPool where a registry is held (arraylist)
that grows indefinitely. The list itself is not used anywhere or consumed. There
exists a hashmap (pool) that returns a list of contexts for existing poolkey
(address/username) that is used instead.
This fixes the issue by removing the arraylist registry, and limiting the
length of the context list for a given poolkey.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
fix marvin test failure test_router_dhcp_optsmarvin, VirtualMachine object's, nic attribute does not have nic's in any
particualr order in the array. so check isdefault attribute to the get non-default nic
earlier test made assumption that nic[0] is default nic, which is not true always
* pr/1801:
CLOUDSTACK-9634: fix marvin test failure test_router_dhcp_opts
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
marvin, VirtualMachine object, nic attribute does not have nic's in any
particualr order in the array. soi check isdefault attribute to the get non-default nic
earlier test made assumption that nic[0] is default nic, which is not true always
CLOUDSTACK-9624 Incorrect hypervisor mapping of guest os Windows 2008 Server R2 (64-bit) for VMware**JIRA ticket**
CLOUDSTACK-9624 Incorrect hypervisor mapping of guest os Windows 2008 Server R2 (64-bit) for VMware
**Issue**
Guest OS Windows Server 2008 R2 (64-bit) is being mapped to incorrect guest os at hypervisor, which is winLonghorn64Guest, same as that of Windows Server 2008 (64-bit).
Due to this the VM's guest os type was set to "Other (64-bit)", which would not represent the guest OS accurately on hypervisor.
**Solution**
Fix is to update incorrect guest_os_name field value in DB table cloud.guest_os_hypervisor.
Th query is,
UPDATE IGNORE `cloud`.`guest_os_hypervisor` SET guest_os_name = 'windows7Server64Guest' WHERE guest_os_id IN (SELECT id FROM guest_os WHERE display_name LIKE 'windows%2008%r2%64%') AND hypervisor_type = 'VMware' AND hypervisor_version != 'default';
After running above query, the 6 updated rows looks like
UPDATE IGNORE `cloud`.`guest_os_hypervisor` SET guest_os_name = 'windows7Server64Guest' WHERE guest_os_id IN (SELECT id FROM guest_os WHERE display_name LIKE 'windows%2008%r2%64%') AND hypervisor_type = 'VMware' AND hypervisor_version != 'default';
Query OK, 6 rows affected (0.01 sec)
Rows matched: 6 Changed: 6 Warnings: 0
mysql> select * from guest_os_hypervisor where guest_os_id in (select id from guest_os where display_name like 'windows%2008%r2%64%') and hypervisor_type = 'VMware' and hypervisor_version != 'default';
+------+-----------------+-----------------------+-------------+--------------------+--------------------------------------+---------------------+---------+-----------------+
| id | hypervisor_type | guest_os_name | guest_os_id | hypervisor_version | uuid | created | removed | is_user_defined |
+------+-----------------+-----------------------+-------------+--------------------+--------------------------------------+---------------------+---------+-----------------+
| 1307 | VMware | windows7Server64Guest | 54 | 4.0 | 98fce372-b271-11e6-b56b-4e61adb7c6b1 | 2016-11-24 23:42:44 | NULL | 0 |
| 1448 | VMware | windows7Server64Guest | 54 | 4.1 | 990abdcc-b271-11e6-b56b-4e61adb7c6b1 | 2016-11-24 23:42:45 | NULL | 0 |
| 1589 | VMware | windows7Server64Guest | 54 | 5.0 | 99166f75-b271-11e6-b56b-4e61adb7c6b1 | 2016-11-24 23:42:45 | NULL | 0 |
| 1730 | VMware | windows7Server64Guest | 54 | 5.1 | 9930ff30-b271-11e6-b56b-4e61adb7c6b1 | 2016-11-24 23:42:45 | NULL | 0 |
| 1871 | VMware | windows7Server64Guest | 54 | 5.5 | 993acb18-b271-11e6-b56b-4e61adb7c6b1 | 2016-11-24 23:42:45 | NULL | 0 |
| 2381 | VMware | windows7Server64Guest | 54 | 6.0 | 9cb53675-b271-11e6-b56b-4e61adb7c6b1 | 2016-11-24 18:12:51 | NULL | 0 |
+------+-----------------+-----------------------+-------------+--------------------+--------------------------------------+---------------------+---------+-----------------+
6 rows in set (0.01 sec)
**Tests**
Registered a template with Windows 2008 R2 (64-bit) guest OS and deployed an instance from the template. Found that the VM appeared in vCenter with valid guest OS type instead of "Other (64-bit)" shown up before the fix.
* pr/1793:
CLOUDSTACK-9624 Incorrect hypervisor mapping of guest os Windows 2008 Server R2 (64-bit) for VMware
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9538: FIX failure in Deleting Snapshot From Primary Storage RBD Storage if vm has been removed
* pr/1710:
CLOUDSTACK-9538: FIX failure in Deleting Snapshot From Primary Storage RBD Storage if vm has been removed
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-9584: run component tests in Travis runThis would run additional component tests in Travis run.
* pr/1755:
CLOUDSTACK-9584: run component tests in Travis run
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Issue
Guest OS Windows Server 2008 R2 (64-bit) is being mapped to incorrect guest os at hypervisor, which is winLonghorn64Guest, same as that of Windows Server 2008 (64-bit).
Due to this the VM's guest os type was set to "Other (64-bit)", which would not represent the guest OS accurately on hypervisor.
Solution
Fix is to update incorrect guest_os_name field value in DB table cloud.guest_os_hypervisor.
Th query is,
UPDATE IGNORE cloud.guest_os_hypervisor SET guest_os_name = 'windows7Server64Guest' WHERE guest_os_id IN (SELECT id FROM guest_os WHERE display_name LIKE 'windows%2008%r2%64%') AND hypervisor_type = 'VMware' AND hypervisor_version != 'default';
