29953 Commits

Author SHA1 Message Date
Will Stevens
9320f858f3 Merge pull request #1460 from shapeblue/jsb/jenv-pyenv
CLOUDSTACK-9334: Support jenv and pyenv to manage Java and Python versions  * Adds .java-version to specify Java 1.7 as the JDK version to use
    for CloudStack development.  jenv does not support comments in
    this file.  Therefore, no license header is present.
  * Adds .python-vresion to specify Python 2.7.11 as the Python to
    use for CloudStack development.  pyenv does not support comments
    in this file.  Therefore, no license header is present.
  * Adds requirements.txt to specify the default dependencies used
    for CloudStack development.  It does not include Marvin's
    dependencies because it's bundle already defines the dependencies
    which are installed with Marvin.
  * Updates INSTALL.md to explain how to utilize jenv and pyenv for
    CloudStack development

* pr/1460:
  CLOUDSTACK-9334: Support jenv and pyenv to manage Java and Python versions

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:50:40 -04:00
Will Stevens
8c3722d953 Merge pull request #1444 from rafaelweingartner/workAroundPR780
CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VMThis PR introduces the changes proposed in PR #780 with some work to make the code null safe.

During this PR, I have also removed some unused code.

* pr/1444:
  Removed unnecessary check when creating the “userVmResponse” object.
  Fixed issues from CLOUDSTACK-8800 that were introduced in PR 780
  CLOUDSTACK-8800 : Improved the listVirtualMachines API call to include memory utilization information for a VM for xenserver,kvm and for vmware.

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:49:14 -04:00
Will Stevens
20cf8b23c0 Merge pull request #1453 from GabrielBrascher/brascher-removeUnusedClasses2
Remove classes with no referencesI used UCDetector (http://www.ucdetector.org/) as a plugin for Eclipse.  With this tool, I discovered a lot of code without any reference (variables, methods and classes).

Following the work that was done at [https://github.com/apache/cloudstack/pull/1448]; this pull request had the goal of removing some of these classes. To check if I wasn't missing anything I searched for any file that could reference some of those classes. As I haven't found any way of these classes being used, they were removed. Note that some of them I found other references, but references such as commented lines or tests, nothing that could indicate their use (as XML files configuring beans or another class instantiating an object with "new").

Waiting for tests. Please tell me if I am missing something.

Removed Classes:
- org.apache.cloudstack.framework.jobs.JobCancellationException (**Note:** removed
variable JobCancellationException in com.cloud.utils.SerialVersionUID)
- org.apache.cloudstack.ldap.NoSuchLdapUserException (**Note:** removed test file
/cloud-plugin-user-authenticator-ldap/test/groovy/org/apache/cloudstack/ldap/NoSuchLdapUserExceptionSpec.groovy)
- com.cloud.agent.api.storage.CreateVolumeOVAAnswer
- com.cloud.exception.MissingParameterValueException
- org.apache.cloudstack.api.response.StatusResponse
- org.apache.cloudstack.api.response.VolumeDetailResponse
- org.apache.cloudstack.api.response.UpgradeVmResponse
- org.apache.cloudstack.api.response.AddIpToVmNicResponse
- org.apache.cloudstack.api.response.TemplateZoneResponse (**Note:** at
org.apache.cloudstack.api.response.TemplateResponse, there is this
comment "To avoid breaking backwards compatibility, we still treat a
template at different zones as different templates, so not embedding
template_zone information in this TemplateZoneResponse set. `private
Set<TemplateZoneResponse> zones;`" but right now it is not used)
- org.apache.cloudstack.api.response.NicDetailResponse

* pr/1453:
  Removed classes with no reference

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:48:03 -04:00
Will Stevens
3f97908417 Merge release branch 4.8 to master
* 4.8:
  writeIfNotHere requires an array of strings, not a string
2016-05-12 16:47:09 -04:00
Will Stevens
a1e181cd7e Merge release branch 4.7 to 4.8
* 4.7:
  writeIfNotHere requires an array of strings, not a string
2016-05-12 16:46:28 -04:00
Will Stevens
170765e90d Merge pull request #1456 from greenqloud/pr-4.7-baremetal_metadata_parameter_bugfix
writeIfNotHere requires an array of strings, not a string

* pr/1456:
  writeIfNotHere requires an array of strings, not a string

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:45:34 -04:00
Will Stevens
f88cb88097 Merge pull request #1538 from shapeblue/nio-test-timeoutfix
Disable NioTestHistorically NioTest has caused issue in CI environment and several developer machines due to network requirements which could be disabled by firewall or security enforcers such as selinux. This disables the test once again using a historic commit 881a6e1
Signed-off-by: Rohit Yadav <rohit.yada

To build and just run this test: mvn clean install -pl utils -Dtest=NioTest

* pr/1538:
  utils: Disable NioTest

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 16:38:37 -04:00
Will Stevens
688522ecd4 Merge pull request #1385 from DaanHoogland/CLOUDSTACK-9265
CLOUDSTACK-9265 cleanup around httpclient versionssome cleanup done
- replaced HttpStatus from org.apache.commons.httpclient with that from org.apache.http
- removed unthrown HttpException
- left auto reformat in place

* pr/1385:
  CLOUDSTACK-9265 cleanup around httpclient versions

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:11:28 -04:00
Will Stevens
c79affaefb Merge pull request #1528 from mike-tutkowski/marvin_methods
CLOUDSTACK-9373: Class methods over-shawdowing instance methodsWe have some methods in base.py that are named the same.

Per my findings below, Python methods in a class should not be named the same even if one is a class method and the other is an instance method.

The solution discussed on dev@ is to remove the instance versions (reason listed in e-mail text, which is listed in JIRA ticket).

https://issues.apache.org/jira/browse/CLOUDSTACK-9373

* pr/1528:
  CLOUDSTACK-9373: Removing a few instance methods where there are class methods that are overshadowing them

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:08:53 -04:00
Will Stevens
7e41747c33 Merge pull request #816 from mike-tutkowski/addremovehosts2
Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

This PR addresses the following JIRA ticket:

https://issues.apache.org/jira/browse/CLOUDSTACK-8813

The problem is that there needs to be notifications sent when a host is added to, about to be removed from, and removed from a cluster.

Such notifications can be used for many purposes. For example, it can allow storage plug-ins to update ACLs on their storage systems. Also, it can allow us to clean up IQNs from ESXi hosts that are no longer needed.

* pr/816:
  CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:07:36 -04:00
Will Stevens
143265c49f Merge pull request #1502 from shapeblue/outofband-master
CLOUDSTACK-9299: Out-of-band Management for CloudStackSupport access to a hosts out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature, please install `ipmitool` (using yum/apt/brew) and `ipmisim`:
https://pypi.python.org/pypi/ipmisim

The default ipmitool location is assumed in /usr/bin, if this is different in your env please fix the global setting, see FS for details on various global settings.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

/cc @jburwell @swill @abhinandanprateek @murali-reddy @borisstoyanov

* pr/1502:
  maven: ignore utils/testsmallfileinactive for rat checking
  CLOUDSTACK-9378: Fix for #1497
  HypervisorUtilsTest: increate timeout to 8seconds
  travis: Use patched version of ipmitool for tests
  CLOUDSTACK-9299: Out-of-band Management for CloudStack

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:06:13 -04:00
Will Stevens
103d62ee02 Merge release branch 4.8 to master
* 4.8:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:04:53 -04:00
Will Stevens
dbbc37ecc7 Merge release branch 4.7 to 4.8
* 4.7:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  Handle private gateways more reliably
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists
  Add private gateway IP to router initialization config
  apply static routes on change to master state
2016-05-12 11:03:57 -04:00
Will Stevens
3fab75772f Merge pull request #1474 from remibergsma/47_private_gw_initial_config
Handle private gateways more reliablyWhen initialising a VPC router we need to know which IP/device corresponds to a private gateway. This is to solve a problem when stop/starting a VPC router (which gets the private gateway config as a guest network and as a result breaks the functionality). You read it right, the private gateway is sent as type=guest after reboot and type=public initially.

Before this change, you could add a private gw to a running router but you couldn't restart it (it would mix up the tiers). Now the private gateway is detected properly and it works just fine.

Booting without private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.0.42",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "None",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

Booting with private gateway:
```
root@r-167-VM:~# cat /etc/cloudstack/cmdline.json
{
    "config": {
        "baremetalnotificationapikey": "V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw",
        "baremetalnotificationsecuritykey": "OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w",
        "disable_rp_filter": "true",
        "dns1": "8.8.8.8",
        "domain": "cs2cloud",
        "eth0ip": "169.254.2.227",
        "eth0mask": "255.255.0.0",
        "host": "192.168.22.61",
        "name": "r-167-VM",
        "port": "8080",
        "privategateway": "10.201.10.1",
        "redundant_router": "false",
        "template": "domP",
        "type": "vpcrouter",
        "vpccidr": "10.0.0.0/24"
    },
    "id": "cmdline"
```

And:
```
cat cmdline
vpccidr=10.0.0.0/24 domain=cs2cloud dns1=8.8.8.8 privategateway=10.201.10.1 template=domP name=r-167-VM eth0ip=169.254.2.227 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=OXI16srCrxFBi-xOtEwcYqwLlMfSFTlTg66YHtXBBqR7HNN1us3HP5zWOKxfVmz4a3C1kUNLPrUH13gNmZlu4w baremetalnotificationapikey=V2l1u3wKJVan01h8kq63-5Y5Ia3VLEW1v_Z6i-31QIRJXlt5vkqaqf6DVcdK0jP3u79SW6X9pqJSLSwQP2c2Rw host=192.168.22.61 port=8080
```

Logs:
```
2016-02-24 20:08:45,723 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-458d4c52 job-1402/job-1403 ctx-d5355fca) (logid:5772906c) Set privategateway field in cmd_line.json to 10.201.10.1
```

* pr/1474:
  Handle private gateways more reliably
  Add private gateway IP to router initialization config

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:02:55 -04:00
Will Stevens
919660d093 Merge pull request #1472 from remibergsma/47_fix_static_router_master_change
Apply static routes on change to master stateRefactored static routes for private gateways so they also get loaded when the router switches to master state. Otherwise they're lost and connections drop after fail over.

* pr/1472:
  apply static routes on change to master state

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:01:55 -04:00
Will Stevens
309a60ea71 Merge pull request #1483 from remibergsma/pr1413-wilder-47
CLOUDSTACK-9287 - Fix unique mac address per rVPC routerThis is work by @wilderrodrigues, see PR #1413 It contains important fixes and I think it needs to be included so I send the PR again.

* pr/1483:
  CLOUDSTACK-9287 - Improve test by checking if pvt gw is removed and fix typos
  CLOUDSTACK-9287 - Fix RVR public interface
  CLOUDSTACK-9287 - Add integration test to cover the private gateway related changes
  CLOUDSTACK-9287 - Refactor the interface state configuration
  CLOUDSTACK-9287 - Check if the nic profile has already been removed from a certain router
  CLOUDSTACK-9287 - Bring up the private gw interface on state change to master
  CLOUDSTACK-9287 - Make sure private gw interface is not used for default gw
  CLOUDSTACK-9287 - Add integration test to cover the private gw interface/mac address issues
  CLOUDSTACK-9287 - Put private gateway interface down on backup router
  CLOUDSTACK-9287 - Generate new mac address if router is redundant and nic profile exists

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-12 11:00:40 -04:00
Paul Angus
11e3582fdf Remove extraneous log directory and add catalina.out log rotation 2016-05-12 08:41:39 +01:00
Rohit Yadav
d4cb05b8f9 utils: Disable NioTest
Disable the NioTest, it is far to dependent on the network configuration of the
machine running the test. Cherry-picked change from a historic commit where
this test was disabled.

(cherry picked from commit 881a6e164ece8e34154555b71e97c9f1763674f5)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:50:50 +05:30
Mike Tutkowski
1ad0da0390 CLOUDSTACK-9373: Removing a few instance methods where there are class methods that are overshadowing them 2016-05-11 23:57:09 -06:00
Rohit Yadav
12fff7d8bd maven: ignore utils/testsmallfileinactive for rat checking
This fixes several Jenkins failures as previous runs don't cleanup this
file created by one of the unit tests.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:16:12 +05:30
Rohit Yadav
6135f6d98f CLOUDSTACK-9378: Fix for #1497
Reorder cleanup items so cleanup won't fail

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav
e122bbfbfc HypervisorUtilsTest: increate timeout to 8seconds
Increases timeout to a larger value to avoid failures in VM environments such as
TravisCI.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav
4d5e8df2f9 travis: Use patched version of ipmitool for tests
- For out-of-band management feature (CLOUDSTACK-9299) use patched version of
  ipmitool that would work on trusty travis machines
- The ipmitool used is from xenial/16.04 release with patch from RedHat
  https://bugzilla.redhat.com/show_bug.cgi?id=1286035
- Installs ipmitool from xenial repositories to get all the dependencies
  and then install patched deb version
- Skip test if the known failure occurs

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Rohit Yadav
07564469e9 CLOUDSTACK-9299: Out-of-band Management for CloudStack
Support access to a host’s out-of-band management interface (e.g. IPMI, iLO,
DRAC, etc.) to manage host power operations (on/off etc.) and querying current
power state in CloudStack.

Given the wide range of out-of-band management interfaces such as iLO and iDRA,
the service implementation allows for development of separate drivers as plugins.
This feature comes with a ipmitool based driver that uses the
ipmitool (http://linux.die.net/man/1/ipmitool) to communicate with any
out-of-band management interface that support IPMI 2.0.

This feature allows following common use-cases:
- Restarting stalled/failed hosts
- Powering off under-utilised hosts
- Powering on hosts for provisioning or to increase capacity
- Allowing system administrators to see the current power state of the host

For testing this feature `ipmisim` can be used:
https://pypi.python.org/pypi/ipmisim

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-12 11:05:40 +05:30
Mike Tutkowski
dad9e5d868 CLOUDSTACK-8813: Notify listeners when a host has been added to a cluster, is about to be removed from a cluster, or has been removed from a cluster 2016-05-11 08:02:46 -06:00
Will Stevens
bee2bdc299 Merge pull request #1489 from shapeblue/dynamicroles-master
CLOUDSTACK-8562: Dynamic Role-Based API Checker for CloudStack### CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker

This feature allows root administrators to define new roles and associate API
permissions to them.

A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack

DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
  current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
  Admin, Domain Admin and User) which maintains this association by requiring
  all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
  removal of roles and/or modifications of permissions, without the need
  of restarting management server(s)

Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
  deployments will continue to use the older static role based api access checker
  with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
  roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
  to add existing set of permissions to the default roles. cloud.account
  will have a new role_id column which will be populated based on default roles
  as well

Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature

* pr/1489:
  maven: Fix jstl version usage
  CLOUDSTACK-8562: Deprecate commands.properties
  CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
  CLOUDSTACK-9361: Centrally handle API validations

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-11 02:03:50 -04:00
rahul singal
9df51faa8a Marvin script for cloudstack-9365 2016-05-11 07:59:38 +02:00
Nick Livens
71c9c90e0c CLOUDSTACK-9365 : updateVirtualMachine with userdata should not error when a VM is attached to multiple networks from which one or more doesn't support userdata 2016-05-11 07:59:38 +02:00
Will Stevens
dc5b52937b Merge release branch 4.8 to master
* 4.8:
  When no zone name is available display a default
2016-05-11 01:32:02 -04:00
Will Stevens
72811b4058 Merge release branch 4.7 to 4.8
* 4.7:
  When no zone name is available display a default
2016-05-11 01:31:21 -04:00
Will Stevens
5a79c2b6f5 Merge pull request #1477 from remibergsma/47_default_zone_name_s3
When no zone name is available display a defaultWhen a zone name is available, the previous behaviour is still there:
![screen shot 2016-03-30 at 21 00 44](https://cloud.githubusercontent.com/assets/1630096/14154026/ba41a4bc-f6ba-11e5-9f88-19cf36bfbd4f.png)

When there is no zone name, it used to display an empty name (where you had to click on to see details):
![screen shot 2016-03-30 at 21 03 06](https://cloud.githubusercontent.com/assets/1630096/14154048/d31a7b08-f6ba-11e5-9f67-f716e8d9fbf2.png)

With this change, a default name `All` is displayed (because this happens when S3 storage is used that is region wide aka all zones):
![screen shot 2016-03-30 at 20 53 50](https://cloud.githubusercontent.com/assets/1630096/14154060/e20d1d0a-f6ba-11e5-9b0a-1b5e502a2964.png)

Region wide S3:
![screen shot 2016-03-30 at 21 04 55](https://cloud.githubusercontent.com/assets/1630096/14154108/2222ff54-f6bb-11e5-845a-c22ddc745b98.png)

FYI: Screenshot shows 'All', later renamed to 'All Zones', see code.

* pr/1477:
  When no zone name is available display a default

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-11 01:30:44 -04:00
Rohit Yadav
003b97bcff maven: Fix jstl version usage
This would fix regression from recent mvn version changes. Without this
patch users get redirected to error.jsp as jstl-1.2 jar is not installed

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:58:27 +05:30
Rohit Yadav
afcbd270b2 CLOUDSTACK-8562: Deprecate commands.properties
- Removes commands.properties file
- Fixes apidocs and marvin to be independent of commands.properties usage
- Removes bundling of commands.properties in deb/rpm packaging
- Removes file references across codebase

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:45:19 +05:30
Rohit Yadav
4347776ac6 CLOUDSTACK-8562: DB-Backed Dynamic Role Based API Access Checker
This feature allows root administrators to define new roles and associate API
permissions to them.

A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.

FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack

DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
  current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
  Admin, Domain Admin and User) which maintains this association by requiring
  all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
  removal of roles and/or modifications of permissions, without the need
  of restarting management server(s)

Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
  deployments will continue to use the older static role based api access checker
  with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
  roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
  to add existing set of permissions to the default roles. cloud.account
  will have a new role_id column which will be populated based on default roles
  as well

Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:45:19 +05:30
Rohit Yadav
95abb6efc3 CLOUDSTACK-9361: Centrally handle API validations
Validate API arguments based on annotations. Introduces:
- NotNullOrEmpty: for doing null and empty string checks
- PositiveNumber: number > 0 (natural number)

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-05-11 09:40:22 +05:30
Will Stevens
456680dbb2 Merge pull request #880 from SudharmaJain/cs-8901
CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads The thread pool was hardcoded to use 8 threads,
com.cloud.template.TemplateManagerImpl.configure(String, Map<String, Object>):
_preloadExecutor = Executors.newFixedThreadPool(8, new NamedThreadFactory("Template-Preloader"));

Added the change to pick threadpool size from configuration.

* pr/880:
  CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-11 00:08:45 -04:00
Will Stevens
10bcf2955a Merge pull request #1529 from mike-tutkowski/marvin_replace_sleep
Marvin: Replace a timer.sleep(30) with pulling logichttps://issues.apache.org/jira/browse/CLOUDSTACK-9374

From the ticket:

In the base.py file, there is a Host class with a delete instance method.

This method first attempts to transition the host into the maintenance resource state.

The first step in this process is to transition the host into the prepare-for-maintenance resource state.

A while later, the host can be transitioned completely into the maintenance resource state.

In an attempt to wait for this transition to occur, the delete method has a timer.sleep(30) call.

The hope is that the host will have transitioned from the prepare-for-maintenance resource state to the maintenance resource state within 30 seconds, but this does not always happen.

We should correct this problem by putting in logic to query the management server for the resource state of the host. If it's in the expected state, move on; else, sleep for a bit and try again (up to a certain limit).

* pr/1529:
  Replace a timer.sleep(30) with pulling logic

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-11 00:03:59 -04:00
Will Stevens
ccf2265460 Merge pull request #1466 from nvazquez/graldboptimization
CLOUDSTACK-9340: General DB Optimization## Description
In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken and some deficiencies were discovered, we can group them in 4 groups:
* Incorrect PRIMARY key
* Duplicate PRIMARY KEY
* Missing indexes (Add indexes to avoid full table scans)
* Some views query (Change view to improve account retrieval speed)

* pr/1466:
  CLOUDSTACK-9340: General DB Optimization

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:57:39 -04:00
Will Stevens
726ee47e98 Merge pull request #1488 from wido/agent-ipv6-enable
agent: Enable IPv6 connectivity for KVM Agent to Management ServerIPv4 is still preferred, so if the hostname of the Management Server
returns a A and AAAA-record the Agent will still connect to the
server over IPv4.

This situation will however allow to use a hostname which only has
a AAAA-record. In that case the Agent will connect to the Management
Server over IPv6.

* pr/1488:
  agent: Enable IPv6 connectivity for KVM Agent to Management Server

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:55:28 -04:00
Will Stevens
d2a95b4c69 Merge pull request #1054 from wido/CLOUDSTACK-8818
CLOUDSTACK-8818: Use MySQL native connector with PythonMySQLdb has been deprecated and is also not supported in Python 3.

mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.

https://dev.mysql.com/doc/connector-python/en/

* pr/1054:
  CLOUDSTACK-8818: Use MySQL native connector with Python

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:54:23 -04:00
Will Stevens
bbfd69fbca Merge pull request #1428 from myENA/mysqlhapatch
Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fai

* pr/1428:
  Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fails to load successfully

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:51:59 -04:00
Will Stevens
baecaa8c8e Merge pull request #1440 from rodrigo93/Removed-Unused-Void-class
Removed Unused Void ClassRemoved the class _Void.java_ from _/cloud-framework-ipc/src/org/apache/cloudstack/framework/async/Void.java_ since the method _public void AsyncMethod(Object realParam,
AsyncCompletionCallback<Void> callback)_ does not exist and it does not seem to be used in another place.
Also, removed an unnecessary comment in
_/cloudstack-framework/ipc/test/org/apache/cloudstack/framework/codestyle/AsyncSampleListenerStyleCaller.java_

* pr/1440:
  Removed Unused Void Class

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:50:57 -04:00
Will Stevens
de582a41c0 Merge pull request #1497 from nvazquez/addidsparam
CLOUDSTACK-9351: Add ids parameter to resource listing API calls## General behaviour
A new parameter is added in each method, its type a list of IDs of the entity, it will be mutually exclusive with id. (Similar to <code>id</code> and <code>ids</code> parameters in <code>listVirtualMachines</code> method)

### API Methods affected
* <code>listTemplates</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVolumes</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listSnapshots</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVMSnapshots</code>: new parameter **<code>vmsnapshotids</code>**, mutually exclusive with <code>vmsnapshotid</code>

* pr/1497:
  CLOUDSTACK-9351: Add marvin test and add it to travis file
  CLOUDSTACK-9351: Add ids parameter to resource listing API calls

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:49:55 -04:00
Will Stevens
fa3bce5a83 Merge pull request #1496 from shapeblue/kvm-ha
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage- KVM-HA- Fix CheckOnHost for Local storage
 - Also skip HA on VMs that are using local storage

* pr/1496:
  CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage     - Also skip HA on VMs that are using local storage

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:49:04 -04:00
Will Stevens
8a3fd10615 Merge pull request #1124 from rafaelweingartner/CID-1338387
CID-1338387: Deletion of method endPointSelector.selectHypervisorHostFollowing the discussions and analysis presented on PR #1056 create by @DaanHoogland
This PR is intended to push those changes that were discussed there regarding the of endPointSelector.selectHypervisorHost method.

* pr/1124:
  Deletion of method endPointSelector.selectHypervisorHost

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:47:50 -04:00
Will Stevens
a368cb703f Merge release branch 4.8 to master
* 4.8:
  Update L10N resource files with 4.8 strings from Transifex (20160504) Force "translator" mode with the transifex client. Update Transifex client config file for 4.8 resources/L10N ref. (generated by Tx client)
2016-05-10 23:45:43 -04:00
Will Stevens
1c2400f4d8 Merge pull request #1515 from milamberspace/L10N-update-4.8-20160422
L10n update 4.8 20160422@swill the good PR for 4.8 branch.

* pr/1515:
  Update L10N resource files with 4.8 strings from Transifex (20160504) Force "translator" mode with the transifex client. Update Transifex client config file for 4.8 resources/L10N ref. (generated by Tx client)

Signed-off-by: Will Stevens <williamstevens@gmail.com>
2016-05-10 23:44:29 -04:00
nvazquez
b9d75c6ffa CLOUDSTACK-9351: Add marvin test and add it to travis file 2016-05-10 12:35:55 -07:00
Syed
7b5d5648d8 Fix create template from snapshot returning null in case of region store 2016-05-09 15:59:23 -04:00
dean.close
38b3bdd488 CLOUDSTACK-6975: Prevent dnsmasq from starting on backup redundant RvR. 2016-05-09 11:34:47 +01:00