- Removes commands.properties file
- Fixes apidocs and marvin to be independent of commands.properties usage
- Removes bundling of commands.properties in deb/rpm packaging
- Removes file references across codebase
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This feature allows root administrators to define new roles and associate API
permissions to them.
A limited form of role-based access control for the CloudStack management server
API is provided through a properties file, commands.properties, embedded in the
WAR distribution. Therefore, customizing API permissions requires unpacking the
distribution and modifying this file consistently on all servers. The old system
also does not permit the specification of additional roles.
FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
DB-Backed Dynamic Role Based API Access Checker for CloudStack brings following
changes, features and use-cases:
- Moves the API access definitions from commands.properties to the mgmt server DB
- Allows defining custom roles (such as a read-only ROOT admin) beyond the
current set of four (4) roles
- All roles will resolve to one of the four known roles types (Admin, Resource
Admin, Domain Admin and User) which maintains this association by requiring
all new defined roles to specify a role type.
- Allows changes to roles and API permissions per role at runtime including additions or
removal of roles and/or modifications of permissions, without the need
of restarting management server(s)
Upgrade/installation notes:
- The feature will be enabled by default for new installations, existing
deployments will continue to use the older static role based api access checker
with an option to enable this feature
- During fresh installation or upgrade, the upgrade paths will add four default
roles based on the four default role types
- For ease of migration, at the time of upgrade commands.properties will be used
to add existing set of permissions to the default roles. cloud.account
will have a new role_id column which will be populated based on default roles
as well
Dynamic-roles migration tool: scripts/util/migrate-dynamicroles.py
- Allows admins to migrate to the dynamic role based checker at a future date
- Performs a harder one-way migrate and update
- Migrates rules from existing commands.properties file into db and deprecates it
- Enables an internal hidden switch to enable dynamic role based checker feature
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Validate API arguments based on annotations. Introduces:
- NotNullOrEmpty: for doing null and empty string checks
- PositiveNumber: number > 0 (natural number)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads The thread pool was hardcoded to use 8 threads,
com.cloud.template.TemplateManagerImpl.configure(String, Map<String, Object>):
_preloadExecutor = Executors.newFixedThreadPool(8, new NamedThreadFactory("Template-Preloader"));
Added the change to pick threadpool size from configuration.
* pr/880:
CLOUDSTACK-8901: PrepareTemplate job thread hard-coded to max 8 threads
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Marvin: Replace a timer.sleep(30) with pulling logichttps://issues.apache.org/jira/browse/CLOUDSTACK-9374
From the ticket:
In the base.py file, there is a Host class with a delete instance method.
This method first attempts to transition the host into the maintenance resource state.
The first step in this process is to transition the host into the prepare-for-maintenance resource state.
A while later, the host can be transitioned completely into the maintenance resource state.
In an attempt to wait for this transition to occur, the delete method has a timer.sleep(30) call.
The hope is that the host will have transitioned from the prepare-for-maintenance resource state to the maintenance resource state within 30 seconds, but this does not always happen.
We should correct this problem by putting in logic to query the management server for the resource state of the host. If it's in the expected state, move on; else, sleep for a bit and try again (up to a certain limit).
* pr/1529:
Replace a timer.sleep(30) with pulling logic
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9340: General DB Optimization## Description
In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken and some deficiencies were discovered, we can group them in 4 groups:
* Incorrect PRIMARY key
* Duplicate PRIMARY KEY
* Missing indexes (Add indexes to avoid full table scans)
* Some views query (Change view to improve account retrieval speed)
* pr/1466:
CLOUDSTACK-9340: General DB Optimization
Signed-off-by: Will Stevens <williamstevens@gmail.com>
agent: Enable IPv6 connectivity for KVM Agent to Management ServerIPv4 is still preferred, so if the hostname of the Management Server
returns a A and AAAA-record the Agent will still connect to the
server over IPv4.
This situation will however allow to use a hostname which only has
a AAAA-record. In that case the Agent will connect to the Management
Server over IPv6.
* pr/1488:
agent: Enable IPv6 connectivity for KVM Agent to Management Server
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8818: Use MySQL native connector with PythonMySQLdb has been deprecated and is also not supported in Python 3.
mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.
https://dev.mysql.com/doc/connector-python/en/
* pr/1054:
CLOUDSTACK-8818: Use MySQL native connector with Python
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fai
* pr/1428:
Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fails to load successfully
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Removed Unused Void ClassRemoved the class _Void.java_ from _/cloud-framework-ipc/src/org/apache/cloudstack/framework/async/Void.java_ since the method _public void AsyncMethod(Object realParam,
AsyncCompletionCallback<Void> callback)_ does not exist and it does not seem to be used in another place.
Also, removed an unnecessary comment in
_/cloudstack-framework/ipc/test/org/apache/cloudstack/framework/codestyle/AsyncSampleListenerStyleCaller.java_
* pr/1440:
Removed Unused Void Class
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9351: Add ids parameter to resource listing API calls## General behaviour
A new parameter is added in each method, its type a list of IDs of the entity, it will be mutually exclusive with id. (Similar to <code>id</code> and <code>ids</code> parameters in <code>listVirtualMachines</code> method)
### API Methods affected
* <code>listTemplates</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVolumes</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listSnapshots</code>: new parameter **<code>ids</code>**, mutually exclusive with <code>id</code>
* <code>listVMSnapshots</code>: new parameter **<code>vmsnapshotids</code>**, mutually exclusive with <code>vmsnapshotid</code>
* pr/1497:
CLOUDSTACK-9351: Add marvin test and add it to travis file
CLOUDSTACK-9351: Add ids parameter to resource listing API calls
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage- KVM-HA- Fix CheckOnHost for Local storage
- Also skip HA on VMs that are using local storage
* pr/1496:
CLOUDSTACK-9350: KVM-HA- Fix CheckOnHost for Local storage - Also skip HA on VMs that are using local storage
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CID-1338387: Deletion of method endPointSelector.selectHypervisorHostFollowing the discussions and analysis presented on PR #1056 create by @DaanHoogland
This PR is intended to push those changes that were discussed there regarding the of endPointSelector.selectHypervisorHost method.
* pr/1124:
Deletion of method endPointSelector.selectHypervisorHost
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
Update L10N resource files with 4.8 strings from Transifex (20160504) Force "translator" mode with the transifex client. Update Transifex client config file for 4.8 resources/L10N ref. (generated by Tx client)
L10n update 4.8 20160422@swill the good PR for 4.8 branch.
* pr/1515:
Update L10N resource files with 4.8 strings from Transifex (20160504) Force "translator" mode with the transifex client. Update Transifex client config file for 4.8 resources/L10N ref. (generated by Tx client)
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9348: Optimize NioTest and NioConnection main loop- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef788ce0d51be06db0ef10f6815f9b563b
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
/cc @swill please review and merge.
Sorry about the previous values, they were not optimized for virtualized env. The aggressive selector.wakeup will ensure main IO loop does not get blocked even by malicious users, even for any timeout (ssl handshake etc).
* pr/1534:
CLOUDSTACK-9348: Optimize NioTest and NioConnection main loop
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- Reduces SSL handshake timeout to 15s, previously this was only 10s in
commit debfcdef788ce0d51be06db0ef10f6815f9b563b
- Adds an aggresive explicit wakeup to save the Nio main IO loop/handler from
getting blocked
- Fix NioTest to fail/succeed in about 60s, previously this was 300s
- Due to aggresive wakeup usage, NioTest should complete in less than 5s on most
systems. On virtualized environment this may slightly increase due to thread,
CPU burst/scheduling delays.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
MySQLdb has been deprecated and is also not supported in Python 3.
mysql.connector is a connector written in Python which talks the
native MySQL protocol without any external code.
https://dev.mysql.com/doc/connector-python/en/
Update L10N resource files with 4.7 strings from Transifex (20160502)Force "translator" mode with the transifex client.
cc @swill the new PR.
* pr/1527:
Update L10N resource files with 4.7 strings from Transifex (20160502) Force "translator" mode with the transifex client.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
kvm: Aqcuire lock when running security group Python scriptIt could happen that when multiple instances are starting at the same
time on a KVM host the Agent spawns multiple instances of security_group.py
which both try to modify iptables/ebtables rules.
This fails with on of the two processes failing.
The instance is still started, but it doesn't have any IP connectivity due
to the failed programming of the security groups.
This modification lets the script aqcuire a exclusive lock on a file so that
only one instance of the scripts talks to iptables/ebtables at once.
Other instances of the script which start will poll every 500ms if they can
obtain the lock and otherwise execute anyway after 15 seconds.
* pr/1408:
kvm: Aqcuire lock when running security group Python script
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9348: Use non-blocking SSL handshake in NioConnection/Link- Uses non-blocking socket config in NioClient and NioServer/NioConnection
- Scalable connectivity from agents and peer clustered-management server
- Removes blocking ssl handshake code with a non-blocking code
- Protects from denial-of-service issues that can degrade mgmt server responsiveness
due to an aggressive/malicious client
- Uses separate executor services for handling connect/accept events
Changes are covered the NioTest so I did not write a new test, advise how we can improve this. Further, I tried to invest time on writing a benchmark test to reproduce a degraded server but could not write it deterministic-ally (sometimes fails/passes but not always). Review, CI testing and feedback requested /cc @swill @jburwell @DaanHoogland @wido @remibergsma @rafaelweingartner @GabrielBrascher
* pr/1493:
CLOUDSTACK-9348: Use non-blocking SSL handshake
CLOUDSTACK-9348: Unit test to demonstrate denial of service attack
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9336 surround the execution of baremetal-vr.py with condition
* pr/1463:
CLOUDSTACK-9336 surround the execution of baremetal-vr.py with condition
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Quota: consolidated lockable account check to a method. Added unit tests to check lockablity of various accounts.
Currently normal user and domain admin accounts are eligible for locking.
* pr/1350:
Quota: consolidated lockable account check to a method. Added unit tests to check lockablity of various accounts
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8302: Removing snapshots on RBDSnapshot removing implemented if primary datastore is RBD
https://issues.apache.org/jira/browse/CLOUDSTACK-8302
* pr/1230:
CLOUDSTACK-8302 - Cleanup snapshot on KVM with RBD Snapshot removing implemented on RBD. 1. On management side: when created new shanpshot we checking if our primary storage is RBD, then do not remove record from cloud.snapshot_store_ref with link to Ceph image via 'install_path' field. 2. On management side: when removing snapshot, also send command to agent 'DeleteCommand'. 3. On agent side: method implemented 'public Answer deleteSnapshot(final DeleteCommand cmd)'
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabledWith DB HA enabled in db.properties, the cloudstack-usage service restarts every 10 seconds. Making the suggested change has fixed it for me. Cloudstack 4.8 on Centos7
* pr/1433:
Cloudstack Usage Breaks if DB HA enabled
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Set default networkDomain to empty instead of usernameThe 10th field of `createUserAccount` is `networkDomain` (See `AccountService.java`) and it is set to a var named `admin`, which is the user name.
So, the first user that is created in a domain that links to LDAP, creates the account within the domain, and sets the `networkDomain` field to the username. All next users are created in the same account.
Then we have the situation that in domain SBP we have a user `rbergsma` that logs in first, gets an account created and then (unless you override) all VMs started in the SBP domain will have network domain `rbergsma`. That is highly confusing and not what is should be.
The `linkDomainToLdap` api call has no `networkDomain` field, so I propose to make this field empty (set it to null). It's a sting and null / empty is allowed.
One can also specify the networkDomain when creating a VPC and also there it is allowed to be null.
When te networkDomain is needed (and is not set in the domain and not in the VPC) it is constructed by using `guest.domain.suffix` so there always is a networkDomain to be used.
It makes more sense to manually set it on a domain level, or specify it on the VPC and in the final case end up with something that is clearly generated (like cs342cloud.local) rather than the username of someone else.
* pr/1485:
Set default networkDomain to empty instead of username
Signed-off-by: Will Stevens <williamstevens@gmail.com>
- Uses non-blocking socket config in NioClient and NioServer/NioConnection
- Scalable connectivity from agents and peer clustered-management server
- Removes blocking ssl handshake code with a non-blocking code
- Protects from denial-of-service issues that can degrade mgmt server responsiveness
due to an aggressive/malicious client
- Uses separate executor services for handling ssl handshakes
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
The NioConnection uses blocking handlers for various events such as connect,
accept, read, write. In case a client connects NioServer (used by
agent mgr to service agents on port 8250) but fails to participate in SSL
handshake or just sits idle, this would block the main IO/selector loop in
NioConnection. Such a client could be either malicious or aggresive.
This unit test demonstrates such a malicious client that can perform a
denial-of-service attack on NioServer that blocks it to serve any other client.
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Bump ssh retries to prevent false positives of test_loadbalanceNo more false positives after this change.
```
[root@cs1 integration]# cat /tmp//MarvinLogs/test_loadbalance_TR5RJD/results.txt
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 3 tests in 930.418s
OK
```
* pr/1473:
bump ssh retries to prevent false positives of test_loadbalance
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8847: ListServiceOfferings is returning incompatible tagged offerings when called with VM idWhen calling listServiceOfferings with VM id as parameter. It is returning incompatible tagged offerings. It should only list all compatible tagged offerings. Compatible means the new service offering should contain all the tags of the existing service offering(Existing offering SUBSET of new offering). If that is the case It should list in the result and can be upgraded to that offering.
* pr/1321:
CLOUDSTACK-8847: ListServiceOfferings is returning incompatible tagged offerings when called with VM id
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Installing bzip2 since it is required for extracting templatesIf you do not install bzip2, then installing templates that are bzip2 compressed will result in Cloudstack not being able to extract the contents and ending up copying the template in compressed form. This will result in VMs not being able to start.
* pr/1490:
Installing bzip2 since it is required for extracting templates.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[4.7] vmware: improve support for disks- Improve disk chain usage while attaching, migrating disks
- Gets root disk controller based diskDeviceBusName from volume's chain info
* pr/1365:
vmware: improve support for disks
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9283: add pid to java arguments in cloudstack-usage.servicecloudstack-usage fails to start throwing Integer exception during PID retrieval, and the service keeps restarting after 10s (as defined in the systemd service definition).
Adding the pid to the java arguments in the systemd service definition makes it stop looping in centos7
* pr/1409:
CLOUDSTACK-9283: add pid to java arguments in systemd/cloudstack-usage.service
Signed-off-by: Will Stevens <williamstevens@gmail.com>
engine/schema: fix upgrade path to work with MySQL 5.7Found this issue when using MySQL 5.7 with Ubuntu 16.04. The upgrade path fix removes an invalid `IGNORE` param that is deprecated now, in the upgrade path we run the alter statement to add an index only if it does not exist so we're good.
For MySQL 5.7, we'll also need to update the docs at some point to include `server-id` along with other parameters. Some of the SQL statements used throughout engine/schema don't adhere to SQL 99 standard which is enforced by default in MySQL 5.7, therefore the following sql-mode (for backward compatibility with mysql 5.6 modes) will be necessary for anyone willing to use MySQL 5.7 (until we fix codebase wide raw and generated sql statements to be SQL99 compliant):
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
server-id = 1
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'
/cc @swill @jburwell @agneya2001 @wido @DaanHoogland and others
* pr/1517:
engine/schema: fix upgrade path to work with MySQL 5.7
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9323: Fix cancel host maintenance canFix cancel host maintenance so that if maintenance is cancelled the host come back to normal state gracefully.
Added marvin tests for host maintennace.
* pr/1454:
CLOUDSTACK-9323: Fix Cancel maintenance so that if maintenance is cancelled the host come back to normal state gracefully. Added marvin tests for host maintennace.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
Removed sleeps and used validateList as requested.
Added required_hardware="false" attr above test_02_root_volume_attach_detach
Modified test_volumes.py to include a hypervisor test for root attach/detach testing
Let hypervisor type KVM and Simulator detach root volumes. Updated test_volumes.py to include a test for detaching and reattaching a root volume from a vm. I also had to update base.py to allow attach_volume to have the parameter deviceid to be passed as needed.
