1075 Commits

Author SHA1 Message Date
Daan Hoogland
1d05fead49 Merge branch '4.11' 2018-06-21 13:08:55 +02:00
nvazquez
faf2a7760d Add unit tests 2018-06-12 11:56:41 -03:00
Rafael Weingärtner
9b83337658 Create unit test cases for 'ConfigDriveBuilder' class (#2674)
* Create unit test cases for 'ConfigDriveBuilder' class

* add method 'getProgramToGenerateIso' as suggested by rohit and Daan

* fix encoding for base64 to StandardCharsets.US_ASCII

* fix MockServerTest.testIsMockServerCanUpgradeConnectionToSsl()

This is another method that is causing Jenkins to fail for almost a month
2018-06-04 13:20:09 +02:00
Rohit Yadav
9c1eabfc28 Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 16:27:51 +05:30
Gabriel Beims Bräscher
02ece53375 addNicToVirtualMachine: Fixes #2540 handle invalid MAC address arg (#2653)
Look for the next available MAC address if the given MAC address in command addNicToVirtualMachine is invalid (null, empty, blank). Fixes #2540
2018-05-21 16:24:21 +05:30
Rohit Yadav
acc5fdcdbd
CLOUDSTACK-10290: allow config drives on primary storage for KVM (#2651)
This introduces a new global setting `vm.configdrive.primarypool.enabled` to toggle creation/hosting of config drive iso files on primary storage, the default will be false causing them to be hosted on secondary storage. The current support is limited from hypervisor resource side and in current implementation limited to `KVM` only. The next big change is that config drive is created at a temporary location by management server and shipped to either KVM or SSVM agent via cmd-answer pattern, the data of which is not logged in logs. This saves us from adding genisoimage dependency on cloudstack-agent pkg.

The APIs to reset ssh public key, password and user-data (via update VM API) requires that VM should be shutdown. Therefore, in the refactoring I removed the case of updation of existing ISO. If there are objections I'll re-put the strategy to detach+attach new config iso as a way of updation. In the refactored implementation, the folder name is changed to lower-cased configdrive. And during VM start, migration or shutdown/removal if primary storage is enable for use, the KVM agent will handle cleanup tasks otherwise SSVM agent will handle them.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-21 14:27:23 +05:30
Nicolas Vazquez
06f7e495dc Host Affinity plugin (#2630)
This implements a new host-affinity plugin.
2018-05-21 12:49:08 +05:30
Rafael Weingärtner
b9ed42bd29
Fix primary storage count when deleting volumes (#2629)
* Primary Storage count for an account does not decrease when a Data Disk is deleted

When a data disk is created and not attached in a running VM, the "deleteVolume" will not decrement the count for used primary storage in the VMs accounting information. The property that is not being decremented is called "primarystoragetotal"; this information can be retrieved via "listAccounts" API method.

Steps to reproduce this issue:
1 - Create an account, deploy a VM in it
2 - Check the primary storage count for the account with listAccounts API
3 - Create a data disk
4 - Check the primary storage count for the account with listAccounts API
5 - Delete the Data disk
6 - Check the primary storage count for the account with listAccounts API - It is the same as before deleting the data disk (it should not be the same as the value in step 2!)

* formatting and cleanups

* fix imports that were wrongly changed during rebase
2018-05-16 15:28:28 -03:00
Rohit Yadav
f663b926c7
config-drive: use hostname of VM instance of internal VM id (#2645)
This fixes config drive to use VM's user provided host-name instead of
the internal VM instance ID for hostname related config in both
cloudstack and openstack metadata bundled in the ISO.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-16 13:19:21 +05:30
Rohit Yadav
a77ed56b86
CLOUDSTACK-9114: Reduce VR downtime during network restart (#2508)
This introduces a rolling restart of VRs when networks are restarted
with cleanup option for isolated and VPC networks. A make redundant option is
shown for isolated networks now in UI.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-11 12:48:07 +05:30
Rohit Yadav
4534cefa40
backports for 4.11.1 from master (#2621)
* CLOUDSTACK-10147 Disabled Xenserver Cluster can still deploy VM's. Added code to skip disabled clusters when selecting a host (#2442)

(cherry picked from commit c3488a51db4bce4ec32c09e6fef78193d360cf3f)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10318: Bug on sorting ACL rules list in chrome (#2478)

(cherry picked from commit 4412563f19ec8b808fe4c79e2baf658507a84873)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10284:Creating a snapshot from VM Snapshot generates error if hypervisor is not KVM.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10221: Allow IPv6 when creating a Basic Network (#2397)

Since CloudStack 4.10 Basic Networking supports IPv6 and thus
should be allowed to be specified when creating a network.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
(cherry picked from commit 9733a10ecda5f1af0f2c0fa863fc976a3e710946)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10214: Unable to remove local primary storage (#2390)

Allow admins to remove primary storage pool.
Cherry-picked from eba2e1d8a1ce4e86b4df144db03e96739da455e5

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* dateutil: constistency of tzdate input and output (#2392)

Signed-off-by: Yoan Blanc <yoan.blanc@exoscale.ch>
Signed-off-by: Daan Hoogland <daan.hoogland@shapeblue.com>
(cherry picked from commit 2ad520282319da9a03061b8c744e51a4ffdf94a2)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* CLOUDSTACK-10054:Volume download times out in 3600 seconds (#2244)

(cherry picked from commit bb607d07a97476dc4fb934b3d75df6affba47086)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

* When creating a new account (via domain admin) it is possible to select “root admin” as the role for the new user (#2606)

* create account with domain admin showing 'root admin' role

Domain admins should not be able to assign the role of root admin to new users. Therefore, the role ‘root admin’ (or any other of the same type) should not be visible to domain admins.

* License and formatting

* Break long sentence into multiple lines

* Fix wording of method 'getCurrentAccount'

* fix typo in variable name

* [CLOUDSTACK-10259] Missing float part of secondary storage data in listAccounts

* [CLOUDSTACK-9338] ACS not accounting resources of VMs with custom service offering

ACS is accounting the resources properly when deploying VMs with custom service offerings. However, there are other methods (such as updateResourceCount) that do not execute the resource accounting properly, and these methods update the resource count for an account in the database. Therefore, if a user deploys VMs with custom service offerings, and later this user calls the “updateResourceCount” method, it (the method) will only account for VMs with normal service offerings, and update this as the number of resources used by the account. This will result in a smaller number of resources to be accounted for the given account than the real used value. The problem becomes worse because if the user starts to delete these VMs, it is possible to reach negative values of resources allocated (breaking all of the resource limiting for accounts). This is a very serious attack vector for public cloud providers!

* [CLOUDSTACK-10230] User should not be able to use removed “Guest OS type” (#2404)

* [CLOUDSTACK-10230] User is able to change to “Guest OS type” that has been removed

Users are able to change the OS type of VMs to “Guest OS type” that has been removed. This becomes a security issue when we try to force users to use HVM VMs (Meltdown/Spectre thing). A removed “guest os type” should not be usable by any users in the cloud.

* Remove trailing lines that are breaking build due to checkstyle compliance

* Remove unused imports

* fix classes that were in the wrong folder structure

* Updates to capacity management
2018-05-09 15:20:19 +05:30
Frank Maximus
85203248a4 ConfigDrive fixes: CLOUDSTACK-10288, CLOUDSTACK-10289 (#2566)
* CLOUDSTACK-10289: Config Drive Metadata: Use VM UUID instead of VM id

* CLOUDSTACK-10288: Config Drive Userdata: support for binary userdata

* CLOUDSTACK-10358: SSH keys are missing on Config Drive disk in some cases
2018-04-26 10:10:23 -03:00
Rafael Weingärtner
91d9821149 [CLOUDSTACK-10230] User should not be able to use removed “Guest OS type” (#2404)
* [CLOUDSTACK-10230] User is able to change to “Guest OS type” that has been removed

Users are able to change the OS type of VMs to “Guest OS type” that has been removed. This becomes a security issue when we try to force users to use HVM VMs (Meltdown/Spectre thing). A removed “guest os type” should not be usable by any users in the cloud.
2018-04-12 09:48:59 +02:00
Rohit Yadav
8ef131745a Merge branch '4.11' 2018-03-15 16:46:50 +05:30
Rohit Yadav
30175d6879
CLOUDSTACK-10132: Extend support for management servers LB for agents (#2469)
The new CA framework introduced basic support for comma-separated
list of management servers for agent, which makes an external LB
unnecessary.

This extends that feature to implement LB sorting algorithms that
sorts the management server list before they are sent to the agents.
This adds a central intelligence in the management server and adds
additional enhancements to Agent class to be algorithm aware and
have a background mechanism to check/fallback to preferred management
server (assumed as the first in the list). This is support for any
indirect agent such as the KVM, CPVM and SSVM agent, and would
provide support for management server host migration during upgrade
(when instead of in-place, new hosts are used to setup new mgmt server).

This FR introduces two new global settings:

- `indirect.agent.lb.algorithm`: The algorithm for the indirect agent LB.
- `indirect.agent.lb.check.interval`: The preferred host check interval
  for the agent's background task that checks and switches to agent's
  preferred host.

The indirect.agent.lb.algorithm supports following algorithm options:

- static: use the list as provided.
- roundrobin: evenly spreads hosts across management servers based on
  host's id.
- shuffle: (pseudo) randomly sorts the list (not recommended for production).

Any changes to the global settings - `indirect.agent.lb.algorithm` and
`host` does not require restarting of the mangement server(s) and the
agents. A message bus based system dynamically reacts to change in these
global settings and propagates them to all connected agents.

Comma-separated management server list is propagated to agents on
following cases:
- Addition of a host (including ssvm, cpvm systevms).
- Connection or reconnection by the agents to a management server.
- After admin changes the 'host' and/or the
  'indirect.agent.lb.algorithm' global settings.

On the agent side, the 'host' setting is saved in its properties file as:
`host=<comma separated addresses>@<algorithm name>`.

First the agent connects to the management server and sends its current
management server list, which is compared by the management server and
in case of failure a new/update list is sent for the agent to persist.

From the agent's perspective, the first address in the propagated list
will be considered the preferred host. A new background task can be
activated by configuring the `indirect.agent.lb.check.interval` which is
a cluster level global setting from CloudStack and admins can also
override this by configuring the 'host.lb.check.interval' in the
`agent.properties` file.

Every time agent gets a ms-host list and the algorithm, the host specific
background check interval is also sent and it dynamically reconfigures
the background task without need to restart agents.

Note: The 'static' and 'roundrobin' algorithms, strictly checks for the
order as expected by them, however, the 'shuffle' algorithm just checks
for content and not the order of the comma separate ms host addresses.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-03-15 16:34:03 +05:30
Nicolas Vazquez
74db647dbb CLOUDSTACK-10321: CPU Cap for KVM (#2482) 2018-03-14 18:21:24 +00:00
Marc-Aurèle Brothier
893a88d225 CLOUDSTACK-10105: Use maven standard project structure in all projects (#2283)
Remove maven standard module (which only a few were using) and get ride of maven customization for the projects structure.

- moved all directories to src/main/java, src/main/resources, src/main/scripts, src/test/java, src/test/resources
- grep scan to search for src/com and src/org left over
- grep for <project>/scripts to fix pom.xml configuration
- remove custom <build> configuration in pom.xml

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2018-01-20 03:19:27 +05:30
Mike Tutkowski
a30a31c9b7 CLOUDSTACK-9620: Enhancements for managed storage (#2298)
Allowed zone-wide primary storage based on a custom plug-in to be added via the GUI in a KVM-only environment (previously this only worked for XenServer and VMware)

Added support for root disks on managed storage with KVM

Added support for volume snapshots with managed storage on KVM

Enable creating a template directly from a volume (i.e. without having to go through a volume snapshot) on KVM with managed storage

Only allow the resizing of a volume for managed storage on KVM if the volume in question is either not attached to a VM or is attached to a VM in the Stopped state.

Included support for Reinstall VM on KVM with managed storage

Enabled offline migration on KVM from non-managed storage to managed storage and vice versa

Included support for online storage migration on KVM with managed storage (NFS and Ceph to managed storage)

Added support to download (extract) a managed-storage volume to a QCOW2 file

When uploading a file from outside of CloudStack to CloudStack, set the min and max IOPS, if applicable.

Included support for the KVM auto-convergence feature

The compression flag was actually added in version 1.0.3 (1000003) as opposed to version 1.3.0 (1003000) (changed this to reflect the correct version)

On KVM when using iSCSI-based managed storage, if the user shuts a VM down from the guest OS (as opposed to doing so from CloudStack), we need to pass to the KVM agent a list of applicable iSCSI volumes that need to be disconnected.

Added a new Global Setting: kvm.storage.live.migration.wait

For XenServer, added a check to enforce that only volumes from zone-wide managed storage can be storage motioned from a host in one cluster to a host in another cluster (cannot do so at the time being with volumes from cluster-scoped managed storage)

Don’t allow Storage XenMotion on a VM that has any managed-storage volume with one or more snapshots.

Enabled for managed storage with VMware: Template caching, create snapshot, delete snapshot, create volume from snapshot, and create template from snapshot

Added an SIOC API plug-in to support VMware SIOC

When starting a VM that uses managed storage in a cluster other than the one it last was running in, we need to remove the reference to the iSCSI volume from the original cluster.

Added the ability to revert a volume to a snapshot

Enabled cluster-scoped managed storage

Added support for VMware dynamic discovery
2018-01-15 00:05:52 +05:30
Frank Maximus
b176648f90 CLOUDSTACK-9813: Extending Config Drive support (#2097)
Extending Config Drive support

* Added support for VMware
* Build configdrive.iso on ssvm
* Added support for VPC and Isolated Networks
* Moved implementation to new Service Provider
* UI fix: add support for urlencoded userdata
* Add support for building systemvm behind a proxy

Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>
2018-01-12 15:14:40 +05:30
Abhinandan Prateek
64832fd70a CLOUDSTACK-4757: Support OVA files with multiple disks for templates (#2146)
CloudStack volumes and templates are one single virtual disk in case of XenServer/XCP and KVM hypervisors since the files used for templates and volumes are virtual disks (VHD, QCOW2). However, VMware volumes and templates are in OVA format, which are archives that can contain a complete VM including multiple VMDKs and other files such as ISOs. And currently, Cloudstack only supports Template creation based on OVA files containing a single disk. If a user creates a template from a OVA file containing more than 1 disk and launches an instance using this template, only the first disk is attached to the new instance and other disks are ignored.
Similarly with uploaded volumes, attaching an uploaded volume that contains multiple disks to a VM will result in only one VMDK to being attached to the VM.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Support+OVA+files+containing+multiple+disks

This behavior needs to be improved in VMWare to support OVA files with multiple disks for both uploaded volumes and templates. i.e. If a user creates a template from a OVA file containing more than 1 disk and launches an instance using this template, the first disk should be attached to the new instance as the ROOT disk and volumes should be created based on other VMDK disks in the OVA file and should be attached to the instance.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-10 22:10:41 +05:30
Wido den Hollander
35b4339946 CLOUDSTACK-10212: Update Netmask/Gateway when Changing IPv4 address (#2388)
This can otherwise cause problems in Basic Networking where multiple
IPv4 ranges are configured in a POD.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-01-09 13:53:16 +05:30
Nicolas Vazquez
e86bb41e0e CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates (#2379)
This feature allows using templates and ISOs avoiding secondary storage as intermediate cache on KVM. The virtual machine deployment process is enhanced to supported bypassed registered templates and ISOs, delegating the work of downloading them to primary storage to the KVM agent instead of the SSVM agent.

Template and ISO registration:
- When hypervisor is KVM, a checkbox is displayed with 'Direct Download' label.
- API methods registerTemplate and registerISO are both extended with this new parameter directdownload.
- On template or ISO registration, no download job is sent to SSVM agent, CloudStack would only persist an entry on template_store_ref indicating that template or ISO has been marked as 'Direct Download' (bypassing Secondary Storage). These entries are persisted as:
template_id = Template or ISO id on vm_template table
store_id NULL
download_state = BYPASSED
state = Ready
(Note: these entries allow users to deploy virtual machine from registered templates or ISOs)
- An URL validation command is sent to a random KVM host to check if template/ISO location can be reached. Metalink are also supported by this feature. In case of a metalink, it is fetched and URL check is performed on each of its URLs.
- Checksum should be provided as indicated on #2246: {ALGORITHM}CHKSUMHASH
- After template or ISO is registered, it would be displayed in the UI

Virtual machine deployment:
When a 'Direct Download' template is selected for deployment, CloudStack would delegate template downloading to destination storage pool via destination host by a new pluggable download manager.
Download manager would handle template downloading depending on URL protocol. In case of HTTP, request headers can be set by the user via vm_template_details. Those details should be persisted as:
Key: HTTP_HEADER
Value: HEADERNAME:HEADERVALUE

In case of HTTPS, a new API method is added uploadTemplateDirectDownloadCertificate to allow user importing a client certificate into all KVM hosts' keystore before deployment.
After template or ISO is downloaded to primary storage, usual entry would be persisted on template_spool_ref indicating the mapping between template/ISO and storage pool.
2018-01-09 12:22:18 +05:30
Nicolas Vazquez
90ef67bab9 CLOUDSTACK-10109: Enable dedication of public IPs to SSVM and CPVM (#2295)
This feature allow admins to dedicate a range of public IP addresses to the SSVM and CPVM, such that they can be subject to specific external firewall rules. The option to dedicate a public IP range to the System VMs (SSVM & CPVM) is added to the createVlanIpRange API method and the UI.

Solution:
Global setting 'system.vm.public.ip.reservation.mode.strictness' is added to determine if the use of the system VM reservation is strict (when true) or preferred (false), false by default.
When a range has been dedicated to System VMs, CloudStack should apply IPs from that range to
the public interfaces of the CPVM and the SSVM depending on global setting's value:

If the global setting is set to false: then CloudStack will use any unused and unreserved public IP
addresses for system VMs only when the pool of reserved IPs has been exhausted
If the global setting is set to true: then CloudStack will fail to deploy the system VM when the pool
of reserved IPs has been exhausted, citing the lack of available IPs.
UI Changes
Under Infrastructure -> Zone -> Physical Network -> Public -> IP Ranges, button 'Account' label is refactored to 'Set reservation'.

When that button is clicked, dialog displayed is also refactored, including a new checkbox 'System VMs' which indicates if range should be dedicated for CPVM and SSVM, and a note indicating its usage.

When clicking on button for any created range, UI dialog displayed indicates whether IP range is dedicated for system vms or not.
2018-01-06 23:14:30 +05:30
dahn
45df928e04 CLOUDSTACK-10117: Account ldap binding (#2381)
Map an ldap group to an account. Ldap related settings on a domain level.
2018-01-06 22:58:28 +05:30
PranaliM
5fad84174a CLOUDSTACK-10104: Optimize database transactions in ListDomain API to improve performance (#2282)
While creating the response object for the 'listDomain' API, several database calls are triggered to fetch details like parent domain, project limit, IP limit, etc. These database calls are triggered for each record found in the main fetch query, which is causing the response to slow down.

Fix:
The database transactions are reduced to improve response of the Listdomain API
2018-01-05 21:51:46 +05:30
Nicolas Vazquez
bf4f1bbb90 CLOUDSTACK-10126: Separate Subnet for SSVM and CPVM (#2368)
This extends work presented on #2048 on which the ability to extend the management range is provided.

Aim
This PR allows separating the management network subnet on which SSVM and CPVM are from the virtual routers management subnet.

Detailed use case
PCI compliance requires that network elements are defined as ‘in scope’ or ‘out of scope’, for compliance purposes. The SSVM and CPVM are both in scope as they allow public HTTP or HTTPS connections. The virtual routers have been defined as out of scope as they have been placed entirely in a firewalled network's segment. However, all of the system VM types share management network. As SSVM and CPVM are both in scope this would bring the virtual routers into scope as well, requiring individual audits of every virtual router. As this is not practical, the ‘management network’ which the SSVM and CPVM are on, and the management network which the virtual routers are on, must be separated by a firewall.

Description
By this feature it is possible to dedicate a created range for SSVM and CPVM (system vms) and provide a VLAN ID for its range.

A new boolean global configuration is added: system.vm.management.ip.reservation.mode.strictness. If enabled, the use of System VMs management IP reservation is strict, preferred if not. Default value is false (preferred).

Strict reservation: System VMs should try to get a private IP from a range marked for system vms. If not available, deployment fails
Preferred reservation: System VMS will try to get a private IP from a range marked for system vms. If not available, IP for range not marked for system vms is taken.
2018-01-05 13:19:03 +05:30
PranaliM
7253969088 CLOUDSTACK-10065: Optimize SQL queries in listTemplate API to improve performance (#2260)
The db queries in listTemplateAPI could be optimized to get unique results from the database which could help in reducing the listTemplate API response time.
2018-01-05 11:27:41 +05:30
mrunalinikankariya
ebb7a5279d CLOUDSTACK-10108: ConfigKey based approach for reading 'ping' configuaration (#2292)
In CLOUDSTACK-9886, we are reading ping.interval and ping.timeout using configdao which involves direct reading of DB. So, replaced it with ConfigKey based approach.
2018-01-05 11:23:42 +05:30
subhash yedugundla
8eca04e1f6 CLOUDSTACK-9572: Snapshot on primary storage not cleaned up after Storage migration (#1740)
Snapshot on primary storage not cleaned up after Storage migration. This happens in the following scenario:

Steps To Reproduce
Create an instance on the local storage on any host
Create a scheduled snapshot of the volume:
Wait until ACS created the snapshot. ACS is creating a snapshot on local storage and is transferring this snapshot to secondary storage. But the latest snapshot on local storage will stay there. This is as expected.
Migrate the instance to another XenServer host with ACS UI and Storage Live Migration
The Snapshot on the old host on local storage will not be cleaned up and is staying on local storage. So local storage will fill up with unneeded snapshots.
2018-01-05 11:19:56 +05:30
ernjvr
d7e334b1b4 CLOUDSTACK-9599: isdynamicallyscalable field missing in updateTemplate Response (#2383)
Using cloudmonkey, when invoking the update template api call, it does not display the isdynamicallyscalable field as part of its template response.
fix done:
org.apache.cloudstack.api.response.TemplateResponse isdynamicallyscalable field is now populated in the server/src/com/cloud/api/query/dao/TemplateJoinDaoImpl.java.newUpdateResponse method.
Unit test:
the Unit test server/test/com/cloud/api/query/dao/TemplateJoinDaoImplTest.java testNewUpdateResponse() verifies that the TemplateResponse is populated correctly.
Marvin test:
the Marvin nosetest integration/smoke/test_templates.py test_02_edit_template(self) confirms that the template_response.isdynamicallyscalable field gets populated with the correct user data.
Test scenario:
Using cloudmonkey, when invoking the 'update template' API call, it should now display the isdynamicallyscalable field as part of its template response.
2018-01-04 11:03:05 +05:30
Mowgli
25b63f5e73 CLOUDSTACK-9607: Preventing template deletion when template is in use (#1773)
Consider this scenario:
1. User launches a VM from Template and keep it running
2. Admin logins and deleted that template [CloudPlatform does not check existing / running VM etc. while the deletion is done]
3. User resets the VM
4. CloudPlatform fails to star the VM as it cannot find the corresponding template.

It throws error as 
java.lang.RuntimeException: Job failed due to exception Resource [Host:11] is unreachable: Host 11: Unable to start instance due to can't find ready template: 209 for data center 1
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:113)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)

Client is requesting better handing of this scenario. We need to check existing / running VM's when the template is deleted and warn admin about the possible issue that may occur.

REPRO STEPS
==================
1. Launches a VM from Template and keep it running
2. Now delete that template 
3. Reset the VM
4. CloudPlatform fails to star the VM as it cannot find the corresponding template.

EXPECTED BEHAVIOR
==================
Cloud platform should throw some warning message while the template is deleted if that template is being used by existing / running VM's

ACTUAL BEHAVIOR
==================
Cloud platform does not throw as waring etc.
2018-01-04 10:59:39 +05:30
Wido den Hollander
9988c269b2 CLOUDSTACK-7958: Add configuration for limit to CIDRs for Admin API calls (#2046)
* Cleanup and Improve NetUtils

This class had many unused methods, inconsistent names and redundant code.

This commit cleans up code, renames a few methods and constants.

The global/account setting 'api.allowed.source.cidr.list' is set
to 0.0.0.0/0,::/0 by default preserve the current behavior and thus
allow API calls for accounts from all IPv4 and IPv6 subnets.

Users can set it to a comma-separated list of IPv4/IPv6 subnets to
restrict API calls for Admin accounts to certain parts of their network(s).

This is to improve Security. Should an attacker steal the Access/Secret key
of an account he/she still needs to be in a subnet from where accounts are
allowed to perform API calls.

This is a good security measure for APIs which are connected to the public internet.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-01-04 10:56:30 +05:30
dahn
6724a47122
CLOUDSTACK-10121 moveUser (#2301)
* internal service call for moveUser
* expose moveUser as API
* move uuid to external entity
2017-12-30 11:40:14 +00:00
Marc-Aurèle Brothier
e27b3e120d CLOUDSTACK-9602: Improve resource limits comprehension in listResources (#1554)
Add resource type name in request and response for listResources API call.
This adds in the response a new attribute typename with the String value for the corresponding resource enum.

    {
      "capacitytotal": 0,
      "capacityused": 0,
      "percentused": "0",
      "type": 19,
      "typename": "gpu",
      "zoneid": "381d0a95-ed4a-4ad9-b41c-b97073c1a433",
      "zonename": "ch-dk-2"
    }

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
2017-12-28 15:46:24 +05:30
Marc-Aurèle Brothier
a26a502345 CLOUDSTACK-9593: userdata: enforce data is a multiple of 4 characters (#1760)
Python base64 requires that the string is a multiple of 4 characters but
the Apache codec does not. RFC states is not mandatory so the data should
not fail the VR script (vmdata.py).

Signed-off-by: Marc-Aurèle Brothier <m@brothier.org>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-28 15:27:33 +05:30
Sigert Goeminne
26759d1d13 CLOUDSTACK-10189: Adding nuage VSD managed network support to CloudStack (#2360)
Exposing externalId en domainId field in the UI to CS users.

Co-Authored-By: Sigert Goeminne sigert.goeminne@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net
2017-12-28 14:55:15 +05:30
Sigert Goeminne
d49765619d CLOUDSTACK-10024: Network migration support
Co-Authored-By: Frank Maximus frank.maximus@nuagenetworks.net
Co-Authored-By: Raf Smeets raf.smeets@nuagenetworks.net

New API’s:

* migrateNetwork
* migrateVpc
2017-12-21 11:25:17 +01:00
Bitworks Software, Ltd
80a6961b5d CLOUDSTACK-10168: VR duplicate entries in /etc/hosts when reusing VM name (#2366)
Bug is fixed. Fixed typo 'adress' to 'address' everywhere which leads to correct generation for agent communication json dhcp command.
2017-12-20 23:19:12 +05:30
Nitin Kumar Maharana
8acb0908c4 CLOUDSTACK-9880: Expansion of Management IP Range. (#2048)
At present, The management IP range can only be expanded under the same subnet. According to existing range, either the last IP can be forward extended or the first IP can be backward extended. But we cannot add an entirely different range from the same subnet. So the expansion of range is subnet bound, which is fixed. But when the range gets exhausted and a user wants to deploy more system VMs, then the operation would fail. The purpose of this feature is to expand the range of management network IPs within the existing subnet. It can also delete and list the IP ranges.

Please refer the FS here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Expansion+of+Management+IP+Range
2017-12-20 08:36:53 +00:00
Gabriel Beims Bräscher
bd5604428b
Merge pull request #2124 from GabrielBrascher/CLOUDSTACK-9432
CLOUDSTACK-9432: cluster/host dedicated to a domain is owned by the root domain
2017-11-28 10:31:23 -02:00
Sigert Goeminne
77864992fe CLOUDSTACK-9776: extra DHCP options support for Nuage VSP
Co-Authored-By: Frank Maximus <frank.maximus@nuagenetworks.net>
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9776

Design-Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+extra+DHCP+option+support
2017-11-21 11:44:39 +01:00
Frank Maximus
14ab69e19a
Merge pull request #2005 from nuagenetworks/feature/nw_offering_for_vpc
CLOUDSTACK-9450: Network Offering for VPC based on DB flag
2017-11-16 17:33:07 +01:00
Rohit Yadav
eda3b35bfa CLOUDSTACK-10012: Migrate to Embedded Jetty
- Migrate to embedded Jetty server.
- Improve ServerDaemon implementation.
- Introduce a new server.properties file for easier configuration.
- Have a single /etc/default/cloudstack-management to configure env.
- Reduce shaded jar file, removing unnecessary dependencies.
- Upgrade to Spring 5.x, upgrade several jar dependencies.
- Does not shade and include mysql-connector, used from classpath instead.
- Upgrade and use bountcastle as a separate un-shaded jar dependency.
- Remove tomcat related configuration and files.
- Have both embedded UI assets in uber jar and separate webapp directory.
- Refactor systemd and init scripts, cleanup packaging.
- Made cloudstack-setup-databases faster, using `urandom`.
- Remove unmaintained distro packagings.
- Moves creation and usage of server keystore in CA manager, this
  deprecates the need to create/store cloud.jks in conf folder and
  the db.cloud.keyStorePassphrase in db.properties file. This also
  remove the need of the --keystore-passphrase in the
  cloudstack-setup-encryption script.
- GZip contents dynamically in embedded Jetty

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-11-03 23:57:25 +05:30
Frank Maximus
d467a6b8ec CLOUDSTACK-9450: Network Offering for VPC based on DB flag
Co-Authored-By: Raf Smeets <raf.smeets@nuagenetworks.net>
2017-11-03 12:39:23 +01:00
Frank Maximus
d077b3efc6
Merge pull request #2004 from nuagenetworks/feature/vr_without_public_ip
CLOUDSTACK-9832: Do not assign public IP NIC to the VPC VR when the VPC offering does not contain VpcVirtualRouter as a SourceNat provider
2017-11-02 11:56:05 +01:00
Sigert Goeminne
587b66db71 CLOUDSTACK-10057: listNetworkOfferings now returns the correct number of offerings (#2250)
Fix paginated response to send correct total resource counts.
2017-10-31 15:05:29 +05:30
Rohit Yadav
41fdb88970 CLOUDSTACK-10047: DVSwitch fixes and improvements (#2293)
Allow security policies to apply on port groups:
- Accepts security policies while creating network offering
- Deployed network will have security policies from the network offering
  applied on the port group (in vmware environment)
- Global settings as fallback when security policies are not defined for a network
  offering
- Default promiscuous mode security policy set to REJECT as it's the default
  for standard/default vswitch

Portgroup vlan-trunking options for dvswitch: This allows admins to define
a network with comma separated vlan id and vlan
range such as vlan://200-400,21,30-50 and use the provided vlan range to
configure vlan-trunking for a portgroup in dvswitch based environment.

VLAN overlap checks are performed for:
- isolated network against existing shared and isolated networks
- dedicated vlan ranges for the physical/public network for the zone
- shared network against existing isolated network

Allow shared networks to bypass vlan overlap checks: This allows admins
to create shared networks with a `bypassvlanoverlapcheck` API flag
which when set to 'true' will create a shared network without
performing vlan overlap checks against isolated network and against
the vlans allocated to the datacenter's physical network (vlan ranges).

Notes:
- No vlan-range overlap checks are performed when creating shared networks
- Multiple vlan id/ranges should include the vlan:// scheme prefix

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-25 15:14:42 +05:30
Mike Tutkowski
4c89b5b97a Merge branch 'pr-2081' 2017-10-18 14:25:45 -06:00
Rohit Yadav
c3ed1b38e5 CLOUDSTACK-9993: Have basic constraint in CA certificate (#2286)
- Refactors V3 x509 cert generator to put basic constraint and key usage
  extensions when CA cert is created
- Refactors root CA provider to use V3 generator to generate CA cert

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-10-11 17:41:37 +05:30
Frank Maximus
1d382e0cb4 CLOUDSTACK-9832: Remove public interface from VPC Virtual Router
Co-Authored-By: Prashanth Manthena <prashanth.manthena@nuagenetworks.net>
Co-Authored-By: Sigert Goeminne <sigert.goeminne@nuagenetworks.net>

Bug: https://issues.apache.org/jira/browse/CLOUDSTACK-9832

Detail:
When the VPC offering does not contain VpcVirtualRouter as a SourceNat provider,
then we will not add the interface in the public network to the VpcVR.

CLOUDSTACK-9832: Move isSrcNat check to VpcManager
2017-10-11 11:35:53 +02:00