965 Commits

Author SHA1 Message Date
Wido den Hollander
ac73e7e671 kvm: Security Group enhancements and refactor old code (#3113)
* security group: Replace deprecated optparse by argparse

Starting with Python 2.7 the library optparse has been replaced by
argpase.

This commit replaces the use of optparse by argparse

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Remove LXC support from security_group.py

LXC does not work and has been partially removed from CloudStack already

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Refactor libvirt code

Use a single function which properly throws an Exception when the
connection to libvirt fails.

Also simplify some logic, make it PEP-8 compatible and remove a unused
function from the code.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Raise Exception on execute() failure

If the executed command exists with a non-zero exit status we should
still return the output to the command, but also raise an Exception.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: Use a function to determin the physical device of a bridge

We can not safely assume that the first device listed under a bridge is the
physical device.

With VXLAN isolation a vnet device can be attached to a bridge prior to the
vxlanXXXX device being attached.

We need to filter out those devices and then fetch the physical device attached
to the bridge.

In addition use the 'bridge' command instead of 'brctl'. 'bridge' is part of the
iproute2 utils just like 'ip' and should be considered as the new default.

This command is also available on EL6 and does not break any backwards compat.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: --set is deprecated, use --match-set

These messages are seen in the KVM Agent log:

  --set option deprecated, please use --match-set

Functionality does not change

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* security group: PEP-8 and indentation fixes

There were a lot of styling problems in the code:

- Missing whitespace or exess whitespace
- CaMelCaSe function names and variables
- 2-space indentation instead of 4 spaces

This commit addresses those issues.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-16 13:35:18 -02:00
Wido den Hollander
d3e95b98fc kvm: Refactory VXLAN script and add IPv6 support (#3070)
* vxlan: Code indentation and styling fixes

This script was using TAB instead of 4 spaces and had many blank
lines containing whitespace.

This commit also fixes some Bash styling, but it does not touch the
functionality of the script.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* vxlan: Improve Bash if-statement logic

Bash suggest using double brackets instead of single brackets in
if-statement test logic

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* vxlan: Disable IPv6 on bridge and VXLAN devices

They are only transport devices and should not be interacting
in the IPv6 traffic.

If IPv6 is enabled Instances can connect to the Hypervisor over
Link-Local IPv6 which is a potential security issue.

By disabling IPv6 on the Bridge and VXLAN device they still forward
Layer 2 packets as intended, but they do not respond on anything.

IPv4 and IPv6 traffic towards the Instances is untouched and works
as before.

Signed-off-by: Wido den Hollander <wido@widodh.nl>

* vxlan: Refactor modifyvxlan.sh for KVM by using only iproute2

This commit refactors the modifyvxlan.sh script by using only iproute2,
the 'ip' command for all functions.

brctl is deprecated and most bridge functionality can be performed with
the 'ip' command.

This commit also fixes various Bash coding fixes and removes a lot of exit
status checking which was redundant.

In addition it add IPv6 underlay for VXLAN transport. If the caller (KVM Agent)
adds the '-6' flag it will generate IPv6 multicast groups and routes which will
transport the VXLAN encapsulated packaes over IPv6 multicast groups.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2019-01-09 13:21:07 +01:00
Rohit Yadav
ac9562a4a1 Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-11-29 15:06:06 +05:30
Rene Diepstraten
d425a409fc sg: add secondary ips to the correct ipset based on ip family (#2990)
Currently secondary ipv6 addresses are added to the ipv4 ipset in security_group.py.
This doesn't work, so this patch adds a function to split a set of ips in ipv4 and ipv6 addresses.
Both the default_network_rules and network_rules_vmSecondaryIp functions now utilise this function and add the ips to the appropriate ipsets.
2018-11-28 19:30:13 +05:30
Rohit Yadav
7479e2877f Merge remote-tracking branch 'origin/4.11' 2018-10-30 15:15:35 +05:30
Rohit Yadav
c6e53f6cc6
kvm: reset KVM host on heartbeat failure (#2984)
On actual testing, I could see that kvmheartbeat.sh script fails on NFS
server failure and stops the agent only. Any HA VMs could be launched
in different hosts, and recovery of NFS server could lead to a state
where a HA enabled VM runs on two hosts and can potentially cause
disk corruptions. In most cases, VM disk corruption will be worse than
VM downtime. I've kept the sleep interval between check/rounds but
reduced it to 10s. The change in behaviour was introduced in #2722.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-30 15:13:59 +05:30
Rohit Yadav
323d381767 Merge remote-tracking branch 'origin/4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-10-29 16:27:08 +05:30
alexanderbazhenoff
a87acf93d8 kvm: improved performance on creating VM (#2923)
Improved performance on creating VM for KVM virtualization.

On a huge hosts every "ifconfig | grep" takes a lot of time (about 2.5-3 minutes on hosts with 500 machines). For example: ip link show dev $vlanDev > /dev/null is faster than ifconfig |grep -w $vlanDev > /dev/null. But using ip command is much better. Using this patch you can create 500s machine in 10 seconds. You don't need slow ifconfig prints anymore.
2018-10-25 16:28:13 +05:30
Rohit Yadav
5313c04b21 Merge remote-tracking branch 'origin/4.11' 2018-10-05 18:36:37 +05:30
Simon Weller
5db65a6363 kvm: Fixes #2868 libvirt resize notify failure (#2878)
Incorrect diskpath information was being sent to virsh blockresize, so the block device size was never refreshed to reflect the new disk size.
Fixes #2868
2018-10-05 18:35:09 +05:30
Khosrow Moossavi
769a0085be tools: Remove deprecated buildw and wix plugin (#2849)
Fixes #2771
This PR removes deprecated and obsolete wix plugin and corresponding buildw to build Windows related artifacts.
2018-10-04 06:12:33 +05:30
Rohit Yadav
3614f5dd8b Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-08-22 15:55:21 +05:30
Slair1
023dcec5ef CLOUDSTACK-10310 Fix KVM reboot on storage issue (#2722) 2018-08-20 10:28:03 +02:00
Wido den Hollander
1ccb32f814 security_group: Use execute() function instead of non-existing bash() (#2732)
In 6233a77d15adde86302a11b2bcda604313833e91 as a part of PR #2432 the
bash() function was replaced by the execute() function.

Somehow this last calling of the bash() function was not caught by testing
and is still in there.

This causes Exceptions to be thrown by the Security Group script.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-07-20 09:07:29 +02:00
Rafael Weingärtner
d0c6cacd06
[CLOUDSTACK-9261] Upgrate jQuery-UI to 1.11 (JQuery UI 1.8.4 prone to XSS) (#2524)
* [CLOUDSTACK-9261] Upgrate jQuery-UI to 1.11 (JQuery UI 1.8.4 prone to XSS)

* fix problems in the UI for lbCertificatePolicy and StaticNAT

* force jenkins build

* Fix about dialog

* Fix position of network service offering
2018-07-16 13:18:55 -03:00
Rohit Yadav
e7bd73e72b Merge branch '4.11' 2018-05-04 12:39:53 +05:30
Rohit Yadav
eb75c1eff5 ca: Fixes #2530 have all IPs from KVM host in issued X509 cert
This ensures that certificate setup includes all the IP addresses (v4
and v6) when a (KVM) host is added to CloudStack. This fixes #2530.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-05-04 12:37:23 +05:30
Rafael Weingärtner
b3c22df71d Forward merge branch '4.11' to master
ConfigDrive fixes: CLOUDSTACK-10288, CLOUDSTACK-10289 (#2566)
CLOUDSTACK-9677: Adding storage policy support for swift as secondary
storage (#2412)
2018-04-26 10:14:49 -03:00
Khosrow Moossavi
b6d420bec3 CLOUDSTACK-9677: Adding storage policy support for swift as secondary storage (#2412)
Original-Author: @pdube on PR Fixes #1830.
2018-04-26 00:42:15 +02:00
Rohit Yadav
5ff2f172cb Merge branch '4.11' 2018-04-24 15:05:38 +02:00
Rohit Yadav
ea55a002cf
network: Fix security groups for CentOS (#2590)
On RHEL/CentOS/Fedora the bridge related sysctl rules are enabled
in kernel by default but can only be disabled. Enabling those keys
will fail, causing iptables/ebtables tables to not be created
and fails SG on CentOS.

This also fixes an integration test case, which assumes first few
tests complete within 3 minutes. In nested env the value may be large,
this increases the value to 20 minutes.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-23 16:54:03 +02:00
Rohit Yadav
644b0910cd Merge branch '4.11'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-20 00:46:43 +05:30
Rohit Yadav
8da2462469
CLOUDSTACK-10333: Secure Live VM Migration for KVM (#2505)
This extends securing of KVM hosts to securing of libvirt on KVM
host as well for TLS enabled live VM migration. To simplify implementation
securing of host implies that both host and libvirtd processes are
secured with management server's CA plugin issued certificates.

Based on whether keystore and certificates files are available at
/etc/cloudstack/agent, the KVM agent determines whether to use TLS or
TCP based uris for live VM migration. It is also enforced that a secured
host will allow live VM migration to/from other secured host, and an
unsecured hosts will allow live VM migration to/from other unsecured
host only.

Post upgrade the KVM agent on startup will expose its security state
(secured detail is sent as true or false) to the managements server that
gets saved in host_details for the host. This host detail can be accesed
via the listHosts response, and in the UI unsecured KVM hosts will show
up with the host state of ‘unsecured’. Further, a button has been added
that allows admins to provision/renew certificates to KVM hosts and can
be used to secure any unsecured KVM host.

The `cloudstack-setup-agent` was modified to accept a new flag `-s`
which will reconfigure libvirtd with following settings:

    listen_tcp=0
    listen_tls=1
    tcp_port="16509"
    tls_port="16514"
    auth_tcp="none"
    auth_tls="none"
    key_file = "/etc/pki/libvirt/private/serverkey.pem"
    cert_file = "/etc/pki/libvirt/servercert.pem"
    ca_file = "/etc/pki/CA/cacert.pem"

For a connected KVM host agent, when the certificate are
renewed/provisioned a background task is scheduled that waits until all
of the agent tasks finish after which libvirt process is restarted and
finally the agent is restarted via AgentShell.

There are no API or DB changes.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-04-20 00:36:18 +05:30
Olivier Lemasle
3e62ce9c1b CLOUDSTACK-10331: Remove reference to deleted script vm_snapshots.js (#2497) 2018-03-21 08:00:17 -03:00
Wido den Hollander
6233a77d15 CLOUDSTACK-10294: PEP-8 fixes and enhancements to security_group.py (#2432)
- We should return a boolean and not a String 'true' or 'false'. Although this output is never checked by the calling function(s).
- Do not use == False or == None as that is not according to the Python specs.
- Calling just print 'hello' is deprecated and won't work in newer Python versions. We should use the print() function.
- Remove unused and commented function.
- Use logging.warning() instead of logging.warn()
- Use subprocess.check_output() for execution. This is the Python way of executing commands.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2018-02-15 14:28:27 +01:00
Rohit Yadav
170b6ce20d
CLOUDSTACK-10236: Enable dynamic roles for missing props file (#2426)
Automate dynamic roles migration for missing props file

- In case commands.properties file is missing, enables dynamic roles.
- Adds a new -D or --default flag to migrate-dynamicroles.py script
  to simply update the global setting and use the default role-rule
  permissions.
- Add warning message, ask admins to move to dynamic roles during upgrade

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-24 13:11:08 +01:00
Wido den Hollander
f80f00ebcb CLOUDSTACK-10242: Properly parse incoming rules to Sec Group (#2418)
With merge of PR #2028 the separator for lines to the Security Group
Python script changed from : to ; to support IPv6 addresses.

This broke certain situations where rules were parsed improperly. This
commit fixes the issue.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-23 20:21:49 +01:00
Pierre-Luc Dion
d84b268b4c CLOUDSTACK-10229: improve xenserver swift logging, removed usued code (#2152)
This script is used to upload snapshots to swift and is executed on dom0 of XenServer. The PR make logging from /var/log/cloud/swiftxenserver.log more meaningful as the below example;

2017-06-15 10:26:32    DEBUG [root] #### CLOUD enter  swift ####
2017-06-15 10:26:32    DEBUG [root] #### CLOUD upload begin    S-12522/d841b62a-7f83-4d5d-9e9d-2940115f7fa9.vhd to swift ####
2017-06-15 10:27:13    DEBUG [root] #### CLOUD upload complete S-12522/d841b62a-7f83-4d5d-9e9d-2940115f7fa9.vhd to swift: 0:00:40 @ 45 MB/s ####
2017-06-15 10:27:13    DEBUG [root] #### CLOUD exit   swift ####
2018-01-12 19:53:35 +05:30
Rohit Yadav
b0d7844cf0
CLOUDSTACK-10109: Fix regression from PR #2295 (#2394)
This fixes regression introduced in PR #2295:
- Pass assign=true to fetch new public IP
- Use wait_until instead of sleep+wait in tests
- Loop through list of public IP ranges to match the systemvm gateway
- Fix potential NPE seen when adding simulator host(s)
- Removes aria2 installation from setup_agent.sh using yum, it's already
  dependency for cloudstack-agent package

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2018-01-10 00:44:00 +05:30
Nicolas Vazquez
e86bb41e0e CLOUDSTACK-10146: Bypass Secondary Storage for KVM templates (#2379)
This feature allows using templates and ISOs avoiding secondary storage as intermediate cache on KVM. The virtual machine deployment process is enhanced to supported bypassed registered templates and ISOs, delegating the work of downloading them to primary storage to the KVM agent instead of the SSVM agent.

Template and ISO registration:
- When hypervisor is KVM, a checkbox is displayed with 'Direct Download' label.
- API methods registerTemplate and registerISO are both extended with this new parameter directdownload.
- On template or ISO registration, no download job is sent to SSVM agent, CloudStack would only persist an entry on template_store_ref indicating that template or ISO has been marked as 'Direct Download' (bypassing Secondary Storage). These entries are persisted as:
template_id = Template or ISO id on vm_template table
store_id NULL
download_state = BYPASSED
state = Ready
(Note: these entries allow users to deploy virtual machine from registered templates or ISOs)
- An URL validation command is sent to a random KVM host to check if template/ISO location can be reached. Metalink are also supported by this feature. In case of a metalink, it is fetched and URL check is performed on each of its URLs.
- Checksum should be provided as indicated on #2246: {ALGORITHM}CHKSUMHASH
- After template or ISO is registered, it would be displayed in the UI

Virtual machine deployment:
When a 'Direct Download' template is selected for deployment, CloudStack would delegate template downloading to destination storage pool via destination host by a new pluggable download manager.
Download manager would handle template downloading depending on URL protocol. In case of HTTP, request headers can be set by the user via vm_template_details. Those details should be persisted as:
Key: HTTP_HEADER
Value: HEADERNAME:HEADERVALUE

In case of HTTPS, a new API method is added uploadTemplateDirectDownloadCertificate to allow user importing a client certificate into all KVM hosts' keystore before deployment.
After template or ISO is downloaded to primary storage, usual entry would be persisted on template_spool_ref indicating the mapping between template/ISO and storage pool.
2018-01-09 12:22:18 +05:30
Rohit Yadav
d19629a115 CLOUDSTACK-10013: Fixes based on code review and test failures
This includes test related fixes and code review fixes based on
reviews from @rafaelweingartner, @marcaurele, @wido and @DaanHoogland.

This also includes VMware disk-resize limitation bug fix based on comments
from @sateesh-chodapuneedi and @priyankparihar.

This also includes the final changes to systemvmtemplate and fixes to
code based on issues found via test failures.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-12-23 17:51:42 +05:30
Rafael Weingärtner
3c6df7c970 CLOUDSTACK-10196: Remove ejb-api 3.0 dependency (#2348)
Fixed QuotaAlertManagerImplTest, which was injecting mock objects manually.
2017-12-18 23:59:06 +05:30
Wido den Hollander
632479d8f8 CLOUDSTACK-9853: Add support for Secondary IPv6 Addresses and Subnets (#2028)
This commit adds support for passing IPv6 Addresses and/or Subnets as
Secondary IPs.

This is groundwork for CLOUDSTACK-9853 where IPv6 Subnets have to be
allowed in the Security Groups of Instances to we can add DHCPv6
Prefix Delegation.

Use ; instead of : for separating addresses, otherwise it would cause
problems with IPv6 Addresses.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-11-22 17:30:33 +05:30
dahn
ed7811a9a2 CLOUDSTACK-10046 checksum validation for any java supported Digests-type (#2246)
* CLOUDSTACK-10046 digest helper for calculating checksums

* CLOUDSTACK-10046 cleanup unused checksum code

* CLOUDSTACK-10046 padding method proof of concept

* CLOUDSTACK-10046 only compare checksums if old value is valid

* Adding positive and negative tests for md5, sha-1 and sha-256, for xen, vmware and kvm hypervisors.
KVM Results:

 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 189, in test_02_1_create_template_with_checksum_sha1_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{sha-1}bf580a13f791d86acf3449a7b457a91a14389264" didn\'t match the given value, "{sha-1}someInvalidValue"\n']
=== TestName: test_02_1_create_template_with_checksum_sha1_negative | Status : SUCCESS ===
=== TestName: test_02_create_template_with_checksum_sha1 | Status : SUCCESS ===.
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 203, in test_03_1_create_template_with_checksum_sha256_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{SHA-256}efc03633f2b8f5db08acbcc5dc1be9028572dfd8f1c6c8ea663f0ef94b458c5" didn\'t match the given value, "{SHA-256}someInvalidValue"\n']
=== TestName: test_03_1_create_template_with_checksum_sha256_negative | Status : SUCCESS ===
=== TestName: test_03_create_template_with_checksum_sha256 | Status : SUCCESS ===
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 217, in test_04_1_create_template_with_checksum_md5_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{md5}ada77653dcf1e59495a9e1ac670ad95f" didn\'t match the given value, "{md5}someInvalidValue"\n']
=== TestName: test_04_1_create_template_with_checksum_md5_negative | Status : SUCCESS ===
=== TestName: test_04_create_template_with_checksum_md5 | Status : SUCCESS ===

* CLOUDSTACK-10046 digest helper for calculating checksums

* CLOUDSTACK-10046 cleanup unused checksum code

* CLOUDSTACK-10046 padding method proof of concept

* CLOUDSTACK-10046 only compare checksums if old value is valid

* Adding positive and negative tests for md5, sha-1 and sha-256, for xen, vmware and kvm hypervisors.
KVM Results:

 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 189, in test_02_1_create_template_with_checksum_sha1_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{sha-1}bf580a13f791d86acf3449a7b457a91a14389264" didn\'t match the given value, "{sha-1}someInvalidValue"\n']
=== TestName: test_02_1_create_template_with_checksum_sha1_negative | Status : SUCCESS ===
=== TestName: test_02_create_template_with_checksum_sha1 | Status : SUCCESS ===.
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 203, in test_03_1_create_template_with_checksum_sha256_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{SHA-256}efc03633f2b8f5db08acbcc5dc1be9028572dfd8f1c6c8ea663f0ef94b458c5" didn\'t match the given value, "{SHA-256}someInvalidValue"\n']
=== TestName: test_03_1_create_template_with_checksum_sha256_negative | Status : SUCCESS ===
=== TestName: test_03_create_template_with_checksum_sha256 | Status : SUCCESS ===
 Negative Test Passed - Exception Occurred Under template download ['Traceback (most recent call last):\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 217, in test_04_1_create_template_with_checksum_md5_negative\n    self.download(self.apiclient, template.id)\n', '  File "/Users/bstoyanov/Documents/sb2/cloudstack/test/integration/smoke/test_templates.py", line 260, in download\n    template.status)\n', 'Exception: Failed to download template: status - Failed post download script: checksum "{md5}ada77653dcf1e59495a9e1ac670ad95f" didn\'t match the given value, "{md5}someInvalidValue"\n']
=== TestName: test_04_1_create_template_with_checksum_md5_negative | Status : SUCCESS ===
=== TestName: test_04_create_template_with_checksum_md5 | Status : SUCCESS ===

* Adding additional test with no checksum added when registering template
Result:
test_05_create_template_with_no_checksum (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_05_create_template_with_no_checksum | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 1 test in 42.320s

OK

* Fixing negative tests exception handling

* Adding tests for ISO checksum validation and fixing a zero prefix failure test in templates

* CLOUDSTACK-10046 padding

* CLOUDSTACK-10046 usability additions

* yet another IDE artifact hindering checkstyle
2017-10-11 11:49:06 +02:00
Rafael Weingärtner
b044f1bc96 CLOUDSTACK-9859: Retirement of midonet plugin (final removal) (#2285)
Following the component retirement process defined in [1], a vote thread was started in [2]. The community decided to retire this Midonet plugin. This task represents the final step of the retirement, which is the removal of the plugin from CloudStacks code base.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68720798
[2] http://markmail.org/message/qigrtfirwnmct4hr
2017-10-10 10:08:23 +05:30
Boris Stoyanov
f917ab660e CLOUDSTACK-9782: Improve host HA tests
- All tests should pass on KVM, Simulator
- Add test cases covering FSM state transitions and actions

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav
212e5ccfa7 CLOUDSTACK-9782: Host HA and KVM HA provider
Host-HA offers investigation, fencing and recovery mechanisms for host that for
any reason are malfunctioning. It uses Activity and Health checks to determine
current host state based on which it may degrade a host or try to recover it. On
failing to recover it, it may try to fence the host.

The core feature is implemented in a hypervisor agnostic way, with two separate
implementations of the driver/provider for Simulator and KVM hypervisors. The
framework also allows for implementation of other hypervisor specific provider
implementation in future.

The Host-HA provider implementation for KVM hypervisor uses the out-of-band
management sub-system to issue IPMI calls to reset (recover) or poweroff (fence)
a host.

The Host-HA provider implementation for Simulator provides a means of testing
and validating the core framework implementation.

Signed-off-by: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-30 18:06:48 +02:00
Rohit Yadav
7ce54bf7a8 CLOUDSTACK-9993: Securing Agents Communications (#2239)
This introduces a new certificate authority framework that allows
pluggable CA provider implementations to handle certificate operations
around issuance, revocation and propagation. The framework injects
itself to `NioServer` to handle agent connections securely. The
framework adds assumptions in `NioClient` that a keystore if available
with known name `cloud.jks` will be used for SSL negotiations and
handshake.

This includes a default 'root' CA provider plugin which creates its own
self-signed root certificate authority on first run and uses it for
issuance and provisioning of certificate to CloudStack agents such as
the KVM, CPVM and SSVM agents and also for the management server for
peer clustering.

Additional changes and notes:
- Comma separate list of management server IPs can be set to the 'host'
  global setting. Newly provisioned agents (KVM/CPVM/SSVM etc) will get
  radomized comma separated list to which they will attempt connection
  or reconnection in provided order. This removes need of a TCP LB on
  port 8250 (default) of the management server(s).
- All fresh deployment will enforce two-way SSL authentication where
  connecting agents will be required to present certificates issued
  by the 'root' CA plugin.
- Existing environment on upgrade will continue to use one-way SSL
  authentication and connecting agents will not be required to present
  certificates.
- A script `keystore-setup` is responsible for initial keystore setup
  and CSR generation on the agent/hosts.
- A script `keystore-cert-import` is responsible for import provided
  certificate payload to the java keystore file.
- Agent security (keystore, certificates etc) are setup initially using
  SSH, and later provisioning is handled via an existing agent connection
  using command-answers. The supported clients and agents are limited to
  CPVM, SSVM, and KVM agents, and clustered management server (peering).
- Certificate revocation does not revoke an existing agent-mgmt server
  connection, however rejects a revoked certificate used during SSL
  handshake.
- Older `cloudstackmanagement.keystore` is deprecated and will no longer
  be used by mgmt server(s) for SSL negotiations and handshake. New
  keystores will be named `cloud.jks`, any additional SSL certificates
  should not be imported in it for use with tomcat etc. The `cloud.jks`
  keystore is stricly used for agent-server communications.
- Management server keystore are validated and renewed on start up only,
  the validity of them are same as the CA certificates.

New APIs:
- listCaProviders: lists all available CA provider plugins
- listCaCertificate: lists the CA certificate(s)
- issueCertificate: issues X509 client certificate with/without a CSR
- provisionCertificate: provisions certificate to a host
- revokeCertificate: revokes a client certificate using its serial

Global settings for the CA framework:
- ca.framework.provider.plugin: The configured CA provider plugin
- ca.framework.cert.keysize: The key size for certificate generation
- ca.framework.cert.signature.algorithm: The certificate signature algorithm
- ca.framework.cert.validity.period: Certificate validity in days
- ca.framework.cert.automatic.renewal: Certificate auto-renewal setting
- ca.framework.background.task.delay: CA background task delay/interval
- ca.framework.cert.expiry.alert.period: Days to check and alert expiring certificates

Global settings for the default 'root' CA provider:
- ca.plugin.root.private.key: (hidden/encrypted) CA private key
- ca.plugin.root.public.key: (hidden/encrypted) CA public key
- ca.plugin.root.ca.certificate: (hidden/encrypted) CA certificate
- ca.plugin.root.issuer.dn: The CA issue distinguished name
- ca.plugin.root.auth.strictness: Are clients required to present certificates
- ca.plugin.root.allow.expired.cert: Are clients with expired certificates allowed

UI changes:
- Button to download/save the CA certificates.

Misc changes:
- Upgrades bountycastle version and uses newer classes
- Refactors SAMLUtil to use new CertUtils

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-08-28 12:15:11 +02:00
SudharmaJain
5397106a76 CLOUDSTACK-8906: /var/log/cloud/ doesn't get logrotated on xenserver (#883)
After integrating XS with CCP the following folder gets created: /var/log/cloud/ however the logs in that are not rotated resulting in root file system fill up. It was a known issue and link http://support.citrix.com/article/CTX138064 describes the issue and solution. Used the article and added corresponding changes to Cloudstack.
2017-08-03 00:51:16 +02:00
Wido den Hollander
990e4a6e40 ipv6: Allow all ICMPv6 traffic if -1 is provided as a ICMP type
ip6tables no longer takes '--icmpv6-type any' as a argument.

To allow all ICMPv6 traffic with ip6tables it has to be invoked this way:

  $ ip6tables -I i-2-14-VM -p icmpv6 -s ::/0 -j ACCEPT

All ICMPv6 traffic is now allow into the Instance.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-07-25 16:31:35 +02:00
Rajani Karuturi
c95b47b740 Merge pull request #1824 from jayapalu/SGipset
CLOUDSTACK-9657: Fixed security group ipset issues with long vm name
2017-05-19 14:05:51 +05:30
Daan Hoogland
70ef0788c9 CLOUDSTACK-9408: Fix download urls in sql and scripts
This fixes the agreed upon url on download.cloudstack.org in various
sql files and misc scripts.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2017-04-20 12:33:33 +05:30
Jayapal
43a991d2eb CLOUDSTACK-9821: Fixed issue in deploying vm in basic zone 2017-03-06 19:01:15 +05:30
Jayapal
1cff18fcfa CLOUDSTACK-9657: Fixed security group ipset issues with long vm name 2017-02-24 15:20:04 +05:30
Jayapal
175c8d83b8 CLOUDSTACK-8871: fixed issue with the xenserver 6.2 ipset nethash 2017-02-20 18:34:13 +05:30
Wido den Hollander
115d6d5dc7
CLOUDSTACK-676: IPv6 In -and Egress filtering for Basic Networking
This commit implements Ingress and Egress filtering for IPv6 in
Basic Networking.

It allows for opening and closing ports just as can be done with IPv4.

Rules have to be specified twice, once for IPv4 and once for IPv6, for
example:

- 22 until 22: 0.0.0.0/0
- 22 until 22: ::/0

Egress filtering works the same as with IPv4. When no rule is applied all
traffic is allowed. Otherwise only the specified traffic (with DNS being
the exception) is allowed.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-01-26 15:36:20 +01:00
Wido den Hollander
84e496b4f9
CLOUDSTACK-676: IPv6 Basic Security Grouping for KVM
This commit implements basic Security Grouping for KVM in
Basic Networking.

It does not implement full Security Grouping yet, but it does:
- Prevent IP-Address source spoofing
- Allow DHCPv6 clients, but disallow DHCPv6 servers
- Disallow Instances to send out Router Advertisements

The Security Grouping allows ICMPv6 packets as described by RFC4890
as they are essential for IPv6 connectivity.

Following RFC4890 it allows:
- Router Solicitations
- Router Advertisements (incoming only)
- Neighbor Advertisements
- Neighbor Solicitations
- Packet Too Big
- Time Exceeded
- Destination Unreachable
- Parameter Problem
- Echo Request

ICMPv6 is a essential part of IPv6, without it connectivity will break or be very
unreliable.

For now it allows any UDP and TCP packet to be send in to the Instance which
effectively opens up the firewall completely.

Future commits will implement Security Grouping further which allows controlling UDP and TCP
ports for IPv6 like can be done with IPv4.

Regardless of the egress filtering (which can't be done yet) it will always allow outbound DNS
to port 53 over UDP or TCP.

Signed-off-by: Wido den Hollander <wido@widodh.nl>
2017-01-26 15:36:08 +01:00
Rohit Yadav
0db4471be0
Merge branch '4.9' 2016-12-26 21:01:20 +05:30
Rohit Yadav
2ce02dc937 CLOUDSTACK-9662: Fix failing iscsi test
Use separate lvcreate command on XenServer7 hosts, that checks and passes
different parameters based on the xenserver release version.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-25 23:53:50 +05:30
Rohit Yadav
342162bad7 Merge branch '4.9'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
2016-12-22 11:59:02 +05:30