CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe wayThe problem arises when the origin hypervisor has an ip addres that ends with 1, like '10.10.10.1' and the qemu VM description is containing an address that has that as part of its address, '10.10.10.100' for instance.
now migrating to '10.10.10.10' will change both addresses in the xml description file for qemu. It is fixed and unit tests are added. I am not sure yet how to integration test this. Regression will probably work so creating a PR now.
* pr/1348:
CLOUDSTACK-9142 Migrate VM changes xmlDesc in a safe way
Signed-off-by: Will Stevens <williamstevens@gmail.com>
It could happen that when multiple instances are starting at the same
time on a KVM host the Agent spawns multiple instances of security_group.py
which both try to modify iptables/ebtables rules.
This fails with on of the two processes failing.
The instance is still started, but it doesn't have any IP connectivity due
to the failed programming of the security groups.
This modification lets the script aqcuire a exclusive lock on a file so that
only one instance of the scripts talks to iptables/ebtables at once.
Other instances of the script which start will poll every 500ms if they can
obtain the lock and otherwise execute anyway after 15 seconds.
The lock will be released as soon as the script exists, which is usually within
a few hundred ms.
Fixed: Error given when creating VPN user in one network if VR for another network is stopped.
Problem: Adding a VPN user to an IP on an isolated network fails with an exception if the end-user has two isolated networks and one of them is not in use (i.e. VR is stopped).
Fix: Presently in the code, all the VR which are not in Running state are forced to throw error. Added a check for the Stopped and Stopping state routers. Configurations will be applied to them when they get restarted.
* pr/826:
Fixed Coverity Issue: Unintentional Integer FLow
Fixed: Error given when creating VPN user in one network if VR for another network is stopped
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9352: Test fails in Widows as the file separator "/" is different from "\"**Problem:**
File separator in windows ("\") is different from the expected in the test ("/"); thus, the test *com.cloud.utils.SwiftUtilTest.testSplitSwiftPath()* will fail in Windows systems.
The problem is that the input of the test is "*container/object*" but the tested method uses the *File.separator* (that depends from the OS), in windows systems the tested method (*com.cloud.utils.SwiftUtil.splitSwiftPath(String)*) looks for a "\", as the string does not contain "\" it returns an empty string and consequently results in a test failure.
**Solution:**
Create a string `String input = "container" + File.separator + "object";`, with that the test will validate the tested method verifying if the method splits the string around matches of the given regular expression (in this case *File.separator*).
*JIRA link: https://issues.apache.org/jira/browse/CLOUDSTACK-9352*
* pr/1498:
Test fails in Widows as the file separator "/" is different from "\"
Signed-off-by: Will Stevens <williamstevens@gmail.com>
BUG-ID:CLOUDSTACK-9331:added code in marvin frame&new config file for advBaremetal support Added code in marvin framework&new config file to support baremetal advanced testcases
* pr/1458:
BUG-ID:CLOUDSTACK-9331:added code in marvin frame&new config file to support baremetal advanced testcases
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9322: Support for Internal LB fuctionality with Nuage VSP SDN Plugin including Marvin test coverageTask: https://issues.apache.org/jira/browse/CLOUDSTACK-9322
PR contents:
1) UI changes to support LB provider InternalLbVm during VPC offering creation.
2) Bug fix for CLOUDSTACK-9320.
3) Nuage VSP SDN Plugin related enhancements for VPC network functionality.
4) Marvin test coverage for Internal LB support on master with Nuage VSP SDN Plugin.
5) Enhancements on our exiting Marvin test code (nuagevsp plugins directory).
6) PEP8 & PyFlakes compliance with our Marvin test code.
Test run:
CloudStack$ nosetests --with-marvin --marvin-config=nuage_ant.cfg test/integration/plugins/nuagevsp/ -a tags=nuagevsp
Test results:
Test user data and password reset functionality with Nuage VSP SDN plugin ... === TestName: test_nuage_UserDataPasswordReset | Status : SUCCESS ===
ok
Test Nuage VSP VPC Offering with different combinations of LB service providers ... === TestName: test_01_nuage_internallb_vpc_Offering | Status : SUCCESS ===
ok
Test Nuage VSP VPC Network Offering with and without Internal LB service ... === TestName: test_02_nuage_internallb_vpc_network_offering | Status : SUCCESS ===
ok
Test Nuage VSP VPC Networks with and without Internal LB service ... === TestName: test_03_nuage_internallb_vpc_networks | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with different combinations of Internal LB rules ... === TestName: test_04_nuage_internallb_rules | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality by performing (wget) traffic tests within a VPC ... === TestName: test_05_nuage_internallb_traffic | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with different LB algorithms by performing (wget) traffic tests ... === TestName: test_06_nuage_internallb_algorithms_traffic | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with restarts of VPC network components by performing (wget) ... === TestName: test_07_nuage_internallb_vpc_network_restarts_traffic | Status : SUCCESS ===
ok
Test Nuage VSP VPC Internal LB functionality with InternalLbVm appliance operations by performing (wget) ... === TestName: test_08_nuage_internallb_appliance_operations_traffic | Status : SUCCESS ===
ok
Test Basic VPC Network Functionality with Nuage VSP SDN plugin ... === TestName: test_nuage_vpc_network | Status : SUCCESS ===
ok
Test Nuage VSP SDN plugin with basic Isolated Network functionality ... === TestName: test_nuage_vsp | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 11 tests in 12094.705s
OK
Test run logs:
[results.txt](https://github.com/apache/cloudstack/files/187587/results.txt)
[runinfo.txt](https://github.com/apache/cloudstack/files/187588/runinfo.txt)
Test config file:
[nuage_ant.txt](https://github.com/apache/cloudstack/files/222711/nuage_ant.txt)
Note: Attached the Marvin config file as .txt instead of .cfg.
PEP8 & PyFlakes Compliance:
CloudStack$ pep8 --max-line-length=150 test/integration/plugins/nuagevsp/*.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/nuageTestCase.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_password_reset.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_vpc_internal_lb.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_vpc_network.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/test_nuage_vsp.py
CloudStack$ pyflakes test/integration/plugins/nuagevsp/*.py
#CLOUDSTACK-9322
* pr/1452:
CLOUDSTACK-9322 : Marvin tests for Internal Lb with Nuage VSP
CLOUDSTACK-9320 : InternalLBVM is not getting destroyed when the last Internal Load Balancer rule is removed for the corresponding source IP address
CLOUDSTACK-9322 : Changes to support InternalLbVm with Nuage VSP plugin
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9164: Prevent firefox's quick search from opening in VM consolehttps://issues.apache.org/jira/browse/CLOUDSTACK-9164
To test:
In Firefox open any VM conosle, and try typing "/".
It should be typed in VM console and quick search of firefox should not open.
* pr/1271:
CLOUDSTACK-9164: Prevent firefox's quick search from opening in VM console
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-8745 : verify usage after root disk migrationput storage in maintenance mode and start ha vm and check usage ... === TestName: test_ha_with_storage_maintenance | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 842.294s
OK
* pr/713:
CLOUDSTACK-8745 : verify usage after root disk migration
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Snapshot removing implemented on RBD.
1. On management side: when created new shanpshot we checking if our primary storage is RBD,
then do not remove record from cloud.snapshot_store_ref with link to Ceph
image via 'install_path' field.
2. On management side: when removing snapshot, also send command to agent 'DeleteCommand'.
3. On agent side: method implemented 'public Answer deleteSnapshot(final DeleteCommand cmd)'
Found this issue when using MySQL 5.7 with Ubuntu 16.04 with following settings:
sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION"
server-id = 1
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
- Fix jetty dependency that is compatible with Java7
- Upgrade minor revisions of dependencies
- Upgrade vmware mvn sdk dependency to 6.0
- Downgrade bounty castle version to 1.46 (same as before)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Updated most dependencies to latest minor releases, EXCEPT:
- Gson 2.x
- Major spring framework version
- Servlet version
- Embedded jetty version
- Mockito version (beta)
- Mysql lib minor version upgrade (breaks mysql-ha plugin)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
This closes#561
CLOUDSTACK-8611:Handle SSH if server "forget" to send exit statusContinuing the work started by @likitha, I did not cherry-picked the
commit (b9181c689e0e7b5f1e28c81d73710196dfabd0ba) from PR <https://github.com/apache/cloudstack/pull/561> due to the fact that the path of that SshHelper class was different of the current SshHelper; that is because the fact that by cherry-picking it would seem that I had changed all the class as the code is from another file.
I made some changes from the cherry-picked commit adding @wilderrodrigues suggestions (create simple methods to have reusable code, make unit tests and create the `WAITING_OPEN_SSH_SESSION` variable to manipulate with the delay of 1000 milliseconds).
Also, I tried to simplify the logic by assuming that ....
if ((conditions & ChannelCondition.EXIT_STATUS) != 0) {
if ((conditions & (ChannelCondition.STDOUT_DATA | ChannelCondition.STDERR_DATA)) == 0) {
break;
}
}
... is the same as `((conditions & ChannelCondition.EXIT_STATUS) != 0) && ((conditions & (ChannelCondition.STDOUT_DATA | ChannelCondition.STDERR_DATA)) == 0)`. This expression has the following results according to each possible condition.
|Condition|Value|result
|-----------------|-------|------|
TIMEOUT | 0000001|false
CLOSED | 0000010 |false
STDERR_DATA | 0000100 | false
STDERR_DATA | 0001000 | false
EOF | 0010000 | false
EXIT_STATUS | 0100000 | **true**
EXIT_SIGNAL | 1000000 | false
After testing all the possibilities we can note that the condition of `(conditions & ChannelCondition.EXIT_STATUS) != 0` is sufficient; thus, the simplified "if" conditional can be:
`if ((conditions & ChannelCondition.EXIT_STATUS) != 0) {
break;
}`
This proposed work can be explained by quoting @likitha:
>CheckS2SVpnConnectionsCommand execution involves executing a script (checkbatchs2svpn.sh) in the virtual router. Once CS has opened a session to a virtual router and executed a script in the router, it waits indefinitely till the session either times out or the exit status of the remote process is available. But it is possible that an EOF is reached by the process in the router and the router never set the exit status.
>References -
>1. Some servers never send the exit status, or occasionally "forget" to do so (http://grepcode.com/file/repo1.maven.org/maven2/org.jvnet.hudson/trilead-ssh2/build212-hudson-1/com/trilead/ssh2/ChannelCondition.java).
>2. Get the exit code/status from the remote command - if available. Be careful - not all server implementations return this value - (http://grepcode.com/file/repo1.maven.org/maven2/org.jvnet.hudson/trilead-ssh2/build212-hudson-1/com/trilead/ssh2/Session.java#Session.waitForCondition%28int%2Clong%29).
* pr/1459:
Handle SSH if server "forget" to send exit status
Signed-off-by: Will Stevens <williamstevens@gmail.com>
File separator in windows is different from linux (the expected in the
test); thus, the test
*com.cloud.utils.SwiftUtilTest.testSplitSwiftPath()* will fail in
windows. The problem is that the input of the test is
*"container/object"* but the tested method uses the *File.separator*
(that depends from the OS), in the windows the tested method
(*com.cloud.utils.SwiftUtil.splitSwiftPath(String)*) looks for a "\" in
windows systems, resulting in an empty string and consequently a failure
in the test.
Some solutions:
- the simple way is to create a string `String input = "container" +
File.separator + "object";`, thus independent of the OS, the test will
succeed.
- a tricky solution is to mock the final static variable
*File.separator* and return "/".
I picked the easy way.
Test to create vpn customer gateway with hostnameWhile adding vpn customer gateway for site to site vpn connection, cs should also accept host name apart from gateway ip address. It should not be restricted to just ip address.
* pr/1308:
Added few validation steps after adding vpncustomer gateway with hostname Changes are as per review comments in PR#1308
Test to verify CS-45057 Bug-Id: CS-45057 Reviewed-by: Self
Signed-off-by: Will Stevens <williamstevens@gmail.com>
New test to validate starting vm after nic removal and attachPlease refer bug CLOUDSTACK-9219 for more details.
Test Results:
==========
Test to verify vm start after NIC removal and reattach ... === TestName: test_30_remove_nic_reattach | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 277.478s
OK
* pr/1326:
New test to validate starting vm after nic removal and attach Bug-Id: CLOUDSTACK-9219
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9088: Update the description for migrateVirtualMachineWithVolume apihttps://issues.apache.org/jira/browse/CLOUDSTACK-9088
* pr/1126:
CLOUDSTACK-9088: Update the description for migrateVirtualMachineWithVolume api.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
[CLOUDSTACK-9218]Test to verify restart network after master VR destroyedPlease refer CLOUDSTACK-9218 for more details
Test Results:
===========
Test restarting RvR network without cleanup after destroying master VR ... === TestName: test_restart_ntwk_MVR_destroyed | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 1 test in 581.194s
OK
* pr/1323:
Added new test to verify restart network after destorying master VR Bug-Id: CLOUDSTACK-9218
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Fixing an issue in Marvin around creating a template from a snapshotThis fixes the following ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-9354
The problem was that Marvin was requiring you to pass in the "ispublic" parameter when creating a template from a snapshot.
As the ticket notes, this issue was introduced by the following commit: https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=bbe0fc4be9527d51820b067a602886003991db4d
The solution I've provided is simply to check if the "ispublic" property is in the dictionary before referencing it.
* pr/1501:
CLOUDSTACK-9354 - Fixing an issue in Marvin around creating a template from a snapshot (if “is public” is not provided, there was a problem)
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag
RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.
* pr/1200:
CLOUDSTACK-9130: Make RebootCommand similar to start/stop/migrate agent commands w.r.t. "execute in sequence" flag RebootCommand now behaves in the same way as start/stop/migrate agent commands w.r.t. to sequential/parallel execution.
Signed-off-by: Will Stevens <williamstevens@gmail.com>
* 4.8:
CLOUDSTACK-9172 Added cross zones check to delete template and iso
Check the existence of 'forceencap' parameter before use
systemvm: set default umask 022 in injectkeys.sh
* 4.7:
CLOUDSTACK-9172 Added cross zones check to delete template and iso
Check the existence of 'forceencap' parameter before use
systemvm: set default umask 022 in injectkeys.sh
CLOUDSTACK-9172 Added cross zones check to delete template and isoAdded a check to ignore the zoneid, in the delete template UI, if the template is cross zones.
reference : CLOUDSTACK-9172
* pr/1505:
CLOUDSTACK-9172 Added cross zones check to delete template and iso
Signed-off-by: Will Stevens <williamstevens@gmail.com>
Check the existence of 'forceencap' parameter before useCheck the existence of 'forceencap' parameter before use.
Error seen:
```
Traceback (most recent call last):
File "/opt/cloud/bin/update_config.py", line 140, in <module>
process_file()
File "/opt/cloud/bin/update_config.py", line 54, in process_file
finish_config()
File "/opt/cloud/bin/update_config.py", line 44, in finish_config
returncode = configure.main(sys.argv)
File "/opt/cloud/bin/configure.py", line 1003, in main
vpns.process()
File "/opt/cloud/bin/configure.py", line 488, in process
self.configure_ipsec(self.dbag[vpn])
File "/opt/cloud/bin/configure.py", line 544, in configure_ipsec
file.addeq(" forceencaps=%s" % CsHelper.bool_to_yn(obj['encap']))
KeyError: 'encap'
```
* pr/1402:
Check the existence of 'forceencap' parameter before use
Signed-off-by: Will Stevens <williamstevens@gmail.com>
systemvm: preserve file permissions, set default umask- In injectkeys.sh which is used to inject new public keys everytime cloudstack
starts; while copying files preserve the mode/ownership. This ensures the
scripts have same mode bits as originally configured in the iso file
- The default umask of 0022 is set in Ubuntu and other packages. Set the same
in case of CentOS startup scripts
cc @abhinandanprateek @wido @remibergsma @DaanHoogland @jburwell
* pr/1420:
systemvm: set default umask 022 in injectkeys.sh
Signed-off-by: Will Stevens <williamstevens@gmail.com>
CLOUDSTACK-9100: ISO.CREATE/TEMPLATE.CREATE event missing for usage_event by template sync thread
If there is a Management server restart while template is in downloading or installing state. Template Sync does not push event into usage_event table.
I have verified the fix manually. Here is a snapshot.
I have registered 4 templates. template id 207 and 208(ISO.CREATE event is missing) before applying the fix. and template id 209 and 210 after applying the fix.
Repro Steps (3 cases)
==========
Case - 1 (private template)
-------------
1. register a private template/iso.
2. restart management server when template is in downloading state.
3. After management server restart, template_store_ref entry is removed if download was not yet completed.
4. on next management server restart, if download would have completed, template_store_ref entry will get populated, but TEMPLATE.CREATE event is missing in usage_event.
Case - 2 (Public template)
--------------------------------
1. register public template.
2. restart management server when in downloading state.
3. after restart template download reinitiates.
4. template goes to ready state, but there is no usage event.
case -3 (public/private template)
---------------------------------
1. register a template
2. restart management server when template is in installing state.
3. after restart template goes to ready, but there is no usage event.
* pr/1157:
CLOUDSTACK-9100: ISO.CREATE/TEMPLATE.CREATE event missing for usage_event by template sync thread
Signed-off-by: Koushik Das <koushik@apache.org>
* 4.8:
CLOUDSTACK-9272: No option in UI to add GSLB with service type "HTTP"
CLOUDSTACK-9270: UI alignment gone bad in multiple places - VM Instance, Network, Egress rules
* 4.7:
CLOUDSTACK-9272: No option in UI to add GSLB with service type "HTTP"
CLOUDSTACK-9270: UI alignment gone bad in multiple places - VM Instance, Network, Egress rules
CLOUDSTACK-9272: No option in UI to add GSLB with service type "HTTP"Steps to Repro:
============
Go to Regions -> Local -> View GSLB -> Add GSLB
Click on the service type dropdown
Observe http is missing. Please see the attached snapshot.
Expected Behaviour:
================
As it supports http also, So http should be in the list.
Actual Behaviour:
==============
http is missing from the list.
Fix:
===
It supports http also. Added http to the list.
Snapshot:
========
<img width="531" alt="gslb-http-missing-nitin" src="https://cloud.githubusercontent.com/assets/12583725/12772818/21513dc0-ca5b-11e5-822e-e2dd2426da65.png">
* pr/1399:
CLOUDSTACK-9272: No option in UI to add GSLB with service type "HTTP"
Signed-off-by: Koushik Das <koushik@apache.org>
Continued the work started by https://github.com/likitha
commit (b9181c6) from PR #561.
CS waits indefinitely for CheckS2SVpnConnectionsComm and to return.
While remote executing commands through ssh, handle channel condition of
EOF because we wait for the the condition.
The SshHelper of the PR #561 is of another path from the
current master, its path was
https://github.com/likitha/cloudstack/commits/CLOUDSTACK-8611/utils/src/com/cloud/utils/ssh/SshHelper.java;
thus, although this commit brings changes from PR #561, I did not
cherry-picked to keep the master file, otherwise it would look that I
had changed all the file.
by me.
