apache/cloudstack
1
0
Fork 0
mirror of https://github.com/apache/cloudstack.git synced 2025-11-03 04:12:31 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #935 from nvazquez/from4.5.1

Browse Source 
CLOUDSTACK-8956: NSX/Nicira Plugin does not support NSX v4.2.1JIRA Ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-8956

### Description of the problem:
Prior to version 4.2. Nicira/VmWare NSX used a variation of Open vSwitch as means of integrating SDN into hypervisor layer. Cloudstack NiciraNVP plugin was written to support OVS as a bridge to NSX.
In version 4.2 VMware introduced NSX vSwitch as a replacement for OVS in ESX hypervisors. It is a fork of distributed vSwitch leveraging one of the recent features of ESX called opaque networks. Because of that change the current version of NiciraNVP plugin doesnt support versions of NSX-MH above 4.2 specifically in Vsphere environment. Proposed fix will analyze a version of NVP/NSX API and use proper support for ESX hypervisors.

vSphere hypervisor mode operations when NV is deployed onto NSX managed network changes:
* Current mode. A portgroup = UUID of CS VM NIC is created on a local standard switch of the Hypervisor where VM is starting. VM nic is attached to that port group.
* New mode. No additional port group is created on a HW. No port group cleanup is needed after VM/NIC is destroyed. VM is attached to 1st port group having the following attributes:
** opaqueNetworkId string "br-int
** opaqueNetworkType string "nsx.network"

If portgroup with such attributes is not found a deployment should fail with exception.

### VMware vSphere API version from 5.1 to 5.5:
Since vSphere API version 5.5, [OpaqueNetworks](https://www.vmware.com/support/developer/converter-sdk/conv55_apireference/vim.OpaqueNetwork.html) are introduced.
Its description says:
> This interface defines an opaque network, in the sense that the detail and configuration of the network is unknown to vShpere and is managed by a management plane outside of vSphere. However, the identifier and name of these networks is made available to vSphere so that host and virtual machine virtual ethernet device can connect to them.

In order to connect a vm's virtual ethernet device to the proper opaque network when deploying a vm into a NSX managed network, we first need to look for a particular opaque network on hosts. This opaque network's id has to be **"br-int"** and its type **"nsx.network"**.

Since vSphere API version 5.5 [HostNetworkInfo](https://www.vmware.com/support/developer/converter-sdk/conv55_apireference/vim.host.NetworkInfo.html#opaqueNetwork) introduces a list of available opaque networks for each host.
If NSX API version >= 4.2 we look for a [OpaqueNetworkInfo](https://www.vmware.com/support/developer/converter-sdk/conv55_apireference/vim.host.OpaqueNetworkInfo.html) which satisfies:
* opaqueNetworkId = "br-int"
* opaqueNetworkType = "nsx.netork"

If that opaque network is found, then we need to attach vm's NIC to a virtual ethernet device which support this, so we use [VirtualEthernetCardOpaqueNetworkBackingInfo](https://www.vmware.com/support/developer/converter-sdk/conv55_apireference/vim.vm.device.VirtualEthernetCard.OpaqueNetworkBackingInfo.html) setting:
* opaqueNetworkId = "br-int"
* opaqueNetworkType = "nsx.netork"

* pr/935:
  CLOUDSTACK-8956: Remove assert(false) on opaque network and ping method on NiciraNvpApiVersion
  CLOUDSTACK-8956: Deploy VM on NSX managed network changes if NSX Api Version >= 4.2: has to connect to "br-int" of "nsx.network" type
  CLOUDSTACK-8956: Log NSX Api Version
  CLOUDSTACK-8956: Add VMware Api v5.5 and change pom.xml to use VMware Api v5.5

Signed-off-by: Remi Bergsma <github@remi.nl>
This commit is contained in:
Remi Bergsma 2015-11-18 13:11:25 +01:00
commit 219da64027
10 changed files with 209 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
deps/install-non-oss.sh vendored
View File

@ -32,3 +32,6 @@ mvn install:install-file -Dfile=manageontap.jar -DgroupId=com.cloud.com.netapp
# From https://my.vmware.com/group/vmware/get-download?downloadGroup=VSP510-WEBSDK-510
# Version: 5.1, Release-date: 2012-09-10, Build: 774886
mvn install:install-file -Dfile=vim25_51.jar -DgroupId=com.cloud.com.vmware -DartifactId=vmware-vim25 -Dversion=5.1 -Dpackaging=jar
# From https://my.vmware.com/group/vmware/get-download?downloadGroup=WEBSDK550
mvn install:install-file -Dfile=vim25_55.jar -DgroupId=com.cloud.com.vmware -DartifactId=vmware-vim25 -Dversion=5.5 -Dpackaging=jar

42
plugins/hypervisors/vmware/src/com/cloud/hypervisor/vmware/resource/VmwareResource.java
View File

@ -83,6 +83,7 @@ import com.vmware.vim25.VirtualDisk;
import com.vmware.vim25.VirtualEthernetCard;
import com.vmware.vim25.VirtualEthernetCardDistributedVirtualPortBackingInfo;
import com.vmware.vim25.VirtualEthernetCardNetworkBackingInfo;
import com.vmware.vim25.VirtualEthernetCardOpaqueNetworkBackingInfo;
import com.vmware.vim25.VirtualMachineConfigSpec;
import com.vmware.vim25.VirtualMachineFileInfo;
import com.vmware.vim25.VirtualMachineFileLayoutEx;
@ -270,6 +271,7 @@ import com.cloud.utils.exception.ExceptionUtil;
import com.cloud.utils.mgmt.JmxUtil;
import com.cloud.utils.mgmt.PropertyMapDynamicBean;
import com.cloud.utils.net.NetUtils;
import com.cloud.utils.nicira.nvp.plugin.NiciraNvpApiVersion;
import com.cloud.utils.ssh.SshHelper;
import com.cloud.vm.VirtualMachine;
import com.cloud.vm.VirtualMachine.PowerState;
@ -1677,26 +1679,35 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
if (s_logger.isDebugEnabled())
s_logger.debug("VM " + vmInternalCSName + " will be started with NIC device type: " + nicDeviceType);
NiciraNvpApiVersion.logNiciraApiVersion();
for (NicTO nicTo : sortNicsByDeviceId(nics)) {
s_logger.info("Prepare NIC device based on NicTO: " + _gson.toJson(nicTo));
boolean configureVServiceInNexus = (nicTo.getType() == TrafficType.Guest) && (vmSpec.getDetails().containsKey("ConfigureVServiceInNexus"));
VirtualMachine.Type vmType = cmd.getVirtualMachine().getType();
Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, configureVServiceInNexus, vmType);
if (VmwareHelper.isDvPortGroup(networkInfo.first())) {
String dvSwitchUuid;
ManagedObjectReference dcMor = hyperHost.getHyperHostDatacenter();
DatacenterMO dataCenterMo = new DatacenterMO(context, dcMor);
ManagedObjectReference dvsMor = dataCenterMo.getDvSwitchMor(networkInfo.first());
dvSwitchUuid = dataCenterMo.getDvSwitchUuid(dvsMor);
s_logger.info("Preparing NIC device on dvSwitch : " + dvSwitchUuid);
nic =
VmwareHelper.prepareDvNicDevice(vmMo, networkInfo.first(), nicDeviceType, networkInfo.second(), dvSwitchUuid, nicTo.getMac(), nicUnitNumber++,
i + 1, true, true);
} else {
s_logger.info("Preparing NIC device on network " + networkInfo.second());
nic =
VmwareHelper.prepareNicDevice(vmMo, networkInfo.first(), nicDeviceType, networkInfo.second(), nicTo.getMac(), nicUnitNumber++, i + 1, true, true);
if ((nicTo.getBroadcastType() != BroadcastDomainType.Lswitch) ||
(nicTo.getBroadcastType() == BroadcastDomainType.Lswitch && NiciraNvpApiVersion.isApiVersionLowerThan("4.2"))){
if (VmwareHelper.isDvPortGroup(networkInfo.first())) {
String dvSwitchUuid;
ManagedObjectReference dcMor = hyperHost.getHyperHostDatacenter();
DatacenterMO dataCenterMo = new DatacenterMO(context, dcMor);
ManagedObjectReference dvsMor = dataCenterMo.getDvSwitchMor(networkInfo.first());
dvSwitchUuid = dataCenterMo.getDvSwitchUuid(dvsMor);
s_logger.info("Preparing NIC device on dvSwitch : " + dvSwitchUuid);
nic =
VmwareHelper.prepareDvNicDevice(vmMo, networkInfo.first(), nicDeviceType, networkInfo.second(), dvSwitchUuid, nicTo.getMac(), nicUnitNumber++,
i + 1, true, true);
} else {
s_logger.info("Preparing NIC device on network " + networkInfo.second());
nic =
VmwareHelper.prepareNicDevice(vmMo, networkInfo.first(), nicDeviceType, networkInfo.second(), nicTo.getMac(), nicUnitNumber++, i + 1, true, true);
}
}
else{
//if NSX API VERSION >= 4.2, connect to br-int (nsx.network), do not create portgroup else previous behaviour
nic = VmwareHelper.prepareNicOpaque(vmMo, nicDeviceType, networkInfo.second(), nicTo.getMac(), nicUnitNumber++, i + 1, true, true);
}
deviceConfigSpecArray[i] = new VirtualDeviceConfigSpec();
@ -2076,6 +2087,9 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
} else if (backing instanceof VirtualEthernetCardNetworkBackingInfo) {
// This NIC is connected to a Virtual Switch
// Nothing to do
} else if (backing instanceof VirtualEthernetCardOpaqueNetworkBackingInfo) {
//if NSX API VERSION >= 4.2, connect to br-int (nsx.network), do not create portgroup else previous behaviour
//OK, connected to OpaqueNetwork
} else {
s_logger.error("nic device backing is of type " + backing.getClass().getName());
throw new Exception("Incompatible backing for a VirtualDevice for nic " + nicIndex); //FIXME Generic exceptions are bad

18
plugins/network-elements/nicira-nvp/src/main/java/com/cloud/network/nicira/ControlClusterStatus.java
View File

@ -29,6 +29,11 @@ public class ControlClusterStatus {
private Stats zoneStats;
private Stats routerStats;
private Stats securityProfileStats;
private ClusterRoleConfig[] configuredRoles;
public ClusterRoleConfig[] getConfiguredRoles() {
return configuredRoles;
}
public String getClusterStatus() {
return clusterStatus;
@ -84,4 +89,17 @@ public class ControlClusterStatus {
}
}
public class ClusterRoleConfig {
public String majorityVersion;
public String role;
public String getMajorityVersion(){
return majorityVersion;
}
public String getRole(){
return role;
}
}
}

23
plugins/network-elements/nicira-nvp/src/main/java/com/cloud/network/resource/NiciraNvpResource.java
View File

@ -37,6 +37,7 @@ import com.cloud.agent.api.StartupNiciraNvpCommand;
import com.cloud.host.Host;
import com.cloud.host.Host.Type;
import com.cloud.network.nicira.ControlClusterStatus;
import com.cloud.network.nicira.ControlClusterStatus.ClusterRoleConfig;
import com.cloud.network.nicira.DestinationNatRule;
import com.cloud.network.nicira.Match;
import com.cloud.network.nicira.NatRule;
@ -47,6 +48,7 @@ import com.cloud.network.utils.CommandRetryUtility;
import com.cloud.resource.ServerResource;
import com.cloud.utils.rest.CloudstackRESTException;
import com.cloud.utils.rest.HttpClientHelper;
import com.cloud.utils.nicira.nvp.plugin.NiciraNvpApiVersion;
public class NiciraNvpResource implements ServerResource {
@ -172,6 +174,7 @@ public class NiciraNvpResource implements ServerResource {
public PingCommand getCurrentStatus(final long id) {
try {
final ControlClusterStatus ccs = niciraNvpApi.getControlClusterStatus();
getApiProviderMajorityVersion(ccs);
if (!"stable".equals(ccs.getClusterStatus())) {
s_logger.error("ControlCluster state is not stable: " + ccs.getClusterStatus());
return null;
@ -183,6 +186,24 @@ public class NiciraNvpResource implements ServerResource {
return new PingCommand(Host.Type.L2Networking, id);
}
private void getApiProviderMajorityVersion(ControlClusterStatus ccs) {
ClusterRoleConfig[] configuredRoles = ccs.getConfiguredRoles();
if (configuredRoles != null){
String apiProviderMajorityVersion = searchApiProvider(configuredRoles);
NiciraNvpApiVersion.setNiciraApiVersion(apiProviderMajorityVersion);
NiciraNvpApiVersion.logNiciraApiVersion();
}
}
private String searchApiProvider(ClusterRoleConfig[] configuredRoles) {
for (int i = 0; i < configuredRoles.length; i++) {
if (configuredRoles[i].getRole() != null && configuredRoles[i].getRole().equals("api_provider")){
return configuredRoles[i].getMajorityVersion();
}
}
return null;
}
@Override
public Answer executeRequest(final Command cmd) {
final NiciraNvpRequestWrapper wrapper = NiciraNvpRequestWrapper.getInstance();
@ -332,4 +353,4 @@ public class NiciraNvpResource implements ServerResource {
// TODO Auto-generated method stub
}
}
}

2
pom.xml
View File

@ -92,7 +92,7 @@
<cs.servlet.version>2.5</cs.servlet.version>
<cs.jstl.version>1.2</cs.jstl.version>
<cs.selenium.server.version>1.0-20081010.060147</cs.selenium.server.version>
<cs.vmware.api.version>5.1</cs.vmware.api.version>
<cs.vmware.api.version>5.5</cs.vmware.api.version>
<org.springframework.version>3.2.12.RELEASE</org.springframework.version>
<cs.mockito.version>1.9.5</cs.mockito.version>
<cs.powermock.version>1.5.3</cs.powermock.version>

45
utils/src/main/java/com/cloud/utils/nicira/nvp/plugin/NiciraNvpApiVersion.java Executable file
View File

@ -0,0 +1,45 @@
//
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
//
package com.cloud.utils.nicira.nvp.plugin;
import org.apache.log4j.Logger;
import com.cloud.maint.Version;
public class NiciraNvpApiVersion {
private static final Logger s_logger = Logger.getLogger(NiciraNvpApiVersion.class);
private static String niciraApiVersion;
public static synchronized void setNiciraApiVersion(String apiVersion){
niciraApiVersion = apiVersion;
}
public static synchronized boolean isApiVersionLowerThan(String apiVersion){
if (niciraApiVersion == null) return false;
int compare = Version.compare(niciraApiVersion, apiVersion);
return (compare < 0);
}
public static synchronized void logNiciraApiVersion(){
s_logger.info("NSX API VERSION: " + ((niciraApiVersion != null) ? niciraApiVersion : " NOT PRESENT"));
}
}

19
vmware-base/src/com/cloud/hypervisor/vmware/mo/HostMO.java
View File

@ -46,6 +46,7 @@ import com.vmware.vim25.HostNetworkInfo;
import com.vmware.vim25.HostNetworkPolicy;
import com.vmware.vim25.HostNetworkSecurityPolicy;
import com.vmware.vim25.HostNetworkTrafficShapingPolicy;
import com.vmware.vim25.HostOpaqueNetworkInfo;
import com.vmware.vim25.HostPortGroup;
import com.vmware.vim25.HostPortGroupSpec;
import com.vmware.vim25.HostRuntimeInfo;
@ -62,7 +63,6 @@ import com.vmware.vim25.PropertySpec;
import com.vmware.vim25.TraversalSpec;
import com.vmware.vim25.VirtualMachineConfigSpec;
import com.vmware.vim25.VirtualNicManagerNetConfig;
import com.cloud.hypervisor.vmware.util.VmwareContext;
import com.cloud.hypervisor.vmware.util.VmwareHelper;
import com.cloud.utils.Pair;
@ -357,6 +357,23 @@ public class HostMO extends BaseMO implements VmwareHypervisorHost {
return null;
}
public boolean hasOpaqueNSXNetwork() throws Exception{
HostNetworkInfo netInfo = getHostNetworkInfo();
List<HostOpaqueNetworkInfo> opaqueNetworks = netInfo.getOpaqueNetwork();
if (opaqueNetworks != null){
for (HostOpaqueNetworkInfo opaqueNetwork : opaqueNetworks){
if (opaqueNetwork.getOpaqueNetworkId() != null && opaqueNetwork.getOpaqueNetworkId().equals("br-int")
&& opaqueNetwork.getOpaqueNetworkType() != null && opaqueNetwork.getOpaqueNetworkType().equals("nsx.network")){
return true;
}
}
throw new Exception("NSX API VERSION >= 4.2 BUT br-int (nsx.network) NOT FOUND");
}
else {
throw new Exception("NSX API VERSION >= 4.2 BUT br-int (nsx.network) NOT FOUND");
}
}
public boolean hasPortGroup(HostVirtualSwitch vSwitch, String portGroupName) throws Exception {
ManagedObjectReference morNetwork = getNetworkMor(portGroupName);
if (morNetwork != null)

45
vmware-base/src/com/cloud/hypervisor/vmware/mo/HypervisorHostHelper.java
View File

@ -43,6 +43,7 @@ import com.cloud.utils.cisco.n1kv.vsm.VsmCommand.SwitchPortMode;
import com.cloud.utils.db.GlobalLock;
import com.cloud.utils.exception.CloudRuntimeException;
import com.cloud.utils.net.NetUtils;
import com.cloud.utils.nicira.nvp.plugin.NiciraNvpApiVersion;
import com.vmware.vim25.AlreadyExistsFaultMsg;
import com.vmware.vim25.BoolPolicy;
import com.vmware.vim25.CustomFieldStringValue;
@ -1061,12 +1062,16 @@ public class HypervisorHostHelper {
boolean bWaitPortGroupReady = false;
if (broadcastDomainType == BroadcastDomainType.Lswitch) {
if (!hostMo.hasPortGroup(vSwitch, networkName)) {
createNvpPortGroup(hostMo, vSwitch, networkName, shapingPolicy);
//if NSX API VERSION >= 4.2, connect to br-int (nsx.network), do not create portgroup else previous behaviour
if (NiciraNvpApiVersion.isApiVersionLowerThan("4.2")){
//Previous behaviour
if (!hostMo.hasPortGroup(vSwitch, networkName)) {
createNvpPortGroup(hostMo, vSwitch, networkName, shapingPolicy);
bWaitPortGroupReady = true;
} else {
bWaitPortGroupReady = false;
bWaitPortGroupReady = true;
} else {
bWaitPortGroupReady = false;
}
}
} else {
if (!hostMo.hasPortGroup(vSwitch, networkName)) {
@ -1081,20 +1086,24 @@ public class HypervisorHostHelper {
}
}
ManagedObjectReference morNetwork;
if (bWaitPortGroupReady)
morNetwork = waitForNetworkReady(hostMo, networkName, timeOutMs);
else
morNetwork = hostMo.getNetworkMor(networkName);
if (morNetwork == null) {
String msg = "Failed to create guest network " + networkName;
s_logger.error(msg);
throw new Exception(msg);
}
ManagedObjectReference morNetwork = null;
if (createGCTag) {
NetworkMO networkMo = new NetworkMO(hostMo.getContext(), morNetwork);
networkMo.setCustomFieldValue(CustomFieldConstants.CLOUD_GC, "true");
if (broadcastDomainType != BroadcastDomainType.Lswitch ||
(broadcastDomainType == BroadcastDomainType.Lswitch && NiciraNvpApiVersion.isApiVersionLowerThan("4.2"))) {
if (bWaitPortGroupReady)
morNetwork = waitForNetworkReady(hostMo, networkName, timeOutMs);
else
morNetwork = hostMo.getNetworkMor(networkName);
if (morNetwork == null) {
String msg = "Failed to create guest network " + networkName;
s_logger.error(msg);
throw new Exception(msg);
}
if (createGCTag) {
NetworkMO networkMo = new NetworkMO(hostMo.getContext(), morNetwork);
networkMo.setCustomFieldValue(CustomFieldConstants.CLOUD_GC, "true");
}
}
if (syncPeerHosts) {

2
vmware-base/src/com/cloud/hypervisor/vmware/mo/VirtualDiskManagerMO.java
View File

@ -131,7 +131,7 @@ public class VirtualDiskManagerMO extends BaseMO {
public void moveVirtualDisk(String srcName, ManagedObjectReference morSrcDc, String destName, ManagedObjectReference morDestDc, boolean force) throws Exception {
ManagedObjectReference morTask = _context.getService().moveVirtualDiskTask(_mor, srcName, morSrcDc, destName, morDestDc, force);
ManagedObjectReference morTask = _context.getService().moveVirtualDiskTask(_mor, srcName, morSrcDc, destName, morDestDc, force, null);
boolean result = _context.getVimClient().waitForTask(morTask);
if (!result)

47
vmware-base/src/com/cloud/hypervisor/vmware/util/VmwareHelper.java
View File

@ -56,12 +56,12 @@ import com.vmware.vim25.VirtualE1000;
import com.vmware.vim25.VirtualEthernetCard;
import com.vmware.vim25.VirtualEthernetCardDistributedVirtualPortBackingInfo;
import com.vmware.vim25.VirtualEthernetCardNetworkBackingInfo;
import com.vmware.vim25.VirtualEthernetCardOpaqueNetworkBackingInfo;
import com.vmware.vim25.VirtualMachineConfigSpec;
import com.vmware.vim25.VirtualMachineSnapshotTree;
import com.vmware.vim25.VirtualPCNet32;
import com.vmware.vim25.VirtualVmxnet2;
import com.vmware.vim25.VirtualVmxnet3;
import com.cloud.hypervisor.vmware.mo.HostMO;
import com.cloud.hypervisor.vmware.mo.LicenseAssignmentManagerMO;
import com.cloud.hypervisor.vmware.mo.VirtualEthernetCardType;
@ -79,6 +79,51 @@ public class VmwareHelper {
return deviceNumber == 7;
}
public static VirtualDevice prepareNicOpaque(VirtualMachineMO vmMo, VirtualEthernetCardType deviceType, String portGroupName,
String macAddress, int deviceNumber, int contextNumber, boolean conntected, boolean connectOnStart) throws Exception {
assert(vmMo.getRunningHost().hasOpaqueNSXNetwork());
VirtualEthernetCard nic;
switch (deviceType) {
case E1000:
nic = new VirtualE1000();
break;
case PCNet32:
nic = new VirtualPCNet32();
break;
case Vmxnet2:
nic = new VirtualVmxnet2();
break;
case Vmxnet3:
nic = new VirtualVmxnet3();
break;
default:
nic = new VirtualE1000();
}
VirtualEthernetCardOpaqueNetworkBackingInfo nicBacking = new VirtualEthernetCardOpaqueNetworkBackingInfo();
nicBacking.setOpaqueNetworkId("br-int");
nicBacking.setOpaqueNetworkType("nsx.network");
nic.setBacking(nicBacking);
VirtualDeviceConnectInfo connectInfo = new VirtualDeviceConnectInfo();
connectInfo.setAllowGuestControl(true);
connectInfo.setConnected(conntected);
connectInfo.setStartConnected(connectOnStart);
nic.setAddressType("Manual");
nic.setConnectable(connectInfo);
nic.setMacAddress(macAddress);
nic.setUnitNumber(deviceNumber);
nic.setKey(-contextNumber);
return nic;
}
public static VirtualDevice prepareNicDevice(VirtualMachineMO vmMo, ManagedObjectReference morNetwork, VirtualEthernetCardType deviceType, String portGroupName,
String macAddress, int deviceNumber, int contextNumber, boolean conntected, boolean connectOnStart) throws Exception {