NSX integration (#7919)

* NSX integration - skeletal code

* Fix module not loading on startup

* add upgrade path and daos
\n add nsx controller command

* add support for adding and listing nsx provider to a zone

* add license

* add default VPC offering and update upgrade path

* add global setting to enable nsx plugin

* add delete nsx controller operation

* add nsxresource

* add NSX resource , api client, create tier1 gw

* update db

* update response and add license

* Add support to create and delete nsx tier-1 gateway

* add license

* cleanup and add skeletal code for network creation

* add create/delete segment and UI integration

* add license

* address code smells - part 1

* fix test / build failure

* NSX integration - skeletal code

* Fix module not loading on startup

* add upgrade path and daos
\n add nsx controller command

* add support for adding and listing nsx provider to a zone

* add license

* add default VPC offering and update upgrade path

* add global setting to enable nsx plugin

* add delete nsx controller operation

* add nsxresource

* add NSX resource , api client, create tier1 gw

* update db

* update response and add license

* Add support to create and delete nsx tier-1 gateway

* add license

* cleanup and add skeletal code for network creation

* add create/delete segment and UI integration

* add license

* address code smells - part 1

* fix test / build failure

* add ui changes + update nsx_provider table transport zones + use NSX broadcast domain for add nics to router

* ui: fix password field, and backend changes

* add route advertisement

* update offering

* update offering

* add sleep before deletion of vpc / tier g/w for ports to be removed

* move creation of segments to design phase

* change provider to VPC router for Dhcp & dns service in an nsx offering

* Add public nic for NSX

* reserve first IP (after g/w) of subnet for router nic - NSX

* revert reserving 1st IP in vpc segments

* [NSX] Create a DHCP relay and add it to a VPC tier segment (#107)

* Create DHCP relay command and execute request

* In progress integrate with networking

* Create DHCP relay config on the network VR allocation

* Revert domain router dao changes

* Create DHCP relay con VR nic plug to NSX network

* Link DHCP relay config to segment after creation

* [NSX] Cleanup DHCP Relay config on segment deletion (#108)

* Cleanup DHCP Relay config on segment deletion

* update segment & relay name generators and call delete dhcprelay after deletion of segment

* address comment

* [NSX] Fix DHCP relay config deletion was missing zone name (#8068)

* [NSX] Refactor API wrapper operations (#8059)

* [NSX] Refactor API wrapper operations

* Big refactor

* Address review comment

* change network cidr to cidr to prevent NPE

* add domain and zone names to the various networks - vpc & tier

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* Nsx unit tests (#8090)

* Add tests

* add test for NsxGuestNetworkGuru

* add unit tests for NsxResource

* add unti tests for NsxElement

* cleanup

* [NSX] Refactor API wrapper operations

* update tests

* update tests - add nsxProviderServiceImpl test

* add unit test - NsxServiceImpl

* add license

* Big refactor

* Address review comment

* change network cidr to cidr to prevent NPE

* add domain and zone names to the various networks - vpc & tier

* fix tests

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* modify NSX resource naming convention (#8095)

* modify NSX resource naming convention

* remove unused imports

* add a setup phase between desgin and implementation of a network for intermediary steps

* add method to all classes

* NSX: Refactor Network & VPC offering (#8110)

* [NSX] Refactor API wrapper operations

* Network offering changes for NSX

* fix services and provider combination

* address comments: rename param

* update nsx_mode parameter

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* fix test

* [NSX] Allow NSX isolated networks (#8132)

* Add network offerings for NSX on isolated networks

* Fix offerings creation

* In progress NSX isolated network

* Fixes

* Fix NIC allocation to router

* NSX: Add Step for Adding Public traffic network for NSX During zone creation (#8126)

* NSX: Add Step for Adding Public traffic network for NSX

* address comments and cleanup

* address comment

* remove indent

* NSX: Create and Delete static NAT & Port forward  rules (#8131)

* NSX: Create and delete NSX Static Nat rules

* fix issues with static nat

* add static nat

* Support to add and delete Port forward rules

* add license

* fix adding multiple pf rules

* cleanup

* fix lint check

* fix smoke tests

* fix smoke tests

* Nsx add lb rule (#8161)

* NSX: Create and delete NSX Static Nat rules

* fix issues with static nat

* add static nat

* Support to add and delete Port forward rules

* add license

* fix adding multiple pf rules

* cleanup

* NSX: Add support to create and delete Load balancer rules

* fix deletion of lb rules

* add header file and update protocol detail

* build failure fix

* [NSX] Add SNAT support (#8100)

* In progress add source NAT

* Fix after merge

* Fix tests

* Fix NPE on isolated network deletion

* Reserve source NAT IP when its not passed for NSX VPC

* Create source NAT rule on VR NIC allocation

* Fix update VPC and remove VPC to update and remove SNAT rule

* Fix packaging

* Address review comment

* Fix build

* fix build - unused import

* Add defensive checks

* Add missing design to NSX public guru

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* NSX: Fix VR public NIC allocation (#8166)

* NSX: fix LB member addition and deletion and add defensive checks (#8167)

* Fix public NIC NPE on broadcast URI

* NSX: Router Public nic to get IP from systemVM Ip range (#8172)

* NSX: Router Public nic to get IP from systemVM Ip range

* Fix VR IP address and setSourceNatIp command

* NSX: hide systemVM reserved IP range SourceNAT

* fix test

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* fix test failure

* test failure fix

* [NSX] Fix update source NAT IP (#8176)

* [NSX] Fix update source NAT IP

* Fix startup

* Fix API result

* NSX - add LB route Advertizement (#8192)

* [NSX] Add ACL types support (#8224)

* NSX: Create segment group on segment creation

* Add unit tests

* Remove group for segment before removing segment

* Create Distributed Firewall rules

* Remove distributed firewall policy on segment deletion

* Fix policy rule ID and add more unit tests

* Fix DROP action rules and transform tests

* Add new ACL rules

* Fixes

* associate security policies with groups and not to DFW and add deletion of rules

* Fix name convention

---------

Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>

* NSX: Fix creation of VPCs (#8320)

* Fix ACL rules creation (#8323)

* [NSX] Fix database views (#8325)

* NSX: Add CKS Support & Firewall rules for Isolated Networks (#8189)

* NSX: Add ALL LB IP to the list of route advertisements in tier1

* NSX: Support Source NAT on NSX Isolated networks

* NSX: Cks Support

* NSX: Create segment group on segment creation

* Add unit tests

* Remove group for segment before removing segment

* Create Distributed Firewall rules

* Remove distributed firewall policy on segment deletion

* Fix policy rule ID and add more unit tests

* Add support for routed NSX Isolated networks \n and non RFC 1918 compliant IPs

* Add support for routed NSX Isolated networks \n and non RFC 1918 compliant IPs

* Add Firewall rules

* build failure - fix unit test

* fix npes

* Add support to delete firewall rules

* update nsx cks offering

* add license

* update order of ports in PF & FW rules

* fix filter for getting transport zones

* CKS support changed - MTU updated, etc

* add LB for CKS on VPC

* address comments

* adapt upstream cks logic for vpc

* rever mtu hack

* update UI changes as per upstream fix

* change display test for CKS n/w offerings for isolated and VPC tiers

* add extra line for linter

* address comment

* revert list change

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* fix ui build failure

* [NSX] Address SonarCloud Bugs (#8341)

* [NSX] Address SonarCloud Bugs

* Fix NSX API connection issues

* NSX: Add unit tests to increase coverage (#8355)

* NSX: Add unit tests

* cleanup unused imports

* add more unit tests

* add tests for publicnsxnetworkguru

* add license

* fix build failures

* address sonar comment

* fix security hotspots

* NSX: Add more unit tests (#8381)

* NSX : Unit tests

* remove unused imports

* remove unused import causing build failure

* fix build failures due to unused imports

* fix build failure

* fix test assertion

* remove unused imports

* remove unused import

* Nsx UI zone bug (#8398)

* NSX: Attempt to fix NSX Zone creation bug for public networks

* fix zone wizard public traffic issue

* add proper filtering of offerings based on VPC nsx mode

* clean up console logs

* NSX: Fix code smells and reported bugs (#8409)

* NSX: Fix code smells and reported bugs

* fox override issue

* remove unused imports

* fix test

* refactor code to reduce complexity

* add lisence

* cleanup

* fix build failure

* fix build failure

* address comments

* test - add config to ignore certain files from test coverage

* test exclusion of classes from test cov

* rever pom changes

* [NSX] Add more unit tests (#8431)

* [NSX] Add more unit tests

* More tests

* Fix build errors

* NSX: Prevent creation of L2 and Shared networks for NSX (#8463)

* NSX: Prevent creation of L2 and Shared networks for NSX

* add checks to backend to prevent creation of l2 and shared networks in nsx zones and filter only nsx offerings when creating isolated networks

* cleanup

* NSX: Fix code smells (#8436)

* NSX: Fix code smells

* Add changes to service creation logic

* CKS: Add action to during firewall rule creation (#8498)

* NSX,UI: Deduplicate network list when creating kubernetes clusters (#8513)

* NSX: Make LB service selectable in network offering (#8512)

* NSX: Make LB service selectable in network offering

* fix label

* address comments

* address comments

* NSX: Add appropriate error message when icmp type is set to -1 for NSX (#8504)

* NSX: Add appropriate error message when icmp type is set to -1 for NSX

* address comments

* update text

* fix test

* fix test - build failure

* fix test - build failure

* NSX: Cleanup NSX resources during k8s cluster cleanup (#8528)

* fix test failure

* NSX: Improve segment deletion process (#8538)

* NSX: Add passive monitor for NSX LB to test whether a server is available (#8533)

* NSX: Add passive monitor for NSX LB to test whether a server is available

* Add active monitors too

* fix build failure

* NSX: Add check for ICMP code / type for NSX zones (#8542)

* NSX: Fix Routed Mode for Isolated and VPC networks (#8534)

* NSX: Fix Routed Mode for Isolated and VPC networks

* NSX: Fix Routed mode - add checks for ports added for FW rules

* clean up code

* fix build failure

* NSX: Add retry logic with sleep to delete segments (#8554)

* NSX: Add retry logic with sleep to delete segments

* add logs

* NSX: Fix custom ACL check (#2)

* NSX: Fix custom ACL check

* NSX: Fix custom ACL check

* Nsx vpc routed mode (#5)

* NSX: Fix VPC routed mode

* NSX: VPC route mode

* remove unnecessary changes

* Nsx: Support internal LB (#4)

* NSX: Support internal LB service in NSX

* add lb removal logic

* Fix UI issue hiding internal LB tab

* Refactor method name

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* NSX: Improve NSX resource cleanup process (#3)

* Fix unit test

* NSX: Add SourceNAT service to the default Routed offering for VPC (#13)

* Fix VPC restart with cleanup (#12)

* NSX: Fix ACL rule removal on replacement and fix rule order (#11)

* NSX: fix smoke test failure for ACLs (#9)

* Fix unit tests

* Fix NSX plugin pom XML

* NSX: Add support to re-order ACL rules (NSX FW rules) (#14)

* [WIP] NSX: Add support to re-order ACL rules (NSX FW rules)

* fix reordering of acl rules on all networks that it is associated to

* clean up and attempt test fix

* Fix tests

* Remove unused import

* tweak reorder logic

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

* Fix zone creation issue for internal load balancer

* Fix

* Fix unit test

* fix logger

* fix logger

* fix logger

* NSX: Fix VPC form to ignore source NAT IP when creating VPCs and fix label

* Move SQL changes to the newest schema file

* NSX: Last Fixes

* Fix build

---------

Co-authored-by: nvazquez <nicovazquez90@gmail.com>

api/src/main/java/com/cloud/agent/api/to/NicTO.java

@@ -32,6 +32,9 @@ public class NicTO extends NetworkTO {
     Map<NetworkOffering.Detail, String> details;
     boolean dpdkEnabled;
     Integer mtu;
+    Long networkId;
+
+    String networkSegmentName;
 
     public NicTO() {
         super();
@@ -127,4 +130,20 @@ public class NicTO extends NetworkTO {
     public void setMtu(Integer mtu) {
         this.mtu = mtu;
     }
+
+    public Long getNetworkId() {
+        return networkId;
+    }
+
+    public void setNetworkId(Long networkId) {
+        this.networkId = networkId;
+    }
+
+    public String getNetworkSegmentName() {
+        return networkSegmentName;
+    }
+
+    public void setNetworkSegmentName(String networkSegmentName) {
+        this.networkSegmentName = networkSegmentName;
+    }
 }

api/src/main/java/com/cloud/agent/api/to/VirtualMachineTO.java

@@ -82,6 +82,7 @@ public class VirtualMachineTO {
 
     Map<String, String> guestOsDetails = new HashMap<String, String>();
     Map<String, String> extraConfig = new HashMap<>();
+    Map<Long, String> networkIdToNetworkNameMap = new HashMap<>();
     DeployAsIsInfoTO deployAsIsInfo;
 
     public VirtualMachineTO(long id, String instanceName, VirtualMachine.Type type, int cpus, Integer speed, long minRam, long maxRam, BootloaderType bootloader,
@@ -392,6 +393,14 @@ public class VirtualMachineTO {
         return extraConfig;
     }
 
+    public Map<Long, String> getNetworkIdToNetworkNameMap() {
+        return networkIdToNetworkNameMap;
+    }
+
+    public void setNetworkIdToNetworkNameMap(Map<Long, String> networkIdToNetworkNameMap) {
+        this.networkIdToNetworkNameMap = networkIdToNetworkNameMap;
+    }
+
     public String getBootType() {
         return bootType;
     }

api/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterHelper.java

@@ -22,4 +22,5 @@ import org.apache.cloudstack.acl.ControlledEntity;
 public interface KubernetesClusterHelper extends Adapter {
 
     ControlledEntity findByUuid(String uuid);
+    ControlledEntity findByVmId(long vmId);
 }

api/src/main/java/com/cloud/network/IpAddress.java

@@ -97,4 +97,6 @@ public interface IpAddress extends ControlledEntity, Identity, InternalIdentity,
 
     void setRuleState(State ruleState);
 
+    boolean isForSystemVms();
+
 }

api/src/main/java/com/cloud/network/Network.java

@@ -205,6 +205,8 @@ public interface Network extends ControlledEntity, StateObject<Network.State>, I
         //Add Tungsten Fabric provider
         public static final Provider Tungsten = new Provider("Tungsten", false);
 
+        public static final Provider Nsx = new Provider("Nsx", false);
+
         private final String name;
         private final boolean isExternal;
 
@@ -427,6 +429,8 @@ public interface Network extends ControlledEntity, StateObject<Network.State>, I
 
     long getDataCenterId();
 
+    long getAccountId();
+
     long getNetworkOfferingId();
 
     @Override

api/src/main/java/com/cloud/network/NetworkService.java

@@ -19,6 +19,7 @@ package com.cloud.network;
 import java.util.List;
 import java.util.Map;
 
+import com.cloud.dc.DataCenter;
 import org.apache.cloudstack.api.command.admin.address.ReleasePodIpCmdByAdmin;
 import org.apache.cloudstack.api.command.admin.network.DedicateGuestVlanRangeCmd;
 import org.apache.cloudstack.api.command.admin.network.ListDedicatedGuestVlanRangesCmd;
@@ -55,6 +56,7 @@ import com.cloud.utils.Pair;
 import com.cloud.utils.exception.CloudRuntimeException;
 import com.cloud.vm.Nic;
 import com.cloud.vm.NicSecondaryIp;
+import org.apache.cloudstack.network.element.InternalLoadBalancerElementService;
 
 /**
  * The NetworkService interface is the "public" api to entities that make requests to the orchestration engine
@@ -87,6 +89,8 @@ public interface NetworkService {
 
     IpAddress reserveIpAddress(Account account, Boolean displayIp, Long ipAddressId) throws ResourceAllocationException;
 
+    IpAddress reserveIpAddressWithVlanDetail(Account account, DataCenter zone, Boolean displayIp, String vlanDetailKey) throws ResourceAllocationException;
+
     boolean releaseReservedIpAddress(long ipAddressId) throws InsufficientAddressCapacityException;
 
     boolean releaseIpAddress(long ipAddressId) throws InsufficientAddressCapacityException;
@@ -254,4 +258,9 @@ public interface NetworkService {
     PublicIpQuarantine updatePublicIpAddressInQuarantine(UpdateQuarantinedIpCmd cmd);
 
     void removePublicIpAddressFromQuarantine(RemoveQuarantinedIpCmd cmd);
+
+    InternalLoadBalancerElementService getInternalLoadBalancerElementByType(VirtualRouterProvider.Type type);
+    InternalLoadBalancerElementService getInternalLoadBalancerElementByNetworkServiceProviderId(long networkProviderId);
+    InternalLoadBalancerElementService getInternalLoadBalancerElementById(long providerId);
+    List<InternalLoadBalancerElementService> getInternalLoadBalancerElements();
 }

api/src/main/java/com/cloud/network/Networks.java

@@ -128,7 +128,8 @@ public class Networks {
         },
         UnDecided(null, null),
         OpenDaylight("opendaylight", String.class),
-        TUNGSTEN("tf", String.class);
+        TUNGSTEN("tf", String.class),
+        NSX("nsx", String.class);
 
         private final String scheme;
         private final Class<?> type;

api/src/main/java/com/cloud/network/VirtualRouterProvider.java

@@ -21,7 +21,7 @@ import org.apache.cloudstack.api.InternalIdentity;
 
 public interface VirtualRouterProvider extends InternalIdentity, Identity {
     public enum Type {
-        VirtualRouter, ElasticLoadBalancerVm, VPCVirtualRouter, InternalLbVm, NetScalerVm
+        VirtualRouter, ElasticLoadBalancerVm, VPCVirtualRouter, InternalLbVm, NetScalerVm, Nsx
     }
 
     public Type getType();

api/src/main/java/com/cloud/network/element/NetworkACLServiceProvider.java

@@ -21,6 +21,7 @@ import java.util.List;
 import com.cloud.exception.ResourceUnavailableException;
 import com.cloud.network.Network;
 import com.cloud.network.vpc.NetworkACLItem;
+import com.cloud.network.vpc.Vpc;
 
 public interface NetworkACLServiceProvider extends NetworkElement {
 
@@ -32,4 +33,6 @@ public interface NetworkACLServiceProvider extends NetworkElement {
      */
     boolean applyNetworkACLs(Network config, List<? extends NetworkACLItem> rules) throws ResourceUnavailableException;
 
+    boolean reorderAclRules(Vpc vpc, List<? extends Network> networks, List<? extends NetworkACLItem> networkACLItems);
+
 }

api/src/main/java/com/cloud/network/guru/NetworkGuru.java

@@ -79,20 +79,24 @@ public interface NetworkGuru extends Adapter {
      * be used to make determination can be isolation methods, services
      * provided on the guest network and the service provider that's on the
      * guest network.
-     *
+     * <p>
      * If a network is already fully substantiated with the necessary resources
      * during this design phase, then the state should be set to Setup.  If
      * the resources are not allocated at this point, the state should be set
      * to Allocated.
      *
-     * @param offering network offering that contains the package of services
+     * @param offering      network offering that contains the package of services
-     *                 the end user intends to use on that network.
+     *                      the end user intends to use on that network.
-     * @param plan where is this network being deployed.
+     * @param plan          where is this network being deployed.
      * @param userSpecified user specified parameters for this network.
-     * @param owner owner of this network.
+     * @param name
+     * @param vpcId
+     * @param owner         owner of this network.
      * @return Network
      */
-    Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, Account owner);
+    Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, String name, Long vpcId, Account owner);
+
+    void setup(Network network, long networkId);
 
     /**
      * For guest networks that are in Allocated state after the design stage,

api/src/main/java/com/cloud/network/nsx/NsxProvider.java

@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.network.nsx;
+
+import org.apache.cloudstack.api.Identity;
+import org.apache.cloudstack.api.InternalIdentity;
+
+public interface NsxProvider extends InternalIdentity, Identity {
+    String getHostname();
+
+    String getPort();
+    String getProviderName();
+    String getUsername();
+    long getZoneId();
+
+    String getTier0Gateway();
+    String getEdgeCluster();
+
+    String getTransportZone();
+}

api/src/main/java/com/cloud/network/nsx/NsxService.java

@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.network.nsx;
+
+import com.cloud.network.IpAddress;
+import com.cloud.network.vpc.Vpc;
+
+public interface NsxService {
+
+    boolean createVpcNetwork(Long zoneId, long accountId, long domainId, Long vpcId, String vpcName, boolean sourceNatEnabled);
+    boolean updateVpcSourceNatIp(Vpc vpc, IpAddress address);
+}

api/src/main/java/com/cloud/network/vpc/VpcOffering.java

@@ -29,6 +29,8 @@ public interface VpcOffering extends InternalIdentity, Identity {
     public static final String defaultVPCOfferingName = "Default VPC offering";
     public static final String defaultVPCNSOfferingName = "Default VPC  offering with Netscaler";
     public static final String redundantVPCOfferingName = "Redundant VPC offering";
+    public static final String DEFAULT_VPC_NAT_NSX_OFFERING_NAME = "VPC offering with NSX - NAT Mode";
+    public static final String DEFAULT_VPC_ROUTE_NSX_OFFERING_NAME = "VPC offering with NSX - Route Mode";
 
     /**
      *
@@ -53,6 +55,10 @@ public interface VpcOffering extends InternalIdentity, Identity {
      */
     boolean isDefault();
 
+    boolean isForNsx();
+
+    String getNsxMode();
+
     /**
      * @return service offering id used by VPC virtual router
      */

api/src/main/java/com/cloud/network/vpc/VpcProvisioningService.java

@@ -36,7 +36,8 @@ public interface VpcProvisioningService {
     VpcOffering createVpcOffering(String name, String displayText, List<String> supportedServices,
                                   Map<String, List<String>> serviceProviders,
                                   Map serviceCapabilitystList, NetUtils.InternetProtocol internetProtocol,
-                                  Long serviceOfferingId, List<Long> domainIds, List<Long> zoneIds, VpcOffering.State state);
+                                  Long serviceOfferingId, Boolean forNsx, String mode,
+                                  List<Long> domainIds, List<Long> zoneIds, VpcOffering.State state);
 
     Pair<List<? extends VpcOffering>,Integer> listVpcOfferings(ListVPCOfferingsCmd cmd);
 

api/src/main/java/com/cloud/offering/NetworkOffering.java

@@ -43,6 +43,11 @@ public interface NetworkOffering extends InfrastructureEntity, InternalIdentity,
         InternalLbProvider, PublicLbProvider, servicepackageuuid, servicepackagedescription, PromiscuousMode, MacAddressChanges, ForgedTransmits, MacLearning, RelatedNetworkOffering, domainid, zoneid, pvlanType, internetProtocol
     }
 
+    public enum NsxMode {
+        NATTED,
+        ROUTED
+    }
+
     public final static String SystemPublicNetwork = "System-Public-Network";
     public final static String SystemControlNetwork = "System-Control-Network";
     public final static String SystemManagementNetwork = "System-Management-Network";
@@ -52,6 +57,11 @@ public interface NetworkOffering extends InfrastructureEntity, InternalIdentity,
 
     public final static String DefaultSharedNetworkOfferingWithSGService = "DefaultSharedNetworkOfferingWithSGService";
     public static final String DEFAULT_TUNGSTEN_SHARED_NETWORK_OFFERING_WITH_SGSERVICE = "DefaultTungstenSharedNetworkOfferingWithSGService";
+    public static final String DEFAULT_NAT_NSX_OFFERING_FOR_VPC = "DefaultNATNSXNetworkOfferingForVpc";
+    public static final String DEFAULT_NAT_NSX_OFFERING_FOR_VPC_WITH_ILB = "DefaultNATNSXNetworkOfferingForVpcWithInternalLB";
+    public static final String DEFAULT_ROUTED_NSX_OFFERING_FOR_VPC = "DefaultRoutedNSXNetworkOfferingForVpc";
+    public static final String DEFAULT_NAT_NSX_OFFERING = "DefaultNATNSXNetworkOffering";
+    public static final String DEFAULT_ROUTED_NSX_OFFERING = "DefaultRoutedNSXNetworkOffering";
     public final static String QuickCloudNoServices = "QuickCloudNoServices";
     public final static String DefaultIsolatedNetworkOfferingWithSourceNatService = "DefaultIsolatedNetworkOfferingWithSourceNatService";
     public final static String OvsIsolatedNetworkOfferingWithSourceNatService = "OvsIsolatedNetworkOfferingWithSourceNatService";
@@ -90,6 +100,10 @@ public interface NetworkOffering extends InfrastructureEntity, InternalIdentity,
 
     boolean isForTungsten();
 
+    boolean isForNsx();
+
+    String getNsxMode();
+
     TrafficType getTrafficType();
 
     boolean isSpecifyVlan();

api/src/main/java/org/apache/cloudstack/api/ApiConstants.java

@@ -303,6 +303,8 @@ public class ApiConstants {
     public static final String MIGRATIONS = "migrations";
     public static final String MEMORY = "memory";
     public static final String MODE = "mode";
+    public static final String NSX_MODE = "nsxmode";
+    public static final String NSX_ENABLED = "isnsxenabled";
     public static final String NAME = "name";
     public static final String METHOD_NAME = "methodname";
     public static final String NETWORK_DOMAIN = "networkdomain";
@@ -704,6 +706,12 @@ public class ApiConstants {
     public static final String VSWITCH_TYPE_PUBLIC_TRAFFIC = "publicvswitchtype";
     public static final String VSWITCH_NAME_GUEST_TRAFFIC = "guestvswitchname";
     public static final String VSWITCH_NAME_PUBLIC_TRAFFIC = "publicvswitchname";
+
+    // NSX
+    public static final String EDGE_CLUSTER = "edgecluster";
+    public static final String TIER0_GATEWAY = "tier0gateway";
+
+    public static final String TRANSPORT_ZONE = "transportzone";
     // Tungsten-Fabric
     public static final String TUNGSTEN_VIRTUAL_ROUTER_UUID = "tungstenvirtualrouteruuid";
     public static final String TUNGSTEN_PROVIDER_HOSTNAME = "tungstenproviderhostname";
@@ -825,6 +833,9 @@ public class ApiConstants {
     public static final String FORCE_ENCAP = "forceencap";
     public static final String SPLIT_CONNECTIONS = "splitconnections";
     public static final String FOR_VPC = "forvpc";
+    public static final String FOR_NSX = "fornsx";
+    public static final String NSX_SUPPORT_LB = "nsxsupportlb";
+    public static final String NSX_SUPPORTS_INTERNAL_LB = "nsxsupportsinternallb";
     public static final String FOR_TUNGSTEN = "fortungsten";
     public static final String SHRINK_OK = "shrinkok";
     public static final String NICIRA_NVP_DEVICE_ID = "nvpdeviceid";
@@ -834,6 +845,11 @@ public class ApiConstants {
     public static final String NICIRA_NVP_L2_GATEWAYSERVICE_UUID = "l2gatewayserviceuuid";
     public static final String NSX_LOGICAL_SWITCH = "nsxlogicalswitch";
     public static final String NSX_LOGICAL_SWITCH_PORT = "nsxlogicalswitchport";
+    public static final String NSX_PROVIDER_UUID = "nsxprovideruuid";
+    public static final String NSX_PROVIDER_HOSTNAME = "nsxproviderhostname";
+
+    public static final String NSX_PROVIDER_PORT = "nsxproviderport";
+    public static final String NSX_CONTROLLER_ID = "nsxcontrollerid";
     public static final String S3_ACCESS_KEY = "accesskey";
     public static final String S3_SECRET_KEY = "secretkey";
     public static final String S3_END_POINT = "endpoint";
@@ -958,6 +974,7 @@ public class ApiConstants {
     public static final String SUPPORTS_REGION_LEVEL_VPC = "supportsregionLevelvpc";
     public static final String SUPPORTS_STRECHED_L2_SUBNET = "supportsstrechedl2subnet";
     public static final String SUPPORTS_PUBLIC_ACCESS = "supportspublicaccess";
+    public static final String SUPPORTS_INTERNAL_LB = "supportsinternallb";
     public static final String SUPPORTS_VM_AUTOSCALING = "supportsvmautoscaling";
     public static final String REGION_LEVEL_VPC = "regionlevelvpc";
     public static final String STRECHED_L2_SUBNET = "strechedl2subnet";
@@ -1076,14 +1093,13 @@ public class ApiConstants {
     public static final String SOURCE_NAT_IP = "sourcenatipaddress";
     public static final String SOURCE_NAT_IP_ID = "sourcenatipaddressid";
     public static final String HAS_RULES = "hasrules";
+    public static final String NSX_DETAIL_KEY = "forNsx";
     public static final String DISK_PATH = "diskpath";
     public static final String IMPORT_SOURCE = "importsource";
     public static final String TEMP_PATH = "temppath";
     public static final String OBJECT_STORAGE = "objectstore";
-
     public static final String HEURISTIC_RULE = "heuristicrule";
     public static final String HEURISTIC_TYPE_VALID_OPTIONS = "Valid options are: ISO, SNAPSHOT, TEMPLATE and VOLUME.";
-
     public static final String MANAGEMENT = "management";
     public static final String IS_VNF = "isvnf";
     public static final String VNF_NICS = "vnfnics";

api/src/main/java/org/apache/cloudstack/api/BaseCmd.java

@@ -38,7 +38,6 @@ import org.apache.cloudstack.affinity.AffinityGroupService;
 import org.apache.cloudstack.alert.AlertService;
 import org.apache.cloudstack.annotation.AnnotationService;
 import org.apache.cloudstack.context.CallContext;
-import org.apache.cloudstack.network.element.InternalLoadBalancerElementService;
 import org.apache.cloudstack.network.lb.ApplicationLoadBalancerService;
 import org.apache.cloudstack.network.lb.InternalLoadBalancerVMService;
 import org.apache.cloudstack.query.QueryService;
@@ -201,8 +200,6 @@ public abstract class BaseCmd {
     @Inject
     public AffinityGroupService _affinityGroupService;
     @Inject
-    public InternalLoadBalancerElementService _internalLbElementSvc;
-    @Inject
     public InternalLoadBalancerVMService _internalLbSvc;
     @Inject
     public NetworkModel _ntwkModel;

api/src/main/java/org/apache/cloudstack/api/command/admin/internallb/ConfigureInternalLoadBalancerElementCmd.java

@@ -17,11 +17,6 @@
 
 package org.apache.cloudstack.api.command.admin.internallb;
 
-import java.util.List;
-
-import javax.inject.Inject;
-
-
 import org.apache.cloudstack.api.APICommand;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.api.ApiErrorCode;
@@ -47,9 +42,6 @@ import com.cloud.user.Account;
             responseHasSensitiveInfo = false)
 public class ConfigureInternalLoadBalancerElementCmd extends BaseAsyncCmd {
 
-    @Inject
-    private List<InternalLoadBalancerElementService> _service;
-
     /////////////////////////////////////////////////////
     //////////////// API parameters /////////////////////
     /////////////////////////////////////////////////////
@@ -98,7 +90,8 @@ public class ConfigureInternalLoadBalancerElementCmd extends BaseAsyncCmd {
     @Override
     public void execute() throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
         CallContext.current().setEventDetails("Internal load balancer element: " + id);
-        VirtualRouterProvider result = _service.get(0).configureInternalLoadBalancerElement(getId(), getEnabled());
+        InternalLoadBalancerElementService service = _networkService.getInternalLoadBalancerElementById(id);
+        VirtualRouterProvider result = service.configureInternalLoadBalancerElement(getId(), getEnabled());
         if (result != null) {
             InternalLoadBalancerElementResponse routerResponse = _responseGenerator.createInternalLbElementResponse(result);
             routerResponse.setResponseName(getCommandName());

api/src/main/java/org/apache/cloudstack/api/command/admin/internallb/CreateInternalLoadBalancerElementCmd.java

@@ -16,11 +16,6 @@
 // under the License.
 package org.apache.cloudstack.api.command.admin.internallb;
 
-import java.util.List;
-
-import javax.inject.Inject;
-
-
 import org.apache.cloudstack.api.APICommand;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.api.ApiErrorCode;
@@ -45,9 +40,6 @@ import com.cloud.user.Account;
             responseHasSensitiveInfo = false)
 public class CreateInternalLoadBalancerElementCmd extends BaseAsyncCreateCmd {
 
-    @Inject
-    private List<InternalLoadBalancerElementService> _service;
-
     /////////////////////////////////////////////////////
     //////////////// API parameters /////////////////////
     /////////////////////////////////////////////////////
@@ -83,7 +75,8 @@ public class CreateInternalLoadBalancerElementCmd extends BaseAsyncCreateCmd {
     @Override
     public void execute() {
         CallContext.current().setEventDetails("Virtual router element Id: " + getEntityId());
-        VirtualRouterProvider result = _service.get(0).getInternalLoadBalancerElement(getEntityId());
+        InternalLoadBalancerElementService service = _networkService.getInternalLoadBalancerElementByNetworkServiceProviderId(getNspId());
+        VirtualRouterProvider result = service.getInternalLoadBalancerElement(getEntityId());
         if (result != null) {
             InternalLoadBalancerElementResponse response = _responseGenerator.createInternalLbElementResponse(result);
             response.setResponseName(getCommandName());
@@ -95,7 +88,8 @@ public class CreateInternalLoadBalancerElementCmd extends BaseAsyncCreateCmd {
 
     @Override
     public void create() throws ResourceAllocationException {
-        VirtualRouterProvider result = _service.get(0).addInternalLoadBalancerElement(getNspId());
+        InternalLoadBalancerElementService service = _networkService.getInternalLoadBalancerElementByNetworkServiceProviderId(getNspId());
+        VirtualRouterProvider result = service.addInternalLoadBalancerElement(getNspId());
         if (result != null) {
             setEntityId(result.getId());
             setEntityUuid(result.getUuid());

api/src/main/java/org/apache/cloudstack/api/command/admin/internallb/ListInternalLoadBalancerElementsCmd.java

@@ -17,11 +17,9 @@
 package org.apache.cloudstack.api.command.admin.internallb;
 
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 
-import javax.inject.Inject;
-
-
 import org.apache.cloudstack.api.APICommand;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.api.BaseListCmd;
@@ -46,9 +44,6 @@ import com.cloud.network.VirtualRouterProvider;
             responseHasSensitiveInfo = false)
 public class ListInternalLoadBalancerElementsCmd extends BaseListCmd {
 
-    @Inject
-    private InternalLoadBalancerElementService _service;
-
     /////////////////////////////////////////////////////
     //////////////// API parameters /////////////////////
     /////////////////////////////////////////////////////
@@ -86,12 +81,21 @@ public class ListInternalLoadBalancerElementsCmd extends BaseListCmd {
     @Override
     public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException,
         ResourceAllocationException {
-        List<? extends VirtualRouterProvider> providers = _service.searchForInternalLoadBalancerElements(getId(), getNspId(), getEnabled());
+        List<InternalLoadBalancerElementService> services;
+        if (id == null && nspId == null) {
+            services = _networkService.getInternalLoadBalancerElements();
+        } else {
+            InternalLoadBalancerElementService elementService = id != null ? _networkService.getInternalLoadBalancerElementById(id) : _networkService.getInternalLoadBalancerElementByNetworkServiceProviderId(nspId);
+            services = Collections.singletonList(elementService);
+        }
         ListResponse<InternalLoadBalancerElementResponse> response = new ListResponse<InternalLoadBalancerElementResponse>();
         List<InternalLoadBalancerElementResponse> providerResponses = new ArrayList<InternalLoadBalancerElementResponse>();
-        for (VirtualRouterProvider provider : providers) {
+        for (InternalLoadBalancerElementService service : services) {
-            InternalLoadBalancerElementResponse providerResponse = _responseGenerator.createInternalLbElementResponse(provider);
+            List<? extends VirtualRouterProvider> providers = service.searchForInternalLoadBalancerElements(getId(), getNspId(), getEnabled());
-            providerResponses.add(providerResponse);
+            for (VirtualRouterProvider provider : providers) {
+                InternalLoadBalancerElementResponse providerResponse = _responseGenerator.createInternalLbElementResponse(provider);
+                providerResponses.add(providerResponse);
+            }
         }
         response.setResponses(providerResponses);
         response.setResponseName(getCommandName());

api/src/main/java/org/apache/cloudstack/api/command/admin/network/CreateNetworkOfferingCmd.java

@@ -24,10 +24,14 @@ import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.stream.Collectors;
 
+import com.cloud.network.Network;
+import com.cloud.network.VirtualRouterProvider;
 import org.apache.cloudstack.api.response.DomainResponse;
 import org.apache.cloudstack.api.response.ZoneResponse;
 import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.BooleanUtils;
 import org.apache.commons.lang3.StringUtils;
 
 import org.apache.cloudstack.api.APICommand;
@@ -46,6 +50,16 @@ import com.cloud.offering.NetworkOffering;
 import com.cloud.offering.NetworkOffering.Availability;
 import com.cloud.user.Account;
 
+import static com.cloud.network.Network.Service.Dhcp;
+import static com.cloud.network.Network.Service.Dns;
+import static com.cloud.network.Network.Service.Lb;
+import static com.cloud.network.Network.Service.StaticNat;
+import static com.cloud.network.Network.Service.SourceNat;
+import static com.cloud.network.Network.Service.PortForwarding;
+import static com.cloud.network.Network.Service.NetworkACL;
+import static com.cloud.network.Network.Service.UserData;
+import static com.cloud.network.Network.Service.Firewall;
+
 @APICommand(name = "createNetworkOffering", description = "Creates a network offering.", responseObject = NetworkOfferingResponse.class, since = "3.0.0",
         requestHasSensitiveInfo = false, responseHasSensitiveInfo = false)
 public class CreateNetworkOfferingCmd extends BaseCmd {
@@ -126,6 +140,30 @@ public class CreateNetworkOfferingCmd extends BaseCmd {
             description = "true if network offering is meant to be used for VPC, false otherwise.")
     private Boolean forVpc;
 
+    @Parameter(name = ApiConstants.FOR_NSX,
+            type = CommandType.BOOLEAN,
+            description = "true if network offering is meant to be used for NSX, false otherwise.",
+            since = "4.20.0")
+    private Boolean forNsx;
+
+    @Parameter(name = ApiConstants.NSX_MODE,
+            type = CommandType.STRING,
+            description = "Indicates the mode with which the network will operate. Valid option: NATTED or ROUTED",
+            since = "4.20.0")
+    private String nsxMode;
+
+    @Parameter(name = ApiConstants.NSX_SUPPORT_LB,
+            type = CommandType.BOOLEAN,
+            description = "true if network offering for NSX network offering supports Load balancer service.",
+            since = "4.20.0")
+    private Boolean nsxSupportsLbService;
+
+    @Parameter(name = ApiConstants.NSX_SUPPORTS_INTERNAL_LB,
+            type = CommandType.BOOLEAN,
+            description = "true if network offering for NSX network offering supports Internal Load balancer service.",
+            since = "4.20.0")
+    private Boolean nsxSupportsInternalLbService;
+
     @Parameter(name = ApiConstants.FOR_TUNGSTEN,
             type = CommandType.BOOLEAN,
             description = "true if network offering is meant to be used for Tungsten-Fabric, false otherwise.")
@@ -210,7 +248,27 @@ public class CreateNetworkOfferingCmd extends BaseCmd {
     }
 
     public List<String> getSupportedServices() {
-        return supportedServices == null ? new ArrayList<String>() : supportedServices;
+        if (!isForNsx()) {
+            return supportedServices == null ? new ArrayList<String>() : supportedServices;
+        } else {
+            List<String> services = new ArrayList<>(List.of(
+                    Dhcp.getName(),
+                    Dns.getName(),
+                    StaticNat.getName(),
+                    SourceNat.getName(),
+                    PortForwarding.getName(),
+                    UserData.getName()
+            ));
+            if (getNsxSupportsLbService()) {
+                services.add(Lb.getName());
+            }
+            if (Boolean.TRUE.equals(forVpc)) {
+                services.add(NetworkACL.getName());
+            } else {
+                services.add(Firewall.getName());
+            }
+            return services;
+        }
     }
 
     public String getGuestIpType() {
@@ -240,6 +298,22 @@ public class CreateNetworkOfferingCmd extends BaseCmd {
         return forVpc;
     }
 
+    public boolean isForNsx() {
+        return BooleanUtils.isTrue(forNsx);
+    }
+
+    public String getNsxMode() {
+        return nsxMode;
+    }
+
+    public boolean getNsxSupportsLbService() {
+        return BooleanUtils.isTrue(nsxSupportsLbService);
+    }
+
+    public boolean getNsxSupportsInternalLbService() {
+        return BooleanUtils.isTrue(nsxSupportsInternalLbService);
+    }
+
     public Boolean getForTungsten() {
         return forTungsten;
     }
@@ -260,9 +334,8 @@ public class CreateNetworkOfferingCmd extends BaseCmd {
     }
 
     public Map<String, List<String>> getServiceProviders() {
-        Map<String, List<String>> serviceProviderMap = null;
+        Map<String, List<String>> serviceProviderMap = new HashMap<>();
-        if (serviceProviderList != null && !serviceProviderList.isEmpty()) {
+        if (serviceProviderList != null && !serviceProviderList.isEmpty() && !isForNsx()) {
-            serviceProviderMap = new HashMap<String, List<String>>();
             Collection servicesCollection = serviceProviderList.values();
             Iterator iter = servicesCollection.iterator();
             while (iter.hasNext()) {
@@ -278,11 +351,37 @@ public class CreateNetworkOfferingCmd extends BaseCmd {
                 providerList.add(provider);
                 serviceProviderMap.put(service, providerList);
             }
+        } else if (Boolean.TRUE.equals(forNsx)) {
+            getServiceProviderMapForNsx(serviceProviderMap);
         }
-
         return serviceProviderMap;
     }
 
+    private void getServiceProviderMapForNsx(Map<String, List<String>> serviceProviderMap) {
+        String routerProvider = Boolean.TRUE.equals(getForVpc()) ? VirtualRouterProvider.Type.VPCVirtualRouter.name() :
+                VirtualRouterProvider.Type.VirtualRouter.name();
+        List<String> unsupportedServices = new ArrayList<>(List.of("Vpn", "SecurityGroup", "Connectivity",
+                "Gateway", "BaremetalPxeService"));
+        List<String> routerSupported = List.of("Dhcp", "Dns", "UserData");
+        List<String> allServices = Service.listAllServices().stream().map(Service::getName).collect(Collectors.toList());
+        if (routerProvider.equals(VirtualRouterProvider.Type.VPCVirtualRouter.name())) {
+            unsupportedServices.add("Firewall");
+        } else {
+            unsupportedServices.add("NetworkACL");
+        }
+        for (String service : allServices) {
+            if (unsupportedServices.contains(service))
+                continue;
+            if (routerSupported.contains(service))
+                serviceProviderMap.put(service, List.of(routerProvider));
+            else
+                serviceProviderMap.put(service, List.of(Network.Provider.Nsx.getName()));
+            if (!getNsxSupportsLbService()) {
+                serviceProviderMap.remove(Lb.getName());
+            }
+        }
+    }
+
     public Map<Capability, String> getServiceCapabilities(Service service) {
         Map<Capability, String> capabilityMap = null;
 

api/src/main/java/org/apache/cloudstack/api/command/admin/vlan/CreateVlanIpRangeCmd.java

@@ -39,6 +39,8 @@ import com.cloud.exception.ResourceAllocationException;
 import com.cloud.exception.ResourceUnavailableException;
 import com.cloud.user.Account;
 
+import java.util.Objects;
+
 @APICommand(name = "createVlanIpRange", description = "Creates a VLAN IP range.", responseObject = VlanIpRangeResponse.class,
         requestHasSensitiveInfo = false, responseHasSensitiveInfo = false)
 public class CreateVlanIpRangeCmd extends BaseCmd {
@@ -112,6 +114,9 @@ public class CreateVlanIpRangeCmd extends BaseCmd {
     @Parameter(name = ApiConstants.FOR_SYSTEM_VMS, type = CommandType.BOOLEAN, description = "true if IP range is set to system vms, false if not")
     private Boolean forSystemVms;
 
+    @Parameter(name = ApiConstants.FOR_NSX, type = CommandType.BOOLEAN, description = "true if the IP range is used for NSX resource", since = "4.20.0")
+    private boolean forNsx;
+
     /////////////////////////////////////////////////////
     /////////////////// Accessors ///////////////////////
     /////////////////////////////////////////////////////
@@ -152,8 +157,12 @@ public class CreateVlanIpRangeCmd extends BaseCmd {
         return startIp;
     }
 
+    public boolean isForNsx() {
+        return !Objects.isNull(forNsx) && forNsx;
+    }
+
     public String getVlan() {
-        if (vlan == null || vlan.isEmpty()) {
+        if ((vlan == null || vlan.isEmpty()) && !isForNsx()) {
             vlan = "untagged";
         }
         return vlan;

api/src/main/java/org/apache/cloudstack/api/command/admin/vpc/CreateVPCOfferingCmd.java

@@ -24,10 +24,15 @@ import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.stream.Collectors;
 
+import com.cloud.exception.InvalidParameterValueException;
+import com.cloud.network.Network;
+import com.cloud.network.VirtualRouterProvider;
 import org.apache.cloudstack.api.response.DomainResponse;
 import org.apache.cloudstack.api.response.ZoneResponse;
 import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.BooleanUtils;
 import org.apache.commons.lang3.StringUtils;
 
 import org.apache.cloudstack.api.APICommand;
@@ -44,6 +49,15 @@ import com.cloud.exception.ResourceAllocationException;
 import com.cloud.network.vpc.VpcOffering;
 import com.cloud.user.Account;
 
+import static com.cloud.network.Network.Service.Dhcp;
+import static com.cloud.network.Network.Service.Dns;
+import static com.cloud.network.Network.Service.Lb;
+import static com.cloud.network.Network.Service.StaticNat;
+import static com.cloud.network.Network.Service.SourceNat;
+import static com.cloud.network.Network.Service.PortForwarding;
+import static com.cloud.network.Network.Service.NetworkACL;
+import static com.cloud.network.Network.Service.UserData;
+
 @APICommand(name = "createVPCOffering", description = "Creates VPC offering", responseObject = VpcOfferingResponse.class,
         requestHasSensitiveInfo = false, responseHasSensitiveInfo = false)
 public class CreateVPCOfferingCmd extends BaseAsyncCreateCmd {
@@ -60,7 +74,6 @@ public class CreateVPCOfferingCmd extends BaseAsyncCreateCmd {
 
     @Parameter(name = ApiConstants.SUPPORTED_SERVICES,
                type = CommandType.LIST,
-               required = true,
                collectionType = CommandType.STRING,
                description = "services supported by the vpc offering")
     private List<String> supportedServices;
@@ -99,6 +112,24 @@ public class CreateVPCOfferingCmd extends BaseAsyncCreateCmd {
             since = "4.13")
     private List<Long> zoneIds;
 
+    @Parameter(name = ApiConstants.FOR_NSX,
+            type = CommandType.BOOLEAN,
+            description = "true if network offering is meant to be used for NSX, false otherwise.",
+            since = "4.20.0")
+    private Boolean forNsx;
+
+    @Parameter(name = ApiConstants.NSX_MODE,
+            type = CommandType.STRING,
+            description = "Indicates the mode with which the network will operate. Valid option: NATTED or ROUTED",
+            since = "4.20.0")
+    private String nsxMode;
+
+    @Parameter(name = ApiConstants.NSX_SUPPORT_LB,
+            type = CommandType.BOOLEAN,
+            description = "true if network offering for NSX VPC offering supports Load balancer service.",
+            since = "4.20.0")
+    private Boolean nsxSupportsLbService;
+
     @Parameter(name = ApiConstants.ENABLE,
             type = CommandType.BOOLEAN,
             description = "set to true if the offering is to be enabled during creation. Default is false",
@@ -118,13 +149,41 @@ public class CreateVPCOfferingCmd extends BaseAsyncCreateCmd {
     }
 
     public List<String> getSupportedServices() {
+        if (!isForNsx() && CollectionUtils.isEmpty(supportedServices)) {
+            throw new InvalidParameterValueException("Supported services needs to be provided");
+        }
+        if (isForNsx()) {
+            supportedServices = new ArrayList<>(List.of(
+                    Dhcp.getName(),
+                    Dns.getName(),
+                    StaticNat.getName(),
+                    SourceNat.getName(),
+                    NetworkACL.getName(),
+                    PortForwarding.getName(),
+                    UserData.getName()
+                    ));
+            if (getNsxSupportsLbService()) {
+                supportedServices.add(Lb.getName());
+            }
+        }
         return supportedServices;
     }
 
+    public boolean isForNsx() {
+        return BooleanUtils.isTrue(forNsx);
+    }
+
+    public String getNsxMode() {
+        return nsxMode;
+    }
+
+    public boolean getNsxSupportsLbService() {
+        return org.apache.commons.lang3.BooleanUtils.isTrue(nsxSupportsLbService);
+    }
+
     public Map<String, List<String>> getServiceProviders() {
-        Map<String, List<String>> serviceProviderMap = null;
+        Map<String, List<String>> serviceProviderMap = new HashMap<>();
-        if (serviceProviderList != null && !serviceProviderList.isEmpty()) {
+        if (serviceProviderList != null && !serviceProviderList.isEmpty() && !isForNsx()) {
-            serviceProviderMap = new HashMap<String, List<String>>();
             Collection<? extends Map<String, String>> servicesCollection = serviceProviderList.values();
             Iterator<? extends Map<String, String>> iter = servicesCollection.iterator();
             while (iter.hasNext()) {
@@ -132,7 +191,7 @@ public class CreateVPCOfferingCmd extends BaseAsyncCreateCmd {
                 if (logger.isTraceEnabled()) {
                     logger.trace("service provider entry specified: " + obj);
                 }
-                HashMap<String, String> services = (HashMap<String, String>)obj;
+                HashMap<String, String> services = (HashMap<String, String>) obj;
                 String service = services.get("service");
                 String provider = services.get("provider");
                 List<String> providerList = null;
@@ -144,11 +203,31 @@ public class CreateVPCOfferingCmd extends BaseAsyncCreateCmd {
                 providerList.add(provider);
                 serviceProviderMap.put(service, providerList);
             }
+        } else if (Boolean.TRUE.equals(forNsx)) {
+            getServiceProviderMapForNsx(serviceProviderMap);
         }
 
         return serviceProviderMap;
     }
 
+    private void getServiceProviderMapForNsx(Map<String, List<String>> serviceProviderMap) {
+        List<String> unsupportedServices = List.of("Vpn", "BaremetalPxeService", "SecurityGroup", "Connectivity",
+                "Gateway", "Firewall");
+        List<String> routerSupported = List.of("Dhcp", "Dns", "UserData");
+        List<String> allServices = Network.Service.listAllServices().stream().map(Network.Service::getName).collect(Collectors.toList());
+        for (String service : allServices) {
+            if (unsupportedServices.contains(service))
+                continue;
+            if (routerSupported.contains(service))
+                serviceProviderMap.put(service, List.of(VirtualRouterProvider.Type.VPCVirtualRouter.name()));
+            else
+                serviceProviderMap.put(service, List.of(Network.Provider.Nsx.getName()));
+        }
+        if (!getNsxSupportsLbService()) {
+            serviceProviderMap.remove(Lb.getName());
+        }
+    }
+
     public Map<String, List<String>> getServiceCapabilityList() {
         return serviceCapabilityList;
     }

api/src/main/java/org/apache/cloudstack/api/command/user/address/ListPublicIpAddressesCmd.java

@@ -19,6 +19,7 @@ package org.apache.cloudstack.api.command.user.address;
 import java.util.ArrayList;
 import java.util.List;
 
+import org.apache.commons.lang.BooleanUtils;
 
 import org.apache.cloudstack.acl.RoleType;
 import org.apache.cloudstack.api.APICommand;
@@ -104,6 +105,9 @@ public class ListPublicIpAddressesCmd extends BaseListRetrieveOnlyResourceCountC
     @Parameter(name = ApiConstants.FOR_DISPLAY, type = CommandType.BOOLEAN, description = "list resources by display flag; only ROOT admin is eligible to pass this parameter", since = "4.4", authorized = {RoleType.Admin})
     private Boolean display;
 
+    @Parameter(name = ApiConstants.FOR_SYSTEM_VMS, type = CommandType.BOOLEAN, description = "true if range is dedicated for system VMs", since = "4.20.0")
+    private Boolean forSystemVMs;
+
     /////////////////////////////////////////////////////
     /////////////////// Accessors ///////////////////////
     /////////////////////////////////////////////////////
@@ -175,6 +179,10 @@ public class ListPublicIpAddressesCmd extends BaseListRetrieveOnlyResourceCountC
         return state;
     }
 
+    public boolean getForSystemVMs() {
+        return BooleanUtils.isTrue(forSystemVMs);
+    }
+
     /////////////////////////////////////////////////////
     /////////////// API Implementation///////////////////
     /////////////////////////////////////////////////////

api/src/main/java/org/apache/cloudstack/api/response/IPAddressResponse.java

@@ -167,6 +167,10 @@ public class IPAddressResponse extends BaseResponseWithAnnotations implements Co
     @Param(description="whether the ip address has Firewall/PortForwarding/LoadBalancing rules defined")
     private boolean hasRules;
 
+    @SerializedName(ApiConstants.FOR_SYSTEM_VMS)
+    @Param(description="true if range is dedicated for System VMs")
+    private boolean forSystemVms;
+
     public void setIpAddress(String ipAddress) {
         this.ipAddress = ipAddress;
     }
@@ -316,4 +320,8 @@ public class IPAddressResponse extends BaseResponseWithAnnotations implements Co
     public void setHasRules(final boolean hasRules) {
         this.hasRules = hasRules;
     }
+
+    public void setForSystemVms(boolean forSystemVms) {
+        this.forSystemVms = forSystemVms;
+    }
 }

api/src/main/java/org/apache/cloudstack/api/response/NetworkOfferingResponse.java

@@ -99,10 +99,18 @@ public class NetworkOfferingResponse extends BaseResponseWithAnnotations {
     @Param(description = "true if network offering can be used by VPC networks only")
     private Boolean forVpc;
 
+    @SerializedName(ApiConstants.FOR_NSX)
+    @Param(description = "true if network offering can be used by NSX networks only")
+    private Boolean forNsx;
+
     @SerializedName(ApiConstants.FOR_TUNGSTEN)
     @Param(description = "true if network offering can be used by Tungsten-Fabric networks only")
     private Boolean forTungsten;
 
+    @SerializedName(ApiConstants.NSX_MODE)
+    @Param(description = "Mode in which the network will operate. This parameter is only relevant for NSX offerings")
+    private String nsxMode;
+
     @SerializedName(ApiConstants.IS_PERSISTENT)
     @Param(description = "true if network offering supports persistent networks, false otherwise")
     private Boolean isPersistent;
@@ -127,6 +135,10 @@ public class NetworkOfferingResponse extends BaseResponseWithAnnotations {
     @Param(description = "true if network offering supports public access for guest networks", since = "4.10.0")
     private Boolean supportsPublicAccess;
 
+    @SerializedName(ApiConstants.SUPPORTS_INTERNAL_LB)
+    @Param(description = "true if network offering supports public access for guest networks", since = "4.20.0")
+    private Boolean supportsInternalLb;
+
     @SerializedName(ApiConstants.DOMAIN_ID)
     @Param(description = "the domain ID(s) this disk offering belongs to. Ignore this information as it is not currently applicable.")
     private String domainId;
@@ -215,10 +227,18 @@ public class NetworkOfferingResponse extends BaseResponseWithAnnotations {
         this.forVpc = forVpc;
     }
 
+    public void setForNsx(Boolean forNsx) {
+        this.forNsx = forNsx;
+    }
+
     public void setForTungsten(Boolean forTungsten) {
         this.forTungsten = forTungsten;
     }
 
+    public void setNsxMode(String nsxMode) {
+        this.nsxMode = nsxMode;
+    }
+
     public void setIsPersistent(Boolean isPersistent) {
         this.isPersistent = isPersistent;
     }
@@ -243,6 +263,10 @@ public class NetworkOfferingResponse extends BaseResponseWithAnnotations {
         this.supportsPublicAccess = supportsPublicAccess;
     }
 
+    public void setSupportsInternalLb(Boolean supportsInternalLb) {
+        this.supportsInternalLb = supportsInternalLb;
+    }
+
     public String getDomainId() {
         return domainId;
     }

api/src/main/java/org/apache/cloudstack/api/response/VlanIpRangeResponse.java

@@ -123,6 +123,10 @@ public class VlanIpRangeResponse extends BaseResponse implements ControlledEntit
     @Param(description = "indicates whether VLAN IP range is dedicated to system vms or not")
     private Boolean forSystemVms;
 
+    @SerializedName(ApiConstants.FOR_NSX)
+    @Param(description = "indicates whether IP range is dedicated to NSX resources or not")
+    private Boolean forNsx;
+
     public void setId(String id) {
         this.id = id;
     }
@@ -235,4 +239,8 @@ public class VlanIpRangeResponse extends BaseResponse implements ControlledEntit
     public void setIp6Cidr(String ip6Cidr) {
         this.ip6Cidr = ip6Cidr;
     }
+
+    public void setForNsx(Boolean forNsx) {
+        this.forNsx = forNsx;
+    }
 }

api/src/main/java/org/apache/cloudstack/api/response/VpcOfferingResponse.java

@@ -63,9 +63,17 @@ public class VpcOfferingResponse extends BaseResponse {
     private Boolean supportsDistributedRouter;
 
     @SerializedName((ApiConstants.SUPPORTS_REGION_LEVEL_VPC))
-    @Param(description = " indicated if the offering can support region level vpc", since = "4.4")
+    @Param(description = "indicated if the offering can support region level vpc", since = "4.4")
     private Boolean supportsRegionLevelVpc;
 
+    @SerializedName(ApiConstants.FOR_NSX)
+    @Param(description = "true if vpc offering can be used by NSX networks only")
+    private Boolean forNsx;
+
+    @SerializedName(ApiConstants.NSX_MODE)
+    @Param(description = "Mode in which the network will operate. This parameter is only relevant for NSX offerings")
+    private String nsxMode;
+
     @SerializedName(ApiConstants.DOMAIN_ID)
     @Param(description = "the domain ID(s) this disk offering belongs to. Ignore this information as it is not currently applicable.")
     private String domainId;
@@ -138,6 +146,14 @@ public class VpcOfferingResponse extends BaseResponse {
         this.domain = domain;
     }
 
+    public void setForNsx(Boolean forNsx) {
+        this.forNsx = forNsx;
+    }
+
+    public void setNsxMode(String nsxMode) {
+        this.nsxMode = nsxMode;
+    }
+
     public String getZoneId() {
         return zoneId;
     }

api/src/main/java/org/apache/cloudstack/api/response/ZoneResponse.java

@@ -145,6 +145,10 @@ public class ZoneResponse extends BaseResponseWithAnnotations implements SetReso
     @Param(description = "the type of the zone - core or edge", since = "4.18.0")
     String type;
 
+    @SerializedName(ApiConstants.NSX_ENABLED)
+    @Param(description = "true, if zone is NSX enabled", since = "4.20.0")
+    private boolean nsxEnabled = false;
+
     public ZoneResponse() {
         tags = new LinkedHashSet<ResourceTagResponse>();
     }
@@ -368,4 +372,8 @@ public class ZoneResponse extends BaseResponseWithAnnotations implements SetReso
     public String getType() {
         return type;
     }
+
+    public void setNsxEnabled(boolean nsxEnabled) {
+        this.nsxEnabled = nsxEnabled;
+    }
 }

api/src/main/java/org/apache/cloudstack/network/element/InternalLoadBalancerElementService.java

@@ -52,4 +52,6 @@ public interface InternalLoadBalancerElementService extends PluggableService {
      * @return
      */
     List<? extends VirtualRouterProvider> searchForInternalLoadBalancerElements(Long id, Long ntwkSvsProviderId, Boolean enabled);
+
+    VirtualRouterProvider.Type getProviderType();
 }

api/src/test/java/org/apache/cloudstack/api/command/admin/offering/CreateNetworkOfferingCmdTest.java

@@ -23,14 +23,16 @@ import org.junit.Test;
 import org.mockito.InjectMocks;
 import org.springframework.test.util.ReflectionTestUtils;
 
+
 public class CreateNetworkOfferingCmdTest {
 
     @InjectMocks
     private CreateNetworkOfferingCmd createNetworkOfferingCmd = new CreateNetworkOfferingCmd();
 
+    String netName = "network";
+
     @Test
     public void createVpcNtwkOffWithEmptyDisplayText() {
-        String netName = "network";
         ReflectionTestUtils.setField(createNetworkOfferingCmd, "networkOfferingName", netName);
         Assert.assertEquals(createNetworkOfferingCmd.getDisplayText(), netName);
     }

api/src/test/java/org/apache/cloudstack/api/command/admin/vpc/CreateVPCOfferingCmdTest.java

@@ -52,15 +52,15 @@ public class CreateVPCOfferingCmdTest {
             IllegalAccessException {
         CreateVPCOfferingCmd cmd = new CreateVPCOfferingCmd();
         ApiCmdTestUtil.set(cmd, ApiConstants.SERVICE_PROVIDER_LIST, new HashMap<String, Map<String, String>>());
-        Assert.assertNull(cmd.getServiceProviders());
+        Assert.assertTrue(cmd.getServiceProviders().isEmpty());
     }
 
     @Test
-    public void getDetailsNull() throws IllegalArgumentException,
+    public void getDetailsEmpty() throws IllegalArgumentException,
             IllegalAccessException {
         CreateVPCOfferingCmd cmd = new CreateVPCOfferingCmd();
         ApiCmdTestUtil.set(cmd, ApiConstants.SERVICE_PROVIDER_LIST, null);
-        Assert.assertNull(cmd.getServiceProviders());
+        Assert.assertTrue(cmd.getServiceProviders().isEmpty());
     }
 
     @Test

client/pom.xml

@@ -296,6 +296,11 @@
             <artifactId>cloud-plugin-network-tungsten</artifactId>
             <version>${project.version}</version>
         </dependency>
+        <dependency>
+            <groupId>org.apache.cloudstack</groupId>
+            <artifactId>cloud-plugin-network-nsx</artifactId>
+            <version>${project.version}</version>
+        </dependency>
         <dependency>
             <groupId>org.apache.cloudstack</groupId>
             <artifactId>cloud-plugin-network-elb</artifactId>

core/src/main/java/com/cloud/agent/api/SetupGuestNetworkCommand.java

@@ -35,6 +35,7 @@ public class SetupGuestNetworkCommand extends NetworkElementCommand {
     String routerIpv6 = null;
     String routerIpv6Gateway = null;
     String routerIpv6Cidr = null;
+    boolean isVrGuestGateway = false;
 
     public NicTO getNic() {
         return nic;
@@ -114,4 +115,12 @@ public class SetupGuestNetworkCommand extends NetworkElementCommand {
     public void setDefaultIp6Dns2(String defaultIp6Dns2) {
         this.defaultIp6Dns2 = defaultIp6Dns2;
     }
+
+    public boolean isVrGuestGateway() {
+        return isVrGuestGateway;
+    }
+
+    public void setVrGuestGateway(boolean vrGuestGateway) {
+        isVrGuestGateway = vrGuestGateway;
+    }
 }

core/src/main/java/com/cloud/agent/resource/virtualnetwork/facade/SetGuestNetworkConfigItem.java

@@ -75,6 +75,7 @@ public class SetGuestNetworkConfigItem extends AbstractConfigItemFacade {
         guestNetwork.setRouterIp6(command.getRouterIpv6());
         guestNetwork.setRouterIp6Gateway(command.getRouterIpv6Gateway());
         guestNetwork.setRouterIp6Cidr(command.getRouterIpv6Cidr());
+        guestNetwork.setVrGuestGateway(command.isVrGuestGateway());
 
         return generateConfigItems(guestNetwork);
     }

core/src/main/java/com/cloud/agent/resource/virtualnetwork/model/GuestNetwork.java

@@ -37,6 +37,7 @@ public class GuestNetwork extends ConfigBase {
     private String routerIp6;
     private String routerIp6Gateway;
     private String routerIp6Cidr;
+    private boolean isVrGuestGateway;
 
     private Integer mtu;
 
@@ -202,4 +203,12 @@ public class GuestNetwork extends ConfigBase {
     public Integer getMtu() {
         return mtu;
     }
+
+    public boolean isVrGuestGateway() {
+        return isVrGuestGateway;
+    }
+
+    public void setVrGuestGateway(boolean vrGuestGateway) {
+        isVrGuestGateway = vrGuestGateway;
+    }
 }

core/src/main/resources/META-INF/cloudstack/core/spring-core-registry-core-context.xml

@@ -350,4 +350,12 @@
     <bean id="clusterDrsAlgorithmRegistry"
           class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
     </bean>
+
+    <bean id="internalLoadBalancerElementServiceRegistry" class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
+        <property name="preRegistered">
+            <list>
+                <ref bean="InternalLbVm" />
+            </list>
+        </property>
+    </bean>
 </beans>

core/src/main/resources/META-INF/cloudstack/network/spring-core-lifecycle-network-context-inheritable.xml

@@ -103,4 +103,9 @@
         <property name="typeClass"
             value="org.apache.cloudstack.region.gslb.GslbServiceProvider" />
     </bean>
+
+    <bean class="org.apache.cloudstack.spring.lifecycle.registry.RegistryLifecycle">
+        <property name="registry" ref="internalLoadBalancerElementServiceRegistry" />
+        <property name="typeClass" value="org.apache.cloudstack.network.element.InternalLoadBalancerElementService" />
+    </bean>
 </beans>

engine/api/src/main/java/org/apache/cloudstack/engine/orchestration/service/NetworkOrchestrationService.java

@@ -105,6 +105,9 @@ public interface NetworkOrchestrationService {
     static final ConfigKey<Boolean> TUNGSTEN_ENABLED = new ConfigKey<>(Boolean.class, "tungsten.plugin.enable", "Advanced", "false",
             "Indicates whether to enable the Tungsten plugin", false, ConfigKey.Scope.Zone, null);
 
+    static final ConfigKey<Boolean> NSX_ENABLED = new ConfigKey<>(Boolean.class, "nsx.plugin.enable", "Advanced", "false",
+            "Indicates whether to enable the NSX plugin", false, ConfigKey.Scope.Zone, null);
+
     List<? extends Network> setupNetwork(Account owner, NetworkOffering offering, DeploymentPlan plan, String name, String displayText, boolean isDefault)
         throws ConcurrentOperationException;
 

engine/components-api/src/main/java/com/cloud/configuration/ConfigurationManager.java

@@ -63,6 +63,9 @@ public interface ConfigurationManager {
     static final String VM_USERDATA_MAX_LENGTH_STRING = "vm.userdata.max.length";
     static final ConfigKey<Integer> VM_USERDATA_MAX_LENGTH = new ConfigKey<>("Advanced", Integer.class, VM_USERDATA_MAX_LENGTH_STRING, "32768",
             "Max length of vm userdata after base64 decoding. Default is 32768 and maximum is 1048576", true);
+    public static final ConfigKey<Boolean> AllowNonRFC1918CompliantIPs = new ConfigKey<>(Boolean.class,
+            "allow.non.rfc1918.compliant.ips", "Advanced", "false",
+            "Allows non-compliant RFC 1918 IPs for Shared, Isolated networks and VPCs", true, null);
 
     /**
      * @param offering
@@ -97,7 +100,6 @@ public interface ConfigurationManager {
 //     * @param volatileVm
 //     * @param hostTag
 //     * @param networkRate
-//     *            TODO
 //     * @param id
 //     * @param useVirtualNetwork
 //     * @param deploymentPlanner
@@ -167,11 +169,9 @@ public interface ConfigurationManager {
      * @param zoneType
      * @param allocationState
      * @param networkDomain
-     *            TODO
      * @param isSecurityGroupEnabled
-     *            TODO
+     * @param ip6Dns1
-     * @param ip6Dns1 TODO
+     * @param ip6Dns2
-     * @param ip6Dns2 TODO
      * @return
      * @throws
      * @throws
@@ -186,7 +186,7 @@ public interface ConfigurationManager {
      *
      * @param userId
      * @param vlanDbId
-     * @param caller TODO
+     * @param caller
      * @return success/failure
      */
     boolean deleteVlanAndPublicIpRange(long userId, long vlanDbId, Account caller);
@@ -197,30 +197,25 @@ public interface ConfigurationManager {
 
     /**
      * Creates a new network offering
+     *
      * @param name
      * @param displayText
      * @param trafficType
      * @param tags
      * @param specifyVlan
      * @param networkRate
-     *            TODO
      * @param serviceProviderMap
-     *            TODO
      * @param isDefault
-     *            TODO
      * @param type
-     *            TODO
      * @param systemOnly
-     *            TODO
      * @param serviceOfferingId
-     * @param conserveMode
+     * @param conserveMode       ;
-     *            ;
      * @param specifyIpRanges
-     *            TODO
+     * @param isPersistent       ;
-     * @param isPersistent
+     * @param details
-     *            ;
-     * @param details TODO
      * @param forVpc
+     * @param forTungsten
+     * @param forNsx
      * @param domainIds
      * @param zoneIds
      * @return network offering object
@@ -230,10 +225,10 @@ public interface ConfigurationManager {
                                             Integer networkRate, Map<Service, Set<Provider>> serviceProviderMap, boolean isDefault, Network.GuestType type, boolean systemOnly, Long serviceOfferingId,
                                             boolean conserveMode, Map<Service, Map<Capability, String>> serviceCapabilityMap, boolean specifyIpRanges, boolean isPersistent,
                                             Map<NetworkOffering.Detail, String> details, boolean egressDefaultPolicy, Integer maxconn, boolean enableKeepAlive, Boolean forVpc,
-                                            Boolean forTungsten, List<Long> domainIds, List<Long> zoneIds, boolean enableOffering, final NetUtils.InternetProtocol internetProtocol);
+                                            Boolean forTungsten, boolean forNsx, String mode, List<Long> domainIds, List<Long> zoneIds, boolean enableOffering, final NetUtils.InternetProtocol internetProtocol);
 
     Vlan createVlanAndPublicIpRange(long zoneId, long networkId, long physicalNetworkId, boolean forVirtualNetwork, boolean forSystemVms, Long podId, String startIP, String endIP,
-        String vlanGateway, String vlanNetmask, String vlanId, boolean bypassVlanOverlapCheck, Domain domain, Account vlanOwner, String startIPv6, String endIPv6, String vlanIp6Gateway, String vlanIp6Cidr)
+        String vlanGateway, String vlanNetmask, String vlanId, boolean bypassVlanOverlapCheck, Domain domain, Account vlanOwner, String startIPv6, String endIPv6, String vlanIp6Gateway, String vlanIp6Cidr, boolean forNsx)
         throws InsufficientCapacityException, ConcurrentOperationException, InvalidParameterValueException;
 
     void createDefaultSystemNetworks(long zoneId) throws ConcurrentOperationException;

engine/components-api/src/main/java/com/cloud/network/addr/PublicIp.java

@@ -269,4 +269,11 @@ public class PublicIp implements PublicIpAddress {
     public void setRuleState(State ruleState) {
         _addr.setRuleState(ruleState);
     }
+
+    @Override
+    public boolean isForSystemVms() {
+        return false;
+    }
+
+
 }

engine/components-api/src/main/java/com/cloud/network/vpc/NetworkACLManager.java

@@ -19,6 +19,7 @@ package com.cloud.network.vpc;
 import java.util.List;
 
 import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.network.Network;
 import com.cloud.network.dao.NetworkVO;
 
 public interface NetworkACLManager {
@@ -91,4 +92,6 @@ public interface NetworkACLManager {
     boolean revokeACLItemsForPrivateGw(PrivateGateway gateway) throws ResourceUnavailableException;
 
     boolean applyACLToPrivateGw(PrivateGateway gateway) throws ResourceUnavailableException;
+
+    boolean reorderAclRules(VpcVO vpc, List<? extends Network> networks, List<? extends NetworkACLItem> networkACLItems);
 }

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

@@ -36,6 +36,7 @@ import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
+import java.util.Objects;
 import java.util.Set;
 import java.util.TimeZone;
 import java.util.UUID;
@@ -48,6 +49,11 @@ import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 import javax.persistence.EntityExistsException;
 
+import com.cloud.domain.Domain;
+import com.cloud.domain.dao.DomainDao;
+import com.cloud.network.vpc.VpcVO;
+import com.cloud.network.vpc.dao.VpcDao;
+import com.cloud.user.dao.AccountDao;
 import com.cloud.event.ActionEventUtils;
 import com.google.gson.Gson;
 import org.apache.cloudstack.affinity.dao.AffinityGroupVMMapDao;
@@ -385,6 +391,12 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
     private DomainRouterJoinDao domainRouterJoinDao;
     @Inject
     private AnnotationDao annotationDao;
+    @Inject
+    private AccountDao accountDao;
+    @Inject
+    private VpcDao vpcDao;
+    @Inject
+    private DomainDao domainDao;
 
     VmWorkJobHandlerProxy _jobHandlerProxy = new VmWorkJobHandlerProxy(this);
 
@@ -599,11 +611,18 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
                 VirtualMachine.Type.ConsoleProxy.equals(vm.getType());
     }
 
-    protected void advanceExpunge(VMInstanceVO vm) throws ResourceUnavailableException, OperationTimedoutException, ConcurrentOperationException {
+    private boolean isVmDestroyed(VMInstanceVO vm) {
         if (vm == null || vm.getRemoved() != null) {
             if (logger.isDebugEnabled()) {
                 logger.debug("Unable to find vm or vm is expunged: " + vm);
             }
+            return true;
+        }
+        return false;
+    }
+
+    protected void advanceExpunge(VMInstanceVO vm) throws ResourceUnavailableException, OperationTimedoutException, ConcurrentOperationException {
+        if (isVmDestroyed(vm)) {
             return;
         }
 
@@ -674,28 +693,31 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
 
         // send hypervisor-dependent commands before removing
         final List<Command> finalizeExpungeCommands = hvGuru.finalizeExpunge(vm);
-        if (CollectionUtils.isNotEmpty(finalizeExpungeCommands) || CollectionUtils.isNotEmpty(nicExpungeCommands)) {
+        handleUnsuccessfulExpungeOperation(finalizeExpungeCommands, nicExpungeCommands, vm, hostId);
-            if (hostId != null) {
-                final Commands cmds = new Commands(Command.OnError.Stop);
-                addAllExpungeCommandsFromList(finalizeExpungeCommands, cmds, vm);
-                addAllExpungeCommandsFromList(nicExpungeCommands, cmds, vm);
-                _agentMgr.send(hostId, cmds);
-                if (!cmds.isSuccessful()) {
-                    for (final Answer answer : cmds.getAnswers()) {
-                        if (!answer.getResult()) {
-                            logger.warn("Failed to expunge vm due to: " + answer.getDetails());
-                            throw new CloudRuntimeException("Unable to expunge " + vm + " due to " + answer.getDetails());
-                        }
-                    }
-                }
-            }
-        }
 
         if (logger.isDebugEnabled()) {
             logger.debug("Expunged " + vm);
         }
     }
 
+    private void handleUnsuccessfulExpungeOperation(List<Command> finalizeExpungeCommands, List<Command> nicExpungeCommands,
+                                                    VMInstanceVO vm, Long hostId) throws OperationTimedoutException, AgentUnavailableException {
+        if (CollectionUtils.isNotEmpty(finalizeExpungeCommands) || CollectionUtils.isNotEmpty(nicExpungeCommands) && (hostId != null)) {
+            final Commands cmds = new Commands(Command.OnError.Stop);
+            addAllExpungeCommandsFromList(finalizeExpungeCommands, cmds, vm);
+            addAllExpungeCommandsFromList(nicExpungeCommands, cmds, vm);
+            _agentMgr.send(hostId, cmds);
+            if (!cmds.isSuccessful()) {
+                for (final Answer answer : cmds.getAnswers()) {
+                    if (!answer.getResult()) {
+                        logger.warn("Failed to expunge vm due to: " + answer.getDetails());
+                        throw new CloudRuntimeException(String.format("Unable to expunge %s due to %s", vm, answer.getDetails()));
+                    }
+                }
+            }
+        }
+    }
+
     protected void handleUnsuccessfulCommands(Commands cmds, VMInstanceVO vm) throws CloudRuntimeException {
         String cmdsStr = cmds.toString();
         String vmToString = vm.toString();
@@ -1277,6 +1299,8 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
                     checkAndSetEnterSetupMode(vmTO, params);
 
                     handlePath(vmTO.getDisks(), vm.getHypervisorType());
+                    setVmNetworkDetails(vm, vmTO);
+
 
                     Commands cmds = new Commands(Command.OnError.Stop);
                     final Map<String, String> sshAccessDetails = _networkMgr.getSystemVMAccessDetails(vm);
@@ -1460,6 +1484,55 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
         }
     }
 
+    public void setVmNetworkDetails(VMInstanceVO vm, VirtualMachineTO vmTO) {
+        Map<Long, String> networkToNetworkNameMap = new HashMap<>();
+        if (VirtualMachine.Type.User.equals(vm.getType())) {
+            List<UserVmJoinVO> userVmJoinVOs = userVmJoinDao.searchByIds(vm.getId());
+            if (userVmJoinVOs != null && !userVmJoinVOs.isEmpty()) {
+                for (UserVmJoinVO userVmJoinVO : userVmJoinVOs) {
+                    addToNetworkNameMap(userVmJoinVO.getNetworkId(), vm.getDataCenterId(), networkToNetworkNameMap);
+                }
+                vmTO.setNetworkIdToNetworkNameMap(networkToNetworkNameMap);
+            }
+        } else if (VirtualMachine.Type.DomainRouter.equals(vm.getType())) {
+            List<DomainRouterJoinVO> routerJoinVO = domainRouterJoinDao.getRouterByIdAndTrafficType(vm.getId(), Networks.TrafficType.Guest);
+            for (DomainRouterJoinVO router : routerJoinVO) {
+                NetworkVO guestNetwork = _networkDao.findById(router.getNetworkId());
+                if (guestNetwork.getVpcId() == null && guestNetwork.getBroadcastDomainType() == Networks.BroadcastDomainType.NSX) {
+                    addToNetworkNameMap(router.getNetworkId(), vm.getDataCenterId(), networkToNetworkNameMap);
+                }
+            }
+            vmTO.setNetworkIdToNetworkNameMap(networkToNetworkNameMap);
+        }
+    }
+
+    private void addToNetworkNameMap(long networkId, long dataCenterId, Map<Long, String> networkToNetworkNameMap) {
+        NetworkVO networkVO = _networkDao.findById(networkId);
+        Account acc = accountDao.findById(networkVO.getAccountId());
+        Domain domain = domainDao.findById(networkVO.getDomainId());
+        DataCenter zone = _dcDao.findById(dataCenterId);
+        if (Objects.isNull(zone)) {
+            throw new CloudRuntimeException(String.format("Failed to find zone with ID: %s", dataCenterId));
+        }
+        if (Objects.isNull(acc)) {
+            throw new CloudRuntimeException(String.format("Failed to find account with ID: %s", networkVO.getAccountId()));
+        }
+        if (Objects.isNull(domain)) {
+            throw new CloudRuntimeException(String.format("Failed to find domain with ID: %s", networkVO.getDomainId()));
+        }
+        String networkName = String.format("D%s-A%s-Z%s", domain.getId(), acc.getId(), zone.getId());
+        if (Objects.isNull(networkVO.getVpcId())) {
+            networkName += "-S" + networkVO.getId();
+        } else {
+            VpcVO vpc = vpcDao.findById(networkVO.getVpcId());
+            if (Objects.isNull(vpc)) {
+                throw new CloudRuntimeException(String.format("Failed to find VPC with ID: %s", networkVO.getVpcId()));
+            }
+            networkName = String.format("%s-V%s-S%s", networkName, vpc.getId(), networkVO.getId());
+        }
+        networkToNetworkNameMap.put(networkVO.getId(), networkName);
+    }
+
     /**
      * Setting pod id to null can result in migration of Volumes across pods. This is not desirable for VMs which
      * have a volume in Ready state (happens when a VM is shutdown and started again).
@@ -2734,6 +2807,7 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
 
         final VirtualMachineTO to = toVmTO(profile);
         final PrepareForMigrationCommand pfmc = new PrepareForMigrationCommand(to);
+        setVmNetworkDetails(vm, to);
 
         ItWorkVO work = new ItWorkVO(UUID.randomUUID().toString(), _nodeId, State.Migrating, vm.getType(), vm.getId());
         work.setStep(Step.Prepare);

engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java

@@ -38,6 +38,9 @@ import java.util.stream.Collectors;
 import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 
+import com.cloud.dc.VlanDetailsVO;
+import com.cloud.dc.dao.VlanDetailsDao;
+import com.cloud.network.dao.NsxProviderDao;
 import org.apache.cloudstack.acl.ControlledEntity.ACLType;
 import org.apache.cloudstack.annotation.AnnotationService;
 import org.apache.cloudstack.annotation.dao.AnnotationDao;
@@ -56,6 +59,7 @@ import org.apache.cloudstack.managed.context.ManagedContextRunnable;
 import org.apache.cloudstack.network.dao.NetworkPermissionDao;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.lang3.BooleanUtils;
+import org.apache.commons.lang3.ObjectUtils;
 import org.apache.commons.lang3.StringUtils;
 
 import com.cloud.agent.AgentManager;
@@ -253,6 +257,7 @@ import com.cloud.vm.dao.NicSecondaryIpVO;
 import com.cloud.vm.dao.UserVmDao;
 import com.cloud.vm.dao.VMInstanceDao;
 import com.googlecode.ipv6.IPv6Address;
+import org.jetbrains.annotations.NotNull;
 
 /**
  * NetworkManagerImpl implements NetworkManager.
@@ -337,8 +342,12 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
     Ipv6Service ipv6Service;
     @Inject
     RouterNetworkDao routerNetworkDao;
+    @Inject
+    private VlanDetailsDao vlanDetailsDao;
 
     List<NetworkGuru> networkGurus;
+    @Inject
+    private NsxProviderDao nsxProviderDao;
 
     @Override
     public List<NetworkGuru> getNetworkGurus() {
@@ -500,6 +509,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
         defaultTungstenSharedSGEnabledNetworkOfferingProviders.put(Service.UserData, tungstenProvider);
         defaultTungstenSharedSGEnabledNetworkOfferingProviders.put(Service.SecurityGroup, tungstenProvider);
 
+
         final Map<Network.Service, Set<Network.Provider>> defaultIsolatedSourceNatEnabledNetworkOfferingProviders = new HashMap<Network.Service, Set<Network.Provider>>();
         defaultProviders.clear();
         defaultProviders.add(Network.Provider.VirtualRouter);
@@ -536,27 +546,27 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.QuickCloudNoServices) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.QuickCloudNoServices, "Offering for QuickCloud with no services", TrafficType.Guest, null, true,
                             Availability.Optional, null, new HashMap<Network.Service, Set<Network.Provider>>(), true, Network.GuestType.Shared, false, null, true, null, true,
-                            false, null, false, null, true, false, false, null, null, true, null);
+                            false, null, false, null, true, false, false, false, null, null, null, true, null);
                 }
 
                 //#2 - SG enabled network offering
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DefaultSharedNetworkOfferingWithSGService) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultSharedNetworkOfferingWithSGService, "Offering for Shared Security group enabled networks",
                             TrafficType.Guest, null, true, Availability.Optional, null, defaultSharedNetworkOfferingProviders, true, Network.GuestType.Shared, false, null, true,
-                            null, true, false, null, false, null, true, false, false, null, null, true, null);
+                            null, true, false, null, false, null, true, false, false, false, null, null, null, true, null);
                 }
 
                 //#3 - shared network offering with no SG service
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DefaultSharedNetworkOffering) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultSharedNetworkOffering, "Offering for Shared networks", TrafficType.Guest, null, true,
                             Availability.Optional, null, defaultSharedNetworkOfferingProviders, true, Network.GuestType.Shared, false, null, true, null, true, false, null, false,
-                            null, true, false, false, null, null, true, null);
+                            null, true, false, false, false, null,null, null, true, null);
                 }
 
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DEFAULT_TUNGSTEN_SHARED_NETWORK_OFFERING_WITH_SGSERVICE) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DEFAULT_TUNGSTEN_SHARED_NETWORK_OFFERING_WITH_SGSERVICE, "Offering for Tungsten Shared Security group enabled networks",
                             TrafficType.Guest, null, true, Availability.Optional, null, defaultTungstenSharedSGEnabledNetworkOfferingProviders, true, Network.GuestType.Shared, false, null, true,
-                            null, true, false, null, false, null, true, false, true,null, null, true, null);
+                            null, true, false, null, false, null, true, false, true, false, null, null,null, true, null);
                     offering.setState(NetworkOffering.State.Enabled);
                     _networkOfferingDao.update(offering.getId(), offering);
                 }
@@ -566,14 +576,14 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultIsolatedNetworkOfferingWithSourceNatService,
                             "Offering for Isolated networks with Source Nat service enabled", TrafficType.Guest, null, false, Availability.Required, null,
                             defaultIsolatedSourceNatEnabledNetworkOfferingProviders, true, Network.GuestType.Isolated, false, null, true, null, false, false, null, false, null,
-                            true, false, false, null, null, true, null);
+                            true, false, false, false, null, null,null, true, null);
                 }
 
                 //#5 - default vpc offering with LB service
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DefaultIsolatedNetworkOfferingForVpcNetworks) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultIsolatedNetworkOfferingForVpcNetworks,
                             "Offering for Isolated VPC networks with Source Nat service enabled", TrafficType.Guest, null, false, Availability.Optional, null,
-                            defaultVPCOffProviders, true, Network.GuestType.Isolated, false, null, false, null, false, false, null, false, null, true, true, false, null, null, true, null);
+                            defaultVPCOffProviders, true, Network.GuestType.Isolated, false, null, false, null, false, false, null, false, null, true, true, false, false, null, null, null,true, null);
                 }
 
                 //#6 - default vpc offering with no LB service
@@ -582,14 +592,14 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                     defaultVPCOffProviders.remove(Service.Lb);
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultIsolatedNetworkOfferingForVpcNetworksNoLB,
                             "Offering for Isolated VPC networks with Source Nat service enabled and LB service disabled", TrafficType.Guest, null, false, Availability.Optional,
-                            null, defaultVPCOffProviders, true, Network.GuestType.Isolated, false, null, false, null, false, false, null, false, null, true, true, false, null, null, true, null);
+                            null, defaultVPCOffProviders, true, Network.GuestType.Isolated, false, null, false, null, false, false, null, false, null, true, true, false, false, null, null, null,true, null);
                 }
 
                 //#7 - isolated offering with source nat disabled
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DefaultIsolatedNetworkOffering) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultIsolatedNetworkOffering, "Offering for Isolated networks with no Source Nat service",
                             TrafficType.Guest, null, true, Availability.Optional, null, defaultIsolatedNetworkOfferingProviders, true, Network.GuestType.Isolated, false, null,
-                            true, null, true, false, null, false, null, true, false, false, null, null, true, null);
+                            true, null, true, false, null, false, null, true, false, false, false, null, null, null, true, null);
                 }
 
                 //#8 - network offering with internal lb service
@@ -611,7 +621,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DefaultIsolatedNetworkOfferingForVpcNetworksWithInternalLB) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultIsolatedNetworkOfferingForVpcNetworksWithInternalLB,
                             "Offering for Isolated VPC networks with Internal Lb support", TrafficType.Guest, null, false, Availability.Optional, null, internalLbOffProviders,
-                            true, Network.GuestType.Isolated, false, null, false, null, false, false, null, false, null, true, true, false, null, null, true, null);
+                            true, Network.GuestType.Isolated, false, null, false, null, false, false, null, false, null, true, true, false, false, null, null, null, true, null);
                     offering.setInternalLb(true);
                     offering.setPublicLb(false);
                     _networkOfferingDao.update(offering.getId(), offering);
@@ -642,7 +652,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                 if (_networkOfferingDao.findByUniqueName(NetworkOffering.DefaultSharedEIPandELBNetworkOffering) == null) {
                     offering = _configMgr.createNetworkOffering(NetworkOffering.DefaultSharedEIPandELBNetworkOffering,
                             "Offering for Shared networks with Elastic IP and Elastic LB capabilities", TrafficType.Guest, null, true, Availability.Optional, null,
-                            netscalerServiceProviders, true, Network.GuestType.Shared, false, null, true, serviceCapabilityMap, true, false, null, false, null, true, false, false, null, null, true, null);
+                            netscalerServiceProviders, true, Network.GuestType.Shared, false, null, true, serviceCapabilityMap, true, false, null, false, null, true, false, false, false, null, null, null, true, null);
                     offering.setDedicatedLB(false);
                     _networkOfferingDao.update(offering.getId(), offering);
                 }
@@ -738,20 +748,8 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                     .getBroadcastDomainType() == BroadcastDomainType.Vlan || predefined.getBroadcastDomainType() == BroadcastDomainType.Lswitch || predefined
                     .getBroadcastDomainType() == BroadcastDomainType.Vxlan)) {
                 final List<NetworkVO> configs = _networksDao.listBy(owner.getId(), offering.getId(), plan.getDataCenterId());
-                if (configs.size() > 0) {
+                if (!configs.isEmpty()) {
-                    if (logger.isDebugEnabled()) {
+                    return existingConfiguration(offering, configs, errorIfAlreadySetup);
-                        logger.debug("Found existing network configuration for offering " + offering + ": " + configs.get(0));
-                    }
-
-                    if (errorIfAlreadySetup) {
-                        final InvalidParameterValueException ex = new InvalidParameterValueException(
-                                "Found existing network configuration (with specified id) for offering (with specified id)");
-                        ex.addProxyObject(offering.getUuid(), "offeringId");
-                        ex.addProxyObject(configs.get(0).getUuid(), "networkConfigId");
-                        throw ex;
-                    } else {
-                        return configs;
-                    }
                 }
             }
 
@@ -760,7 +758,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
             long related = -1;
 
             for (final NetworkGuru guru : networkGurus) {
-                final Network network = guru.design(offering, plan, predefined, owner);
+                final Network network = guru.design(offering, plan, predefined, name, vpcId, owner);
                 if (network == null) {
                     continue;
                 }
@@ -783,11 +781,8 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                 Transaction.execute(new TransactionCallbackNoReturn() {
                     @Override
                     public void doInTransactionWithoutResult(final TransactionStatus status) {
-                        final NetworkVO vo = new NetworkVO(id, network, offering.getId(), guru.getName(), owner.getDomainId(), owner.getId(), relatedFile, name, displayText, predefined
+                        final NetworkVO vo = getNetworkVO(id, offering, plan, predefined,
-                                .getNetworkDomain(), offering.getGuestType(), plan.getDataCenterId(), plan.getPhysicalNetworkId(), aclType, offering.isSpecifyIpRanges(),
+                                network, guru, owner, name, displayText,relatedFile, aclType,vpcId, isDisplayNetworkEnabled);
-                                vpcId, offering.isRedundantRouter(), predefined.getExternalId());
-                        vo.setDisplayNetwork(isDisplayNetworkEnabled == null ? true : isDisplayNetworkEnabled);
-                        vo.setStrechedL2Network(offering.isSupportingStrechedL2());
                         final NetworkVO networkPersisted = _networksDao.persist(vo, vo.getGuestType() == Network.GuestType.Isolated,
                                 finalizeServicesAndProvidersForNetwork(offering, plan.getPhysicalNetworkId()));
                         networks.add(networkPersisted);
@@ -804,13 +799,14 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                         }
 
                         if (domainId != null && aclType == ACLType.Domain) {
-                            _networksDao.addDomainToNetwork(id, domainId, subdomainAccess == null ? true : subdomainAccess);
+                            _networksDao.addDomainToNetwork(id, domainId, subdomainAccess == null || subdomainAccess);
                         }
                     }
                 });
+                guru.setup(network, relatedFile);
             }
 
-            if (networks.size() < 1) {
+            if (networks.isEmpty()) {
                 // see networkOfferingVO.java
                 final CloudRuntimeException ex = new CloudRuntimeException("Unable to convert network offering with specified id to network profile");
                 ex.addProxyObject(offering.getUuid(), "offeringId");
@@ -824,6 +820,37 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
         }
     }
 
+    @NotNull
+    private static NetworkVO getNetworkVO(long id, final NetworkOffering offering, final DeploymentPlan plan, final Network predefined,
+                                          Network network, final NetworkGuru guru, final Account owner,
+                                          final String name, final String displayText, long relatedFile, final ACLType aclType,
+                                          final Long vpcId, final Boolean isDisplayNetworkEnabled) {
+        final NetworkVO vo = new NetworkVO(id, network, offering.getId(), guru.getName(), owner.getDomainId(), owner.getId(),
+                relatedFile, name, displayText, predefined.getNetworkDomain(), offering.getGuestType(),
+                plan.getDataCenterId(), plan.getPhysicalNetworkId(), aclType, offering.isSpecifyIpRanges(),
+                vpcId, offering.isRedundantRouter(), predefined.getExternalId());
+        vo.setDisplayNetwork(isDisplayNetworkEnabled == null || isDisplayNetworkEnabled);
+        vo.setStrechedL2Network(offering.isSupportingStrechedL2());
+        return vo;
+    }
+
+    private List<? extends Network> existingConfiguration(final NetworkOffering offering, List<NetworkVO> configs,
+                                                          final boolean errorIfAlreadySetup) {
+        if (logger.isDebugEnabled()) {
+            logger.debug("Found existing network configuration for offering " + offering + ": " + configs.get(0));
+        }
+
+        if (errorIfAlreadySetup) {
+            final InvalidParameterValueException ex = new InvalidParameterValueException(
+                    "Found existing network configuration (with specified id) for offering (with specified id)");
+            ex.addProxyObject(offering.getUuid(), "offeringId");
+            ex.addProxyObject(configs.get(0).getUuid(), "networkConfigId");
+            throw ex;
+        } else {
+            return configs;
+        }
+    }
+
     @Override
     @DB
     public void allocate(final VirtualMachineProfile vm, final LinkedHashMap<? extends Network, List<? extends NicProfile>> networks, final Map<String, Map<Integer, String>> extraDhcpOptions) throws InsufficientCapacityException,
@@ -1025,6 +1052,12 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
             return null;
         }
 
+        if (isNicAllocatedForNsxPublicNetworkOnVR(network, profile, vm)) {
+            String guruName = "NsxPublicNetworkGuru";
+            NetworkGuru nsxGuru = AdapterBase.getAdapterByName(networkGurus, guruName);
+            nsxGuru.allocate(network, profile, vm);
+        }
+
         if (isDefaultNic != null) {
             profile.setDefaultNic(isDefaultNic);
         }
@@ -1057,6 +1090,36 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
         return new Pair<NicProfile, Integer>(vmNic, Integer.valueOf(deviceId));
     }
 
+    private boolean isNicAllocatedForNsxPublicNetworkOnVR(Network network, NicProfile requested, VirtualMachineProfile vm) {
+        if (ObjectUtils.anyNull(network, requested, vm)) {
+            return false;
+        }
+        boolean isVirtualRouter = vm.getType() == Type.DomainRouter;
+        boolean isPublicTraffic = network.getTrafficType() == TrafficType.Public;
+        if (!isVirtualRouter || !isPublicTraffic || requested.getIPv4Address() == null) {
+            return false;
+        }
+        long dataCenterId = vm.getVirtualMachine().getDataCenterId();
+        if (nsxProviderDao.findByZoneId(dataCenterId) == null) {
+            return false;
+        }
+
+        Long vpcId = _ipAddressDao.findByIp(requested.getIPv4Address()).getVpcId();
+        List<IPAddressVO> ips = _ipAddressDao.listByAssociatedVpc(vpcId, true);
+
+        if (CollectionUtils.isEmpty(ips)) {
+            return false;
+        }
+        ips = ips.stream().filter(x -> !x.getAddress().addr().equals(requested.getIPv4Address())).collect(Collectors.toList());
+        IPAddressVO ip = ips.get(0);
+        VlanDetailsVO vlanDetail = vlanDetailsDao.findDetail(ip.getVlanId(), ApiConstants.NSX_DETAIL_KEY);
+        if (vlanDetail == null) {
+            return false;
+        }
+        boolean isForNsx = vlanDetail.getValue().equalsIgnoreCase("true");
+        return isForNsx && !ip.isForSystemVms();
+    }
+
     private void setMtuDetailsInVRNic(final Pair<NetworkVO, VpcVO> networks, Network network, NicVO vo) {
         if (TrafficType.Public == network.getTrafficType()) {
             if (networks == null) {
@@ -2814,10 +2877,9 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
         }
 
         // Check if cidr is RFC1918 compliant if the network is Guest Isolated for IPv4
-        if (cidr != null && ntwkOff.getGuestType() == Network.GuestType.Isolated && ntwkOff.getTrafficType() == TrafficType.Guest) {
+        if (cidr != null && (ntwkOff.getGuestType() == Network.GuestType.Isolated && ntwkOff.getTrafficType() == TrafficType.Guest) &&
-            if (!NetUtils.validateGuestCidr(cidr)) {
+                !NetUtils.validateGuestCidr(cidr, !ConfigurationManager.AllowNonRFC1918CompliantIPs.value())) {
                 throw new InvalidParameterValueException("Virtual Guest Cidr " + cidr + " is not RFC 1918 or 6598 compliant");
-            }
         }
 
         final String networkDomainFinal = networkDomain;
@@ -3844,7 +3906,8 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
 
     private boolean cleanupNetworkResources(final long networkId, final Account caller, final long callerUserId) {
         boolean success = true;
-        final Network network = _networksDao.findById(networkId);
+        final NetworkVO network = _networksDao.findById(networkId);
+        final NetworkOfferingVO networkOffering= _networkOfferingDao.findById(network.getNetworkOfferingId());
 
         //remove all PF/Static Nat rules for the network
         try {
@@ -4724,6 +4787,6 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
         return new ConfigKey<?>[]{NetworkGcWait, NetworkGcInterval, NetworkLockTimeout,
                 GuestDomainSuffix, NetworkThrottlingRate, MinVRVersion,
                 PromiscuousMode, MacAddressChanges, ForgedTransmits, MacLearning, RollingRestartEnabled,
-                TUNGSTEN_ENABLED };
+                TUNGSTEN_ENABLED, NSX_ENABLED };
     }
 }

engine/orchestration/src/test/java/com/cloud/vm/VirtualMachineManagerImplTest.java

@@ -39,6 +39,18 @@ import java.util.Map;
 import java.util.Random;
 import java.util.stream.Collectors;
 
+import com.cloud.agent.api.to.VirtualMachineTO;
+import com.cloud.api.query.vo.UserVmJoinVO;
+import com.cloud.dc.DataCenterVO;
+import com.cloud.dc.dao.DataCenterDao;
+import com.cloud.domain.DomainVO;
+import com.cloud.domain.dao.DomainDao;
+import com.cloud.network.dao.NetworkDao;
+import com.cloud.network.dao.NetworkVO;
+import com.cloud.network.vpc.VpcVO;
+import com.cloud.network.vpc.dao.VpcDao;
+import com.cloud.user.AccountVO;
+import com.cloud.user.dao.AccountDao;
 import org.apache.cloudstack.context.CallContext;
 import org.apache.cloudstack.engine.subsystem.api.storage.StoragePoolAllocator;
 import org.apache.cloudstack.framework.config.ConfigKey;
@@ -182,6 +194,16 @@ public class VirtualMachineManagerImplTest {
     @Mock
     private UserVmVO userVmMock;
     @Mock
+    private NetworkDao networkDao;
+    @Mock
+    private AccountDao accountDao;
+    @Mock
+    private DomainDao domainDao;
+    @Mock
+    private DataCenterDao dcDao;
+    @Mock
+    private VpcDao vpcDao;
+    @Mock
     private EntityManager _entityMgr;
     @Mock
     private DeploymentPlanningManager _dpMgr;
@@ -938,6 +960,48 @@ public class VirtualMachineManagerImplTest {
         virtualMachineManagerImpl.checkAndAttemptMigrateVmAcrossCluster(vm, destinationClusterId, map);
     }
 
+    @Test
+    public void checkIfVmNetworkDetailsReturnedIsCorrect() {
+        VMInstanceVO vm = new VMInstanceVO(1L, 1L, "VM1", "i-2-2-VM",
+                VirtualMachine.Type.User, 1L, HypervisorType.KVM, 1L, 1L, 1L,
+                1L, false, false);
+
+        VirtualMachineTO vmTO = new VirtualMachineTO() {
+        };
+        UserVmJoinVO userVm = new UserVmJoinVO();
+        NetworkVO networkVO = mock(NetworkVO.class);
+        AccountVO accountVO = mock(AccountVO.class);
+        DomainVO domainVO = mock(DomainVO.class);
+        domainVO.setName("testDomain");
+        DataCenterVO dataCenterVO = mock(DataCenterVO.class);
+        VpcVO vpcVO = mock(VpcVO.class);
+
+        networkVO.setAccountId(1L);
+        networkVO.setName("testNet");
+        networkVO.setVpcId(1L);
+
+        accountVO.setAccountName("testAcc");
+
+        vpcVO.setName("VPC1");
+
+
+        List<UserVmJoinVO> userVms = List.of(userVm);
+        Mockito.when(userVmJoinDaoMock.searchByIds(anyLong())).thenReturn(userVms);
+        Mockito.when(networkDao.findById(anyLong())).thenReturn(networkVO);
+        Mockito.when(accountDao.findById(anyLong())).thenReturn(accountVO);
+        Mockito.when(domainDao.findById(anyLong())).thenReturn(domainVO);
+        Mockito.when(dcDao.findById(anyLong())).thenReturn(dataCenterVO);
+        Mockito.when(vpcDao.findById(anyLong())).thenReturn(vpcVO);
+        Mockito.when(dataCenterVO.getId()).thenReturn(1L);
+        when(accountVO.getId()).thenReturn(2L);
+        Mockito.when(domainVO.getId()).thenReturn(3L);
+        Mockito.when(vpcVO.getId()).thenReturn(4L);
+        Mockito.when(networkVO.getId()).thenReturn(5L);
+        virtualMachineManagerImpl.setVmNetworkDetails(vm, vmTO);
+        assertEquals(1, vmTO.getNetworkIdToNetworkNameMap().size());
+        assertEquals("D3-A2-Z1-V4-S5", vmTO.getNetworkIdToNetworkNameMap().get(5L));
+    }
+
     @Test
     public void testOrchestrateStartNonNullPodId() throws Exception {
         VMInstanceVO vmInstance = new VMInstanceVO();

engine/schema/src/main/java/com/cloud/network/dao/NetworkVO.java

@@ -367,6 +367,10 @@ public class NetworkVO implements Network {
         return mode;
     }
 
+    public void setAccountId(long accountId) {
+        this.accountId = accountId;
+    }
+
     @Override
     public long getAccountId() {
         return accountId;

engine/schema/src/main/java/com/cloud/network/dao/NsxProviderDao.java

@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.network.dao;
+
+import com.cloud.network.element.NsxProviderVO;
+import com.cloud.utils.db.GenericDao;
+
+import java.util.List;
+
+public interface NsxProviderDao extends GenericDao<NsxProviderVO, Long> {
+    NsxProviderVO findByZoneId(long zoneId);
+
+    NsxProviderVO findByUuid(String uuid);
+
+    List<NsxProviderVO> findAll();
+}

engine/schema/src/main/java/com/cloud/network/dao/NsxProviderDaoImpl.java

@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.network.dao;
+
+import com.cloud.network.element.NsxProviderVO;
+import com.cloud.utils.db.DB;
+import com.cloud.utils.db.GenericDaoBase;
+import com.cloud.utils.db.SearchBuilder;
+import com.cloud.utils.db.SearchCriteria;
+import org.springframework.stereotype.Component;
+
+import java.util.List;
+
+@Component
+@DB()
+public class NsxProviderDaoImpl extends GenericDaoBase<NsxProviderVO, Long>
+        implements NsxProviderDao {
+
+    final SearchBuilder<NsxProviderVO> allFieldsSearch;
+
+    public NsxProviderDaoImpl() {
+        super();
+        allFieldsSearch = createSearchBuilder();
+        allFieldsSearch.and("id", allFieldsSearch.entity().getId(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.and("uuid", allFieldsSearch.entity().getUuid(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.and("hostname", allFieldsSearch.entity().getHostname(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.and("provider_name", allFieldsSearch.entity().getProviderName(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.and("tier0_gateway", allFieldsSearch.entity().getTier0Gateway(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.and("zone_id", allFieldsSearch.entity().getZoneId(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.and("edge_cluster", allFieldsSearch.entity().getEdgeCluster(),
+                SearchCriteria.Op.EQ);
+        allFieldsSearch.done();
+    }
+    @Override
+    public NsxProviderVO findByZoneId(long zoneId) {
+        SearchCriteria<NsxProviderVO> sc = allFieldsSearch.create();
+        sc.setParameters("zone_id", zoneId);
+        return findOneBy(sc);
+    }
+
+    @Override
+    public List<NsxProviderVO> findAll() {
+        return listAll();
+    }
+}

engine/schema/src/main/java/com/cloud/network/element/NsxProviderVO.java

@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.network.element;
+
+import com.cloud.network.nsx.NsxProvider;
+import com.cloud.utils.db.Encrypt;
+
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.Id;
+import javax.persistence.Table;
+import java.util.Date;
+import java.util.UUID;
+
+@Entity
+@Table(name = "nsx_providers")
+public class NsxProviderVO implements NsxProvider {
+
+    @Id
+    @GeneratedValue(strategy = GenerationType.IDENTITY)
+    @Column(name = "id")
+    long id;
+
+    @Column(name = "zone_id")
+    private long zoneId;
+
+    @Column(name = "host_id")
+    private long hostId;
+
+    @Column(name = "uuid")
+    private String uuid;
+
+    @Column(name = "provider_name")
+    private String providerName;
+
+    @Column(name = "hostname")
+    private String hostname;
+
+    @Column(name = "port")
+    private String port = "443";
+
+    @Column(name = "username")
+    private String username;
+
+    @Encrypt
+    @Column(name = "password")
+    private String password;
+
+    @Column(name = "tier0_gateway")
+    private String tier0Gateway;
+
+    @Column(name = "edge_cluster")
+    private String edgeCluster;
+
+    @Column(name = "transport_zone")
+    private String transportZone;
+
+    @Column(name = "created")
+    private Date created;
+
+    @Column(name = "removed")
+    private Date removed;
+    public NsxProviderVO() {
+        this.uuid = UUID.randomUUID().toString();
+    }
+
+    @Override
+    public long getId() {
+        return id;
+    }
+
+    public void setId(long id) {
+        this.id = id;
+    }
+
+    @Override
+    public long getZoneId() {
+        return zoneId;
+    }
+
+    public void setZoneId(long zoneId) {
+        this.zoneId = zoneId;
+    }
+
+    public long getHostId() {
+        return hostId;
+    }
+
+    public void setHostId(long hostId) {
+        this.hostId = hostId;
+    }
+
+    @Override
+    public String getUuid() {
+        return uuid;
+    }
+
+    public void setUuid(String uuid) {
+        this.uuid = uuid;
+    }
+
+    @Override
+    public String getProviderName() {
+        return providerName;
+    }
+
+    public void setProviderName(String providerName) {
+        this.providerName = providerName;
+    }
+
+    @Override
+    public String getHostname() {
+        return hostname;
+    }
+
+    public void setPort(String port) {
+        this.port = port;
+    }
+
+    @Override
+    public String getPort() {
+        return port;
+    }
+
+    public void setHostname(String hostname) {
+        this.hostname = hostname;
+    }
+
+    @Override
+    public String getUsername() {
+        return username;
+    }
+
+    public void setUsername(String username) {
+        this.username = username;
+    }
+
+    public String getPassword() {
+        return password;
+    }
+
+    public void setPassword(String password) {
+        this.password = password;
+    }
+
+    public String getTier0Gateway() {
+        return tier0Gateway;
+    }
+
+    public void setTier0Gateway(String tier0Gateway) {
+        this.tier0Gateway = tier0Gateway;
+    }
+
+    public String getEdgeCluster() {
+        return edgeCluster;
+    }
+
+    public void setEdgeCluster(String edgeCluster) {
+        this.edgeCluster = edgeCluster;
+    }
+
+    public String getTransportZone() {
+        return transportZone;
+    }
+
+    public void setTransportZone(String transportZone) {
+        this.transportZone = transportZone;
+    }
+
+    public Date getCreated() {
+        return created;
+    }
+
+    public void setCreated(Date created) {
+        this.created = created;
+    }
+
+    public Date getRemoved() {
+        return removed;
+    }
+
+    public void setRemoved(Date removed) {
+        this.removed = removed;
+    }
+
+    public static final class Builder {
+        private long zoneId;
+        private long hostId;
+        private String providerName;
+        private String hostname;
+        private String port;
+        private String username;
+        private String password;
+        private String tier0Gateway;
+        private String edgeCluster;
+        private String transportZone;
+
+
+        public Builder() {
+            // Default constructor
+        }
+
+        public Builder setZoneId(long zoneId) {
+            this.zoneId = zoneId;
+            return this;
+        }
+
+        public Builder setHostId(long hostId) {
+            this.hostId = hostId;
+            return this;
+        }
+
+        public Builder setProviderName(String providerName) {
+            this.providerName = providerName;
+            return this;
+        }
+
+        public Builder setHostname(String hostname) {
+            this.hostname = hostname;
+            return this;
+        }
+
+        public Builder setPort(String port) {
+            this.port = port;
+            return this;
+        }
+
+        public Builder setUsername(String username) {
+            this.username = username;
+            return this;
+        }
+
+        public Builder setPassword(String password) {
+            this.password = password;
+            return this;
+        }
+
+        public Builder setTier0Gateway(String tier0Gateway) {
+            this.tier0Gateway = tier0Gateway;
+            return this;
+        }
+
+        public Builder setEdgeCluster(String edgeCluster) {
+            this.edgeCluster = edgeCluster;
+            return this;
+        }
+
+        public Builder setTransportZone(String transportZone) {
+            this.transportZone = transportZone;
+            return this;
+        }
+        public NsxProviderVO build() {
+            NsxProviderVO provider = new NsxProviderVO();
+            provider.setZoneId(this.zoneId);
+            provider.setHostId(this.hostId);
+            provider.setUuid(UUID.randomUUID().toString());
+            provider.setProviderName(this.providerName);
+            provider.setHostname(this.hostname);
+            provider.setPort(this.port);
+            provider.setUsername(this.username);
+            provider.setPassword(this.password);
+            provider.setTier0Gateway(this.tier0Gateway);
+            provider.setEdgeCluster(this.edgeCluster);
+            provider.setTransportZone(this.transportZone);
+            provider.setCreated(new Date());
+            return provider;
+        }
+    }
+}

engine/schema/src/main/java/com/cloud/network/vpc/VpcOfferingVO.java

@@ -58,6 +58,12 @@ public class VpcOfferingVO implements VpcOffering {
     @Column(name = "default")
     boolean isDefault = false;
 
+    @Column(name = "for_nsx")
+    boolean forNsx = false;
+
+    @Column(name = "nsx_mode")
+    String nsxMode;
+
     @Column(name = GenericDao.REMOVED_COLUMN)
     Date removed;
 
@@ -144,6 +150,22 @@ public class VpcOfferingVO implements VpcOffering {
         return isDefault;
     }
 
+    public boolean isForNsx() {
+        return forNsx;
+    }
+
+    public void setForNsx(boolean forNsx) {
+        this.forNsx = forNsx;
+    }
+
+    public String getNsxMode() {
+        return nsxMode;
+    }
+
+    public void setNsxMode(String nsxMode) {
+        this.nsxMode =  nsxMode;
+    }
+
     public void setUniqueName(String uniqueName) {
         this.uniqueName = uniqueName;
     }

engine/schema/src/main/java/com/cloud/offerings/NetworkOfferingVO.java

@@ -136,6 +136,12 @@ public class NetworkOfferingVO implements NetworkOffering {
     @Column(name = "for_tungsten")
     boolean forTungsten = false;
 
+    @Column(name = "for_nsx")
+    boolean forNsx = false;
+
+    @Column(name = "nsx_mode")
+    String nsxMode;
+
     @Column(name = "egress_default_policy")
     boolean egressdefaultpolicy;
 
@@ -195,6 +201,24 @@ public class NetworkOfferingVO implements NetworkOffering {
         this.forTungsten = forTungsten;
     }
 
+    @Override
+    public boolean isForNsx() {
+        return forNsx;
+    }
+
+    public void setForNsx(boolean forNsx) {
+        this.forNsx = forNsx;
+    }
+
+    @Override
+    public String getNsxMode() {
+        return nsxMode;
+    }
+
+    public void setNsxMode(String nsxMode) {
+        this.nsxMode = nsxMode;
+    }
+
     @Override
     public long getId() {
         return id;

engine/schema/src/main/java/com/cloud/vm/dao/NicDao.java

@@ -44,6 +44,8 @@ public interface NicDao extends GenericDao<NicVO, Long> {
 
     NicVO findByNetworkIdAndType(long networkId, VirtualMachine.Type vmType);
 
+    NicVO findByNetworkIdAndTypeIncludingRemoved(long networkId, VirtualMachine.Type vmType);
+
     NicVO findByIp4AddressAndNetworkId(String ip4Address, long networkId);
 
     NicVO findByNetworkIdAndMacAddress(long networkId, String mac);

engine/schema/src/main/java/com/cloud/vm/dao/NicDaoImpl.java

@@ -176,12 +176,21 @@ public class NicDaoImpl extends GenericDaoBase<NicVO, Long> implements NicDao {
         return findOneIncludingRemovedBy(sc);
     }
 
-    @Override
+    private NicVO findByNetworkIdAndTypeInternal(long networkId, VirtualMachine.Type vmType, boolean includingRemoved) {
-    public NicVO findByNetworkIdAndType(long networkId, VirtualMachine.Type vmType) {
         SearchCriteria<NicVO> sc = AllFieldsSearch.create();
         sc.setParameters("network", networkId);
         sc.setParameters("vmType", vmType);
-        return findOneBy(sc);
+        return includingRemoved ? findOneIncludingRemovedBy(sc) : findOneBy(sc);
+    }
+
+    @Override
+    public NicVO findByNetworkIdAndType(long networkId, VirtualMachine.Type vmType) {
+        return findByNetworkIdAndTypeInternal(networkId, vmType, false);
+    }
+
+    @Override
+    public NicVO findByNetworkIdAndTypeIncludingRemoved(long networkId, VirtualMachine.Type vmType) {
+        return findByNetworkIdAndTypeInternal(networkId, vmType, true);
     }
 
     @Override

engine/schema/src/main/resources/META-INF/cloudstack/core/spring-engine-schema-core-daos-context.xml

@@ -136,6 +136,7 @@
   <bean id="nicIpAliasDaoImpl" class="com.cloud.vm.dao.NicIpAliasDaoImpl" />
   <bean id="objectInDataStoreDaoImpl" class="org.apache.cloudstack.storage.db.ObjectInDataStoreDaoImpl" />
   <bean id="ovsProviderDaoImpl" class="com.cloud.network.dao.OvsProviderDaoImpl" />
+  <bean id="nsxControllerDaoImpl" class="com.cloud.network.dao.NsxProviderDaoImpl" />
   <bean id="tungstenControllerDaoImpl" class="com.cloud.network.dao.TungstenProviderDaoImpl"/>
   <bean id="physicalNetworkDaoImpl" class="com.cloud.network.dao.PhysicalNetworkDaoImpl" />
   <bean id="physicalNetworkIsolationMethodDaoImpl" class="com.cloud.network.dao.PhysicalNetworkIsolationMethodDaoImpl" />

engine/schema/src/main/resources/META-INF/db/schema-41900to42000.sql

@@ -33,4 +33,32 @@ ADD UNIQUE INDEX `i_resource_count__type_tag_domaintId` (`type`,`tag`,`domain_id
 UPDATE `cloud`.`service_offering` SET ram_size = 512 WHERE unique_name IN ("Cloud.Com-SoftwareRouter", "Cloud.Com-SoftwareRouter-Local",
                                                                            "Cloud.Com-InternalLBVm", "Cloud.Com-InternalLBVm-Local",
                                                                            "Cloud.Com-ElasticLBVm", "Cloud.Com-ElasticLBVm-Local")
-                                                    AND system_use = 1 AND ram_size < 512;
+                                                       AND system_use = 1 AND ram_size < 512;
+
+-- NSX Plugin --
+CREATE TABLE `cloud`.`nsx_providers` (
+                                         `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
+                                         `uuid` varchar(40),
+                                         `zone_id` bigint unsigned NOT NULL COMMENT 'Zone ID',
+                                         `host_id` bigint unsigned NOT NULL COMMENT 'Host ID',
+                                         `provider_name` varchar(40),
+                                         `hostname` varchar(255) NOT NULL,
+                                         `port` varchar(255),
+                                         `username` varchar(255) NOT NULL,
+                                         `password` varchar(255) NOT NULL,
+                                         `tier0_gateway` varchar(255),
+                                         `edge_cluster` varchar(255),
+                                         `transport_zone` varchar(255),
+                                         `created` datetime NOT NULL COMMENT 'date created',
+                                         `removed` datetime COMMENT 'date removed if not null',
+                                         PRIMARY KEY (`id`),
+                                         CONSTRAINT `fk_nsx_providers__zone_id` FOREIGN KEY `fk_nsx_providers__zone_id` (`zone_id`) REFERENCES `data_center`(`id`) ON DELETE CASCADE,
+                                         INDEX `i_nsx_providers__zone_id`(`zone_id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+-- NSX Plugin --
+CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.network_offerings','for_nsx', 'int(1) unsigned DEFAULT "0" COMMENT "is nsx enabled for the resource"');
+CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.network_offerings','nsx_mode', 'varchar(32) COMMENT "mode in which the network would route traffic"');
+CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.vpc_offerings','for_nsx', 'int(1) unsigned DEFAULT "0" COMMENT "is nsx enabled for the resource"');
+CALL `cloud`.`IDEMPOTENT_ADD_COLUMN`('cloud.vpc_offerings','nsx_mode', 'varchar(32) COMMENT "mode in which the network would route traffic"');
+

engine/schema/src/main/resources/META-INF/db/views/cloud.network_offering_view.sql

@@ -60,6 +60,8 @@ SELECT
     `network_offerings`.`supports_vm_autoscaling` AS `supports_vm_autoscaling`,
     `network_offerings`.`for_vpc` AS `for_vpc`,
     `network_offerings`.`for_tungsten` AS `for_tungsten`,
+    `network_offerings`.`for_nsx` AS `for_nsx`,
+    `network_offerings`.`nsx_mode` AS `nsx_mode`,
     `network_offerings`.`service_package_id` AS `service_package_id`,
     GROUP_CONCAT(DISTINCT(domain.id)) AS domain_id,
     GROUP_CONCAT(DISTINCT(domain.uuid)) AS domain_uuid,

engine/schema/src/main/resources/META-INF/db/views/cloud.vpc_offering_view.sql

@@ -17,7 +17,6 @@
 
 -- cloud.vpc_offering_view source
 
-
 DROP VIEW IF EXISTS `cloud`.`vpc_offering_view`;
 
 CREATE VIEW `cloud`.`vpc_offering_view` AS
@@ -29,6 +28,8 @@ select
     `vpc_offerings`.`display_text` AS `display_text`,
     `vpc_offerings`.`state` AS `state`,
     `vpc_offerings`.`default` AS `default`,
+    `vpc_offerings`.`for_nsx` AS `for_nsx`,
+    `vpc_offerings`.`nsx_mode` AS `nsx_mode`,
     `vpc_offerings`.`created` AS `created`,
     `vpc_offerings`.`removed` AS `removed`,
     `vpc_offerings`.`service_offering_id` AS `service_offering_id`,

plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/manager/VmwareManagerImpl.java

@@ -462,7 +462,7 @@ public class VmwareManagerImpl extends ManagerBase implements VmwareManager, Vmw
                 }
             }
             HypervisorHostHelper.prepareNetwork(vSwitchName, "cloud.private", hostMo, vlanId, null, null, null, 180000,
-                    vsType, portsPerDvPortGroup, null, false, BroadcastDomainType.Vlan, null, null);
+                    vsType, portsPerDvPortGroup, null, false, BroadcastDomainType.Vlan, null, null, null);
         }
     }
 

plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

@@ -730,7 +730,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
         HostMO hostMO = new HostMO(context, host.getMor());
 
         try {
-            prepareNetworkFromNicInfo(hostMO, cmd.getNic(), false, null);
+            prepareNetworkFromNicInfo(hostMO, cmd.getNic(), false, null, null);
             hostname =  host.getHyperHostName();
         } catch (Exception e) {
             return new SetupPersistentNetworkAnswer(cmd, false, "failed to setup port-group due to: "+ e.getLocalizedMessage());
@@ -1472,7 +1472,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
         deviceNumber++;
 
         VirtualDevice nic;
-        Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, false, vmType);
+        Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, false, nicTo.getNetworkSegmentName(), vmType);
         String dvSwitchUuid = null;
         if (VmwareHelper.isDvPortGroup(networkInfo.first())) {
             ManagedObjectReference dcMor = hyperHost.getHyperHostDatacenter();
@@ -1534,7 +1534,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
                 return new ReplugNicAnswer(cmd, false, "Nic to replug not found");
             }
 
-            Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, false, cmd.getVMType());
+            Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, false, null, cmd.getVMType());
             String dvSwitchUuid = null;
             if (VmwareHelper.isDvPortGroup(networkInfo.first())) {
                 ManagedObjectReference dcMor = hyperHost.getHyperHostDatacenter();
@@ -1616,7 +1616,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
         } else {
             networkInfo =
                     HypervisorHostHelper.prepareNetwork(_publicTrafficInfo.getVirtualSwitchName(), "cloud.public", vmMo.getRunningHost(), vlanId, null, ipAddressTO.getNetworkRate(), null,
-                            _opsTimeout, vSwitchType, _portsPerDvPortGroup, null, false, BroadcastDomainType.Vlan, _vsmCredentials, null);
+                            _opsTimeout, vSwitchType, _portsPerDvPortGroup, null, false, BroadcastDomainType.Vlan, _vsmCredentials, null, null);
         }
 
         int nicIndex = allocPublicNicIndex(vmMo);
@@ -2525,7 +2525,8 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
                 }
                 boolean configureVServiceInNexus = (nicTo.getType() == TrafficType.Guest) && (vmSpec.getDetails().containsKey("ConfigureVServiceInNexus"));
                 VirtualMachine.Type vmType = cmd.getVirtualMachine().getType();
-                Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, configureVServiceInNexus, vmType);
+                Pair<ManagedObjectReference, String> networkInfo = prepareNetworkFromNicInfo(vmMo.getRunningHost(), nicTo, configureVServiceInNexus,
+                        vmSpec.getNetworkIdToNetworkNameMap().getOrDefault(nicTo.getNetworkId(), null), vmType);
                 if ((nicTo.getBroadcastType() != BroadcastDomainType.Lswitch)
                         || (nicTo.getBroadcastType() == BroadcastDomainType.Lswitch && NiciraNvpApiVersion.isApiVersionLowerThan("4.2"))) {
                     if (VmwareHelper.isDvPortGroup(networkInfo.first())) {
@@ -3981,7 +3982,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
         return defaultVlan;
     }
 
-    private Pair<ManagedObjectReference, String> prepareNetworkFromNicInfo(HostMO hostMo, NicTO nicTo, boolean configureVServiceInNexus, VirtualMachine.Type vmType)
+    private Pair<ManagedObjectReference, String> prepareNetworkFromNicInfo(HostMO hostMo, NicTO nicTo, boolean configureVServiceInNexus, String networkName, VirtualMachine.Type vmType)
             throws Exception {
 
         Ternary<String, String, String> switchDetails = getTargetSwitch(nicTo);
@@ -4011,7 +4012,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
             }
             networkInfo = HypervisorHostHelper.prepareNetwork(switchName, namePrefix, hostMo, vlanId, svlanId,
                     nicTo.getNetworkRateMbps(), nicTo.getNetworkRateMulticastMbps(), _opsTimeout, switchType,
-                    _portsPerDvPortGroup, nicTo.getGateway(), configureVServiceInNexus, nicTo.getBroadcastType(), _vsmCredentials, nicTo.getDetails());
+                    _portsPerDvPortGroup, nicTo.getGateway(), configureVServiceInNexus, nicTo.getBroadcastType(), _vsmCredentials, nicTo.getDetails(), networkName);
         }
 
         return networkInfo;
@@ -4602,7 +4603,8 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
             NicTO[] nics = vm.getNics();
             for (NicTO nic : nics) {
                 // prepare network on the host
-                prepareNetworkFromNicInfo(new HostMO(getServiceContext(), _morHyperHost), nic, false, cmd.getVirtualMachine().getType());
+                prepareNetworkFromNicInfo(new HostMO(getServiceContext(), _morHyperHost), nic, false,
+                        vm.getNetworkIdToNetworkNameMap().getOrDefault(nic.getNetworkId(), null), cmd.getVirtualMachine().getType());
             }
 
             List<Pair<String, Long>> secStoreUrlAndIdList = mgr.getSecondaryStorageStoresUrlAndIdList(Long.parseLong(_dcId));
@@ -5680,7 +5682,7 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
     }
 
     protected Answer execute(MaintainCommand cmd) {
-        return new MaintainAnswer(cmd, "Put host in maintaince");
+        return new MaintainAnswer(cmd, "Put host in maintenance");
     }
 
     protected Answer execute(PingTestCommand cmd) {
@@ -7326,7 +7328,8 @@ public class VmwareResource extends ServerResourceBase implements StoragePoolRes
                 NicTO[] nics = vmTo.getNics();
                 for (NicTO nic : nics) {
                     // prepare network on the host
-                    prepareNetworkFromNicInfo((HostMO)targetHyperHost, nic, false, vmTo.getType());
+                    prepareNetworkFromNicInfo((HostMO)targetHyperHost, nic, false,
+                            vmTo.getNetworkIdToNetworkNameMap().get(nic.getNetworkId()), vmTo.getType());
                 }
 
                 if (targetHyperHost == null) {

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterHelperImpl.java

@@ -17,6 +17,7 @@
 package com.cloud.kubernetes.cluster;
 
 import com.cloud.kubernetes.cluster.dao.KubernetesClusterDao;
+import com.cloud.kubernetes.cluster.dao.KubernetesClusterVmMapDao;
 import com.cloud.utils.component.AdapterBase;
 import org.apache.cloudstack.acl.ControlledEntity;
 import org.apache.cloudstack.framework.config.ConfigKey;
@@ -24,18 +25,30 @@ import org.apache.cloudstack.framework.config.Configurable;
 import org.springframework.stereotype.Component;
 
 import javax.inject.Inject;
+import java.util.Objects;
 
 @Component
 public class KubernetesClusterHelperImpl extends AdapterBase implements KubernetesClusterHelper, Configurable {
 
     @Inject
     private KubernetesClusterDao kubernetesClusterDao;
+    @Inject
+    private KubernetesClusterVmMapDao kubernetesClusterVmMapDao;
 
     @Override
     public ControlledEntity findByUuid(String uuid) {
         return kubernetesClusterDao.findByUuid(uuid);
     }
 
+    @Override
+    public ControlledEntity findByVmId(long vmId) {
+        KubernetesClusterVmMapVO clusterVmMapVO = kubernetesClusterVmMapDao.getClusterMapFromVmId(vmId);
+        if (Objects.isNull(clusterVmMapVO)) {
+            return null;
+        }
+        return kubernetesClusterDao.findById(clusterVmMapVO.getClusterId());
+    }
+
     @Override
     public String getConfigComponentName() {
         return KubernetesClusterHelper.class.getSimpleName();

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java

@@ -182,6 +182,11 @@ import org.apache.logging.log4j.Level;
 public class KubernetesClusterManagerImpl extends ManagerBase implements KubernetesClusterService {
 
     private static final String DEFAULT_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME = "DefaultNetworkOfferingforKubernetesService";
+    private static final String DEFAULT_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_DISPLAY_TEXT = "Network Offering used for CloudStack Kubernetes service";
+    private static final String DEFAULT_NSX_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME = "DefaultNSXNetworkOfferingforKubernetesService";
+    private static final String DEFAULT_NSX_VPC_TIER_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME = "DefaultNSXVPCNetworkOfferingforKubernetesService";
+    private static final String DEFAULT_NSX_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_DISPLAY_TEXT = "Network Offering for NSX CloudStack Kubernetes Service";
+    private static final String DEFAULT_NSX_VPC_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_DISPLAY_TEXT = "Network Offering for NSX CloudStack Kubernetes service on VPC";
 
     protected StateMachine2<KubernetesCluster.State, KubernetesCluster.Event, KubernetesCluster> _stateMachine = KubernetesCluster.State.getStateMachine();
 
@@ -1893,26 +1898,54 @@ public class KubernetesClusterManagerImpl extends ManagerBase implements Kuberne
 
     @Override
     public boolean start() {
+        createNetworkOfferingForKubernetes(DEFAULT_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME,
+                DEFAULT_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_DISPLAY_TEXT, false, false);
+
+        createNetworkOfferingForKubernetes(DEFAULT_NSX_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME,
+                DEFAULT_NSX_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_DISPLAY_TEXT, true, false);
+
+        createNetworkOfferingForKubernetes(DEFAULT_NSX_VPC_TIER_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME,
+                DEFAULT_NSX_VPC_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_DISPLAY_TEXT , true, true);
+
+        _gcExecutor.scheduleWithFixedDelay(new KubernetesClusterGarbageCollector(), 300, 300, TimeUnit.SECONDS);
+        _stateScanner.scheduleWithFixedDelay(new KubernetesClusterStatusScanner(), 300, 30, TimeUnit.SECONDS);
+
+        return true;
+    }
+
+    private void createNetworkOfferingForKubernetes(String offeringName, String offeringDesc, boolean forNsx, boolean forVpc) {
         final Map<Network.Service, Network.Provider> defaultKubernetesServiceNetworkOfferingProviders = new HashMap<Service, Network.Provider>();
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Dhcp, Network.Provider.VirtualRouter);
+        Network.Provider provider = forVpc ? Network.Provider.VPCVirtualRouter : Network.Provider.VirtualRouter;
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Dns, Network.Provider.VirtualRouter);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Dhcp, provider);
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.UserData, Network.Provider.VirtualRouter);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Dns, provider);
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Firewall, Network.Provider.VirtualRouter);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.UserData, provider);
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Gateway, Network.Provider.VirtualRouter);
+        if (forVpc) {
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Lb, Network.Provider.VirtualRouter);
+            defaultKubernetesServiceNetworkOfferingProviders.put(Service.NetworkACL, forNsx ? Network.Provider.Nsx : provider);
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.SourceNat, Network.Provider.VirtualRouter);
+        } else {
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.StaticNat, Network.Provider.VirtualRouter);
+            defaultKubernetesServiceNetworkOfferingProviders.put(Service.Firewall, forNsx ? Network.Provider.Nsx : provider);
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.PortForwarding, Network.Provider.VirtualRouter);
+        }
-        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Vpn, Network.Provider.VirtualRouter);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.Lb, forNsx ? Network.Provider.Nsx : provider);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.SourceNat, forNsx ? Network.Provider.Nsx : provider);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.StaticNat, forNsx ? Network.Provider.Nsx : provider);
+        defaultKubernetesServiceNetworkOfferingProviders.put(Service.PortForwarding, forNsx ? Network.Provider.Nsx : provider);
+
+        if (!forNsx) {
+            defaultKubernetesServiceNetworkOfferingProviders.put(Service.Gateway, Network.Provider.VirtualRouter);
+            defaultKubernetesServiceNetworkOfferingProviders.put(Service.Vpn, Network.Provider.VirtualRouter);
+        }
 
         NetworkOfferingVO defaultKubernetesServiceNetworkOffering =
-                new NetworkOfferingVO(DEFAULT_NETWORK_OFFERING_FOR_KUBERNETES_SERVICE_NAME,
+                new NetworkOfferingVO(offeringName,
-                        "Network Offering used for CloudStack Kubernetes service", Networks.TrafficType.Guest,
+                        offeringDesc, Networks.TrafficType.Guest,
                         false, false, null, null, true,
                         NetworkOffering.Availability.Required, null, Network.GuestType.Isolated, true,
                         true, false, false, false, false,
                         false, false, false, true, true, false,
-                        false, true, false, false);
+                        forVpc, true, false, false);
+        if (forNsx) {
+            defaultKubernetesServiceNetworkOffering.setNsxMode(NetworkOffering.NsxMode.NATTED.name());
+            defaultKubernetesServiceNetworkOffering.setForNsx(true);
+        }
         defaultKubernetesServiceNetworkOffering.setSupportsVmAutoScaling(true);
         defaultKubernetesServiceNetworkOffering.setState(NetworkOffering.State.Enabled);
         defaultKubernetesServiceNetworkOffering = networkOfferingDao.persistDefaultNetworkOffering(defaultKubernetesServiceNetworkOffering);
@@ -1924,11 +1957,6 @@ public class KubernetesClusterManagerImpl extends ManagerBase implements Kuberne
             networkOfferingServiceMapDao.persist(offService);
             logger.trace("Added service for the network offering: " + offService);
         }
-
-        _gcExecutor.scheduleWithFixedDelay(new KubernetesClusterGarbageCollector(), 300, 300, TimeUnit.SECONDS);
-        _stateScanner.scheduleWithFixedDelay(new KubernetesClusterStatusScanner(), 300, 30, TimeUnit.SECONDS);
-
-        return true;
     }
 
     @Override

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java

@@ -31,6 +31,8 @@ import java.util.stream.Collectors;
 
 import javax.inject.Inject;
 
+import com.cloud.offering.NetworkOffering;
+import com.cloud.offerings.dao.NetworkOfferingDao;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.api.BaseCmd;
 import org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd;
@@ -149,6 +151,8 @@ public class KubernetesClusterResourceModifierActionWorker extends KubernetesClu
     protected VolumeApiService volumeService;
     @Inject
     protected VolumeDao volumeDao;
+    @Inject
+    protected NetworkOfferingDao networkOfferingDao;
 
     protected String kubernetesClusterNodeNamePrefix;
 
@@ -738,12 +742,24 @@ public class KubernetesClusterResourceModifierActionWorker extends KubernetesClu
     protected void setupKubernetesClusterVpcTierRules(IpAddress publicIp, Network network, List<Long> clusterVMIds) throws ManagementServerException {
         // Create ACL rules
         createVpcTierAclRules(network);
-        // Add port forwarding for API access
+
-        try {
+        NetworkOffering offering = networkOfferingDao.findById(network.getNetworkOfferingId());
-            provisionPublicIpPortForwardingRule(publicIp, network, owner, clusterVMIds.get(0), CLUSTER_API_PORT, CLUSTER_API_PORT);
+        if (offering.isConserveMode()) {
-        } catch (ResourceUnavailableException | NetworkRuleConflictException e) {
+            // Add load balancing for API access
-            throw new ManagementServerException(String.format("Failed to activate API port forwarding rules for the Kubernetes cluster : %s", kubernetesCluster.getName()), e);
+            try {
+                provisionLoadBalancerRule(publicIp, network, owner, clusterVMIds, CLUSTER_API_PORT);
+            } catch (InsufficientAddressCapacityException e) {
+                throw new ManagementServerException(String.format("Failed to activate API load balancing rules for the Kubernetes cluster : %s", kubernetesCluster.getName()), e);
+            }
+        } else {
+            // Add port forwarding for API access
+            try {
+                provisionPublicIpPortForwardingRule(publicIp, network, owner, clusterVMIds.get(0), CLUSTER_API_PORT, CLUSTER_API_PORT);
+            } catch (ResourceUnavailableException | NetworkRuleConflictException e) {
+                throw new ManagementServerException(String.format("Failed to activate API port forwarding rules for the Kubernetes cluster : %s", kubernetesCluster.getName()), e);
+            }
         }
+
         // Add port forwarding rule for SSH access on each node VM
         try {
             provisionSshPortForwardingRules(publicIp, network, owner, clusterVMIds);

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/dao/KubernetesClusterVmMapDao.java

@@ -23,6 +23,8 @@ import java.util.List;
 
 public interface KubernetesClusterVmMapDao extends GenericDao<KubernetesClusterVmMapVO, Long> {
     public List<KubernetesClusterVmMapVO> listByClusterId(long clusterId);
+
+    public KubernetesClusterVmMapVO getClusterMapFromVmId(long vmId);
     public List<KubernetesClusterVmMapVO> listByClusterIdAndVmIdsIn(long clusterId, List<Long> vmIds);
 
     int removeByClusterIdAndVmIdsIn(long clusterId, List<Long> vmIds);

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/dao/KubernetesClusterVmMapDaoImpl.java

@@ -31,12 +31,17 @@ import com.cloud.utils.db.SearchCriteria;
 public class KubernetesClusterVmMapDaoImpl extends GenericDaoBase<KubernetesClusterVmMapVO, Long> implements KubernetesClusterVmMapDao {
 
     private final SearchBuilder<KubernetesClusterVmMapVO> clusterIdSearch;
+    private final SearchBuilder<KubernetesClusterVmMapVO> vmIdSearch;
 
     public KubernetesClusterVmMapDaoImpl() {
         clusterIdSearch = createSearchBuilder();
         clusterIdSearch.and("clusterId", clusterIdSearch.entity().getClusterId(), SearchCriteria.Op.EQ);
         clusterIdSearch.and("vmIdsIN", clusterIdSearch.entity().getVmId(), SearchCriteria.Op.IN);
         clusterIdSearch.done();
+
+        vmIdSearch = createSearchBuilder();
+        vmIdSearch.and("vmId", vmIdSearch.entity().getVmId(), SearchCriteria.Op.EQ);
+        vmIdSearch.done();
     }
 
     @Override
@@ -47,6 +52,13 @@ public class KubernetesClusterVmMapDaoImpl extends GenericDaoBase<KubernetesClus
         return listBy(sc, filter);
     }
 
+    @Override
+    public KubernetesClusterVmMapVO getClusterMapFromVmId(long vmId) {
+        SearchCriteria<KubernetesClusterVmMapVO> sc = vmIdSearch.create();
+        sc.setParameters("vmId", vmId);
+        return findOneBy(sc);
+    }
+
     @Override
     public List<KubernetesClusterVmMapVO> listByClusterIdAndVmIdsIn(long clusterId, List<Long> vmIds) {
         SearchCriteria<KubernetesClusterVmMapVO> sc = clusterIdSearch.create();

plugins/network-elements/bigswitch/src/main/java/com/cloud/network/element/BigSwitchBcfElement.java

@@ -698,6 +698,11 @@ NetworkACLServiceProvider, FirewallServiceProvider, ResourceStateAdapter {
         return true;
     }
 
+    @Override
+    public boolean reorderAclRules(Vpc vpc, List<? extends Network> networks, List<? extends NetworkACLItem> networkACLItems) {
+        return true;
+    }
+
     @Override
     public boolean applyFWRules(Network network,
             List<? extends FirewallRule> rules)

plugins/network-elements/bigswitch/src/main/java/com/cloud/network/guru/BigSwitchBcfGuestNetworkGuru.java

@@ -143,7 +143,7 @@ public class BigSwitchBcfGuestNetworkGuru extends GuestNetworkGuru implements Ne
     }
 
     @Override
-    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, Account owner) {
+    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, String name, Long vpcId, Account owner) {
         // Check if the isolation type of the physical network is BCF_SEGMENT, then delegate GuestNetworkGuru to design
         PhysicalNetworkVO physnet = _physicalNetworkDao.findById(plan.getPhysicalNetworkId());
         if (physnet == null || physnet.getIsolationMethods() == null || !physnet.getIsolationMethods().contains("BCF_SEGMENT")) {
@@ -162,7 +162,7 @@ public class BigSwitchBcfGuestNetworkGuru extends GuestNetworkGuru implements Ne
         }
 
         logger.debug("Physical isolation type is BCF_SEGMENT, asking GuestNetworkGuru to design this network");
-        NetworkVO networkObject = (NetworkVO)super.design(offering, plan, userSpecified, owner);
+        NetworkVO networkObject = (NetworkVO)super.design(offering, plan, userSpecified, name, vpcId, owner);
         if (networkObject == null) {
             return null;
         }

plugins/network-elements/brocade-vcs/src/main/java/com/cloud/network/guru/BrocadeVcsGuestNetworkGuru.java

@@ -95,7 +95,7 @@ public class BrocadeVcsGuestNetworkGuru extends GuestNetworkGuru {
     }
 
     @Override
-    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, Account owner) {
+    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, String name, Long vpcId, Account owner) {
         // Check of the isolation type of the related physical network is VLAN
         PhysicalNetworkVO physnet = _physicalNetworkDao.findById(plan.getPhysicalNetworkId());
         DataCenter dc = _dcDao.findById(plan.getDataCenterId());
@@ -103,8 +103,9 @@ public class BrocadeVcsGuestNetworkGuru extends GuestNetworkGuru {
             logger.debug("Refusing to design this network");
             return null;
         }
+
         logger.debug("Physical isolation type is VCS, asking GuestNetworkGuru to design this network");
-        NetworkVO networkObject = (NetworkVO)super.design(offering, plan, userSpecified, owner);
+        NetworkVO networkObject = (NetworkVO)super.design(offering, plan, userSpecified, name, vpcId, owner);
         if (networkObject == null) {
             return null;
         }

plugins/network-elements/brocade-vcs/src/test/java/com/cloud/network/guru/BrocadeVcsGuestNetworkGuruTest.java

@@ -170,7 +170,7 @@ public class BrocadeVcsGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork != null);
         assertTrue(designednetwork.getBroadcastDomainType() == BroadcastDomainType.Vcs);
     }
@@ -191,7 +191,7 @@ public class BrocadeVcsGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork == null);
     }
 
@@ -213,7 +213,7 @@ public class BrocadeVcsGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork == null);
     }
 

plugins/network-elements/internal-loadbalancer/src/main/java/org/apache/cloudstack/network/element/InternalLoadBalancerElement.java

@@ -83,7 +83,9 @@ import com.cloud.vm.VirtualMachine;
 import com.cloud.vm.VirtualMachineProfile;
 import com.cloud.vm.dao.DomainRouterDao;
 import com.cloud.network.router.NetworkHelper;
+import org.springframework.stereotype.Component;
 
+@Component
 public class InternalLoadBalancerElement extends AdapterBase implements LoadBalancingServiceProvider, InternalLoadBalancerElementService, IpDeployer {
     protected static final Map<Service, Map<Capability, String>> capabilities = setCapabilities();
     private static InternalLoadBalancerElement internalLbElement = null;
@@ -112,14 +114,7 @@ public class InternalLoadBalancerElement extends AdapterBase implements LoadBala
     @Qualifier("networkHelper")
     protected NetworkHelper _networkHelper;
 
-    protected InternalLoadBalancerElement() {
+    public InternalLoadBalancerElement() {
-    }
-
-    public static InternalLoadBalancerElement getInstance() {
-        if (internalLbElement == null) {
-            internalLbElement = new InternalLoadBalancerElement();
-        }
-        return internalLbElement;
     }
 
     private boolean canHandle(Network config, Scheme lbScheme) {
@@ -517,6 +512,11 @@ public class InternalLoadBalancerElement extends AdapterBase implements LoadBala
         return sc.list();
     }
 
+    @Override
+    public Type getProviderType() {
+        return Type.InternalLbVm;
+    }
+
     @Override
     public boolean applyIps(Network network, List<? extends PublicIpAddress> ipAddress, Set<Service> services) throws ResourceUnavailableException {
         //do nothing here; this element just has to extend the ip deployer

plugins/network-elements/juniper-contrail/src/main/java/org/apache/cloudstack/network/contrail/management/ContrailGuru.java

@@ -117,7 +117,7 @@ public class ContrailGuru extends AdapterBase implements NetworkGuru {
     }
 
     @Override
-    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, Account owner) {
+    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, String name, Long vpcId, Account owner) {
         // Check of the isolation type of the related physical network is L3VPN
         PhysicalNetworkVO physnet = _physicalNetworkDao.findById(plan.getPhysicalNetworkId());
         DataCenter dc = _dcDao.findById(plan.getDataCenterId());
@@ -136,6 +136,11 @@ public class ContrailGuru extends AdapterBase implements NetworkGuru {
         return network;
     }
 
+    @Override
+    public void setup(Network network, long networkId) {
+        // do nothing
+    }
+
     @Override
     public Network implement(Network network, NetworkOffering offering, DeployDestination destination, ReservationContext context)
             throws InsufficientVirtualNetworkCapacityException {

plugins/network-elements/juniper-contrail/src/main/java/org/apache/cloudstack/network/contrail/management/ContrailManagerImpl.java

@@ -217,7 +217,7 @@ public class ContrailManagerImpl extends ManagerBase implements ContrailManager
         ConfigurationManager configMgr = (ConfigurationManager) _configService;
         NetworkOfferingVO voffer = configMgr.createNetworkOffering(offeringName, offeringDisplayText,
                 TrafficType.Public, null, true, Availability.Optional, null, serviceProviderMap, true,
-                Network.GuestType.Shared, false, null, false, null, true, false, null, true, null, false, false, false, null, null, true, null);
+                Network.GuestType.Shared, false, null, false, null, true, false, null, true, null, false, false, false, false, null, null, null, true, null);
         long id = voffer.getId();
         _networkOfferingDao.update(id, voffer);
         return _networkOfferingDao.findById(id);
@@ -252,7 +252,7 @@ public class ContrailManagerImpl extends ManagerBase implements ContrailManager
         ConfigurationManager configMgr = (ConfigurationManager)_configService;
         NetworkOfferingVO voffer =
                 configMgr.createNetworkOffering(offeringName, offeringDisplayText, TrafficType.Guest, null, false, Availability.Optional, null, serviceProviderMap, true,
-                        Network.GuestType.Isolated, false, null, false, null, false, true, null, true, null, false, offeringName.equals(vpcRouterOfferingName), false, null, null, true, null);
+                        Network.GuestType.Isolated, false, null, false, null, false, true, null, true, null, false, offeringName.equals(vpcRouterOfferingName), false, false, null,  null, null, true, null);
         if (offeringName.equals(vpcRouterOfferingName)) {
             voffer.setInternalLb(true);
         }
@@ -293,7 +293,7 @@ public class ContrailManagerImpl extends ManagerBase implements ContrailManager
             }
             serviceProviderMap.put(svc, providerSet);
         }
-        vpcOffer = _vpcProvSvc.createVpcOffering(juniperVPCOfferingName, juniperVPCOfferingDisplayText, services, serviceProviderMap, null, null, null, null, null, VpcOffering.State.Enabled);
+        vpcOffer = _vpcProvSvc.createVpcOffering(juniperVPCOfferingName, juniperVPCOfferingDisplayText, services, serviceProviderMap, null, null, null, false, null, null, null, VpcOffering.State.Enabled);
         long id = vpcOffer.getId();
         _vpcOffDao.update(id, (VpcOfferingVO)vpcOffer);
         return _vpcOffDao.findById(id);

plugins/network-elements/juniper-contrail/src/main/java/org/apache/cloudstack/network/contrail/management/ContrailVpcElementImpl.java

@@ -182,6 +182,11 @@ public class ContrailVpcElementImpl extends ContrailElementImpl implements Netwo
         return true;
     }
 
+    @Override
+    public boolean reorderAclRules(Vpc vpc, List<? extends Network> networks, List<? extends NetworkACLItem> networkACLItems) {
+        return true;
+    }
+
     @Override
     public boolean applyACLItemsToPrivateGw(PrivateGateway privateGateway,
             List<? extends NetworkACLItem> rules)

plugins/network-elements/juniper-contrail/src/main/java/org/apache/cloudstack/network/contrail/management/ManagementNetworkGuru.java

@@ -109,7 +109,7 @@ public class ManagementNetworkGuru extends ContrailGuru {
     }
 
     @Override
-    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, Account owner) {
+    public Network design(NetworkOffering offering, DeploymentPlan plan, Network userSpecified, String name, Long vpcId, Account owner) {
 
         if (!canHandle(offering)) {
             return null;

plugins/network-elements/nicira-nvp/src/main/java/com/cloud/network/guru/NiciraNvpGuestNetworkGuru.java

@@ -136,7 +136,7 @@ public class NiciraNvpGuestNetworkGuru extends GuestNetworkGuru implements Netwo
     }
 
     @Override
-    public Network design(final NetworkOffering offering, final DeploymentPlan plan, final Network userSpecified, final Account owner) {
+    public Network design(final NetworkOffering offering, final DeploymentPlan plan, final Network userSpecified, String name, Long vpcId, final Account owner) {
         // Check of the isolation type of the related physical network is supported
         final PhysicalNetworkVO physnet = physicalNetworkDao.findById(plan.getPhysicalNetworkId());
         final DataCenter dc = _dcDao.findById(plan.getDataCenterId());
@@ -153,7 +153,7 @@ public class NiciraNvpGuestNetworkGuru extends GuestNetworkGuru implements Netwo
         logger.debug("Nicira Nvp " + devices.get(0).getUuid() + " found on physical network " + physnet.getId());
 
         logger.debug("Physical isolation type is supported, asking GuestNetworkGuru to design this network");
-        final NetworkVO networkObject = (NetworkVO) super.design(offering, plan, userSpecified, owner);
+        final NetworkVO networkObject = (NetworkVO) super.design(offering, plan, userSpecified, name, vpcId, owner);
         if (networkObject == null) {
             return null;
         }

plugins/network-elements/nicira-nvp/src/test/java/com/cloud/network/guru/NiciraNvpGuestNetworkGuruTest.java

@@ -168,7 +168,7 @@ public class NiciraNvpGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork != null);
         assertTrue(designednetwork.getBroadcastDomainType() == BroadcastDomainType.Lswitch);
     }
@@ -192,7 +192,7 @@ public class NiciraNvpGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork == null);
     }
 
@@ -215,7 +215,7 @@ public class NiciraNvpGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork == null);
     }
 
@@ -241,7 +241,7 @@ public class NiciraNvpGuestNetworkGuruTest {
         final Network network = mock(Network.class);
         final Account account = mock(Account.class);
 
-        final Network designednetwork = guru.design(offering, plan, network, account);
+        final Network designednetwork = guru.design(offering, plan, network, "", 1L, account);
         assertTrue(designednetwork == null);
     }
 

plugins/network-elements/nsx/pom.xml

@@ -0,0 +1,59 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>cloud-plugin-network-nsx</artifactId>
+    <name>Apache CloudStack Plugin - NSX Network</name>
+
+    <parent>
+        <groupId>org.apache.cloudstack</groupId>
+        <artifactId>cloudstack-plugins</artifactId>
+        <version>4.20.0.0-SNAPSHOT</version>
+        <relativePath>../../pom.xml</relativePath>
+    </parent>
+    <dependencies>
+        <dependency>
+            <groupId>com.vmware</groupId>
+            <artifactId>nsx-java-sdk</artifactId>
+            <version>4.1.0.2.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.vmware</groupId>
+            <artifactId>nsx-gpm-java-sdk</artifactId>
+            <version>4.1.0.2.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.vmware</groupId>
+            <artifactId>nsx-policy-java-sdk</artifactId>
+            <version>4.1.0.2.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.vmware.vapi</groupId>
+            <artifactId>vapi-authentication</artifactId>
+            <version>2.40.0</version>
+        </dependency>
+        <dependency>
+            <groupId>com.vmware.vapi</groupId>
+            <artifactId>vapi-runtime</artifactId>
+            <version>2.40.0</version>
+        </dependency>
+    </dependencies>
+</project>

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/NsxAnswer.java

@@ -0,0 +1,31 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+
+public class NsxAnswer extends Answer {
+    public NsxAnswer(final Command command, final boolean success, final String details) {
+        super(command, success, details);
+    }
+
+    public NsxAnswer(final Command command, final Exception e) {
+        super(command, e);
+    }
+
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/StartupNsxCommand.java

@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack;
+
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.host.Host;
+
+public class StartupNsxCommand extends StartupCommand {
+    public StartupNsxCommand() {
+        super(Host.Type.L2Networking);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxDhcpRelayConfigCommand.java

@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.List;
+import java.util.Objects;
+
+public class CreateNsxDhcpRelayConfigCommand extends NsxCommand {
+
+    private Long vpcId;
+    private String vpcName;
+    private long networkId;
+    private String networkName;
+    private List<String> addresses;
+
+    public CreateNsxDhcpRelayConfigCommand(long domainId, long accountId, long zoneId,
+                                           Long vpcId, String vpcName, long networkId, String networkName,
+                                           List<String> addresses) {
+        super(domainId, accountId, zoneId);
+        this.vpcId = vpcId;
+        this.vpcName = vpcName;
+        this.networkId = networkId;
+        this.networkName = networkName;
+        this.addresses = addresses;
+    }
+
+    public Long getVpcId() {
+        return vpcId;
+    }
+
+    public String getVpcName() {
+        return vpcName;
+    }
+
+    public long getNetworkId() {
+        return networkId;
+    }
+
+    public String getNetworkName() {
+        return networkName;
+    }
+
+    public List<String> getAddresses() {
+        return addresses;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        CreateNsxDhcpRelayConfigCommand that = (CreateNsxDhcpRelayConfigCommand) o;
+        return networkId == that.networkId && Objects.equals(vpcId, that.vpcId) && Objects.equals(vpcName, that.vpcName) && Objects.equals(networkName, that.networkName) && Objects.equals(addresses, that.addresses);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), vpcId, vpcName, networkId, networkName, addresses);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxDistributedFirewallRulesCommand.java

@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import org.apache.cloudstack.resource.NsxNetworkRule;
+
+import java.util.List;
+import java.util.Objects;
+
+public class CreateNsxDistributedFirewallRulesCommand extends NsxCommand {
+
+    private Long vpcId;
+    private long networkId;
+    private List<NsxNetworkRule> rules;
+
+    public CreateNsxDistributedFirewallRulesCommand(long domainId, long accountId, long zoneId,
+                                                    Long vpcId, long networkId,
+                                                    List<NsxNetworkRule> rules) {
+        super(domainId, accountId, zoneId);
+        this.vpcId = vpcId;
+        this.networkId = networkId;
+        this.rules = rules;
+    }
+
+    public Long getVpcId() {
+        return vpcId;
+    }
+
+    public long getNetworkId() {
+        return networkId;
+    }
+
+    public List<NsxNetworkRule> getRules() {
+        return rules;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        CreateNsxDistributedFirewallRulesCommand that = (CreateNsxDistributedFirewallRulesCommand) o;
+        return networkId == that.networkId && Objects.equals(vpcId, that.vpcId) && Objects.equals(rules, that.rules);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), vpcId, networkId, rules);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxLoadBalancerRuleCommand.java

@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import org.apache.cloudstack.resource.NsxLoadBalancerMember;
+
+import java.util.List;
+import java.util.Objects;
+
+public class CreateNsxLoadBalancerRuleCommand extends NsxNetworkCommand {
+
+    private final String publicPort;
+    private final String privatePort;
+    private final String algorithm;
+    private final String protocol;
+    List<NsxLoadBalancerMember> memberList;
+
+    private final long lbId;
+    public CreateNsxLoadBalancerRuleCommand(long domainId, long accountId, long zoneId, Long networkResourceId,
+                                            String networkResourceName, boolean isResourceVpc,
+                                            List<NsxLoadBalancerMember> memberList, long lbId, String publicPort,
+                                            String privatePort, String algorithm, String protocol) {
+        super(domainId, accountId, zoneId, networkResourceId, networkResourceName, isResourceVpc);
+        this.lbId = lbId;
+        this.memberList = memberList;
+        this.publicPort = publicPort;
+        this.privatePort = privatePort;
+        this.algorithm = algorithm;
+        this.protocol = protocol;
+    }
+
+
+    public long getLbId() {
+        return lbId;
+    }
+
+    public String getPublicPort() {
+        return publicPort;
+    }
+
+    public String getPrivatePort() {
+        return privatePort;
+    }
+
+    public List<NsxLoadBalancerMember> getMemberList() {
+        return memberList;
+    }
+
+    public String getAlgorithm() {
+        return algorithm;
+    }
+
+    public String getProtocol() {
+        return protocol;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        CreateNsxLoadBalancerRuleCommand command = (CreateNsxLoadBalancerRuleCommand) o;
+        return lbId == command.lbId && Objects.equals(publicPort, command.publicPort) && Objects.equals(privatePort, command.privatePort) && Objects.equals(algorithm, command.algorithm) && Objects.equals(protocol, command.protocol) && Objects.equals(memberList, command.memberList);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), publicPort, privatePort, algorithm, protocol, memberList, lbId);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxPortForwardRuleCommand.java

@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class CreateNsxPortForwardRuleCommand extends NsxNetworkCommand {
+    private final String publicPort;
+    private final String privatePort;
+    private final String protocol;
+    private final long ruleId;
+
+
+    public CreateNsxPortForwardRuleCommand(long domainId, long accountId, long zoneId, Long networkResourceId,
+                                           String networkResourceName, boolean isResourceVpc, Long vmId,
+                                           long ruleId, String publicIp, String vmIp, String publicPort, String privatePort, String protocol) {
+        super(domainId, accountId, zoneId, networkResourceId, networkResourceName, isResourceVpc, vmId, publicIp, vmIp);
+        this.publicPort = publicPort;
+        this.privatePort = privatePort;
+        this.ruleId = ruleId;
+        this.protocol = protocol;
+
+    }
+
+    public String getPublicPort() {
+        return publicPort;
+    }
+
+    public String getPrivatePort() {
+        return privatePort;
+    }
+
+    public long getRuleId() {
+        return ruleId;
+    }
+
+    public String getProtocol() {
+        return protocol;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        CreateNsxPortForwardRuleCommand that = (CreateNsxPortForwardRuleCommand) o;
+        return ruleId == that.ruleId && Objects.equals(publicPort, that.publicPort) && Objects.equals(privatePort, that.privatePort) && Objects.equals(protocol, that.protocol);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), publicPort, privatePort, protocol, ruleId);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxSegmentCommand.java

@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class CreateNsxSegmentCommand extends NsxCommand {
+
+    private Long vpcId;
+    private String vpcName;
+    private long networkId;
+    private String networkName;
+    private String networkGateway;
+    private String networkCidr;
+
+    public CreateNsxSegmentCommand(long domainId, long accountId, long zoneId,
+                                   Long vpcId, String vpcName, long networkId, String networkName,
+                                   String networkGateway, String networkCidr) {
+        super(domainId, accountId, zoneId);
+        this.vpcId = vpcId;
+        this.vpcName = vpcName;
+        this.networkId = networkId;
+        this.networkName = networkName;
+        this.networkGateway = networkGateway;
+        this.networkCidr = networkCidr;
+    }
+
+    public Long getVpcId() {
+        return vpcId;
+    }
+
+    public String getVpcName() {
+        return vpcName;
+    }
+
+    public long getNetworkId() {
+        return networkId;
+    }
+
+    public String getNetworkName() {
+        return networkName;
+    }
+
+    public String getNetworkGateway() {
+        return networkGateway;
+    }
+
+    public String getNetworkCidr() {
+        return networkCidr;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        if (!super.equals(o)) return false;
+        CreateNsxSegmentCommand command = (CreateNsxSegmentCommand) o;
+        return Objects.equals(networkName, command.networkName);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), networkName);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxStaticNatCommand.java

@@ -0,0 +1,25 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+public class CreateNsxStaticNatCommand extends NsxNetworkCommand {
+
+    public CreateNsxStaticNatCommand(long domainId, long accountId, long zoneId, Long networkResourceId, String networkResourceName,
+                                     boolean isResourceVpc, Long vmId, String publicIp, String vmIp) {
+        super(domainId, accountId, zoneId, networkResourceId, networkResourceName, isResourceVpc, vmId, publicIp, vmIp);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateNsxTier1GatewayCommand.java

@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class CreateNsxTier1GatewayCommand extends NsxCommand {
+
+    private Long networkResourceId;
+    private String networkResourceName;
+    private boolean isResourceVpc;
+    private boolean sourceNatEnabled;
+
+    public CreateNsxTier1GatewayCommand(long domainId, long accountId, long zoneId,
+                                        Long networkResourceId, String networkResourceName, boolean isResourceVpc,
+                                        boolean sourceNatEnabled) {
+        super(domainId, accountId, zoneId);
+        this.networkResourceId = networkResourceId;
+        this.networkResourceName = networkResourceName;
+        this.isResourceVpc = isResourceVpc;
+        this.sourceNatEnabled = sourceNatEnabled;
+    }
+
+    public Long getNetworkResourceId() {
+        return networkResourceId;
+    }
+
+    public boolean isResourceVpc() {
+        return isResourceVpc;
+    }
+
+    public String getNetworkResourceName() {
+        return networkResourceName;
+    }
+
+    public boolean isSourceNatEnabled() {
+        return sourceNatEnabled;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        if (!super.equals(o)) return false;
+        CreateNsxTier1GatewayCommand that = (CreateNsxTier1GatewayCommand) o;
+        return Objects.equals(networkResourceName, that.networkResourceName);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), networkResourceName);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/CreateOrUpdateNsxTier1NatRuleCommand.java

@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class CreateOrUpdateNsxTier1NatRuleCommand extends NsxCommand {
+
+    private String tier1GatewayName;
+    private String action;
+    private String translatedIpAddress;
+    private String natRuleId;
+
+    public CreateOrUpdateNsxTier1NatRuleCommand(long domainId, long accountId, long zoneId,
+                                                String tier1GatewayName, String action, String translatedIpAddress, String natRuleId) {
+        super(domainId, accountId, zoneId);
+        this.tier1GatewayName = tier1GatewayName;
+        this.action = action;
+        this.translatedIpAddress = translatedIpAddress;
+        this.natRuleId = natRuleId;
+    }
+
+    public String getTier1GatewayName() {
+        return tier1GatewayName;
+    }
+
+    public String getAction() {
+        return action;
+    }
+
+    public String getTranslatedIpAddress() {
+        return translatedIpAddress;
+    }
+
+    public String getNatRuleId() {
+        return natRuleId;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        CreateOrUpdateNsxTier1NatRuleCommand that = (CreateOrUpdateNsxTier1NatRuleCommand) o;
+        return Objects.equals(tier1GatewayName, that.tier1GatewayName) && Objects.equals(action, that.action) && Objects.equals(translatedIpAddress, that.translatedIpAddress) && Objects.equals(natRuleId, that.natRuleId);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), tier1GatewayName, action, translatedIpAddress, natRuleId);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/DeleteNsxDistributedFirewallRulesCommand.java

@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import org.apache.cloudstack.resource.NsxNetworkRule;
+
+import java.util.List;
+
+public class DeleteNsxDistributedFirewallRulesCommand extends CreateNsxDistributedFirewallRulesCommand {
+    public DeleteNsxDistributedFirewallRulesCommand(long domainId, long accountId, long zoneId, Long vpcId, long networkId, List<NsxNetworkRule> rules) {
+        super(domainId, accountId, zoneId, vpcId, networkId, rules);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/DeleteNsxLoadBalancerRuleCommand.java

@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import org.apache.cloudstack.resource.NsxLoadBalancerMember;
+
+import java.util.List;
+import java.util.Objects;
+
+public class DeleteNsxLoadBalancerRuleCommand extends NsxNetworkCommand {
+    private long lbId;
+    List<NsxLoadBalancerMember> memberList;
+
+    public DeleteNsxLoadBalancerRuleCommand(long domainId, long accountId, long zoneId, Long networkResourceId,
+                                            String networkResourceName, boolean isResourceVpc,
+                                            List<NsxLoadBalancerMember> memberList, long lbId, long vmId) {
+        super(domainId, accountId, zoneId, networkResourceId, networkResourceName, isResourceVpc, vmId);
+        this.lbId = lbId;
+        this.memberList = memberList;
+    }
+
+    public long getLbId() {
+        return lbId;
+    }
+
+    public List<NsxLoadBalancerMember> getMemberList() { return memberList; }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        DeleteNsxLoadBalancerRuleCommand that = (DeleteNsxLoadBalancerRuleCommand) o;
+        return lbId == that.lbId && Objects.equals(memberList, that.memberList);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), lbId, memberList);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/DeleteNsxNatRuleCommand.java

@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import com.cloud.network.Network;
+
+import java.util.Objects;
+
+public class DeleteNsxNatRuleCommand extends NsxNetworkCommand {
+    private Long ruleId;
+    private Network.Service service;
+
+    private String privatePort;
+    private String protocol;
+    public DeleteNsxNatRuleCommand(long domainId, long accountId, long zoneId, Long networkResourceId, String networkResourceName,
+                                   boolean isResourceVpc, Long vmId, Long ruleId, String privatePort, String protocol) {
+        super(domainId, accountId, zoneId, networkResourceId, networkResourceName, isResourceVpc, vmId);
+        this.ruleId = ruleId;
+        this.privatePort = privatePort;
+        this.protocol = protocol;
+    }
+
+    public Long getRuleId() {
+        return ruleId;
+    }
+
+    public Network.Service getService() {
+        return service;
+    }
+
+    public void setService(Network.Service service) {
+        this.service = service;
+    }
+
+    public String getPrivatePort() {
+        return privatePort;
+    }
+
+    public String getProtocol() {
+        return protocol;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        DeleteNsxNatRuleCommand that = (DeleteNsxNatRuleCommand) o;
+        return Objects.equals(ruleId, that.ruleId) && Objects.equals(service, that.service) && Objects.equals(privatePort, that.privatePort) && Objects.equals(protocol, that.protocol);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), ruleId, service, privatePort, protocol);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/DeleteNsxSegmentCommand.java

@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class DeleteNsxSegmentCommand extends NsxCommand {
+
+    private Long vpcId;
+    private String vpcName;
+
+    private long networkId;
+    private String networkName;
+
+    public DeleteNsxSegmentCommand(long domainId, long accountId, long zoneId, Long vpcId,
+                                   String vpcName, long networkId, String networkName) {
+        super(domainId, accountId, zoneId);
+        this.vpcId = vpcId;
+        this.vpcName = vpcName;
+        this.networkId = networkId;
+        this.networkName = networkName;
+    }
+
+    public Long getVpcId() {
+        return vpcId;
+    }
+
+    public String getVpcName() {
+        return vpcName;
+    }
+
+    public long getNetworkId() {
+        return networkId;
+    }
+
+    public String getNetworkName() {
+        return networkName;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        DeleteNsxSegmentCommand command = (DeleteNsxSegmentCommand) o;
+        return networkId == command.networkId && Objects.equals(vpcId, command.vpcId) && Objects.equals(vpcName, command.vpcName) && Objects.equals(networkName, command.networkName);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), vpcId, vpcName, networkId, networkName);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/DeleteNsxTier1GatewayCommand.java

@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class DeleteNsxTier1GatewayCommand extends NsxCommand {
+
+    private Long networkResourceId;
+    private String networkResourceName;
+    private boolean isResourceVpc;
+
+    public DeleteNsxTier1GatewayCommand(long domainId, long accountId, long zoneId,
+                                        Long networkResourceId, String networkResourceName, boolean isResourceVpc) {
+        super(domainId, accountId, zoneId);
+        this.networkResourceId = networkResourceId;
+        this.networkResourceName = networkResourceName;
+        this.isResourceVpc = isResourceVpc;
+    }
+
+    public Long getNetworkResourceId() {
+        return networkResourceId;
+    }
+
+    public String getNetworkResourceName() {
+        return networkResourceName;
+    }
+
+    public boolean isResourceVpc() {
+        return isResourceVpc;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass() || !super.equals(o)) {
+            return false;
+        }
+        DeleteNsxTier1GatewayCommand that = (DeleteNsxTier1GatewayCommand) o;
+        return isResourceVpc == that.isResourceVpc && Objects.equals(networkResourceId, that.networkResourceId) && Objects.equals(networkResourceName, that.networkResourceName);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), networkResourceId, networkResourceName, isResourceVpc);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/NsxCommand.java

@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import com.cloud.agent.api.Command;
+
+import java.util.Objects;
+
+public class NsxCommand extends Command {
+    private long zoneId;
+    private long accountId;
+    private long domainId;
+
+    public NsxCommand() {
+    }
+
+    public NsxCommand(long domainId, long accountId, long zoneId) {
+        this.zoneId = zoneId;
+        this.accountId = accountId;
+        this.domainId = domainId;
+    }
+
+    public long getZoneId() {
+        return zoneId;
+    }
+
+    public long getAccountId() {
+        return accountId;
+    }
+
+    public long getDomainId() {
+        return domainId;
+    }
+
+    @Override
+    public boolean executeInSequence() {
+        return false;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        if (!super.equals(o)) return false;
+        NsxCommand that = (NsxCommand) o;
+        return Objects.equals(zoneId, that.zoneId) && Objects.equals(accountId, that.accountId) && Objects.equals(domainId, that.domainId);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), zoneId, accountId, domainId);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/NsxNetworkCommand.java

@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import java.util.Objects;
+
+public class NsxNetworkCommand extends NsxCommand {
+    private Long networkResourceId;
+    private String networkResourceName;
+    private boolean isResourceVpc;
+    private Long vmId;
+    private String publicIp;
+    private String vmIp;
+
+    public NsxNetworkCommand(long domainId, long accountId, long zoneId, Long networkResourceId, String networkResourceName,
+                             boolean isResourceVpc, Long vmId, String publicIp, String vmIp) {
+        super(domainId, accountId, zoneId);
+        this.networkResourceId = networkResourceId;
+        this.networkResourceName = networkResourceName;
+        this.isResourceVpc = isResourceVpc;
+        this.vmId = vmId;
+        this.publicIp = publicIp;
+        this.vmIp = vmIp;
+    }
+
+    public NsxNetworkCommand(long domainId, long accountId, long zoneId, Long networkResourceId, String networkResourceName,
+                             boolean isResourceVpc) {
+        super(domainId, accountId, zoneId);
+        this.networkResourceId = networkResourceId;
+        this.networkResourceName = networkResourceName;
+        this.isResourceVpc = isResourceVpc;
+    }
+
+    public NsxNetworkCommand(long domainId, long accountId, long zoneId, Long networkResourceId, String networkResourceName,
+                            boolean isResourceVpc, Long vmId) {
+        this(domainId, accountId, zoneId, networkResourceId, networkResourceName, isResourceVpc);
+        this.vmId = vmId;
+    }
+
+    public Long getNetworkResourceId() {
+        return networkResourceId;
+    }
+
+    public void setNetworkResourceId(long networkResourceId) {
+        this.networkResourceId = networkResourceId;
+    }
+
+    public String getNetworkResourceName() {
+        return networkResourceName;
+    }
+
+    public void setNetworkResourceName(String networkResourceName) {
+        this.networkResourceName = networkResourceName;
+    }
+
+    public boolean isResourceVpc() {
+        return isResourceVpc;
+    }
+
+    public void setResourceVpc(boolean resourceVpc) {
+        isResourceVpc = resourceVpc;
+    }
+
+    public Long getVmId() {
+        return vmId;
+    }
+
+    public void setVmId(Long vmId) {
+        this.vmId = vmId;
+    }
+
+    public String getPublicIp() {
+        return publicIp;
+    }
+
+    public void setPublicIp(String publicIp) {
+        this.publicIp = publicIp;
+    }
+
+    public String getVmIp() {
+        return vmIp;
+    }
+
+    public void setVmIp(String vmIp) {
+        this.vmIp = vmIp;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        if (!super.equals(o)) return false;
+        NsxNetworkCommand that = (NsxNetworkCommand) o;
+        return networkResourceId == that.networkResourceId && vmId == that.vmId &&
+                Objects.equals(networkResourceName, that.networkResourceName) && Objects.equals(publicIp, that.publicIp)
+                && Objects.equals(vmIp, that.vmIp);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(super.hashCode(), networkResourceId, networkResourceName, vmId, publicIp, vmIp);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/agent/api/StartupNsxCommand.java

@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.agent.api;
+
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.host.Host;
+
+public class StartupNsxCommand extends StartupCommand {
+
+    public StartupNsxCommand() {
+        super(Host.Type.L2Networking);
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/api/command/AddNsxControllerCmd.java

@@ -0,0 +1,130 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command;
+
+import com.cloud.network.nsx.NsxProvider;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.response.NsxControllerResponse;
+import org.apache.cloudstack.api.response.ZoneResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.service.NsxProviderService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+
+
+@APICommand(name = AddNsxControllerCmd.APINAME, description = "Add NSX Controller to CloudStack",
+        responseObject = NsxControllerResponse.class, requestHasSensitiveInfo = false,
+        responseHasSensitiveInfo = false, since = "4.19.0")
+public class AddNsxControllerCmd extends BaseCmd {
+    public static final String APINAME = "addNsxController";
+    public static final Logger LOGGER = LoggerFactory.getLogger(AddNsxControllerCmd.class.getName());
+
+    @Inject
+    NsxProviderService nsxProviderService;
+
+    @Parameter(name = ApiConstants.ZONE_ID, type = CommandType.UUID, entityType = ZoneResponse.class, required = true,
+            description = "the ID of zone")
+    private Long zoneId;
+
+    @Parameter(name = ApiConstants.NAME, type = CommandType.STRING, required = true, description = "NSX controller / provider name")
+    private String name;
+
+    @Parameter(name = ApiConstants.NSX_PROVIDER_HOSTNAME, type = CommandType.STRING, required = true, description = "NSX controller hostname / IP address")
+    private String hostname;
+
+    @Parameter(name = ApiConstants.NSX_PROVIDER_PORT, type = CommandType.STRING, description = "NSX controller port")
+    private String port;
+    @Parameter(name = ApiConstants.USERNAME, type = CommandType.STRING, required = true, description = "Username to log into NSX controller")
+    private String username;
+    @Parameter(name = ApiConstants.PASSWORD, type = CommandType.STRING, required = true, description = "Password to login into NSX controller")
+    private String password;
+
+    @Parameter(name = ApiConstants.TIER0_GATEWAY, type = CommandType.STRING, required = true, description = "Tier-0 Gateway address")
+    private String tier0Gateway;
+
+    @Parameter(name = ApiConstants.EDGE_CLUSTER, type = CommandType.STRING, required = true, description = "Edge Cluster name")
+    private String edgeCluster;
+
+    @Parameter(name = ApiConstants.TRANSPORT_ZONE, type = CommandType.STRING, required = true, description = "Transport Zone controls to which hosts a logical switch can reach")
+    private String transportZone;
+
+    public NsxProviderService getNsxProviderService() {
+        return nsxProviderService;
+    }
+
+    public Long getZoneId() {
+        return zoneId;
+    }
+
+    public String getName() {
+        return name;
+    }
+
+    public String getHostname() {
+        return hostname;
+    }
+
+    public String getPort() {
+        return port;
+    }
+
+    public String getUsername() {
+        return username;
+    }
+
+    public String getPassword() {
+        return password;
+    }
+
+    public String getTier0Gateway() {
+        return tier0Gateway;
+    }
+
+    public String getEdgeCluster() {
+        return edgeCluster;
+    }
+
+    public String getTransportZone() {
+        return transportZone;
+    }
+
+    @Override
+    public void execute() throws ServerApiException {
+        NsxProvider nsxProvider = nsxProviderService.addProvider(this);
+        NsxControllerResponse nsxControllerResponse =
+                nsxProviderService.createNsxControllerResponse(
+                        nsxProvider);
+        if (nsxControllerResponse == null)
+            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Failed to add NSX controller");
+        else {
+            nsxControllerResponse.setResponseName(getCommandName());
+            setResponseObject(nsxControllerResponse);
+        }
+    }
+
+    @Override
+    public long getEntityOwnerId() {
+        return CallContext.current().getCallingAccount().getId();
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/api/command/DeleteNsxControllerCmd.java

@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command;
+
+import com.cloud.exception.InvalidParameterValueException;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.response.NsxControllerResponse;
+import org.apache.cloudstack.api.response.SuccessResponse;
+import org.apache.cloudstack.service.NsxProviderService;
+
+import javax.inject.Inject;
+
+import static org.apache.cloudstack.api.command.DeleteNsxControllerCmd.APINAME;
+
+@APICommand(name = APINAME, description = "delete NSX Controller to CloudStack",
+        responseObject = NsxControllerResponse.class, requestHasSensitiveInfo = false,
+        responseHasSensitiveInfo = false, since = "4.19.0")
+public class DeleteNsxControllerCmd extends BaseCmd {
+    public static final String APINAME = "deleteNsxController";
+
+    @Inject
+    protected NsxProviderService nsxProviderService;
+/////////////////////////////////////////////////////
+    //////////////// API parameters /////////////////////
+    /////////////////////////////////////////////////////
+
+    @Parameter(name = ApiConstants.NSX_CONTROLLER_ID, type = CommandType.UUID, entityType = NsxControllerResponse.class,
+            required = true, description = "NSX Controller ID")
+    private Long nsxControllerId;
+
+    /////////////////////////////////////////////////////
+    /////////////////// Accessors ///////////////////////
+    /////////////////////////////////////////////////////
+
+    public Long getNsxControllerId() {
+        return nsxControllerId;
+    }
+
+    /////////////////////////////////////////////////////
+    /////////////// API Implementation///////////////////
+    /////////////////////////////////////////////////////
+
+
+    @Override
+    public void execute() throws ServerApiException, ConcurrentOperationException {
+        try {
+            boolean deleted = nsxProviderService.deleteNsxController(getNsxControllerId());
+            if (deleted) {
+                SuccessResponse response = new SuccessResponse(getCommandName());
+                response.setResponseName(getCommandName());
+                setResponseObject(response);
+            } else {
+                throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, "Failed to remove NSX Controller from Zone");
+            }
+        } catch (InvalidParameterValueException e) {
+            throw new ServerApiException(ApiErrorCode.PARAM_ERROR, e.getMessage());
+        } catch (CloudRuntimeException e) {
+            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, e.getMessage());
+        }
+    }
+
+    @Override
+    public long getEntityOwnerId() {
+        return 0;
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/api/command/ListNsxControllersCmd.java

@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.utils.StringUtils;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.BaseResponse;
+import org.apache.cloudstack.api.BaseListCmd;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ListResponse;
+import org.apache.cloudstack.api.response.NsxControllerResponse;
+import org.apache.cloudstack.api.response.ZoneResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.service.NsxProviderService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.inject.Inject;
+import java.util.List;
+
+import static org.apache.cloudstack.api.command.ListNsxControllersCmd.APINAME;
+
+@APICommand(name = APINAME, description = "list all NSX controllers added to CloudStack",
+        responseObject = NsxControllerResponse.class, requestHasSensitiveInfo = false,
+        responseHasSensitiveInfo = false, since = "4.19.0")
+public class ListNsxControllersCmd extends BaseListCmd {
+    public static final String APINAME = "listNsxControllers";
+    public static final Logger LOGGER = LoggerFactory.getLogger(ListNsxControllersCmd.class.getName());
+
+    @Inject
+    private NsxProviderService nsxProviderService;
+
+    @Parameter(name = ApiConstants.ZONE_ID, description = "NSX controller added to the specific zone",
+            type = CommandType.UUID, entityType = ZoneResponse.class)
+    Long zoneId;
+
+    @Override
+    public void execute() throws ServerApiException, ConcurrentOperationException {
+        List<BaseResponse> baseResponseList = nsxProviderService.listNsxProviders(zoneId);
+        List<BaseResponse> pagingList = StringUtils.applyPagination(baseResponseList, this.getStartIndex(), this.getPageSizeVal());
+        ListResponse<BaseResponse> listResponse = new ListResponse<>();
+        listResponse.setResponses(pagingList);
+        listResponse.setResponseName(getCommandName());
+        setResponseObject(listResponse);
+    }
+
+    @Override
+    public long getEntityOwnerId() {
+        return CallContext.current().getCallingAccount().getId();
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/api/response/NsxControllerResponse.java

@@ -0,0 +1,136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.response;
+
+import com.cloud.network.nsx.NsxProvider;
+import com.cloud.serializer.Param;
+import com.google.gson.annotations.SerializedName;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseResponse;
+import org.apache.cloudstack.api.EntityReference;
+
+@EntityReference(value = {NsxProvider.class})
+public class NsxControllerResponse extends BaseResponse {
+    @SerializedName(ApiConstants.NSX_PROVIDER_UUID)
+    @Param(description = "NSX controller ID")
+    private String uuid;
+    @SerializedName(ApiConstants.NAME)
+    @Param(description = "NSX controller name")
+    private String name;
+
+    @SerializedName(ApiConstants.ZONE_ID)
+    @Param(description = "Zone ID to which the NSX controller is associated with")
+    private String zoneId;
+
+    @SerializedName(ApiConstants.ZONE_NAME)
+    @Param(description = "Zone name to which the NSX controller is associated with")
+    private String zoneName;
+
+    @SerializedName(ApiConstants.HOST_NAME)
+    @Param(description = "NSX controller hostname or IP address")
+    private String hostname;
+
+    @SerializedName(ApiConstants.PORT)
+    @Param(description = "NSX controller port")
+    private String port;
+
+    @SerializedName(ApiConstants.TIER0_GATEWAY)
+    @Param(description = "The tier-0 gateway network. Tier-0 gateway is responsible for handling" +
+            " traffic between logical and physical networks"
+    )
+    private String tier0Gateway;
+
+    @SerializedName(ApiConstants.EDGE_CLUSTER)
+    @Param(description = "The name of the edge cluster. An edge cluster is a logical grouping of edge nodes in NSX")
+    private String edgeCluster;
+
+    @SerializedName(ApiConstants.TRANSPORT_ZONE)
+    @Param(description = "The name of the transport zone. A transport zone controls to which hosts a logical switch can reach")
+    private String transportZone;
+
+    public String getUuid() {
+        return uuid;
+    }
+
+    public void setUuid(String uuid) {
+        this.uuid = uuid;
+    }
+
+    public String getName() {
+        return name;
+    }
+
+    public void setName(String name) {
+        this.name = name;
+    }
+
+    public String getZoneId() {
+        return zoneId;
+    }
+
+    public void setZoneId(String zoneId) {
+        this.zoneId = zoneId;
+    }
+
+    public String getZoneName() {
+        return zoneName;
+    }
+
+    public void setZoneName(String zoneName) {
+        this.zoneName = zoneName;
+    }
+
+    public String getHostname() {
+        return hostname;
+    }
+
+    public void setHostname(String hostname) {
+        this.hostname = hostname;
+    }
+
+    public String getPort() {
+        return port;
+    }
+
+    public void setPort(String port) {
+        this.port = port;
+    }
+
+    public String getTier0Gateway() {
+        return tier0Gateway;
+    }
+
+    public void setTier0Gateway(String tier0Gateway) {
+        this.tier0Gateway = tier0Gateway;
+    }
+
+    public String getEdgeCluster() {
+        return edgeCluster;
+    }
+
+    public void setEdgeCluster(String edgeCluster) {
+        this.edgeCluster = edgeCluster;
+    }
+
+    public String getTransportZone() {
+        return transportZone;
+    }
+
+    public void setTransportZone(String transportZone) {
+        this.transportZone = transportZone;
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/resource/NsxLoadBalancerMember.java

@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.resource;
+
+public class NsxLoadBalancerMember {
+    private long vmId;
+    private String vmIp;
+    private int port;
+
+    public NsxLoadBalancerMember(long vmId, String vmIp, int port) {
+        this.vmId = vmId;
+        this.vmIp = vmIp;
+        this.port = port;
+    }
+
+    public long getVmId() {
+        return vmId;
+    }
+
+    public String getVmIp() {
+        return vmIp;
+    }
+
+    public int getPort() {
+        return port;
+    }
+}

plugins/network-elements/nsx/src/main/java/org/apache/cloudstack/resource/NsxNetworkRule.java

@@ -0,0 +1,397 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.resource;
+
+import com.cloud.network.Network;
+
+import java.util.List;
+
+public class NsxNetworkRule {
+
+    public enum NsxRuleAction {
+        ALLOW, DROP
+    }
+
+    private long domainId;
+    private long accountId;
+    private long zoneId;
+    private Long networkResourceId;
+    private String networkResourceName;
+    private boolean isVpcResource;
+    private long vmId;
+    private long ruleId;
+    private String publicIp;
+    private String vmIp;
+    private String publicPort;
+    private String privatePort;
+    private String protocol;
+    private String algorithm;
+    private List<NsxLoadBalancerMember> memberList;
+    private NsxRuleAction aclAction;
+    private List<String> sourceCidrList;
+    private List<String> destinationCidrList;
+    private Integer icmpCode;
+
+    private Integer icmpType;
+    private String trafficType;
+    private Network.Service service;
+
+    public long getDomainId() {
+        return domainId;
+    }
+
+    public void setDomainId(long domainId) {
+        this.domainId = domainId;
+    }
+
+    public long getAccountId() {
+        return accountId;
+    }
+
+    public void setAccountId(long accountId) {
+        this.accountId = accountId;
+    }
+
+    public long getZoneId() {
+        return zoneId;
+    }
+
+    public void setZoneId(long zoneId) {
+        this.zoneId = zoneId;
+    }
+
+    public Long getNetworkResourceId() {
+        return networkResourceId;
+    }
+
+    public void setNetworkResourceId(Long networkResourceId) {
+        this.networkResourceId = networkResourceId;
+    }
+
+    public String getNetworkResourceName() {
+        return networkResourceName;
+    }
+
+    public void setNetworkResourceName(String networkResourceName) {
+        this.networkResourceName = networkResourceName;
+    }
+
+    public boolean isVpcResource() {
+        return isVpcResource;
+    }
+
+    public void setVpcResource(boolean vpcResource) {
+        isVpcResource = vpcResource;
+    }
+
+    public long getVmId() {
+        return vmId;
+    }
+
+    public void setVmId(long vmId) {
+        this.vmId = vmId;
+    }
+
+    public long getRuleId() {
+        return ruleId;
+    }
+
+    public void setRuleId(long ruleId) {
+        this.ruleId = ruleId;
+    }
+
+    public String getPublicIp() {
+        return publicIp;
+    }
+
+    public void setPublicIp(String publicIp) {
+        this.publicIp = publicIp;
+    }
+
+    public String getVmIp() {
+        return vmIp;
+    }
+
+    public void setVmIp(String vmIp) {
+        this.vmIp = vmIp;
+    }
+
+    public String getPublicPort() {
+        return publicPort;
+    }
+
+    public void setPublicPort(String publicPort) {
+        this.publicPort = publicPort;
+    }
+
+    public String getPrivatePort() {
+        return privatePort;
+    }
+
+    public void setPrivatePort(String privatePort) {
+        this.privatePort = privatePort;
+    }
+
+    public String getProtocol() {
+        return protocol;
+    }
+
+    public void setProtocol(String protocol) {
+        this.protocol = protocol;
+    }
+
+    public void setAlgorithm(String algorithm) {
+        this.algorithm = algorithm;
+    }
+
+    public String getAlgorithm() {
+        return algorithm;
+    }
+
+    public List<NsxLoadBalancerMember> getMemberList() {
+        return memberList;
+    }
+
+    public void setMemberList(List<NsxLoadBalancerMember> memberList) {
+        this.memberList = memberList;
+    }
+
+    public NsxRuleAction getAclAction() {
+        return aclAction;
+    }
+
+    public void setAclAction(NsxRuleAction aclAction) {
+        this.aclAction = aclAction;
+    }
+
+    public Network.Service getService() {
+        return service;
+    }
+
+    public void setService(Network.Service service) {
+        this.service = service;
+    }
+
+    public Integer getIcmpCode() {
+        return icmpCode;
+    }
+
+    public void setIcmpCode(Integer icmpCode) {
+        this.icmpCode = icmpCode;
+    }
+
+    public Integer getIcmpType() {
+        return icmpType;
+    }
+
+    public void setIcmpType(Integer icmpType) {
+        this.icmpType = icmpType;
+    }
+
+    public List<String> getSourceCidrList() {
+        return sourceCidrList;
+    }
+
+    public void setSourceCidrList(List<String> sourceCidrList) {
+        this.sourceCidrList = sourceCidrList;
+    }
+
+    public List<String> getDestinationCidrList() {
+        return destinationCidrList;
+    }
+
+    public void setDestinationCidrList(List<String> destinationCidrList) {
+        this.destinationCidrList = destinationCidrList;
+    }
+
+    public String getTrafficType() {
+        return trafficType;
+    }
+
+    public void setTrafficType(String trafficType) {
+        this.trafficType = trafficType;
+    }
+
+    public static final class Builder {
+        private long domainId;
+        private long accountId;
+        private long zoneId;
+        private Long networkResourceId;
+        private String networkResourceName;
+        private boolean isVpcResource;
+        private long vmId;
+
+        private long ruleId;
+        private String publicIp;
+        private String vmIp;
+        private String publicPort;
+        private String privatePort;
+        private String protocol;
+        private String algorithm;
+        private List<NsxLoadBalancerMember> memberList;
+        private NsxRuleAction aclAction;
+        private List<String> sourceCidrList;
+        private List<String> destinationidrList;
+        private String trafficType;
+        private Integer icmpType;
+        private Integer icmpCode;
+        private Network.Service service;
+
+        public Builder() {
+            // Default constructor
+        }
+
+        public Builder setDomainId(long domainId) {
+            this.domainId = domainId;
+            return this;
+        }
+
+        public Builder setAccountId(long accountId) {
+            this.accountId = accountId;
+            return this;
+        }
+
+        public Builder setZoneId(long zoneId) {
+            this.zoneId = zoneId;
+            return this;
+        }
+
+        public Builder setNetworkResourceId(Long networkResourceId) {
+            this.networkResourceId = networkResourceId;
+            return this;
+        }
+
+        public Builder setNetworkResourceName(String networkResourceName) {
+            this.networkResourceName = networkResourceName;
+            return this;
+        }
+
+        public Builder setVpcResource(boolean isVpcResource) {
+            this.isVpcResource = isVpcResource;
+            return this;
+        }
+
+
+        public Builder setVmId(long vmId) {
+            this.vmId = vmId;
+            return this;
+        }
+
+        public Builder setRuleId(long ruleId) {
+            this.ruleId = ruleId;
+            return this;
+        }
+
+        public Builder setPublicIp(String publicIp) {
+            this.publicIp = publicIp;
+            return this;
+        }
+
+        public Builder setVmIp(String vmIp) {
+            this.vmIp = vmIp;
+            return this;
+        }
+
+        public Builder setPublicPort(String publicPort) {
+            this.publicPort = publicPort;
+            return this;
+        }
+
+        public Builder setPrivatePort(String privatePort) {
+            this.privatePort = privatePort;
+            return this;
+        }
+
+        public Builder setProtocol(String protocol) {
+            this.protocol = protocol;
+            return this;
+        }
+
+        public Builder setAlgorithm(String algorithm) {
+            this.algorithm = algorithm;
+            return this;
+        }
+
+        public Builder setMemberList(List<NsxLoadBalancerMember> memberList) {
+            this.memberList = memberList;
+            return this;
+        }
+
+
+        public Builder setAclAction(NsxRuleAction aclAction) {
+            this.aclAction = aclAction;
+            return this;
+        }
+
+        public Builder setTrafficType(String trafficType) {
+            this.trafficType = trafficType;
+            return this;
+        }
+
+        public Builder setIcmpType(Integer icmpType) {
+            this.icmpType = icmpType;
+            return this;
+        }
+
+        public Builder setIcmpCode(Integer icmpCode) {
+            this.icmpCode = icmpCode;
+            return this;
+        }
+
+        public Builder setSourceCidrList(List<String> sourceCidrList) {
+            this.sourceCidrList = sourceCidrList;
+            return this;
+        }
+
+        public Builder setDestinationCidrList(List<String> destinationCidrList) {
+            this.destinationidrList = destinationCidrList;
+            return this;
+        }
+
+        public Builder setService(Network.Service service) {
+            this.service = service;
+            return this;
+        }
+
+        public NsxNetworkRule build() {
+            NsxNetworkRule rule = new NsxNetworkRule();
+            rule.setDomainId(this.domainId);
+            rule.setAccountId(this.accountId);
+            rule.setZoneId(this.zoneId);
+            rule.setNetworkResourceId(this.networkResourceId);
+            rule.setNetworkResourceName(this.networkResourceName);
+            rule.setVpcResource(this.isVpcResource);
+            rule.setVmId(this.vmId);
+            rule.setVmIp(this.vmIp);
+            rule.setPublicIp(this.publicIp);
+            rule.setPublicPort(this.publicPort);
+            rule.setPrivatePort(this.privatePort);
+            rule.setProtocol(this.protocol);
+            rule.setRuleId(this.ruleId);
+            rule.setAlgorithm(this.algorithm);
+            rule.setMemberList(this.memberList);
+            rule.setAclAction(this.aclAction);
+            rule.setIcmpType(this.icmpType);
+            rule.setIcmpCode(this.icmpCode);
+            rule.setSourceCidrList(this.sourceCidrList);
+            rule.setDestinationCidrList(this.destinationidrList);
+            rule.setTrafficType(this.trafficType);
+            rule.setService(service);
+            return rule;
+        }
+    }
+}