%BOOK_ENTITIES; ]>

This section describes my progress with the project titled "Add Xen/XCP support for GRE SDN controller"

It has been a half way of GSoC2013 journey which I am getting more familiar with its activities. Personally, the previous one-and-a-half month has surprisingly passed by in a blink with lots of pressure. In this first time joining in GSoC2013, I have found it totally new and interesting in its working methods and challenges. Along with those stressful moments, I appreciated all wonderful experiences and knowledge that I have luckily gained from this commitment. It is time to review it all and present in time order. My project named “Add Xen/XCP support for GRE SDN controller”, the proposal can be found here: Proposal Specifically, I need to improve the current GRE SDN controller to work with XCP, a free version of XenServer. Then, as mentioning with my two mentor Sebastien Goasguen and Hugo, I continue to work in next missions as below: re-factor GRE source code by following NiciraNVP plugin design. add GRE support for KVM hypervisor. develop a new ODL plugin using Opendaylight controller for controlling and managing network services via OpenFlow protocol. At the beginning, I started to explore frameworks and tools that CloudStack uses such as Spring framework, marven, git and Reviewboard. In my country developers are more familiar with svn than git, however these tools are also such easy to use so I don't write more about them. I want to note about using Spring in CloudStack and what happen in the Management Server startup process.

Spring provides a Container which contains pre-loaded components CloudStack use. At startup, these components are loaded to Container via two ways: components are declared as beans in componentcontext.xml and applicationcontext.xml <bean id="accountDaoImpl" class="com.cloud.user.dao.AccountDaoImpl" /> <bean id="accountDetailsDaoImpl" class="com.cloud.user.AccountDetailsDaoImpl" /> <bean id="accountJoinDaoImpl" class="com.cloud.api.query.dao.AccountJoinDaoImpl" /> <bean id="accountGuestVlanMapDaoImpl" class="com.cloud.network.dao.AccountGuestVlanMapDaoImpl" /> <bean id="accountVlanMapDaoImpl" class="com.cloud.dc.dao.AccountVlanMapDaoImpl" /> ... components are marked with @Component annotation @Component @Local(value = { NetworkManager.class}) public class NetworkManagerImpl extends ManagerBase implements NetworkManager, Listener { static final Logger s_logger = Logger.getLogger(NetworkManagerImpl.class); As I know recently @Component is not recommended. The fundamental functionality provided by the Spring Container is Dependency Injection (DI). To decouple Java components from other Java components the dependency to a certain other class should get injected into them rather that the class inself creates or finds this object. The general concept between dependency injection is called Inversion of Control. A class should not configure itself but should be configured from outside. A design based on independent classes / components increases the re-usability and possibility to test the software. Example of using DI in CloudStack is showed below: public class NetworkManagerImpl extends ManagerBase implements NetworkManager, Listener { static final Logger s_logger = Logger.getLogger(NetworkManagerImpl.class); @Inject DataCenterDao _dcDao = null; @Inject VlanDao _vlanDao = null; @Inject IPAddressDao _ipAddressDao = null; @Inject AccountDao _accountDao = null;

The MS startup process is defined in cloud-client-ui/WEB-INF/web.xml. The following items will be loaded sequentially: Log4jConfigListener. ContextLoaderListener. CloudStartupServlet. ConsoleServlet. ApiServlet. Of which, CloudStartupServlet will call to ComponentContext to init all of pre-defined components life cycle including configure() and start() phase. The components are divided into seven levels to consecutively startup. Of course, they must override configure() and start() methods. public interface ComponentLifecycle { public static final int RUN_LEVEL_SYSTEM_BOOTSTRAP = 0; // for system level bootstrap components public static final int RUN_LEVEL_SYSTEM = 1; // for system level service components (i.e., DAOs) public static final int RUN_LEVEL_FRAMEWORK_BOOTSTRAP = 2; // for framework startup checkers (i.e., DB migration check) public static final int RUN_LEVEL_FRAMEWORK = 3; // for framework bootstrap components(i.e., clustering management components) public static final int RUN_LEVEL_COMPONENT_BOOTSTRAP = 4; // general manager components public static final int RUN_LEVEL_COMPONENT = 5; // regular adapters, plugin components public static final int RUN_LEVEL_APPLICATION_MAINLOOP = 6; public static final int MAX_RUN_LEVELS = 7; // configuration phase Map<String, String> avoidMap = new HashMap<String, String>(); for(int i = 0; i < ComponentLifecycle.MAX_RUN_LEVELS; i++) { for(Map.Entry<String, ComponentLifecycle> entry : ((Map<String, ComponentLifecycle>)classifiedComponents[i]).entrySet()) { ComponentLifecycle component = entry.getValue(); String implClassName = ComponentContext.getTargetClass(component).getName(); s_logger.info("Configuring " + implClassName); if(avoidMap.containsKey(implClassName)) { s_logger.info("Skip configuration of " + implClassName + " as it is already configured"); continue; } try { component.configure(component.getName(), component.getConfigParams()); } catch (ConfigurationException e) { s_logger.error("Unhandled exception", e); throw new RuntimeException("Unable to configure " + implClassName, e); } avoidMap.put(implClassName, implClassName); } } // starting phase avoidMap.clear(); for(int i = 0; i < ComponentLifecycle.MAX_RUN_LEVELS; i++) { for(Map.Entry<String, ComponentLifecycle> entry : ((Map<String, ComponentLifecycle>)classifiedComponents[i]).entrySet()) { ComponentLifecycle component = entry.getValue(); String implClassName = ComponentContext.getTargetClass(component).getName(); s_logger.info("Starting " + implClassName); if(avoidMap.containsKey(implClassName)) { s_logger.info("Skip configuration of " + implClassName + " as it is already configured"); continue; } try { component.start(); if(getTargetObject(component) instanceof ManagementBean) registerMBean((ManagementBean)getTargetObject(component)); } catch (Exception e) { s_logger.error("Unhandled exception", e); throw new RuntimeException("Unable to start " + implClassName, e); } avoidMap.put(implClassName, implClassName); } }

Networking is the most important component in CloudStack, which serves network services from layer 2 to layer 7. In GsoC, fortunately I have a chance to learn about CloudsStack network architecture. It's really amazing. CloudStack's networking is divided to three parts: NetworkGuru NetworkGuru are responsible for: Design and implementation of virtual networks. IP adress management. See full description about Network Guru on my wiki post: Add Xen/XCP support for GRE SDN controller NetworkElement NetworkElement in my opinion is the most important in CloudStack's networking. It represents components that are present in network. Such components can provide any kind of network service or support the virtual networking infrastructure and their interface is defined by com.cloud.network.element.NetworkElement. There are two things we attend in NetworkElement: services and elements. CloudStack currently support network services below: Dhcp service. Connectivity service. Firewall service. Load Balancing service. Network ACL service. Port Forwarding service. SourceNat service. StaticNat service. UerData service. Vpc service. Many Element implemented these above services. They are: MidonetElement. BigSwitchVnsElement. NiciraNvpElement. BaremetalElement. VirtualRouterElement. VpcVirtualRouterElement. CiscoVnmcElement. JuniperSrxExternalFirewallElement. ElasticLbElement. F5ExternalLbElement. CloudZoneNetworkElement. BaremetalPxeElement. BaremetalUserdataElement. DnsNotifier. OvsElement. SecurityGroupElement. See full description about Network Element on my wiki post: Add Xen/XCP support for GRE SDN controller In addition, Elements willing to support network services have to implement corresponding methods from ServicesProvider interfaces. For example, NiciraNvpElement want to support staticNat rule so it has to override applyStaticNats method. NetworkManager Network Manager handle the resources managed by the network elements. They are also implemented as many other "resource" managers in CloudStack. For instance, the manager for setting up L2-in-L3 networks with Open vSwitch is OvsTunnelManagerImpl, whereas Virtual Router lifecycle is managed by VirtualApplianceManagerImpl. In the project, I'm going to implement L3 services for sdn controller, so I need to understand how network services implement.

As I said in previous session, network services are represented in ServiceProvider interfaces. There are currently 12 service providers including: Dhcp, Firewall, IpDeployer, LoadBalancing, NetworkACL, PortForwarding, RemoteAccessVpn, Site2siteVpn, SourceNat, StaticNat, UserData and Vpc. In this session, I'll focus on L3 services implemented in CloudStack such as FirewallRule, PortForwardingRule, StaticNatRules, etc. All services are implemented at NetworkElement and every elements including network plugins (nicira nvp, bigswitch vns,...), which is willing to support them, must override from NetworkElement. For a clearly exlaination, I'll take the StaticNat service implemented in Nicira NVP plugin, source code can be found in NiciraNvpElement.java. NiciraNvpElement firstly has to check whether it can handle the StaticNat service via canHandle() method: if (!canHandle(network, Service.StaticNat)) { return false; } protected boolean canHandle(Network network, Service service) { s_logger.debug("Checking if NiciraNvpElement can handle service " + service.getName() + " on network " + network.getDisplayText()); //Check if network has right broadcast domain type if (network.getBroadcastDomainType() != BroadcastDomainType.Lswitch) { return false; } //Check if NiciraNVP is the provider of the network if (!_networkModel.isProviderForNetwork(getProvider(), network.getId())) { s_logger.debug("NiciraNvpElement is not a provider for network " + network.getDisplayText()); return false; } //Check if NiciraNVP support StaticNat service if (!_ntwkSrvcDao.canProviderSupportServiceInNetwork(network.getId(), service, Network.Provider.NiciraNvp)) { s_logger.debug("NiciraNvpElement can't provide the " + service.getName() + " service on network " + network.getDisplayText()); return false; } return true; } NiciraNvp checks whether it is the provider of the network and it can support StaticNat service or not. After the checking, it makes a staticNat rely on their own Logical Router, that I won't report detail here. The sequence diagram for applying a L3 service is described below: network_service.png: Network services implementation sequence diagram. After understanding network architecture and services implementation, I decided to improve Ovs plugin to support L3 services. Because it's the native sdn controller, I want to use Virtual Router for L3 services deployment. This work will be done when I call L3 services execution from OvsElement to VirtualRouterManager. With Xen hosts, VirtualRouterElement execute L3 services via xapi plugin calls. I make a flow which describes more detail about the process below l3_services.png: Layer 3 services implementation in Ovs plugin. In Xen, all of L3 services are executed via a Xapi plugin naming "vmops". Default, Virtual Routers (VR) control and manage network services. In this case, "vmops" forwards request to network-responsibility shellscripts such as call_firewall.sh or call_loadbalancer.sh. They then parse parameters and call to shellscripts placed in VR via ssh. For example, if we define a staticNat rule, the process occurs as follow: VR Manager (VirtualNetworkApplianceManager) send staticNat command to AgentManager: try { answers = _agentMgr.send(router.getHostId(), cmds); } catch (OperationTimedoutException e) { s_logger.warn("Timed Out", e); throw new AgentUnavailableException("Unable to send commands to virtual router ", router.getHostId(), e); } AgentManager makes a xapi plugin call to host containing the VR String result = callHostPlugin(conn, "vmops", "setFirewallRule", "args", args.toString()); "vmops" forwards the request to "call_firewall" shellscript @echo def setFirewallRule(session, args): sargs = args['args'] cmd = sargs.split(' ') cmd.insert(0, "/usr/lib/xcp/bin/call_firewall.sh") cmd.insert(0, "/bin/bash") try: txt = util.pread2(cmd) txt = 'success' except: util.SMlog(" set firewall rule failed " ) txt = '' return txt "call_firewall" parses the parameters and directly request to a shellscript placed in VR via ssh command ssh -p 3922 -q -o StrictHostKeyChecking=no -i $cert root@$domRIp "/root/firewall.sh $*" That's all. "firewall" script set some iptable rules for executing the staticNat rule

The project need to add an open source Openflow controller, and I decided to choose Opendaylight. Opendaylight (ODL) is an interesting experience that I have in GSoC. Before starting project, I still confused between many open source OpenFlow controller such as POX, NOX, Beacon, Floodlight, Opendaylight... Honestly, I do not have large knowledge of OpenFlow protocol and also open source SDN controller at the beginning of project. When the project was in progress, I chose Floodlight, a safe solution because of its rich of functionality and good documents. However, Sebastien Goasguen, CloudStack GSoC manager, recommended me to try Opendaylight. From the collected information, I found that Opendaylight are getting a lot of attentions from the community. At the moment, ODL has three main projects: Opendaylight Controller. Opendaylight Network Virtualization Platform. Opendaylight Virtual Tennant Network. It also has six incubating projects: YANG Tools. LISP Flow Mapping. OVSDB Integration. Openflow Protocol Library. BGP-LS/PCEP. Defense4All. For integrating Opendaylight to control and manage network services, I chose ODL Controller project, which is developed by Cisco programmers. The ODL controller is a pure software and as a JVM it can be run on any OS as long as it supports Java. The structure of the ODL controller is shown below: odl_structure.jpg: Opendaylight Controller architecture. The structure is separated to three layers: Network Apps and Orchestration: the top layer consists of applications that utilize the network for normal network communications. Also included in this layer are business and network logic applications that control and monitor network behavior. Controller Platform: the middle layer is the framework in which the SDN abstractions can manifest; providing a set of common APIs to the application layer (commonly referred to as the northbound interface), while implementing one or more protocols for command and control of the physical hardware within the network (typically referred to as the southbound interface). Physical and Virtual Network Devices: The bottom layer consists of the physical and virtual devices, switches, routers, etc., that make up the connective fabric between all endpoints within the network. This controller is implemented strictly in software and is contained within its own Java Virtual Machine (JVM). Source code can be cloned from git: git clone https://git.opendaylight.org/gerrit/p/controller.git Applications make request to ODL Northbound API via HTTP. Currently, ODL supports not too much services. All REST API we can find here: ODL Controller REST API For example, we can add query list of exist flows configured on a Node in a give container. GET http://controller-ip/controller/nb/v2/flow/{containerName}/{nodeType}/{nodeId} {containername}: name of the container. The container name for the base controller is “default” {nodeType}: type of the node being programmed {nodeId}: node identifier Or we can add a new flow POST http://controller-ip/controller/nb/v2/flow/{containerName}/{nodeType}/{nodeId}/{name} with request body in XML or JSON format { "actions" : [ "...", ... ], "nwDst" : "...", "hardTimeout" : "...", "installInHw" : "...", "tosBits" : "...", "cookie" : "...", "node" : { "id" : "...", "type" : "..." }, "dlDst" : "...", "name" : "...", "nwSrc" : "...", "vlanPriority" : "...", "protocol" : "...", "priority" : "...", "vlanId" : "...", "tpDst" : "...", "etherType" : "...", "tpSrc" : "...", "ingressPort" : "...", "idleTimeout" : "...", "dlSrc" : "..." } The following python client writen by Dwcarder describe more specific about using REST API:https://github.com/dwcarder/python-OpenDaylight/blob/master/OpenDaylight.py In project, I learnt how to make HTTP request from CloudStack to ODL for controlling and managing network services. However, there is a problem that ODL currently don't support L2 configuration, while integration ODL to CloudStack requires this. I found an incubating project, led by Brent Salisbury and Evan Zeller from the University of Kentucky, is currently trying to integrate OpenvSwitch database management protocol to ODL, which will allow ODL to view, modify and delete OpenvSwitch object such as bridges and ports by way of the OpenvSwitch databse. In short, this project mainly creates a module acts like OVSDB-client and uses JSON-RPC for remote management. I talked to them and jumped into this project. Thus, I'll do an extra work on ODL community to improve ODL Controller support L2 configuration while still integrate ODL to CloudStack by making a new ODL plugin with the same behavior of NiciraNvp and Ovs. Full information about the incubating project can be found here:https://wiki.opendaylight.org/view/Project_Proposals:OVSDB-Integration The next session I will take a short description about XenAPI (also called Xapi), which applications use to interact virtualization resources in Xen hosts.

There are many tool stacks we can use to manage Xen hosts, such as: XL, Xapi, libvirt or Xend. Of which, Xapi is the default. Xapi (or Xen API) is called from applications to control and manage virtualization resources in Xen hosts via XML-RPC. Xapi is the core component of XCP and XenServer and writen by Ocaml language. It's possible to talk directly to Xapi using XML-RPC. This is a way to make remote procedure calls using http requests. In fact, it's possible to send and receive messages using telnet but this is not recommended. The XML-RPC calls are the fixed standard, but we also have bindings to that XML-RPC for Python, C and Java. For example about using XML-RPC calls, I make a simple request written by python to list all VMs on a Xen host. First thing we need to import XenAPI lib: >>> import XenAPI Then we have to authenticate to XenServer or XCP addressed from url with user and password >>> session = XenAPI.Session('https://url') >>> session.login_with_password('user','password') If this works, we've done the hard bit and established communications with our server. Function bellow will list all Vms on this server. >>> session.xenapi.VM.get_all() The answer should be something like: ['OpaqueRef:7b737e4f-58d8-b493-ea31-31324a2de528', 'OpaqueRef:7237b8af-b80c-c021-fbdc-68146d98d7f5', ........., 'OpaqueRef:c3b752b9-1926-9ceb-f36a-408497c3478b'] Which is a list of strings, each of which represents a unique identifier for a particular 'object' on the server. In this case of each 'OpaqueRef' represents a virtual machine. For each VM we can get the name (name_label) >>> [session.xenapi.VM.get_name_label(x) for x in session.xenapi.VM.get_all()] There are a lot of machines in this list. Some of them however are 'template Vms', frozen copies which can't actually run, but which can be cloned in oder to make real virtual machines. We can find out which Vms are templates by calling the VM.get_is_a_template() function. So let's combinate the two in order to produce a list of all the real Vms on my server: >>> [session.xenapi.VM.get_name_label(x) for x in session.xenapi.VM.get_all() if not session.xenapi.VM.get_is_a_template(x)] The answer should be something like: ['Debian Etch 4.0 (2)', 'Debian Etch 4.0 (1)', 'test9', 'test4', 'Control domain on host: ebony', 'Control domain on host: localhost.localdomain', 'test3', 'Debian Sarge 3.1 (1)', 'test2', 'Debian Etch 4.0 (3)', 'test1', 'test3', 'test7', 'test5'] Finally it's only polite to log out of the server. This allows it to garbage collect the no-longer active session. >>> session.logout() Full python script can be found here: Xapi python client We can find Xapi source code from: https://github.com/xen-org/xen-api Xapi come with some main classes, each of them refer to a virtual resource object in Xen such as: VM: refer to virtual machine. VIF: refer to virtual NIC. VDI: refer to virtual volume or hard disk. ... Full information about Xapi source code we can find here. http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ Click on each item we can see more detail. Xapi plugin Xapi has an extension mechanism that allows one to install a Python script (usually but it can be any executable) on the Xen host, and then call that through the Xapi. Writing a Xapi plugin in Python is simplified by using the XenAPIPlugin module, which is by default installed in dom0 in XCP. In my GsoC project, I have to call some plugin scripts to control and manage virtual switches. For example, I inserted a new function to get network name-label in vmops script. Then, we can call it directly from XE command line or via XML-RPC. Here is a simple call from XE: $xe host-call-plugin host-uuid=host-uuid plugin=vmops fn=getLabel If the plugins has some arguments, it should be inserted with "args:" keyword. In ACS, almost plugins are called from CitrixResourceBase.java. With my above function, I inserted a new method into CitrixResourceBase.java and called to the plugin as below: private String getLabel() { Connection conn = getConnection(); String result = callHostPlugin(conn, "ovstunnel", "getLabel"); return result; } Of which, Connection class will init a session to Xen host and callHostPlugin method executes a XML-RPC call to plugin. Note that every Xapi plugin scripts must be placed into /etc/xapi.d/plugins.

In one-and-a-half month, I have understood all of above knowledge and finished two things: improve gre controller to support XCP. re-factor GRE source code by following NiciraNVP plugin design. improve gre controller to support XCP From the understanding of how the native SDN works, a small patch has been made to help it works with Xen Cloud Platform (XCP) version 1.6. Without the patch, this controller can serve XenServer only, the commercial version of XCP. I did try SDN with XCP and debug to find out what errors are and why they occur. After some efforts, I figured out following problems: The SDN controller has to know what interface it'll deploy GRE tunnels. To do this check, it looks into network to find out the PIF's interface. It has a network name-label, which user defined in the deploy zone phase. If not, it will be replaced by a default label. However, XCP's network has no user-defined or default name-label. Therefore in this step I have made a trick. I used whatever name-label found in the XCP host to bypass this check. When creating an OVS bridge, the controller creates a new dom0 vif, plugs to the bridge and immediately unplugs it. This action aims to ask XenServer create the bridge without running ovs-vsctl or brctl script. I saw that it is not very important to XCP hosts and also generates an error from xenopsd daemon, so I ignored this step. The script playing a direct role to interact with openvswitch is ovstunnel. It requires a lib named cloudstack_pluginlib, which does not exist in XCP. Thus, I inserted this file into copying process from CloudStack to XCP when add-host phase occurs. The "setup_ovs_bridge" function in ovstunnel takes a look into XenServer version to act a blocking IPv6. However, product_version parameter does not exist on XCP. It uses platform_version parameter instead. So, I decided to ignore this step. The patch is already committed to sdnextensions branch. It is also the primary branch I have been working on this GSoC period. re-factor GRE source code by following NiciraNVP plugin design GRE source code was re-factored with following changes: add Connectivity service checking: All of L2 configuration methods now have to check whether Ovs plugin can handle Connectivity service.. move commands / answers to a new package: com.cloud.agent.api. add new NetworkProvider: Ovs. add L3 services to Ovs Capabilities: Ovs Capability now is set enabled to such L3 services as SourceNat, StaticNat, PortForwarding, RedundantRouter, Gateway. L2 service Connectivity is also set enabled. add L3 services prototype code to OvsElement.java With the knowledge about CloudStack's network architecture I have learned and represented above, I made a patch which permits guest networks can reach each other via private IPaddress without using VPC mode. Proposal can be found here: Routing between guest networks In next days, I will done the following things: implement L3 services with Virtual Router. improve Ovs to support KVM hypervisor. add new ODL plugin using ODL controller to control and manager network services.