bug 14042: Don't set dhcp:router option on DHCP server for non-default network on CentOS/RHEL

The routing table with two nics may be messed up, due to we sent same
router(gateway) information from different DHCP server, in order to specify
default gateway. E.g.

Network A: 192.168.1.0/24, gw 192.168.1.1
Network B: 192.168.2.0/24, gw 192.168.2.1

User VM: Nic 1 connect to network A, get ip 192.168.1.10; nic 2 connect to
network B, get ip 192.168.2.10.

Set network A as the default network of user VM.

Currently we would send this information to user VM through DHCP offer:
In network A: dhcp-option:router 192.168.1.1
In network B: dhcp-option:router 192.168.1.1

So both NIC in the guest VM would receive 192.168.1.1 as router(gateway).

But, in CentOS 5.6, dhclient-scripts try to tell if the gateway is reachable
for current subnet.

So when we try to enable nic 2(eth1) of user VM, dhclient would receive:
IP: 192.168.2.10
Mask: 255.255.255.0
Router: 192.168.1.1

Then it would found that the specified gateway(router) is not within its own
subnet(192.168.2.0/24). But since we send out this ip(192.168.1.1) as the
gateway for it, dhclient thought that it should got someway to access the
network through this IP. So it would execute:

ip route add 192.168.1.1 dev eth1
ip route replace default via 192.168.1.1 dev eth1

But it can never reach 192.168.1.1(which is in the eth0's subnet and the
gateway of eth0) by go through eth1 interface. So it is messed up.

We've tested Windows 2008 R2, CentOS 5.3, CentOS 5.6 and Ubuntu 10.04. Windows
and Ubuntu are fine with above policy.

To solve this, we send different dhcp:router option according to the guest OS
type now.

We may need expand this list later, but for now we only know that CentOS and
RHEL would behavior in this way.

status 14042: resolved fixed

patches/systemvm/debian/config/root/edithosts.sh

@@ -93,17 +93,23 @@ echo "$ip $host" >> $HOSTS
 
 if [ "$dflt" != "" ]
 then
-  logger -t cloud "$0: setting default router to $dflt"
   #make sure dnsmasq looks into options file
   sed -i /dhcp-optsfile/d /etc/dnsmasq.conf
   echo "dhcp-optsfile=$DHCP_OPTS" >> /etc/dnsmasq.conf
 
   tag=$(echo $ip | tr '.' '_')
   sed -i /$tag/d $DHCP_OPTS
-  echo "$tag,3,$dflt" >> $DHCP_OPTS
+  if [ "$dflt" != "0.0.0.0" ]
+  then
+      logger -t cloud "$0: setting default router for $ip to $dflt"
+      echo "$tag,3,$dflt" >> $DHCP_OPTS
+  else
+      logger -t cloud "$0: unset default router for $ip"
+      echo "$tag,3," >> $DHCP_OPTS
+  fi
   if [ "$dns" != "" ] 
   then
-    logger -t cloud "$0: setting dns server to $dns"
+    logger -t cloud "$0: setting dns server for $ip to $dns"
     echo "$tag,6,$dns" >> $DHCP_OPTS
   fi
   [ "$routes" != "" ] && echo "$tag,121,$routes" >> $DHCP_OPTS

server/src/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java

@@ -167,6 +167,7 @@ import com.cloud.offerings.dao.NetworkOfferingDao;
 import com.cloud.resource.ResourceManager;
 import com.cloud.service.ServiceOfferingVO;
 import com.cloud.service.dao.ServiceOfferingDao;
+import com.cloud.storage.GuestOSVO;
 import com.cloud.storage.StorageManager;
 import com.cloud.storage.VMTemplateVO;
 import com.cloud.storage.Volume.Type;
@@ -2528,7 +2529,16 @@ public class VirtualNetworkApplianceManagerImpl implements VirtualNetworkApplian
     private void createDhcpEntryCommand(VirtualRouter router, UserVm vm, NicVO nic, Commands cmds) {
         DhcpEntryCommand dhcpCommand = new DhcpEntryCommand(nic.getMacAddress(), nic.getIp4Address(), vm.getHostName());
         DataCenterVO dcVo = _dcDao.findById(router.getDataCenterIdToDeployIn());
-        dhcpCommand.setDefaultRouter(findGatewayIp(vm.getId()));
+        String gatewayIp = findGatewayIp(vm.getId());
+        if (!gatewayIp.equals(nic.getGateway())) {
+            GuestOSVO guestOS = _guestOSDao.findById(vm.getGuestOSId());
+            // Don't set dhcp:router option for non-default nic on CentOS/RHEL, because they would set routing on wrong interface
+            // This is tricky, we may need to update this when we have more information on various OS's behavior
+            if (guestOS.getDisplayName().startsWith("CentOS") || guestOS.getDisplayName().startsWith("Red Hat Enterprise")) {
+                gatewayIp = "0.0.0.0";
+            }
+        }
+        dhcpCommand.setDefaultRouter(gatewayIp);
         dhcpCommand.setDefaultDns(findDefaultDnsIp(vm.getId()));
 
         dhcpCommand.setAccessDetail(NetworkElementCommand.ROUTER_IP, getRouterControlIp(router.getId()));