SystemVM optimizations (#5831)

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* refactor patching logic - certs

* move logic of fixing rootdiskcontroller from upgrade to kubernetes service

* add livepatch option to restart network & vpc

* smooth upgrade of cks clusters

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* address comments

* add temp links of download.cloudstack.org

* address part of the comments

* address comments

* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0

* address comments - simplify

* fix vue3 related icon changes

* allow network commands when router template version is lower but is patched

* add internal LB to the list of routers to be patched on network restart with live patch

* add unit tests for API param validations and new helper utilities - file scp & checksum validations

* perform patching only for non-user i.e., system VMs

* add test to validate params

* remove unused import

* add column to domain_router to display software version and support networkrestart with livePatch from router view

* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false

* use router software version instead of checksum

* show N/A if no software version reported i.e., in upgraded envs

* fix deb failure

* update pom to official links of systemVM template

agent/src/main/java/com/cloud/agent/Agent.java

@@ -763,8 +763,10 @@ public class Agent implements HandlerFactory, IAgentControl {
             throw new CloudRuntimeException("Unable to save received agent client and ca certificates", e);
         }
 
+        String ksPassphrase = _shell.getPersistentProperty(null, KeyStoreUtils.KS_PASSPHRASE_PROPERTY);
         Script script = new Script(_keystoreCertImportPath, 300000, s_logger);
         script.add(agentFile.getAbsolutePath());
+        script.add(ksPassphrase);
         script.add(keyStoreFile);
         script.add(KeyStoreUtils.AGENT_MODE);
         script.add(certFile);

api/src/main/java/com/cloud/event/EventTypes.java

@@ -637,6 +637,9 @@ public class EventTypes {
     // Storage Policies
     public static final String EVENT_IMPORT_VCENTER_STORAGE_POLICIES = "IMPORT.VCENTER.STORAGE.POLICIES";
 
+    // SystemVM
+    public static final String EVENT_LIVE_PATCH_SYSTEMVM = "LIVE.PATCH.SYSTEM.VM";
+
     static {
 
         // TODO: need a way to force author adding event types to declare the entity details as well, with out braking
@@ -1048,6 +1051,7 @@ public class EventTypes {
         entityEventDetails.put(EVENT_IMPORT_VCENTER_STORAGE_POLICIES, "StoragePolicies");
 
         entityEventDetails.put(EVENT_IMAGE_STORE_DATA_MIGRATE, ImageStore.class);
+        entityEventDetails.put(EVENT_LIVE_PATCH_SYSTEMVM, "SystemVMs");
     }
 
     public static String getEntityForEvent(String eventName) {

api/src/main/java/com/cloud/network/NetworkService.java

@@ -81,7 +81,7 @@ public interface NetworkService {
 
     boolean deleteNetwork(long networkId, boolean forced);
 
-    boolean restartNetwork(Long networkId, boolean cleanup, boolean makeRedundant, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException;
+    boolean restartNetwork(Long networkId, boolean cleanup, boolean makeRedundant, boolean livePatch, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException;
 
     boolean restartNetwork(RestartNetworkCmd cmd) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException;
 

api/src/main/java/com/cloud/network/router/VirtualRouter.java

@@ -52,4 +52,6 @@ public interface VirtualRouter extends VirtualMachine {
     Long getVpcId();
 
     String getTemplateVersion();
+
+    String getScriptsVersion();
 }

api/src/main/java/com/cloud/network/vpc/VpcService.java

@@ -137,7 +137,7 @@ public interface VpcService {
      */
     boolean restartVpc(RestartVPCCmd cmd) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException;
 
-    boolean restartVpc(Long networkId, boolean cleanup, boolean makeRedundant, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException;
+    boolean restartVpc(Long networkId, boolean cleanup, boolean makeRedundant, boolean livePatch, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException;
 
     /**
      * Returns a Private gateway found in the VPC by id

api/src/main/java/com/cloud/server/ManagementService.java

@@ -40,6 +40,7 @@ import org.apache.cloudstack.api.command.admin.resource.ListCapacityCmd;
 import org.apache.cloudstack.api.command.admin.resource.UploadCustomCertificateCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.ListSystemVMsCmd;
+import org.apache.cloudstack.api.command.admin.systemvm.PatchSystemVMCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.RebootSystemVmCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.ScaleSystemVMCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.StopSystemVmCmd;
@@ -428,5 +429,6 @@ public interface ManagementService {
 
     void cleanupVMReservations();
 
+    Pair<Boolean, String> patchSystemVM(PatchSystemVMCmd cmd);
 
 }

api/src/main/java/org/apache/cloudstack/api/ApiConstants.java

@@ -36,6 +36,7 @@ public class ApiConstants {
     public static final String USER_API_KEY = "userapikey";
     public static final String APPLIED = "applied";
     public static final String LIST_LB_VMIPS = "lbvmips";
+    public static final String LIVE_PATCH = "livepatch";
     public static final String AVAILABLE = "available";
     public static final String BACKUP_ID = "backupid";
     public static final String BACKUP_OFFERING_NAME = "backupofferingname";

api/src/main/java/org/apache/cloudstack/api/command/admin/systemvm/PatchSystemVMCmd.java

@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.api.command.admin.systemvm;
+
+import com.cloud.event.EventTypes;
+import com.cloud.user.Account;
+import com.cloud.utils.Pair;
+import com.cloud.vm.VirtualMachine;
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.SuccessResponse;
+import org.apache.cloudstack.api.response.SystemVmResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+@APICommand(name = PatchSystemVMCmd.APINAME, description = "Attempts to live patch systemVMs - CPVM, SSVM ",
+        responseObject = SuccessResponse.class, requestHasSensitiveInfo = false,
+        responseHasSensitiveInfo = false, authorized = { RoleType.Admin }, since = "4.17.0")
+public class PatchSystemVMCmd extends BaseAsyncCmd {
+    public static final Logger s_logger = Logger.getLogger(PatchSystemVMCmd.class.getName());
+    public static final String APINAME = "patchSystemVm";
+
+    /////////////////////////////////////////////////////
+    //////////////// API parameters /////////////////////
+    /////////////////////////////////////////////////////
+    @Parameter(name = ApiConstants.ID, type = CommandType.UUID, entityType = SystemVmResponse.class,
+            description = "patches systemVM - CPVM/SSVM with the specified ID")
+    private Long id;
+
+    @Parameter(name = ApiConstants.FORCED, type = CommandType.BOOLEAN,
+            description = "If true, initiates copy of scripts and restart of the agent, even if the scripts version matches." +
+                    "To be used with ID parameter only")
+    private Boolean force;
+
+    /////////////////////////////////////////////////////
+    /////////////////// Accessors ///////////////////////
+    /////////////////////////////////////////////////////
+
+
+    public Long getId() {
+        return id;
+    }
+
+    public boolean isForced() {
+        return force != null && force;
+    }
+
+    /////////////////////////////////////////////////////
+    /////////////// API Implementation///////////////////
+    /////////////////////////////////////////////////////
+
+    @Override
+    public String getEventType() {
+        return EventTypes.EVENT_LIVE_PATCH_SYSTEMVM;
+    }
+
+    @Override
+    public String getEventDescription() {
+        return String.format("Attempting to live patch System VM with Id: %s ", this._uuidMgr.getUuid(VirtualMachine.class, getId()));
+    }
+
+    @Override
+    public String getCommandName() {
+        return APINAME.toLowerCase() + BaseCmd.RESPONSE_SUFFIX;
+    }
+
+    @Override
+    public long getEntityOwnerId() {
+        Account account = CallContext.current().getCallingAccount();
+        if (account != null) {
+            return account.getId();
+        }
+
+        return Account.ACCOUNT_ID_SYSTEM;
+    }
+
+    @Override
+    public void execute() {
+        Pair<Boolean, String> patched = _mgr.patchSystemVM(this);
+        if (patched.first()) {
+            final SuccessResponse response = new SuccessResponse(getCommandName());
+            response.setDisplayText(patched.second());
+            setResponseObject(response);
+        } else {
+            throw new ServerApiException(ApiErrorCode.INTERNAL_ERROR, patched.second());
+        }
+    }
+}

api/src/main/java/org/apache/cloudstack/api/command/user/network/RestartNetworkCmd.java

@@ -60,6 +60,11 @@ public class RestartNetworkCmd extends BaseAsyncCmd {
     @Parameter(name = ApiConstants.MAKEREDUNDANT, type = CommandType.BOOLEAN, required = false, description = "Turn the network into a network with redundant routers.", since = "4.11.1")
     private Boolean makeRedundant = false;
 
+    @Parameter(name = ApiConstants.LIVE_PATCH, type = CommandType.BOOLEAN, required = false,
+            description = "Live patches the router software before restarting it. This parameter will only work when 'cleanup' is false.",
+            since = "4.17.0")
+    private Boolean livePatch = false;
+
     /////////////////////////////////////////////////////
     /////////////////// Accessors ///////////////////////
     /////////////////////////////////////////////////////
@@ -81,6 +86,8 @@ public class RestartNetworkCmd extends BaseAsyncCmd {
         return makeRedundant;
     }
 
+    public Boolean getLivePatch() { return livePatch; }
+
     /////////////////////////////////////////////////////
     /////////////// API Implementation///////////////////
     /////////////////////////////////////////////////////

api/src/main/java/org/apache/cloudstack/api/command/user/vpc/RestartVPCCmd.java

@@ -54,6 +54,11 @@ public class RestartVPCCmd extends BaseAsyncCmd {
     @Parameter(name = ApiConstants.MAKEREDUNDANT, type = CommandType.BOOLEAN, required = false, description = "Turn a single VPC into a redundant one.")
     private Boolean makeredundant = false;
 
+    @Parameter(name = ApiConstants.LIVE_PATCH, type = CommandType.BOOLEAN, required = false,
+            description = "Live patches the router software before restarting it. This parameter will only work when 'cleanup' is false.",
+            since = "4.17.0")
+    private Boolean livePatch = false;
+
     /////////////////////////////////////////////////////
     /////////////////// Accessors ///////////////////////
     /////////////////////////////////////////////////////
@@ -70,6 +75,8 @@ public class RestartVPCCmd extends BaseAsyncCmd {
         return makeredundant;
     }
 
+    public Boolean getLivePatch() { return livePatch; }
+
     /////////////////////////////////////////////////////
     /////////////// API Implementation///////////////////
     /////////////////////////////////////////////////////

api/src/main/java/org/apache/cloudstack/api/response/DomainRouterResponse.java

@@ -233,6 +233,10 @@ public class DomainRouterResponse extends BaseResponseWithAnnotations implements
     @Param(description = "Last executed health check result for the router", responseObject = RouterHealthCheckResultResponse.class, since = "4.14")
     List<RouterHealthCheckResultResponse> healthCheckResults;
 
+    @SerializedName("softwareversion")
+    @Param(description = "the version of the code / software in the router")
+    private String softwareVersion;
+
     public DomainRouterResponse() {
         nics = new LinkedHashSet<NicResponse>();
     }
@@ -490,4 +494,12 @@ public class DomainRouterResponse extends BaseResponseWithAnnotations implements
     public void setHealthCheckResults(List<RouterHealthCheckResultResponse> healthCheckResults) {
         this.healthCheckResults = healthCheckResults;
     }
+
+    public String getSoftwareVersion() {
+        return softwareVersion;
+    }
+
+    public void setSoftwareVersion(String softwareVersion) {
+        this.softwareVersion = softwareVersion;
+    }
 }

api/src/test/java/org/apache/cloudstack/api/command/admin/systemvm/PatchSystemVMCmdTest.java

@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.A
+package org.apache.cloudstack.api.command.admin.systemvm;
+
+import com.cloud.server.ManagementService;
+import com.cloud.user.Account;
+import com.cloud.utils.Pair;
+import org.apache.cloudstack.context.CallContext;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.InjectMocks;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import org.powermock.api.mockito.PowerMockito;
+import org.powermock.core.classloader.annotations.PowerMockIgnore;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.test.util.ReflectionTestUtils;
+
+@RunWith(PowerMockRunner.class)
+@PrepareForTest(CallContext.class)
+@PowerMockIgnore({"javax.xml.*", "org.w3c.dom.*", "org.apache.xerces.*", "org.xml.*"})
+public class PatchSystemVMCmdTest {
+
+    @Mock
+    private ManagementService _mgr;
+
+    @InjectMocks
+    PatchSystemVMCmd cmd = new PatchSystemVMCmd();
+
+    @Before
+    public void setUp() throws Exception {
+        MockitoAnnotations.initMocks(this);
+    }
+
+    @Test
+    public void patchValidSystemVM() {
+        ReflectionTestUtils.setField(cmd, "id", 1L);
+        Pair successResponse = new Pair<>(true, "");
+        Mockito.doReturn(successResponse).when(_mgr).patchSystemVM(cmd);
+        try {
+            cmd.execute();
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    @Test
+    public void patchInvalidSystemVM() {
+        ReflectionTestUtils.setField(cmd, "id", null);
+        Pair<Boolean, String> failureResponse = new Pair<>(false, "Please provide a valid ID of a system VM to be patched");
+        Mockito.doReturn(failureResponse).when(_mgr).patchSystemVM(cmd);
+        try {
+            cmd.execute();
+        } catch (Exception e) {
+            Assert.assertEquals(failureResponse.second(), e.getMessage());
+        }
+    }
+
+    @Test
+    public void validateArgsForPatchSystemVMApi() {
+        PowerMockito.mockStatic(CallContext.class);
+        CallContext callContextMock = PowerMockito.mock(CallContext.class);
+        PowerMockito.when(CallContext.current()).thenReturn(callContextMock);
+        Account accountMock = PowerMockito.mock(Account.class);
+        PowerMockito.when(callContextMock.getCallingAccount()).thenReturn(accountMock);
+        Mockito.when(accountMock.getId()).thenReturn(2L);
+        ReflectionTestUtils.setField(cmd, "id", 1L);
+        Assert.assertEquals((long)cmd.getId(), 1L);
+        Assert.assertFalse(cmd.isForced());
+        Assert.assertEquals(cmd.getEntityOwnerId(), 2L);
+
+
+    }
+}

client/pom.xml

@@ -890,32 +890,6 @@
                 </dependency>
             </dependencies>
             <build>
-                <plugins>
-                    <plugin>
-                        <artifactId>maven-antrun-plugin</artifactId>
-                        <version>1.7</version>
-                        <executions>
-                            <!-- Copy the systemvm in the package phase as it is generated by console-proxy in the package
-                                phase. -->
-                            <execution>
-                                <id>copy-systemvm</id>
-                                <phase>process-resources</phase>
-                                <goals>
-                                    <goal>run</goal>
-                                </goals>
-                                <configuration>
-                                    <target>
-                                        <copy todir="${basedir}/target/common/vms">
-                                            <fileset dir="${basedir}/../systemvm/dist">
-                                                <include name="systemvm.iso" />
-                                            </fileset>
-                                        </copy>
-                                    </target>
-                                </configuration>
-                            </execution>
-                        </executions>
-                    </plugin>
-                </plugins>
             </build>
         </profile>
         <profile>

core/src/main/java/com/cloud/agent/api/PatchSystemVmAnswer.java

@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+public class PatchSystemVmAnswer extends Answer {
+
+    String templateVersion;
+    String scriptsVersion;
+
+    public PatchSystemVmAnswer() {
+    }
+
+    public PatchSystemVmAnswer(PatchSystemVmCommand cmd, String details, String templateVersion, String scriptsVersion) {
+        super(cmd, true, details);
+        this.templateVersion = templateVersion;
+        this.scriptsVersion = scriptsVersion;
+    }
+
+    public PatchSystemVmAnswer(PatchSystemVmCommand cmd, String details) {
+        super(cmd, false, details);
+    }
+
+    public String getTemplateVersion() {
+        return this.templateVersion;
+    }
+
+    public String getScriptsVersion() {
+        return this.scriptsVersion;
+    }
+}

core/src/main/java/com/cloud/agent/api/PatchSystemVmCommand.java

@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class PatchSystemVmCommand extends Command {
+    boolean forced;
+    HashMap<String, String> accessDetails = new HashMap<String, String>(0);
+
+    public boolean isForced() {
+        return forced;
+    }
+
+    public void setForced(boolean forced) {
+        this.forced = forced;
+    }
+
+    public void setAccessDetail(final Map<String, String> details) {
+        if (details == null) {
+            return;
+        }
+        for (final Map.Entry<String, String> detail : details.entrySet()) {
+            if (detail == null) {
+                continue;
+            }
+            setAccessDetail(detail.getKey(), detail.getValue());
+        }
+    }
+
+    public void setAccessDetail(final String name, final String value) {
+        accessDetails.put(name, value);
+    }
+
+    public String getAccessDetail(final String name) {
+        return accessDetails.get(name);
+    }
+
+    @Override
+    public boolean executeInSequence() {
+        return false;
+    }
+}

core/src/main/java/com/cloud/agent/resource/virtualnetwork/VRScripts.java

@@ -41,6 +41,7 @@ public class VRScripts {
     public static final String IP_ALIAS_CONFIG = "ip_aliases.json";
     public static final String LOAD_BALANCER_CONFIG = "load_balancer.json";
 
+    public static final String SYSTEM_VM_PATCHED = "patched.sh";
     public final static String CONFIG_CACHE_LOCATION = "/var/cache/cloud/";
     public final static Duration VR_SCRIPT_EXEC_TIMEOUT = Duration.standardMinutes(10);
     public final static Duration CONNECTION_TIMEOUT = Duration.standardMinutes(1);

core/src/main/java/com/cloud/agent/resource/virtualnetwork/VirtualRoutingResource.java

@@ -34,6 +34,7 @@ import java.util.concurrent.locks.ReentrantLock;
 
 import javax.naming.ConfigurationException;
 
+import com.cloud.utils.PasswordGenerator;
 import org.apache.cloudstack.ca.SetupCertificateAnswer;
 import org.apache.cloudstack.ca.SetupCertificateCommand;
 import org.apache.cloudstack.ca.SetupKeyStoreCommand;
@@ -174,11 +175,12 @@ public class VirtualRoutingResource {
     }
 
     private Answer execute(final SetupCertificateCommand cmd) {
-        final String args = String.format("/usr/local/cloud/systemvm/conf/agent.properties " +
+        final String args = String.format("/usr/local/cloud/systemvm/conf/agent.properties %s " +
                         "/usr/local/cloud/systemvm/conf/%s %s " +
                         "/usr/local/cloud/systemvm/conf/%s \"%s\" " +
                         "/usr/local/cloud/systemvm/conf/%s \"%s\" " +
                         "/usr/local/cloud/systemvm/conf/%s \"%s\"",
+                PasswordGenerator.generateRandomPassword(16),
                 KeyStoreUtils.KS_FILENAME,
                 KeyStoreUtils.SSH_MODE,
                 KeyStoreUtils.CERT_FILENAME,
@@ -582,4 +584,23 @@ public class VirtualRoutingResource {
         }
         return new Answer(cmd, false, "Fail to recognize aggregation action " + action.toString());
     }
+
+    public boolean isSystemVMSetup(String vmName, String controlIp) throws InterruptedException {
+        if (vmName.startsWith("s-") || vmName.startsWith("v-")) {
+            ScriptConfigItem scriptConfigItem = new ScriptConfigItem(VRScripts.SYSTEM_VM_PATCHED, "/opt/cloud/bin/keystore*");
+            ExecutionResult result = new ExecutionResult(false, "");
+            int retries = 0;
+            while (!result.isSuccess() && retries < 600) {
+                result = applyConfigToVR(controlIp, scriptConfigItem, VRScripts.VR_SCRIPT_EXEC_TIMEOUT);
+                if (result.isSuccess()) {
+                    return true;
+                }
+
+                retries++;
+                Thread.sleep(1000);
+            }
+            return false;
+        }
+        return true;
+    }
 }

core/src/main/java/com/cloud/resource/ServerResource.java

@@ -26,12 +26,19 @@ import com.cloud.agent.api.PingCommand;
 import com.cloud.agent.api.StartupCommand;
 import com.cloud.host.Host;
 import com.cloud.utils.component.Manager;
+import org.apache.cloudstack.utils.security.KeyStoreUtils;
 
 /**
  * ServerResource is a generic container to execute commands sent
  */
 public interface ServerResource extends Manager {
 
+    String[] systemVmPatchFiles = new String[] { "agent.zip", "cloud-scripts.tgz", "patch-sysvms.sh" };
+    String[] certificateFiles = new String[] {KeyStoreUtils.CERT_FILENAME, KeyStoreUtils.CACERT_FILENAME, KeyStoreUtils.PKEY_FILENAME};
+
+    String SSHKEYSPATH = "/root/.ssh";
+    String SSHPRVKEYPATH = SSHKEYSPATH +"/id_rsa.cloud";
+
     /**
      * @return Host.Type type of the computing server we have.
      */

debian/cloudstack-common.install

@@ -15,7 +15,6 @@
 # specific language governing permissions and limitations
 # under the License.
 
-/usr/share/cloudstack-common/vms/systemvm.iso
 /usr/share/cloudstack-common/scripts/installer/*
 /usr/share/cloudstack-common/scripts/network/*
 /usr/share/cloudstack-common/scripts/storage/*
@@ -29,6 +28,7 @@
 /usr/share/cloudstack-common/scripts/vm/hypervisor/vmware/*
 /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/*
 /usr/share/cloudstack-common/lib/*
+/usr/share/cloudstack-common/vms/*
 /usr/bin/cloudstack-set-guest-password
 /usr/bin/cloudstack-set-guest-sshkey
 /usr/share/pyshared

debian/rules

@@ -117,11 +117,13 @@ override_dh_auto_install:
 	mkdir $(DESTDIR)/usr/share/$(PACKAGE)-common/scripts
 	mkdir $(DESTDIR)/usr/share/$(PACKAGE)-common/setup
 	mkdir $(DESTDIR)/usr/share/$(PACKAGE)-common/lib
+	mkdir $(DESTDIR)/usr/share/$(PACKAGE)-common/vms
 	cp -r scripts/installer $(DESTDIR)/usr/share/$(PACKAGE)-common/scripts
 	cp -r scripts/network $(DESTDIR)/usr/share/$(PACKAGE)-common/scripts
 	cp -r scripts/storage $(DESTDIR)/usr/share/$(PACKAGE)-common/scripts
 	cp -r scripts/util $(DESTDIR)/usr/share/$(PACKAGE)-common/scripts
 	cp -r scripts/vm $(DESTDIR)/usr/share/$(PACKAGE)-common/scripts
+	cp -r systemvm/dist/* $(DESTDIR)/usr/share/$(PACKAGE)-common/vms
 	install -D client/target/utilities/bin/cloud-migrate-databases $(DESTDIR)/usr/bin/cloudstack-migrate-databases
 	install -D client/target/utilities/bin/cloud-set-guest-password $(DESTDIR)/usr/bin/cloudstack-set-guest-password
 	install -D client/target/utilities/bin/cloud-set-guest-sshkey $(DESTDIR)/usr/bin/cloudstack-set-guest-sshkey
@@ -129,7 +131,7 @@ override_dh_auto_install:
 	install -D client/target/utilities/bin/cloud-setup-management $(DESTDIR)/usr/bin/cloudstack-setup-management
 	install -D client/target/utilities/bin/cloud-setup-encryption $(DESTDIR)/usr/bin/cloudstack-setup-encryption
 	install -D client/target/utilities/bin/cloud-sysvmadm $(DESTDIR)/usr/bin/cloudstack-sysvmadm
-	install -D systemvm/dist/systemvm.iso $(DESTDIR)/usr/share/$(PACKAGE)-common/vms/systemvm.iso
+	install -D systemvm/dist/* $(DESTDIR)/usr/share/$(PACKAGE)-common/vms/
 	# We need jasypt for cloud-install-sys-tmplt, so this is a nasty hack to get it into the right place
 	install -D agent/target/dependencies/jasypt-1.9.3.jar $(DESTDIR)/usr/share/$(PACKAGE)-common/lib
 

engine/api/src/main/java/com/cloud/vm/VirtualMachineGuru.java

@@ -20,7 +20,14 @@ import com.cloud.agent.api.Answer;
 import com.cloud.agent.manager.Commands;
 import com.cloud.deploy.DeployDestination;
 import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.utils.PasswordGenerator;
+import com.cloud.utils.exception.CloudRuntimeException;
+import org.apache.cloudstack.ca.CAManager;
+import org.apache.cloudstack.framework.ca.Certificate;
+import org.apache.cloudstack.utils.security.CertUtils;
+import org.apache.cloudstack.utils.security.KeyStoreUtils;
 
+import java.io.IOException;
 import java.nio.charset.StandardCharsets;
 import java.util.Base64;
 
@@ -72,4 +79,22 @@ public interface VirtualMachineGuru {
         }
         return base64EncodedPublicKey;
     }
+
+    public static String getEncodedString(String certificate) {
+        return Base64.getEncoder().encodeToString(certificate.replace("\n", KeyStoreUtils.CERT_NEWLINE_ENCODER).replace(" ", KeyStoreUtils.CERT_SPACE_ENCODER).getBytes(StandardCharsets.UTF_8));
+    }
+
+    static void appendCertificateDetails(StringBuilder buf, Certificate certificate) {
+        try {
+            buf.append(" certificate=").append(getEncodedString(CertUtils.x509CertificateToPem(certificate.getClientCertificate())));
+            buf.append(" cacertificate=").append(getEncodedString(CertUtils.x509CertificatesToPem(certificate.getCaCertificates())));
+            if (certificate.getPrivateKey() != null) {
+                buf.append(" privatekey=").append(getEncodedString(CertUtils.privateKeyToPem(certificate.getPrivateKey())));
+            }
+        } catch (IOException e) {
+            throw new CloudRuntimeException("Failed to transform X509 cert to PEM format", e);
+        }
+        buf.append(" keystore_password=").append(getEncodedString(PasswordGenerator.generateRandomPassword(16)));
+        buf.append(" validity=").append(CAManager.CertValidityPeriod.value());
+    }
 }

engine/api/src/main/java/org/apache/cloudstack/engine/orchestration/service/NetworkOrchestrationService.java

@@ -253,7 +253,7 @@ public interface NetworkOrchestrationService {
 
     NetworkProfile convertNetworkToNetworkProfile(long networkId);
 
-    boolean restartNetwork(Long networkId, Account callerAccount, User callerUser, boolean cleanup) throws ConcurrentOperationException, ResourceUnavailableException,
+    boolean restartNetwork(Long networkId, Account callerAccount, User callerUser, boolean cleanup, boolean livePatch) throws ConcurrentOperationException, ResourceUnavailableException,
         InsufficientCapacityException;
 
     boolean shutdownNetworkElementsAndResources(ReservationContext context, boolean b, Network network);

engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java

@@ -552,7 +552,7 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
         advanceExpunge(vm);
     }
 
-    private boolean expungeCommandCanBypassHostMaintenance(VirtualMachine vm) {
+    private boolean isValidSystemVMType(VirtualMachine vm) {
         return VirtualMachine.Type.SecondaryStorageVm.equals(vm.getType()) ||
                 VirtualMachine.Type.ConsoleProxy.equals(vm.getType());
     }
@@ -604,7 +604,7 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
             final Commands cmds = new Commands(Command.OnError.Stop);
 
             for (final Command volumeExpungeCommand : volumeExpungeCommands) {
-                volumeExpungeCommand.setBypassHostMaintenance(expungeCommandCanBypassHostMaintenance(vm));
+                volumeExpungeCommand.setBypassHostMaintenance(isValidSystemVMType(vm));
                 cmds.addCommand(volumeExpungeCommand);
             }
 
@@ -690,7 +690,7 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
             return;
         }
         for (final Command command : cmdList) {
-            command.setBypassHostMaintenance(expungeCommandCanBypassHostMaintenance(vm));
+            command.setBypassHostMaintenance(isValidSystemVMType(vm));
             if (s_logger.isTraceEnabled()) {
                 s_logger.trace(String.format("Adding expunge command [%s] for VM [%s]", command.toString(), vm.toString()));
             }
@@ -1192,8 +1192,12 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
                     handlePath(vmTO.getDisks(), vm.getHypervisorType());
 
                     Commands cmds = new Commands(Command.OnError.Stop);
+                    final Map<String, String> sshAccessDetails = _networkMgr.getSystemVMAccessDetails(vm);
+                    final Map<String, String> ipAddressDetails = new HashMap<>(sshAccessDetails);
+                    ipAddressDetails.remove(NetworkElementCommand.ROUTER_NAME);
 
-                    cmds.addCommand(new StartCommand(vmTO, dest.getHost(), getExecuteInSequence(vm.getHypervisorType())));
+                    StartCommand command = new StartCommand(vmTO, dest.getHost(), getExecuteInSequence(vm.getHypervisorType()));
+                    cmds.addCommand(command);
 
                     vmGuru.finalizeDeployment(cmds, vmProfile, dest, ctx);
 
@@ -1247,12 +1251,16 @@ public class VirtualMachineManagerImpl extends ManagerBase implements VirtualMac
                             final Host vmHost = _hostDao.findById(destHostId);
                             if (vmHost != null && (VirtualMachine.Type.ConsoleProxy.equals(vm.getType()) ||
                                     VirtualMachine.Type.SecondaryStorageVm.equals(vm.getType())) && caManager.canProvisionCertificates()) {
-                                final Map<String, String> sshAccessDetails = _networkMgr.getSystemVMAccessDetails(vm);
                                 for (int retries = 3; retries > 0; retries--) {
                                     try {
-                                        setupAgentSecurity(vmHost, sshAccessDetails, vm);
+                                        final Certificate certificate = caManager.issueCertificate(null, Arrays.asList(vm.getHostName(), vm.getInstanceName()),
+                                                new ArrayList<>(ipAddressDetails.values()), CAManager.CertValidityPeriod.value(), null);
+                                        final boolean result = caManager.deployCertificate(vmHost, certificate, false, sshAccessDetails);
+                                        if (!result) {
+                                            s_logger.error("Failed to setup certificate for system vm: " + vm.getInstanceName());
+                                        }
                                         return;
-                                    } catch (final AgentUnavailableException | OperationTimedoutException e) {
+                                    } catch (final Exception e) {
                                         s_logger.error("Retrying after catching exception while trying to secure agent for systemvm id=" + vm.getId(), e);
                                     }
                                 }

engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java

@@ -38,6 +38,7 @@ import java.util.stream.Collectors;
 import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 
+import com.cloud.server.ManagementServer;
 import org.apache.cloudstack.acl.ControlledEntity.ACLType;
 import org.apache.cloudstack.annotation.AnnotationService;
 import org.apache.cloudstack.annotation.dao.AnnotationDao;
@@ -306,7 +307,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
     @Inject
     VMNetworkMapDao _vmNetworkMapDao;
     @Inject
-    DomainRouterDao _routerDao;
+    DomainRouterDao routerDao;
     @Inject
     RemoteAccessVpnDao _remoteAccessVpnDao;
     @Inject
@@ -320,6 +321,8 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
     @Inject
     private AnnotationDao annotationDao;
     @Inject
+    public ManagementServer mgr;
+    @Inject
     NetworkPermissionDao networkPermissionDao;
 
     List<NetworkGuru> networkGurus;
@@ -1625,8 +1628,8 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
                 throw new UnsupportedOperationException("Cannot update the network resources in sequence when providers other than virtualrouter are used");
         }
         //check if routers are in correct state before proceeding with the update
-        List<DomainRouterVO> routers = _routerDao.listByNetworkAndRole(network.getId(), VirtualRouter.Role.VIRTUAL_ROUTER);
+        List<DomainRouterVO> routers = routerDao.listByNetworkAndRole(network.getId(), VirtualRouter.Role.VIRTUAL_ROUTER);
-        for (DomainRouterVO router : routers) {
+        for (DomainRouterVO router : routers){
             if (router.getRedundantState() == VirtualRouter.RedundantState.UNKNOWN) {
                 if (!forced) {
                     throw new CloudRuntimeException("Domain router: " + router.getInstanceName() + " is in unknown state, Cannot update network. set parameter forced to true for forcing an update");
@@ -3310,7 +3313,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
     }
 
     @Override
-    public boolean restartNetwork(final Long networkId, final Account callerAccount, final User callerUser, final boolean cleanup) throws ConcurrentOperationException, ResourceUnavailableException,
+    public boolean restartNetwork(final Long networkId, final Account callerAccount, final User callerUser, final boolean cleanup, final boolean livePatch) throws ConcurrentOperationException, ResourceUnavailableException,
             InsufficientCapacityException {
         boolean status = true;
         boolean restartRequired = false;
@@ -3329,6 +3332,24 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
             }
             setRestartRequired(network, restartRequired);
             return status;
+        } else if (livePatch) {
+            List<DomainRouterVO> domainRouters = routerDao.listByNetworkAndRole(network.getId(), VirtualRouter.Role.VIRTUAL_ROUTER, VirtualRouter.Role.INTERNAL_LB_VM);
+            for (DomainRouterVO router: domainRouters) {
+                try {
+                    VMInstanceVO instanceVO = _vmDao.findById(router.getId());
+                    if (instanceVO == null) {
+                        s_logger.info("Did not find a virtual router instance for the network");
+                        continue;
+                    }
+                    Pair<Boolean, String> patched = mgr.updateSystemVM(instanceVO, true);
+                    if (patched.first()) {
+                        s_logger.info(String.format("Successfully patched router %s", router));
+                    }
+                } catch (CloudRuntimeException e) {
+                    throw new CloudRuntimeException(String.format("Failed to live patch router: %s", router), e);
+                }
+
+            }
         }
 
         s_logger.debug("Implementing the network " + network + " elements and resources as a part of network restart without cleanup");
@@ -3438,10 +3459,10 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
             return false;
         }
         s_logger.debug("Performing rolling restart of routers of network " + network);
-        destroyExpendableRouters(_routerDao.findByNetwork(network.getId()), context);
+        destroyExpendableRouters(routerDao.findByNetwork(network.getId()), context);
 
         final List<Provider> providersToImplement = getNetworkProviders(network.getId());
-        final List<DomainRouterVO> oldRouters = _routerDao.findByNetwork(network.getId());
+        final List<DomainRouterVO> oldRouters = routerDao.findByNetwork(network.getId());
 
         // Deploy a new router
         if (oldRouters.size() > 0) {
@@ -3474,7 +3495,7 @@ public class NetworkOrchestrator extends ManagerBase implements NetworkOrchestra
             implementNetworkElementsAndResources(dest, context, network, offering);
         }
 
-        return areRoutersRunning(_routerDao.findByNetwork(network.getId()));
+        return areRoutersRunning(routerDao.findByNetwork(network.getId()));
     }
 
     private void setRestartRequired(final NetworkVO network, final boolean restartRequired) {

engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java

@@ -681,8 +681,8 @@ public class SystemVmTemplateRegistration {
 
     /**
      * This method parses the metadata file consisting of the systemVM templates information
-     * @return the version of the systemvm template that is to be used. This is done to in order
+     * @return the version of the systemvm template that is to be used. This is done in order
-     * to fallback on the latest available version of the systemVM template when there does not
+     * to fallback on the latest available version of the systemVM template when there doesn't
      * exist a template corresponding to the current code version.
      */
     public static String parseMetadataFile() {

engine/schema/src/main/java/com/cloud/vm/DomainRouterVO.java

@@ -74,6 +74,9 @@ public class DomainRouterVO extends VMInstanceVO implements VirtualRouter {
     @Enumerated(EnumType.STRING)
     private UpdateState updateState;
 
+    @Column(name= "software_version")
+    private String softwareVersion;
+
     public DomainRouterVO(final long id, final long serviceOfferingId, final long elementId, final String name, final long templateId, final HypervisorType hypervisorType, final long guestOSId, final long domainId,
             final long accountId, final long userId, final boolean isRedundantRouter, final RedundantState redundantState, final boolean haEnabled, final boolean stopPending,
             final Long vpcId) {
@@ -211,4 +214,12 @@ public class DomainRouterVO extends VMInstanceVO implements VirtualRouter {
     public String getName() {
         return instanceName;
     }
+
+    public String getSoftwareVersion() {
+        return softwareVersion;
+    }
+
+    public void setSoftwareVersion(String softwareVersion) {
+        this.softwareVersion = softwareVersion;
+    }
 }

engine/schema/src/main/java/com/cloud/vm/dao/ConsoleProxyDaoImpl.java

@@ -52,7 +52,7 @@ public class ConsoleProxyDaoImpl extends GenericDaoBase<ConsoleProxyVO, Long> im
         + " AS runningVm ON c.id = runningVm.proxy_id WHERE i.state='Running' " + " GROUP BY c.id";
 
     //
-    // query SQL for returnning running VM count at data center basis
+    // query SQL for returning running VM count at data center basis
     //
     private static final String DATACENTER_VM_MATRIX = "SELECT d.id, d.name, count(v.id) AS count"
         + " FROM data_center AS d LEFT JOIN vm_instance AS v ON v.data_center_id=d.id "

engine/schema/src/main/java/com/cloud/vm/dao/DomainRouterDao.java

@@ -121,7 +121,7 @@ public interface DomainRouterDao extends GenericDao<DomainRouterVO, Long> {
 
     List<DomainRouterVO> listByNetworkAndPodAndRole(long networkId, long podId, Role role);
 
-    List<DomainRouterVO> listByNetworkAndRole(long networkId, Role role);
+    List<DomainRouterVO> listByNetworkAndRole(long networkId, Role... roles);
 
     List<DomainRouterVO> listByElementId(long elementId);
 

engine/schema/src/main/java/com/cloud/vm/dao/DomainRouterDaoImpl.java

@@ -79,7 +79,7 @@ public class DomainRouterDaoImpl extends GenericDaoBase<DomainRouterVO, Long> im
         AllFieldsSearch = createSearchBuilder();
         AllFieldsSearch.and("dc", AllFieldsSearch.entity().getDataCenterId(), Op.EQ);
         AllFieldsSearch.and("account", AllFieldsSearch.entity().getAccountId(), Op.EQ);
-        AllFieldsSearch.and("role", AllFieldsSearch.entity().getRole(), Op.EQ);
+        AllFieldsSearch.and("role", AllFieldsSearch.entity().getRole(), Op.IN);
         AllFieldsSearch.and("domainId", AllFieldsSearch.entity().getDomainId(), Op.EQ);
         AllFieldsSearch.and("host", AllFieldsSearch.entity().getHostId(), Op.EQ);
         AllFieldsSearch.and("lastHost", AllFieldsSearch.entity().getLastHostId(), Op.EQ);
@@ -338,10 +338,10 @@ public class DomainRouterDaoImpl extends GenericDaoBase<DomainRouterVO, Long> im
     }
 
     @Override
-    public List<DomainRouterVO> listByNetworkAndRole(final long networkId, final Role role) {
+    public List<DomainRouterVO> listByNetworkAndRole(final long networkId, final Role... roles) {
         final SearchCriteria<DomainRouterVO> sc = AllFieldsSearch.create();
         sc.setJoinParameters("networkRouter", "networkId", networkId);
-        sc.setParameters("role", role);
+        sc.setParameters("role", (Object[])roles);
         return listBy(sc);
     }
 

engine/schema/src/main/resources/META-INF/db/schema-41610to41700.sql

@@ -558,6 +558,44 @@ FROM
         LEFT JOIN `user_vm_details` `custom_ram_size` ON (((`custom_ram_size`.`vm_id` = `vm_instance`.`id`)
         AND (`custom_ram_size`.`name` = 'memory'))));
 
+INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) SELECT UUID(), 3, 'listConfigurations', 'ALLOW', (SELECT MAX(`sort_order`)+1 FROM `cloud`.`role_permissions`) ON DUPLICATE KEY UPDATE rule=rule;
+INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) SELECT UUID(), 3, 'updateConfiguration', 'ALLOW', (SELECT MAX(`sort_order`)+1 FROM `cloud`.`role_permissions`) ON DUPLICATE KEY UPDATE rule=rule;
+
+-- table for network permissions
+CREATE TABLE  `cloud`.`network_permissions` (
+  `id` bigint unsigned NOT NULL auto_increment,
+  `network_id` bigint unsigned NOT NULL,
+  `account_id` bigint unsigned NOT NULL,
+  PRIMARY KEY  (`id`),
+  INDEX `i_network_permission_network_id`(`network_id`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+INSERT INTO `cloud`.`user_vm_details`(`vm_id`, `name`, `value`)
+    SELECT `user_vm_details`.`vm_id`, 'SSH.KeyPairNames', `ssh_keypairs`.`keypair_name`
+        FROM `cloud`.`user_vm_details`
+        INNER JOIN `cloud`.`ssh_keypairs` ON ssh_keypairs.public_key = user_vm_details.value
+        INNER JOIN `cloud`.`vm_instance` ON vm_instance.id = user_vm_details.vm_id
+        WHERE ssh_keypairs.account_id = vm_instance.account_id;
+
+ALTER TABLE `cloud`.`kubernetes_cluster` ADD COLUMN `security_group_id` bigint unsigned DEFAULT NULL,
+ADD CONSTRAINT `fk_kubernetes_cluster__security_group_id` FOREIGN KEY `fk_kubernetes_cluster__security_group_id`(`security_group_id`) REFERENCES `security_group`(`id`) ON DELETE CASCADE;
+
+-- PR#5984 Create table to persist VM stats.
+DROP TABLE IF EXISTS `cloud`.`vm_stats`;
+CREATE TABLE `cloud`.`vm_stats` (
+  `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
+  `vm_id` bigint unsigned NOT NULL,
+  `mgmt_server_id` bigint unsigned NOT NULL,
+  `timestamp` datetime NOT NULL,
+  `vm_stats_data` text NOT NULL,
+  PRIMARY KEY (`id`)
+  ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
+-- PR#5984 Update name for global configuration vm.stats.increment.metrics
+Update configuration set name='vm.stats.increment.metrics' where name='vm.stats.increment.metrics.in.memory';
+
+ALTER TABLE `cloud`.`domain_router` ADD COLUMN `software_version` varchar(100) COMMENT 'Software version';
+
 DROP VIEW IF EXISTS `cloud`.`domain_router_view`;
 CREATE VIEW `cloud`.`domain_router_view` AS
     select
@@ -629,7 +667,8 @@ CREATE VIEW `cloud`.`domain_router_view` AS
         domain_router.is_redundant_router is_redundant_router,
         domain_router.redundant_state redundant_state,
         domain_router.stop_pending stop_pending,
-        domain_router.role role
+        domain_router.role role,
+        domain_router.software_version software_version
     from
         `cloud`.`domain_router`
             inner join
@@ -660,38 +699,3 @@ CREATE VIEW `cloud`.`domain_router_view` AS
         `cloud`.`async_job` ON async_job.instance_id = vm_instance.id
             and async_job.instance_type = 'DomainRouter'
             and async_job.job_status = 0;
-INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) SELECT UUID(), 3, 'listConfigurations', 'ALLOW', (SELECT MAX(`sort_order`)+1 FROM `cloud`.`role_permissions`) ON DUPLICATE KEY UPDATE rule=rule;
-INSERT INTO `cloud`.`role_permissions` (`uuid`, `role_id`, `rule`, `permission`, `sort_order`) SELECT UUID(), 3, 'updateConfiguration', 'ALLOW', (SELECT MAX(`sort_order`)+1 FROM `cloud`.`role_permissions`) ON DUPLICATE KEY UPDATE rule=rule;
-
--- table for network permissions
-CREATE TABLE  `cloud`.`network_permissions` (
-  `id` bigint unsigned NOT NULL auto_increment,
-  `network_id` bigint unsigned NOT NULL,
-  `account_id` bigint unsigned NOT NULL,
-  PRIMARY KEY  (`id`),
-  INDEX `i_network_permission_network_id`(`network_id`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-
-INSERT INTO `cloud`.`user_vm_details`(`vm_id`, `name`, `value`)
-    SELECT `user_vm_details`.`vm_id`, 'SSH.KeyPairNames', `ssh_keypairs`.`keypair_name`
-        FROM `cloud`.`user_vm_details`
-        INNER JOIN `cloud`.`ssh_keypairs` ON ssh_keypairs.public_key = user_vm_details.value
-        INNER JOIN `cloud`.`vm_instance` ON vm_instance.id = user_vm_details.vm_id
-        WHERE ssh_keypairs.account_id = vm_instance.account_id;
-
-ALTER TABLE `cloud`.`kubernetes_cluster` ADD COLUMN `security_group_id` bigint unsigned DEFAULT NULL,
-ADD CONSTRAINT `fk_kubernetes_cluster__security_group_id` FOREIGN KEY `fk_kubernetes_cluster__security_group_id`(`security_group_id`) REFERENCES `security_group`(`id`) ON DELETE CASCADE;
-
--- PR#5984 Create table to persist VM stats.
-DROP TABLE IF EXISTS `cloud`.`vm_stats`;
-CREATE TABLE `cloud`.`vm_stats` (
-  `id` bigint unsigned NOT NULL auto_increment COMMENT 'id',
-  `vm_id` bigint unsigned NOT NULL,
-  `mgmt_server_id` bigint unsigned NOT NULL,
-  `timestamp` datetime NOT NULL,
-  `vm_stats_data` text NOT NULL,
-  PRIMARY KEY (`id`)
-  ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-
--- PR#5984 Update name for global configuration vm.stats.increment.metrics
-Update configuration set name='vm.stats.increment.metrics' where name='vm.stats.increment.metrics.in.memory';

packaging/centos7/cloud.spec

@@ -236,7 +236,7 @@ mkdir -p ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms
 mkdir -p ${RPM_BUILD_ROOT}%{python_sitearch}/
 mkdir -p ${RPM_BUILD_ROOT}/usr/bin
 cp -r scripts/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/scripts
-install -D systemvm/dist/systemvm.iso ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms/systemvm.iso
+install -D systemvm/dist/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms/
 install python/lib/cloud_utils.py ${RPM_BUILD_ROOT}%{python_sitearch}/cloud_utils.py
 cp -r python/lib/cloudutils ${RPM_BUILD_ROOT}%{python_sitearch}/
 python3 -m py_compile ${RPM_BUILD_ROOT}%{python_sitearch}/cloud_utils.py
@@ -627,7 +627,9 @@ pip3 install --upgrade urllib3
 %dir %attr(0755,root,root) %{_datadir}/%{name}-common/vms
 %attr(0755,root,root) %{_datadir}/%{name}-common/scripts
 %attr(0755,root,root) /usr/bin/cloudstack-sccs
-%attr(0644, root, root) %{_datadir}/%{name}-common/vms/systemvm.iso
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/agent.zip
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/cloud-scripts.tgz
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/patch-sysvms.sh
 %attr(0644,root,root) %{python_sitearch}/cloud_utils.py
 %attr(0644,root,root) %{python_sitearch}/__pycache__/*
 %attr(0644,root,root) %{python_sitearch}/cloudutils/*

packaging/centos8/cloud.spec

@@ -229,7 +229,7 @@ mkdir -p ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms
 mkdir -p ${RPM_BUILD_ROOT}%{python_sitearch}/
 mkdir -p ${RPM_BUILD_ROOT}/usr/bin
 cp -r scripts/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/scripts
-install -D systemvm/dist/systemvm.iso ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms/systemvm.iso
+install -D systemvm/dist/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms/
 install python/lib/cloud_utils.py ${RPM_BUILD_ROOT}%{python_sitearch}/cloud_utils.py
 cp -r python/lib/cloudutils ${RPM_BUILD_ROOT}%{python_sitearch}/
 python3 -m py_compile ${RPM_BUILD_ROOT}%{python_sitearch}/cloud_utils.py
@@ -615,7 +615,9 @@ pip install --upgrade /usr/share/cloudstack-marvin/Marvin-*.tar.gz
 %dir %attr(0755,root,root) %{_datadir}/%{name}-common/vms
 %attr(0755,root,root) %{_datadir}/%{name}-common/scripts
 %attr(0755,root,root) /usr/bin/cloudstack-sccs
-%attr(0644, root, root) %{_datadir}/%{name}-common/vms/systemvm.iso
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/agent.zip
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/cloud-scripts.tgz
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/patch-sysvms.sh
 %attr(0644,root,root) %{python_sitearch}/cloud_utils.py
 %attr(0644,root,root) %{python_sitearch}/__pycache__/*
 %attr(0644,root,root) %{python_sitearch}/cloudutils/*

packaging/suse15/cloud.spec

@@ -231,7 +231,7 @@ mkdir -p ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms
 mkdir -p ${RPM_BUILD_ROOT}%{python_sitearch}/
 mkdir -p ${RPM_BUILD_ROOT}/usr/bin
 cp -r scripts/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/scripts
-install -D systemvm/dist/systemvm.iso ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms/systemvm.iso
+install -D systemvm/dist/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-common/vms/
 install python/lib/cloud_utils.py ${RPM_BUILD_ROOT}%{python_sitearch}/cloud_utils.py
 cp -r python/lib/cloudutils ${RPM_BUILD_ROOT}%{python_sitearch}/
 python3 -m py_compile ${RPM_BUILD_ROOT}%{python_sitearch}/cloud_utils.py
@@ -609,7 +609,9 @@ pip install --upgrade /usr/share/cloudstack-marvin/Marvin-*.tar.gz
 %dir %attr(0755,root,root) %{_datadir}/%{name}-common/vms
 %attr(0755,root,root) %{_datadir}/%{name}-common/scripts
 %attr(0755,root,root) /usr/bin/cloudstack-sccs
-%attr(0644, root, root) %{_datadir}/%{name}-common/vms/systemvm.iso
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/agent.zip
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/cloud-scripts.tgz
+%attr(0644, root, root) %{_datadir}/%{name}-common/vms/patch-sysvms.sh
 %attr(0644,root,root) %{python_sitearch}/cloud_utils.py
 %attr(0644,root,root) %{python_sitearch}/__pycache__/*
 %attr(0644,root,root) %{python_sitearch}/cloudutils/*

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java

@@ -283,6 +283,7 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
     private static final String AARCH64 = "aarch64";
 
     public static final String RESIZE_NOTIFY_ONLY = "NOTIFYONLY";
+    public static final String BASEPATH = "/usr/share/cloudstack-common/vms/";
 
     private String _modifyVlanPath;
     private String _versionstringpath;
@@ -309,8 +310,6 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
     private static final int NUMMEMSTATS =2;
 
     private KVMHAMonitor _monitor;
-    public static final String SSHKEYSPATH = "/root/.ssh";
-    public static final String SSHPRVKEYPATH = SSHKEYSPATH + File.separator + "id_rsa.cloud";
     public static final String SSHPUBKEYPATH = SSHKEYSPATH + File.separator + "id_rsa.pub.cloud";
     public static final String DEFAULTDOMRSSHPORT = "3922";
 
@@ -412,7 +411,7 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
         s_powerStatesTable.put(DomainState.VIR_DOMAIN_SHUTDOWN, PowerState.PowerOff);
     }
 
-    private VirtualRoutingResource _virtRouterResource;
+    public VirtualRoutingResource _virtRouterResource;
 
     private String _pingTestPath;
 
@@ -472,7 +471,7 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
         try {
             SshHelper.scpTo(routerIp, 3922, "root", permKey, null, path, content.getBytes(), filename, null);
         } catch (final Exception e) {
-            s_logger.warn("Fail to create file " + path + filename + " in VR " + routerIp, e);
+            s_logger.warn("Failed to create file " + path + filename + " in VR " + routerIp, e);
             details = e.getMessage();
             success = false;
         }
@@ -1171,20 +1170,6 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
 
         _storagePoolMgr = new KVMStoragePoolManager(_storage, _monitor);
 
-        _sysvmISOPath = (String)params.get("systemvm.iso.path");
-        if (_sysvmISOPath == null) {
-            final String[] isoPaths = {"/usr/share/cloudstack-common/vms/systemvm.iso"};
-            for (final String isoPath : isoPaths) {
-                if (_storage.exists(isoPath)) {
-                    _sysvmISOPath = isoPath;
-                    break;
-                }
-            }
-            if (_sysvmISOPath == null) {
-                s_logger.debug("Can't find system vm ISO");
-            }
-        }
-
         final Map<String, String> bridges = new HashMap<String, String>();
 
         params.put("libvirt.host.bridges", bridges);
@@ -2942,7 +2927,6 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
         }
 
         if (vmSpec.getType() != VirtualMachine.Type.User) {
-            if (_sysvmISOPath != null) {
             final DiskDef iso = new DiskDef();
             iso.defISODisk(_sysvmISOPath);
             if (_guestCpuArch != null && _guestCpuArch.equals("aarch64")) {
@@ -2950,7 +2934,6 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
             }
             vm.getDevices().addDevice(iso);
         }
-        }
 
         // For LXC, find and add the root filesystem, rbd data disks
         if (HypervisorType.LXC.toString().toLowerCase().equals(vm.getHvsType())) {

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtPatchSystemVmCommandWrapper.java

@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.hypervisor.kvm.resource.wrapper;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.PatchSystemVmAnswer;
+import com.cloud.agent.api.PatchSystemVmCommand;
+import com.cloud.agent.api.routing.NetworkElementCommand;
+import com.cloud.agent.resource.virtualnetwork.VRScripts;
+import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
+import com.cloud.resource.CommandWrapper;
+import com.cloud.resource.ResourceWrapper;
+import com.cloud.utils.ExecutionResult;
+import com.cloud.utils.FileUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.cloud.utils.validation.ChecksumUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.log4j.Logger;
+
+import java.io.File;
+
+@ResourceWrapper(handles = PatchSystemVmCommand.class)
+public class LibvirtPatchSystemVmCommandWrapper extends CommandWrapper<PatchSystemVmCommand, Answer, LibvirtComputingResource> {
+    private static final Logger s_logger = Logger.getLogger(LibvirtPatchSystemVmCommandWrapper.class);
+    private static int sshPort = Integer.parseInt(LibvirtComputingResource.DEFAULTDOMRSSHPORT);
+    private static File pemFile = new File(LibvirtComputingResource.SSHPRVKEYPATH);
+
+    @Override
+    public Answer execute(PatchSystemVmCommand cmd, LibvirtComputingResource serverResource) {
+        final String controlIp = cmd.getAccessDetail(NetworkElementCommand.ROUTER_IP);
+        final String sysVMName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+        ExecutionResult result;
+        try {
+            result = getSystemVmVersionAndChecksum(serverResource, controlIp);
+        } catch (CloudRuntimeException e) {
+            return new PatchSystemVmAnswer(cmd, e.getMessage());
+        }
+
+        final String[] lines = result.getDetails().split("&");
+        // TODO: do we fail, or patch anyway??
+        if (lines.length != 2) {
+            return new PatchSystemVmAnswer(cmd, result.getDetails());
+        }
+
+        String scriptChecksum = lines[1].trim();
+        String checksum = ChecksumUtil.calculateCurrentChecksum(sysVMName, "vms/cloud-scripts.tgz").trim();
+
+        if (!StringUtils.isEmpty(checksum) && checksum.equals(scriptChecksum) && !cmd.isForced()) {
+            String msg = String.format("No change in the scripts checksum, not patching systemVM %s", sysVMName);
+            s_logger.info(msg);
+            return new PatchSystemVmAnswer(cmd, msg, lines[0], lines[1]);
+        }
+
+        Pair<Boolean, String> patchResult = null;
+        try {
+            FileUtil.scpPatchFiles(controlIp, "/tmp/", sshPort, pemFile, serverResource.systemVmPatchFiles, LibvirtComputingResource.BASEPATH);
+            patchResult = SshHelper.sshExecute(controlIp, sshPort, "root",
+                    pemFile, null, "/tmp/patch-sysvms.sh", 10000, 10000, 600000);
+        } catch (Exception e) {
+            return new PatchSystemVmAnswer(cmd, e.getMessage());
+        }
+
+        if (patchResult.first()) {
+            String scriptVersion = lines[1];
+            if (StringUtils.isNotEmpty(patchResult.second())) {
+                String res = patchResult.second().replace("\n", " ");
+                String[] output = res.split(":");
+                if (output.length != 2) {
+                    s_logger.warn("Failed to get the latest script version");
+                } else {
+                    scriptVersion = output[1].split(" ")[0];
+                }
+            }
+            return new PatchSystemVmAnswer(cmd, String.format("Successfully patched systemVM %s ", sysVMName), lines[0], scriptVersion);
+        }
+        return new PatchSystemVmAnswer(cmd, patchResult.second());
+    }
+
+    private ExecutionResult getSystemVmVersionAndChecksum(LibvirtComputingResource serverResource, String controlIp) {
+        ExecutionResult result;
+        try {
+            result = serverResource.executeInVR(controlIp, VRScripts.VERSION, null);
+            if (!result.isSuccess()) {
+                String errMsg = String.format("GetSystemVMVersionCmd on %s failed, message %s", controlIp, result.getDetails());
+                s_logger.error(errMsg);
+                throw new CloudRuntimeException(errMsg);
+            }
+        } catch (final Exception e) {
+            final String msg = "GetSystemVMVersionCmd failed due to " + e;
+            s_logger.error(msg, e);
+            throw new CloudRuntimeException(msg, e);
+        }
+        return result;
+    }
+}
+

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java

@@ -19,8 +19,10 @@
 
 package com.cloud.hypervisor.kvm.resource.wrapper;
 
+import java.io.File;
 import java.net.URISyntaxException;
 
+import com.cloud.utils.FileUtil;
 import org.apache.log4j.Logger;
 import org.libvirt.Connect;
 import org.libvirt.DomainInfo.DomainState;
@@ -34,8 +36,8 @@ import com.cloud.agent.api.to.VirtualMachineTO;
 import com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource;
 import com.cloud.exception.InternalErrorException;
 import com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
-import com.cloud.hypervisor.kvm.resource.LibvirtVMDef;
 import com.cloud.hypervisor.kvm.resource.LibvirtKvmAgentHook;
+import com.cloud.hypervisor.kvm.resource.LibvirtVMDef;
 import com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager;
 import com.cloud.network.Networks.TrafficType;
 import com.cloud.resource.CommandWrapper;
@@ -115,6 +117,20 @@ public final class LibvirtStartCommandWrapper extends CommandWrapper<StartComman
                             break;
                         }
                     }
+
+                    try {
+                        File pemFile = new File(LibvirtComputingResource.SSHPRVKEYPATH);
+                        FileUtil.scpPatchFiles(controlIp, "/tmp/", Integer.parseInt(LibvirtComputingResource.DEFAULTDOMRSSHPORT), pemFile, LibvirtComputingResource.systemVmPatchFiles, LibvirtComputingResource.BASEPATH);
+                        if (!virtRouterResource.isSystemVMSetup(vmName, controlIp)) {
+                            String errMsg = "Failed to patch systemVM";
+                            s_logger.error(errMsg);
+                            return new StartAnswer(command, errMsg);
+                        }
+                    } catch (Exception e) {
+                        String errMsg = "Failed to scp files to system VM. Patching of systemVM failed";
+                        s_logger.error(errMsg, e);
+                        return new StartAnswer(command, String.format("%s due to: %s", errMsg, e.getMessage()));
+                    }
                 }
             }
 

plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResourceTest.java

@@ -54,6 +54,7 @@ import javax.xml.xpath.XPathConstants;
 import javax.xml.xpath.XPathExpressionException;
 import javax.xml.xpath.XPathFactory;
 
+import com.cloud.utils.ssh.SshHelper;
 import org.apache.cloudstack.storage.command.AttachAnswer;
 import org.apache.cloudstack.storage.command.AttachCommand;
 import org.apache.cloudstack.utils.linux.CPUStat;
@@ -212,7 +213,7 @@ import org.apache.cloudstack.utils.bytescale.ByteScaleUtils;
 import org.libvirt.VcpuInfo;
 
 @RunWith(PowerMockRunner.class)
-@PrepareForTest(value = {MemStat.class})
+@PrepareForTest(value = {MemStat.class, SshHelper.class})
 @PowerMockIgnore({"javax.xml.*", "org.w3c.dom.*", "org.apache.xerces.*"})
 public class LibvirtComputingResourceTest {
 
@@ -5280,7 +5281,9 @@ public class LibvirtComputingResourceTest {
     }
 
     @Test
-    public void testStartCommand() {
+    public void testStartCommand() throws Exception {
+        PowerMockito.mockStatic(SshHelper.class);
+        PowerMockito.doNothing().when(SshHelper.class, "scpTo", Mockito.anyString(), Mockito.anyInt(), Mockito.anyString(), Mockito.any(File.class), nullable(String.class), Mockito.anyString(), Mockito.any(String[].class), Mockito.anyString());
         final VirtualMachineTO vmSpec = Mockito.mock(VirtualMachineTO.class);
         final com.cloud.host.Host host = Mockito.mock(com.cloud.host.Host.class);
         final boolean executeInSequence = false;
@@ -5332,6 +5335,7 @@ public class LibvirtComputingResourceTest {
             when(nic.getType()).thenReturn(TrafficType.Control);
             when(libvirtComputingResource.getVirtRouterResource()).thenReturn(virtRouterResource);
             when(virtRouterResource.connect(controlIp, 1, 5000)).thenReturn(true);
+            when(virtRouterResource.isSystemVMSetup(vmName, controlIp)).thenReturn(true);
         } catch (final InternalErrorException e) {
             fail(e.getMessage());
         } catch (final LibvirtException e) {
@@ -5354,7 +5358,9 @@ public class LibvirtComputingResourceTest {
     }
 
     @Test
-    public void testStartCommandIsolationEc2() {
+    public void testStartCommandIsolationEc2() throws Exception {
+        PowerMockito.mockStatic(SshHelper.class);
+        PowerMockito.doNothing().when(SshHelper.class, "scpTo", Mockito.anyString(), Mockito.anyInt(), Mockito.anyString(), Mockito.any(File.class), nullable(String.class), Mockito.anyString(), Mockito.any(String[].class), Mockito.anyString());
         final VirtualMachineTO vmSpec = Mockito.mock(VirtualMachineTO.class);
         final com.cloud.host.Host host = Mockito.mock(com.cloud.host.Host.class);
         final boolean executeInSequence = false;
@@ -5410,6 +5416,7 @@ public class LibvirtComputingResourceTest {
             when(nic.getType()).thenReturn(TrafficType.Control);
             when(libvirtComputingResource.getVirtRouterResource()).thenReturn(virtRouterResource);
             when(virtRouterResource.connect(controlIp, 1, 5000)).thenReturn(true);
+            when(virtRouterResource.isSystemVMSetup(vmName, controlIp)).thenReturn(true);
         } catch (final InternalErrorException e) {
             fail(e.getMessage());
         } catch (final LibvirtException e) {

plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java

@@ -47,6 +47,11 @@ import java.util.stream.Collectors;
 import javax.naming.ConfigurationException;
 import javax.xml.datatype.XMLGregorianCalendar;
 
+import com.cloud.agent.api.PatchSystemVmAnswer;
+import com.cloud.agent.api.PatchSystemVmCommand;
+import com.cloud.resource.ServerResourceBase;
+import com.cloud.utils.FileUtil;
+import com.cloud.utils.validation.ChecksumUtil;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.storage.command.CopyCommand;
 import org.apache.cloudstack.storage.command.StorageSubSystemCommand;
@@ -361,10 +366,11 @@ import com.vmware.vim25.VmConfigSpec;
 import com.vmware.vim25.VmwareDistributedVirtualSwitchPvlanSpec;
 import com.vmware.vim25.VmwareDistributedVirtualSwitchVlanIdSpec;
 
-public class VmwareResource implements StoragePoolResource, ServerResource, VmwareHostService, VirtualRouterDeployer {
+public class VmwareResource extends ServerResourceBase implements StoragePoolResource, ServerResource, VmwareHostService, VirtualRouterDeployer {
     private static final Logger s_logger = Logger.getLogger(VmwareResource.class);
     public static final String VMDK_EXTENSION = ".vmdk";
     private static final String EXECUTING_RESOURCE_COMMAND = "Executing resource command %s: [%s].";
+    public static final String BASEPATH = "/usr/share/cloudstack-common/vms/";
 
     private static final Random RANDOM = new Random(System.nanoTime());
 
@@ -466,7 +472,9 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
             mbean.addProp("Name", cmd.getClass().getSimpleName());
 
             Class<? extends Command> clz = cmd.getClass();
-            if (cmd instanceof NetworkElementCommand) {
+            if (clz == PatchSystemVmCommand.class) {
+                answer = execute((PatchSystemVmCommand) cmd);
+            } else if (cmd instanceof NetworkElementCommand) {
                 return _vrResource.executeRequest((NetworkElementCommand) cmd);
             } else if (clz == ReadyCommand.class) {
                 answer = execute((ReadyCommand) cmd);
@@ -631,6 +639,77 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
         return answer;
     }
 
+    private ExecutionResult getSystemVmVersionAndChecksum(String controlIp) {
+        ExecutionResult result;
+        try {
+            result = executeInVR(controlIp, VRScripts.VERSION, null);
+            if (!result.isSuccess()) {
+                String errMsg = String.format("GetSystemVMVersionCmd on %s failed, message %s", controlIp, result.getDetails());
+                s_logger.error(errMsg);
+                throw new CloudRuntimeException(errMsg);
+            }
+        } catch (final Exception e) {
+            final String msg = "GetSystemVMVersionCmd failed due to " + e;
+            s_logger.error(msg, e);
+            throw new CloudRuntimeException(msg, e);
+        }
+        return result;
+    }
+
+    private Answer execute(PatchSystemVmCommand cmd) {
+        String controlIp = cmd.getAccessDetail((NetworkElementCommand.ROUTER_IP));
+        String sysVMName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+        String homeDir = System.getProperty("user.home");
+        File pemFile = new File(homeDir + "/.ssh/id_rsa");
+        ExecutionResult result;
+        try {
+            result = getSystemVmVersionAndChecksum(controlIp);
+            FileUtil.scpPatchFiles(controlIp, "/tmp/", DefaultDomRSshPort, pemFile, systemVmPatchFiles, BASEPATH);
+        } catch (CloudRuntimeException e) {
+            return new PatchSystemVmAnswer(cmd, e.getMessage());
+        }
+
+        final String[] lines = result.getDetails().split("&");
+        // TODO: do we fail, or patch anyway??
+        if (lines.length != 2) {
+            return new PatchSystemVmAnswer(cmd, result.getDetails());
+        }
+
+        String scriptChecksum = lines[1].trim();
+        String checksum = ChecksumUtil.calculateCurrentChecksum(sysVMName, "vms/cloud-scripts.tgz").trim();
+
+        if (!org.apache.commons.lang3.StringUtils.isEmpty(checksum) && checksum.equals(scriptChecksum) && !cmd.isForced()) {
+            String msg = String.format("No change in the scripts checksum, not patching systemVM %s", sysVMName);
+            s_logger.info(msg);
+            return new PatchSystemVmAnswer(cmd, msg, lines[0], lines[1]);
+        }
+
+        Pair<Boolean, String> patchResult = null;
+        try {
+            patchResult = SshHelper.sshExecute(controlIp, DefaultDomRSshPort, "root",
+                    pemFile, null, "/tmp/patch-sysvms.sh", 10000, 10000, 600000);
+        } catch (Exception e) {
+            return new PatchSystemVmAnswer(cmd, e.getMessage());
+        }
+
+        String scriptVersion = lines[1];
+        if (StringUtils.isNotEmpty(patchResult.second())) {
+            String res = patchResult.second().replace("\n", " ");
+            String[] output = res.split(":");
+            if (output.length != 2) {
+                s_logger.warn("Failed to get the latest script version");
+            } else {
+                scriptVersion = output[1].split(" ")[0];
+            }
+
+        }
+        if (patchResult.first()) {
+            return new PatchSystemVmAnswer(cmd, String.format("Successfully patched systemVM %s ", sysVMName), lines[0], scriptVersion);
+        }
+        return new PatchSystemVmAnswer(cmd, patchResult.second());
+
+    }
+
     private Answer execute(SetupPersistentNetworkCommand cmd) {
         VmwareHypervisorHost host = getHyperHost(getServiceContext());
         String hostname = null;
@@ -2124,7 +2203,6 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
                     String msg = "secondary storage for dc " + _dcId + " is not ready yet?";
                     throw new Exception(msg);
                 }
-                mgr.prepareSecondaryStorageStore(secStoreUrl, secStoreId);
 
                 ManagedObjectReference morSecDs = prepareSecondaryDatastoreOnHost(secStoreUrl);
                 if (morSecDs == null) {
@@ -2135,7 +2213,7 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
 
                 deviceConfigSpecArray[i] = new VirtualDeviceConfigSpec();
                 Pair<VirtualDevice, Boolean> isoInfo = VmwareHelper.prepareIsoDevice(vmMo,
-                        String.format("[%s] systemvm/%s", secDsMo.getName(), mgr.getSystemVMIsoFileNameOnDatastore()), secDsMo.getMor(), true, true, ideUnitNumber++, i + 1);
+                        null, secDsMo.getMor(), true, true, ideUnitNumber++, i + 1);
                 deviceConfigSpecArray[i].setDevice(isoInfo.first());
                 if (isoInfo.second()) {
                     if (s_logger.isDebugEnabled())
@@ -2487,6 +2565,32 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
 
             startAnswer.setIqnToData(iqnToData);
 
+            if (vmSpec.getType() != VirtualMachine.Type.User) {
+                String controlIp = getControlIp(nics);
+                // check if the router is up?
+                for (int count = 0; count < 60; count++) {
+                    final boolean result = _vrResource.connect(controlIp, 1, 5000);
+                    if (result) {
+                        break;
+                    }
+                }
+
+                try {
+                    String homeDir = System.getProperty("user.home");
+                    File pemFile = new File(homeDir + "/.ssh/id_rsa");
+                    FileUtil.scpPatchFiles(controlIp, "/tmp/", DefaultDomRSshPort, pemFile, systemVmPatchFiles, BASEPATH);
+                    if (!_vrResource.isSystemVMSetup(vmInternalCSName, controlIp)) {
+                        String errMsg = "Failed to patch systemVM";
+                        s_logger.error(errMsg);
+                        return new StartAnswer(cmd, errMsg);
+                    }
+                } catch (Exception e) {
+                    String errMsg = "Failed to scp files to system VM. Patching of systemVM failed";
+                    s_logger.error(errMsg, e);
+                    return new StartAnswer(cmd, String.format("%s due to: %s", errMsg, e.getMessage()));
+                }
+            }
+
             // Since VM was successfully powered-on, if there was an existing VM in a different cluster that was unregistered, delete all the files associated with it.
             if (existingVmName != null && existingVmFileLayout != null) {
                 List<String> vmDatastoreNames = new ArrayList<String>();
@@ -3878,6 +3982,17 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
         }
     }
 
+    private String getControlIp(NicTO[] nicTOs) {
+        String controlIpAddress = null;
+        for (NicTO nic : nicTOs) {
+            if ((TrafficType.Management == nic.getType() || TrafficType.Control == nic.getType()) && nic.getIp() != null) {
+                controlIpAddress = nic.getIp();
+                break;
+            }
+        }
+        return controlIpAddress;
+    }
+
     private VirtualMachineMO takeVmFromOtherHyperHost(VmwareHypervisorHost hyperHost, String vmName) throws Exception {
 
         VirtualMachineMO vmMo = hyperHost.findVmOnPeerHyperHost(vmName);
@@ -4399,10 +4514,6 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
                     throw new Exception(msg);
                 }
 
-                if (vm.getType() != VirtualMachine.Type.User) {
-                    mgr.prepareSecondaryStorageStore(secStoreUrl, secStoreId);
-                }
-
                 ManagedObjectReference morSecDs = prepareSecondaryDatastoreOnHost(secStoreUrl);
                 if (morSecDs == null) {
                     String msg = "Failed to prepare secondary storage on host, secondary store url: " + secStoreUrl;
@@ -6685,6 +6796,11 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
         return _name;
     }
 
+    @Override
+    protected String getDefaultScriptsDir() {
+        return null;
+    }
+
     @Override
     public boolean start() {
         return true;
@@ -7355,7 +7471,6 @@ public class VmwareResource implements StoragePoolResource, ServerResource, Vmwa
                     String msg = "secondary storage for dc " + _dcId + " is not ready yet?";
                     throw new Exception(msg);
                 }
-                mgr.prepareSecondaryStorageStore(secStoreUrl, secStoreId);
                 ManagedObjectReference morSecDs = prepareSecondaryDatastoreOnSpecificHost(secStoreUrl, targetHyperHost);
                 if (morSecDs == null) {
                     throw new Exception(String.format("Failed to prepare secondary storage on host, secondary store url: %s", secStoreUrl));

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java

@@ -51,6 +51,7 @@ import javax.naming.ConfigurationException;
 import javax.xml.parsers.DocumentBuilderFactory;
 import javax.xml.parsers.ParserConfigurationException;
 
+import com.cloud.resource.ServerResourceBase;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.diagnostics.CopyToSecondaryStorageAnswer;
 import org.apache.cloudstack.diagnostics.CopyToSecondaryStorageCommand;
@@ -179,7 +180,7 @@ import com.xensource.xenapi.XenAPIObject;
  * before you do any changes in this code here.
  *
  */
-public abstract class CitrixResourceBase implements ServerResource, HypervisorResource, VirtualRouterDeployer {
+public abstract class CitrixResourceBase extends ServerResourceBase implements ServerResource, HypervisorResource, VirtualRouterDeployer {
     /**
      * used to describe what type of resource a storage device is of
      */
@@ -215,6 +216,7 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
     private final static String VM_NAME_ISO_SUFFIX = "-ISO";
 
     private final static String VM_FILE_ISO_SUFFIX = ".iso";
+    public final static int DEFAULTDOMRSSHPORT = 3922;
 
     private static final XenServerConnectionPool ConnPool = XenServerConnectionPool.getInstance();
     // static min values for guests on xenserver
@@ -225,6 +227,8 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
     protected static final HashMap<VmPowerState, PowerState> s_powerStatesTable;
 
     public static final String XS_TOOLS_ISO_AFTER_70 = "guest-tools.iso";
+    public static final String BASEPATH = "/opt/xensource/packages/resources/";
+
     protected static final String PLATFORM_CORES_PER_SOCKET_KEY = "cores-per-socket";
 
     static {
@@ -339,6 +343,11 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
         return new StorageSubsystemCommandHandlerBase(processor);
     }
 
+    @Override
+    protected String getDefaultScriptsDir() {
+        return null;
+    }
+
     public String callHostPlugin(final Connection conn, final String plugin, final String cmd, final String... params) {
         final Map<String, String> args = new HashMap<String, String>();
         String msg;
@@ -904,11 +913,15 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
         }
     }
 
-    public String connect(final Connection conn, final String vmname, final String ipAddress) {
+    public String connect(final Connection conn, final String vmname, final String ipAddress, int sleep) {
-        return connect(conn, vmname, ipAddress, 3922);
+        return connect(conn, vmname, ipAddress, DEFAULTDOMRSSHPORT, sleep);
+    }
+
+    public String connect(final Connection conn, final String vmName, final String ipAddress, final int port, int sleep) {
+        if (sleep == 0) {
+            sleep = _sleep;
         }
 
-    public String connect(final Connection conn, final String vmName, final String ipAddress, final int port) {
         for (int i = 0; i <= _retry; i++) {
             try {
                 final Set<VM> vms = VM.getByNameLabel(conn, vmName);
@@ -929,7 +942,7 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
                 return null;
             }
             try {
-                Thread.sleep(_sleep);
+                Thread.sleep(sleep);
             } catch (final InterruptedException e) {
             }
         }
@@ -974,8 +987,25 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
             s_logger.warn("scp VR config file into host " + _host.getIp() + " failed with exception " + e.getMessage().toString());
         }
 
-        final String rc = callHostPlugin(conn, "vmops", "createFileInDomr", "domrip", routerIp, "srcfilepath", hostPath + filename, "dstfilepath", path);
+        final String rc = callHostPlugin(conn, "vmops", "createFileInDomr", "domrip", routerIp, "srcfilepath", hostPath + filename, "dstfilepath", path, "cleanup", "true");
-        s_logger.debug("VR Config file " + filename + " got created in VR, ip " + routerIp + " with content \n" + content);
+        s_logger.debug("VR Config file " + filename + " got created in VR, IP: " + routerIp + " with content \n" + content);
+
+        return new ExecutionResult(rc.startsWith("succ#"), rc.substring(5));
+    }
+
+    public ExecutionResult copyPatchFilesToVR(final String routerIp, final String path) {
+        final Connection conn = getConnection();
+        final String hostPath = "/opt/xensource/packages/resources/";
+        String rc = "";
+        for (String file: systemVmPatchFiles) {
+            rc = callHostPlugin(conn, "vmops", "createFileInDomr", "domrip", routerIp, "srcfilepath", hostPath.concat(file), "dstfilepath", path, "cleanup", "false");
+            if (rc.startsWith("fail#")) {
+                s_logger.error(String.format("Failed to scp file %s required for patching the systemVM", file));
+                break;
+            }
+        }
+
+        s_logger.debug("VR Config files at " + hostPath + " got created in VR, IP: " + routerIp);
 
         return new ExecutionResult(rc.startsWith("succ#"), rc.substring(5));
     }
@@ -1092,9 +1122,6 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
                     _host.setSystemvmisouuid(vdi.getRecord(conn).uuid);
                 }
             }
-            if (_host.getSystemvmisouuid() == null) {
-                throw new CloudRuntimeException("can not find systemvmiso");
-            }
         }
 
         final VBD.Record cdromVBDR = new VBD.Record();
@@ -1104,10 +1131,8 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
         cdromVBDR.userdevice = "3";
         cdromVBDR.mode = Types.VbdMode.RO;
         cdromVBDR.type = Types.VbdType.CD;
-        final VBD cdromVBD = VBD.create(conn, cdromVBDR);
-        cdromVBD.insert(conn, VDI.getByUuid(conn, _host.getSystemvmisouuid()));
 
-        return cdromVBD;
+        return VBD.create(conn, cdromVBDR);
     }
 
     protected boolean createSecondaryStorageFolder(final Connection conn, final String remoteMountPath, final String newFolder, final String nfsVersion) {
@@ -1396,7 +1421,7 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
                 final DiskTO[] disks = vmSpec.getDisks();
                 for (final DiskTO disk : disks) {
                     if (disk.getType() == Volume.Type.ISO) {
-                        final TemplateObjectTO iso = (TemplateObjectTO)disk.getData();
+                        final TemplateObjectTO iso = (TemplateObjectTO) disk.getData();
                         final String osType = iso.getGuestOsType();
                         if (osType != null) {
                             final String isoGuestOsName = getGuestOsType(vmSpec.getPlatformEmulator());
@@ -4893,7 +4918,7 @@ public abstract class CitrixResourceBase implements ServerResource, HypervisorRe
                     throw new CloudRuntimeException("Unable to authenticate");
                 }
 
-                final String cmd = "mkdir -p /opt/cloud/bin /var/log/cloud";
+                final String cmd = "mkdir -p /opt/cloud/bin /var/log/cloud /opt/xensource/packages/resources/";
                 if (!SSHCmdHelper.sshExecuteCmd(sshConnection, cmd)) {
                     throw new CloudRuntimeException("Cannot create directory /opt/cloud/bin on XenServer hosts");
                 }

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixCheckSshCommandWrapper.java

@@ -46,7 +46,7 @@ public final class CitrixCheckSshCommandWrapper extends CommandWrapper<CheckSshC
         }
 
         try {
-            final String result = citrixResourceBase.connect(conn, command.getName(), privateIp, cmdPort);
+            final String result = citrixResourceBase.connect(conn, command.getName(), privateIp, cmdPort, 0);
             if (result != null) {
                 return new CheckSshAnswer(command, "Can not ping System vm " + vmName + "due to:" + result);
             }

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixNetworkElementCommandWrapper.java

@@ -25,10 +25,11 @@ import com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource;
 import com.cloud.hypervisor.xenserver.resource.CitrixResourceBase;
 import com.cloud.resource.CommandWrapper;
 import com.cloud.resource.ResourceWrapper;
+import org.apache.log4j.Logger;
 
 @ResourceWrapper(handles =  NetworkElementCommand.class)
 public final class CitrixNetworkElementCommandWrapper extends CommandWrapper<NetworkElementCommand, Answer, CitrixResourceBase> {
-
+    private static final Logger s_logger = Logger.getLogger(CitrixNetworkElementCommandWrapper.class);
     @Override
     public Answer execute(final NetworkElementCommand command, final CitrixResourceBase citrixResourceBase) {
         final VirtualRoutingResource routingResource = citrixResourceBase.getVirtualRoutingResource();

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixPatchSystemVmCommandWrapper.java

@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.hypervisor.xenserver.resource.wrapper.xenbase;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.PatchSystemVmAnswer;
+import com.cloud.agent.api.PatchSystemVmCommand;
+import com.cloud.agent.api.routing.NetworkElementCommand;
+import com.cloud.agent.resource.virtualnetwork.VRScripts;
+import com.cloud.hypervisor.xenserver.resource.CitrixResourceBase;
+import com.cloud.resource.CommandWrapper;
+import com.cloud.resource.ResourceWrapper;
+import com.cloud.utils.ExecutionResult;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.validation.ChecksumUtil;
+import com.xensource.xenapi.Connection;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.log4j.Logger;
+
+import java.io.File;
+
+@ResourceWrapper(handles = PatchSystemVmCommand.class)
+public class CitrixPatchSystemVmCommandWrapper extends CommandWrapper<PatchSystemVmCommand, Answer, CitrixResourceBase> {
+    private static final Logger s_logger = Logger.getLogger(CitrixPatchSystemVmCommandWrapper.class);
+    private static int sshPort = CitrixResourceBase.DEFAULTDOMRSSHPORT;
+    private static File pemFile = new File(CitrixResourceBase.SSHPRVKEYPATH);
+
+    @Override
+    public Answer execute(PatchSystemVmCommand command, CitrixResourceBase serverResource) {
+        final String controlIp = command.getAccessDetail(NetworkElementCommand.ROUTER_IP);
+        final String sysVMName = command.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+        final Connection conn = serverResource.getConnection();
+
+        ExecutionResult result;
+        try {
+            result = getSystemVmVersionAndChecksum(serverResource, controlIp);
+        } catch (CloudRuntimeException e) {
+            return new PatchSystemVmAnswer(command, e.getMessage());
+        }
+
+        final String[] lines = result.getDetails().split("&");
+        // TODO: do we fail, or patch anyway??
+        if (lines.length != 2) {
+            return new PatchSystemVmAnswer(command, result.getDetails());
+        }
+
+        String scriptChecksum = lines[1].trim();
+        String checksum = ChecksumUtil.calculateCurrentChecksum(sysVMName, "vms/cloud-scripts.tgz").trim();
+        if (!StringUtils.isEmpty(checksum) && checksum.equals(scriptChecksum) && !command.isForced()) {
+            String msg = String.format("No change in the scripts checksum, not patching systemVM %s", sysVMName);
+            s_logger.info(msg);
+            return new PatchSystemVmAnswer(command, msg, lines[0], lines[1]);
+        }
+
+        String patchResult = null;
+        try {
+            serverResource.copyPatchFilesToVR(controlIp, "/tmp/");
+            patchResult = serverResource.callHostPlugin(conn, "vmops", "runPatchScriptInDomr", "domrip", controlIp);
+        } catch (Exception e) {
+            return new PatchSystemVmAnswer(command, e.getMessage());
+        }
+
+        if (patchResult.startsWith("succ#")) {
+            String scriptVersion = lines[1];
+            String res = patchResult.replace("\n", " ");
+            String[] output = res.split(":");
+            if (output.length != 2) {
+                s_logger.warn("Failed to get the latest script version");
+            } else {
+                scriptVersion = output[1].split(" ")[0];
+            }
+
+            return new PatchSystemVmAnswer(command, String.format("Successfully patched systemVM %s ", sysVMName), lines[0], scriptVersion);
+        }
+        return new PatchSystemVmAnswer(command, patchResult.substring(5));
+
+    }
+
+    private ExecutionResult getSystemVmVersionAndChecksum(CitrixResourceBase serverResource, String controlIp) {
+        ExecutionResult result;
+        try {
+            result = serverResource.executeInVR(controlIp, VRScripts.VERSION, null);
+            if (!result.isSuccess()) {
+                String errMsg = String.format("GetSystemVMVersionCmd on %s failed, message %s", controlIp, result.getDetails());
+                s_logger.error(errMsg);
+                throw new CloudRuntimeException(errMsg);
+            }
+        } catch (final Exception e) {
+            final String msg = "GetSystemVMVersionCmd failed due to " + e;
+            s_logger.error(msg, e);
+            throw new CloudRuntimeException(msg, e);
+        }
+        return result;
+    }
+
+
+}

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixRebootRouterCommandWrapper.java

@@ -40,7 +40,7 @@ public final class CitrixRebootRouterCommandWrapper extends CommandWrapper<Reboo
         final Answer answer = wrapper.execute(rebootCommand, citrixResourceBase);
 
         if (answer.getResult()) {
-            final String cnct = citrixResourceBase.connect(conn, command.getVmName(), command.getPrivateIpAddress());
+            final String cnct = citrixResourceBase.connect(conn, command.getVmName(), command.getPrivateIpAddress(), 0);
             citrixResourceBase.networkUsage(conn, command.getPrivateIpAddress(), "create", null);
 
             if (cnct == null) {

plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/wrapper/xenbase/CitrixStartCommandWrapper.java

@@ -25,6 +25,8 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 
+import com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.log4j.Logger;
 
 import com.cloud.agent.api.Answer;
@@ -180,6 +182,35 @@ public final class CitrixStartCommandWrapper extends CommandWrapper<StartCommand
 
             state = VmPowerState.RUNNING;
 
+            if (vmSpec.getType() != VirtualMachine.Type.User) {
+                String controlIp = null;
+                for (final NicTO nic : vmSpec.getNics()) {
+                    if (nic.getType() == Networks.TrafficType.Control) {
+                        controlIp = nic.getIp();
+                        break;
+                    }
+                }
+
+                String result2 = citrixResourceBase.connect(conn, vmName, controlIp, 1000);
+                if (StringUtils.isEmpty(result2)) {
+                    s_logger.info(String.format("Connected to SystemVM: %s", vmName));
+                }
+
+                try {
+                    citrixResourceBase.copyPatchFilesToVR(controlIp, "/tmp/");
+                    VirtualRoutingResource vrResource = citrixResourceBase.getVirtualRoutingResource();
+                    if (!vrResource.isSystemVMSetup(vmName, controlIp)) {
+                        String errMsg = "Failed to patch systemVM";
+                        s_logger.error(errMsg);
+                        return new StartAnswer(command, errMsg);
+                    }
+                } catch (Exception e) {
+                    String errMsg = "Failed to scp files to system VM. Patching of systemVM failed";
+                    s_logger.error(errMsg, e);
+                    return new StartAnswer(command, String.format("%s due to: %s", errMsg, e.getMessage()));
+                }
+            }
+
             final StartAnswer startAnswer = new StartAnswer(command);
 
             startAnswer.setIqnToData(iqnToData);

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java

@@ -444,7 +444,8 @@ public class KubernetesClusterManagerImpl extends ManagerBase implements Kuberne
         if (!((dockerRegistryUserName != null && !dockerRegistryUserName.isEmpty()) &&
                 (dockerRegistryPassword != null && !dockerRegistryPassword.isEmpty()) &&
                 (dockerRegistryUrl != null && !dockerRegistryUrl.isEmpty()))) {
-            throw new InvalidParameterValueException("All the docker private registry parameters (username, password, url, email) required are specified");
+
+            throw new InvalidParameterValueException("All the docker private registry parameters (username, password, url) required are specified");
         }
 
         try {
@@ -787,7 +788,7 @@ public class KubernetesClusterManagerImpl extends ManagerBase implements Kuberne
                 addKubernetesClusterDetailIfIsNotEmpty(details, kubernetesClusterId, ApiConstants.DOCKER_REGISTRY_PASSWORD, dockerRegistryPassword, false);
                 addKubernetesClusterDetailIfIsNotEmpty(details, kubernetesClusterId, ApiConstants.DOCKER_REGISTRY_URL, dockerRegistryUrl, true);
 
-                details.add(new KubernetesClusterDetailsVO(kubernetesClusterId, ApiConstants.USERNAME, "admin", true));
+                details.add(new KubernetesClusterDetailsVO(kubernetesCluster.getId(), ApiConstants.USERNAME, "admin", true));
                 SecureRandom random = new SecureRandom();
                 String randomPassword = new BigInteger(130, random).toString(32);
                 details.add(new KubernetesClusterDetailsVO(kubernetesClusterId, ApiConstants.PASSWORD, randomPassword, false));

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java

@@ -87,6 +87,7 @@ import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
 
+
 public class KubernetesClusterActionWorker {
 
     public static final String CLUSTER_NODE_VM_USER = "cloud";

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java

@@ -76,6 +76,7 @@ import com.cloud.vm.UserVmManager;
 import com.cloud.vm.VirtualMachine;
 import com.cloud.vm.VmDetailConstants;
 import com.cloud.vm.dao.VMInstanceDao;
+
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.cloudstack.api.BaseCmd;
 import org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd;
@@ -187,6 +188,7 @@ public class KubernetesClusterResourceModifierActionWorker extends KubernetesClu
                 registryUrl = detail.getValue();
             }
         }
+
         if (StringUtils.isNoneEmpty(registryUsername, registryPassword, registryUrl)) {
             // Update runcmd in the cloud-init configuration to run a script that updates the containerd config with provided registry details
             String runCmd = "- bash -x /opt/bin/setup-containerd";

plugins/integrations/kubernetes-service/src/main/resources/conf/k8s-control-node-add.yml

@@ -262,7 +262,7 @@ write_files:
 runcmd:
   - chown -R cloud:cloud /home/cloud/.ssh
   - containerd config default > /etc/containerd/config.toml
-  - sed -i '/\[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options\]/a '"\\            SystemdCgroup=true"'' /etc/containerd/config.toml
+  - sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
   - systemctl daemon-reload
   - systemctl restart containerd
   - until [ -f /etc/systemd/system/deploy-kube-system.service ]; do sleep 5; done

plugins/integrations/kubernetes-service/src/main/resources/conf/k8s-control-node.yml

@@ -289,7 +289,6 @@ write_files:
       echo "Restarting containerd service"
       systemctl restart containerd
 
-
   - path: /etc/systemd/system/setup-kube-system.service
     permissions: '0755'
     owner: root:root
@@ -320,7 +319,7 @@ write_files:
 runcmd:
   - chown -R cloud:cloud /home/cloud/.ssh
   - containerd config default > /etc/containerd/config.toml
-  - sed -i '/\[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options\]/a '"\\            SystemdCgroup=true"'' /etc/containerd/config.toml
+  - sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
   - systemctl daemon-reload
   - systemctl restart containerd
   - until [ -f /etc/systemd/system/deploy-kube-system.service ]; do sleep 5; done

plugins/integrations/kubernetes-service/src/main/resources/conf/k8s-node.yml

@@ -262,7 +262,7 @@ write_files:
 runcmd:
   - chown -R cloud:cloud /home/cloud/.ssh
   - containerd config default > /etc/containerd/config.toml
-  - sed -i '/\[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options\]/a '"\\            SystemdCgroup=true"'' /etc/containerd/config.toml
+  - sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
   - systemctl daemon-reload
   - systemctl restart containerd
   - until [ -f /etc/systemd/system/deploy-kube-system.service ]; do sleep 5; done

pom.xml

@@ -49,7 +49,7 @@
         <!-- keep in alphabetic order -->
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
-        <project.systemvm.template.version>4.16.1.0</project.systemvm.template.version>
+        <project.systemvm.template.version>4.17.0.0</project.systemvm.template.version>
         <sonar.organization>apachecloudstack</sonar.organization>
         <sonar.host.url>https://sonarcloud.io</sonar.host.url>
 

scripts/installer/export-templates.sh

@@ -0,0 +1,192 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+METADATA_FILE="metadata.ini"
+IMAGE_PATH=${3:-"/usr/share/cloudstack-management/templates/systemvm/"}
+TEMPLATE_VERSION=$(awk -F "=" '/version/ {print $2}' ${IMAGE_PATH}${METADATA_FILE} | xargs)
+TEMPLATE_PATH="/usr/share/cloudstack-management/templates/systemvm/"
+VERSION="${TEMPLATE_VERSION%.*}"
+PREFIX=${4:-"systemvmtemplate-$VERSION"}
+CLEANUP=${2:-1}
+TEMP_IMAGE_PATH="/tmp/sysvm_convert/"
+
+initial_setup() {
+  mkdir -p $TEMP_IMAGE_PATH
+  cp -r $IMAGE_PATH/* $TEMP_IMAGE_PATH
+  cd $TEMP_IMAGE_PATH
+  if [ ! -f ${TEMP_IMAGE_PATH}${PREFIX}-kvm.qcow2 ]; then
+    bzip2 -dc $PREFIX-kvm.qcow2.bz2 > $PREFIX-kvm.qcow2
+  fi
+}
+
+export_vmware() {
+  initial_setup
+  # Export for KVM
+  virt-sparsify $PREFIX-kvm.qcow2 --compress -o compat=0.10 $PREFIX-kvm-temp.qcow2
+  # Export for VMware
+  qemu-img convert -f qcow2 -O vmdk -o adapter_type=lsilogic,subformat=streamOptimized,compat6 $PREFIX-kvm-temp.qcow2 $PREFIX-vmware.vmdk
+  size=$(stat --printf="%s" $PREFIX-vmware.vmdk)
+
+cat <<EOF > $PREFIX-vmware.ovf
+<?xml version="1.0" encoding="UTF-8"?>
+<!--Generated by VMware ovftool 4.1.0 (build-2459827), UTC time: 2021-11-24T18:59:40.381083Z-->
+<Envelope vmw:buildId="build-2459827" xmlns="http://schemas.dmtf.org/ovf/envelope/1" xmlns:cim="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  <References>
+    <File ovf:href="$PREFIX-vmware.vmdk" ovf:id="file1" ovf:size="$size"/>
+  </References>
+  <DiskSection>
+    <Info>Virtual disk information</Info>
+    <Disk ovf:capacity="5" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>
+  </DiskSection>
+  <VirtualSystem ovf:id="vm">
+    <Info>A virtual machine</Info>
+    <Name>$PREFIX-vmware</Name>
+    <OperatingSystemSection ovf:id="101" vmw:osType="otherLinux64Guest">
+      <Info>The kind of installed guest operating system</Info>
+    </OperatingSystemSection>
+    <VirtualHardwareSection>
+      <Info>Virtual hardware requirements</Info>
+      <System>
+        <vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
+        <vssd:InstanceID>0</vssd:InstanceID>
+        <vssd:VirtualSystemIdentifier>$PREFIX-vmware</vssd:VirtualSystemIdentifier>
+        <vssd:VirtualSystemType>vmx-11</vssd:VirtualSystemType>
+      </System>
+      <Item>
+        <rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits>
+        <rasd:Description>Number of Virtual CPUs</rasd:Description>
+        <rasd:ElementName>1 virtual CPU(s)</rasd:ElementName>
+        <rasd:InstanceID>1</rasd:InstanceID>
+        <rasd:ResourceType>3</rasd:ResourceType>
+        <rasd:VirtualQuantity>1</rasd:VirtualQuantity>
+      </Item>
+      <Item>
+        <rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits>
+        <rasd:Description>Memory Size</rasd:Description>
+        <rasd:ElementName>256MB of memory</rasd:ElementName>
+        <rasd:InstanceID>2</rasd:InstanceID>
+        <rasd:ResourceType>4</rasd:ResourceType>
+        <rasd:VirtualQuantity>256</rasd:VirtualQuantity>
+      </Item>
+      <Item>
+        <rasd:Address>0</rasd:Address>
+        <rasd:Description>SCSI Controller</rasd:Description>
+        <rasd:ElementName>scsiController0</rasd:ElementName>
+        <rasd:InstanceID>3</rasd:InstanceID>
+        <rasd:ResourceSubType>lsilogic</rasd:ResourceSubType>
+        <rasd:ResourceType>6</rasd:ResourceType>
+      </Item>
+      <Item>
+        <rasd:Address>0</rasd:Address>
+        <rasd:Description>IDE Controller</rasd:Description>
+        <rasd:ElementName>ideController0</rasd:ElementName>
+        <rasd:InstanceID>4</rasd:InstanceID>
+        <rasd:ResourceType>5</rasd:ResourceType>
+      </Item>
+      <Item ovf:required="false">
+        <rasd:AddressOnParent>0</rasd:AddressOnParent>
+        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
+        <rasd:ElementName>cdrom0</rasd:ElementName>
+        <rasd:InstanceID>5</rasd:InstanceID>
+        <rasd:Parent>4</rasd:Parent>
+        <rasd:ResourceType>15</rasd:ResourceType>
+      </Item>
+      <Item>
+        <rasd:AddressOnParent>0</rasd:AddressOnParent>
+        <rasd:ElementName>disk0</rasd:ElementName>
+        <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
+        <rasd:InstanceID>6</rasd:InstanceID>
+        <rasd:Parent>3</rasd:Parent>
+        <rasd:ResourceType>17</rasd:ResourceType>
+      </Item>
+      <Item ovf:required="false">
+        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
+        <rasd:ElementName>video</rasd:ElementName>
+        <rasd:InstanceID>7</rasd:InstanceID>
+        <rasd:ResourceType>24</rasd:ResourceType>
+        <vmw:Config ovf:required="false" vmw:key="enable3DSupport" vmw:value="false"/>
+        <vmw:Config ovf:required="false" vmw:key="useAutoDetect" vmw:value="false"/>
+        <vmw:Config ovf:required="false" vmw:key="videoRamSizeInKB" vmw:value="4096"/>
+      </Item>
+      <Item ovf:required="false">
+        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
+        <rasd:ElementName>vmci</rasd:ElementName>
+        <rasd:InstanceID>8</rasd:InstanceID>
+        <rasd:ResourceSubType>vmware.vmci</rasd:ResourceSubType>
+        <rasd:ResourceType>1</rasd:ResourceType>
+      </Item>
+      <vmw:Config ovf:required="false" vmw:key="cpuHotAddEnabled" vmw:value="false"/>
+      <vmw:Config ovf:required="false" vmw:key="cpuHotRemoveEnabled" vmw:value="false"/>
+      <vmw:Config ovf:required="false" vmw:key="firmware" vmw:value="bios"/>
+      <vmw:Config ovf:required="false" vmw:key="memoryHotAddEnabled" vmw:value="false"/>
+    </VirtualHardwareSection>
+    <AnnotationSection ovf:required="false">
+      <Info>A human-readable annotation</Info>
+      <Annotation>$PREFIX-vmware</Annotation>
+    </AnnotationSection>
+  </VirtualSystem>
+</Envelope>
+EOF
+cat <<EOF > $PREFIX-vmware.mf
+SHA1($PREFIX-vmware.ovf)= $(sha1sum $PREFIX-vmware.ovf|awk '{print $1}')
+SHA1($PREFIX-vmware.vmdk)= $(sha1sum $PREFIX-vmware.vmdk |awk '{print $1}')
+EOF
+
+  tar -cvf $PREFIX-vmware.ova $PREFIX-vmware.ovf $PREFIX-vmware.mf $PREFIX-vmware.vmdk
+  checksum=$(md5sum $PREFIX-vmware.ova | awk '{print $1}')
+  sed -i '/^\['"vmware"']/,/^\[/{s/^checksum[[:space:]]*=.*/checksum = '"$checksum"'/}' ./$METADATA_FILE
+  rm -rf *.mf  *.ovf  *.vmdk
+  sudo cp $TEMP_IMAGE_PATH/$PREFIX-vmware.ova $TEMP_IMAGE_PATH/metadata.ini $IMAGE_PATH
+  cleanup
+}
+
+export_xen() {
+  # Export for XenServer/XCP-ng
+  initial_setup
+  qemu-img convert -f qcow2 -O vpc $PREFIX-kvm.qcow2 $PREFIX-xen.vhd
+  bzip2 $PREFIX-xen.vhd
+  checksum=$(md5sum $PREFIX-xen.vhd.bz2 | awk '{print $1}')
+  sed -i '/^\['"xenserver"']/,/^\[/{s/^checksum[[:space:]]*=.*/checksum = '"$checksum"'/}' $METADATA_FILE
+  rm -rf $PREFIX-xen.vhd
+  sudo cp $TEMP_IMAGE_PATH/$PREFIX-xen* $TEMP_IMAGE_PATH/metadata.ini $IMAGE_PATH
+  cleanup
+}
+
+cleanup() {
+  cd /var/cloudstack/management/
+  if [ $CLEANUP == 1 ]; then
+    cd /var/cloudstack/management/
+    rm -rf $TEMP_IMAGE_PATH
+  fi
+}
+
+if [ "$#" -lt 1 ] ; then
+   echo "Usage: $0 <hypervisor: vmware/xenserver> [cleanup: 0/1; default: 1] [imagepath: default:/usr/share/cloudstack-management/templates/systemvm/] [templateprefix: default:systemvmtemplate-$VERSION]" >&2
+   exit 1
+fi
+
+if [ $1 == "vmware" ]; then
+  echo "exporting vmware template"
+  export_vmware
+elif [ $1 == "xenserver" ]; then
+  echo "exporting xenserver template"
+  export_xen
+else
+  echo "Conversion of template to $1's compatible format not supported "
+fi
+

scripts/util/keystore-cert-import

@@ -17,19 +17,34 @@
 # under the License.
 
 PROPS_FILE="$1"
-KS_FILE="$2"
+KS_PASS="$2"
-MODE="$3"
+KS_FILE="$3"
-CERT_FILE="$4"
+MODE="$4"
-CERT=$(echo "$5" | tr '^' '\n' | tr '~' ' ')
+CERT_FILE="$5"
-CACERT_FILE="$6"
+CERT=$(echo "$6" | tr '^' '\n' | tr '~' ' ')
-CACERT=$(echo "$7" | tr '^' '\n' | tr '~' ' ')
+CACERT_FILE="$7"
-PRIVKEY_FILE="$8"
+CACERT=$(echo "$8" | tr '^' '\n' | tr '~' ' ')
-PRIVKEY=$(echo "$9" | tr '^' '\n' | tr '~' ' ')
+PRIVKEY_FILE="$9"
+PRIVKEY=$(echo "${10}" | tr '^' '\n' | tr '~' ' ')
 
 ALIAS="cloud"
 SYSTEM_FILE="/var/cache/cloud/cmdline"
 LIBVIRTD_FILE="/etc/libvirt/libvirtd.conf"
 
+if [ ! -f "$LIBVIRTD_FILE" ]; then
+  # Re-use existing password or use the one provided
+  while [ ! -d /usr/local/cloud/systemvm/conf ]; do sleep 1; done
+  if [ -f "$PROPS_FILE" ]; then
+      OLD_PASS=$(sed -n '/keystore.passphrase/p' "$PROPS_FILE" 2>/dev/null  | sed 's/keystore.passphrase=//g' 2>/dev/null)
+      if [ ! -z "${OLD_PASS// }" ]; then
+          KS_PASS="$OLD_PASS"
+      else
+          sed -i "/keystore.passphrase.*/d" $PROPS_FILE 2> /dev/null || true
+          echo "keystore.passphrase=$KS_PASS" >> $PROPS_FILE
+      fi
+  fi
+fi
+
 # Find keystore password
 KS_PASS=$(sed -n '/keystore.passphrase/p' "$PROPS_FILE" 2>/dev/null  | sed 's/keystore.passphrase=//g' 2>/dev/null)
 
@@ -41,11 +56,17 @@ fi
 # Import certificate
 if [ ! -z "${CERT// }" ]; then
     echo "$CERT" > "$CERT_FILE"
+elif [ ! -f "$CERT_FILE" ]; then
+   echo "Cannot find certificate file: $CERT_FILE, exiting"
+   exit
 fi
 
 # Import ca certs
 if [ ! -z "${CACERT// }" ]; then
     echo "$CACERT" > "$CACERT_FILE"
+elif [ ! -f "$CACERT_FILE" ]; then
+    echo "Cannot find ca certificate file: $CACERT_FILE, exiting!"
+    exit
 fi
 
 # Import cacerts into the keystore
@@ -64,6 +85,11 @@ fi
 # Import private key if available
 if [ ! -z "${PRIVKEY// }" ]; then
     echo "$PRIVKEY" > "$PRIVKEY_FILE"
+else
+    > "$PRIVKEY_FILE"
+fi
+
+if [ -f "$PRIVKEY_FILE" ] && [ -s "$PRIVKEY_FILE" ]; then
     # Re-initialize keystore when private key is provided
     keytool -delete -noprompt -alias "$ALIAS" -keystore "$KS_FILE" -storepass "$KS_PASS" 2>/dev/null || true
     openssl pkcs12 -export -name "$ALIAS" -in "$CERT_FILE" -inkey "$PRIVKEY_FILE" -out "$KS_FILE.p12" -password pass:"$KS_PASS" > /dev/null 2>&1

scripts/vm/hypervisor/xenserver/vmops

@@ -235,10 +235,12 @@ def createFileInDomr(session, args):
     src_filepath = args['srcfilepath']
     dst_path = args['dstfilepath']
     domrip = args['domrip']
+    cleanup = 'true' if 'cleanup' not in args else args['cleanup']
     txt=""
     try:
         target = "root@" + domrip + ":" + dst_path
         txt = util.pread2(['scp','-P','3922','-q','-o','StrictHostKeyChecking=no','-i','/root/.ssh/id_rsa.cloud',src_filepath, target])
+        if cleanup == 'true' or not cleanup:
             util.pread2(['rm',src_filepath])
         txt = 'succ#' + txt
     except:
@@ -246,6 +248,19 @@ def createFileInDomr(session, args):
         txt = 'fail#' + txt
     return txt
 
+@echo
+def runPatchScriptInDomr(session, args):
+    domrip = args['domrip']
+    txt=""
+    try:
+        target = "root@" + domrip
+        txt = util.pread2(['ssh','-p','3922','-i','/root/.ssh/id_rsa.cloud', target, "/bin/bash","/tmp/patch-sysvms.sh"])
+        txt = 'succ#' + txt
+    except:
+        logging.debug("failed to run patch script in systemVM with IP:  " + domrip)
+        txt = 'fail#' + txt
+    return txt
+
 @echo
 def deleteFile(session, args):
     file_path = args["filepath"]
@@ -1588,4 +1603,5 @@ if __name__ == "__main__":
                             "cleanup_rules":cleanup_rules,
                             "createFileInDomr":createFileInDomr,
                             "kill_copy_process":kill_copy_process,
-                            "secureCopyToHost":secureCopyToHost})
+                            "secureCopyToHost":secureCopyToHost,
+                            "runPatchScriptInDomr": runPatchScriptInDomr})

scripts/vm/hypervisor/xenserver/xcposs/patch

@@ -31,7 +31,9 @@ vmops=..,0755,/usr/lib/xcp/plugins
 ovsgre=..,0755,/usr/lib/xcp/plugins
 ovstunnel=..,0755,/usr/lib/xcp/plugins
 vmopsSnapshot=..,0755,/usr/lib/xcp/plugins
-systemvm.iso=../../../../../vms,0644,/usr/share/xcp/packages/iso/
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/hypervisor/xenserver/xcpserver/patch

@@ -31,7 +31,9 @@ NFSSR.py=/opt/xensource/sm
 vmops=..,0755,/etc/xapi.d/plugins
 ovstunnel=..,0755,/etc/xapi.d/plugins
 vmopsSnapshot=..,0755,/etc/xapi.d/plugins
-systemvm.iso=../../../../../vms,0644,/opt/xensource/packages/iso
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/hypervisor/xenserver/xenserver56/patch

@@ -30,7 +30,9 @@ NFSSR.py=/opt/xensource/sm
 vmops=..,0755,/etc/xapi.d/plugins
 vmopsSnapshot=..,0755,/etc/xapi.d/plugins
 cloudstack_pluginlib.py=..,0755,/etc/xapi.d/plugins
-systemvm.iso=../../../../../vms,0644,/opt/xensource/packages/iso
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/hypervisor/xenserver/xenserver56fp1/patch

@@ -30,7 +30,9 @@ NFSSR.py=/opt/xensource/sm
 vmops=..,0755,/etc/xapi.d/plugins
 vmopsSnapshot=..,0755,/etc/xapi.d/plugins
 cloudstack_pluginlib.py=..,0755,/etc/xapi.d/plugins
-systemvm.iso=../../../../../vms,0644,/opt/xensource/packages/iso
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/hypervisor/xenserver/xenserver60/patch

@@ -34,7 +34,9 @@ cloudstack_plugins.conf=..,0644,/etc/xensource
 cloudstack_pluginlib.py=..,0755,/etc/xapi.d/plugins
 ovstunnel=..,0755,/etc/xapi.d/plugins
 vmopsSnapshot=..,0755,/etc/xapi.d/plugins
-systemvm.iso=../../../../../vms,0644,/opt/xensource/packages/iso
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/hypervisor/xenserver/xenserver62/patch

@@ -35,7 +35,9 @@ cloudstack_plugins.conf=..,0644,/etc/xensource
 cloudstack_pluginlib.py=..,0755,/etc/xapi.d/plugins
 ovstunnel=..,0755,/etc/xapi.d/plugins
 cloud-plugin-storage=..,0755,/etc/xapi.d/plugins
-systemvm.iso=../../../../../vms,0644,/opt/xensource/packages/iso
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/hypervisor/xenserver/xenserver65/patch

@@ -35,7 +35,9 @@ cloudstack_plugins.conf=..,0644,/etc/xensource
 cloudstack_pluginlib.py=..,0755,/etc/xapi.d/plugins
 ovstunnel=..,0755,/etc/xapi.d/plugins
 cloud-plugin-storage=..,0755,/etc/xapi.d/plugins
-systemvm.iso=../../../../../vms,0644,/opt/xensource/packages/iso
+agent.zip=../../../../../vms,0644,/opt/xensource/packages/resources/
+cloud-scripts.tgz=../../../../../vms,0644,/opt/xensource/packages/resources/
+patch-sysvms.sh=../../../../../vms,0644,/opt/xensource/packages/resources/
 id_rsa.cloud=../../../systemvm,0600,/root/.ssh
 network_info.sh=..,0755,/opt/cloud/bin
 setupxenserver.sh=..,0755,/opt/cloud/bin

scripts/vm/systemvm/injectkeys.sh

@@ -24,14 +24,8 @@
 set -e
 
 TMP=/tmp
-MOUNTPATH=${HOME}/systemvm_mnt
-TMPDIR=${TMP}/cloud/systemvm
 umask 022
 
-clean_up() {
-  $SUDO umount $MOUNTPATH
-}
-
 copy_priv_key() {
   local newprivkey=$1
   diff -q $newprivkey $(dirname $0)/id_rsa.cloud && return 0
@@ -45,8 +39,6 @@ then
    SUDO="sudo -n "
 fi
 
-$SUDO mkdir -p $MOUNTPATH
-
 [ $# -ne 1 ] && echo "Usage: $(basename $0) <new private key file>" && exit 3
 newprivkey=$1
 [ ! -f $newprivkey ] && echo "$(basename $0): Could not open $newprivkey" && exit 3

server/src/main/java/com/cloud/api/ResponseObjectTypeAdapter.java

@@ -22,6 +22,7 @@ import java.lang.reflect.Type;
 import org.apache.cloudstack.api.ResponseObject;
 import org.apache.cloudstack.api.response.ExceptionResponse;
 import org.apache.cloudstack.api.response.SuccessResponse;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.log4j.Logger;
 
 import com.google.gson.JsonElement;
@@ -38,6 +39,9 @@ public class ResponseObjectTypeAdapter implements JsonSerializer<ResponseObject>
 
         if (responseObj instanceof SuccessResponse) {
             obj.addProperty("success", ((SuccessResponse)responseObj).getSuccess());
+            if (!StringUtils.isEmpty(((SuccessResponse) responseObj).getDisplayText())) {
+                obj.addProperty("details", ((SuccessResponse)responseObj).getDisplayText());
+            }
             return obj;
         } else if (responseObj instanceof ExceptionResponse) {
             obj.addProperty("errorcode", ((ExceptionResponse)responseObj).getErrorCode());

server/src/main/java/com/cloud/api/query/dao/DomainRouterJoinDaoImpl.java

@@ -24,6 +24,7 @@ import javax.inject.Inject;
 import org.apache.cloudstack.annotation.AnnotationService;
 import org.apache.cloudstack.annotation.dao.AnnotationDao;
 import org.apache.cloudstack.context.CallContext;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.log4j.Logger;
 import org.springframework.stereotype.Component;
 
@@ -90,13 +91,28 @@ public class DomainRouterJoinDaoImpl extends GenericDaoBase<DomainRouterJoinVO,
         routerResponse.setState(router.getState());
         routerResponse.setIsRedundantRouter(router.isRedundantRouter());
         routerResponse.setScriptsVersion(router.getScriptsVersion());
+        routerResponse.setSoftwareVersion(router.getSoftwareVersion());
         if (router.getRedundantState() != null) {
             routerResponse.setRedundantState(router.getRedundantState().toString());
         }
         if (router.getTemplateVersion() != null) {
             String routerVersion = CloudStackVersion.trimRouterVersion(router.getTemplateVersion());
             routerResponse.setVersion(routerVersion);
-            routerResponse.setRequiresUpgrade((CloudStackVersion.compare(routerVersion, NetworkOrchestrationService.MinVRVersion.valueIn(router.getDataCenterId())) < 0));
+            boolean isTempVersionLower = (CloudStackVersion.compare(routerVersion, NetworkOrchestrationService.MinVRVersion.valueIn(router.getDataCenterId())) < 0);
+            if (!isTempVersionLower) {
+                routerResponse.setRequiresUpgrade(false);
+            } else {
+                boolean requiresUpgrade = true;
+                String currentCodeVersion = this.getClass().getPackage().getImplementationVersion();
+                if (StringUtils.isNotEmpty(currentCodeVersion)) {
+                    currentCodeVersion = CloudStackVersion.parse(currentCodeVersion).toString();
+                    String routerSoftwareVersion = router.getSoftwareVersion();
+                    if (StringUtils.isNotEmpty(routerSoftwareVersion)) {
+                        requiresUpgrade = !(currentCodeVersion.equals(routerSoftwareVersion));
+                    }
+                }
+                routerResponse.setRequiresUpgrade(requiresUpgrade);
+            }
         } else {
             routerResponse.setVersion("UNKNOWN");
             routerResponse.setRequiresUpgrade(true);

server/src/main/java/com/cloud/api/query/vo/DomainRouterJoinVO.java

@@ -251,6 +251,9 @@ public class DomainRouterJoinVO extends BaseViewVO implements ControlledViewEnti
     @Enumerated(value = EnumType.STRING)
     private VirtualRouter.Role role;
 
+    @Column(name = "software_version")
+    private String softwareVersion;
+
     public DomainRouterJoinVO() {
     }
 
@@ -534,4 +537,8 @@ public class DomainRouterJoinVO extends BaseViewVO implements ControlledViewEnti
     public Class<?> getEntityType() {
         return VirtualMachine.class;
     }
+
+    public String getSoftwareVersion() {
+        return softwareVersion;
+    }
 }

server/src/main/java/com/cloud/consoleproxy/ConsoleProxyManagerImpl.java

@@ -29,9 +29,12 @@ import java.util.Map;
 import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 
+import com.cloud.utils.PasswordGenerator;
 import org.apache.cloudstack.agent.lb.IndirectAgentLB;
+import org.apache.cloudstack.ca.CAManager;
 import org.apache.cloudstack.context.CallContext;
 import org.apache.cloudstack.engine.orchestration.service.NetworkOrchestrationService;
+import org.apache.cloudstack.framework.ca.Certificate;
 import org.apache.cloudstack.framework.config.ConfigKey;
 import org.apache.cloudstack.framework.config.Configurable;
 import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
@@ -221,6 +224,10 @@ public class ConsoleProxyManagerImpl extends ManagerBase implements ConsoleProxy
     private VirtualMachineManager virtualMachineManager;
     @Inject
     private IndirectAgentLB indirectAgentLB;
+    @Inject
+    private CAManager caManager;
+    @Inject
+    private NetworkOrchestrationService networkMgr;
 
     private ConsoleProxyListener consoleProxyListener;
 
@@ -1204,7 +1211,11 @@ public class ConsoleProxyManagerImpl extends ManagerBase implements ConsoleProxy
 
     @Override
     public boolean finalizeVirtualMachineProfile(VirtualMachineProfile profile, DeployDestination dest, ReservationContext context) {
-
+        final Map<String, String> sshAccessDetails = networkMgr.getSystemVMAccessDetails(profile.getVirtualMachine());
+        final Map<String, String> ipAddressDetails = new HashMap<>(sshAccessDetails);
+        ipAddressDetails.remove("router.name");
+        final Certificate certificate = caManager.issueCertificate(null, Arrays.asList(profile.getHostName(), profile.getInstanceName()),
+                new ArrayList<>(ipAddressDetails.values()), CAManager.CertValidityPeriod.value(), null);
         ConsoleProxyVO vm = consoleProxyDao.findById(profile.getId());
         Map<String, String> details = userVmDetailsDao.listDetailsKeyPairs(vm.getId());
         vm.setDetails(details);
@@ -1276,7 +1287,7 @@ public class ConsoleProxyManagerImpl extends ManagerBase implements ConsoleProxy
         if (dc.getDns2() != null) {
             buf.append(" dns2=").append(dc.getDns2());
         }
-
+        buf.append(" keystore_password=").append(VirtualMachineGuru.getEncodedString(PasswordGenerator.generateRandomPassword(16)));
         String bootArgs = buf.toString();
         if (s_logger.isDebugEnabled()) {
             s_logger.debug("Boot Args for " + profile + ": " + bootArgs);

server/src/main/java/com/cloud/hypervisor/kvm/discoverer/LibvirtServerDiscoverer.java

@@ -16,26 +16,6 @@
 // under the License.
 package com.cloud.hypervisor.kvm.discoverer;
 
-import java.net.InetAddress;
-import java.net.URI;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-
-import javax.inject.Inject;
-import javax.naming.ConfigurationException;
-
-import org.apache.cloudstack.agent.lb.IndirectAgentLB;
-import org.apache.cloudstack.ca.CAManager;
-import org.apache.cloudstack.ca.SetupCertificateCommand;
-import org.apache.cloudstack.direct.download.DirectDownloadManager;
-import org.apache.cloudstack.framework.ca.Certificate;
-import org.apache.cloudstack.utils.security.KeyStoreUtils;
-import org.apache.log4j.Logger;
-
 import com.cloud.agent.AgentManager;
 import com.cloud.agent.Listener;
 import com.cloud.agent.api.AgentControlAnswer;
@@ -68,6 +48,24 @@ import com.cloud.utils.StringUtils;
 import com.cloud.utils.exception.CloudRuntimeException;
 import com.cloud.utils.ssh.SSHCmdHelper;
 import com.trilead.ssh2.Connection;
+import org.apache.cloudstack.agent.lb.IndirectAgentLB;
+import org.apache.cloudstack.ca.CAManager;
+import org.apache.cloudstack.ca.SetupCertificateCommand;
+import org.apache.cloudstack.direct.download.DirectDownloadManager;
+import org.apache.cloudstack.framework.ca.Certificate;
+import org.apache.cloudstack.utils.security.KeyStoreUtils;
+import org.apache.log4j.Logger;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+import java.net.InetAddress;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
 
 import static com.cloud.configuration.ConfigurationManagerImpl.ADD_HOST_ON_SERVICE_RESTART_KVM;
 
@@ -163,6 +161,7 @@ public abstract class LibvirtServerDiscoverer extends DiscovererBase implements
             validityPeriod = 1;
         }
 
+        String keystorePassword = PasswordGenerator.generateRandomPassword(16);
         final SSHCmdHelper.SSHCmdResult keystoreSetupResult = SSHCmdHelper.sshExecuteCmdWithResult(sshConnection,
                 String.format("sudo /usr/share/cloudstack-common/scripts/util/%s " +
                                 "/etc/cloudstack/agent/agent.properties " +
@@ -171,7 +170,7 @@ public abstract class LibvirtServerDiscoverer extends DiscovererBase implements
                                 "/etc/cloudstack/agent/%s",
                         KeyStoreUtils.KS_SETUP_SCRIPT,
                         KeyStoreUtils.KS_FILENAME,
-                        PasswordGenerator.generateRandomPassword(16),
+                        keystorePassword,
                         validityPeriod,
                         KeyStoreUtils.CSR_FILENAME));
 
@@ -187,12 +186,13 @@ public abstract class LibvirtServerDiscoverer extends DiscovererBase implements
         final SetupCertificateCommand certificateCommand = new SetupCertificateCommand(certificate);
         final SSHCmdHelper.SSHCmdResult setupCertResult = SSHCmdHelper.sshExecuteCmdWithResult(sshConnection,
                 String.format("sudo /usr/share/cloudstack-common/scripts/util/%s " +
-                                    "/etc/cloudstack/agent/agent.properties " +
+                                "/etc/cloudstack/agent/agent.properties %s " +
                                 "/etc/cloudstack/agent/%s %s " +
                                 "/etc/cloudstack/agent/%s \"%s\" " +
                                 "/etc/cloudstack/agent/%s \"%s\" " +
                                 "/etc/cloudstack/agent/%s \"%s\"",
                         KeyStoreUtils.KS_IMPORT_SCRIPT,
+                        keystorePassword,
                         KeyStoreUtils.KS_FILENAME,
                         KeyStoreUtils.SSH_MODE,
                         KeyStoreUtils.CERT_FILENAME,

server/src/main/java/com/cloud/network/NetworkServiceImpl.java

@@ -2345,9 +2345,9 @@ public class NetworkServiceImpl extends ManagerBase implements NetworkService, C
 
     @Override
     @ActionEvent(eventType = EventTypes.EVENT_NETWORK_RESTART, eventDescription = "restarting network", async = true)
-    public boolean restartNetwork(Long networkId, boolean cleanup, boolean makeRedundant, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
+    public boolean restartNetwork(Long networkId, boolean cleanup, boolean makeRedundant, boolean livePatch, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
         NetworkVO network = getNetworkVO(networkId, "Network with specified id doesn't exist");
-        return restartNetwork(network, cleanup, makeRedundant, user);
+        return restartNetwork(network, cleanup, makeRedundant, livePatch, user);
     }
 
     private NetworkVO getNetworkVO(Long networkId, String errMsgFormat) {
@@ -2359,7 +2359,7 @@ public class NetworkServiceImpl extends ManagerBase implements NetworkService, C
     }
 
     @ActionEvent(eventType = EventTypes.EVENT_NETWORK_RESTART, eventDescription = "restarting network", async = true)
-    public boolean restartNetwork(NetworkVO network, boolean cleanup, boolean makeRedundant, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
+    public boolean restartNetwork(NetworkVO network, boolean cleanup, boolean makeRedundant, boolean livePatch, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
 
         // Don't allow to restart network if it's not in Implemented/Setup state
         if (!(network.getState() == Network.State.Implemented || network.getState() == Network.State.Setup)) {
@@ -2384,9 +2384,11 @@ public class NetworkServiceImpl extends ManagerBase implements NetworkService, C
             }
             cleanup = true;
         }
-
+        if (cleanup) {
+            livePatch = false;
+        }
         long id = network.getId();
-        boolean success = _networkMgr.restartNetwork(id, callerAccount, user, cleanup);
+        boolean success = _networkMgr.restartNetwork(id, callerAccount, user, cleanup, livePatch);
         if (success) {
             s_logger.debug(String.format("Network id=%d is restarted successfully.",id));
         } else {
@@ -2406,8 +2408,9 @@ public class NetworkServiceImpl extends ManagerBase implements NetworkService, C
             throwInvalidIdException("Cannot restart a VPC tier with cleanup, please restart the whole VPC.", network.getUuid(), "network tier");
         }
         boolean makeRedundant = cmd.getMakeRedundant();
+        boolean livePatch = cmd.getLivePatch();
         User callerUser = _accountMgr.getActiveUser(CallContext.current().getCallingUserId());
-        return restartNetwork(network, cleanup, makeRedundant, callerUser);
+        return restartNetwork(network, cleanup, makeRedundant, livePatch, callerUser);
     }
 
     @Override

server/src/main/java/com/cloud/network/router/NetworkHelper.java

@@ -65,6 +65,7 @@ public interface NetworkHelper {
      * @return
      */
     public abstract boolean checkRouterVersion(VirtualRouter router);
+    public abstract boolean checkRouterTemplateVersion(VirtualRouter router);
 
     public abstract List<DomainRouterVO> startRouters(
             RouterDeploymentDefinition routerDeploymentDefinition)

server/src/main/java/com/cloud/network/router/NetworkHelperImpl.java

@@ -27,6 +27,7 @@ import java.util.Map;
 import javax.annotation.PostConstruct;
 import javax.inject.Inject;
 
+import com.cloud.utils.validation.ChecksumUtil;
 import org.apache.cloudstack.api.ApiConstants;
 import org.apache.log4j.Logger;
 
@@ -269,6 +270,27 @@ public class NetworkHelperImpl implements NetworkHelper {
 
     @Override
     public boolean checkRouterVersion(final VirtualRouter router) {
+        if (!VirtualNetworkApplianceManager.RouterVersionCheckEnabled.value()) {
+            // Router version check is disabled.
+            return true;
+        }
+        if (router.getTemplateVersion() == null) {
+            return false;
+        }
+        final long dcid = router.getDataCenterId();
+        String routerVersion = CloudStackVersion.trimRouterVersion(router.getTemplateVersion());
+        String currentCheckSum = ChecksumUtil.calculateCurrentChecksum(router.getName(), "vms/cloud-scripts.tgz");
+        String routerChecksum = router.getScriptsVersion() == null ? "" : router.getScriptsVersion();
+        boolean routerVersionMatch = CloudStackVersion.compare(routerVersion, NetworkOrchestrationService.MinVRVersion.valueIn(dcid)) >= 0;
+        if (routerVersionMatch) {
+            return true;
+        }
+        boolean routerCheckSumMatch = currentCheckSum.equals(routerChecksum);
+        return routerCheckSumMatch;
+    }
+
+    @Override
+    public boolean checkRouterTemplateVersion(final VirtualRouter router) {
         if (!VirtualNetworkApplianceManager.RouterVersionCheckEnabled.value()) {
             // Router version check is disabled.
             return true;

server/src/main/java/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java

@@ -46,6 +46,7 @@ import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 
 import com.cloud.offering.DiskOffering;
+import com.cloud.server.ManagementServer;
 import org.apache.cloudstack.alert.AlertService;
 import org.apache.cloudstack.alert.AlertService.AlertType;
 import org.apache.cloudstack.api.command.admin.router.RebootRouterCmd;
@@ -66,6 +67,7 @@ import org.apache.cloudstack.managed.context.ManagedContextRunnable;
 import org.apache.cloudstack.network.router.deployment.RouterDeploymentDefinitionBuilder;
 import org.apache.cloudstack.network.topology.NetworkTopology;
 import org.apache.cloudstack.network.topology.NetworkTopologyContext;
+import org.apache.cloudstack.utils.CloudStackVersion;
 import org.apache.cloudstack.utils.identity.ManagementServerNode;
 import org.apache.cloudstack.utils.usage.UsageUtils;
 import org.apache.commons.lang3.StringUtils;
@@ -364,6 +366,7 @@ Configurable, StateListener<VirtualMachine.State, VirtualMachine.Event, VirtualM
 
     @Inject protected CommandSetupHelper _commandSetupHelper;
     @Inject protected RouterDeploymentDefinitionBuilder _routerDeploymentManagerBuilder;
+    @Inject private ManagementServer mgr;
 
     private int _routerRamSize;
     private int _routerCpuMHz;
@@ -1269,7 +1272,7 @@ Configurable, StateListener<VirtualMachine.State, VirtualMachine.Event, VirtualM
             ActionEventUtils.onActionEvent(User.UID_SYSTEM, Account.ACCOUNT_ID_SYSTEM,
                     Domain.ROOT_DOMAIN, EventTypes.EVENT_ROUTER_HEALTH_CHECKS,
                     "Recreating router " + router.getUuid() + " by restarting VPC " + router.getVpcUuid());
-            return vpcService.restartVpc(router.getVpcId(), true, false, user);
+            return vpcService.restartVpc(router.getVpcId(), true, false, false, user);
         } catch (Exception e) {
             s_logger.error("Failed to restart VPC for router recreation " +
                     router.getVpcName() + " ,router " + router.getUuid(), e);
@@ -1293,7 +1296,7 @@ Configurable, StateListener<VirtualMachine.State, VirtualMachine.Event, VirtualM
             ActionEventUtils.onActionEvent(User.UID_SYSTEM, Account.ACCOUNT_ID_SYSTEM,
                     Domain.ROOT_DOMAIN, EventTypes.EVENT_ROUTER_HEALTH_CHECKS,
                     "Recreating router " + router.getUuid() + " by restarting network " + router.getNetworkUuid());
-            return networkService.restartNetwork(router.getNetworkId(), true, false, user);
+            return networkService.restartNetwork(router.getNetworkId(), true, false, false, user);
         } catch (Exception e) {
             s_logger.error("Failed to restart network " + router.getNetworkName() +
                     " for router recreation " + router.getNetworkName(), e);
@@ -2679,6 +2682,11 @@ Configurable, StateListener<VirtualMachine.State, VirtualMachine.Event, VirtualM
         final GetDomRVersionAnswer versionAnswer = (GetDomRVersionAnswer) cmds.getAnswer("getDomRVersion");
         router.setTemplateVersion(versionAnswer.getTemplateVersion());
         router.setScriptsVersion(versionAnswer.getScriptsVersion());
+        String codeVersion = mgr.getVersion();
+        if (StringUtils.isNotEmpty(codeVersion)) {
+            codeVersion = CloudStackVersion.parse(codeVersion).toString();
+        }
+        router.setSoftwareVersion(codeVersion);
         _routerDao.persist(router, guestNetworks);
 
         final List<? extends Nic> routerNics = _nicDao.listByVmId(profile.getId());
@@ -3209,7 +3217,7 @@ Configurable, StateListener<VirtualMachine.State, VirtualMachine.Event, VirtualM
     private List<Long> rebootRouters(final List<DomainRouterVO> routers) {
         final List<Long> jobIds = new ArrayList<Long>();
         for (final DomainRouterVO router : routers) {
-            if (!_nwHelper.checkRouterVersion(router)) {
+            if (!_nwHelper.checkRouterTemplateVersion(router)) {
                 s_logger.debug("Upgrading template for router: " + router.getId());
                 final Map<String, String> params = new HashMap<String, String>();
                 params.put("ctxUserId", "1");

server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

@@ -1718,13 +1718,14 @@ public class VpcManagerImpl extends ManagerBase implements VpcManager, VpcProvis
         final long vpcId = cmd.getId();
         final boolean cleanUp = cmd.getCleanup();
         final boolean makeRedundant = cmd.getMakeredundant();
+        final boolean livePatch = cmd.getLivePatch();
         final User callerUser = _accountMgr.getActiveUser(CallContext.current().getCallingUserId());
-        return restartVpc(vpcId, cleanUp, makeRedundant, callerUser);
+        return restartVpc(vpcId, cleanUp, makeRedundant, livePatch, callerUser);
     }
 
     @Override
     @ActionEvent(eventType = EventTypes.EVENT_VPC_RESTART, eventDescription = "restarting vpc")
-    public boolean restartVpc(Long vpcId, boolean cleanUp, boolean makeRedundant, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
+    public boolean restartVpc(Long vpcId, boolean cleanUp, boolean makeRedundant, boolean livePatch, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
         Vpc vpc = getActiveVpc(vpcId);
         if (vpc == null) {
             final InvalidParameterValueException ex = new InvalidParameterValueException("Unable to find Enabled VPC by id specified");
@@ -1767,7 +1768,11 @@ public class VpcManagerImpl extends ManagerBase implements VpcManager, VpcProvis
                 return true;
             }
 
-            restartVPCNetworks(vpcId, callerAccount, user, cleanUp);
+            if (cleanUp) {
+                livePatch = false;
+            }
+
+            restartVPCNetworks(vpcId, callerAccount, user, cleanUp, livePatch);
 
             s_logger.debug("Starting VPC " + vpc + " as a part of VPC restart process without cleanup");
             if (!startVpc(vpcId, false)) {
@@ -1785,11 +1790,11 @@ public class VpcManagerImpl extends ManagerBase implements VpcManager, VpcProvis
         }
     }
 
-    private void restartVPCNetworks(long vpcId, Account callerAccount, User callerUser, boolean cleanUp) throws InsufficientCapacityException, ResourceUnavailableException {
+    private void restartVPCNetworks(long vpcId, Account callerAccount, User callerUser, boolean cleanUp, boolean livePatch) throws InsufficientCapacityException, ResourceUnavailableException {
         List<? extends Network> networks = _ntwkModel.listNetworksByVpc(vpcId);
         for (Network network: networks) {
-            if (network.isRestartRequired()) {
+            if (network.isRestartRequired() || livePatch) {
-                _ntwkMgr.restartNetwork(network.getId(), callerAccount, callerUser, cleanUp);
+                _ntwkMgr.restartNetwork(network.getId(), callerAccount, callerUser, cleanUp, livePatch);
             }
         }
     }

server/src/main/java/com/cloud/server/ConfigurationServerImpl.java

@@ -667,7 +667,6 @@ public class ConfigurationServerImpl extends ManagerBase implements Configuratio
             s_logger.info("Keypairs already in database, updating local copy");
             updateKeyPairsOnDisk(homeDir);
         }
-        s_logger.info("Going to update systemvm iso with generated keypairs if needed");
         try {
             copyPrivateKeyToHosts(pubkeyfile.getAbsolutePath(), privkeyfile.getAbsolutePath());
         } catch (CloudRuntimeException e) {
@@ -742,13 +741,10 @@ public class ConfigurationServerImpl extends ManagerBase implements Configuratio
         s_logger.info("Trying to copy private keys to hosts");
         String injectScript = getInjectScript();
         String scriptPath = Script.findScript("", injectScript);
-        String systemVmIsoPath = Script.findScript("", "vms/systemvm.iso");
         if (scriptPath == null) {
             throw new CloudRuntimeException("Unable to find key inject script " + injectScript);
         }
-        if (systemVmIsoPath == null) {
+
-            throw new CloudRuntimeException("Unable to find systemvm iso vms/systemvm.iso");
-        }
         Script command = null;
         if(isOnWindows()) {
             command = new Script("python", s_logger);

server/src/main/java/com/cloud/server/ManagementServer.java

@@ -22,6 +22,7 @@ import com.cloud.storage.GuestOSHypervisorVO;
 import com.cloud.storage.GuestOSVO;
 import com.cloud.utils.Pair;
 import com.cloud.utils.component.PluggableService;
+import com.cloud.vm.VMInstanceVO;
 import com.cloud.vm.VirtualMachine;
 
 /**
@@ -67,4 +68,6 @@ public interface ManagementServer extends ManagementService, PluggableService {
 
     public long getMemoryOrCpuCapacityByHost(Long hostId, short capacityType);
 
+    Pair<Boolean, String> updateSystemVM(VMInstanceVO systemVM, boolean forced);
+
 }

server/src/main/java/com/cloud/server/ManagementServerImpl.java

@@ -43,6 +43,22 @@ import javax.crypto.spec.SecretKeySpec;
 import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.PatchSystemVmAnswer;
+import com.cloud.agent.api.PatchSystemVmCommand;
+import com.cloud.agent.api.routing.NetworkElementCommand;
+import com.cloud.agent.manager.Commands;
+import com.cloud.dc.DomainVlanMapVO;
+import com.cloud.dc.dao.DomainVlanMapDao;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.network.Networks;
+import com.cloud.utils.db.UUIDManager;
+import com.cloud.utils.fsm.StateMachine2;
+import com.cloud.vm.DomainRouterVO;
+import com.cloud.vm.NicVO;
+import com.cloud.vm.dao.DomainRouterDao;
+import com.cloud.vm.dao.NicDao;
 import org.apache.cloudstack.acl.ControlledEntity;
 import org.apache.cloudstack.acl.SecurityChecker;
 import org.apache.cloudstack.affinity.AffinityGroupProcessor;
@@ -226,6 +242,7 @@ import org.apache.cloudstack.api.command.admin.swift.ListSwiftsCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.ListSystemVMsCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd;
+import org.apache.cloudstack.api.command.admin.systemvm.PatchSystemVMCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.RebootSystemVmCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.ScaleSystemVMCmd;
 import org.apache.cloudstack.api.command.admin.systemvm.StartSystemVMCmd;
@@ -585,6 +602,7 @@ import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
 import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
 import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
 import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.cloudstack.utils.CloudStackVersion;
 import org.apache.cloudstack.utils.identity.ManagementServerNode;
 import org.apache.commons.codec.binary.Base64;
 import org.apache.commons.collections.CollectionUtils;
@@ -614,7 +632,6 @@ import com.cloud.consoleproxy.ConsoleProxyManager;
 import com.cloud.dc.AccountVlanMapVO;
 import com.cloud.dc.ClusterVO;
 import com.cloud.dc.DataCenterVO;
-import com.cloud.dc.DomainVlanMapVO;
 import com.cloud.dc.HostPodVO;
 import com.cloud.dc.Pod;
 import com.cloud.dc.PodVlanMapVO;
@@ -624,7 +641,6 @@ import com.cloud.dc.VlanVO;
 import com.cloud.dc.dao.AccountVlanMapDao;
 import com.cloud.dc.dao.ClusterDao;
 import com.cloud.dc.dao.DataCenterDao;
-import com.cloud.dc.dao.DomainVlanMapDao;
 import com.cloud.dc.dao.HostPodDao;
 import com.cloud.dc.dao.PodVlanMapDao;
 import com.cloud.dc.dao.VlanDao;
@@ -769,11 +785,13 @@ import com.cloud.vm.dao.VMInstanceDao;
 
 public class ManagementServerImpl extends ManagerBase implements ManagementServer, Configurable {
     public static final Logger s_logger = Logger.getLogger(ManagementServerImpl.class.getName());
+    protected StateMachine2<State, VirtualMachine.Event, VirtualMachine> _stateMachine;
 
     static final ConfigKey<Integer> vmPasswordLength = new ConfigKey<Integer>("Advanced", Integer.class, "vm.password.length", "6", "Specifies the length of a randomly generated password", false);
     static final ConfigKey<Integer> sshKeyLength = new ConfigKey<Integer>("Advanced", Integer.class, "ssh.key.length", "2048", "Specifies custom SSH key length (bit)", true, ConfigKey.Scope.Global);
     static final ConfigKey<Boolean> humanReadableSizes = new ConfigKey<Boolean>("Advanced", Boolean.class, "display.human.readable.sizes", "true", "Enables outputting human readable byte sizes to logs and usage records.", false, ConfigKey.Scope.Global);
     public static final ConfigKey<String> customCsIdentifier = new ConfigKey<String>("Advanced", String.class, "custom.cs.identifier", UUID.randomUUID().toString().split("-")[0].substring(4), "Custom identifier for the cloudstack installation", true, ConfigKey.Scope.Global);
+    private static final VirtualMachine.Type []systemVmTypes = { VirtualMachine.Type.SecondaryStorageVm, VirtualMachine.Type.ConsoleProxy};
 
     @Inject
     public AccountManager _accountMgr;
@@ -836,7 +854,7 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
     @Inject
     private StoragePoolJoinDao _poolJoinDao;
     @Inject
-    private NetworkDao _networkDao;
+    private NetworkDao networkDao;
     @Inject
     private StorageManager _storageMgr;
     @Inject
@@ -909,10 +927,17 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
     private AnnotationDao annotationDao;
     @Inject
     private DomainVlanMapDao _domainVlanMapDao;
+    @Inject
+    private NicDao nicDao;
+    @Inject
+    DomainRouterDao routerDao;
+    @Inject
+    public UUIDManager uuidMgr;
 
     private LockControllerListener _lockControllerListener;
     private final ScheduledExecutorService _eventExecutor = Executors.newScheduledThreadPool(1, new NamedThreadFactory("EventChecker"));
     private final ScheduledExecutorService _alertExecutor = Executors.newScheduledThreadPool(1, new NamedThreadFactory("AlertChecker"));
+    private static final int patchCommandTimeout = 600000;
 
     private Map<String, String> _configs;
 
@@ -952,6 +977,7 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
 
     public ManagementServerImpl() {
         setRunLevel(ComponentLifecycle.RUN_LEVEL_APPLICATION_MAINLOOP);
+        setStateMachine();
     }
 
     public List<UserAuthenticator> getUserAuthenticators() {
@@ -1010,6 +1036,10 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
         return true;
     }
 
+    private void setStateMachine() {
+        _stateMachine = VirtualMachine.State.getStateMachine();
+    }
+
     @Override
     public boolean start() {
         s_logger.info("Startup CloudStack management server...");
@@ -2197,9 +2227,9 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
                 if (ip == null) {
                     throw new InvalidParameterValueException("Please specify a valid ipaddress id");
                 }
-                network = _networkDao.findById(ip.getSourceNetworkId());
+                network = networkDao.findById(ip.getSourceNetworkId());
             } else {
-                network = _networkDao.findById(networkId);
+                network = networkDao.findById(networkId);
             }
             if (network == null || network.getGuestType() != Network.GuestType.Shared) {
                 throw new InvalidParameterValueException("Please specify a valid network id");
@@ -2271,7 +2301,7 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
         }
 
         if (associatedNetworkId != null) {
-            _accountMgr.checkAccess(caller, null, false, _networkDao.findById(associatedNetworkId));
+            _accountMgr.checkAccess(caller, null, false, networkDao.findById(associatedNetworkId));
             sc.setParameters("associatedNetworkIdEq", associatedNetworkId);
         }
         if (vpcId != null) {
@@ -2292,7 +2322,7 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
                 owner = _accountMgr.finalizeOwner(CallContext.current().getCallingAccount(), cmd.getAccountName(), cmd.getDomainId(), null);
             }
             if (associatedNetworkId != null) {
-                NetworkVO guestNetwork = _networkDao.findById(associatedNetworkId);
+                NetworkVO guestNetwork = networkDao.findById(associatedNetworkId);
                 if (zoneId == null) {
                     zoneId = guestNetwork.getDataCenterId();
                 } else if (zoneId != guestNetwork.getDataCenterId()) {
@@ -3580,6 +3610,7 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
         cmdList.add(UploadResourceIconCmd.class);
         cmdList.add(DeleteResourceIconCmd.class);
         cmdList.add(ListResourceIconCmd.class);
+        cmdList.add(PatchSystemVMCmd.class);
         cmdList.add(ListGuestVlansCmd.class);
 
         // Out-of-band management APIs for admins
@@ -3986,7 +4017,7 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
         boolean elasticLoadBalancerEnabled = false;
         boolean KVMSnapshotEnabled = false;
         String supportELB = "false";
-        final List<NetworkVO> networks = _networkDao.listSecurityGroupEnabledNetworks();
+        final List<NetworkVO> networks = networkDao.listSecurityGroupEnabledNetworks();
         if (networks != null && !networks.isEmpty()) {
             securityGroupsEnabled = true;
             final String elbEnabled = _configDao.getValue(Config.ElasticLoadBalancerEnabled.key());
@@ -4717,6 +4748,104 @@ public class ManagementServerImpl extends ManagerBase implements ManagementServe
         _dpMgr.cleanupVMReservations();
     }
 
+    @Override
+    public Pair<Boolean, String> patchSystemVM(PatchSystemVMCmd cmd) {
+        Long systemVmId = cmd.getId();
+        boolean forced = cmd.isForced();
+
+        if (systemVmId == null) {
+            throw new InvalidParameterValueException("Please provide a valid ID of a system VM to be patched");
+        }
+
+        final VMInstanceVO systemVm = _vmInstanceDao.findByIdTypes(systemVmId, systemVmTypes);
+        if (systemVm == null) {
+            throw new InvalidParameterValueException(String.format("Unable to find SystemVm with id %s. patchSystemVm API can be used to patch CPVM / SSVM only.", systemVmId));
+        }
+
+        return updateSystemVM(systemVm, forced);
+    }
+
+
+    private String getControlIp(final long systemVmId) {
+        String controlIpAddress = null;
+        final List<NicVO> nics = nicDao.listByVmId(systemVmId);
+        for (final NicVO n : nics) {
+            final NetworkVO nc = networkDao.findById(n.getNetworkId());
+            if (nc != null && nc.getTrafficType() == Networks.TrafficType.Control) {
+                controlIpAddress = n.getIPv4Address();
+                // router will have only one control IP
+                break;
+            }
+        }
+
+        if (controlIpAddress == null) {
+            s_logger.warn(String.format("Unable to find systemVm's control ip in its attached NICs!. systemVmId: %s", systemVmId));
+            VMInstanceVO systemVM = _vmInstanceDao.findById(systemVmId);
+            return systemVM.getPrivateIpAddress();
+        }
+
+        return controlIpAddress;
+    }
+
+    public Pair<Boolean, String> updateSystemVM(VMInstanceVO systemVM, boolean forced) {
+        String msg = String.format("Unable to patch SystemVM: %s as it is not in Running state. Please destroy and recreate the SystemVM.", systemVM);
+        if (systemVM.getState() != State.Running) {
+            s_logger.error(msg);
+            return new Pair<>(false, msg);
+        }
+        return patchSystemVm(systemVM, forced);
+    }
+
+    private boolean updateRouterDetails(Long routerId, String scriptVersion, String templateVersion) {
+        DomainRouterVO router = routerDao.findById(routerId);
+        if (router == null) {
+            throw new CloudRuntimeException(String.format("Failed to find router with id: %s", routerId));
+        }
+
+        router.setTemplateVersion(templateVersion);
+        router.setScriptsVersion(scriptVersion);
+        String codeVersion = getVersion();
+        if (StringUtils.isNotEmpty(codeVersion)) {
+            codeVersion = CloudStackVersion.parse(codeVersion).toString();
+        }
+        router.setSoftwareVersion(codeVersion);
+        return routerDao.update(routerId, router);
+    }
+
+    private Pair<Boolean, String> patchSystemVm(VMInstanceVO systemVM, boolean forced) {
+        PatchSystemVmAnswer answer;
+        final PatchSystemVmCommand command = new PatchSystemVmCommand();
+        command.setAccessDetail(NetworkElementCommand.ROUTER_IP, getControlIp(systemVM.getId()));
+        command.setAccessDetail(NetworkElementCommand.ROUTER_NAME, systemVM.getInstanceName());
+        command.setForced(forced);
+        try {
+            Commands cmds = new Commands(Command.OnError.Stop);
+            cmds.addCommand(command);
+            Answer[] answers = _agentMgr.send(systemVM.getHostId(), cmds, patchCommandTimeout);
+            answer = (PatchSystemVmAnswer) answers[0];
+            if (!answer.getResult()) {
+                String errMsg = String.format("Failed to patch systemVM %s due to %s", systemVM.getInstanceName(), answer.getDetails());
+                s_logger.error(errMsg);
+                return new Pair<>(false, errMsg);
+            }
+        } catch (AgentUnavailableException | OperationTimedoutException e) {
+            String errMsg = "SystemVM live patch failed";
+            s_logger.error(errMsg, e);
+            return new Pair<>(false,  String.format("%s due to: %s", errMsg, e.getMessage()));
+        }
+        s_logger.info(String.format("Successfully patched system VM %s", systemVM.getInstanceName()));
+        List<VirtualMachine.Type> routerTypes = new ArrayList<>();
+        routerTypes.add(VirtualMachine.Type.DomainRouter);
+        routerTypes.add(VirtualMachine.Type.InternalLoadBalancerVm);
+        if (routerTypes.contains(systemVM.getType())) {
+            boolean updated = updateRouterDetails(systemVM.getId(), answer.getScriptsVersion(), answer.getTemplateVersion());
+            if (!updated) {
+                s_logger.warn("Failed to update router's script and template version details");
+            }
+        }
+        return new Pair<>(true, answer.getDetails());
+    }
+
     public List<StoragePoolAllocator> getStoragePoolAllocators() {
         return _storagePoolAllocators;
     }

server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java

@@ -184,12 +184,15 @@ public class CAManagerImpl extends ManagerBase implements CAManager {
         }
         CallContext.current().setEventDetails("host id: " + host.getId());
         CallContext.current().putContextParameter(Host.class, host.getUuid());
-        final String csr;
+        String csr = null;
+
         try {
+            if (host.getType() != Host.Type.ConsoleProxy && host.getType() != Host.Type.SecondaryStorageVM) {
                 csr = generateKeyStoreAndCsr(host, null);
                 if (StringUtils.isEmpty(csr)) {
                     return false;
                 }
+            }
             final Certificate certificate = issueCertificate(csr, Arrays.asList(host.getName(), host.getPrivateIpAddress()), Arrays.asList(host.getPrivateIpAddress(), host.getPublicIpAddress(), host.getStorageIpAddress()), CAManager.CertValidityPeriod.value(), caProvider);
             return deployCertificate(host, certificate, reconnect, null);
         } catch (final AgentUnavailableException | OperationTimedoutException e) {
@@ -209,6 +212,11 @@ public class CAManagerImpl extends ManagerBase implements CAManager {
         return answer.getCsr();
     }
 
+    private boolean isValidSystemVMType(Host.Type type) {
+        return Host.Type.SecondaryStorageVM.equals(type) ||
+                Host.Type.ConsoleProxy.equals(type);
+    }
+
     @Override
     public boolean deployCertificate(final Host host, final Certificate certificate, final Boolean reconnect, final Map<String, String> sshAccessDetails)
             throws AgentUnavailableException, OperationTimedoutException {

server/src/test/java/com/cloud/vpc/MockNetworkManagerImpl.java

@@ -231,7 +231,7 @@ public class MockNetworkManagerImpl extends ManagerBase implements NetworkOrches
     }
 
     @Override
-    public boolean restartNetwork(Long networkId, boolean cleanup, boolean makeRedundant, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
+    public boolean restartNetwork(Long networkId, boolean cleanup, boolean makeRedundant, boolean livePatch, User user) throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
         return false;
     }
 
@@ -777,7 +777,7 @@ public class MockNetworkManagerImpl extends ManagerBase implements NetworkOrches
      * @see com.cloud.network.NetworkManager#restartNetwork(java.lang.Long, com.cloud.user.Account, com.cloud.user.User, boolean)
      */
     @Override
-    public boolean restartNetwork(Long networkId, Account callerAccount, User callerUser, boolean cleanup) throws ConcurrentOperationException,
+    public boolean restartNetwork(Long networkId, Account callerAccount, User callerUser, boolean cleanup, boolean livePatch) throws ConcurrentOperationException,
         ResourceUnavailableException, InsufficientCapacityException {
         // TODO Auto-generated method stub
         return false;

services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxy.java

@@ -31,6 +31,7 @@ import java.util.Map;
 import java.util.Properties;
 import java.util.concurrent.Executor;
 
+import org.apache.commons.lang3.ArrayUtils;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.log4j.xml.DOMConfigurator;
 import org.eclipse.jetty.websocket.api.Session;
@@ -74,6 +75,7 @@ public class ConsoleProxy {
     static boolean standaloneStart = false;
 
     static String encryptorPassword = "Dummy";
+    static final String[] skipProperties = new String[]{"certificate", "cacertificate", "keystore_password", "privatekey"};
 
     private static void configLog4j() {
         final ClassLoader loader = Thread.currentThread().getContextClassLoader();
@@ -109,6 +111,9 @@ public class ConsoleProxy {
         s_logger.info("Configure console proxy...");
         for (Object key : conf.keySet()) {
             s_logger.info("Property " + (String)key + ": " + conf.getProperty((String)key));
+            if (!ArrayUtils.contains(skipProperties, key)) {
+                s_logger.info("Property " + (String)key + ": " + conf.getProperty((String)key));
+            }
         }
 
         String s = conf.getProperty("consoleproxy.httpListenPort");
@@ -247,7 +252,9 @@ public class ConsoleProxy {
 
         if (conf != null) {
             for (Object key : conf.keySet()) {
-                s_logger.info("Context property " + (String)key + ": " + conf.getProperty((String)key));
+                if (!ArrayUtils.contains(skipProperties, key)) {
+                    s_logger.info("Context property " + (String) key + ": " + conf.getProperty((String) key));
+                }
             }
         }
 

services/secondary-storage/controller/src/main/java/org/apache/cloudstack/secondarystorage/SecondaryStorageManagerImpl.java

@@ -30,12 +30,15 @@ import java.util.Map;
 import javax.inject.Inject;
 import javax.naming.ConfigurationException;
 
+import com.cloud.utils.PasswordGenerator;
 import org.apache.cloudstack.agent.lb.IndirectAgentLB;
+import org.apache.cloudstack.ca.CAManager;
 import org.apache.cloudstack.context.CallContext;
 import org.apache.cloudstack.engine.orchestration.service.NetworkOrchestrationService;
 import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
 import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
 import org.apache.cloudstack.engine.subsystem.api.storage.ZoneScope;
+import org.apache.cloudstack.framework.ca.Certificate;
 import org.apache.cloudstack.framework.config.ConfigKey;
 import org.apache.cloudstack.framework.config.Configurable;
 import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
@@ -245,6 +248,8 @@ public class SecondaryStorageManagerImpl extends ManagerBase implements Secondar
     private ImageStoreDetailsUtil imageStoreDetailsUtil;
     @Inject
     private IndirectAgentLB indirectAgentLB;
+    @Inject
+    private CAManager caManager;
 
     private long _capacityScanInterval = DEFAULT_CAPACITY_SCAN_INTERVAL_IN_MILLISECONDS;
     private int _secStorageVmMtuSize;
@@ -1070,6 +1075,12 @@ public class SecondaryStorageManagerImpl extends ManagerBase implements Secondar
             return false;
         }
 
+        final Map<String, String> sshAccessDetails = _networkMgr.getSystemVMAccessDetails(profile.getVirtualMachine());
+        final Map<String, String> ipAddressDetails = new HashMap<>(sshAccessDetails);
+        ipAddressDetails.remove("router.name");
+        final Certificate certificate = caManager.issueCertificate(null, Arrays.asList(profile.getHostName(), profile.getInstanceName()),
+                new ArrayList<>(ipAddressDetails.values()), CAManager.CertValidityPeriod.value(), null);
+
         StringBuilder buf = profile.getBootArgsBuilder();
         buf.append(" template=domP type=secstorage");
         buf.append(" host=").append(com.cloud.utils.StringUtils.toCSVList(indirectAgentLB.getManagementServerList(dest.getHost().getId(), dest.getDataCenter().getId(), null)));
@@ -1157,7 +1168,7 @@ public class SecondaryStorageManagerImpl extends ManagerBase implements Secondar
         }
         String nfsVersion = imageStoreDetailsUtil != null ? imageStoreDetailsUtil.getNfsVersion(secStore.getId()) : null;
         buf.append(" nfsVersion=").append(nfsVersion);
-
+        buf.append(" keystore_password=").append(VirtualMachineGuru.getEncodedString(PasswordGenerator.generateRandomPassword(16)));
         String bootArgs = buf.toString();
         if (s_logger.isDebugEnabled()) {
             s_logger.debug(String.format("Boot args for machine profile [%s]: [%s].", profile.toString(), bootArgs));

systemvm/debian/etc/systemd/system/cloud-early-config.service

@@ -2,11 +2,8 @@
 Description=CloudStack post-boot patching service using cmdline
 DefaultDependencies=no
 
-Before=network-pre.target
+Requires=local-fs.target cloud-preinit.service
-Wants=network-pre.target
+After=local-fs.target cloud-preinit.service
-
-Requires=local-fs.target
-After=local-fs.target
 
 [Install]
 WantedBy=multi-user.target

systemvm/debian/etc/systemd/system/cloud-postinit.service

@@ -1,7 +1,6 @@
 [Unit]
 Description=CloudStack post-patching init script
 After=cloud-early-config.service network.target local-fs.target
-Before=ssh.service
 
 [Install]
 WantedBy=multi-user.target

systemvm/debian/etc/systemd/system/cloud-preinit.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=CloudStack service to initialize interfaces
+DefaultDependencies=no
+
+Before=network-pre.target
+Wants=network-pre.target
+
+Requires=local-fs.target
+After=local-fs.target
+
+[Install]
+WantedBy=multi-user.target
+
+[Service]
+Type=oneshot
+ExecStart=/opt/cloud/bin/setup/init.sh
+RemainAfterExit=true
+TimeoutStartSec=5min

systemvm/debian/opt/cloud/bin/patched.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ls -lrt $1

systemvm/debian/opt/cloud/bin/setup/bootstrap.sh

@@ -15,7 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
+set -x
 PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
 CMDLINE=/var/cache/cloud/cmdline
 
@@ -29,124 +29,6 @@ log_it() {
   log_action_msg "$@"
 }
 
-hypervisor() {
-  if [ -d /proc/xen ]; then
-    mount -t xenfs none /proc/xen
-    $(dmesg | grep -q "Xen HVM")
-    if [ $? -eq 0 ]; then  # 1=PV,0=HVM
-      echo "xen-hvm" && return 0
-    else
-      echo "xen-pv" && return 0
-    fi
-  fi
-
-  [ -x /usr/sbin/virt-what ] && local facts=( $(virt-what) )
-  if [ "$facts" != "" ]; then
-    # Xen HVM is recognized as Hyperv when Viridian extensions are enabled
-    if [ "${facts[-1]}" == "xen-domU" ] && [ "${facts[0]}" == "hyperv" ]; then
-      echo "xen-hvm" && return 0
-    else
-      echo ${facts[-1]} && return 0
-    fi
-  fi
-
-  grep -q QEMU /proc/cpuinfo  && echo "kvm" && return 0
-  grep -q QEMU /var/log/messages && echo "kvm" && return 0
-
-  vmware-checkvm &> /dev/null && echo "vmware" && return 0
-
-  echo "unknown" && return 1
-}
-
-config_guest() {
-  [ ! -d /proc/xen ] && sed -i 's/^vc/#vc/' /etc/inittab && telinit q
-  [ -d /proc/xen ] && sed -i 's/^#vc/vc/' /etc/inittab && telinit q
-
-  systemctl daemon-reload
-
-  case $HYPERVISOR in
-     xen-pv|xen-domU)
-          systemctl stop ntpd
-          systemctl disable ntpd
-          systemctl enable xe-daemon
-          systemctl start xe-daemon
-
-          cat /proc/cmdline > $CMDLINE
-          sed -i "s/%/ /g" $CMDLINE
-          ;;
-     xen-hvm)
-          systemctl stop ntpd
-          systemctl disable ntpd
-          systemctl enable xe-daemon
-          systemctl start xe-daemon
-
-          if [ ! -f /usr/bin/xenstore-read ]; then
-            log_it "ERROR: xentools not installed, cannot found xenstore-read" && exit 5
-          fi
-          /usr/bin/xenstore-read vm-data/cloudstack/init > $CMDLINE
-          sed -i "s/%/ /g" $CMDLINE
-          ;;
-     kvm)
-          # Configure kvm hotplug support
-          if grep -E 'CONFIG_HOTPLUG_PCI=y|CONFIG_HOTPLUG_PCI_ACPI=y' /boot/config-`uname -r`; then
-            log_it "acpiphp and pci_hotplug module already compiled in"
-          else
-            modprobe acpiphp 2> /dev/null && log_it "acpiphp module loaded" || true
-            modprobe pci_hotplug 2> /dev/null && log_it "pci_hotplug module loaded" || true
-          fi
-
-          sed -i -e "/^s0:2345:respawn.*/d" /etc/inittab
-          sed -i -e "/6:23:respawn/a\s0:2345:respawn:/sbin/getty -L 115200 ttyS0 vt102" /etc/inittab
-          systemctl enable qemu-guest-agent
-          systemctl start qemu-guest-agent
-
-          # Wait for $CMDLINE file to be written by the qemu-guest-agent
-          for i in {1..60}; do
-            if [ -s $CMDLINE ]; then
-              log_it "Received a new non-empty cmdline file from qemu-guest-agent"
-              # Remove old configuration files in /etc/cloudstack if VR is booted from cloudstack
-              rm -rf /etc/cloudstack/*.json
-              log_it "Booting from cloudstack, remove old configuration files in /etc/cloudstack/"
-              break
-            fi
-            sleep 1
-          done
-          if [ ! -s $CMDLINE  ]; then
-            log_it "Failed to receive the cmdline file via the qemu-guest-agent"
-          fi
-          ;;
-     vmware)
-          # system time sync'd with host via vmware tools
-          systemctl stop ntpd
-          systemctl disable ntpd
-          systemctl enable open-vm-tools
-          systemctl start open-vm-tools
-
-          vmtoolsd --cmd 'machine.id.get' > $CMDLINE
-          ;;
-     virtualpc|hyperv)
-          # Hyper-V is recognized as virtualpc hypervisor type. Boot args are passed using KVP Daemon
-          systemctl enable hyperv-daemons.hv-fcopy-daemon.service hyperv-daemons.hv-kvp-daemon.service hyperv-daemons.hv-vss-daemon.service
-          systemctl start hyperv-daemons.hv-fcopy-daemon.service hyperv-daemons.hv-kvp-daemon.service hyperv-daemons.hv-vss-daemon.service
-          sleep 5
-          cp -f /var/opt/hyperv/.kvp_pool_0 $CMDLINE
-          cat /dev/null > /var/opt/hyperv/.kvp_pool_0
-          ;;
-     virtualbox)
-          # Virtualbox is used to test the virtual router
-          # get the commandline from a dmistring  (yes, hacky!)
-          dmidecode | grep cmdline | sed 's/^.*cmdline://' > $CMDLINE
-          RV=$?
-          if [ $RV -ne 0 ] ; then
-            log_it "Failed to get cmdline from a virtualbox dmi property"
-          fi
-          ;;
-  esac
-
-  # Find and export guest type
-  export TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
-}
-
 patch_systemvm() {
   local patchfile=$1
   local backupfolder="/tmp/.conf.backup"
@@ -158,6 +40,8 @@ patch_systemvm() {
   fi
   rm /usr/local/cloud/systemvm -rf
   mkdir -p /usr/local/cloud/systemvm
+  ls -lrt $patchfile
+
   echo "All" | unzip $patchfile -d /usr/local/cloud/systemvm >$logfile 2>&1
   find /usr/local/cloud/systemvm/ -name \*.sh | xargs chmod 555
   if [ -f $backupfolder/cloud.jks ]; then
@@ -171,7 +55,7 @@ patch_systemvm() {
 }
 
 patch() {
-  local PATCH_MOUNT=/media/cdrom
+  local PATCH_MOUNT=/tmp/
   local logfile="/var/log/patchsystemvm.log"
 
   if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ]  && [ -f ${PATCH_MOUNT}/agent.zip ] && [ -f /var/cache/cloud/patch.required ]
@@ -188,11 +72,7 @@ patch() {
   rm -f /var/cache/cloud/patch.required
   chmod -x /etc/systemd/system/cloud*.service
   systemctl daemon-reload
-  umount $PATCH_MOUNT || true
 
-  if [ -f /mnt/cmdline ]; then
-    cat /mnt/cmdline > $CMDLINE
-  fi
   return 0
 }
 
@@ -212,11 +92,7 @@ config_sysctl() {
 bootstrap() {
   log_it "Bootstrapping systemvm appliance"
 
-  export HYPERVISOR=$(hypervisor)
+  export TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
-  [ $? -ne 0 ] && log_it "Failed to detect hypervisor type, bailing out" && exit 10
-  log_it "Starting guest services for $HYPERVISOR"
-
-  config_guest
   patch
   config_sysctl
 

systemvm/debian/opt/cloud/bin/setup/cksnode.sh

@@ -28,6 +28,7 @@ setup_k8s_node() {
 
     # set default ssh port and restart sshd service
     sed -i 's/3922/22/g' /etc/ssh/sshd_config
+    systemctl restart ssh
 
     # Prevent root login
     > /root/.ssh/authorized_keys
@@ -39,7 +40,6 @@ setup_k8s_node() {
     log_it "Swap disabled"
 
     log_it "Setting up interfaces"
-    setup_common eth0
     setup_system_rfc1918_internal
 
     log_it "Setting up entry in hosts"

systemvm/debian/opt/cloud/bin/setup/cloud-early-config

@@ -15,7 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
+set -x
 PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
 
 # Clear boot up flag, it would be created by rc.local after boot up done
@@ -31,37 +31,43 @@ log_it() {
   log_action_msg "$@"
 }
 
+validate_checksums() {
+  local oldmd5=
+  [ -f ${1} ] && oldmd5=$(cat ${1})
+  local newmd5=
+  [ -f ${2} ] && newmd5=$(md5sum ${2} | awk '{print $1}')
+  log_it "Scripts checksum detected: oldmd5=$oldmd5 newmd5=$newmd5" >> /dev/null 2>&1
+  echo "oldmd5='${oldmd5}'; newmd5='${newmd5}'"
+}
+
 patch() {
-  local PATCH_MOUNT=/media/cdrom
+  local PATCH_MOUNT=/tmp
-  local patchfile=$PATCH_MOUNT/cloud-scripts.tgz
+  local PATCH_SCRIPTS=cloud-scripts.tgz
+  local oldpatchfile=/usr/share/cloud/$PATCH_SCRIPTS
+  local patchfile=$PATCH_MOUNT/$PATCH_SCRIPTS
   local privkey=$PATCH_MOUNT/authorized_keys
   local md5file=/var/cache/cloud/cloud-scripts-signature
-  local cdrom_dev=
   mkdir -p $PATCH_MOUNT
 
-  if [ -e /dev/xvdd ]; then
-       cdrom_dev=/dev/xvdd
-  elif [ -e /dev/cdrom ]; then
-       cdrom_dev=/dev/cdrom
-  elif [ -e /dev/cdrom1 ]; then
-       cdrom_dev=/dev/cdrom1
-  elif [ -e /dev/cdrom2 ]; then
-       cdrom_dev=/dev/cdrom2
-  elif [ -e /dev/cdrom3 ]; then
-       cdrom_dev=/dev/cdrom3
-  fi
-
   if [ -f /var/cache/cloud/authorized_keys ]; then
     privkey=/var/cache/cloud/authorized_keys
   fi
 
-  if [ -n "$cdrom_dev" ]; then
+  eval $(validate_checksums $md5file $oldpatchfile)
-    mount -o ro $cdrom_dev $PATCH_MOUNT
+  if [ "$oldmd5" == "$newmd5" ] && [ -d /usr/local/cloud/systemvm ] && [ "$(ls -A /usr/local/cloud/systemvm)" ]; then
-    local oldmd5=
+    log_it "Checksum matches, no need to patch"
-    [ -f ${md5file} ] && oldmd5=$(cat ${md5file})
+    return 0
-    local newmd5=
+  fi
-    [ -f ${patchfile} ] && newmd5=$(md5sum ${patchfile} | awk '{print $1}')
+
-    log_it "Scripts checksum detected: oldmd5=$oldmd5 newmd5=$newmd5"
+  CMDLINE=/var/cache/cloud/cmdline
+  export TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
+  retry=60
+  local patched=false
+  if [ "$TYPE" != "cksnode" ]; then
+    while [ $retry -gt 0 ]
+    do
+      if [ -f $patchfile ]; then
+        eval $(validate_checksums $md5file $patchfile)
         if [ "$oldmd5" != "$newmd5" ] && [ -f ${patchfile} ] && [ "$newmd5" != "" ]
         then
           tar xzf $patchfile -C /
@@ -74,9 +80,25 @@ patch() {
           cp -f $privkey /root/.ssh/
           chmod go-rwx /root/.ssh/authorized_keys
         fi
+        patched=true
+        break
       fi
 
+      sleep 2
+      retry=$(($retry-1))
+      log_it "Could not find patch file, retrying"
+    done
+
+    if [ $retry -eq 0 ] && [ "$patched" == "false" ]; then
+      return 2
+    fi
     return 0
+  fi
+}
+
+cleanup() {
+  rm -rf /tmp/agent.zip
+  mv /tmp/cloud-scripts.tgz /usr/share/cloud/cloud-scripts.tgz
 }
 
 start() {
@@ -99,6 +121,7 @@ start() {
   patch
   sync
   /opt/cloud/bin/setup/bootstrap.sh
+  cleanup
 
   log_it "Finished setting up systemvm"
   exit 0

systemvm/debian/opt/cloud/bin/setup/common.sh

@@ -15,7 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
+set -x
 PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
 
 . /lib/lsb/init-functions
@@ -767,6 +767,21 @@ parse_cmd_line() {
         authorized_key)
             export AUTHORIZED_KEYS=$VALUE
             ;;
+        keystore_password)
+            export KEYSTORE_PSSWD=$VALUE
+            ;;
+        validity)
+          export KS_VALIDITY=$VALUE
+          ;;
+        certificate)
+          export CERTIFICATE=$VALUE
+          ;;
+        cacertificate)
+          export CACERTIFICATE=$VALUE
+          ;;
+        privatekey)
+          export PRIVATEKEY=$VALUE
+          ;;
       esac
   done
   echo -e "\n\t}\n}" >> ${CHEF_TMP_FILE}

systemvm/debian/opt/cloud/bin/setup/consoleproxy.sh

@@ -25,7 +25,6 @@ setup_console_proxy() {
   echo "haproxy dnsmasq apache2 nfs-common portmap" > /var/cache/cloud/disabled_svcs
   mkdir -p /var/log/cloud
 
-  setup_common eth0 eth1 eth2
   setup_system_rfc1918_internal
 
   log_it "Setting up entry in hosts"
@@ -33,21 +32,11 @@ setup_console_proxy() {
   public_ip=`getPublicIp`
   echo "$public_ip $NAME" >> /etc/hosts
 
-  log_it "Applying iptables rules"
-  cp /etc/iptables/iptables-consoleproxy /etc/iptables/rules.v4
-
-  log_it "Configuring sshd"
-  local hyp=$HYPERVISOR
-  if [ "$hyp" == "vmware" ] || [ "$hyp" == "hyperv" ]; then
-    setup_sshd $ETH1_IP "eth1"
-  else
-    setup_sshd $ETH0_IP "eth0"
-  fi
-
   disable_rpfilter
   enable_fwding 0
   enable_irqbalance 0
   rm -f /etc/logrotate.d/cloud
+
 }
 
 setup_console_proxy

systemvm/debian/opt/cloud/bin/setup/dhcpsrvr.sh

@@ -25,7 +25,6 @@ dhcpsrvr_svcs() {
 
 setup_dhcpsrvr() {
   log_it "Setting up dhcp server system vm"
-  setup_common eth0 eth1
   setup_dnsmasq
   setup_apache2 $ETH0_IP
 
@@ -36,18 +35,10 @@ setup_dhcpsrvr() {
   enable_irqbalance 0
   enable_fwding 0
 
-  cp /etc/iptables/iptables-router /etc/iptables/rules.v4
-
   #Only allow DNS service for current network
   sed -i "s/-A INPUT -i eth0 -p udp -m udp --dport 53 -j ACCEPT/-A INPUT -i eth0 -p udp -m udp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" /etc/iptables/rules.v4
   sed -i "s/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" /etc/iptables/rules.v4
 
-  if [ "$SSHONGUEST" == "true" ]
-  then
-    setup_sshd $ETH0_IP "eth0"
-  else
-    setup_sshd $ETH1_IP "eth1"
-  fi
 }
 
 dhcpsrvr_svcs

systemvm/debian/opt/cloud/bin/setup/elbvm.sh

@@ -25,20 +25,11 @@ elbvm_svcs() {
 
 setup_elbvm() {
   log_it "Setting up Elastic Load Balancer system vm"
-  setup_common eth0 eth1
   sed -i  /$NAME/d /etc/hosts
   public_ip=$ETH2_IP
   [ "$ETH2_IP" == "0.0.0.0" ] || [ "$ETH2_IP" == "" ] && public_ip=$ETH0_IP
   echo "$public_ip $NAME" >> /etc/hosts
 
-  cp /etc/iptables/iptables-elbvm /etc/iptables/rules.v4
-  if [ "$SSHONGUEST" == "true" ]
-  then
-    setup_sshd $ETH0_IP "eth0"
-  else
-    setup_sshd $ETH1_IP "eth1"
-  fi
-
   enable_fwding 0
   enable_irqbalance 0
 }

systemvm/debian/opt/cloud/bin/setup/ilbvm.sh

@@ -25,7 +25,6 @@ ilbvm_svcs() {
 
 setup_ilbvm() {
   log_it "Setting up Internal Load Balancer system vm"
-  setup_common eth0 eth1
   #eth0 = guest network, eth1=control network
 
   sed -i  /$NAME/d /etc/hosts

systemvm/debian/opt/cloud/bin/setup/init.sh

@@ -0,0 +1,217 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -x
+PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
+CMDLINE=/var/cache/cloud/cmdline
+
+hypervisor() {
+  if [ -d /proc/xen ]; then
+    mount -t xenfs none /proc/xen
+    $(dmesg | grep -q "Xen HVM")
+    if [ $? -eq 0 ]; then  # 1=PV,0=HVM
+      echo "xen-hvm" && return 0
+    else
+      echo "xen-pv" && return 0
+    fi
+  fi
+
+  [ -x /usr/sbin/virt-what ] && local facts=( $(virt-what) )
+  if [ "$facts" != "" ]; then
+    # Xen HVM is recognized as Hyperv when Viridian extensions are enabled
+    if [ "${facts[-1]}" == "xen-domU" ] && [ "${facts[0]}" == "hyperv" ]; then
+      echo "xen-hvm" && return 0
+    else
+      echo ${facts[-1]} && return 0
+    fi
+  fi
+
+  grep -q QEMU /proc/cpuinfo  && echo "kvm" && return 0
+  grep -q QEMU /var/log/messages && echo "kvm" && return 0
+
+  vmware-checkvm &> /dev/null && echo "vmware" && return 0
+
+  echo "unknown" && return 1
+}
+
+config_guest() {
+  [ ! -d /proc/xen ] && sed -i 's/^vc/#vc/' /etc/inittab && telinit q
+  [ -d /proc/xen ] && sed -i 's/^#vc/vc/' /etc/inittab && telinit q
+
+  systemctl daemon-reload
+
+  case $HYPERVISOR in
+     xen-pv|xen-domU)
+          systemctl stop ntpd
+          systemctl disable ntpd
+          systemctl enable xe-daemon
+          systemctl start xe-daemon
+
+          cat /proc/cmdline > $CMDLINE
+          sed -i "s/%/ /g" $CMDLINE
+          ;;
+     xen-hvm)
+          systemctl stop ntpd
+          systemctl disable ntpd
+          systemctl enable xe-daemon
+          systemctl start xe-daemon
+
+          if [ ! -f /usr/bin/xenstore-read ]; then
+            log_it "ERROR: xentools not installed, cannot found xenstore-read" && exit 5
+          fi
+          /usr/bin/xenstore-read vm-data/cloudstack/init > $CMDLINE
+          sed -i "s/%/ /g" $CMDLINE
+          ;;
+     kvm)
+          # Configure kvm hotplug support
+          if grep -E 'CONFIG_HOTPLUG_PCI=y|CONFIG_HOTPLUG_PCI_ACPI=y' /boot/config-`uname -r`; then
+            log_it "acpiphp and pci_hotplug module already compiled in"
+          else
+            modprobe acpiphp 2> /dev/null && log_it "acpiphp module loaded" || true
+            modprobe pci_hotplug 2> /dev/null && log_it "pci_hotplug module loaded" || true
+          fi
+
+          sed -i -e "/^s0:2345:respawn.*/d" /etc/inittab
+          sed -i -e "/6:23:respawn/a\s0:2345:respawn:/sbin/getty -L 115200 ttyS0 vt102" /etc/inittab
+          systemctl enable qemu-guest-agent
+          systemctl start qemu-guest-agent
+
+          # Wait for $CMDLINE file to be written by the qemu-guest-agent
+          for i in {1..60}; do
+            if [ -s $CMDLINE ]; then
+              log_it "Received a new non-empty cmdline file from qemu-guest-agent"
+              # Remove old configuration files in /etc/cloudstack if VR is booted from cloudstack
+              rm -rf /etc/cloudstack/*.json
+              log_it "Booting from cloudstack, remove old configuration files in /etc/cloudstack/"
+              break
+            fi
+            sleep 1
+          done
+          if [ ! -s $CMDLINE  ]; then
+            log_it "Failed to receive the cmdline file via the qemu-guest-agent"
+          fi
+          ;;
+     vmware)
+          # system time sync'd with host via vmware tools
+          systemctl stop ntpd
+          systemctl disable ntpd
+          systemctl enable open-vm-tools
+          systemctl start open-vm-tools
+
+          vmtoolsd --cmd 'machine.id.get' > $CMDLINE
+          ;;
+     virtualpc|hyperv)
+          # Hyper-V is recognized as virtualpc hypervisor type. Boot args are passed using KVP Daemon
+          systemctl enable hyperv-daemons.hv-fcopy-daemon.service hyperv-daemons.hv-kvp-daemon.service hyperv-daemons.hv-vss-daemon.service
+          systemctl start hyperv-daemons.hv-fcopy-daemon.service hyperv-daemons.hv-kvp-daemon.service hyperv-daemons.hv-vss-daemon.service
+          sleep 5
+          cp -f /var/opt/hyperv/.kvp_pool_0 $CMDLINE
+          cat /dev/null > /var/opt/hyperv/.kvp_pool_0
+          ;;
+     virtualbox)
+          # Virtualbox is used to test the virtual router
+          # get the commandline from a dmistring  (yes, hacky!)
+          dmidecode | grep cmdline | sed 's/^.*cmdline://' > $CMDLINE
+          RV=$?
+          if [ $RV -ne 0 ] ; then
+            log_it "Failed to get cmdline from a virtualbox dmi property"
+          fi
+          ;;
+  esac
+
+  if [ -f /mnt/cmdline ]; then
+    cat /mnt/cmdline > $CMDLINE
+  fi
+
+  # Find and export guest type
+  export TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
+}
+
+setup_interface_sshd() {
+
+  if [ "$TYPE" != "cksnode" ]; then
+    log_it "Applying iptables rules"
+    if [ "$TYPE" != "dhcpsrvr" ]; then
+      cp /etc/iptables/iptables-$TYPE /etc/iptables/rules.v4
+    else
+      cp /etc/iptables/iptables-router /etc/iptables/rules.v4
+    fi
+  fi
+
+  if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ]; then
+    setup_common eth0 eth1 eth2
+    log_it "Configuring sshd"
+    local hyp=$HYPERVISOR
+    if [ "$hyp" == "vmware" ] || [ "$hyp" == "hyperv" ]; then
+      setup_sshd $ETH1_IP "eth1"
+    else
+      setup_sshd $ETH0_IP "eth0"
+    fi
+
+  elif [ "$TYPE" == "router" ]; then
+    if [ -n "$ETH2_IP" ]; then
+      setup_common eth0 eth1 eth2
+
+      if [ -n "$EXTRA_PUBNICS" ]; then
+        for ((i = 3; i < 3 + $EXTRA_PUBNICS; i++)); do
+          setup_interface "$i" "0.0.0.0" "255.255.255.255" $GW "force"
+        done
+      fi
+    else
+      setup_common eth0 eth1
+      if [ -n "$EXTRA_PUBNICS" ]; then
+        for ((i = 2; i < 2 + $EXTRA_PUBNICS; i++)); do
+          setup_interface "$i" "0.0.0.0" "255.255.255.255" $GW "force"
+        done
+      fi
+    fi
+    setup_sshd $ETH1_IP "eth1"
+
+  elif [ "$TYPE" == "vpcrouter" ]; then
+    setup_interface "0" $ETH0_IP $ETH0_MASK $GW
+    setup_sshd $ETH0_IP "eth0"
+
+  elif [ "$TYPE" == "ilbvm" ]; then
+    setup_common eth0 eth1
+    setup_sshd $ETH1_IP "eth1"
+
+  elif [ "$TYPE" == "elbvm" ] || [ "$TYPE" == "dhcpsrvr" ]; then
+    setup_common eth0 eth1
+    if [ "$SSHONGUEST" == "true" ]; then
+      setup_sshd $ETH0_IP "eth0"
+    else
+      setup_sshd $ETH1_IP "eth1"
+    fi
+  elif [ "$TYPE" == "cksnode" ]; then
+    setup_common eth0
+  fi
+
+  systemctl restart systemd-journald
+  # Patch known systemd/sshd memory leak - https://github.com/systemd/systemd/issues/8015#issuecomment-476160981
+  echo '@include null' >> /etc/pam.d/systemd-user
+  # Enable and Start SSH
+  systemctl enable --now --no-block ssh
+}
+
+export HYPERVISOR=$(hypervisor)
+[ $? -ne 0 ] && log_it "Failed to detect hypervisor type, bailing out" && exit 10
+log_it "Starting guest services for $HYPERVISOR"
+
+config_guest
+source /opt/cloud/bin/setup/common.sh
+setup_interface_sshd

systemvm/debian/opt/cloud/bin/setup/postinit.sh

@@ -23,17 +23,11 @@ log_it() {
   log_action_msg "$@"
 }
 
-# Eject cdrom if any
-CMDLINE=/var/cache/cloud/cmdline
-export TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
-if [ "$TYPE" != "cksnode" ]; then
-  eject || true
-fi
-
 # Restart journald for setting changes to apply
 systemctl restart systemd-journald
 
-TYPE=$(grep -Po 'type=\K[a-zA-Z]*' /var/cache/cloud/cmdline)
+CMDLINE=/var/cache/cloud/cmdline
+TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
 if [ "$TYPE" == "router" ] || [ "$TYPE" == "vpcrouter" ] || [ "$TYPE" == "dhcpsrvr" ]
 then
   if [ -x /opt/cloud/bin/update_config.py ]
@@ -71,10 +65,4 @@ then
   ip6tables-restore < $ipv6
 fi
 
-# Patch known systemd/sshd memory leak - https://github.com/systemd/systemd/issues/8015#issuecomment-476160981
-echo '@include null' >> /etc/pam.d/systemd-user
-
-# Enable and Start SSH
-systemctl enable --now --no-block ssh
-
 date > /var/cache/cloud/boot_up_done

systemvm/debian/opt/cloud/bin/setup/router.sh

@@ -43,23 +43,6 @@ setup_router() {
   oldmd5=
   [ -f "/etc/udev/rules.d/70-persistent-net.rules" ] && oldmd5=$(md5sum "/etc/udev/rules.d/70-persistent-net.rules" | awk '{print $1}')
 
-  if [ -n "$ETH2_IP" ]; then
-    setup_common eth0 eth1 eth2
-
-    if [ -n "$EXTRA_PUBNICS" ]; then
-      for ((i = 3; i < 3 + $EXTRA_PUBNICS; i++)); do
-        setup_interface "$i" "0.0.0.0" "255.255.255.255" $GW "force"
-      done
-    fi
-  else
-    setup_common eth0 eth1
-    if [ -n "$EXTRA_PUBNICS" ]; then
-      for ((i = 2; i < 2 + $EXTRA_PUBNICS; i++)); do
-        setup_interface "$i" "0.0.0.0" "255.255.255.255" $GW "force"
-      done
-    fi
-  fi
-
   log_it "Checking udev NIC assignment order changes"
   if [ "$NIC_MACS" != "" ]
   then
@@ -88,8 +71,6 @@ setup_router() {
   enable_fwding 1
   enable_rpsrfs 1
   enable_passive_ftp 1
-  cp /etc/iptables/iptables-router /etc/iptables/rules.v4
-  setup_sshd $ETH1_IP "eth1"
 
   # Only allow DNS service for current network
   sed -i "s/-A INPUT -i eth0 -p udp -m udp --dport 53 -j ACCEPT/-A INPUT -i eth0 -p udp -m udp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" /etc/iptables/rules.v4

systemvm/debian/opt/cloud/bin/setup/secstorage.sh

@@ -25,7 +25,6 @@ setup_secstorage() {
   echo "conntrackd keepalived haproxy dnsmasq" > /var/cache/cloud/disabled_svcs
   mkdir -p /var/log/cloud
 
-  setup_common eth0 eth1 eth2
   setup_storage_network
   setup_system_rfc1918_internal
 
@@ -37,14 +36,6 @@ setup_secstorage() {
   log_it "Applying iptables rules"
   cp /etc/iptables/iptables-secstorage /etc/iptables/rules.v4
 
-  log_it "Configuring sshd"
-  local hyp=$HYPERVISOR
-  if [ "$hyp" == "vmware" ] || [ "$hyp" == "hyperv" ]; then
-    setup_sshd $ETH1_IP "eth1"
-  else
-    setup_sshd $ETH0_IP "eth0"
-  fi
-
   log_it "Configuring apache2"
   setup_apache2 $ETH2_IP
 

systemvm/debian/opt/cloud/bin/setup/vpcrouter.sh

@@ -29,7 +29,6 @@ setup_vpcrouter() {
 auto lo eth0
 iface lo inet loopback
 EOF
-  setup_interface "0" $ETH0_IP $ETH0_MASK $GW
 
   echo $NAME > /etc/hostname
   echo 'AVAHI_DAEMON_DETECT_LOCAL=0' > /etc/default/avahi-daemon
@@ -86,7 +85,6 @@ EOF
   enable_fwding 1
   enable_passive_ftp 1
   cp /etc/iptables/iptables-vpcrouter /etc/iptables/rules.v4
-  setup_sshd $ETH0_IP "eth0"
   cp /etc/vpcdnsmasq.conf /etc/dnsmasq.conf
   cp /etc/cloud-nic.rules /etc/udev/rules.d/cloud-nic.rules
   echo "" > /etc/dnsmasq.d/dhcphosts.txt

systemvm/patch-sysvms.sh

@@ -0,0 +1,132 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
+backupfolder=/tmp/bkpup_live_patch
+logfile="/var/log/livepatchsystemvm.log"
+newpath="/tmp/"
+CMDLINE=/var/cache/cloud/cmdline
+md5file=/var/cache/cloud/cloud-scripts-signature
+svcfile=/var/cache/cloud/enabled_svcs
+TYPE=$(grep -Po 'type=\K[a-zA-Z]*' $CMDLINE)
+patchfailed=0
+backuprestored=0
+
+backup_old_package() {
+  mkdir -p $backupfolder
+  if [ -d /usr/local/cloud/systemvm/conf/ ]; then
+    echo "Backing up keystore file and certificates" > $logfile 2>&1
+    mkdir -p $backupfolder/conf
+    cp -r /usr/local/cloud/systemvm/conf/* $backupfolder/conf
+  fi
+  if [ -d /usr/local/cloud/systemvm/ ]; then
+    echo "Backing up agent package" >> $logfile 2>&1
+    cd /usr/local/cloud/systemvm/
+    zip -r $backupfolder/agent.zip * >> $logfile 2>&1 2>&1
+    cd -
+  fi
+  cp $md5file $backupfolder
+  echo "Backing up cloud-scripts file" >> $logfile 2>&1
+  tar -zcvf $backupfolder/cloud-scripts.tgz /etc/ /var/ /opt/ /root/  >> $logfile 2>&1
+}
+
+restore_backup() {
+  echo "Restoring cloud scripts" >> $logfile 2>&1
+  tar -xvf $backupfolder/cloud-scripts.tar -C / >> $logfile 2>&1
+  echo "Restoring agent package" >> $logfile 2>&1
+  if [ -f $backupfolder/agent.zip ]; then
+    unzip $backupfolder/agent.zip -d /usr/local/cloud/systemvm/ >> $logfile 2>&1
+    echo "Restore keystore file and certificates" >> $logfile 2>&1
+    mkdir -p "/usr/local/cloud/systemvm/conf/"
+    cp -r $backupfolder/conf/* /usr/local/cloud/systemvm/conf/
+  fi
+  backuprestored=1
+  restart_services
+  cp $backupfolder/cloud-scripts-signature $md5file
+}
+
+update_checksum() {
+  newmd5=$(md5sum $1 | awk '{print $1}')
+  echo "checksum: " ${newmd5} >> $logfile 2>&1
+  echo ${newmd5} > ${md5file}
+}
+
+restart_services() {
+  systemctl daemon-reload
+  while IFS= read -r line
+    do
+      for svc in ${line}; do
+        systemctl is-active --quiet "$svc"
+        if [ $? -eq 0 ]; then
+          systemctl restart "$svc"
+          systemctl is-active --quiet "$svc"
+          if [ $? -gt 0 ]; then
+            echo "Failed to start "$svc" service. Patch Failed. Retrying again" >> $logfile 2>&1
+            if [ $backuprestored == 0 ]; then
+              restore_backup
+            fi
+            patchfailed=1
+            break
+          fi
+        fi
+      done	
+      if [ $patchfailed == 1 ]; then
+        return
+      fi
+    done < "$svcfile"
+}
+
+cleanup_systemVM() {
+  rm -rf $backupfolder
+  mv "$newpath"cloud-scripts.tgz /usr/share/cloud/cloud-scripts.tgz
+  rm -rf "$newpath""agent.zip" "$newpath""patch-sysvms.sh"
+}
+
+patch_systemvm() {
+  rm -rf /usr/local/cloud/systemvm
+
+  if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ]; then
+    echo "All" | unzip $newpath/agent.zip -d /usr/local/cloud/systemvm >> $logfile 2>&1
+    mkdir -p /usr/local/cloud/systemvm
+    find /usr/local/cloud/systemvm/ -name \*.sh | xargs chmod 555
+  fi
+  echo "Extracting cloud scripts" >> $logfile 2>&1
+  tar -xvf $newpath/cloud-scripts.tgz -C / >> $logfile 2>&1
+
+  if [ -f $backupfolder/conf/cloud.jks ]; then
+    cp -r $backupfolder/conf/* /usr/local/cloud/systemvm/conf/
+    echo "Restored keystore file and certs using backup" >> $logfile 2>&1
+  fi
+
+  update_checksum $newpath/cloud-scripts.tgz
+
+  if [ "$TYPE" == "consoleproxy" ] || [ "$TYPE" == "secstorage" ] || [[ "$TYPE" == *router ]]; then
+    restart_services
+  fi
+}
+
+
+backup_old_package
+patch_systemvm
+cleanup_systemVM
+
+if [ $patchfailed == 0 ]; then
+  echo "version:$(cat ${md5file}) "
+fi
+
+exit $patchfailed

systemvm/pom.xml

@@ -88,6 +88,12 @@
                                         <include>agent.zip</include>
                                     </includes>
                                 </resource>
+                                <resource>
+                                    <directory>${basedir}</directory>
+                                    <includes>
+                                        <include>patch-sysvms.sh</include>
+                                    </includes>
+                                </resource>
                             </resources>
                         </configuration>
                     </execution>
@@ -138,31 +144,6 @@
                     </execution>
                 </executions>
             </plugin>
-            <plugin>
-                <groupId>org.codehaus.mojo</groupId>
-                <artifactId>exec-maven-plugin</artifactId>
-                <version>1.2.1</version>
-                <executions>
-                    <execution>
-                        <phase>package</phase>
-                        <goals>
-                            <goal>exec</goal>
-                        </goals>
-                    </execution>
-                </executions>
-                <configuration>
-                    <executable>${mkisofs}</executable>
-                    <workingDirectory>dist</workingDirectory>
-                    <arguments>
-                        <argument>-quiet</argument>
-                        <argument>-r</argument>
-                        <argument>-o</argument>
-                        <argument>systemvm.iso</argument>
-                        <argument>agent.zip</argument>
-                        <argument>cloud-scripts.tgz</argument>
-                    </arguments>
-                </configuration>
-            </plugin>
         </plugins>
     </build>
     <profiles>

tools/appliance/systemvmtemplate/scripts/configure_systemvm_services.sh

@@ -19,7 +19,7 @@
 set -e
 set -x
 
-CLOUDSTACK_RELEASE=4.16.1
+CLOUDSTACK_RELEASE=4.17.0
 
 function configure_apache2() {
    # Enable ssl, rewrite and auth
@@ -52,7 +52,6 @@ function configure_cacerts() {
   # Add LetsEncrypt ca-cert
   wget https://letsencrypt.org/certs/lets-encrypt-r3.der
   wget https://letsencrypt.org/certs/isrgrootx1.der
-
   keytool -trustcacerts -keystore /etc/ssl/certs/java/cacerts -storepass changeit -noprompt -importcert -alias letsencryptauthorityr3 -file lets-encrypt-r3.der
   keytool -trustcacerts -keystore /etc/ssl/certs/java/cacerts -storepass changeit -noprompt -importcert -alias letsencryptauthorityx1 -file isrgrootx1.der
   rm -f lets-encrypt-r3.der isrgrootx1.der
@@ -71,6 +70,7 @@ function install_cloud_scripts() {
   chmod -x /etc/systemd/system/* || true
 
   systemctl daemon-reload
+  systemctl enable cloud-preinit
   systemctl enable cloud-early-config
   systemctl enable cloud-postinit
 }

ui/public/locales/en.json

@@ -46,6 +46,7 @@
 "label.accept": "Accept",
 "label.accept.project.invitation": "Accept project invitation",
 "label.access": "Access",
+"label.acess.kubernetes.nodes": "Access Kubernetes Nodes",
 "label.accesskey": "Access Key",
 "label.acess.kubernetes.nodes": "Access Kubernetes Nodes",
 "label.account": "Account",
@@ -231,6 +232,8 @@
 "label.action.migrate.systemvm": "Migrate System VM",
 "label.action.migrate.systemvm.processing": "Migrating System VM....",
 "label.action.migrate.systemvm.to.ps": "Migrate system VM to another primary storage",
+"label.action.patch.systemvm": "Patch System VM",
+"label.action.patch.systemvm.processing": "Patching System VM....",
 "label.action.project.add.account": "Add Account to Project",
 "label.action.project.add.user": "Add User to Project",
 "label.action.reboot.instance": "Reboot Instance",
@@ -1329,6 +1332,7 @@
 "label.list.nodes": "List nodes",
 "label.list.pods": "List pods",
 "label.list.services": "List services",
+"label.livepatch": "Live Patch network's router(s)",
 "label.load.balancer": "Load Balancer",
 "label.load.balancing.policies": "Load balancing policies",
 "label.loadbalancerinstance": "Assigned VMs",
@@ -2081,6 +2085,7 @@
 "label.snmpcommunity": "SNMP Community",
 "label.snmpport": "SNMP Port",
 "label.sockettimeout": "Socket Timeout",
+"label.softwareversion": "Software Version",
 "label.source.based": "SourceBased",
 "label.source.nat.supported": "SourceNAT Supported",
 "label.sourcecidr": "Source CIDR",
@@ -2222,6 +2227,7 @@
 "label.templatesubject": "Subject",
 "label.templatetotal": "Template",
 "label.templatetype": "Template Type",
+"label.templateversion": "Template Version",
 "label.tftp.dir": "TFTP Directory",
 "label.tftpdir": "Tftp root directory",
 "label.theme.alert": "The setting is only visible to the current browser. To apply the setting, please download the JSON file and replace its content in the `theme` section of the `config.json` file under the path: `/public/config.json`",
@@ -2581,6 +2587,8 @@
 "message.action.host.enable.maintenance.mode": "Enabling maintenance mode will cause a live migration of all running instances on this host to any available host.",
 "message.action.instance.reset.password": "Please confirm that you want to change the ROOT password for this virtual machine.",
 "message.action.manage.cluster": "Please confirm that you want to manage the cluster.",
+"message.action.patch.router": "Please confirm that you want to live patch the router. <br> This operation is equivalent updating the router packages and restarting the network without cleanup.",
+"message.action.patch.systemvm": "Please confirm that you want to patch the System VM.",
 "message.action.primarystorage.enable.maintenance.mode": "Warning: placing the primary storage into maintenance mode will cause all VMs using volumes from it to be stopped.  Do you want to continue?",
 "message.action.reboot.instance": "Please confirm that you want to reboot this instance.",
 "message.action.reboot.router": "All services provided by this virtual router will be interrupted. Please confirm that you want to reboot this router.",