cloudstack/scripts/installer/export-templates.sh
Pearl Dsilva 830f3061bc
SystemVM optimizations (#5831)
* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* refactor patching logic - certs

* move logic of fixing rootdiskcontroller from upgrade to kubernetes service

* add livepatch option to restart network & vpc

* smooth upgrade of cks clusters

* Support for live patching systemVMs and deprecating systemVM.iso. Includes:
- fix systemVM template version
- Include agent.zip, cloud-scripts.tgz to the commons package
- Support for live-patching systemVMs - CPVM, SSVM, Routers
- Fix Unit test
- Remove systemvm.iso dependency

* The following commit:
- refactors logic added to support SystemVM deployment on KVM
- Adds support to copy specific files (required for patching) to the hosts on Xenserver
- Modifies vmops method - createFileInDomr to take cleanup param
- Adds configuratble sleep param to CitrixResourceBase::connect() used to verify if telnet to specifc port is possible (if sleep is 0, then default to _sleep = 10000ms)
- Adds Command/Answer for patch systemVMs on XenServer/Xcp

* - Support to patch SystemVMs - VMWare
- Remove attaching systemvm.iso to systemVMs
- Modify / Refactor VMware start command to copy patch related files to the systemvms
- cleanup

* Commit comprises of:
- remove docker from systemvm template - use containerd as container runtime
- update create-k8s-binaries script to use ctr for all docker operations
- Update userdata sent to the k8s nodes
- update cksnode script, run during patching of the cks/k8s nodes

* Add ssh to k8s nodes details in the Access tab on the UI

* test

* Refactor ca/cert patching logic

* Commit comprises of the following changes:
- Use restart network/VPC API to patch routers
- use livePatch API support patching of only cpvm/ssvm
- add timeout to the keystore setup/import script

* remove all references of systemvm.iso

* Fix keystore-cert-import invocation + refactor cert timeout in CP/SS VMs

* fix script timeout

* Refactor cert patching for systemVMs + update keystore-cert-import script + patch-sysvms script + remove patchSysvmCommand from networkelementcommand

* remove commented code + change core user to cloud for cks nodes

* Update ownership of ssh directory

* NEED TO DISCUSS - add on the fly template conversion as an ExecStartPre action (systemd)

* Add UI changes + move changes from patch file to runcmd

* test: validate performance for template modification during seeding

* create vms folder in cloudstack-commons directory - debian rules

* remove logic for on the fly template convert + update k8s test

* fix syntax issue - causing issue with shared network tests

* Code cleanup

* add cgroup config for containerd

* add systemd config for kubelet

* add additional info during image registry config

* address comments

* add temp links of download.cloudstack.org

* address part of the comments

* address comments

* update containerd config - as version has upgraded to 1.5 from 1.4.12 in 4.17.0

* address comments - simplify

* fix vue3 related icon changes

* allow network commands when router template version is lower but is patched

* add internal LB to the list of routers to be patched on network restart with live patch

* add unit tests for API param validations and new helper utilities - file scp & checksum validations

* perform patching only for non-user i.e., system VMs

* add test to validate params

* remove unused import

* add column to domain_router to display software version and support networkrestart with livePatch from router view

* Requires upgrade column to consider package (cloud-scripts) checksum to identify if true/false

* use router software version instead of checksum

* show N/A if no software version reported i.e., in upgraded envs

* fix deb failure

* update pom to official links of systemVM template
2022-04-21 13:40:19 -03:00

193 lines
8.3 KiB
Bash

#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
METADATA_FILE="metadata.ini"
IMAGE_PATH=${3:-"/usr/share/cloudstack-management/templates/systemvm/"}
TEMPLATE_VERSION=$(awk -F "=" '/version/ {print $2}' ${IMAGE_PATH}${METADATA_FILE} | xargs)
TEMPLATE_PATH="/usr/share/cloudstack-management/templates/systemvm/"
VERSION="${TEMPLATE_VERSION%.*}"
PREFIX=${4:-"systemvmtemplate-$VERSION"}
CLEANUP=${2:-1}
TEMP_IMAGE_PATH="/tmp/sysvm_convert/"
initial_setup() {
mkdir -p $TEMP_IMAGE_PATH
cp -r $IMAGE_PATH/* $TEMP_IMAGE_PATH
cd $TEMP_IMAGE_PATH
if [ ! -f ${TEMP_IMAGE_PATH}${PREFIX}-kvm.qcow2 ]; then
bzip2 -dc $PREFIX-kvm.qcow2.bz2 > $PREFIX-kvm.qcow2
fi
}
export_vmware() {
initial_setup
# Export for KVM
virt-sparsify $PREFIX-kvm.qcow2 --compress -o compat=0.10 $PREFIX-kvm-temp.qcow2
# Export for VMware
qemu-img convert -f qcow2 -O vmdk -o adapter_type=lsilogic,subformat=streamOptimized,compat6 $PREFIX-kvm-temp.qcow2 $PREFIX-vmware.vmdk
size=$(stat --printf="%s" $PREFIX-vmware.vmdk)
cat <<EOF > $PREFIX-vmware.ovf
<?xml version="1.0" encoding="UTF-8"?>
<!--Generated by VMware ovftool 4.1.0 (build-2459827), UTC time: 2021-11-24T18:59:40.381083Z-->
<Envelope vmw:buildId="build-2459827" xmlns="http://schemas.dmtf.org/ovf/envelope/1" xmlns:cim="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<References>
<File ovf:href="$PREFIX-vmware.vmdk" ovf:id="file1" ovf:size="$size"/>
</References>
<DiskSection>
<Info>Virtual disk information</Info>
<Disk ovf:capacity="5" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>
</DiskSection>
<VirtualSystem ovf:id="vm">
<Info>A virtual machine</Info>
<Name>$PREFIX-vmware</Name>
<OperatingSystemSection ovf:id="101" vmw:osType="otherLinux64Guest">
<Info>The kind of installed guest operating system</Info>
</OperatingSystemSection>
<VirtualHardwareSection>
<Info>Virtual hardware requirements</Info>
<System>
<vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
<vssd:InstanceID>0</vssd:InstanceID>
<vssd:VirtualSystemIdentifier>$PREFIX-vmware</vssd:VirtualSystemIdentifier>
<vssd:VirtualSystemType>vmx-11</vssd:VirtualSystemType>
</System>
<Item>
<rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits>
<rasd:Description>Number of Virtual CPUs</rasd:Description>
<rasd:ElementName>1 virtual CPU(s)</rasd:ElementName>
<rasd:InstanceID>1</rasd:InstanceID>
<rasd:ResourceType>3</rasd:ResourceType>
<rasd:VirtualQuantity>1</rasd:VirtualQuantity>
</Item>
<Item>
<rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits>
<rasd:Description>Memory Size</rasd:Description>
<rasd:ElementName>256MB of memory</rasd:ElementName>
<rasd:InstanceID>2</rasd:InstanceID>
<rasd:ResourceType>4</rasd:ResourceType>
<rasd:VirtualQuantity>256</rasd:VirtualQuantity>
</Item>
<Item>
<rasd:Address>0</rasd:Address>
<rasd:Description>SCSI Controller</rasd:Description>
<rasd:ElementName>scsiController0</rasd:ElementName>
<rasd:InstanceID>3</rasd:InstanceID>
<rasd:ResourceSubType>lsilogic</rasd:ResourceSubType>
<rasd:ResourceType>6</rasd:ResourceType>
</Item>
<Item>
<rasd:Address>0</rasd:Address>
<rasd:Description>IDE Controller</rasd:Description>
<rasd:ElementName>ideController0</rasd:ElementName>
<rasd:InstanceID>4</rasd:InstanceID>
<rasd:ResourceType>5</rasd:ResourceType>
</Item>
<Item ovf:required="false">
<rasd:AddressOnParent>0</rasd:AddressOnParent>
<rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
<rasd:ElementName>cdrom0</rasd:ElementName>
<rasd:InstanceID>5</rasd:InstanceID>
<rasd:Parent>4</rasd:Parent>
<rasd:ResourceType>15</rasd:ResourceType>
</Item>
<Item>
<rasd:AddressOnParent>0</rasd:AddressOnParent>
<rasd:ElementName>disk0</rasd:ElementName>
<rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
<rasd:InstanceID>6</rasd:InstanceID>
<rasd:Parent>3</rasd:Parent>
<rasd:ResourceType>17</rasd:ResourceType>
</Item>
<Item ovf:required="false">
<rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
<rasd:ElementName>video</rasd:ElementName>
<rasd:InstanceID>7</rasd:InstanceID>
<rasd:ResourceType>24</rasd:ResourceType>
<vmw:Config ovf:required="false" vmw:key="enable3DSupport" vmw:value="false"/>
<vmw:Config ovf:required="false" vmw:key="useAutoDetect" vmw:value="false"/>
<vmw:Config ovf:required="false" vmw:key="videoRamSizeInKB" vmw:value="4096"/>
</Item>
<Item ovf:required="false">
<rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
<rasd:ElementName>vmci</rasd:ElementName>
<rasd:InstanceID>8</rasd:InstanceID>
<rasd:ResourceSubType>vmware.vmci</rasd:ResourceSubType>
<rasd:ResourceType>1</rasd:ResourceType>
</Item>
<vmw:Config ovf:required="false" vmw:key="cpuHotAddEnabled" vmw:value="false"/>
<vmw:Config ovf:required="false" vmw:key="cpuHotRemoveEnabled" vmw:value="false"/>
<vmw:Config ovf:required="false" vmw:key="firmware" vmw:value="bios"/>
<vmw:Config ovf:required="false" vmw:key="memoryHotAddEnabled" vmw:value="false"/>
</VirtualHardwareSection>
<AnnotationSection ovf:required="false">
<Info>A human-readable annotation</Info>
<Annotation>$PREFIX-vmware</Annotation>
</AnnotationSection>
</VirtualSystem>
</Envelope>
EOF
cat <<EOF > $PREFIX-vmware.mf
SHA1($PREFIX-vmware.ovf)= $(sha1sum $PREFIX-vmware.ovf|awk '{print $1}')
SHA1($PREFIX-vmware.vmdk)= $(sha1sum $PREFIX-vmware.vmdk |awk '{print $1}')
EOF
tar -cvf $PREFIX-vmware.ova $PREFIX-vmware.ovf $PREFIX-vmware.mf $PREFIX-vmware.vmdk
checksum=$(md5sum $PREFIX-vmware.ova | awk '{print $1}')
sed -i '/^\['"vmware"']/,/^\[/{s/^checksum[[:space:]]*=.*/checksum = '"$checksum"'/}' ./$METADATA_FILE
rm -rf *.mf *.ovf *.vmdk
sudo cp $TEMP_IMAGE_PATH/$PREFIX-vmware.ova $TEMP_IMAGE_PATH/metadata.ini $IMAGE_PATH
cleanup
}
export_xen() {
# Export for XenServer/XCP-ng
initial_setup
qemu-img convert -f qcow2 -O vpc $PREFIX-kvm.qcow2 $PREFIX-xen.vhd
bzip2 $PREFIX-xen.vhd
checksum=$(md5sum $PREFIX-xen.vhd.bz2 | awk '{print $1}')
sed -i '/^\['"xenserver"']/,/^\[/{s/^checksum[[:space:]]*=.*/checksum = '"$checksum"'/}' $METADATA_FILE
rm -rf $PREFIX-xen.vhd
sudo cp $TEMP_IMAGE_PATH/$PREFIX-xen* $TEMP_IMAGE_PATH/metadata.ini $IMAGE_PATH
cleanup
}
cleanup() {
cd /var/cloudstack/management/
if [ $CLEANUP == 1 ]; then
cd /var/cloudstack/management/
rm -rf $TEMP_IMAGE_PATH
fi
}
if [ "$#" -lt 1 ] ; then
echo "Usage: $0 <hypervisor: vmware/xenserver> [cleanup: 0/1; default: 1] [imagepath: default:/usr/share/cloudstack-management/templates/systemvm/] [templateprefix: default:systemvmtemplate-$VERSION]" >&2
exit 1
fi
if [ $1 == "vmware" ]; then
echo "exporting vmware template"
export_vmware
elif [ $1 == "xenserver" ]; then
echo "exporting xenserver template"
export_xen
else
echo "Conversion of template to $1's compatible format not supported "
fi