vyos/vyos-build
1
0
Fork 0
mirror of https://github.com/vyos/vyos-build.git synced 2025-10-01 20:28:40 +02:00
Code Issues Packages Projects Releases Wiki Activity

build-packages: remove obsolete script and link to VyOS documentation

Browse Source 
The referenced scripts/build-packages script was a proof of concept from me
some time ago to build VyOS packages through a more or less tiny wrapper. The
script was not maintained at any time and randomly caused more harm then good.

The script has been abandoned and the documentation has been updated on how to
build individual VyOS packages.
This commit is contained in:
Christian Poessinger 2020-08-03 15:10:32 +02:00
parent 2e0d072fdd
commit f0f368e80d
3 changed files with 17 additions and 580 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,12 @@
# Contributing to VyOS
You wan't to help us improve VyOS? This is awesome. We accept any kind of Pull
Requests on GitHub. To make the life of the maintainers and you as future
contributor (or maybe maintainer) much easier we have come up with some basic
rules. Instead of copy/pasting or maintaining two instances of how to contribute
to VyOS you can find the entire process documented in our online documentation:
https://docs.vyos.io/en/latest/contributing/development.html
Also this guide might not be complete so any PR is much appreciated.
It might also worth browsing our blog: https://blog.vyos.io

215
README.md
View File

@ -47,218 +47,13 @@ There are several directories with their own purpose:
purposes, but not during ISO build process purposes, but not during ISO build process
* `vars/` Jenkins Pipeline library for reusable functions * `vars/` Jenkins Pipeline library for reusable functions
# Building images using Docker # Building VyOS
Using our [Dockerfile](docker/Dockerfile) you create your own Docker container In order to have a single manual and not maintining multiple copies the
that is used to build a VyOS ISO image or other required VyOS packages. The instructions on how to build VyOS either in a Docker container or natively can
[Dockerfile](docker/Dockerfile) contains some of the most used packages needed be found in our [Documentation - Build VyOS](https://docs.vyos.io/en/latest/contributing/build-vyos.html).
to build a VyOS ISO, a qemu image, and several of the submodules.
To build the docker image ensure you have a working [Docker](https://www.docker.com) # Development Branches
environment and then run the following commands:
```bash
$ docker build -t vyos-builder docker
```
Run newly built container:
```bash
$ docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos-builder bash
```
This will drop you into a bash shell with this vyos-build repo mounted at
`/vyos`. Then follow the instructions bellow to build the VyOS ISO and QEMU
image.
```bash
vyos_bld@948a2be7c52c:/vyos$ uname -a
Linux 948a2be7c52c 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03) x86_64 GNU/Linux
```
**NOTE:**
* Docker container must be run with `--privileged` flag
* We recommended to run the container with a volume mapped in order to easy
export built VyOS ISO images to the "external" world
* UNIX ownership is automatically inherited from your host directory but can be
altered by specifying the following environment variables when running the
container: `-e GOSU_UID=$(id -u)` and/or `-e GOSU_GID=$(id -g)`
After the Docker container is running you can follow up the instructions below in
order to build the VyOS ISO image.
## Building the ISO image
The `./configure` script will warn you if any dependencies are missing. Individual
packages may have other build dependencies. If some dependencies are missing,
package build scripts will tell you.
```bash
$ ./configure --help
usage: configure [-h] [--build-by BUILD_BY] [--version VERSION]
[--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR]
[--debian-security-mirror DEBIAN_SECURITY_MIRROR]
[--architecture ARCHITECTURE] [--vyos-mirror VYOS_MIRROR]
[--build-type BUILD_TYPE] [--debian-mirror DEBIAN_MIRROR]
[--debug] [--custom-apt-entry CUSTOM_APT_ENTRY]
[--custom-apt-key CUSTOM_APT_KEY]
[--custom-package CUSTOM_PACKAGE]
optional arguments:
-h, --help show this help message and exit
--build-by BUILD_BY Builder identifier (e.g. jrandomhacker@example.net)
--version VERSION Version number (release builds only)
--pbuilder-debian-mirror PBUILDER_DEBIAN_MIRROR
Debian repository mirror for pbuilder env bootstrap
--debian-security-mirror DEBIAN_SECURITY_MIRROR
Debian security updated mirror
--architecture ARCHITECTURE
Image target architecture (amd64 or i386 or armhf)
--vyos-mirror VYOS_MIRROR
VyOS package mirror
--build-type BUILD_TYPE
Build type, release or development
--debian-mirror DEBIAN_MIRROR
Debian repository mirror for ISO build
--debug Enable debug output
--custom-apt-entry CUSTOM_APT_ENTRY
Custom APT entry
--custom-apt-key CUSTOM_APT_KEY
Custom APT key file
--custom-package CUSTOM_PACKAGE
Custom package to install from repositories
```
Before you can build an image, you need to configure your build.
Each build needs to run the `./configure` step first where you can extend your
ISO by additional packages (`--custom-package`) or specify who build this nice
ISO image (`--build-by`). If you have local Debian mirrors, you can select them
by `--debian-mirror` or `--debian-security-mirror`.
```bash
$ ./configure --custom-package vim --build-by jrandom@hacker.com
$ sudo make iso
```
After some time you will find the resulting ISO image in the `build` folder.
### Building images for virtualization platforms
#### QEMU
Run following command after building the ISO image.
```bash
$ make qemu
```
#### VMware
Run following command after building the QEMU image.
```bash
$ make vmware
```
## Building subpackages inside Docker
VyOS requires a bunch of packages which are VyOS specific and thus can not be
found in any Debian Upstream mirrror. Those packages can be found at the VyOS
GitHub project (https://github.com/vyos) and there is a nice helper script
available to build and list those individual packages.
[scripts/build-packages](scripts/build-packages) provides an easy interface
to automate the process of building all VyOS related packages that are not part
of the upstream Debian version. Execute it in the root of the `vyos-build`
directory to start compilation.
```bash
$ scripts/build-packages -h
usage: build-packages [-h] [-c | -k | -f] [-v] [-l] [-b BUILD [BUILD ...]]
[-p] [--blacklist BLACKLIST [BLACKLIST ...]]
optional arguments:
-h, --help show this help message and exit
-c, --clean Re-clone required Git repositories
-k, --keep Keep modified Git repositories
-f, --fetch Fetch sources only, no build
-v, --verbose Increase logging verbosity for each occurance
-l, --list-packages List all packages to build
-b BUILD [BUILD ...], --build BUILD [BUILD ...]
Whitespace separated list of packages to build
-p, --parallel Build on all CPUs
--blacklist BLACKLIST [BLACKLIST ...]
Do not build/report packages when calling --list
```
Git repositories are automatically fetched and build on demand. If you want to
work offline you can fetch all source code first with the `-f` option.
The easiest way to compile is with the above mentioned [Docker](docker/Dockerfile)
container, it includes all dependencies for compiling supported packages.
```bash
$ docker run --rm -it -v $(pwd):/vyos -w /vyos \
--sysctl net.ipv6.conf.lo.disable_ipv6=0 \
vyos-builder scripts/build-packages
```
**NOTE:** `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is required to build the
`vyos-strongswan` package
**NOTE:** Prior to executing this script you need to create or build the Docker
container and checkout all packages you want to compile.
### Building single package(s)
To build a single package use the same script as above but specify packages with `-b`:
Executed from the root of `vyos-build`
```bash
$ docker run --rm -it -v $(pwd):/vyos -w /vyos \
--sysctl net.ipv6.conf.lo.disable_ipv6=0 \
vyos-builder scripts/build-packages -b <package>
```
**NOTE:** `--sysctl net.ipv6.conf.lo.disable_ipv6=0` is only needed when
building `vyos-strongswan` and can be ignored on other packages.
**NOTE:** `vyos-strongswan` will only compile on a Linux system, running on macOS
or Windows might result in a unittest deadlock (it never exits).
#### Building single packages from your own repositories
You can also build packages that are not from the default git repositories,
for example from your own forks of the official vyos repositories.
First create a directory "packages" at the top level of the vyos-build repository
and clone your package into it (creating a subdirectory with the package contents).
Then checkout the correct branch or commit you want to build before building the package.
Example using `git@github.com:myname/vyos-1x.git` repository to build vyos-1x:
```bash
$ mkdir packages
$ cd packages
$ git clone git@github.com:myname/vyos-1x.git
$ cd ..
$ docker run --rm -it -v $(pwd):/vyos -w /vyos \
--sysctl net.ipv6.conf.lo.disable_ipv6=0 \
vyos-builder scripts/build-packages -b vyos-1x
```
**NOTE:** You need to git pull manually after you commit to the remote and before rebuilding,
the local repository won't be updated automatically.
**WARNING:** Any packages in the packages directory will be added to the iso during build,
replacing the upstream ones. Make sure you delete them (both the source directories and built
deb packages) if you want to build an iso from purely upstream packages.
# Development process
## Git branches
The default branch that contains the most recent VyOS code is called `current` The default branch that contains the most recent VyOS code is called `current`
rather than `master`. We know it's confusing, but it's not easy to fix. In a rather than `master`. We know it's confusing, but it's not easy to fix. In a

370
scripts/build-packages
View File

@ -1,370 +0,0 @@
#!/usr/bin/env python3
import os
import subprocess
import sys
import shutil
import argparse
import logging
current_working_directory = os.getcwd()
repo_root = subprocess.check_output('git rev-parse --show-toplevel', shell=True, universal_newlines=True).rstrip('\n')
repo_sha = subprocess.check_output('git rev-parse --short=12 HEAD', shell=True, universal_newlines=True).rstrip('\n')
def add_package(name, url=None, commit=None, branch='current', tag=None, shallow=True, custombuild_cmd=None):
"""
Build up source package with URL and build commands executed during the later
called build_package step.
If no additional information is passed we will use the latest commit from current
branch
If no URL is passed we assume it's a regular VyOS package from the VyOS Github
namespace at https://github.com/vyos
"""
if not url:
url = 'https://github.com/vyos/' + name + '.git'
package = {
'name': name,
'url': url,
'commit': commit,
'tag': tag,
'branch': branch,
'shallow': shallow,
'path': repo_root + '/packages/' + name,
'custombuild_cmd': custombuild_cmd
}
return package
def call(bashCommand, log, **kwargs):
"""
Run command with arguments. Wait for command to complete.
Send output to logging module passed as 'log'.
"""
from subprocess import Popen, PIPE, STDOUT, check_output, CalledProcessError
from tempfile import TemporaryFile
from time import sleep
log.debug("Executing '{}'".format(bashCommand))
# code borrowsed from:
# https://stackoverflow.com/questions/38374063/python-can-we-use-tempfile-with-subprocess-to-get-non-buffering-live-output-in-p
# the temp file will be automatically cleaned up
output = TemporaryFile()
error = TemporaryFile()
kwargs['stdout'] = output
kwargs['stderr'] = error
kwargs['shell'] = True
kwargs['universal_newlines'] = True
sub = Popen(bashCommand, **kwargs)
while sub.poll() is None:
where = output.tell()
lines = output.readline()
if not lines:
sleep(0.3)
output.seek(where)
else:
log.debug(lines.decode().rstrip('\n'))
where = error.tell()
lines = error.readline()
if not lines:
sleep(0.3)
error.seek(where)
else:
log.info(lines.decode().rstrip('\n'))
# A last write needed after subprocess ends
log.debug(output.read().decode().rstrip('\n'))
log.info(error.read().decode().rstrip('\n'))
error.close()
output.close()
return sub.returncode
def clone_package(pkg, log):
"""
Clone Git repository from URL embedded in pkg to local disk
First cleanup any possible leftovers from previous builds
"""
if args.keep and os.path.isdir(pkg['path']):
log.debug("Keep possibly modified package '{}'".format(pkg['path']))
return False
elif args.clean:
# delete repository from disk
if os.path.isdir(pkg['path']):
log.debug("Cleaning '{}'".format(pkg['path']))
shutil.rmtree(pkg['path'])
else:
if os.path.isdir(pkg['path']):
# Change current directory into Git repo for this package
os.chdir(pkg['path'])
bashCommand = 'git clean -d -x --force && git reset --hard ' + pkg['commit']
return call(bashCommand, log)
if pkg['commit'] and pkg['shallow']:
log.debug("Package '{}' has both commit and shallow set, unsetting shallow to do a full clone.".format(pkg['name']))
pkg['shallow']=False
bashCommand = 'git clone ' + pkg['url']
if pkg['shallow']:
bashCommand += ' --depth 1'
if pkg['tag']:
bashCommand += ' --branch ' + pkg['tag']
elif pkg['branch']:
bashCommand += ' --branch ' + pkg['branch']
bashCommand += ' ' + pkg['path']
if pkg['commit']:
bashCommand += ' && cd ' + pkg['path'] + ' && git reset --hard ' + pkg['commit']
return call(bashCommand, log)
def build_package(pkg, log=None):
"""
Generate Debian package from passed 'pkg'
"""
# Change current directory into Git repo for this package
os.chdir(pkg['path'])
# Overwrite custom build command if required, e.g. libyang
bashCommand = ''
if pkg['custombuild_cmd']:
bashCommand = pkg['custombuild_cmd']
else:
# Build package
bashCommand = 'dpkg-buildpackage -uc -us -tc -b'
if args.parallel:
bashCommand += ' -j' + str(os.cpu_count())
return call(bashCommand, log)
# a List of all Vyatta/VyOS based packages
vyos_packages = ['vyatta-bash',
'vyatta-cfg',
'vyatta-op',
'vyatta-cfg-system',
'vyatta-cfg-firewall',
'vyatta-op-firewall',
'vyatta-cfg-vpn',
'vyatta-op-vpn',
'vyatta-cfg-qos',
'vyatta-op-qos',
'vyatta-cfg-op-pppoe',
'vyatta-nat',
'vyatta-config-mgmt',
'vyatta-config-migrate',
'vyatta-zone',
'vyatta-cluster',
'vyatta-eventwatch',
'vyatta-webproxy',
'vyatta-cfg-quagga',
'vyatta-op-quagga',
'vyatta-wanloadbalance',
'vyatta-ravpn',
'vyos-nhrp',
'vyos-world',
'vyatta-iproute',
'vyatta-biosdevname',
'vyos-opennhrp',
'udp-broadcast-relay',
'vyos-1x',
'vyatta-conntrack',
'vyatta-conntrack-sync',
'vyos-xe-guest-utilities',
'vyos-netplug',
'ddclient',
'live-boot']
# Special packages mean packages which are located no in the VyOS namespace
# or require fancy build instructions
pkg_special = []
# libvyosconfig/ipaddrcheck uses a different default branch
libvyosconfig_build_cmd = "eval $(opam env --root=/opt/opam --set-root) && " \
"dpkg-buildpackage -b -us -uc -tc"
pkg_special.append( add_package('libvyosconfig', branch='master', custombuild_cmd=libvyosconfig_build_cmd))
pkg_special.append( add_package('ipaddrcheck', branch='master'))
# Packages where we directly build the upstream source
pkg_special.append( add_package('hvinfo', url='https://github.com/dmbaturin/hvinfo.git', branch='master') )
# VyOS strongswan ships additional python3-vici packages required by vyos-1x and this is not build by default
vyos_strongswan_build_cmd = "dpkg-buildpackage -b -us -uc -tc && " \
"autoreconf -i && ./configure --enable-python-eggs && " \
"cd src/libcharon/plugins/vici/python && make && " \
"python3 setup.py --command-packages=stdeb.command bdist_deb && " \
"mv ./deb_dist/*.deb " + repo_root + "/packages"
pkg_special.append( add_package('vyos-strongswan', custombuild_cmd=vyos_strongswan_build_cmd) )
#
# FreeRangeRouting (FRR) packages
#
pkg_special.append( add_package('rtrlib', url='https://github.com/rtrlib/rtrlib.git', branch='master', tag='v0.6.3') )
frr_build_cmd = './tools/tarsource.sh -V && dpkg-buildpackage -us -uc -Ppkg.frr.rtrlib -d'
pkg_special.append( add_package('frr', url='https://github.com/FRRouting/frr.git', branch='master', tag='frr-7.0', custombuild_cmd=frr_build_cmd) )
#libyang_build_cmd = 'mkdir build && cd build && cmake .. && make build-deb && mv debs/* ' + repo_root + '/packages'
#pkg_special.append( add_package('libyang', url='https://github.com/opensourcerouting/libyang.git', commit='179da47', branch='master', custombuild_cmd=libyang_build_cmd) )
#
# We use keepalived from Debian Buster
#
pkg_special.append( add_package('keepalived', url='https://salsa.debian.org/ipvs-team/pkg-keepalived.git', branch='master', commit='eae91c81'))
#
# Linux (VyOS) Kernel
#
kernel_build_cmd = "make x86_64_vyos_defconfig && " \
"sed -i 's/\"kernel_version\": \"[0-9].[0-9][0-9].[0-9]*\"/\"kernel_version\": \"'$(make kernelversion)'\"/' " + repo_root + "/data/defaults.json && " \
"make bindeb-pkg LOCALVERSION='-amd64-vyos' KDEB_PKGVERSION=$(make kernelversion)-1 -j $(getconf _NPROCESSORS_ONLN)"
pkg_special.append( add_package('vyos-kernel', branch='linux-vyos-4.19.y', custombuild_cmd=kernel_build_cmd) )
#
# Accell-PPP Package and Kernel Module
#
accel_ppp_build_cmd = "echo 'lib/modules/'$(cat " + repo_root + "/data/defaults.json | jq '.kernel_version' | tr -d \\\")-amd64-vyos/extra/*.ko > debian/vyos-accel-ppp-ipoe-kmod.install && " \
"sed -i 's#[0-9].[0-9][0-9].[0-9]*-amd64-vyos#'$(cat " + repo_root + "/data/defaults.json | jq '.kernel_version' | tr -d \\\")'-amd64-vyos#g' debian/rules && " \
"KERNELDIR=" + repo_root + "/packages/vyos-kernel dpkg-buildpackage -b -us -uc -tc -j" + str(os.cpu_count())
pkg_special.append( add_package('vyos-accel-ppp', custombuild_cmd=accel_ppp_build_cmd) )
#
# mdns-repeater branch for current is master
#
pkg_special.append( add_package('mdns-repeater', branch='master') )
# A list of all packages we will build in the end
pkg_build = []
if __name__ == '__main__':
parser = argparse.ArgumentParser()
exclusive = parser.add_mutually_exclusive_group(required=False)
exclusive.add_argument('-c', '--clean', action='store_true', help='Re-clone required Git repositories')
exclusive.add_argument('-k', '--keep', action='store_true', help='Keep modified Git repositories')
exclusive.add_argument('-f', '--fetch', action='store_true', help='Fetch sources only, no build')
parser.add_argument('-v', '--verbose', action='count', default=0, help='Increase logging verbosity for each occurance')
parser.add_argument('-l', '--list-packages', action='store_true', help='List all packages to build')
parser.add_argument('-b', '--build', nargs='+', help='Whitespace separated list of packages to build')
parser.add_argument('-p', '--parallel', action='store_true', help='Build on all CPUs')
parser.add_argument('--blacklist', nargs='+', help='Do not build/report packages when calling --list')
args = parser.parse_args()
levels = [ logging.INFO, logging.WARNING, logging.DEBUG ]
level = levels[min(len(levels)-1,args.verbose)] # capped to number of levels
logging.basicConfig(level=level, format="%(asctime)s %(name)s %(message)s")
print("Using vyos-build repository ('{}') commit '{}'\n".format(repo_root, repo_sha))
#
# Exclude packages from build process,
# also do not list them on --list
#
if args.blacklist:
for exclude in args.blacklist:
if exclude in vyos_packages:
vyos_packages.remove(exclude)
continue
found = False
for pkg in pkg_special:
if exclude == pkg['name']:
found = True
# package already formed
pkg_special.remove(pkg)
break
if not found:
print("Invalid choice '" + exclude + "', --list-packages for complete list!")
sys.exit(1)
#
# List all available (to be build) packages
#
if args.list_packages:
print("Individual packages available for build:")
for pkg in vyos_packages:
print(' * ' + pkg)
for pkg in pkg_special:
print(' * ' + pkg['name'])
sys.exit(0)
#
# Only add selective packages to the build list
#
if args.build:
# NOTE: remove double added packages from list
for target in args.build:
if target in vyos_packages:
pkg_build.append(add_package( target ))
continue
found = False
for pkg in pkg_special:
if target == pkg['name']:
found = True
# package already formed
pkg_build.append( pkg )
break
if not found:
print("Invalid choice '" + target + "', for -b/--build use --list-packages for complete list!")
sys.exit(1)
else:
# Add all VyOS packages to the package list
for pkg in vyos_packages:
pkg_build.append(add_package( pkg ))
# We also wan't to build all of our special packages
for pkg in pkg_special:
pkg_build.append( pkg )
# Build all VyOS packages (packages found on https://github.com/vyos
# and referenced in vyos_packages)
for pkg in pkg_build:
# Create a logging instance per package
log = logging.getLogger(pkg['name'])
ret = clone_package(pkg, log)
if ret:
log.error("ERROR cloning source")
sys.exit(1)
else:
# only build packages if fetch flag is not set
if not args.fetch:
ret = build_package(pkg, log)
if ret:
log.error("ERROR building source")
sys.exit(1)
sys.exit(0)