mirror of
https://github.com/vyos/vyos-documentation.git
synced 2025-10-26 08:41:46 +01:00
testing: DOS -> UNIX line endings
This commit is contained in:
Christian Poessinger
parent
cdbf5a45a5
commit
eee5da4e8b
414
docs/testing.rst
414
docs/testing.rst
@ -1,207 +1,207 @@
|
|||||||
.. _testing:
|
.. _testing:
|
||||||
|
|
||||||
#######
|
#######
|
||||||
Testing
|
Testing
|
||||||
#######
|
#######
|
||||||
|
|
||||||
One of the major advantages introduced in VyOS 1.3 is an autmated test framework.
|
One of the major advantages introduced in VyOS 1.3 is an autmated test framework.
|
||||||
When assembling an ISO image multiple things can go wrong badly and publishing
|
When assembling an ISO image multiple things can go wrong badly and publishing
|
||||||
a faulty ISO makes no sense. The user is disappointed by the quality of the image
|
a faulty ISO makes no sense. The user is disappointed by the quality of the image
|
||||||
and the developers get flodded with bug reports over and over again.
|
and the developers get flodded with bug reports over and over again.
|
||||||
|
|
||||||
As the VyOS documentation is not only for users but also for the developers -
|
As the VyOS documentation is not only for users but also for the developers -
|
||||||
and we keep no secret documentation - this section describes how the automated
|
and we keep no secret documentation - this section describes how the automated
|
||||||
testing works.
|
testing works.
|
||||||
|
|
||||||
Jenkins CI
|
Jenkins CI
|
||||||
==========
|
==========
|
||||||
|
|
||||||
Our `VyOS CI`_ system is based on Jenkins and builds all our required packages
|
Our `VyOS CI`_ system is based on Jenkins and builds all our required packages
|
||||||
for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build
|
for VyOS 1.2 to 1.4. In addition to the package build, there is the vyos-build
|
||||||
Job which builds and tests the VyOS ISO image which is published after a
|
Job which builds and tests the VyOS ISO image which is published after a
|
||||||
successfull test drive.
|
successfull test drive.
|
||||||
|
|
||||||
We differentiate in two independent tests, which are both run in parallel by
|
We differentiate in two independent tests, which are both run in parallel by
|
||||||
two separate QEmu instances which are launched via ``make test`` and ``make
|
two separate QEmu instances which are launched via ``make test`` and ``make
|
||||||
testc`` from within the vyos-build_ repository.
|
testc`` from within the vyos-build_ repository.
|
||||||
|
|
||||||
Smoketests
|
Smoketests
|
||||||
==========
|
==========
|
||||||
|
|
||||||
Smoketests executes predefined VyOS CLI commands and checks if the desired
|
Smoketests executes predefined VyOS CLI commands and checks if the desired
|
||||||
daemon/service configuration is rendert - that is how to put it "short".
|
daemon/service configuration is rendert - that is how to put it "short".
|
||||||
|
|
||||||
When and ISO image is assembled by the `VyOS CI`_, the ``BUILD_SMOKETEST``
|
When and ISO image is assembled by the `VyOS CI`_, the ``BUILD_SMOKETEST``
|
||||||
parameter is enabled by default, which will extend the ISO configuration line
|
parameter is enabled by default, which will extend the ISO configuration line
|
||||||
with the following packages:
|
with the following packages:
|
||||||
|
|
||||||
.. code-block:: python
|
.. code-block:: python
|
||||||
|
|
||||||
def CUSTOM_PACKAGES = ''
|
def CUSTOM_PACKAGES = ''
|
||||||
if (params.BUILD_SMOKETESTS)
|
if (params.BUILD_SMOKETESTS)
|
||||||
CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest'
|
CUSTOM_PACKAGES = '--custom-package vyos-1x-smoketest'
|
||||||
|
|
||||||
So if you plan to build your own custom ISO image and wan't to make use of our
|
So if you plan to build your own custom ISO image and wan't to make use of our
|
||||||
smoketests, ensure that you have the `vyos-1x-smoketest` package installed.
|
smoketests, ensure that you have the `vyos-1x-smoketest` package installed.
|
||||||
|
|
||||||
The ``make test`` command from the vyos-build_ repository will launch a new
|
The ``make test`` command from the vyos-build_ repository will launch a new
|
||||||
QEmu instance and the ISO image is first installed to the virtual harddisk.
|
QEmu instance and the ISO image is first installed to the virtual harddisk.
|
||||||
|
|
||||||
After its first boot into the newly installed system the main Smoketest script
|
After its first boot into the newly installed system the main Smoketest script
|
||||||
is executed, it can be found here: `/usr/bin/vyos-smoketest`
|
is executed, it can be found here: `/usr/bin/vyos-smoketest`
|
||||||
|
|
||||||
The script only searches for executable "test-cases" under
|
The script only searches for executable "test-cases" under
|
||||||
``/usr/libexec/vyos/tests/smoke/cli/`` and executes them one by one.
|
``/usr/libexec/vyos/tests/smoke/cli/`` and executes them one by one.
|
||||||
|
|
||||||
.. note:: As Smoketests will alter the system configuration and you are logged
|
.. note:: As Smoketests will alter the system configuration and you are logged
|
||||||
in remote you may loose your connection to the system.
|
in remote you may loose your connection to the system.
|
||||||
|
|
||||||
Manual Smoketest Run
|
Manual Smoketest Run
|
||||||
--------------------
|
--------------------
|
||||||
|
|
||||||
On the other hand - as each test is contain in its own file - one can always
|
On the other hand - as each test is contain in its own file - one can always
|
||||||
execute a single Smoketest by hand by simply running the Python test scripts.
|
execute a single Smoketest by hand by simply running the Python test scripts.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
|
|
||||||
.. code-block:: none
|
.. code-block:: none
|
||||||
|
|
||||||
vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
|
vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
|
||||||
test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok
|
test_bgp_01_simple (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok
|
test_bgp_02_neighbors (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok
|
test_bgp_03_peer_groups (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok
|
test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok
|
test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok
|
test_bgp_06_listen_range (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok
|
test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok
|
test_bgp_08_zebra_route_map (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok
|
test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok
|
test_bgp_10_vrf_simple (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok
|
test_bgp_11_confederation (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok
|
test_bgp_12_v6_link_local (__main__.TestProtocolsBGP) ... ok
|
||||||
test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok
|
test_bgp_13_solo (__main__.TestProtocolsBGP) ... ok
|
||||||
|
|
||||||
----------------------------------------------------------------------
|
----------------------------------------------------------------------
|
||||||
Ran 13 tests in 348.191s
|
Ran 13 tests in 348.191s
|
||||||
|
|
||||||
OK
|
OK
|
||||||
|
|
||||||
Interface based tests
|
Interface based tests
|
||||||
---------------------
|
---------------------
|
||||||
|
|
||||||
Our smoketests not only test daemons and serives, but also check if what we
|
Our smoketests not only test daemons and serives, but also check if what we
|
||||||
configure for an interface works. Thus there is a common base classed named:
|
configure for an interface works. Thus there is a common base classed named:
|
||||||
``base_interfaces_test.py`` which holds all the common code that an interface
|
``base_interfaces_test.py`` which holds all the common code that an interface
|
||||||
supports and is tested.
|
supports and is tested.
|
||||||
|
|
||||||
Those common tests consists out of:
|
Those common tests consists out of:
|
||||||
|
|
||||||
* Add one or more IP addresses
|
* Add one or more IP addresses
|
||||||
* DHCP client and DHCPv6 prefix delegation
|
* DHCP client and DHCPv6 prefix delegation
|
||||||
* MTU size
|
* MTU size
|
||||||
* IP and IPv6 options
|
* IP and IPv6 options
|
||||||
* Port description
|
* Port description
|
||||||
* Port disable
|
* Port disable
|
||||||
* VLANs (QinQ and regular 802.1q)
|
* VLANs (QinQ and regular 802.1q)
|
||||||
* ...
|
* ...
|
||||||
|
|
||||||
.. note:: When you are working on interface configuration and you also wan't to
|
.. note:: When you are working on interface configuration and you also wan't to
|
||||||
test if the Smoketests pass you would normally loose the remote SSH connection
|
test if the Smoketests pass you would normally loose the remote SSH connection
|
||||||
to your :abbr:`DUT (Device Under Test)`. To handle this issue, some of the
|
to your :abbr:`DUT (Device Under Test)`. To handle this issue, some of the
|
||||||
interface based tests can be called with an environment variable beforehand
|
interface based tests can be called with an environment variable beforehand
|
||||||
to limit the number of interfaces used in the test. By default all interface
|
to limit the number of interfaces used in the test. By default all interface
|
||||||
e.g. all Ethernet interfaces are used.
|
e.g. all Ethernet interfaces are used.
|
||||||
|
|
||||||
.. code-block:: none
|
.. code-block:: none
|
||||||
|
|
||||||
vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py
|
vyos@vyos:~$ TEST_ETH="eth1 eth2" /usr/libexec/vyos/tests/smoke/cli/test_interfaces_bonding.py
|
||||||
test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok
|
test_add_multiple_ip_addresses (__main__.BondingInterfaceTest) ... ok
|
||||||
test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok
|
test_add_single_ip_address (__main__.BondingInterfaceTest) ... ok
|
||||||
test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok
|
test_bonding_hash_policy (__main__.BondingInterfaceTest) ... ok
|
||||||
test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok
|
test_bonding_lacp_rate (__main__.BondingInterfaceTest) ... ok
|
||||||
test_bonding_min_links (__main__.BondingInterfaceTest) ... ok
|
test_bonding_min_links (__main__.BondingInterfaceTest) ... ok
|
||||||
test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok
|
test_bonding_remove_member (__main__.BondingInterfaceTest) ... ok
|
||||||
test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok
|
test_dhcpv6_client_options (__main__.BondingInterfaceTest) ... ok
|
||||||
test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok
|
test_dhcpv6pd_auto_sla_id (__main__.BondingInterfaceTest) ... ok
|
||||||
test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok
|
test_dhcpv6pd_manual_sla_id (__main__.BondingInterfaceTest) ... ok
|
||||||
test_interface_description (__main__.BondingInterfaceTest) ... ok
|
test_interface_description (__main__.BondingInterfaceTest) ... ok
|
||||||
test_interface_disable (__main__.BondingInterfaceTest) ... ok
|
test_interface_disable (__main__.BondingInterfaceTest) ... ok
|
||||||
test_interface_ip_options (__main__.BondingInterfaceTest) ... ok
|
test_interface_ip_options (__main__.BondingInterfaceTest) ... ok
|
||||||
test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok
|
test_interface_ipv6_options (__main__.BondingInterfaceTest) ... ok
|
||||||
test_interface_mtu (__main__.BondingInterfaceTest) ... ok
|
test_interface_mtu (__main__.BondingInterfaceTest) ... ok
|
||||||
test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok
|
test_ipv6_link_local_address (__main__.BondingInterfaceTest) ... ok
|
||||||
test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok
|
test_mtu_1200_no_ipv6_interface (__main__.BondingInterfaceTest) ... ok
|
||||||
test_span_mirror (__main__.BondingInterfaceTest) ... ok
|
test_span_mirror (__main__.BondingInterfaceTest) ... ok
|
||||||
test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok
|
test_vif_8021q_interfaces (__main__.BondingInterfaceTest) ... ok
|
||||||
test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok
|
test_vif_8021q_lower_up_down (__main__.BondingInterfaceTest) ... ok
|
||||||
test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok
|
test_vif_8021q_mtu_limits (__main__.BondingInterfaceTest) ... ok
|
||||||
test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok
|
test_vif_8021q_qos_change (__main__.BondingInterfaceTest) ... ok
|
||||||
test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok
|
test_vif_s_8021ad_vlan_interfaces (__main__.BondingInterfaceTest) ... ok
|
||||||
test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok
|
test_vif_s_protocol_change (__main__.BondingInterfaceTest) ... ok
|
||||||
|
|
||||||
----------------------------------------------------------------------
|
----------------------------------------------------------------------
|
||||||
Ran 23 tests in 244.694s
|
Ran 23 tests in 244.694s
|
||||||
|
|
||||||
OK
|
OK
|
||||||
|
|
||||||
This will limit the `bond` interface test to only make use of `eth1` and `eth2`
|
This will limit the `bond` interface test to only make use of `eth1` and `eth2`
|
||||||
as member ports.
|
as member ports.
|
||||||
|
|
||||||
Config Load Tests
|
Config Load Tests
|
||||||
=================
|
=================
|
||||||
|
|
||||||
The other part of our tests are called "config load tests". The config load tests
|
The other part of our tests are called "config load tests". The config load tests
|
||||||
will load - one after another - arbitrary configuration files to test if the
|
will load - one after another - arbitrary configuration files to test if the
|
||||||
configuration migration scripts work as designed and that a given set of
|
configuration migration scripts work as designed and that a given set of
|
||||||
functionality still can be loaded with a fresh VyOS ISO image.
|
functionality still can be loaded with a fresh VyOS ISO image.
|
||||||
|
|
||||||
The configurations are all derived from production systems and can not only act
|
The configurations are all derived from production systems and can not only act
|
||||||
as a testcase but also as reference if one wants to enable a certain feature.
|
as a testcase but also as reference if one wants to enable a certain feature.
|
||||||
The configurations can be found here:
|
The configurations can be found here:
|
||||||
https://github.com/vyos/vyos-1x/tree/current/smoketest/configs
|
https://github.com/vyos/vyos-1x/tree/current/smoketest/configs
|
||||||
|
|
||||||
The entire test is controlled by the main wrapper script ``/usr/bin/vyos-configtest``
|
The entire test is controlled by the main wrapper script ``/usr/bin/vyos-configtest``
|
||||||
which behaves in the same way as the main smoketest script. It scans the folder
|
which behaves in the same way as the main smoketest script. It scans the folder
|
||||||
for potential configuration files and issues a ``load`` command one after another.
|
for potential configuration files and issues a ``load`` command one after another.
|
||||||
|
|
||||||
Manual config load test
|
Manual config load test
|
||||||
-----------------------
|
-----------------------
|
||||||
|
|
||||||
One is not bound to load all configurations one after another but can also load
|
One is not bound to load all configurations one after another but can also load
|
||||||
individual test configurations on his own.
|
individual test configurations on his own.
|
||||||
|
|
||||||
.. code-block:: none
|
.. code-block:: none
|
||||||
|
|
||||||
vyos@vyos:~$ configure
|
vyos@vyos:~$ configure
|
||||||
load[edit]
|
load[edit]
|
||||||
|
|
||||||
vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small
|
vyos@vyos# load /usr/libexec/vyos/tests/config/ospf-small
|
||||||
Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small'
|
Loading configuration from '/usr/libexec/vyos/tests/config/ospf-small'
|
||||||
Load complete. Use 'commit' to make changes effective.
|
Load complete. Use 'commit' to make changes effective.
|
||||||
[edit]
|
[edit]
|
||||||
vyos@vyos# compare
|
vyos@vyos# compare
|
||||||
[edit interfaces ethernet eth0]
|
[edit interfaces ethernet eth0]
|
||||||
-hw-id 00:50:56:bf:c5:6d
|
-hw-id 00:50:56:bf:c5:6d
|
||||||
[edit interfaces ethernet eth1]
|
[edit interfaces ethernet eth1]
|
||||||
+duplex auto
|
+duplex auto
|
||||||
-hw-id 00:50:56:b3:38:c5
|
-hw-id 00:50:56:b3:38:c5
|
||||||
+speed auto
|
+speed auto
|
||||||
[edit interfaces]
|
[edit interfaces]
|
||||||
-ethernet eth2 {
|
-ethernet eth2 {
|
||||||
- hw-id 00:50:56:b3:9c:1d
|
- hw-id 00:50:56:b3:9c:1d
|
||||||
-}
|
-}
|
||||||
-vti vti1 {
|
-vti vti1 {
|
||||||
- address 192.0.2.1/30
|
- address 192.0.2.1/30
|
||||||
-}
|
-}
|
||||||
...
|
...
|
||||||
|
|
||||||
vyos@vyos# commit
|
vyos@vyos# commit
|
||||||
vyos@vyos#
|
vyos@vyos#
|
||||||
|
|
||||||
.. note:: Some of the configurations have preconditions which need to be met.
|
.. note:: Some of the configurations have preconditions which need to be met.
|
||||||
Those most likely include generation of crypographic keys before the config
|
Those most likely include generation of crypographic keys before the config
|
||||||
can be applied - you will get a commit error otherwise. If you are interested
|
can be applied - you will get a commit error otherwise. If you are interested
|
||||||
how those preconditions are fulfilled check the vyos-build_ repository and
|
how those preconditions are fulfilled check the vyos-build_ repository and
|
||||||
the ``scripts/check-qemu-install`` file.
|
the ``scripts/check-qemu-install`` file.
|
||||||
|
|
||||||
.. include:: /_include/common-references.txt
|
.. include:: /_include/common-references.txt
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user