mirror of
https://github.com/vyos/vyos-documentation.git
synced 2025-10-26 08:41:46 +01:00
f03898cbd4
The following options have not been migrated because we
did not find its valid and correct use:
no-vyos-configure
Do not load the config, leaves you with plain Linux system.
no-vyos-rl-system
Disables getting the system into clean state before loading the config.
344 lines
14 KiB
ReStructuredText
344 lines
14 KiB
ReStructuredText
.. _troubleshooting:
|
|
|
|
Troubleshooting
|
|
===============
|
|
|
|
Sometimes things break or don't work as expected. This section describes
|
|
several troubleshooting tools provided by VyOS that can help when something
|
|
goes wrong.
|
|
|
|
Basic Connectivity Verification
|
|
-------------------------------
|
|
|
|
Verifying connectivity can be done with the familiar `ping` and `traceroute`
|
|
commands. The options for each are shown (the options for each command were
|
|
displayed using the built-in help as described in the :ref:`cli`
|
|
section and are omitted from the output here):
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ ping
|
|
Possible completions:
|
|
<hostname> Send Internet Control Message Protocol (ICMP) echo request
|
|
<x.x.x.x>
|
|
<h:h:h:h:h:h:h:h>
|
|
|
|
Several options are available when more extensive troubleshooting is needed:
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ ping 8.8.8.8
|
|
Possible completions:
|
|
<Enter> Execute the current command
|
|
adaptive Ping options
|
|
allow-broadcast
|
|
audible
|
|
bypass-route
|
|
count
|
|
deadline
|
|
flood
|
|
interface
|
|
interval
|
|
mark
|
|
no-loopback
|
|
numeric
|
|
pattern
|
|
quiet
|
|
record-route
|
|
size
|
|
timestamp
|
|
tos
|
|
ttl
|
|
verbose
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ traceroute
|
|
Possible completions:
|
|
<hostname> Track network path to specified node
|
|
<x.x.x.x>
|
|
<h:h:h:h:h:h:h:h>
|
|
ipv4 Track network path to <hostname|IPv4 address>
|
|
ipv6 Track network path to <hostname|IPv6 address>
|
|
|
|
However, another tool, mtr_, is available which combines ping and traceroute
|
|
into a single tool. An example of its output is shown:
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ mtr 10.62.212.12
|
|
|
|
My traceroute [v0.85]
|
|
vyos (0.0.0.0)
|
|
Keys: Help Display mode Restart statistics Order of fields quit
|
|
Packets Pings
|
|
Host Loss% Snt Last Avg Best Wrst StDev
|
|
1. 10.11.110.4 0.0% 34 0.5 0.5 0.4 0.8 0.1
|
|
2. 10.62.255.184 0.0% 34 1.1 1.0 0.9 1.4 0.1
|
|
3. 10.62.255.71 0.0% 34 1.4 1.4 1.3 2.0 0.1
|
|
4. 10.62.212.12 0.0% 34 1.6 1.6 1.6 1.7 0.0
|
|
|
|
.. note:: The output of ``mtr`` consumes the screen and will replace your
|
|
command prompt.
|
|
|
|
Several options are available for changing the display output. Press `h` to
|
|
invoke the built in help system. To quit, just press `q` and you'll be returned
|
|
to the VyOS command prompt.
|
|
|
|
Monitoring
|
|
----------
|
|
|
|
Network Interfaces
|
|
^^^^^^^^^^^^^^^^^^
|
|
|
|
It's possible to monitor network traffic, either at the flow level or protocol
|
|
level. This can be useful when troubleshooting a variety of protocols and
|
|
configurations. The following interface types can be monitored:
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ monitor interfaces
|
|
Possible completions:
|
|
<Enter> Execute the current command
|
|
bonding Monitor a bonding interface
|
|
bridge Monitor a bridge interface
|
|
ethernet Monitor a ethernet interface
|
|
loopback Monitor a loopback interface
|
|
openvpn Monitor an openvpn interface
|
|
pppoe Monitor pppoe interface
|
|
pseudo-ethernet
|
|
Monitor a pseudo-ethernet interface
|
|
tunnel Monitor a tunnel interface
|
|
vrrp Monitor a vrrp interface
|
|
vti Monitor a vti interface
|
|
wireless Monitor wireless interface
|
|
|
|
To monitor traffic flows, issue the :code:`monitor interfaces <type> <name> flow`
|
|
command, replacing `<type>` and `<name>` with your desired interface type and
|
|
name, respectively. Output looks like the following:
|
|
|
|
.. code-block:: none
|
|
|
|
12.5Kb 25.0Kb 37.5Kb 50.0Kb 62.5Kb
|
|
????????????????????????????????????????????????????????????????????????????????????????????????????
|
|
10.11.111.255 => 10.11.110.37 0b 0b 0b
|
|
<= 624b 749b 749b
|
|
10.11.110.29 => 10.62.200.11 0b 198b 198b
|
|
<= 0b 356b 356b
|
|
255.255.255.255 => 10.11.110.47 0b 0b 0b
|
|
<= 724b 145b 145b
|
|
10.11.111.255 => 10.11.110.47 0b 0b 0b
|
|
<= 724b 145b 145b
|
|
10.11.111.255 => 10.11.110.255 0b 0b 0b
|
|
<= 680b 136b 136b
|
|
????????????????????????????????????????????????????????????????????????????????????????????????????
|
|
TX: cumm: 26.7KB peak: 40.6Kb rates: 23.2Kb 21.4Kb 21.4Kb
|
|
RX: 67.5KB 63.6Kb 54.6Kb 54.0Kb 54.0Kb
|
|
TOTAL: 94.2KB 104Kb 77.8Kb 75.4Kb 75.4Kb
|
|
|
|
Several options are available for changing the display output. Press `h` to
|
|
invoke the built in help system. To quit, just press `q` and you'll be returned
|
|
to the VyOS command prompt.
|
|
|
|
To monitor interface traffic, issue the :code:`monitor interfaces <type> <name>
|
|
traffic` command, replacing `<type>` and `<name>` with your desired interface
|
|
type and name, respectively. This command invokes the familiar tshark_ utility
|
|
and the following options are available:
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ monitor interfaces ethernet eth0 traffic
|
|
Possible completions:
|
|
<Enter> Execute the current command
|
|
detail Monitor detailed traffic for the specified ethernet interface
|
|
filter Monitor filtered traffic for the specified ethernet interface
|
|
save Save monitored traffic to a file
|
|
unlimited Monitor traffic for the specified ethernet interface
|
|
|
|
To quit monitoring, press `Ctrl-c` and you'll be returned to the VyOS command
|
|
prompt. The `detail` keyword provides verbose output of the traffic seen on
|
|
the monitored interface. The `filter` keyword accepts valid `PCAP filter
|
|
expressions`_, enclosed in single or double quotes (e.g. "port 25" or "port 161
|
|
and udp"). The `save` keyword allows you to save the traffic dump to a file.
|
|
The `unlimited` keyword is used to specify that an unlimited number of packets
|
|
can be captured (by default, 1,000 packets are captured and you're returned to
|
|
the VyOS command prompt).
|
|
|
|
Interface Bandwith
|
|
^^^^^^^^^^^^^^^^^^
|
|
|
|
to take a quick view on the used bandwith of an interface use the ``monitor bandwith`` command
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ monitor bandwidth interface eth0
|
|
|
|
show the following:
|
|
|
|
.. code-block:: none
|
|
|
|
eth0 bmon 3.5
|
|
Interfaces │ RX bps pps %│ TX bps pps %
|
|
>eth0 │ 141B 2 │ 272B 1
|
|
───────────────────────────────┴───────────────────────┴────────────────────────────────────────────────────────────────
|
|
B (RX Bytes/second)
|
|
198.00 .|....|.....................................................
|
|
165.00 .|....|.....................................................
|
|
132.00 ||..|.|.....................................................
|
|
99.00 ||..|.|.....................................................
|
|
66.00 |||||||.....................................................
|
|
33.00 |||||||.....................................................
|
|
1 5 10 15 20 25 30 35 40 45 50 55 60
|
|
KiB (TX Bytes/second)
|
|
3.67 ......|.....................................................
|
|
3.06 ......|.....................................................
|
|
2.45 ......|.....................................................
|
|
1.84 ......|.....................................................
|
|
1.22 ......|.....................................................
|
|
0.61 :::::||.....................................................
|
|
1 5 10 15 20 25 30 35 40 45 50 55 60
|
|
|
|
───────────────────────────────────────── Press d to enable detailed statistics ────────────────────────────────────────
|
|
─────────────────────────────────────── Press i to enable additional information ───────────────────────────────────────
|
|
Wed Apr 3 14:46:59 2019 Press ? for help
|
|
|
|
| Press ``d`` for more detailed informations or ``i`` for additional information.
|
|
| To exit press ``q`` and than ``y``
|
|
|
|
Interface performance
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
To take a look on the network bandwith between two nodes, the ``monitor bandwidth-test`` command is used to run iperf.
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ monitor bandwidth-test
|
|
Possible completions:
|
|
accept Wait for bandwidth test connections (port TCP/5001)
|
|
initiate Initiate a bandwidth test
|
|
|
|
| The ``accept`` command open a listen iperf server on TCP Port 5001
|
|
| The ``initiate`` command conncet to this server.
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ monitor bandwidth-test initiate
|
|
Possible completions:
|
|
<hostname> Initiate a bandwidth test to specified host (port TCP/5001)
|
|
<x.x.x.x>
|
|
<h:h:h:h:h:h:h:h>
|
|
|
|
|
|
Monitor command
|
|
^^^^^^^^^^^^^^^
|
|
|
|
The ``monitor command`` command allows you to repeatedly run a command to view a continuously refreshed output.
|
|
The command is run and output every 2 seconds, allowing you to monitor the output continuously without having to re-run the command. This can be useful to follow routing adjacency formation.
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@router:~$ monitor command "show interfaces"
|
|
|
|
Will clear the screen and show you the output of ``show interfaces`` every 2 seconds.
|
|
|
|
.. code-block:: none
|
|
|
|
Every 2.0s: /opt/vyatta/bin/vyatta-op-cmd-wrapper s... Sun Mar 26 02:49:46 2019
|
|
|
|
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
|
|
Interface IP Address S/L Description
|
|
--------- ---------- --- -----------
|
|
eth0 192.168.1.1/24 u/u
|
|
eth0.5 198.51.100.4/24 u/u WAN
|
|
lo 127.0.0.1/8 u/u
|
|
::1/128
|
|
vti0 172.32.254.2/30 u/u
|
|
vti1 172.32.254.9/30 u/u
|
|
|
|
Clear Command
|
|
-------------
|
|
|
|
Sometimes you need to clear counters or statistics to troubleshoot better.
|
|
|
|
To do this use the ``clear`` command in Operational mode.
|
|
|
|
to clear the console output
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ clear console
|
|
|
|
to clear interface counters
|
|
|
|
.. code-block:: none
|
|
|
|
# clear all interfaces
|
|
vyos@vyos:~$ clear interface ethernet counters
|
|
# clear specific interface
|
|
vyos@vyos:~$ clear interface ehternet eth0 counters
|
|
|
|
The command follow the same logic as the ``set`` command in configuration mode.
|
|
|
|
.. code-block:: none
|
|
|
|
# clear all counters of a interface type
|
|
vyos@vyos:~$ clear interface <interface_type> counters
|
|
# clear counter of a interface in interface_type
|
|
vyos@vyos:~$ clear interface <interface_type> <interace_name> counters
|
|
|
|
|
|
to clear counters on firewall rulesets or single rules
|
|
|
|
.. code-block:: none
|
|
|
|
vyos@vyos:~$ clear firewall name <ipv4 ruleset name> counters
|
|
vyos@vyos:~$ clear firewall name <ipv4 ruleset name> rule <rule#> counters
|
|
|
|
vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> counters
|
|
vyos@vyos:~$ clear firewall ipv6-name <ipv6 ruleset name> rule <rule#> counters
|
|
|
|
|
|
Basic System Information
|
|
------------------------
|
|
|
|
.. _boot-steps:
|
|
|
|
Boot steps
|
|
^^^^^^^^^^
|
|
|
|
VyOS 1.2.0+ uses `Debian Jessie`_ as the base Linux operating system.
|
|
Jessie was the first version of Debian that uses `systemd`_ as the default init system.
|
|
|
|
These are the boot steps for VyOS 1.2.0+
|
|
|
|
1. The BIOS loads Grub (or isolinux for the Live CD)
|
|
2. Grub then starts the Linux boot and loads the Linux Kernel ``/boot/vmlinuz``
|
|
3. Kernel Launches Systemd ``/lib/systemd/systemd``
|
|
4. Systemd loads the VyOS service file ``/lib/systemd/system/vyos-router.service``
|
|
5. The service file launches the VyOS router init script ``/usr/libexec/vyos/init/vyos-router`` - this is part of the `vyatta-cfg`_ Debian package
|
|
|
|
1. Starts FRR_ - successor to `GNU Zebra`_ and `Quagga`_
|
|
|
|
2. Initialises the boot configuration file - copies over ``config.boot.default`` if there is no configuration
|
|
3. Runs the configuration migration, if the configuration is for an older version of VyOS
|
|
4. Runs The pre-config script, if there is one ``/config/scripts/vyos-preconfig-bootup.script``
|
|
5. If the config file was upgraded, runs any post upgrade scripts ``/config/scripts/post-upgrade.d``
|
|
6. Starts **rl-system** and **firewall**
|
|
7. Mounts the ``/boot`` partition
|
|
8. The boot configuration file is then applied by ``/opt/vyatta/sbin/vyatta-boot-config-loader /opt/vyatta/etc/config/config.boot``
|
|
|
|
1. The config loader script writes log entries to ``/var/log/vyatta-config-loader.log``
|
|
|
|
10. Runs ``telinit q`` to tell the init system to reload ``/etc/inittab``
|
|
11. Finally it runs the post-config script ``/config/scripts/vyos-postconfig-bootup.script``
|
|
|
|
.. _Quagga: http://www.quagga.net/
|
|
.. _`GNU Zebra`: https://www.gnu.org/software/zebra/
|
|
.. _FRR: https://frrouting.org/
|
|
.. _vyatta-cfg: https://github.com/vyos/vyatta-cfg
|
|
.. _systemd: _https://freedesktop.org/wiki/Software/systemd/
|
|
.. _`Debian Jessie`: https://www.debian.org/releases/jessie/
|
|
.. _mtr: http://www.bitwizard.nl/mtr/
|
|
.. _tshark: https://www.wireshark.org/docs/man-pages/tshark.html
|
|
.. _`PCAP filter expressions`: http://www.tcpdump.org/manpages/pcap-filter.7.html
|