diff --git a/docs/vpp/configuration/dataplane/cpu.rst b/docs/vpp/configuration/dataplane/cpu.rst index 97e0fd93..cf5db4da 100644 --- a/docs/vpp/configuration/dataplane/cpu.rst +++ b/docs/vpp/configuration/dataplane/cpu.rst @@ -1,4 +1,4 @@ -:lastproofread: 2025-09-04 +:lastproofread: 2025-09-05 .. _vpp_config_dataplane_cpu: @@ -12,18 +12,30 @@ VPP can utilize multiple CPU cores to enhance packet processing performance. Pro There are several parameters that can be configured to optimize CPU usage for VPP. +.. important:: + + Please read carefully the system configuration settings page before making any changes to CPU settings: :doc:`system`. + +If CPU settings are not configured, VPP will start a single main thread on core 1 (``main-core``), without any additional worker threads. + CPU Configuration Parameters ============================ -main-core -^^^^^^^^^ +Mandatory settings +------------------ -The main core is responsible for handling control plane operations and managing worker threads. It should be set to a core that is not heavily utilized by other processes. If not set, VPP will automatically select a core `1`. +``main-core`` +^^^^^^^^^^^^^ + +The main core is responsible for handling control plane operations, managing worker threads, and processing packets. It should be set to a core that is not heavily utilized by other processes. The option should be always set if you apply any other CPU settings. .. cfgcmd:: set vpp settings cpu main-core -corelist-workers -^^^^^^^^^^^^^^^^ +Manual cores selection +---------------------- + +``corelist-workers`` +^^^^^^^^^^^^^^^^^^^^ This parameter specifies the list of CPU cores that will be used as worker threads for packet processing. @@ -36,8 +48,8 @@ Automatic cores selection There is a possibility to let VPP select CPU cores automatically. This can be done by configuring the following two parameters: -skip-cores -^^^^^^^^^^ +``skip-cores`` +^^^^^^^^^^^^^^ This parameter allows you to specify number of first CPU cores that should be excluded from being used for main or worker threads. The main thread will be assigned to the first available core after the skipped ones, and worker threads will be assigned to subsequent cores. @@ -45,8 +57,8 @@ This parameter allows you to specify number of first CPU cores that should be ex Exclude cores that are reserved for other critical system processes to ensure that VPP does not interfere with their operation. -workers -^^^^^^^ +``workers`` +^^^^^^^^^^^ This parameter allows you to specify the number of worker threads that should be created. Each worker thread will be assigned to a separate CPU core after the skipped and main ones. diff --git a/docs/vpp/configuration/dataplane/lcp.rst b/docs/vpp/configuration/dataplane/lcp.rst index eacf1045..bc3c0d62 100644 --- a/docs/vpp/configuration/dataplane/lcp.rst +++ b/docs/vpp/configuration/dataplane/lcp.rst @@ -26,15 +26,15 @@ Pay attention that disabling this option leads to loss of connectivity to destin Other configuration section crucial for integration between VPP and Kernel is netlink settings. It allows to configure how VPP management listen to netlink events and how it processes them. -.. cfgcmd:: set vpp settings lcp batch-delay-ms +.. cfgcmd:: set vpp settings lcp netlink batch-delay-ms This parameter specifies the delay in milliseconds between processing batch netlink messages. If you expect to get frequent and intensive netlink events, you may need to decrease this value to ensure that VPP processes netlink events in a timely manner. -.. cfgcmd:: set vpp settings lcp batch-size +.. cfgcmd:: set vpp settings lcp netlink batch-size This parameter specifies the maximum number of netlink messages to process in a single batch. If you have a high volume of netlink events, increasing this value can improve throughput by allowing more messages to be processed at once. However, setting it too high may increase latency for individual messages. -.. cfgcmd:: set vpp settings lcp rx-buffer-size +.. cfgcmd:: set vpp settings lcp netlink rx-buffer-size This parameter specifies the size of the receive buffer for netlink messages. Increasing this value can help accommodate bursts of netlink messages, but setting it too high may lead to increased memory usage. diff --git a/docs/vpp/configuration/dataplane/system.rst b/docs/vpp/configuration/dataplane/system.rst index ff8bfb61..5ed3b5e6 100644 --- a/docs/vpp/configuration/dataplane/system.rst +++ b/docs/vpp/configuration/dataplane/system.rst @@ -52,3 +52,115 @@ Both settings are automatically calculated based on configured hugepages. Kernel Tuning ============= +VPP performance greatly benefits from proper kernel tuning, especially CPU isolation and disabling unnecessary kernel features. These optimizations ensure dedicated CPU cores are available exclusively for VPP dataplane processing without interference from the kernel scheduler or other system processes. + +.. warning:: + + Kernel tuning changes require a system reboot to take effect. + + Improper CPU isolation can lead to system instability if essential system processes are starved of CPU resources. + +CPU Isolation and Optimization +------------------------------- + +CPU isolation is crucial for VPP performance as it dedicates specific CPU cores exclusively to VPP dataplane processing. The isolated cores are removed from the kernel scheduler and will not run regular system processes. + +**Disable NMI Watchdog** + +The NMI (Non-Maskable Interrupt) watchdog can interfere with VPP performance by generating interrupts on isolated cores and is not compatible with nohz-full mode: + +.. cfgcmd:: set system option kernel cpu disable-nmi-watchdog + + Disables the NMI watchdog for detecting hard CPU lockups. This prevents unnecessary interrupts on VPP worker cores. + +**CPU Core Isolation** + +.. cfgcmd:: set system option kernel cpu isolate-cpus + + Isolates specified CPUs from the kernel scheduler. Isolated cores will not run regular system processes and are dedicated to applications like VPP. + + The ```` can be: + + * Single core: ``2`` + * Range: ``2-5`` + * Mixed: ``1,3-5,7`` + + ..important:: + + Always reserve at least 1-2 cores for the operating system to ensure system stability. For example, on a 4-core system, isolate cores 2-3 for VPP and leave cores 0-1 for the OS. + + Assign the first isolated core as the VPP main core and the remaining isolated cores as VPP worker cores. Ensure that VPP CPU assignments match the isolated CPU range. + +**Adaptive-Tick Mode** + +.. cfgcmd:: set system option kernel cpu nohz-full + + Enables adaptive-tick mode (NO_HZ_FULL) for specified CPUs. This causes the kernel to avoid sending scheduling-clock interrupts to CPUs that have only one runnable task, significantly reducing interrupt overhead for dedicated workloads like VPP. + + Use the same CPU range as configured for ``isolate-cpus``. + +**RCU Callback Offloading** + +.. cfgcmd:: set system option kernel cpu rcu-no-cbs + + Offloads Read-Copy-Update (RCU) callback processing from specified CPUs. This ensures that RCU callbacks do not prevent the specified CPUs from entering dyntick-idle or adaptive-tick mode, which is essential for nohz-full functionality. + + Use the same CPU range as configured for ``isolate-cpus``. + +System Optimization +-------------------- + +Additional kernel optimizations can further improve VPP performance by disabling unnecessary features and reducing system overhead. + +**Disable High Precision Event Timer** + +.. cfgcmd:: set system option kernel disable-hpet + + Disables the High Precision Event Timer (HPET). HPET can cause additional interrupts and overhead that may impact VPP performance. + +**Disable Machine Check Exceptions** + +.. cfgcmd:: set system option kernel disable-mce + + Disables Machine Check Exception (MCE) reporting and handling. While MCE provides hardware error detection, it can introduce latency in high-performance scenarios. + +**Disable CPU Power Saving** + +.. cfgcmd:: set system option kernel disable-power-saving + + Disables CPU power saving mechanisms (C-states). This keeps CPU cores at maximum performance levels, eliminating latency from power state transitions. + +**Disable Soft Lockup Detection** + +.. cfgcmd:: set system option kernel disable-softlockup + + Disables the soft lockup detector for kernel threads. This prevents false positives when VPP worker threads are busy processing packets. + +**Disable CPU Mitigations** + +.. cfgcmd:: set system option kernel disable-mitigations + + Disables all optional CPU mitigations for security vulnerabilities (e.g., Spectre, Meltdown). This may improve performance on some platforms. + +Optimal Configuration Example +----------------------------- + +For a system with 4 CPU cores (0-3) where cores 2-3 are dedicated to VPP: + +.. code-block:: none + + # Kernel CPU optimizations + set system option kernel cpu disable-nmi-watchdog + set system option kernel cpu isolate-cpus '2-3' + set system option kernel cpu nohz-full '2-3' + set system option kernel cpu rcu-no-cbs '2-3' + + # System optimizations + set system option kernel disable-hpet + set system option kernel disable-mce + set system option kernel disable-power-saving + set system option kernel disable-softlockup + + # VPP CPU assignment (matches isolated cores) + set vpp settings cpu main-core '2' + set vpp settings cpu corelist-workers '3' diff --git a/docs/vpp/configuration/nat/nat44.rst b/docs/vpp/configuration/nat/nat44.rst index 495cc27b..6a52b30f 100644 --- a/docs/vpp/configuration/nat/nat44.rst +++ b/docs/vpp/configuration/nat/nat44.rst @@ -21,7 +21,7 @@ Configuration of NAT44 involves few steps: 2. Create NAT rules for SNAT and/or DNAT. Dynamic and Static Operations -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +============================= NAT44 configuration can be done in one of two ways or in both ways simultaneously: @@ -33,12 +33,12 @@ To configure dynamic NAT, you need to define a pool of public IP addresses that Static rules are more suitable for scenarios where you need to provide consistent and predictable mappings between private and public IP addresses, also they are the only way to configure DNAT. Interfaces Configuration ------------------------- +======================== The first step in configuring NAT44 is defining which interfaces handle inside (private) and outside (public) traffic. VyOS uses these interface designations to determine the direction of translation. Inside Interfaces -^^^^^^^^^^^^^^^^^ +----------------- Inside interfaces connect to private networks where hosts need source NAT to access external networks. @@ -48,8 +48,8 @@ Inside interfaces connect to private networks where hosts need source NAT to acc Traffic flowing **from** inside interfaces gets source NAT applied, translating private source addresses to public addresses from the translation pool. -Outside Interfaces -^^^^^^^^^^^^^^^^^^ +Outside Interfaces +------------------ Outside interfaces connect to public networks where external hosts may need to access internal services. @@ -60,7 +60,7 @@ Outside interfaces connect to public networks where external hosts may need to a Traffic flowing **to** outside interfaces can trigger destination NAT based on static rules, allowing external access to internal services. Interface Roles and Traffic Flow -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------- .. note:: @@ -79,7 +79,7 @@ Interface Roles and Traffic Flow 4. **Static NAT**: Requires explicit configuration for outside→inside traffic Multiple Interface Support -^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------- You can configure multiple interfaces as inside or outside to support complex network topologies: @@ -94,12 +94,12 @@ You can configure multiple interfaces as inside or outside to support complex ne set vpp nat44 interface outside eth3 Address Pool Configuration --------------------------- +========================== Address pools define ranges of IP addresses that can be used for NAT translations. VyOS NAT44 supports two types of address pools, each serving different purposes. Translation Pools -^^^^^^^^^^^^^^^^^ +----------------- Translation pools are used for dynamic source NAT (SNAT). They provide a range of public IP addresses that can be dynamically assigned to private hosts when they access external networks. @@ -125,7 +125,7 @@ Translation pools are used for dynamic source NAT (SNAT). They provide a range o set vpp nat44 address-pool translation interface eth1 Twice-NAT Pools -^^^^^^^^^^^^^^^ +--------------- Twice-NAT pools are used when performing both source and destination NAT on the same traffic flow. This is particularly useful in scenarios where you need to: @@ -152,7 +152,7 @@ Twice-NAT pools are used when performing both source and destination NAT on the set vpp nat44 address-pool twice-nat interface eth2 Pool Requirements -^^^^^^^^^^^^^^^^^ +----------------- .. important:: @@ -162,7 +162,7 @@ Pool Requirements * Interface-based pools automatically include main (first) IP address assigned to the specified interface Pool Selection Priority -^^^^^^^^^^^^^^^^^^^^^^^ +----------------------- When multiple pools are configured, VyOS uses the following selection priority: @@ -175,7 +175,7 @@ When multiple pools are configured, VyOS uses the following selection priority: As soon as you have configured interfaces and pool, the NAT44 is operational. Static Rules Configuration --------------------------- +========================== Static NAT rules provide predictable and consistent mappings between private and public IP addresses. They are essential for: @@ -186,7 +186,7 @@ Static NAT rules provide predictable and consistent mappings between private and Unlike dynamic NAT that uses a pool of addresses, static rules create one-to-one mappings that persist until explicitly removed. Basic Static Rule Configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------- To create a static NAT rule, you need to define the local (internal) and external (public) address mappings: @@ -207,7 +207,7 @@ Where: This basic configuration creates a static one-to-one mapping. Traffic from outside to the external IP will be translated to the internal IP, and vice versa. Port-based Static Rules -^^^^^^^^^^^^^^^^^^^^^^^ +----------------------- For more granular control, you can create port-specific static rules. This is useful when you want to publish specific services: @@ -237,12 +237,12 @@ Where: * ```` specifies the protocol (tcp, udp, icmp) - if not specified, the rule applies to all protocols Advanced Static Rule Options -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------- VyOS NAT44 supports several advanced options for static rules: Twice-NAT -~~~~~~~~~ +^^^^^^^^^ Twice-NAT performs both source and destination NAT. So when an external host accesses an internal service, a source IP of such connection is translated to an address from the twice-NAT address pool. @@ -255,7 +255,7 @@ The twice-NAT option can be enabled with the following command: set vpp nat44 static rule options twice-nat Self Twice-NAT -~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ Self Twice-NAT is used when a local host needs to access itself via the external address: @@ -270,7 +270,7 @@ This option rewrites source IP addresses on packets sent only from a local addre Using self-twice-nat option requires to set interface connected to the local network as both inside and outside interface, because both source and destination NAT need to be applied. Out-to-In Only -~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ Restricts the rule to only apply to traffic from outside to inside interfaces: @@ -281,7 +281,7 @@ Restricts the rule to only apply to traffic from outside to inside interfaces: This prevents the creation of sessions from the inside interface, making it purely a DNAT rule. Force Twice-NAT Address -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ When using twice-nat, you can force the use of a specific IP address from the twice-nat address pool: @@ -339,12 +339,12 @@ Configuration Examples ``set vpp nat44 address-pool twice-nat address `` Advanced NAT44 Settings ------------------------ +======================= VyOS provides additional NAT44 settings for fine-tuning performance and behavior. These settings are configured under the VPP settings hierarchy. Session Timeouts -^^^^^^^^^^^^^^^^ +---------------- NAT44 maintains translation sessions with configurable timeout values for different protocols: @@ -374,7 +374,7 @@ NAT44 maintains translation sessions with configurable timeout values for differ set vpp settings nat44 timeout icmp 30 Session Limits -^^^^^^^^^^^^^^ +-------------- Control the maximum number of concurrent NAT sessions: @@ -392,7 +392,7 @@ This setting helps prevent memory exhaustion and ensures predictable performance set vpp settings nat44 session-limit 100000 Forwarding Behavior -^^^^^^^^^^^^^^^^^^^ +------------------- By default, VyOS NAT44 forwards packets that don't match any NAT rules according to the routing table. This behavior can be controlled: @@ -410,7 +410,7 @@ By default, VyOS NAT44 forwards packets that don't match any NAT rules according * **Security isolation**: Preventing any non-NAT traffic from traversing the device Worker Assignment -^^^^^^^^^^^^^^^^^ +----------------- For advanced performance tuning, you can assign NAT44 processing to specific worker threads: @@ -433,7 +433,7 @@ For advanced performance tuning, you can assign NAT44 processing to specific wor Worker assignment is an advanced feature typically used in high-performance deployments where you want to dedicate specific CPU cores to NAT processing. Most deployments don't require this configuration. Complete Configuration Example ------------------------------- +============================== Here's a complete example showing how to configure VyOS NAT44 for a typical network setup: @@ -500,10 +500,10 @@ Here's a complete example showing how to configure VyOS NAT44 for a typical netw set vpp nat44 static rule 300 description "API service (No Internet access for it)" Best Practices and Troubleshooting ----------------------------------- +================================== Recommendations -^^^^^^^^^^^^^^^ +--------------- * **Use out-to-in-only** for services that do not need access to external networks * **Limit port ranges** in static rules to only necessary ports @@ -512,7 +512,7 @@ Recommendations * **Configure appropriate pool sizes** based on expected concurrent connections in your network Common Configuration Issues -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +--------------------------- **Static rules not working:** @@ -527,7 +527,7 @@ Common Configuration Issues 3. Check that both translation and twice-nat pools are properly defined Operational Commands -^^^^^^^^^^^^^^^^^^^^ +==================== Monitor NAT44 status and active connections using VyOS operational commands: