Repository files navigation

1. Introduction
2. Features
3. How does it work?
4. Installation
   1. Requirements / Dependencies
   2. Debian Package
   3. Container / Docker
   4. Source
5. Usage / Configuration
   1. GUI Integration
   2. Proxmox HA Integration
   3. Options
6. Affinity & Anti-Affinity Rules
   1. Affinity Rules
   2. Anti-Affinity Rules
   3. Ignore VMs
   4. Pin VMs to Hypervisor Nodes
7. Maintenance
8. Misc
   1. Bugs
   2. Contributing
   3. Documentation
   4. Support
9. Author(s)

ProxLB is an advanced load balancing solution specifically designed for Proxmox clusters, addressing the absence of a Dynamic Resource Scheduler (DRS) that is familiar to VMware users. As a third-party solution, ProxLB enhances the management and efficiency of Proxmox clusters by intelligently distributing workloads across available nodes. Workloads can be balanced by different times like the guest's memory, CPU or disk usage or their assignment to avoid overprovisioning and ensuring resources.

One of the key advantages of ProxLB is that it is fully open-source and free, making it accessible for anyone to use, modify, and contribute to. This ensures transparency and fosters community-driven improvements. ProxLB supports filtering and ignoring specific nodes and guests through configuration files and API calls, providing administrators with the flexibility to tailor the load balancing behavior to their specific needs.

A standout feature of ProxLB is its maintenance mode. When enabled, all guest workloads are automatically moved to other nodes within the cluster, ensuring that a node can be safely updated, rebooted, or undergo hardware maintenance without disrupting the overall cluster operation. Additionally, ProxLB supports both affinity and anti-affinity rules, allowing operators to group multiple guests to run together on the same node or ensure that certain guests do not run on the same node, depending on the cluster's node count. This feature is crucial for optimizing performance and maintaining high availability.

ProxLB can also return the best next node for guest placement, which can be integrated into CI/CD pipelines using tools like Ansible or Terraform. This capability streamlines the deployment process and ensures efficient resource utilization. Furthermore, ProxLB leverages the Proxmox API, including the entire ACL (Access Control List) system, for secure and efficient operation. Unlike some solutions, it does not require SSH access, enhancing security and simplifying configuration.

Overall, ProxLB significantly enhances resource management by intelligently distributing workloads, reducing downtime through its maintenance mode, and providing improved flexibility with affinity and anti-affinity rules. Its seamless integration with CI/CD tools and reliance on the Proxmox API make it a robust and secure solution for optimizing Proxmox cluster performance.

ProxLB's key features are by enabling automatic rebalancing of VMs and CTs across a Proxmox cluster based on memory, CPU, and local disk usage while identifying optimal nodes for automation. It supports maintenance mode, affinity rules, and seamless Proxmox API integration with ACL support, offering flexible usage as a one-time operation, a daemon, or through the Proxmox Web GUI.

Features

• Rebalance VMs/CTs in the cluster by:
  • Memory
  • Disk (only local storage)
  • CPU
• Get best nodes for further automation
• Supported Guest Types
  • VMs
  • CTs
• Maintenance Mode
  • Set node(s) into maintenance
  • Move all workloads to different nodes
• Affinity / Anti-Affinity Rules
• Fully based on Proxmox API
  • Fully integrated into the Proxmox ACL
  • No SSH required
• Usage
  • One-Time
  • Daemon
  • Proxmox Web GUI Integration

ProxLB is a load-balancing system designed to optimize the distribution of virtual machines (VMs) and containers (CTs) across a cluster. It works by first gathering resource usage metrics from all nodes in the cluster through the Proxmox API. This includes detailed resource metrics for each VM and CT on every node. ProxLB then evaluates the difference between the maximum and minimum resource usage of the nodes, referred to as "Balanciness." If this difference exceeds a predefined threshold (which is configurable), the system initiates the rebalancing process.

Before starting any migrations, ProxLB validates that rebalancing actions are necessary and beneficial. Depending on the selected balancing mode — such as CPU, memory, or disk — it creates a balancing matrix. This matrix sorts the VMs by their maximum used or assigned resources, identifying the VM with the highest usage. ProxLB then places this VM on the node with the most free resources in the selected balancing type. This process runs recursively until the operator-defined Balanciness is achieved. Balancing can be defined for the used or max. assigned resources of VMs/CTs.

• Python3.x
• proxmoxer
• requests
• urllib3
• pyyaml

The dependencies can simply be installed withpip by running the following command:

pip install -r requirements.txt

Note: Distribution packages, such like the provided.deb package will automatically resolve and install all required dependencies by using already packaged version from the distribution's repository. By using the Docker (container) image or Debian packages, you do not need to take any care of the requirements listed here.

ProxLB is a powerful and flexible load balancer designed to work across various architectures, includingamd64,arm64,rv64 and many other ones that support Python. It runs independently of the underlying hardware, making it a versatile choice for different environments. This chapter covers the step-by-step process to install ProxLB on Debian-based systems, including Debian clones like Ubuntu.

You can simply use this snippet to install the repository and to install ProxLB on your system.

echo"deb https://repo.gyptazy.com/stable /"> /etc/apt/sources.list.d/proxlb.listwget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpgapt-get update&& apt-get -y install proxlbcp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml# Adjust the config to your needsvi /etc/proxlb/proxlb.yamlsystemctl start proxlb

Afterwards, ProxLB is running in the background and balances your cluster by your defined balancing method (default: memory).

ProxLB provides two different repositories:

• https://repo.gyptazy.com/stable (only stable release)
• https://repo.gyptazy.com/testing (bleeding edge - not recommended)

The repository is signed and the GPG key can be found at:

• https://repo.gyptazy.com/repository.gpg

You can also simply import it by running:

# KeyID:  17169F23F9F71A14AD49EDADDB51D3EB01824F4C# UID:    gyptazy Solutions Repository <contact@gyptazy.com># SHA256: 52c267e6f4ec799d40cdbdb29fa518533ac7942dab557fa4c217a76f90d6b0f3  repository.gpgwget -O /etc/apt/trusted.gpg.d/proxlb.asc https://repo.gyptazy.com/repository.gpg

Note: The defined repositoriesrepo.gyptazy.com andrepo.proxlb.de are the same!

If you do not want to use the repository you can also find the debian packages as a .deb file on gyptazy's CDN at:

• https://cdn.gyptazy.com/files/os/debian/proxlb/

Afterwards, you can simply install the package by running:

dpkg -i proxlb_*.debcp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml# Adjust the config to your needsvi /etc/proxlb/proxlb.yamlsystemctl start proxlb

Using the ProxLB container images is straight forward and only requires you to mount the config file.

# Pull the imagedocker pull cr.gyptazy.com/proxlb/proxlb:latest# Download the configwget -O proxlb.yaml https://raw.githubusercontent.com/gyptazy/ProxLB/refs/heads/main/config/proxlb_example.yaml# Adjust the config to your needsvi proxlb.yaml# Start the ProxLB container image with the ProxLB configdocker run -it --rm -v$(pwd)/proxlb.yaml:/etc/proxlb/proxlb.yaml proxlb

Note: ProxLB container images are officially only available at cr.proxlb.de and cr.gyptazy.com.

ProxLB can also easily be used from the provided sources - for traditional systems but also as a Docker/Podman container image.

Setting up and running ProxLB from the sources is simple and requires just a few commands. Ensure Python 3 and the Python dependencies are installed on your system, then run ProxLB using the following command:

git clone https://github.com/gyptazy/ProxLB.gitcd ProxLB

Afterwards simply adjust the config file to your needs:

vi config/proxlb.yaml

Start ProxLB by Python3 on the system:

python3 proxlb/main.py -c config/proxlb.yaml

Creating a container image of ProxLB is straightforward using the provided Dockerfile. The Dockerfile simplifies the process by automating the setup and configuration required to get ProxLB running in an Alpine container. Simply follow the steps in the Dockerfile to build the image, ensuring all dependencies and configurations are correctly applied. For those looking for an even quicker setup, a ready-to-use ProxLB container image is also available, eliminating the need for manual building and allowing for immediate deployment.

git clone https://github.com/gyptazy/ProxLB.gitcd ProxLBdocker build -t proxlb.

Afterwards simply adjust the config file to your needs:

vi config/proxlb.yaml

Finally, start the created container.

docker run -it --rm -v$(pwd)/proxlb.yaml:/etc/proxlb/proxlb.yaml proxlb

Running ProxLB is straightforward and versatile, as it only requiresPython3 and theproxmoxer library. This means ProxLB can be executed directly on a Proxmox node or on dedicated systems such as Debian, RedHat, or even FreeBSD, provided that the Proxmox API is accessible from the client running ProxLB. ProxLB can also run inside a Container - Docker or LXC - and is simply up to you.

ProxLB can also be accessed through the Proxmox Web UI by installing the optionalpve-proxmoxlb-service-ui package, which depends on the proxlb package. For full Web UI integration, this package must be installed on all nodes within the cluster. Once installed, a new menu item -Rebalancing, appears in the cluster level under the HA section. Once installed, it offers two key functionalities:

• Rebalancing VM workloads
• Migrate VM workloads away from a defined node (e.g. maintenance preparation)

Note: This package is currently discontinued and will be readded at a later time. See also:#44: How to install pve-proxmoxlb-service-ui package.

Proxmox HA (High Availability) groups are designed to ensure that virtual machines (VMs) remain running within a Proxmox cluster. HA groups define specific rules for where VMs should be started or migrated in case of node failures, ensuring minimal downtime and automatic recovery.

However, when used in conjunction with ProxLB, the built-in load balancer for Proxmox, conflicts can arise. ProxLB operates with its own logic for workload distribution, taking into account affinity and anti-affinity rules. While it effectively balances guest workloads, it may re-shift and redistribute VMs in a way that does not align with HA group constraints, potentially leading to unsuitable placements.

Due to these conflicts, it is currently not recommended to use both HA groups and ProxLB simultaneously. The interaction between the two mechanisms can lead to unexpected behavior, where VMs might not adhere to HA group rules after being moved by ProxLB.

A solution to improve compatibility between HA groups and ProxLB is under evaluation, aiming to ensure that both features can work together without disrupting VM placement strategies.

See also:#65: Host groups: Honour HA groups.

The following options can be set in the configuration fileproxlb.yaml:

An example of the configuration file looks like:

proxmox_api:  hosts: ['virt01.example.com', '10.10.10.10', 'fe01::bad:code::cafe']  user: root@pam  pass: crazyPassw0rd!  # API Token method  # token_id: proxlb  # token_secret: 430e308f-1337-1337-beef-1337beefcafe  ssl_verification: True  timeout: 10  # API Connection retries  # retries: 1  # wait_time: 1proxmox_cluster:  maintenance_nodes: ['virt66.example.com']  ignore_nodes: []  overprovisioning: Truebalancing:  enable: True  enforce_affinity: False  parallel: False  live: True  with_local_disks: True  balance_types: ['vm', 'ct']  max_job_validation: 1800  balanciness: 5  method: memory  mode: usedservice:  daemon: True  schedule:    interval: 12    format: hours  delay:    enable: False    time: 1    format: hours  log_level: INFO

The following options and parameters are currently supported:

ProxLB provides an advanced mechanism to define affinity and anti-affinity rules, enabling precise control over virtual machine (VM) placement. These rules help manage resource distribution, improve high availability configurations, and optimize performance within a Proxmox Virtual Environment (PVE) cluster. By leveraging Proxmox’s integrated access management, ProxLB ensures that users can only define and manage rules for guests they have permission to access.

ProxLB implements affinity and anti-affinity rules through a tag-based system within the Proxmox web interface. Each guest (virtual machine or container) can be assigned specific tags, which then dictate its placement behavior. This method maintains a streamlined and secure approach to managing VM relationships while preserving Proxmox’s inherent permission model.

Affinity rules are used to group certain VMs together, ensuring that they run on the same host whenever possible. This can be beneficial for workloads requiring low-latency communication, such as clustered databases or application servers that frequently exchange data.

To define an affinity rule which keeps all guests assigned to this tag together on a node, users assign a tag with the prefixplb_affinity_$TAG:

plb_affinity_talos

As a result, ProxLB will attempt to place all VMs with theplb_affinity_web tag on the same host (see also the attached screenshot with the same node).

Conversely, anti-affinity rules ensure that designated VMs do not run on the same physical host. This is particularly useful for high-availability setups, where redundancy is crucial. Ensuring that critical services are distributed across multiple hosts reduces the risk of a single point of failure.

To define an anti-affinity rule that ensures to not move systems within this group to the same node, users assign a tag with the prefix:

plb_anti_affinity_ntp

As a result, ProxLB will try to place the VMs with theplb_anti_affinity_ntp tag on different hosts (see also the attached screenshot with the different nodes).

Note: While this ensures that ProxLB tries distribute these VMs across different physical hosts within the Proxmox cluster this may not always work. If you have more guests attached to the group than nodes in the cluster, we still need to run them anywhere. If this case occurs, the next one with the most free resources will be selected.

Guests, such as VMs or CTs, can also be completely ignored. This means, they won't be affected by any migration (even when (anti-)affinity rules are enforced). To ensure a proper resource evaluation, these guests are still collected and evaluated but simply skipped for balancing actions. Another thing is the implementation. While ProxLB might have a very restricted configuration file including the file permissions, this file is only read- and writeable by the Proxmox administrators. However, we might have user and groups who want to define on their own that their systems shouldn't be moved. Therefore, these users can simpy set a specific tag to the guest object - just like the (anti)affinity rules.

To define a guest to be ignored from the balancing, users assign a tag with the prefixplb_ignore_$TAG:

plb_ignore_dev

As a result, ProxLB will not migrate this guest with theplb_ignore_dev tag to any other node.

Note: Ignored guests are really ignored. Even by enforcing affinity rules this guest will be ignored.

Guests, such as VMs or CTs, can also be pinned to specific (and multiple) nodes in the cluster. This might be usefull when running applications with some special licensing requirements that are only fulfilled on certain nodes. It might also be interesting, when some physical hardware is attached to a node, that is not available in general within the cluster.

To pin a guest to a specific cluster node, users assign a tag with the prefixplb_pin_$nodename to the desired guest:

plb_pin_node03

As a result, ProxLB will pin the guestdev-vm01 to the nodevirt03.

You can also repeat this step multiple times for different node names to create a potential group of allowed hosts where a the guest may be served on. In this case, ProxLB takes the node with the lowest used resources according to the defined balancing values from this group.

Note: The given node names from the tag are validated. This means, ProxLB validated if the given node name is really part of the cluster. In case of a wrongly defined or unavailable node name it continous to use the regular processes to make sure the guest keeps running.

Themaintenance_nodes option allows operators to designate one or more Proxmox nodes for maintenance mode. When a node is set to maintenance, no new guest workloads will be assigned to it, and all existing workloads will be migrated to other available nodes within the cluster. This process ensures that (anti)-affinity rules and resource availability are respected, preventing disruptions while maintaining optimal performance across the infrastructure.

Within the sectionproxmox_cluster you can define the keymaintenance_nodes as a list object. Simply add/remove one or more nodes with their equal name in the cluster and restart the daemon.

proxmox_cluster:  maintenance_nodes: ['virt66.example.com']

Afterwards, all guest objects will be moved to other nodes in the cluster by ensuring the best balancing.

Bugs can be reported via the GitHub issue trackerhere. You may also report bugs via email or deliver PRs to fix them on your own. Therefore, you might also see the contributing chapter.

Feel free to add further documentation, to adjust already existing one or to contribute with code. Please take care about the style guide and naming conventions. You can find more in ourCONTRIBUTING.md file.

You can also find additional and more detailed documentation within thedocs/ directory.

If you need assistance or have any questions, we offer support through our dedicatedchat room in Matrix orDiscord. Join our community for real-time help, advice, and discussions. The Matrix and Discord room are bridged to ensure that the communication is not splitted - so simply feel free to join which fits most to you!

Connect with us in our dedicated chat room for immediate support and live interaction with other users and developers. You can also visit ourGitHub Community to post your queries, share your experiences, and get support from fellow community members and moderators. You may also just open directly an issuehere on GitHub.

Note: Please always keep in mind that this is a one-man show project without any further help. This includes coding, testing, packaging and all the infrastructure around it to keep this project up and running.

• Florian Paul Azim Hoberg @gyptazy (https://gyptazy.com)