ProxmoxVE on Wael Nasreddine

Proxmox VM - Ensure SCSI0 in Boot Order for Talos OS

Mon, 01 Jan 0001 00:00:00 +0000

Proxmox VM - Ensure SCSI0 in Boot Order for Talos OS Link to heading

Symptom Link to heading

After installing Talos Linux to a Proxmox VM's SCSI disk (e.g., scsi0) and removing the installation ISO, the VM may fail to boot, potentially getting stuck in a PXE/no boot loop.

Cause Link to heading

The Proxmox VM's boot order might not have the SCSI disk (e.g., scsi0) enabled or prioritized. This can happen if disks were deleted and re-added during experimentation.

ProxmoxVE - Add Certificate via ACME

Mon, 01 Jan 0001 00:00:00 +0000

ProxmoxVE - Add Certificate via ACME Link to heading

Problem Link to heading

I need to get a valid TLS certificate for my ProxmoxVE so browser don't complain.

Solution Link to heading

The server certificate can be configured to use ACME to install a certificate. Follow this documentation to configure it.

ProxmoxVE - Build Log

ProxmoxVE - Boot without starting any VM

Mon, 01 Jan 0001 00:00:00 +0000

ProxmoxVE - Boot without starting any VM Link to heading

Problem Link to heading

I want to boot ProxmoxVE without booting any guest in order to perform a maintenance task.

Solution Link to heading

During boot interrupt systemd (proxmox-boot) so it does not boot
Press e to edit the command line
Type in the following at the very end of the line: systemd.mask=pve-guests.service
Hit Enter to boot without starting any guests.

ProxmoxVE - Shutdown timeout

Mon, 01 Jan 0001 00:00:00 +0000

Problem Link to heading

In order to give important VM time to shutdown properly, I need to define shutdown timeout for each important VM.

The default timeout is 180s

The timeouts are defined per VM and not per Group. Since TrueNAS takes a while to shut down and is crucial for it to take its time shutting down, I will increase its timeout to 300 seconds. The default timeout is good enough for the rest of the VMs.

ProxmoxVE - Start and Shutdown order

Mon, 01 Jan 0001 00:00:00 +0000

Problem Link to heading

I need to define the Start and Shutdown order for all of my VMs in order to ensure dependencies between vms.

Solution Link to heading

The order defined below per group and not per VM although each VM in the group needs to input the same order number.

Group 5: Basic Infrastructure
- Basic DNS
Group 10: Infrastructure
- TrueNAS
- Tailscale
Group 15: Enhanced Infrastructure
- Full Featured DNS (it depends on NFS to mount local zone files.)
Group 20: Databases and caches
- Postgres
- Redis
- Etc..
Group 30: Kubernetes Control Plane
- All clusters
Group 40: Kubernetes Worker
- All clusters

ProxmoxVE - Startup delay

Mon, 01 Jan 0001 00:00:00 +0000

Problem Link to heading

In order to guarantee dependent VMs boot before they are needed, I need to define a delay to wait before booting the top VMs

Solution Link to heading

The delays are defined per VM and not per Group.

Basic DNS: 60s
- Allow basic DNS to load correctly and become available as other VMs depends on it.
TrueNAS: 60s
- TrueNAS boots pretty quickly (unlike shutdown), so give it enough time to boot before other VMs.

ProxmoxVE - TrueNAS VM - Disable Memory Ballooning

Mon, 01 Jan 0001 00:00:00 +0000

ProxmoxVE - TrueNAS VM - Disable Memory Ballooning Link to heading

ProxmoxVE's memory ballooning feature allows a virtual machine's operating system to dynamically adjust its reported memory usage. This enables the hypervisor (ProxmoxVE) to reclaim memory from the VM if it's not actively being used by the guest OS, and allocate it to other VMs or the host.

However, for TrueNAS virtual machines, memory ballooning should be disabled.

ProxmoxVE - Turn off laptop screen

Mon, 01 Jan 0001 00:00:00 +0000

ProxmoxVE - Turn off laptop screen Link to heading

Problem Link to heading

I have a couple of ProxmoxVE servers running on old laptops and I noticed that the screen never turns off.

Solution Link to heading

The solution is to leverage the consoleblank kernel command line flag:

Edit /etc/kernel/cmdline
Append consoleblank=60 to the end of the line
Run proxmox-boot-tool refresh
Reboot

ProxmoxVE - Build Log
ProxmoxVE - Prevent sleep on lid close

ProxmoxVE - VM - Talos Linux - New Kubernetes Cluster

Mon, 01 Jan 0001 00:00:00 +0000

ProxmoxVE - VM - Talos Linux - New Kubernetes Cluster Link to heading

Talos runs great on ProxmoxVE. Make sure to add the system extension siderolabs/qemu-guest-agent when producing the Talos - Generate ISO and Installer with Custom Extensions via Image Factory. This enhances VM manageability from Proxmox (e.g., proper shutdowns, IP reporting in Proxmox UI).

The required steps for a new Kubernetes Cluster are:

Generate an ISO and Download it on ProxmoxVE.
Setup the Control Plane of Kubernetes.
Create one Talos VM for each Talos Control Plane node.
- Tip: Kubernetes recommends an odd number of nodes. Typically one node for development and three nodes for production for high availability.

Generate an ISO and Download it on ProxmoxVE Link to heading

Generate an ISO by following Talos - Generate ISO and Installer with Custom Extensions via Image Factory.
Tip: Pin the Talos version in your image factory command for reproducible builds.
Download the ISO on ProxmoxVE

Create Talos VM(s) Link to heading

Create a new VM, and be sure to follow the Talos Linux - System Requirements.
Use advanced configuration and disable memory ballooning, that will not be useful with Kubernetes and could in fact be harmful.
Choose OVMF over SeaBIOS as it loads a nicer console font by default. See Proxmox VM - UEFI vs SeaBIOS Consideration.
CPU Type: Set CPU type to host in Proxmox VM options for best performance, unless live migration to different CPU hosts is a strict requirement.
Network Adapter: Use virtio (paravirtualized) for the network device model for best performance.
Resource Allocation: Plan CPU cores and RAM for each VM based on expected load. For control planes: typically 2+ cores, 4GB+ RAM. Workers depend on workload.
(Optional) VM Template: After setting up and testing one worker VM (before applying Talos config), consider converting it to a Proxmox template for faster worker node deployment. You'd still need to boot with the ISO and apply the Talos config.
Gather the NIC Mac Address from the VM Hardware page, this is needed to setup a static IP on the router.
Add the node to the router with fixed IP following UniFi - Add DHCP Reservation for MAC.
Start the VM
Follow the heading below for configuring the cluster.

Configure the cluster Link to heading

Write the controlplane-patch.yaml and worker-patch.yaml with the contents below.
Generate the secrets with talosctl gen secrets -o secrets.yaml
Security: Store secrets.yaml securely. It contains sensitive PKI data. Consider encrypting it at rest (e.g., using GPG, HashiCorp Vault, or similar). Back it up to a secure location.
Generate the configuration with the command talosctl gen config --with-secrets secrets.yaml --config-patch-control-plane @controlplane-patch.yaml --config-patch-worker @worker-patch.yaml <cluster name> https://<IP_of_first_node>:6443 --output-dir . --install-image <url-to-installer-image-from-factory>
- Note: <IP_of_first_node> in the URL for talosctl gen config is used as the initial endpoint for the generated talosconfig and as one of the default cert SANs. Your controlplane-patch.yaml correctly overrides/supplements apiServer.certSANs and defines controlPlane.endpoint to your VIP (<VIP>), which is good practice.
The command generates controlplane.yaml, worker.yaml, and talosconfig. The .yaml files are the machine configs for your nodes.
For each control plane node, apply the config with the command and wait for the node to become ready talosctl apply-config --insecure --file controlplane.yaml --nodes <IP>
First node should start saying "waiting for talosctl bootstrap"
Other nodes are ready as soon you see the number of machines increment on the first node.
Worker plane nodes can be applied all at once with the command talosctl apply-config --insecure --file worker.yaml --nodes <IP> --nodes <IP>
Configure talosconfig with the IP of the first node. Do not add other nodes yet (see Talos - Bootstrap - x509 Unknown Authority)
talosctl config endpoint <IP_of_one_control_plane_node>
talosctl config node <IP_of_one_control_plane_node>
Either copy talosconfig to ~/.talos/config or export TALOSCONFIG environment variable to point to its location.
Security: The talosconfig file contains client certificates that grant administrative access to your cluster. Protect it accordingly.
Bootstrap the cluster with talosctl bootstrap
Wait for talosctl health -n <IP> for each IP to report healthy.
Generate the proper kube config with the command talosctl kubeconfig. This command merges with ~/.kube/config if one is found and automatically sets the context to the cluster so it's ready to use.
Important: After bootstrap and verifying health, update your talosconfig to include all control plane node endpoints for high availability when using talosctl: talosctl config endpoint <CP1_IP>,<CP2_IP>,<CP3_IP> --nodes <CP1_IP>,<CP2_IP>,<CP3_IP> (Or edit the talosconfig YAML file directly to update contexts.<your-cluster>.endpoints and contexts.<your-cluster>.nodes). This ensures talosctl can reach the Talos API even if one control plane node is down.
Run kubectl get node -o wide to confirm things are working fine.
Ensure no node is tainted as uninitialized, use the following command: kubectl get nodes -o custom-columns=NODE:.metadata.name,TAINTS:.spec.taints
Talos Linux - Cloud Controller Manager (or CCM)

controlplane-patch.yaml

ProxmoxVE - VM Performance Tuning

Mon, 01 Jan 0001 00:00:00 +0000

ProxmoxVE - VM Performance Tuning Link to heading

Proxmox VE, being based on KVM/QEMU, offers numerous avenues for tuning Virtual Machine (VM) performance. Optimal settings often depend on the specific workload of the VM (e.g., I/O intensive, CPU intensive, network intensive). This note covers general best practices and specific configurations.

1. CPU Configuration Link to heading

CPU Type:
host: This is generally recommended for best performance. It passes through the host CPU's features and model directly to the VM, avoiding emulation overhead and ensuring access to all instruction sets (e.g., AVX, AES-NI).
Avoid older emulated CPU types unless required for specific compatibility (e.g., live migration to a host with a different CPU).
Cores & Sockets:
Start with what the VM workload reasonably needs. Over-allocating CPU cores can sometimes lead to scheduler contention on the host, potentially degrading performance for all VMs.
For most VMs, configuring sockets=1 and adjusting the number of cores is straightforward. NUMA considerations might influence socket/core layout for very large VMs on multi-socket hosts.
NUMA Pinning: For VMs with significant RAM and CPU allocated on NUMA-aware host hardware, enabling the "NUMA" option in Proxmox and ensuring the VM's vCPUs and memory are pinned to the same NUMA node can improve memory access latency and performance. This is an advanced setting.

2. Memory Configuration Link to heading

Static Allocation: For performance-sensitive VMs, especially those running databases or memory-intensive applications (like TrueNAS with ZFS), assign static memory.
Disable Ballooning: As detailed in ProxmoxVE - TrueNAS VM - Disable Memory Ballooning, memory ballooning can be detrimental for workloads that manage their own memory carefully (e.g., ZFS ARC, database buffer pools). Uncheck the "Ballooning Device" for such VMs.
Hugepages: For certain applications (e.g., databases, Java applications), transparent hugepages (THP) or explicitly configured hugepages can improve memory performance by reducing TLB misses. This is configured at the host level and sometimes within the guest OS.
Proxmox allows enabling hugepages for VMs (e.g., "KSM and Hugepages" settings).

3. Storage Performance Link to heading

Disk Image Format:
raw: Generally offers the best performance as it has minimal overhead.
qcow2: Offers features like snapshots and thin provisioning but can have a slight performance overhead compared to raw. If using qcow2, preallocation can improve performance.
Storage Type on Host:
Local SSDs (NVMe or SATA) will provide the best I/O performance for VM disks.
Ceph or ZFS (if Proxmox is installed on ZFS) can provide robust and performant shared storage, but their performance depends heavily on their own configuration (e.g., number of OSDs, SSD journaling, network).
VirtIO Drivers:
VirtIO SCSI / VirtIO Block (virtio-blk): Both are paravirtualized drivers offering good performance.
- VirtIO SCSI (often with VirtIO SCSI single controller type) is generally recommended for flexibility and features like TRIM/UNMAP support.
- Enable IO Thread for VirtIO SCSI disks. This dedicates a separate I/O thread per disk, which can significantly improve I/O throughput and reduce latency, especially for multi-core VMs and fast storage.
Ensure VirtIO drivers are installed in the guest OS (most modern Linux distros include them; Windows requires separate driver installation).
Cache Mode:
none (or writethrough for certain data integrity needs): Often recommended for VMs with applications that manage their own caching (e.g., databases) or when data integrity is paramount. This bypasses the host page cache.
writeback: Can offer higher performance by using the host page cache, but carries a risk of data loss if the host crashes before data is flushed to disk (unless using features like Ceph's RBD caching with appropriate safety measures).
Carefully consider the implications of cache modes based on the workload and storage backend.
Discard/TRIM: Enable the "Discard" option for virtual disks on SSD-backed storage (and ensure the guest OS also issues TRIM commands). This helps SSDs maintain performance by allowing them to reclaim unused blocks.

4. Network Performance Link to heading

VirtIO NIC: The VirtIO (paravirtualized) network device type offers the best performance for most VMs. Ensure guest OS drivers are installed.
Multiqueue VirtIO: Enable the "Multiqueue" option for VirtIO NICs (and ensure the guest OS supports it, e.g., Linux kernel 3.8+). This allows network packet processing to be distributed across multiple vCPUs, significantly improving network throughput for multi-core VMs. Set the number of queues appropriately (e.g., to match the number of vCPUs, up to a reasonable limit).
vHostNet (Host-side processing): While not a direct Proxmox setting, understanding that VirtIO networking can leverage vHostNet in the host kernel can offload some network processing from QEMU, improving efficiency.
SR-IOV / PCI Passthrough: For extreme network performance needs, passing through a physical NIC or a Virtual Function (VF) via SR-IOV to the VM can provide near bare-metal network performance by bypassing the hypervisor's network stack.

5. Guest OS Optimizations Link to heading

Install Guest Utilities/Drivers:
qemu-guest-agent: Essential for proper communication with the hypervisor (e.g., for clean shutdowns, IP address reporting, freezing filesystem for snapshots).
Latest VirtIO drivers for Windows.
Kernel Tuning (Linux Guests): Adjust kernel parameters within the guest (e.g., I/O schedulers, network buffer sizes, swappiness) based on the workload.
Disable Unused Services: Minimize resource consumption within the guest.

6. General Proxmox Host Considerations Link to heading

Host Kernel: Keep Proxmox VE updated to benefit from the latest KVM/QEMU features, bug fixes, and performance improvements.
Host Resource Monitoring: Regularly monitor host CPU, memory, disk I/O, and network utilization to identify bottlenecks.
Limit KSM (Kernel Samepage Merging): While KSM can save memory by deduplicating identical memory pages between VMs, it consumes CPU resources. If CPU is a bottleneck or for latency-sensitive VMs, consider disabling KSM or tuning its parameters. It's often disabled by default in recent Proxmox versions or can be toggled per VM.

SuperMicro - Install ProxmoxVE

Mon, 01 Jan 0001 00:00:00 +0000

SuperMicro - Install ProxmoxVE Link to heading

Problem Link to heading

I need to install ProxmoxVE on the server.

Solution Link to heading

SuperMicro - BMC - Boot from USB
Perform a normal ProxmoxVE installation using the two SSD in RAID1 (mirror)

Setting up SuperMicro SuperStorage 6048R-E1CR36H
SuperMicro - BMC - Boot from USB

ProxmoxVE on Wael Nasreddine

Proxmox VM - Ensure SCSI0 in Boot Order for Talos OS

Proxmox VM - Ensure SCSI0 in Boot Order for Talos OS Link to heading

Symptom Link to heading

Cause Link to heading

ProxmoxVE - Add Certificate via ACME

ProxmoxVE - Add Certificate via ACME Link to heading

Problem Link to heading

Solution Link to heading

Related Link to heading

ProxmoxVE - Boot without starting any VM

ProxmoxVE - Boot without starting any VM Link to heading

Problem Link to heading

Solution Link to heading

Related Link to heading

ProxmoxVE - Shutdown timeout

Problem Link to heading

Solution Link to heading

ProxmoxVE - Start and Shutdown order

Problem Link to heading

Solution Link to heading

Related Link to heading

ProxmoxVE - Startup delay

Problem Link to heading

Solution Link to heading

Related Link to heading

ProxmoxVE - TrueNAS VM - Disable Memory Ballooning

ProxmoxVE - TrueNAS VM - Disable Memory Ballooning Link to heading

ProxmoxVE - Turn off laptop screen

ProxmoxVE - Turn off laptop screen Link to heading

Problem Link to heading

Solution Link to heading

Related Link to heading

ProxmoxVE - VM - Talos Linux - New Kubernetes Cluster

ProxmoxVE - VM - Talos Linux - New Kubernetes Cluster Link to heading

Generate an ISO and Download it on ProxmoxVE Link to heading

Create Talos VM(s) Link to heading

Configure the cluster Link to heading

ProxmoxVE - VM Performance Tuning

ProxmoxVE - VM Performance Tuning Link to heading

1. CPU Configuration Link to heading

2. Memory Configuration Link to heading

3. Storage Performance Link to heading

4. Network Performance Link to heading

5. Guest OS Optimizations Link to heading

6. General Proxmox Host Considerations Link to heading

Related Link to heading

SuperMicro - Install ProxmoxVE

SuperMicro - Install ProxmoxVE Link to heading

Problem Link to heading

Solution Link to heading

Related Link to heading