Repository files navigation

The fastest and most memory efficient lattice Boltzmann CFD software, running on all GPUs and CPUs viaOpenCL. Free for non-commercial use.

(click on images to show videos on YouTube)

Read theFluidX3D Documentation!

• CFD model: lattice Boltzmann method (LBM)

  • streaming (part 2/2)

    f₀^temp(x,t) =f₀(x,t)
    f_i^temp(x,t) =f_{(t%2 ?i : (i%2 ?i+1 :i-1))}(i%2 ?x :x-e_i,t)   for  i ∈ [1,q-1]

  • collision

    ρ(x,t) = (Σ_if_i^temp(x,t)) + 1
    
    u(x,t) =¹∕_ρ(x,t) Σ_ic_if_i^temp(x,t)
    
    f_i^eq-shifted(x,t) =w_iρ · (^(u_°c_i)2∕_(2c⁴) -^(u_°u)∕_(2c²) +^(u_°c_i)∕_c²) +w_i (ρ-1)
    
    f_i^temp(x,t+Δt) =f_i^temp(x,t) +Ω_i(f_i^temp(x,t),f_i^eq-shifted(x,t),τ)

  • streaming (part 1/2)

    f₀(x,t+Δt) =f₀^temp(x,t+Δt)
    f_{(t%2 ? (i%2 ?i+1 :i-1) :i)}(i%2 ?x+e_i :x,t+Δt) =f_i^temp(x,t+Δt)   for  i ∈ [1,q-1]

  • variables andnotation

    variable	SI units	defining equation	description
    x	m	x = (x,y,z)T	3D position in Cartesian coordinates
    t	s	-	time
    ρ	kg∕m³	ρ = (Σifi)+1	mass density of fluid
    p	kg∕m s²	p =c²ρ	pressure of fluid
    u	m∕s	u =1∕ρ Σicifi	velocity of fluid
    ν	m²∕s	ν =μ∕ρ	kinematic shear viscosity of fluid
    μ	kg∕m s	μ =ρν	dynamic viscosity of fluid
    fi	kg∕m³	-	shifted density distribution functions (DDFs)
    Δx	m	Δx = 1	lattice constant (in LBM units)
    Δt	s	Δt = 1	simulation time step (in LBM units)
    c	m∕s	c =1∕√3Δx∕Δt	lattice speed of sound (in LBM units)
    i	1	0 ≤i <q	LBM streaming direction index
    q	1	q ∈ { 9,15,19,27 }	number of LBM streaming directions
    ei	m	D2Q9 / D3Q15/19/27	LBM streaming directions
    ci	m∕s	ci =ei∕Δt	LBM streaming velocities
    wi	1	Σiwi = 1	LBM velocity set weights
    Ωi	kg∕m³	SRT or TRT	LBM collision operator
    τ	s	τ =ν∕c² +Δt∕2	LBM relaxation time

  • velocity sets: D2Q9, D3Q15, D3Q19 (default), D3Q27

  • collision operators: single-relaxation-time (SRT/BGK) (default), two-relaxation-time (TRT)

  • DDF-shifting and other algebraic optimization to minimize round-off error

• optimized to minimize VRAM footprint to 1/6 of other LBM codes

  • traditional LBM (D3Q19) with FP64 requires ~344 Bytes/cell
    

    • 🟧🟧🟧🟧🟧🟧🟧🟧🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟨🟨🟨🟨🟨🟨🟨🟨🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥
      (density 🟧, velocity 🟦, flags 🟨, 2 copies of DDFs 🟩/🟥; each square = 1 Byte)
    • allows for 3 Million cells per 1 GB VRAM

  • FluidX3D (D3Q19) requires only 55 Bytes/cell withEsoteric-Pull+FP16
    

    • 🟧🟧🟧🟧🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟨🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩
      (density 🟧, velocity 🟦, flags 🟨, DDFs 🟩; each square = 1 Byte)

    • allows for 19 Million cells per 1 GB VRAM

    • in-place streaming withEsoteric-Pull: eliminates redundant copy of density distribution functions (DDFs) in memory; almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries; offers optimal memory access patterns for single-cell in-place streaming

    • decoupled arithmetic precision (FP32) and memory precision (FP32 or FP16S or FP16C): all arithmetic is done in FP32 for compatibility on all hardware, but DDFs in memory can be compressed to FP16S or FP16C: almost cuts memory demand in half again and almost doubles performance, without impacting overall accuracy for most setups

    • only 8 flag bits per lattice point (can be used independently / at the same time)

      • TYPE_S (stationary or moving) solid boundaries
      • TYPE_E equilibrium boundaries (inflow/outflow)
      • TYPE_T temperature boundaries
      • TYPE_F free surface (fluid)
      • TYPE_I free surface (interface)
      • TYPE_G free surface (gas)
      • TYPE_X remaining for custom use or further extensions
      • TYPE_Y remaining for custom use or further extensions

  • large cost saving: comparison of maximum single-GPU grid resolution for D3Q19 LBM

    GPU VRAM capacity	1 GB	2 GB	3 GB	4 GB	6 GB	8 GB	10 GB	11 GB	12 GB	16 GB	20 GB	24 GB	32 GB	40 GB	48 GB	64 GB	80 GB	94 GB	128 GB	192 GB	256 GB
    approximate GPU price	$25 GT 210	$25 GTX 950	$12 GTX 1060	$50 GT 730	$35 GTX 1060	$70 RX 470	$500 RTX 3080	$240 GTX 1080 Ti	$75 Tesla M40	$75 Instinct MI25	$900 RX 7900 XT	$205 Tesla P40	$600 Instinct MI60	$5500 A100	$2400 RTX 8000	$10k Instinct MI210	$11k A100	>$40k H100 NVL	? GPU Max 1550	~$10k MI300X	-
    traditional LBM (FP64)	144³	182³	208³	230³	262³	288³	312³	322³	330³	364³	392³	418³	460³	494³	526³	578³	624³	658³	730³	836³	920³
    FluidX3D (FP32/FP32)	224³	282³	322³	354³	406³	448³	482³	498³	512³	564³	608³	646³	710³	766³	814³	896³	966³	1018³	1130³	1292³	1422³
    FluidX3D (FP32/FP16)	266³	336³	384³	424³	484³	534³	574³	594³	610³	672³	724³	770³	848³	912³	970³	1068³	1150³	1214³	1346³	1540³	1624³

• cross-vendor multi-GPU support on a single computer/server

  • domain decomposition allows pooling VRAM from multiple GPUs for much larger grid resolution
  • GPUs don't have to be identical (not even from the same vendor), but similar VRAM capacity/bandwidth is recommended
  • domain communication architecture (simplified)

    ++   .-----------------------------------------------------------------.   ++++   |                              GPU 0                              |   ++++   |                          LBM Domain 0                           |   ++++   '-----------------------------------------------------------------'   ++++              |                 selective                /|\             ++++             \|/               in-VRAM copy               |              ++++        .-------------------------------------------------------.        ++++        |               GPU 0 - Transfer Buffer 0               |        ++++        '-------------------------------------------------------'        ++!!                            |     PCIe     /|\                           !!!!                           \|/    copy      |                            !!@@        .-------------------------.   .-------------------------.        @@@@        | CPU - Transfer Buffer 0 |   | CPU - Transfer Buffer 1 |        @@@@        '-------------------------'\ /'-------------------------'        @@@@                           pointer  X   swap                             @@@@        .-------------------------./ \.-------------------------.        @@@@        | CPU - Transfer Buffer 1 |   | CPU - Transfer Buffer 0 |        @@@@        '-------------------------'   '-------------------------'        @@!!                           /|\    PCIe      |                            !!!!                            |     copy     \|/                           !!++        .-------------------------------------------------------.        ++++        |               GPU 1 - Transfer Buffer 1               |        ++++        '-------------------------------------------------------'        ++++             /|\                selective                 |              ++++              |                in-VRAM copy              \|/             ++++   .-----------------------------------------------------------------.   ++++   |                              GPU 1                              |   ++++   |                          LBM Domain 1                           |   ++++   '-----------------------------------------------------------------'   ++##                                    |                                    ####                      domain synchronization barrier                     ####                                    |                                    ##||   -------------------------------------------------------------> time   ||

  • domain communication architecture (detailed)

    ++   .-----------------------------------------------------------------.   ++++   |                              GPU 0                              |   ++++   |                          LBM Domain 0                           |   ++++   '-----------------------------------------------------------------'   ++++     |  selective in- /|\  |  selective in- /|\  |  selective in- /|\    ++++    \|/ VRAM copy (X)  |  \|/ VRAM copy (Y)  |  \|/ VRAM copy (Z)  |     ++++   .---------------------.---------------------.---------------------.   ++++   |    GPU 0 - TB 0X+   |    GPU 0 - TB 0Y+   |    GPU 0 - TB 0Z+   |   ++++   |    GPU 0 - TB 0X-   |    GPU 0 - TB 0Y-   |    GPU 0 - TB 0Z-   |   ++++   '---------------------'---------------------'---------------------'   ++!!          | PCIe /|\            | PCIe /|\            | PCIe /|\         !!!!         \|/ copy |            \|/ copy |            \|/ copy |          !!@@   .---------. .---------.---------. .---------.---------. .---------.   @@@@   | CPU 0X+ | | CPU 1X- | CPU 0Y+ | | CPU 3Y- | CPU 0Z+ | | CPU 5Z- |   @@@@   | CPU 0X- | | CPU 2X+ | CPU 0Y- | | CPU 4Y+ | CPU 0Z- | | CPU 6Z+ |   @@@@   '---------\ /---------'---------\ /---------'---------\ /---------'   @@@@      pointer X swap (X)    pointer X swap (Y)    pointer X swap (Z)     @@@@   .---------/ \---------.---------/ \---------.---------/ \---------.   @@@@   | CPU 1X- | | CPU 0X+ | CPU 3Y- | | CPU 0Y+ | CPU 5Z- | | CPU 0Z+ |   @@@@   | CPU 2X+ | | CPU 0X- | CPU 4Y+ | | CPU 0Y- | CPU 6Z+ | | CPU 0Z- |   @@@@   '---------' '---------'---------' '---------'---------' '---------'   @@!!         /|\ PCIe |            /|\ PCIe |            /|\ PCIe |          !!!!          | copy \|/            | copy \|/            | copy \|/         !!++   .--------------------..---------------------..--------------------.   ++++   |   GPU 1 - TB 1X-   ||    GPU 3 - TB 3Y-   ||   GPU 5 - TB 5Z-   |   ++++   :====================::=====================::====================:   ++++   |   GPU 2 - TB 2X+   ||    GPU 4 - TB 4Y+   ||   GPU 6 - TB 6Z+   |   ++++   '--------------------''---------------------''--------------------'   ++++    /|\ selective in-  |  /|\ selective in-  |  /|\ selective in-  |     ++++     |  VRAM copy (X) \|/  |  VRAM copy (Y) \|/  |  VRAM copy (Z) \|/    ++++   .--------------------..---------------------..--------------------.   ++++   |        GPU 1       ||        GPU 3        ||        GPU 5       |   ++++   |    LBM Domain 1    ||    LBM Domain 3     ||    LBM Domain 5    |   ++++   :====================::=====================::====================:   ++++   |        GPU 2       ||        GPU 4        ||        GPU 6       |   ++++   |    LBM Domain 2    ||    LBM Domain 4     ||    LBM Domain 6    |   ++++   '--------------------''---------------------''--------------------'   ++##              |                     |                     |              ####              |      domain synchronization barriers      |              ####              |                     |                     |              ##||   -------------------------------------------------------------> time   ||

• peak performance on GPUs (datacenter/gaming/professional/laptop)

  • single-GPU/CPU benchmarks
  • multi-GPU benchmarks
• powerful model extensions

  • boundary types
    • stationary mid-grid bounce-back boundaries (stationary solid boundaries)
    • moving mid-grid bounce-back boundaries (moving solid boundaries)
    • equilibrium boundaries (non-reflective inflow/outflow)
    • temperature boundaries (fixed temperature)
  • global force per volume (Guo forcing), can be modified on-the-fly
  • local force per volume (force field)
    • optional computation of forces from the fluid on solid boundaries
  • state-of-the-artfree surface LBM (FSLBM) implementation:
    • volume-of-fluid model
    • fully analytic PLIC for efficient curvature calculation
    • improved mass conservation
    • ultra efficient implementation with only4 kernels additionally tostream_collide() kernel
  • thermal LBM to simulate thermal convection
    • D3Q7 subgrid for thermal DDFs
    • in-place streaming withEsoteric-Pull for thermal DDFs
    • optionalFP16S or FP16C compression for thermal DDFs withDDF-shifting
  • Smagorinsky-Lilly subgrid turbulence LES model to keep simulations with very large Reynolds number stable

    Π_αβ = Σ_ie_iαe_iβ (f_i -f_i^eq-shifted)
    
    Q = Σ_αβΠ_αβ²
                         ______________________
    τ = ½ (τ₀ + √ τ₀² +^(16√2)∕_(3π²)^√Q∕_ρ )

  • particles with immersed-boundary method (either passive or 2-way-coupled, single-GPU only)

• FluidX3D can do simulations so large that storing the volumetric data for later rendering becomes unmanageable (like 120GB for a single frame, hundreds of TeraByte for a video)
• instead, FluidX3D allowsrendering raw simulation data directly in VRAM, so no large volumetric files have to be exported to the hard disk (see mytechnical talk)
• the rendering is so fast that it works interactively in real time for both rasterization and raytracing
• rasterization and raytracing are done in OpenCL and work on all GPUs, even the ones without RTX/DXR raytracing cores or without any rendering hardware at all (like A100, MI200, ...)
• if no monitor is available (like on a remote Linux server), there is anASCII rendering mode to interactively visualize the simulation in the terminal (even in WSL and/or through SSH)
• rendering is fully multi-GPU-parallelized via seamless domain decomposition rasterization
• with interactive graphics mode disabled, image resolution can be as large as VRAM allows for (4K/8K/16K and above)
• (interacitive) visualization modes:
  • flag wireframe / solid surface (and force vectors on solid cells or surface pressure if the extension is used)
  • velocity field (with slice mode)
  • streamlines (with slice mode)
  • velocity-colored Q-criterion isosurface
  • rasterized free surface withmarching-cubes
  • raytraced free surface with fast ray-grid traversal and marching-cubes, either 1-4 rays/pixel or 1-10 rays/pixel

• FluidX3D is written in OpenCL 1.2, so it runs on all hardware from all vendors (Nvidia, AMD, Intel, ...):
  • world's fastest datacenter GPUs: MI300X, H100 (NVL), A100, MI200, MI100, V100(S), GPU Max 1100, ...
  • gaming GPUs (desktop/laptop): Nvidia GeForce, AMD Radeon, Intel Arc
  • professional/workstation GPUs: Nvidia Quadro, AMD Radeon Pro / FirePro, Intel Arc Pro
  • integrated GPUs
  • CPUs (requiresinstallation of Intel CPU Runtime for OpenCL)
  • Intel Xeon Phi (requiresinstallation of Intel CPU Runtime for OpenCL)
  • smartphone ARM GPUs
• native cross-vendor multi-GPU implementation
  • uses PCIe communication, so no SLI/Crossfire/NVLink/InfinityFabric required
  • single-node parallelization, so no MPI installation required
  • GPUs don't even have to be from the same vendor, but similar memory capacity and bandwidth are recommended
• works onWindows andLinux with C++17, with limited support also formacOS andAndroid
• supportsimporting and voxelizing triangle meshes from binary.stl files, with fast GPU voxelization
• supportsexporting volumetric data as binary.vtk files
• supportsexporting triangle meshes as binary.vtk files
• supportsexporting rendered images as.png/.qoi/.bmp files; encoding runs in parallel on the CPU while the simulation on GPU can continue without delay

Here areperformance benchmarks on various hardware in MLUPs/s, or how many million lattice cells are updated per second. The settings used for the benchmark are D3Q19 SRT with no extensions enabled (only LBM with implicit mid-grid bounce-back boundaries) and the setup consists of an empty cubic box with sufficient size (typically 256³). Without extensions, a single lattice cell requires:

• a memory capacity of 93 (FP32/FP32) or 55 (FP32/FP16) Bytes
• a memory bandwidth of 153 (FP32/FP32) or 77 (FP32/FP16) Bytes per time step
• 363 (FP32/FP32) or 406 (FP32/FP16S) or 1275 (FP32/FP16C) FLOPs per time step (FP32+INT32 operations counted combined)

In consequence, the arithmetic intensity of this implementation is 2.37 (FP32/FP32) or 5.27 (FP32/FP16S) or 16.56 (FP32/FP16C) FLOPs/Byte. So performance is only limited by memory bandwidth. The table in the left 3 columns shows the hardware specs as found in the data sheets (theoretical peak FP32 compute performance, memory capacity, theoretical peak memory bandwidth). The right 3 columns show the measured FluidX3D performance for FP32/FP32, FP32/FP16S, FP32/FP16C floating-point precision settings, with the (roofline model efficiency) in round brackets, indicating how much % of theoretical peak memory bandwidth are being used.

If your GPU/CPU is not on the list yet, you can report your benchmarkshere.

gantttitle FluidX3D Performance [MLUPs/s] - FP32 arithmetic, (fastest of FP32/FP16S/FP16C) memory storagedateFormat XaxisFormat %s%%{init: {"gantt": {'titleTopMargin': 42,'topPadding': 70,'leftPadding': 260,'rightPadding': 5,'sectionFontSize': 20,'fontSize': 20,'barHeight': 20,'barGap': 3,'numberSectionStyles': 2},'theme': 'forest','themeVariables': {'sectionBkgColor': '#99999999','altSectionBkgColor': '#00000000','titleColor': '#AFAFAF','textColor': '#AFAFAF','taskTextColor': 'black','taskBorderColor': '#487E3A'}}}%%section MI300X41327 :crit, 0, 41327section MI250 (1 GCD)9030 :crit, 0, 9030section MI2109547 :crit, 0, 9547section MI1008542 :crit, 0, 8542section MI605111 :crit, 0, 5111section Radeon VII7778 :crit, 0, 7778section GPU Max 11006209 :done, 0, 6209section GH200 94GB GPU34689 : 0, 34689section H100 NVL32613 : 0, 32613section H100 SXM5 80GB HBM329561 : 0, 29561section H100 PCIe 80GB HBM2e20624 : 0, 20624section A100 SXM4 80GB18448 : 0, 18448section A100 PCIe 80GB17896 : 0, 17896section PG506-242/24315654 : 0, 15654section A100 SXM4 40GB16013 : 0, 16013section A100 PCIe 40GB16035 : 0, 16035section CMP 170HX12392 : 0, 12392section A309721 : 0, 9721section V100 SXM2 32GB8947 : 0, 8947section V100 PCIe 16GB10325 : 0, 10325section GV1006641 : 0, 6641section Titan V7253 : 0, 7253section P100 PCIe 16GB5950 : 0, 5950section P100 PCIe 12GB4141 : 0, 4141section GTX TITAN2500 : 0, 2500section K40m1868 : 0, 1868section K80 (1 GPU)1642 : 0, 1642section K20c1507 : 0, 1507section RX 7900 XTX7716 :crit, 0, 7716section PRO W79005939 :crit, 0, 5939section RX 7900 XT5986 :crit, 0, 5986section PRO W78004426 :crit, 0, 4426section RX 7900 GRE4570 :crit, 0, 4570section PRO W77002943 :crit, 0, 2943section RX 76002561 :crit, 0, 2561section PRO W76002287 :crit, 0, 2287section PRO W75001682 :crit, 0, 1682section RX 6900 XT4227 :crit, 0, 4227section RX 6800 XT4241 :crit, 0, 4241section PRO W68003361 :crit, 0, 3361section RX 6700 XT2908 :crit, 0, 2908section RX 6800M3213 :crit, 0, 3213section RX 6700M2429 :crit, 0, 2429section RX 66001839 :crit, 0, 1839section RX 6500 XT1030 :crit, 0, 1030section RX 5700 XT3253 :crit, 0, 3253section RX 57003167 :crit, 0, 3167section RX 5600 XT2214 :crit, 0, 2214section RX Vega 643227 :crit, 0, 3227section RX 5901688 :crit, 0, 1688section RX 580 4GB1848 :crit, 0, 1848section RX 580 2048SP 8GB1622 :crit, 0, 1622section R9 390X2217 :crit, 0, 2217section HD 7850635 :crit, 0, 635section Arc B580 LE4979 :done, 0, 4979section Arc A770 LE4568 :done, 0, 4568section Arc A750 LE4314 :done, 0, 4314section Arc A5803889 :done, 0, 3889section Arc Pro A40985 :done, 0, 985section Arc A3801115 :done, 0, 1115section RTX 509019141 : 0, 19141section RTX 508010304 : 0, 10304section RTX 50707238 : 0, 7238section RTX 409011496 : 0, 11496section RTX 6000 Ada10293 : 0, 10293section L40S7637 : 0, 7637section L407945 : 0, 7945section RTX 4080 Super8218 : 0, 8218section RTX 40807933 : 0, 7933section RTX 4070 Ti Super7295 : 0, 7295section RTX 4090M6901 : 0, 6901section RTX 4070 Super5554 : 0, 5554section RTX 40705016 : 0, 5016section RTX 4080M5114 : 0, 5114section RTX 4000 Ada4221 : 0, 4221section RTX 40603124 : 0, 3124section RTX 4070M3092 : 0, 3092section RTX 2000 Ada2526 : 0, 2526section RTX 3090 Ti10956 : 0, 10956section RTX 309010732 : 0, 10732section RTX 3080 Ti9832 : 0, 9832section RTX 3080 12GB9657 : 0, 9657section RTX A60008814 : 0, 8814section RTX 3080 10GB8118 : 0, 8118section RTX 3070 Ti6807 : 0, 6807section RTX 3080M Ti5908 : 0, 5908section RTX 30705096 : 0, 5096section RTX 3060 Ti5129 : 0, 5129section RTX A40004945 : 0, 4945section RTX A5000M4461 : 0, 4461section RTX 30604070 : 0, 4070section RTX 3060M4012 : 0, 4012section RTX 3050M Ti2341 : 0, 2341section RTX 3050M2339 : 0, 2339section Titan RTX7554 : 0, 7554section RTX 60006879 : 0, 6879section RTX 8000 Passive5607 : 0, 5607section RTX 2080 Ti6853 : 0, 6853section RTX 2080 Super5284 : 0, 5284section RTX 50004773 : 0, 4773section RTX 20804977 : 0, 4977section RTX 2070 Super4893 : 0, 4893section RTX 20705017 : 0, 5017section RTX 2060 Super5035 : 0, 5035section RTX 40004584 : 0, 4584section RTX 2060 KO3376 : 0, 3376section RTX 20603604 : 0, 3604section GTX 1660 Super3551 : 0, 3551section T42887 : 0, 2887section GTX 1660 Ti3041 : 0, 3041section GTX 16601992 : 0, 1992section GTX 1650M 896C1858 : 0, 1858section GTX 1650M 1024C1400 : 0, 1400section T500665 : 0, 665section Titan Xp5495 : 0, 5495section GTX 1080 Ti4877 : 0, 4877section GTX 10803182 : 0, 3182section GTX 1060 6GB1925 : 0, 1925section GTX 1060M1882 : 0, 1882section GTX 1050M Ti1224 : 0, 1224section P1000839 : 0, 839section GTX 980 Ti2703 : 0, 2703section GTX 9801965 : 0, 1965section GTX 9701721 : 0, 1721section M40001519 : 0, 1519section M60 (1 GPU)1571 : 0, 1571section GTX 960M872 : 0, 872section GTX 7701215 : 0, 1215section GTX 680 4GB1274 : 0, 1274section K2000444 : 0, 444section GT 630 (OEM)185 : 0, 185section NVS 2909 : 0, 9section Arise 10206 :active, 0, 6section M2 Ultra (76-CU, 192GB)8769 :active, 0, 8769section M2 Max (38-CU, 32GB)4641 :active, 0, 4641section M1 Ultra (64-CU, 128GB)8418 :active, 0, 8418section M1 Max (24-CU, 32GB)4496 :active, 0, 4496section M1 Pro (16-CU, 16GB)2329 :active, 0, 2329section M1 (8-CU, 16GB)759 :active, 0, 759section Radeon Graphics (7800X3D)498 :crit, 0, 498section 780M (Z1 Extreme)860 :crit, 0, 860section Vega 8 (4750G)511 :crit, 0, 511section Vega 8 (3500U)288 :crit, 0, 288section Arc 140V GPU (16GB)1282 :done, 0, 1282section Arc Graphics (Ultra 9 185H)724 :done, 0, 724section Iris Xe Graphics (i7-1265U)621 :done, 0, 621section UHD Xe 32EUs245 :done, 0, 245section UHD 770475 :done, 0, 475section UHD 630301 :done, 0, 301section UHD P630288 :done, 0, 288section HD 5500192 :done, 0, 192section HD 4600115 :done, 0, 115section Orange Pi 5 Mali-G610 MP4232 :active, 0, 232section Samsung Mali-G72 MP18230 :active, 0, 230section 2x EPYC 97545179 :crit, 0, 5179section 2x EPYC 96541814 :crit, 0, 1814section 2x EPYC 95542552 :crit, 0, 2552section 2x EPYC 7352739 :crit, 0, 739section 2x EPYC 7313498 :crit, 0, 498section 2x EPYC 7302784 :crit, 0, 784section 2x 6980P7875 :done, 0, 7875section 2x 6979P8135 :done, 0, 8135section 2x Platinum 8592+3135 :done, 0, 3135section 2x CPU Max 94802037 :done, 0, 2037section 2x Platinum 8480+2162 :done, 0, 2162section 2x Platinum 83801410 :done, 0, 1410section 2x Platinum 83581285 :done, 0, 1285section 2x Platinum 8256396 :done, 0, 396section 2x Platinum 8153691 :done, 0, 691section 2x Gold 6248R755 :done, 0, 755section 2x Gold 6128254 :done, 0, 254section Phi 7210415 :done, 0, 415section 4x E5-4620 v4460 :done, 0, 460section 2x E5-2630 v4264 :done, 0, 264section 2x E5-2623 v4125 :done, 0, 125section 2x E5-2680 v3304 :done, 0, 304section GH200 Neoverse-V21323 : 0, 1323section TR PRO 7995WX1715 :crit, 0, 1715section TR 3970X463 :crit, 0, 463section TR 1950X273 :crit, 0, 273section Ryzen 7800X3D363 :crit, 0, 363section Ryzen 5700X3D229 :crit, 0, 229section FX-610022 :crit, 0, 22section Athlon X2 QL-653 :crit, 0, 3section Ultra 7 258V287 :done, 0, 287section Ultra 9 185H317 :done, 0, 317section i9-14900K490 :done, 0, 490section i7-13700K504 :done, 0, 504section i7-1265U128 :done, 0, 128section i9-11900KB208 :done, 0, 208section i9-10980XE286 :done, 0, 286section E-2288G198 :done, 0, 198section i7-9700103 :done, 0, 103section i5-9600147 :done, 0, 147section i7-8700K152 :done, 0, 152section E-2176G201 :done, 0, 201section i7-7700HQ108 :done, 0, 108section E3-1240 v5141 :done, 0, 141section i5-5300U37 :done, 0, 37section i7-4770104 :done, 0, 104section i7-4720HQ80 :done, 0, 80section N28077 :done, 0, 7

Multi-GPU benchmarks are done at the largest possible grid resolution with cubic domains, and either 2x1x1, 2x2x1 or 2x2x2 of these domains together. The (percentages in round brackets) are single-GPUroofline model efficiency, and the (multiplicators in round brackets) are scaling factors relative to benchmarked single-GPU performance.

gantttitle FluidX3D Performance [MLUPs/s] - FP32 arithmetic, (fastest of FP32/FP16S/FP16C) memory storagedateFormat XaxisFormat %s%%{init: {"gantt": {'titleTopMargin': 42,'topPadding': 70,'leftPadding': 260,'rightPadding': 5,'sectionFontSize': 20,'fontSize': 20,'barHeight': 20,'barGap': 3,'numberSectionStyles': 2},'theme': 'forest','themeVariables': {'sectionBkgColor': '#99999999','altSectionBkgColor': '#00000000','titleColor': '#AFAFAF','textColor': '#AFAFAF','taskTextColor': 'black','taskBorderColor': '#487E3A'}}}%%section 8x Instinct MI300X204924 :crit, 0, 204924section 4x Instinct MI300X109546 :crit, 0, 109546section 2x Instinct MI300X61053 :crit, 0, 61053section 1x Instinct MI300X41327 :crit, 0, 41327section 4x Instinct MI250 (8 GCD)53521 :crit, 0, 53521section 2x Instinct MI250 (4 GCD)29627 :crit, 0, 29627section 1x Instinct MI250 (2 GCD17338 :crit, 0, 17338section 1x Instinct MI250 (1 GCD)9030 :crit, 0, 9030section 32x Instinct MI210 GigaIO50952 :crit, 0, 50952section 24x Instinct MI210 GigaIO45033 :crit, 0, 45033section 16x Instinct MI210 GigaIO37922 :crit, 0, 37922section 8x Instinct MI210 GigaIO27996 :crit, 0, 27996section 4x Instinct MI210 GigaIO17232 :crit, 0, 17232section 2x Instinct MI210 GigaIO13539 :crit, 0, 13539section 1x Instinct MI210 GigaIO9105 :crit, 0, 9105section 4x Instinct MI21031408 :crit, 0, 31408section 2x Instinct MI21016156 :crit, 0, 16156section 1x Instinct MI2108757 :crit, 0, 8757section 8x Radeon VII30826 :crit, 0, 30826section 4x Radeon VII24273 :crit, 0, 24273section 2x Radeon VII15591 :crit, 0, 15591section 1x Radeon VII7778 :crit, 0, 7778section 4x DC GPU Max 110022777 :done, 0, 22777section 2x DC GPU Max 110011815 :done, 0, 11815section 1x DC GPU Max 11006209 :done, 0, 6209section 4x H100 SXM5 80GB HBM378462 : 0, 78462section 2x H100 SXM5 80GB HBM346189 : 0, 46189section 1x H100 SXM5 80GB HBM328522 : 0, 28522section 4x A100 PCIe 80GB52056 : 0, 52056section 2x A100 PCIe 80GB27165 : 0, 27165section 1x A100 PCIe 80GB17896 : 0, 17896section 4x PG506-243/24241088 : 0, 41088section 2x PG506-243/24224168 : 0, 24168section 1x PG506-243/24215654 : 0, 15654section 8x A100 SXM4 40GB72965 : 0, 72965section 4x A100 SXM4 40GB42400 : 0, 42400section 2x A100 SXM4 40GB23707 : 0, 23707section 1x A100 SXM4 40GB15917 : 0, 15917section 4x V100 SXM2 32GB26527 : 0, 26527section 2x V100 SXM2 32GB15469 : 0, 15469section 1x V100 SXM2 32GB8947 : 0, 8947section 3x K40m + 1x Titan Xp5174 : 0, 5174section 2x Tesla K40m3300 : 0, 3300section 1x Tesla K40m1868 : 0, 1868section 1x Tesla K80 (2 GPU)3448 : 0, 3448section 1x Tesla K80 (1 GPU)1642 : 0, 1642section 2x L4014164 : 0, 14164section 1x L407945 : 0, 7945section 8x RTX A600040063 : 0, 40063section 4x RTX A600027915 : 0, 27915section 2x RTX A600015026 : 0, 15026section 1x RTX A60008814 : 0, 8814section 2x Quadro RTX 8000 Pa.10214 : 0, 10214section 1x Quadro RTX 8000 Pa.5607 : 0, 5607section 7x 2080 Ti + 1x A100 40GB33857 : 0, 33857section 4x GeForce RTX 2080 Ti18598 : 0, 18598section 2x GeForce RTX 2080 Ti10922 : 0, 10922section 1x GeForce RTX 2080 Ti6853 : 0, 6853section 1x A770 + 1x Titan Xp8380 :done, 0, 8380

• How to learn using FluidX3D?

  
  Follow theFluidX3D Documentation!
  
  
• What physical model does FluidX3D use?

  
  FluidX3D implements the lattice Boltzmann method, a type of direct numerical simulation (DNS), the most accurate type of fluid simulation, but also the most computationally challenging. Optional extension models include volume force (Guo forcing), free surface (volume-of-fluid andPLIC), a temperature model and Smagorinsky-Lilly subgrid turbulence model.
  
  
• FluidX3D only uses FP32 or even FP32/FP16, in contrast to FP64. Are simulation results physically accurate?

  
  Yes, in all but extreme edge cases. The code has been specially optimized to minimize arithmetic round-off errors and make the most out of lower precision. With these optimizations, accuracy in most cases is indistinguishable from FP64 double-precision, even with FP32/FP16 mixed-precision. Details can be found inthis paper.
  
  
• Compared to the benchmark numbers statedhere, efficiency seems much lower but performance is slightly better for most devices. How can this be?

  
  In that paper, the One-Step-Pull swap algorithm is implemented, using only misaligned reads and coalesced writes. On almost all GPUs, the performance penalty for misaligned writes is much larger than for misaligned reads, and sometimes there is almost no penalty for misaligned reads at all. Because of this, One-Step-Pull runs at peak bandwidth and thus peak efficiency.
  Here, a different swap algorithm termedEsoteric-Pull is used, a type of in-place streaming. This makes the LBM require much less memory (93 vs. 169 (FP32/FP32) or 55 vs. 93 (FP32/FP16) Bytes/cell for D3Q19), and also less memory bandwidth (153 vs. 171 (FP32/FP32) or 77 vs. 95 (FP32/FP16) Bytes/cell per time step for D3Q19) due to so-called implicit bounce-back boundaries. However memory access now is half coalesced and half misaligned for both reads and writes, so memory access efficiency is lower. For overall performance, these two effects approximately cancel out. The benefit of Esoteric-Pull - being able to simulate domains twice as large with the same amount of memory - clearly outweights the cost of slightly lower memory access efficiency, especially since performance is not reduced overall.
  
  
• Why don't you use CUDA? Wouldn't that be more efficient?

  
  No, that is a wrong myth. OpenCL is exactly as efficient as CUDA on Nvidia GPUs if optimized properly.Here I did roofline model and analyzed OpenCL performance on various hardware. OpenCL efficiency on modern Nvidia GPUs can be 100% with the right memory access pattern, so CUDA can't possibly be any more efficient. Without any performance advantage, there is no reason to use proprietary CUDA over OpenCL, since OpenCL is compatible with a lot more hardware.
  
  
• Why no multi-relaxation-time (MRT) collision operator?

  
  The idea of MRT is to linearly transform the DDFs into "moment space" by matrix multiplication and relax these moments individually, promising better stability and accuracy. In practice, in the vast majority of cases, it has zero or even negative effects on stability and accuracy, and simple SRT is much superior. Apart from the kinematic shear viscosity and conserved terms, the remaining moments are non-physical quantities and their tuning is a blackbox. Although MRT can be implemented in an efficient manner with only a single matrix-vector multiplication in registers, leading to identical performance compared to SRT by remaining bandwidth-bound, storing the matrices vastly elongates and over-complicates the code for no real benefit.
  
  

• Can FluidX3D run on multiple GPUs at the same time?

  
  Yes. The simulation grid is then split in domains, one for each GPU (domain decomposition method). The GPUs essentially pool their memory, enabling much larger grid resolution and higher performance. Rendering is parallelized across multiple GPUs as well; each GPU renders its own domain with a 3D offset, then rendered frames from all GPUs are overlayed with their z-buffers. Communication between domains is done over PCIe, so no SLI/Crossfire/NVLink/InfinityFabric is required. All GPUs must however be installed in the same node (PC/laptop/server). Even unholy combinations of Nvidia/AMD/Intel GPUs will work, although it is recommended to only use GPUs with similar memory capacity and bandwidth together. Using a fast gaming GPU and slow integrated GPU together would only decrease performance due to communication overhead.
  
  
• I'm on a budget and have only a cheap computer. Can I run FluidX3D on my toaster PC/laptop?

  
  Absolutely. Today even the most inexpensive hardware, like integrated GPUs or entry-level gaming GPUs, support OpenCL. You might be a bit more limited on memory capacity and grid resolution, but you should be good to go. I've tested FluidX3D on very old and inexpensive hardware and even on my Samsung S9+ smartphone, and it runs just fine, although admittedly a bit slower.
  
  
• I don't have an expensive workstation GPU, but only a gaming GPU. Will performance suffer?

  
  No. Efficiency on gaming GPUs is exactly as good as on their "professional"/workstation counterparts. Performance often is even better as gaming GPUs have higher boost clocks.
  
  
• Do I need a GPU with ECC memory?

  
  No. Gaming GPUs work just fine. Some Nvidia GPUs automatically reduce memory clocks for compute applications to almost entirely eliminate memory errors.
  
  
• My GPU does not support CUDA. Can I still use FluidX3D?

  
  Yes. FluidX3D uses OpenCL 1.2 and not CUDA, so it runs on any GPU from any vendor since around 2012.
  
  
• I don't have a dedicated graphics card at all. Can I still run FluidX3D on my PC/laptop?

  
  Yes. FluidX3D also runs on all integrated GPUs since around 2012, and also on CPUs.
  
  
• I need more memory than my GPU can offer. Can I run FluidX3D on my CPU as well?

  
  Yes. You only need to install theIntel OpenCL CPU Runtime.
  
  
• In the benchmarks you list some very expensive hardware. How do you get access to that?

  
  As a PhD candidate in computational physics, I used FluidX3D for my research, so I had access to BZHPC, SuperMUC-NG and JSC JURECA-DC supercomputers.
  
  

• I don't have an RTX/DXR GPU that supports raytracing. Can I still use raytracing graphics in FluidX3D?

  
  Yes, and at full performance. FluidX3D does not use a bounding volume hierarchy (BVH) to accelerate raytracing, but fast ray-grid traversal instead, implemented directly in OpenCL C. This is much faster than BVH for moving isosurfaces in the LBM grid (~N vs. ~N²+log(N) runtime; LBM itself is ~N³), and it does not require any dedicated raytracing hardware. Raytracing in FluidX3D runs on any GPU that supports OpenCL 1.2.
  
  
• I have a datacenter/mining GPU without any video output or graphics hardware. Can FluidX3D still render simulation results?

  
  Yes. FluidX3D does all rendering (rasterization and raytracing) in OpenCL C, so no display output and no graphics features like OpenGL/Vulkan/DirectX are required. Rendering is just another form of compute after all. Rendered frames are passed to the CPU over PCIe and then the CPU can either draw them on screen through dedicated/integrated graphics or write them to the hard drive.
  
  
• I'm running FluidX3D on a remote (super-)computer and only have an SSH terminal. Can I still use graphics somehow?

  
  Yes, either directly as interactive ASCII graphics in the terminal or by storing rendered frames on the hard drive and then copying them over via `scp -r user@server.url:"~/path/to/images/folder" .`.
  
  

• I want to learn about programming/software/physics/engineering. Can I use FluidX3D for free?

  
  Yes. Anyone can use FluidX3D for free for public research, education or personal use. Use by scientists, students and hobbyists is free of charge and well encouraged.
  
  
• I am a scientist/teacher with a paid position at a public institution. Can I use FluidX3D for my research/teaching?

  
  Yes, you can use FluidX3D free of charge. This is considered research/education, not commercial use. To give credit, thereferences listed below should be cited. If you publish data/results generated by altered source versions, the altered source code must be published as well.
  
  
• I work at a company in CFD/consulting/R&D or related fields. Can I use FluidX3D commercially?

  
  No. Commercial use is not allowed with the current license.
  
  
• Is FluidX3D open-source?

  
  No. "Open-source" as a technical term is defined as freely available without any restriction on use, but I am not comfortable with that. I have written FluidX3D in my spare time and no one should milk it for profits while I remain uncompensated, especially considering what other CFD software sells for. The technical term for the type of license I choose is "source-available no-cost non-commercial". The source code is freely available, and you are free to use, to alter and to redistribute it, as long as you do not sell it or make a profit from derived products/services, and as long as you do not use it for any military purposes (see thelicense for details).
  
  
• Will FluidX3D at some point be available with a commercial license?

  
  Maybe I will add the option for a second, commercial license later on. If you are interested in commercial use, let me know. For non-commercial use in science and education, FluidX3D is and will always be free.
  
  

• OpenCL-Headers andC++ Wrapper for GPU parallelization (Khronos Group)
• Win32 API for interactive graphics in Windows (Microsoft)
• X11/Xlib for interactive graphics in Linux (The Open Group)
• marching-cubes tables for isosurface generation on GPU (Paul Bourke)
• src/lodepng.cpp andsrc/lodepng.hpp for.png encoding and decoding (Lode Vandevenne)
• SimplexNoise class insrc/utilities.hpp for generating continuous noise in 2D/3D/4D space (Stefan Gustavson)
• skybox/skybox8k.png for free surface raytracing (HDRI Hub)

• Lehmann, M.:Computational study of microplastic transport at the water-air interface with a memory-optimized lattice Boltzmann method. PhD thesis, (2023)
• Lehmann, M.:Esoteric Pull and Esoteric Push: Two Simple In-Place Streaming Schemes for the Lattice Boltzmann Method on GPUs. Computation, 10, 92, (2022)
• Lehmann, M., Krause, M., Amati, G., Sega, M., Harting, J. and Gekle, S.:Accuracy and performance of the lattice Boltzmann method with 64-bit, 32-bit, and customized 16-bit number formats. Phys. Rev. E 106, 015308, (2022)
• Lehmann, M.:Combined scientific CFD simulation and interactive raytracing with OpenCL. IWOCL'22: International Workshop on OpenCL, 3, 1-2, (2022)
• Lehmann, M., Oehlschlägel, L.M., Häusl, F., Held, A. and Gekle, S.:Ejection of marine microplastics by raindrops: a computational and experimental study. Micropl.&Nanopl. 1, 18, (2021)
• Lehmann, M.:High Performance Free Surface LBM on GPUs. Master's thesis, (2019)
• Lehmann, M. and Gekle, S.:Analytic Solution to the Piecewise Linear Interface Construction Problem and Its Application in Curvature Calculation for Volume-of-Fluid Simulation Codes. Computation, 10, 21, (2022)

• FluidX3D is solo-developed and maintained by Dr. Moritz Lehmann.
• For any questions, feedback or other inquiries, contact me atdr.moritz.lehmann@gmail.com.
• Updates are posted on Mastodon via@ProjectPhysX/#FluidX3D and onYouTube.

I'm developing FluidX3D in my spare time, to make computational fluid dynamics lightning fast, accessible on all hardware, and free for everyone.

• You can support FluidX3D by reporting any bugs or things that don't work in theissues. I'm welcoming feedback!
• If you like FluidX3D, share it with friends and colleagues. Spread the word that CFD is now lightning fast, accessible and free.
• If you want to support FluidX3D financially, you cansponsor me on GitHub orbuy me a coffee. Thank you!