Benchmark System Documentation — Argos

Overview
Benchmark Architecture
Execution Pipeline
Available Metrics
CPU Time (default)
Memory Peak (default)
Global Memory Peak
Bounding Box
Centroid to Face
Edge-Face Intersections
Heatmap
Holes
Vertices
Faces
Measurements Reliability
Interpretation of results
Known limits and precautions
Export of results
Configuration via JSON file

Overview

The Argos benchmark system is designed to objectively evaluate mesh reconstruction algorithms (PointCloud → Mesh). It measures both execution performance (CPU time, memory consumption) and the geometric quality of the reconstructed mesh compared to the original mesh.

The benchmark is powered by the Google Benchmark library, which ensures reliable execution time measurement by repeating iterations until statistical stability is reached.

Scope of measurement : All metrics focus exclusively on the reconstruction phase (PointCloud → Mesh algorithm), and not on the entire pipeline (including sampling). However, it is possible to add metrics to compare another part of the pipeline, or even the entire pipeline.

Benchmark Architecture

BenchmarksContainer
│
├── objFilesPaths[]          ← .obj / .ply files in the input folder
│
└── pipelines[]              ← one Pipeline per reconstruction algorithm
     │
     └── Pipeline
         ├── name            ← algorithm name
         └── runner          ← lambda encapsulating reconstructionBenchmark()

For each combination (algorithm × input file), Google Benchmark saves a separate benchmark and runs it in a controlled manner.

The core class is BenchmarksContainer.h/.cpp. It :

Scans the input folder to collect all .obj and .ply files.
Registers the benchmarks using benchmark::RegisterBenchmark.
Orchestrates the execution and transmits the results to BenchmarkReporter.

Execution Pipeline

Execution is organized into two levels : the outer loop per input file (in run()), and the inner loop for measuring Google Benchmark (in reconstructionBenchmark()).

For each input file (.obj or .ply)
│
├── [Read + deserialize → Original Mesh]
│
├── [Sampling → PointCloud]    ← once per file, common to all algorithms
│
├── [Save Cloud_{file}.obj]
│
└── For each reconstruction algorithm
     │
     ├── [Save Google Benchmark]
     │
     └── ┌──────────────────────────────┐
         │    Google Benchmark Loop     │    ← Repeated until stabilization
         │                              │
         │  memory.start()              │
         │  reconstructor.reconstruct() │    ← Measurement here
         │  memory.stop()               │
         │                              │
         │  [Pause timing]              │
         │  → Metrics calculation       │    ← First iteration only
         │  → Save Mesh.obj             │
         │  [Resumption timing]         │
         └──────────────────────────────┘

[Export results → .xlsx]

Important Note : Sampling is performed only once per file in run(), before benchmarks are saved. The resulting point cloud is therefore identical for all reconstruction algorithms tested on the same file, ensuring fair comparisons. Metric calculations and saving the reconstructed mesh are performed off-timing (state.PauseTiming() / state.ResumeTiming()), without skewing the measurements.

Available Metrics

CPU Time (default)

Property	Value
CLI Command	(automatically enabled)
Unit	milliseconds (CPU time)
Measured by	Google Benchmark

Description : Total CPU time consumed by the reconstruction algorithm, accumulated across all iterations and then converted to a single value per operation.

The distinction is made between :

cpu_time_per_op : CPU time per iteration (multi-threaded processing included)
wall_time_per_op : Actual time elapsed ("wall clock") per iteration

Interpretation :

A cpu_time_per_op significantly higher than the wall_time_per_op indicates effective parallel processing (multiple cores utilized).
A cpu_time_per_op close to or lower than the wall_time_per_op indicates a primarily single-threaded algorithm or limited by I/O.

Memory Peak (default)

Property	Value
CLI Command	(automatically enabled)
Unit	Megabytes (MB/Mo)
Key in report	`MemoryPeak(Mo)`

Description : Maximum amount of resident RAM (VmRSS on Linux, WorkingSetSize on Windows) observed during reconstruction. A monitoring thread samples memory every 500 ms between start() and stop().

Interpretation :

Indicates the instantaneous memory consumption of the algorithm (memory actually used at a given time).
Useful for comparing algorithms on large files or constrained environments.
The 500 ms granularity may miss very brief peaks.

Global Memory Peak (`global_peak`)

Property	Value
CLI Command	`global_peak`
Unit	Megabytes (MB/Mo)
Key in Report	`GlobalMemoryPeak(Mo)`

Description : Maximum memory peak reached since the process started (VmHWM on Linux, PeakWorkingSetSize on Windows). This value is monotonically increasing : if a previous benchmark consumed more memory, it's value will be retained.

Interpretation :

Shows the application's peak memory usage, across all algorithms.
Comparing GlobalMemoryPeak and MemoryPeak helps determine if the global peak originates from the current algorithm or a previous one.
Use with caution in multi-algorithm benchmarks : the value may reflect a previous algorithm.

Bounding Box (`bbox`)

Property	Value
CLI Command	`bbox`
Keys in Report	`BoundingBox.VolumeDiffPct`,
`BoundingBox.CenterDist`

Description : Compares the axis-aligned bounding boxes (AABB) of the original and reconstructed meshes. Both AABBs are calculated from the vertices (getVertices()).

Produced Metrics :

Key	Description
`BoundingBox.VolumeDiffPct`	Relative volume difference, expressed as a percentage: `\|VolumeA - VolumeB\| / VolumeA * 100`
`BoundingBox.CenterDist`	Euclidean distance between the centers of the two AABBs

Interpretation :

VolumeDiffPct ≈ 0 : The reconstructed mesh occupies an overall volume similar to the original. A value > 20 % indicates a significant mismatch.
CenterDist is expressed in mesh units. Divide it manually by the diagonal of the original AABB to get a normalized offset comparable across models.
This metric is quick to calculate and provides a rough initial warning, but does not indicate local geometric accuracy.

Centroid to Face (`ctf`)

Property	Value
CLI Command	`ctf`
Keys in Report	`CentroidToFace_MeanBA`,
`CentroidToFace_MaxBA`,
`CentroidToFace_MinBA`,
`CentroidToFace_NbFacesB`,
`CentroidToFace_NbTrianglesB`

Description : For each triangle in mesh B (the reconstructed mesh, obtained by fan-triangulation of the faces), the distance from its centroid to the nearest triangle in mesh A (the original) is computed using an AABB tree. The metric is therefore measured in the B→A direction (reconstructed → original). The A→B direction is not currently exported.

Produced Metrics :

Key	Description
`CentroidToFace_MeanBA`	Average centroid-to-face distance from the reconstructed mesh to the original
`CentroidToFace_MinBA`	Minimum distance from the reconstructed mesh to the original
`CentroidToFace_MaxBA`	Maximum distance from the reconstructed mesh to the original
`CentroidToFace_NbFacesB`	Number of faces (polygons) of the reconstructed mesh
`CentroidToFace_NbTrianglesB`	Number of triangles (after fan-triangulation) of the reconstructed mesh

Interpretation :

MeanBA close to 0 : The reconstructed mesh is geometrically close to the original.
A high MaxBA reveals areas with strong local deviation (artifacts, poorly reconstructed holes).
This metric is a simplified, one-sided analog of the Hausdorff distance and is one of the best indicators of geometric fidelity among the available metrics.

Edge-Face Intersections (`intersections`)

Property	Value
CLI Command	`intersections`
Unit	Integer (count)
Key in Report	`EdgeFaceIntersects`

Description : Counts the number of times an edge in the reconstructed mesh intersects a face in the original mesh, using the Möller-Trumbore algorithm. Edges are extracted uniquely (no duplicates). Non-triangular faces in the original are decomposed into triangles by fan-triangulation.

Interpretation :

0 : The reconstructed mesh never intersects the original — a sign of overall consistent reconstruction.
High value : Presence of significant geometric artifacts (surfaces that "pass through" the original). This may indicate over-reconstruction, noise, or poor algorithm configuration.
This metric is indicative of the topological consistency between the two meshes.

Caution : The complexity is O(edges × triangles). On dense meshes, the computation can be lengthy. Use on reasonably sized files.

Heatmap (`heatmap`)

Property	Value
CLI Command	`heatmap`
Keys in Report	(no counter, side effect only)

Description : Does not produce a numerical metric. When enabled, this metric injects texture coordinates into the reconstructed mesh proportional to the distance between each reconstructed vertex and the nearest triangle of the original mesh. This information is written into the exported .obj file as vt records, so a viewer (Blender, MeshLab, ...) can render a per-vertex heat map (green = close to the original, red = far from it).

Interpretation :

The texture coordinates are normalized in the [0, 1] range. The actual distances are computed by VertexToFace::compute (see VertexToFace in the API reference).
Useful for a visual, qualitative assessment of the reconstruction.
Adds mid-edge vertices to the reconstructed mesh, which inflates its vertex and face counts.

Holes (`holes`)

Property	Value
CLI Command	`holes`
Keys in Report	`Holes.HoleA`, `Holes.HoleB`,
`Holes.CircumferenceA`, `Holes.CircumferenceB`

Description : Detects the topological holes of both meshes via TopologicMesh::findAllHoles(), then reports the number of holes and the average circumference for each mesh.

Produced Metrics :

Key	Description
`Holes.HoleA`	Number of holes detected in the original mesh (`-1` if the input is a cloud)
`Holes.HoleB`	Number of holes detected in the reconstructed mesh
`Holes.CircumferenceA`	Average circumference of the holes in the original mesh (`-1` if N/A)
`Holes.CircumferenceB`	Average circumference of the holes in the reconstructed mesh

Interpretation :

A higher hole count on the reconstructed mesh than on the original indicates a loss of surface.
This metric is a good complement to ctf to detect topological defects.

Vertices (`vertices`)

Property	Value
CLI Command	`vertices`
Keys in Report	`Original mesh vertices`,
`Reconstructed mesh vertices`

Description : Reports the number of vertices of the original and reconstructed meshes as raw counters (no computation, no analysis).

Interpretation :

Useful to compare mesh complexities and to explain variations of memory / time metrics.
Both counters are integers, not units.

Faces (`faces`)

Property	Value
CLI Command	`faces`
Keys in Report	`Original mesh faces`,
`Reconstructed mesh faces`

Description : Same as the vertices metric, but for faces (polygons) of the original and reconstructed meshes.

Interpretation :

Useful to compare mesh complexities and to explain variations of memory / time metrics.
Both counters are integers, not units.

Measurements Reliability

Execution Time

Google Benchmark automatically repeats iterations until the measured time is statistically stable (low variance). Unmeasured operations (metric calculations, file saving) are explicitly excluded from the timing via PauseTiming() / ResumeTiming(), thus avoiding any bias.

Memory (`MemoryPeak`)

The measurement is performed by a dedicated thread that samples resident memory every 500 ms. This choice offers a good compromise between accuracy and impact on the measurement. However : - Very brief peaks (< 500 ms) may be missed. - The measurement is dependent on the operating system (procfs on Linux, WinAPI on Windows) and reflects the actual physical memory used, not the allocated virtual memory.

Geometric Metrics

The ctf and bbox metrics are calculated only once (during the first iteration, outside of the timing), which guarantees their consistency without increasing the measurement loop's load. They are deterministic for the same pair (original mesh, reconstructed mesh).

The intersections metric is also calculated outside of the loop. It is based on the Möller–Trumbore algorithm, a standard reference for ray-triangle intersection detection.

Interpretation of results

Reading the XLSX report

Each line corresponds to a single execution identified by {algorithm}_{file.obj}. The columns are :

Column	Description
`OBJ file name`	Execution ID
`iterations`	Number of iterations performed by Google Benchmark
`CPU time / op (ms)`	Average CPU time per iteration, in milliseconds
`Overall time / op (ms)`	Average real-time time per iteration, in milliseconds
`MemoryPeak(Mo)`	Memory peak during reconstruction
`GlobalMemoryPeak(Mo)`	Global memory peak of the process
`BoundingBox.VolumeDiffPct`	Relative volume difference between the two AABBs (%)
`BoundingBox.CenterDist`	Euclidean distance between the centers of the two AABBs
`CentroidToFace_MeanBA`	Average centroid-to-face distance (reconstructed → original)
`CentroidToFace_MinBA`	Minimum centroid-to-face distance (reconstructed → original)
`CentroidToFace_MaxBA`	Maximum centroid-to-face distance (reconstructed → original)
`CentroidToFace_NbFacesB`	Number of faces of the reconstructed mesh
`CentroidToFace_NbTrianglesB`	Number of triangles of the reconstructed mesh
`EdgeFaceIntersects`	Number of edge-to-face intersections
`Holes.HoleA` / `Holes.HoleB`	Number of holes detected in each mesh
`Holes.CircumferenceA` / `Holes.CircumferenceB`	Average hole circumference in each mesh
`Original mesh vertices` / `Reconstructed mesh vertices`	Vertex counts
`Original mesh faces` / `Reconstructed mesh faces`	Face counts

Time Unit : The CPU time / op and Overall time / op columns are expressed in milliseconds in the Excel file (unlike the internal raw values, which are in nanoseconds).

Dynamic columns : The metric columns shown above only appear in the XLSX if the corresponding metric is enabled on the command line (or via the JSON config).

Quick Assessment Grid

Metric	Ideal Value	Warning Signal
`CentroidToFace_MeanBA`	≈ 0	> threshold dependent on model scale
`CentroidToFace_MaxBA`	low	value >> Mean
`BoundingBox.VolumeDiffPct`	≈ 0	> 20%
`BoundingBox.CenterDist`	low	> 10% of the original AABB diagonal
`EdgeFaceIntersects`	0	any non-zero value
`Holes.HoleA` vs `Holes.HoleB`	similar	reconstructed has many more holes
`MemoryPeak(Mo)`	minimal	depends on the hardware target

Comparing Two Algorithms

To compare advancing and scale_space on the same file:

Time : Compare CPU time / op (ms) — the fastest algorithm has the lowest value.
Quality : Compare CentroidToFace_MeanBA — the lower mean indicates the best geometric accuracy.
Consistency : Check EdgeFaceIntersects — prefer the algorithm that produces 0 intersections.
Memory : Compare MemoryPeak(Mo) if memory constraint is significant.

Known limits and precautions

Memory Granularity : The memory monitoring thread polls every 500 ms. On very fast reconstructions (< 1 s), the peak may be underestimated.

Monotonous GlobalMemoryPeak : This metric can only increase with each benchmark. For fair comparisons between algorithms, MemoryPeak is preferred.

Complexity of EdgeFaceIntersections : Intersection detection is O(edges × triangles). On high-density meshes, the computation time may exceed the reconstruction time. It is advisable to enable this metric only on moderately sized models or to limit the number of points during sampling.

Repeatability : CGAL (Poisson, Scale Space, Advancing Front) algorithms may produce slightly different results depending on the platform (compiler, system libraries). For reliable comparisons, run all benchmarks on the same machine in the same session.

Shared sampling between algorithms : For the same .obj file, the point cloud is calculated only once and shared between all reconstruction algorithms. This is advantageous for fair comparisons, but it means that the quality of the sampling impacts all algorithms equally — the metrics therefore only reflect the quality of the reconstruction, not the quality of the sampling.

Export of results

At the end of each run, two types of output are generated in the output/{timestamp}/ folder :

.obj Files :

Cloud_{file}.obj: intermediate point cloud, common to all algorithms for this file
{algorithm}_Mesh_{file}.obj: mesh reconstructed by the corresponding algorithm

rapport.xlsx File : summary table of all metrics for each benchmark, with a header containing the date, number of CPU cores, and their frequency. This file can be opened directly in Excel or LibreOffice Calc to facilitate comparison and visualization of the results.

Configuration via JSON file

As an alternative to the command line, you can pass a JSON configuration file using the --config option :

./argos_benchmark --config path/to/config.json

--config is exclusive with other options (-i, algorithms, metrics) : they cannot be combined.

The ConfigParser reads the JSON and dynamically reconstructs an argv equivalent to the CLI command, according to the following rules :

JSON Key	Behavior
`"input"` / `"output"`	Translated to `--input <value>` / `--output <value>`
`"parameters"`	The key is ignored; each subkey becomes a `--subkey` flag followed by its values
Other scalar key	The value is issued as is (subcommand name, option value)
Other object	The key is ignored, the content is recursively flattened

JSON file example :

{
  "input": "/path/to/obj_folder",
  "sampling": {
    "algorithm": "uniform_area",
    "parameters": {
      "points": 5000
    }
  },
  "reconstruct": [
    {
      "algorithm": "scale_space",
      "parameters": {
        "iterations": {
          "min": 2,
          "max": 8,
          "step": 2
        }
      }
    },
    {
      "algorithm": "advancing"
    }
  ],
  "metrics": [
    "bbox",
    "ctf"
  ]
}

This file is equivalent to the command:

./argos_benchmark -i "/path/to/obj_folder" uniform_area -p 5000 scale_space -i 2 8 2 advancing bbox ctf

Benchmark System Documentation — Argos

Table of Contents

Overview

Benchmark Architecture

Execution Pipeline

Available Metrics

CPU Time (default)

Memory Peak (default)

Global Memory Peak (global_peak)

Bounding Box (bbox)

Centroid to Face (ctf)

Edge-Face Intersections (intersections)

Heatmap (heatmap)

Holes (holes)

Vertices (vertices)

Faces (faces)

Measurements Reliability

Execution Time

Memory (MemoryPeak)

Geometric Metrics

Interpretation of results

Reading the XLSX report

Quick Assessment Grid

Comparing Two Algorithms

Known limits and precautions

Export of results

Configuration via JSON file

Global Memory Peak (`global_peak`)

Bounding Box (`bbox`)

Centroid to Face (`ctf`)

Edge-Face Intersections (`intersections`)

Heatmap (`heatmap`)

Holes (`holes`)

Vertices (`vertices`)

Faces (`faces`)

Memory (`MemoryPeak`)