TODO: main result
3D models and scene rendered with our synthesized neural materials.

Abstract

High-quality material synthesis is essential for replicating complex surface properties to create realistic digital scenes. However, existing methods often suffer from inefficiencies in time and memory, require domain expertise, or demand extensive training data, with high-dimensional material data further constraining performance. Additionally, most approaches lack multi-modal guidance capabilities and standardized evaluation metrics, limiting control and comparability in synthesis tasks.
To address these limitations, we propose NeuMaDiff, a novel neural material synthesis framework utilizing hyperdiffusion. Our method employs neural fields as a low-dimensional representation and incorporates a multi-modal conditional hyperdiffusion model to learn the distribution over material weights. This enables flexible guidance through inputs such as material type, text descriptions, or reference images, providing greater control over synthesis.
To support future research, we contribute two new material datasets and introduce two BRDF distributional metrics for more rigorous evaluation. We demonstrate the effectiveness of NeuMaDiff through extensive experiments, including a novel statistics-based constrained synthesis approach, which enables the generation of materials of desired categories.

Dataset and base model

Our NeuMERL dataset are uploaded at AI community Hugging Face NeuMERL dataset, which contains 2400 BRDFs.
For material synthesis, the weights of the pre-trained base models are uploaded at Hugging Face Synthesis model weights.

Material synthesis framework

synthesis framework
An overview of NeuMaDiff, our novel neural material synthesis framework, consisting of three main stages. 1 (top left): Data augmentation using RGB permutation and PCA interpolation to create an expanded dataset, AugMERL; 2 (middle): Neural field fitted to individual materials, resulting in NeuMERL, a dataset of neural material representations; 3 (bottom): Training a multi-modal conditional hyperdiffusion model on NeuMERL to enable conditional synthesis of high-quality, diverse materials guided by inputs such as material type, text descriptions, or reference images. We further propose a novel statistics-based constrained synthesis method to generate materials of a specified type (top right).

Novel metrics for material synthesis

The evaluation mainly focus on the fidelity and diversity of the synthesized materials, which is still an open problem in this field. To fill in the gap, we raise the idea of utilizing Minimum matching distance (MMD), Coverage (COV) and 1-nearest neighbor (1-NNA) with either the image quality metric or the BRDF distribution as underlying distance function.

Please refer to our paper about the details of the BRDF distributional metrics. We demonstrate the effectiveness of NeuMaDiff through extensive experiments, including on this brand-new series of metrics.

Unconditional synthesis evaluation

uncond-vis
Material synthesis across various pipelines. All baseline models fail to capture the underlying distribution effectively, resulting in meaningless samples with severe artefacts in the synthesized materials. In contrast, NeuMaDiff successfully captures the complex neural material distribution, achieving significantly better fidelity and diversity.

unconditional-synthesis-metric-score
Qualitative evaluation of unconditional synthesis with metrics assessing generation fidelity and diversity. ↓ indicates that a lower score is better and ↑ indicates the opposite. NeuMaDiff significantly outperforms all baseline models across these metrics, underscoring its effectiveness in neural material synthesis.

Multi-modal synthesis evaluation

We demonstrate the conditional synthesis capabilities of NeuMaDiff across various modalities of input: material type, text description, or material images.

type-cond
Type-conditioned synthesis. The synthesized materials are diverse and closely align with the specified material type.


text-cond
Text-conditioned synthesis. NeuMaDiff synthesizes materials aligning with the texts and generalizes to unseen inputs: “green metal”, “red plastic”, and “highly specular material”.


img-cond
Image-conditioned synthesis. In each of the six pairs, the left image is the conditioning input, while the right image is the synthesized result. NeuMaDiff effectively generates realistic materials that closely align with the conditioning images.

Constrained synthesis evaluation

We classify materials into seven categories based on their reflective properties: diffuse, metallic, low-specular, medium-specular, high-specular, plastic, and mirror, via a novel approach called constrained synthesis.
It complements our conditional pipeline by enforcing constraints on unconditionally synthesized samples, allowing for targeted material generation according to desired reflective characteristics. Please refer to our paper regarding the statistical constraints details.

constrained-syn-vis
Synthesized materials of seven distinct categories using our novel constrained synthesis. Grounded in BRDF statistical analysis, this approach provides enhanced explainability and interpretability compared to standard conditional synthesis methods.

Citation

If you found the paper or code useful, please consider citing, 
@misc{
    NeuMaDiff2024,
   title={NeuMaDiff: Neural Material Synthesis via Hyperdiffusion}, 
   author={Chenliang Zhou and Zheyuan Hu and Alejandro Sztrajman and Yancheng Cai and Yaru Liu and Cengiz Oztireli},
   year={2024},
   eprint={2411.12015},
   archivePrefix={arXiv},
   primaryClass={cs.GR},
   url={https://arxiv.org/abs/2411.12015}, 
}


The website template was inspired by FrePolad.