ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction

Shaozhe Hao, Kai Han, Zhengyao Lv, Shihao Zhao, Kwan-Yee K. Wong
The University of Hong Kong
ECCV 2024 Oral
PDF arXiv Code
ConceptExpress teaser.

Unsupervised Concept Extraction (UCE): We focus on the unsupervised problem of extracting multiple concepts from a single image. Given an image that contains multiple concepts, we aim to harness a frozen pretrained diffusion model to automatically learn the conceptual tokens. Using the learned conceptual tokens, we can regenerate the extracted concepts with high quality.

Abstract

While personalized text-to-image generation has enabled the learning of a single concept from multiple images, a more practical yet challenging scenario involves learning multiple concepts within a single image. However, existing works tackling this scenario heavily rely on extensive human annotations. In this paper, we introduce a novel task named Unsupervised Concept Extraction (UCE) that considers a fully unsupervised setting without any human knowledge of the concepts. Given an image that contains multiple concepts, the task aims to extract and recreate individual concepts solely relying on the existing knowledge from pretrained diffusion models. To address this problem, we present ConceptExpress that tackles UCE by unleashing the inherent capabilities of pretrained diffusion models in two aspects. Specifically, a concept localization approach automatically locates and disentangles salient concepts by leveraging spatial correspondence provided by diffusion self-attention; and based on the lookup association between a concept and a conceptual token, a concept-wise optimization process learns discriminative tokens that represent each individual concept. Finally, we establish an evaluation protocol tailored for the UCE task. Extensive experiments show the effectiveness of ConceptExpress, demonstrating it to be a promising solution to UCE.

Method


ConceptExpress can disentangle each concept in the compositional scene and learn discriminative conceptual tokens that represent each individual concept.

ConceptExpress presents two major innovations:

  • For concept disentanglement, we propose a concept localization approach that automatically locates salient concepts within the image. This approach involves clustering spatial points on the self-attention map, building upon the observation that stable diffusion has learned good unsupervised spatial correspondence in the self-attention layers.
  • For conceptual token learning, we employ concept-wise masked denoising optimization by reconstructing the located concept. This optimization is based on a token lookup table that associates each located concept with its corresponding conceptual token.

See more method details in our paper!

Results

Unsupervised concept extraction


Bas†: Break-a-Scene adpated in the unsupervised setting by leveraging the instance masks identified by our method as the ground-truth segmentation masks.

Text-guided generation

ConceptExpress is also capable of text-guided generation:









BibTeX

If you find this project useful for your research, please cite the following:

@InProceedings{hao2024conceptexpress,
    title={Concept{E}xpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction}, 
    author={Shaozhe Hao and Kai Han and Zhengyao Lv and Shihao Zhao and Kwan-Yee~K. Wong},
    booktitle={ECCV},
    year={2024},
}

This page was adapted from this source code.