synthesizing photorealistic satellite imagery with semantic layout conditioning using denoising diffusion probabilistic models

Paper number

IAC-23,B1,4,6,x76306

Author

Mr. Orkhan Baghirli, Syn10, Estonia

Coauthor

Mr. Hamid Askarov, Azercosmos, Space Agency of Republic of Azerbaijan, Azerbaijan

Coauthor

Mr. Imran Ibrahimli, Syn10, Estonia

Coauthor

Mr. Ismat Bakhishov, Azercosmos, Space Agency of Republic of Azerbaijan, Azerbaijan

Coauthor

Mr. Nabi Nabiyev, Purdue University, United States

Year

2023

Abstract

Satellite images are an important source of data when it comes to environmental and socio-economic applications on a global scale. Building predictive models that are based on complex scenes often necessitates labor-intensive and expensive annotation efforts to make sure their performance is up to par. To mitigate the labeled data dependency problem, generative models aim to synthesize photorealistic images from semantic layouts, which is the inverse of semantic segmentation.

In the past, research on semantic image synthesis typically focused on Generative Adversarial Networks (GANs), which can yield low-quality or limited diversity of generated images. Contrastingly, Denoising Diffusion Probabilistic Models (DDPMs) have recently achieved impressive results in image generation tasks in comparison to GANs. Nevertheless, DDPM-based synthesis of satellite imagery with semantic layout conditioning is so far an under-explored area of research. 

In this work, we examine the potential of conditional DDPMs for the task of transforming semantic layouts into realistic satellite images. We modify U-Net architecture, which is the current best practice for image diffusion models, to condition the DDPMs with semantic maps by adding embeddings into intermediate layers of the network. Additionally, we incorporate an attention mechanism into our model architecture to make the DDPMs more powerful and versatile generators.

We demonstrate the effectiveness of our proposed model by evaluating it on a dataset acquired through Azersky. We aim to assess visual quality, diversity and learned correspondence of generated images. Our findings indicate the potential for generating high-quality labeled synthetic images that can be used for data augmentation in a data-scarce environment.

We look forward to further explorations of DDPMs in a wider variety of settings and data modalities. We additionally release an open-source reference implementation of the algorithm.

Abstract document

IAC-23,B1,4,6,x76306.brief.pdf

Manuscript document

IAC-23,B1,4,6,x76306.pdf (🔒 authorized access only).

To get the manuscript, please contact IAF Secretariat.