Cycle-Consistent Diverse Image Synthesis from Natural Language

Paper

Chen, Zhi and Luo, Yadan. 2019. "Cycle-Consistent Diverse Image Synthesis from Natural Language." 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). Shanghai, China 08 - 12 Jul 2019 United States. IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/10.1109/ICMEW.2019.00085
Paper/Presentation Title

Cycle-Consistent Diverse Image Synthesis from Natural Language

Presentation TypePaper
AuthorsChen, Zhi and Luo, Yadan
Journal or Proceedings TitleProceedings of 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)
Journal Citationpp. 459-464
Number of Pages6
Year2019
PublisherIEEE (Institute of Electrical and Electronics Engineers)
Place of PublicationUnited States
ISBN9781538692141
Digital Object Identifier (DOI)https://doi.org/10.1109/ICMEW.2019.00085
Web Address (URL) of Paperhttps://ieeexplore.ieee.org/document/8795022
Web Address (URL) of Conference Proceedingshttps://ieeexplore.ieee.org/xpl/conhome/8777006/proceeding
Conference/Event2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)
Event Details
2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)
Parent
IEEE International Conference on Multimedia and Expo
Delivery
In person
Event Date
08 to end of 12 Jul 2019
Event Location
Shanghai, China
Rank
B
B
Abstract

Text-to-image translation has become an attractive yet challenging task in computer vision. Previous approaches tend to generate similar, or even monotonous, images for distinctive texts and overlook the characteristics of specific sentences. In this paper, we aim to generate images from the given texts by preserving diverse appearances and modes of the objects or instances contained. To achieve that, a novel learning model named SuperGAN is proposed, which consists of two major components: an image synthesis network and a captioning model in a Cycle-GAN framework. SuperGAN adopts the cycle-consistent adversarial training strategy to learn an image generator where the feature distribution of the generated images complies with the distribution of the generic images. Meanwhile, a cycle-consistency loss is applied to constrain that the caption of the generated images is closed to the original texts. Extensive experiments on the benchmark dataset Oxford-flowers-102 demonstrate the validity and effectiveness of our proposed method. In addition, a new evaluation metric is proposed to measure the diversity of synthetic results.

KeywordsImage synthesis; image captioning; generative adversarial networks; cycle-consistency loss
Contains Sensitive ContentDoes not contain sensitive content
ANZSRC Field of Research 2020460304. Computer vision
Public Notes

Files associated with this item cannot be displayed due to copyright restrictions.

Byline AffiliationsUniversity of Queensland
Permalink -

https://research.usq.edu.au/item/zyx16/cycle-consistent-diverse-image-synthesis-from-natural-language

Log in to edit
  • 7
    total views
  • 0
    total downloads
  • 2
    views this month
  • 0
    downloads this month

Export as

Related outputs

Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routing
Zhao, Zecheng, Chen, Zhi, Huang, Zi, Sadiq, Shazia and Chen, Tong. 2025. "Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routing." 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '25). Padua, Italy 13 - 18 Jul 2025 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3726302.3729936
Dynamic Target Distribution Estimation for Source-Free Open-Set Domain Adaptation
Yu, Zhiqi, Liao, Zhichao, Li, Jingjing, Chen, Zhi and Zhu, Lei. 2025. "Dynamic Target Distribution Estimation for Source-Free Open-Set Domain Adaptation." 39th AAAI Conference on Artificial Intelligence (AAAI 2025). Philadelphia, Pennsylvania, United States 25 Feb - 04 Mar 2025 United States. Association for the Advancement of Artificial Intelligence (AAAI). https://doi.org/10.1609/aaai.v39i21.34380
Snap and diagnose: An advanced multimodal retrieval system for identifying plant diseases in the wild
Wei, Tianqi, Chen, Zhi and Yu, Xin. 2024. "Snap and diagnose: An advanced multimodal retrieval system for identifying plant diseases in the wild." 6th ACM International Conference on Multimedia in Asia (MMAsia '24). Auckland, New Zealand 03 - 06 Dec 2024 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3696409.3700293
DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection
Lim, Jia Syuen, Chen, Zhuoxiao, Baktashmotlagh, Mahsa, Chen, Zhi, Yu, Xin, Huang, Zi and Luo, Yadan. 2024. "DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection." 38th Annual Conference on Neural Information Processing Systems (NeurIPS 2024). Vancouver, Canada 10 - 15 Dec 2024 Canada.
Towards Cost-Efficient Federated Multi-agent RL with Learnable Aggregation
Zhang, Yi, Wang, Sen, Chen, Zhi, Xu, Xuwei, Funiak, Stano and Liu, Jiajun. 2024. "Towards Cost-Efficient Federated Multi-agent RL with Learnable Aggregation." 28th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2024). Taipei, Taiwan 07 - 10 May 2024 Springer. https://doi.org/10.1007/978-981-97-2253-2_14
Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline
Wei, Tianqi, Chen, Zhi, Huang, Zi and Yu, Xin. 2024. "Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline." 32nd ACM International Conference on Multimedia (MM '24). Melbourne, Australia 28 Oct - 01 Nov 2024 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3664647.3680599
Secondary analysis of newly diagnosed type 2 diabetes subgroups and treatment responses in the MARCH cohort
Wang, Weihao, Li, Xinyao, Chen, Fei, Wei, Ran, Chen, Zhi, Li, Jingjing, Qiao, Jingtao, Pan, Qi, Yang, Wenying and Guo, Lixin. 2024. "Secondary analysis of newly diagnosed type 2 diabetes subgroups and treatment responses in the MARCH cohort." Diabetes and Metabolic Syndrome: Clinical Research and Reviews. 18 (1). https://doi.org/10.1016/j.dsx.2023.102936
Optimizing taxi route planning based on taxi trajectory data analysis
Yang, Xinyi, Chen, Zhi and Luo, Yadan. 2023. "Optimizing taxi route planning based on taxi trajectory data analysis." 34th Australasian Database Conference (ADC 2023). Melbourne, Australia 01 202 - 03 Nov 2023 Switzerland . Springer. https://doi.org/10.1007/978-3-031-47843-7_4
Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error
Wang, Zixin, Luo, Yadan, Chen, Zhi, Wang, Sen and Huang, Zi. 2023. "Cal-SFDA: Source-Free Domain-adaptive Semantic Segmentation with Differentiable Expected Calibration Error." 31st ACM International Conference on Multimedia (MM '23). Ottawa, Canada 29 Oct 202 - 03 Nov 2023 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3581783.3611808
Zero-Shot Learning by Harnessing Adversarial Samples
Chen, Zhi, Zhang, Pengfei, Li, Jingjing, Wang, Sen and Huang, Zi. 2023. "Zero-Shot Learning by Harnessing Adversarial Samples." 31st ACM International Conference on Multimedia (MM '23). Ottawa, Canada 29 Oct 202 - 03 Nov 2023 Association for Computing Machinery (ACM). https://doi.org/10.1145/3581783.3611823
FluMA: An Intelligent Platform for Influenza Monitoring and Analysis
Chen, Xi, Chen, Zhi, Wang, Zijian, Qui, Ruihong and Luo, Yadan. 2022. "FluMA: An Intelligent Platform for Influenza Monitoring and Analysis." 33rd Australasian Database Conference (ADC 2022). Sydney, Australia 02 - 04 Sep 2022 Switzerland . Springer. https://doi.org/10.1007/978-3-031-15512-3_12
GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot Learning
Chen, Zhi, Luo, Yadan, Wang, Sen, Li, Jingjing and Huang, Zi. 2022. "GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot Learning." IEEE Transactions on Multimedia. 25, pp. 5374-5385. https://doi.org/10.1109/TMM.2022.3190678
Application of Novel Subgroups of Chinese Inpatients with Diabetes Based on Machine Learning Paradigm
Wang, Weihao, Chen, Zhi, Wang, Sen, Chen, Fei, Deng, Mingqun, Fan, Qi and Guo, Lixin. 2022. "Application of Novel Subgroups of Chinese Inpatients with Diabetes Based on Machine Learning Paradigm." Diabetes and Metabolic Syndrome: Clinical Research and Reviews. 16 (7). https://doi.org/10.1016/j.dsx.2022.102556
Pixel Exclusion: Uncertainty-aware Boundary Discovery for Active Cross-Domain Semantic Segmentation
You, Fuming, Li, Jingjing, Chen, Zhi and Zhu, Lei. 2022. "Pixel Exclusion: Uncertainty-aware Boundary Discovery for Active Cross-Domain Semantic Segmentation." 30th ACM International Conference on Multimedia (MM '22). Lisbon, Portugal 10 - 14 Oct 2022 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3503161.3548079
Distinguishing Unseen from Seen for Generalized Zero-shot Learning
Su, Hongzu, Li, Jingjing, Chen, Zhi, Zhu, Lei and Lu, Ke. 2022. "Distinguishing Unseen from Seen for Generalized Zero-shot Learning." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, United States 18 - 24 Jun 2022 United States. IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/10.1109/CVPR52688.2022.00773
Semantics Disentangling for Generalized Zero-Shot Learning
Chen, Zhi, Luo, Yadan, Qui, Ruihong, Wang, Sen, Huang, Zi, Li, Jingjing and Zhang, Zheng. 2022. "Semantics Disentangling for Generalized Zero-Shot Learning." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada 10 - 17 Oct 2021 United States. IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/10.1109/ICCV48922.2021.00859
Local graph convolutional networks for cross-modal hashing
Zhang, Yudong, Wang, Sen, Lu, Jianglin, Chen, Zhi, Zhang, Zheng and Huang, Zi. 2021. "Local graph convolutional networks for cross-modal hashing." 29th ACM International Conference on Multimedia (MM '21). 20 - 24 Oct 2021 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3474085.3475346
Domain Adaptive Semantic Segmentation Without Source Data
You, Fuming, Li, Jingjing, Zhu, Lei, Chen, Zhi and Huang, Zi. 2021. "Domain Adaptive Semantic Segmentation Without Source Data." 29th ACM International Conference on Multimedia (MM '21). 20 - 24 Oct 2021 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3474085.3475482
Application of New International Classification of Adult‐Onset Diabetes in Chinese Inpatients with Diabetes Mellitus
Wang, Weihao, Pei, Xiaobei, Zhang, Lina, Chen, Zhi, Lin, Dong, Duan, Xiaoye, Fan, Jingwen, Pan, Qi and Guo, Lixin. 2021. "Application of New International Classification of Adult‐Onset Diabetes in Chinese Inpatients with Diabetes Mellitus." Diabetes - Metabolism: Research and Reviews. 37 (7). https://doi.org/10.1002/dmrr.3427
Mitigating Generation Shifts for Generalized Zero-Shot Learning
Chen, Zhi, Luo, Yadan, Wang, Sen, Qui, Ruihong, Li, Jingjing and Huang, Zi. 2021. "Mitigating Generation Shifts for Generalized Zero-Shot Learning." 29th ACM International Conference on Multimedia (MM '21). 20 - 24 Oct 2021 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3474085.3475258
CausalRec: Causal Inference for Visual Debiasing in Visually-Aware Recommendation
Qui, Ruihong, Wang, Sen, Chen, Zhi, Yin, Hongzhi and Huang, Zi. 2021. "CausalRec: Causal Inference for Visual Debiasing in Visually-Aware Recommendation." 29th ACM International Conference on Multimedia (MM '21). 20 - 24 Oct 2021 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3474085.3475266
Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning
Chen, Zhi, Huang, Zi, Li, Jingjing and Zhang, Zheng. 2021. "Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning." 32nd Australasian Database Conference (ADC 2021). Dunedin, New Zealand 29 Jan - 05 Feb 2021 Switzerland . Springer. https://doi.org/10.1007/978-3-030-69377-0_12
Rethinking Generative Zero-Shot Learning: An Ensemble Learning Perspective for Recognising Visual Patches
Chen, Zhi, Wang, Sen, Li, Jingjing and Huang, Zi. 2020. "Rethinking Generative Zero-Shot Learning: An Ensemble Learning Perspective for Recognising Visual Patches." 28th ACM International Conference on Multimedia (MM '20). Seattle, United States 12 - 16 Oct 2020 United States. Association for Computing Machinery (ACM). https://doi.org/10.1145/3394171.3413813
Canzsl: Cycle-Consistent Adversarial Networks for Zero-Shot Learning from Natural Language
Chen, Zhi, Li, Jingjing, Luo, Yadan, Huang, Zi and Yang, Yang. 2020. "Canzsl: Cycle-Consistent Adversarial Networks for Zero-Shot Learning from Natural Language." 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). Snowmass, United States 01 - 05 Mar 2020 United Stated. IEEE (Institute of Electrical and Electronics Engineers). https://doi.org/10.1109/WACV45572.2020.9093610