DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI - Extreme Situated Interaction Access content directly
Conference Papers Year : 2024

DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI

Xiaohan Peng
Janin Koch
Wendy E Mackay

Abstract

Visually oriented designers often struggle to create effective generative AI (GenAI) prompts. A preliminary study identified specific issues in composing and fine-tuning prompts, as well as needs in accurately translating intentions into rich input. We developed DesignPrompt, a moodboard tool that lets designers combine multiple modalities — images, color, text — into a single GenAI prompt and tweak the results. We ran a comparative structured observation study with 12 professional designers to better understand their intent expression, expectation alignment and transparency perception using DesignPrompt and text input GenAI. We found that multimodal prompt input encouraged designers to explore and express themselves more effectively. Designer’s interaction preferences change according to their overall sense of control over the GenAI and whether they are seeking inspiration or a specific image. Designers developed innovative uses of DesignPrompt, including developing elaborate multimodal prompts and creating a multimodal prompt pattern to maximize novelty while ensuring consistency.
Fichier principal
Vignette du fichier
DIS_24_DesignPrompt_Author_Version.pdf (7.76 Mo) Télécharger le fichier
Origin Files produced by the author(s)
licence
Copyright

Dates and versions

hal-04652020 , version 1 (17-07-2024)

Licence

Copyright

Identifiers

Cite

Xiaohan Peng, Janin Koch, Wendy E Mackay. DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI. DIS 2024 - Designing Interactive Systems Conference, ACM, Jul 2024, Copenhagen, Denmark. pp.1-15, ⟨10.1145/3643834.3661588⟩. ⟨hal-04652020⟩
0 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More