In the past few weeks, I deep dived into an exploration revolving around the use of physical interfaces to feed and interact with a real-time img2img diffusion pipeline using Stream Diffusion and SDXL Turbo. What really captivated me is to use my hands, objects, art supplies, tools, and light to create images and scenes. 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗠𝗲𝗱𝗶𝗮 + 𝗧𝗼𝗼𝗹𝘀 I experimented with clay, manipulating different types and colors along with a selection of prompts. I used a magnifying glass, tracked in real-time, to focus the diffusion process on specific areas. Combining these tools created a dynamic and inspiring experience. Using magic clay to layer shapes and colors as a base for revealing landscapes and hidden worlds, and the magnifying glass to focus and reveal these details, was particularly effective. 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗟𝗶𝗴𝗵𝘁 I used light as my method of interaction with the img2img diffusion. This approach felt special right away. There was something magical about holding a physical light source and seeing it influence the generated visuals. I iterated on this technique with themes like Rococo architecture, flowers, Brutalist architecture, hidden worlds, and origami landscapes. 𝗜𝗻𝗸 + 𝗛𝘆𝗯𝗿𝗶𝗱 𝗙𝗼𝗿𝗺𝗮𝘁𝘀 I also used ink in milk as a means of physical interaction with the diffusion pipeline. As I drop ink into milk, shapes come alive instantly. By learning to manipulate the combination of physical and digital elements, I steered the generated output toward my areas of interest. These iterations extended beyond ink in milk to include the format in which these elements are contained: a circular plate or a triptych of small stainless steel trays. These formats provide a structured yet flexible framework to explore themes and narratives across multiple visual spaces. It's magical. Some of that last iteration has been captured in this insightful article by Fast Company: lnkd.in/e6UUCTyr #stablediffusion #realtime #ai