Imagine having a box full of millions of colored particles and you’re tasked to arrange them to form a stunning, lifelike image. Seems daunting, right? Now, imagine a magical guide that can help you – slowly, step by step – organize those particles into a picture-perfect image. This magical guide is what we call a diffusion model in the world of machine learning.

Diffusion models, or to give them their full name, Denoising Diffusion Probabilistic Models (DDPMs), are a type of generative model that can take an initial state of random noise and, through a sequence of transformations, refine it into something meaningful and coherent.

These models are game-changers in AI and have led to some stunning results, creating high-quality samples that compete with other generative models, like the famous GANs and VAEs.

A Real-World Example

Suppose you’re working on a project that needs a large number of images of cats, but you don’t have any at your disposal. A diffusion model can help you here. Start with a random noise image – think of a TV static screen. The diffusion model then iteratively refines and transforms this noise, and voila! After hundreds or even thousands, of transformations, you’ll get a lifelike image of a cat.

It’s like watching a digital artist start with a blank canvas and then gradually paint an image. But instead of an artist, it’s an AI, and instead of a blank canvas, it’s random noise.

The Drawbacks

While these models sound exciting, they aren’t perfect. The main challenge is that they are computationally heavy. The generation process is slow and can take hundreds or thousands of steps, making them less practical in real-time scenarios.

The potential of diffusion models is vast and the research is evolving fast. As we see advances in computational power and more efficient algorithms, these drawbacks may soon become a thing of the past.

Let’s chat about AI on Discord.

I will share my experience generating art with AI. I am not an artist, but I have ideas and access to a web browser that has access to power Generative AI tech. There are many platforms to experience, but I recommend that you pick one, try out prompts, iterate, and share. I picked Stable Diffusion.

Stable Diffusion is an advanced generative model for creating images, inspired by the diffusion process in physics. When given a text prompt, the model generates an image based on the content and themes described in the prompt.

M. C. Loves Robots

I love the artstyle of M. C. Escher. With Generative AI, I can now get an idea of what Escher would create if he was obsessed with robots.

Let’s unpack the prompt used to generate this image

The prompt “M. C. Escher lithograph, robots, gears, lights, wires” consists of several components that give the model an idea of what elements to include in the generated image.

1. “M. C. Escher lithograph” – This part of the prompt instructs the model to generate an image inspired by the artistic style of M.C. Escher, a Dutch graphic artist known for his mathematically inspired works that feature impossible objects, optical illusions, and intricate patterns.
2. “robots” – This keyword tells the model to include robots as one of the central subjects in the image. The robots may be designed in a way that reflects Escher’s style or is inspired by his works.
3. “gears” – The model should incorporate gears into the image, possibly as part of the robots’ mechanisms or as a separate visual element.
4. “lights” – This term suggests that the image should have a focus on lighting, either by using the lights as a part of the composition or emphasizing the way they interact with the other elements in the scene.
5. “wires” – The presence of wires indicates that the generated image should have a somewhat intricate or complex appearance, with wires possibly connecting various elements or being part of the robots.

When the Stable Diffusion model receives this prompt, it will generate an image that combines these elements. The resulting artwork might be an intricate scene of robots, gears, and wires, all rendered in the distinctive style of M.C. Escher, with a focus on the interplay of light and shadow. The specific details of the generated image will depend on the model’s interpretation of the prompt and its internal knowledge of the various elements mentioned.

Each word in a Generative AI prompt matters. You can get wild variations by changing out the words and experimenting.

Stable Diffusion Prompt Engineering Tips

Stable Diffusion is a groundbreaking generative AI art model that enables the creation of stunning and unique artwork. By providing the model with a carefully designed prompt, you can guide its creative process to generate art that aligns with your vision.

Be Specific and Clear

When writing prompts for Stable Diffusion, it’s crucial to be specific and clear in your description. This ensures that the model can accurately interpret your intentions and generate artwork that matches your vision. For example, instead of writing “a landscape,” provide more detail such as “a serene sunset landscape with a calm lake and a silhouette of a tree.”

Incorporate Emotions and Atmosphere

To create truly captivating art, consider incorporating emotions and atmosphere into your prompts. This can help the model generate art that evokes a certain mood or feeling. For instance, instead of writing “a cityscape,” try “a bustling cityscape with a lively atmosphere, filled with people enjoying their day.”

Experiment with Different Styles

Stable Diffusion is capable of producing artwork in various styles, from abstract to photorealistic. When crafting your prompt, consider specifying a particular artistic style or movement to inspire the model. For example, you could write “a vibrant, impressionistic painting of a Parisian street café.”

Combine Unusual Elements

One of the most exciting aspects of generative AI art is its ability to combine seemingly unrelated elements to create unique and thought-provoking pieces. To harness this potential, try writing prompts that incorporate contrasting or unexpected components. For example, “a surreal scene of a whale floating above a city skyline with hot air balloons in the background.”

Iterative Refinement

Don’t be afraid to iterate and refine your prompts. If the generated artwork isn’t quite what you envisioned, try adjusting the wording or adding more details to your prompt. It’s essential to experiment and adapt your prompts to hone your skills in guiding the model’s creativity.

Stable Diffusion offers an exciting avenue for artistic expression, and writing effective prompts is key to unlocking its full potential. By being specific, incorporating emotions, experimenting with styles, combining unusual elements, and refining your prompts, you can guide the model to generate visually stunning and emotionally evocative artwork. So go ahead, unleash your creativity, and let Stable Diffusion bring your artistic visions to life!

Use ChatGPT if you get stuck

A neat thing that you can do is use ChatGPT if you get stuck. Use AI to write AI prompts. Ask ChatGPT to give you ideas for prompts for an artist that you like. I always loved President Obama’s presidential painting by Kehinde Wiley. A robot is likely to be our future president and Kehinde will be tapped once again to paint the portrait. I used ChatGPT to help me generate ideas for this painting.

Enjoy the journey

We are going through some weird times. Enjoy the journey. Try out prompts and share the results on our Discord server. I would love to see what you come up with.