THE SAME MATH THAT MAKES AI CREATIVE IS WHY SO MANY COUNTRY SONGS ARE ALIKE

What a Luke Combs concert, video games, and a Google research paper taught me about creativity

My mid-week article is about AI as usual, but also about our own creativity, cat pictures, video games, and country music. I had already started drafting this article when I sat at Wembley Stadium having arrived early to watch Luke Combs. I didn’t know two of the opening acts, and they played country songs I hadn’t heard before but at the same time sounded surprisingly familiar, yet unique. This is what this article is about.

When a diffusion model is trained on thousands of cat pictures it does not then regurgitate out the exact cats it learned over and over again. Thankfully it generates brand new cats, that are unique yet recognisable as cats, just like those familiar country songs I mentioned earlier. Cats with whisker configurations and fur patterns that never existed in the training set. A Google Research paper, presented at ICLR 2026, argues that this “creativity” is not magic, nor is it some emergent spark of machine intelligence or consciousness. It is a predictable mathematical consequence of neural networks being too clumsy to learn the sharp, exact function they are supposed to learn, and settling instead for a smoothed-out, interpolated version. Which may be the same dynamic, that describes a surprising amount of human creativity too.

The force field that is supposed to memorize

In order to understand what is going on, we can picture what happens when a diffusion model generates an image. Training starts with real data (cat photos in this case) and intentionally corrupts them with noise until they are unrecognizable. The model then learns to reverse this process, transforming random static back into a coherent picture one tiny denoising step at a time.

The function used to tell each pixel where to go during this cleanup process is called the score function. We can picture this as a force field hanging in space. If we imagine random noise as a cloud of gas particles scattered across a room, the score function is the invisible field that pulls each particle in a specific direction until they all snap into the shape of a meaningful thing, in this instance a cloud shaped like a cute kitten.

Here’s the thing though, if you had the perfect score function, the one computed exactly from the training data with no approximation, every single particle would get pulled to land precisely on top of a training example and we’d get the same cats we have seen before. In a simple one-dimensional world with only two training points at +1 and -1, the perfect score function contains a sharp sign-change at the midpoint, zero. It acts like a continental divide: particles on the left get yanked to -1, particles on the right to +1, and nothing ever lands in between. Pure memorization. The model then becomes a retrieval tool, not a tool that can create something new.

Why neural networks can’t help but blur

In practice though, diffusion models never learn the perfect score function. They learn an approximation, and neural networks are fundamentally bad at learning sharp cliffs.

The reason is weight decay, a regularization technique that penalizes a network for using extreme parameter values. It is a way of telling the model: “keep it simple, do not overcommit to any one pattern.” The moment you apply weight decay, that crisp sign-change at the midpoint gets softened. Instead of a sheer drop, the network learns a gentle slope. Particles in the middle zone now feel a weaker pull. They slow down and eventually settle in the space between the training points, producing something new that is neither +1 nor -1 but a plausible blend.

The stronger the weight decay, the wider that “interpolation zone” becomes. And even without explicit weight decay, gradient-based training has its own implicit preference for simpler, smoother functions. The network cannot help itself. Learning anything means learning a blurred version.

The researchers at Google trained small two-layer networks on a one-dimensional toy problem to confirm this, varying the strength of weight decay in the AdamW optimizer. The results matched the theory cleanly. Sharp score function, memorization. Smoother score function, interpolation. Picture a photograph of a crisp line, ever so slightly defocused. The edge is still there, but the boundary has softened into something more generous.

The quality-vs-novelty balancing act

Real images, of course, do not live in one dimension. A high-resolution photo occupies a pixel space with millions of dimensions, and the vast majority of that space is nonsense, random static that means nothing to a human eye. The images we recognize sit on a thin, crumpled surface tucked inside that enormous volume, what researchers call the data manifold.

Here is where the story gets genuinely elegant. In multiple dimensions, score smoothing does not apply equally in every direction. Along directions that are tangential to the data manifold (running parallel along the surface, sliding from one cat photo toward another), smoothing slows the particles down, just like in the one-dimensional case. Particles drift into the blank spaces between training points. This is where the novelty appears, those brand new cats we mentioned earlier.

But along directions that are normal to the manifold (pointing straight toward the surface from the surrounding noise), the perfect score function is already smooth. In fact, if the manifold is flat, it is just a straight line. Further smoothing in that direction does almost nothing. The pull toward realism stays strong.

This direction-dependent effect is the hidden mechanism that balances quality against novelty. Without it, you get one of two failures: either every particle collapses to a memorized training image (no smoothing at all), or particles stall out in noisy empty space and generate blurry, unrecognizable smudges (smoothing in every direction). What diffusion models actually do is selectively apply the brakes only where it produces newness, without compromising the drive toward the manifold. The result: images that look real and have never been seen before.

The interpolation we do not like to admit

At this point you might be thinking: well, that is not real creativity, it is just blending things the model learned. And you would not be wrong, technically this is exactly what it does. But before dismissing interpolation as a lesser kind of creation, it is worth looking in the mirror.

Humans borrow, remix, and interpolate from existing patterns far more than we like to think. Take country music, which I do like, so bear with me! The genre has been joked about for decades as a machine that takes three inputs (girls, trucks, beer) and churns out an endless stream of variations on the same song. This is funny because there is truth in it, and it is also not remotely unique to country. Pop music runs on the I-IV-V-I chord progression the way a diffusion model runs on a smoothed score function. It is a template. A manifold of acceptable harmonic motion. You can build a career inside it, and many people have, and there’s absolutely nothing wrong with it.

This is not a bug or an uncomfortable issue we have to contend with, this is how we operate, our culture. Templates, tropes, and conventions are how creative traditions actually work. The question is not whether a work interpolates from what came before. Almost everything does. The question is where on the spectrum it sits, between “slight variation on an established formula” and “genuinely novel genre.” The distance between those two poles is a gradient, and most of us, most of the time, are operating somewhere in the middle.

A crash course in creative interpolation

I got a crash course in this when I moved to Europe in the early 90s. Growing up, my video game universe was defined by Japanese gaming giants: Nintendo, Taito, Sega, Bandai, etc… The aesthetic was bright, character-driven, cartoon-like. Then I landed in a world where home computers like the Amiga and Atari ST dominated, and suddenly I was playing games from studios like Bitmap Brothers, Psygnosis, and Team17 that looked entirely alien and even weird to me.

Bitmap Brothers games had a visual signature I had never encountered: metallic color palettes, chunky, moody. Xenon 2, Speedball 2, and Gods. These were not just different; they played rather differently than I was used to. They felt like alien artifacts from a parallel European dimension, one where video games had evolved along a different branch of the creative tree.

What I did not appreciate at the time is that this was the same creative process producing a different flavor, because the starting conditions, reference pool, and overall culture surrounding the creators were different. European developers were drawing on their own influences: sci-fi illustration, European comics, the demoscene aesthetic born from edgy home-computer hobbyists. Japanese developers were drawing on manga, anime, and a different set of arcade conventions. Both ecosystems were highly derivative internally. Everyone was borrowing from everyone else. But because the “training data” (the cultural manifold each team was interpolating along) was not the same, the outputs were strikingly different.

Different interpolation zones produce different worlds. The mechanism is identical. Only the inputs change.

Creativity is a spectrum, not a switch

The Google Research paper gives us a precise vocabulary for something we have always known intuitively. Creativity, whether in a neural network or a human mind, is rarely pure invention from scratch. It is interpolation along a manifold, smoothed by the imperfections of the learning process.

For the diffusion model, the manifold is the set of all realistic images buried in pixel space, and the smoothing comes from the fact that neural networks, by their mathematical nature, cannot learn sharp boundaries. For humans, the manifold is the set of all the influences we have absorbed over a lifetime, and the smoothing comes from the fact that we cannot perfectly reproduce those influences even when we try. We drift. We blend. Have you ever tried to draw a well known character like Sonic, or Mickey Mouse purely from memory? If you are anything like me the character will look familiar, but definitely different to the original.

“What we call the ‘creativity’ of diffusion models might actually be a predictable mathematical result. Because neural networks are never ‘perfectly’ sharp, they create bridges that interpolate between known data.” — Zhengdao Chen, Google Research

Recognizing this does not diminish creativity. It clarifies where the real work happens: in choosing which influences to blend, how far to interpolate, and when to push beyond the existing manifold entirely.

Before we’re quick to dismiss generative AI as incapable of creating anything truly new, we should look at ourselves first. We can absolutely create new things. That’s not in question. But that pop song you love was not the first to use those chords. That retro platformer you rate as the best ever made was not the first platform game ever coded. And that murder mystery you couldn’t put down was not the first murder novel ever written. There’s nothing wrong with existing within the manifold.

For the full derivation and formal results behind these ideas, read the original paper: “On the Interpolation Effect of Score Smoothing in Diffusion Models” (Chen, ICLR 2026).