Interesting methodology here, but the examples cited are ridiculous. First of all Billboard Top 100 in recent years contains the worst quality modern pop music out there. So you are starting out with a highly flawed data set.
#musictheory #cosomusic

//

Hit songs rely on increasing “harmonic surprise” to hook listeners, study finds | Ars Technica
https://arstechnica.com/gaming/2021/08/hit-songs-rely-on-increasing-harmonic-surprise-to-hook-listeners-study-finds/

^ Also, it is VERY difficult to hear the harmonic shifts in "This is America" during the verses (even for someone with my background). I assert that the only thing "surprising" about that song is the video. I never thought that song was anything special on its own; people are conflating it with the (excellent) video.

(If you think I'm just a #cosoclassical nerd looking down on hip-hop, you're mistaken. I've always been a big fan of artists like Wu-Tang Clan, The Roots, Chance the Rapper.)

The idea of harmonic "surprise" driving listener interest is not new at all, and has been studied before. There is plenty of music from the 1500s that still does this.

To be clear: there is a lot of GREAT pop music out there. I am just saying that most of Billboard is formulaic crap, so that anything that is even remotely "different" stands out in context.

Here is a much better example of what this article is discussing. This is a brilliantly written song.

https://strongsongspodcast.com/episode/single-ladies-by-beyonce

@voltronic

I'm trying to recall the time when most popular music wasn't formulaic crap, and not having great luck.

@mcfate
Are you discounting the many excellent bands and singer-songwriters from the 60s and 70s??? You can't honestly say that Carol King, Stevie Wonder, The Beatles, Led Zeppelin, Marvin Gaye, Jimi Hendrix, etc. are formulaic crap.