Is the Billboard Hot 100 Broken?

What are the hottest songs in the country right now?

Maybe it’s “One Dance,” Drake’s latest dance-floor anthem? Or is it “Panda,” the inescapable trap from newbie Brooklyn rapper Desiigner? What about Justin Timberlake’s pop jingle, “Can’t Stop the Feeling”?

Billboard publishes 111 different weekly charts that track the current popularity of songs and albums across a broad variety of genres and distribution platforms. But to answer the question above, you’d typically consult Billboard’s flagship chart, the Hot 100.

Published since 1958, first in the physical Billboard magazine and now also online, the Hot 100 purports to measure the “overall popularity” of all songs commercially distributed in the U.S. While the math and components that constitute the Hot 100 have, naturally, evolved over the course of six decades, the chart has, since its inception, been the authority on any given song’s popularity in the U.S.

The Hot 100 used to measure the popularity of commercial singles — think “Smooth,” the Carlos Santana and Rob Thomas collaboration that rode the no. 1 song spot for 12 straight weeks in 1999 on the strength of massive radio play and platinum cassette and CD sales. But today’s marketplace doesn’t neatly distinguish among official singles (e.g., “One Dance”), album cuts (“Hype”), and mixtape material (“Jumpman”). The guy behind all three of those songs, Drake, has notably dominated the Hot 100 chart by dropping all sorts of releases at once. (For the record: “One Dance,” Drake’s first no. 1 single, spent 10 weeks atop the Hot 100 before being usurped this week by Sia’s “Cheap Thrills.”)

The questions now are whether and how the Hot 100 can sustain its singular usefulness in a marketplace that contains multitudes. Piracy is immeasurable. Sales and streams aren’t comparable to one another. Streaming music exclusives, such as Beyoncé’s single “Formation,” further confound assessment of a song’s true reach. The musical ecosystem is more fractured than ever, and Billboard is trying to create a coherent statement from all the pieces.

According to Silvio Pietroluongo, vice president of charts and data development at Billboard, the Hot 100 is still the industry standard. “The Hot 100 is, by leaps and bounds, our most-trafficked chart,” Pietroluongo says. “It’s still the most-quoted chart among media, labels, PR companies when there is a success story to publicize.” As a record of success, the chart’s credibility persists — despite the fact that a no. 1 hit single in 2016 has achieved a quantifiably different feat than a no. 1 hit single as recently as the turn of the century.

As of 2014, the Billboard Hot 100 is calculated using data from three subcategory singles charts:

• Radio Songs, a Nielsen Audio measure of terrestrial and satellite airplay
• Streaming Songs, a measure of online traffic measured by the traditional music track service Nielsen SoundScan
• Digital Songs, a measure of online sales, also measured by SoundScan

(Billboard charts have relied on Nielsen data since 1991, when the publication made the switch from phone research to SoundScan’s purchase-tracking system.)

At face value, the distinction between what qualifies as success on each of the latter two charts might seem a bit wonky; it’s all just music on the internet. In recent weeks, listeners have purchased Justin Timberlake’s single “Can’t Stop the Feeling,” the top song on the Digital chart, more than they’ve purchased anything else, whereas listeners have streamed the rookie Desiigner’s single “Panda,” the top song on the Streaming chart, more than they’ve streamed anything else.

They’re both successful songs; both no. 1 hit records, in fact. But only one song, “Panda,” held its no. 1 spot on the Hot 100 for multiple weeks, largely on the strength of its total weekly streams — a data point that Billboard and SoundScan have been counting toward the Hot 100 for less than two years. On the Billboard albums chart, which measures sales, 1,500 song streams are equal to one album sale. But for the Hot 100, which is effectively a hybrid chart measuring sales and airplay, it’s a free-for-all: “You’ll have Spotify streams, Apple Music streams, video plays on YouTube, you know, even incorporate user-generated clips that incorporate the actual music into our calculations,” Pietroluongo says. “We have weighted factors, and there’s a formula to it.” But Billboard has yet to reveal exactly how its math works, and what it favors.

Desiigner initially released his street single, “Panda,” via SoundCloud in December 2015. Five months later, the song became a no. 1 hit record. “Panda” had shot onto the Hot 100 shortly after Kanye West incorporated it into “Pt. 2,” a cut from West’s February album release, The Life of Pablo — initially a Tidal exclusive. “Panda,” on the other hand, was free to stream everywhere. And so “Panda,” a viral hit unbound to a single streaming service, quickly eclipsed the chart positions of “Pt. 2” and every other song on Pablo. Kanye is the bigger artist, and yet Desiigner’s single is, by far, bigger than any song on Kanye’s album.

Pietroluongo describes the limited reach of platform-exclusive music — such as Pablo — as a challenge for artists, labels, and music-streaming services to sort out amongst themselves. “The more places your song is, the more likely people are to stream it,” he says. “If you’re limiting it to one place, it may limit your total volume for the week.”

Stream, steal, or buy: Those are your choices. The premium streaming services represent just one batch of countless channels by which consumers can hear music. And so Billboard now bears the complex task of incorporating traffic from an ever-widening variety of platforms — YouTube, Vevo, Spotify, Apple Music, Tidal, Pandora, Vine, Twitter, etc. — into a standardized accounting that ranks all these songs together. Billboard has struggled to do this, though not for lack of some recent, belated attempts to measure streaming beyond the official figures provided by the major players. (Had “Panda” come out before Billboard started counting YouTube views in 2013, the song likely would have never topped the Hot 100.)

The magazine’s Twitter Top Tracks chart, a weekly measure of the most-mentioned songs on the social media service, represents one attempt to make sense of the noise. Take “Formation,” for instance. The first single from Beyoncé’s latest album, Lemonade, initially went live the night of Feb. 6 — a day before her halftime performance at Super Bowl 50 — as an unlisted music video on YouTube. Hours later, “Formation” was available on Tidal. Immediately upon its release, the song generated a deafening volume of buzz given Beyoncé’s stature, her song’s controversial subject matter, and her forthcoming Super Bowl show. Once Beyoncé and her dancers, styled as Black Panthers, performed “Formation” live at Levi’s Stadium, the song’s buzz grew somehow louder.

While “Formation” quickly shot to the no. 2 spot on the Twitter Top Tracks chart the week of Feb. 20, 2016, the song didn’t break into the Hot 100 until May, following Tidal’s commercial release of Lemonade in full. Only by checking the Streaming Songs and Top Twitter Track charts, instead of the Hot 100, would a reader deduce that “Formation” peaked in the public consciousness in the weeks immediately following Super Bowl 50.

Pietroluongo explains that music streaming services do exercise significant control over Billboard’s ability to account for the streams and sales of exclusive releases, like those from Rihanna and Beyoncé (for Tidal) and Drake and Chance the Rapper (for Apple Music). “There are those certain times where a streaming service may not give us the information because it’s exclusive. It’s really nothing new historically — there was a time when Walmart was selling an album exclusively and they didn’t report it because they didn’t want to be transparent about the data for certain releases,” he says. “A lot of times the decision can come from the service itself, it can come from the label, it can come from the artist, the manager — so there are certain times where we may not be getting proper data from these streaming services.”

And that’s not to mention user content–sharing platforms such as SoundCloud (whose data Pietroluongo says Billboard doesn’t incorporate into its charts due to “a combination of factors,” including a failure to strike a partnership with the company) as well as file-hosting piracy hotbeds such as BitTorrent and Zippyshare. Combined with premium streaming, the way we consume music in 2016 has become so informal and porous that official measurement is a crapshoot.

If you’re trying to get a handle on the musical landscape, there are emerging alternatives to Billboard and the Hot 100. Since 2009, the song-identification service Shazam has published its own set of music-streaming charts, including a Top 100 chart that uses Shazam app traffic to determine the most popular songs in the U.S. as well as 50 other countries; and a Future Hits chart that tracks Shazam user interest in new and emerging releases. Spotify has a similar charts model, and on top of that the service provides hit-predictive curation based on user behavior. “Rap Caviar,” a hip-hop playlist managed by in-house curation expert Tuma Basa, is a mix of big, established singles (Drake’s “Controlla”) and more speculative picks (Travis Scott’s “Pick Up the Phone,” French Montana and Kodak Black’s “Lockjaw”). (Spotify declined to comment for this article.)

In a recent BuzzFeed report about the curation experts employed by Spotify, Apple Music, and Google, journalist Reggie Ugwu explores the ways in which digital playlists have drastically reorganized the music economy. “Spotify says 50% of its more than 100 million users globally are listening to its human-curated playlists (not counting those in the popular, algorithmically personalized ‘Discover Weekly’), which cumulatively generate more than a billion plays per week,” Ugwu writes. “According to an industry estimate, 1 out of every 5 plays across all streaming services today happens inside of a playlist.” (Spotify’s recent partnership with the website Genius, which is providing the streaming platform with lyrics annotations, playlist curation, editorial content, and artists’ commentary, could signal a trend of music channels further evolving toward the record stores of yore.)

Programming and tracking are two different functions, of course; the Billboard charts measure airplay but aren’t themselves a radio station. Mind you, FM radio is still massively popular, reaching millions of listeners, despite the digital paradigm shift, and so the Hot 100 is still a useful, unique, and authoritative measure of traditional airplay. It’s just no longer the all-encompassing measure that it once was. Now more than ever, determining the hottest song in America in any given week will depend on who you ask.