However, he is best known today for his appearance in one of the first "talkies"—The Jazz Singer—the first feature film with sound to enjoy wide commercial success, in 1927. The film is the story of a young Jewish man, Jakie Rabinowitz, who defies the traditions of his devout Jewish family when he seeks to become a popular singer of secular music. In The Jazz Singer Jolson performed the song "Mammy" in blackface. In fact, Jolson's singing should not be considered as jazz, for his style remained forever rooted in the style of the vaudeville stage at the turn of twentieth century. Jolson ad-libbed his best-known catchphrase, "You ain't heard nothing yet!," in The Jazz Singer. It is also the name of a song he co-wrote earlier.

Among the many songs he popularized were "You Made Me Love You (I Didn't Want to Do It)," "Rock-A-Bye Your Baby With A Dixie Melody," "Swanee" (songwriter George Gershwin's first success), "April Showers," "Toot, Toot, Tootsie, Goodbye," "California, Here I Come," "When the Red, Red Robin Comes Bob-Bob-Bobbin' Along," "Sonny Boy," and "Avalon."

Jolson was the first musical artist to sell over ten million records in total. While no official Billboard magazine chart existed during his career, its staff archivist Joel Whitburn used a variety of sources to estimate the hits of 1890-1954. By his reckoning, Jolson had the equivalent of 23 Number-One hits, the fourth-highest total ever, trailing only Bing Crosby, Paul Whiteman, and Guy Lombardo. Whitburn calculated that Jolson would have topped one chart or another for 114 weeks.

Jolson's legacy is considered by many to be severely neglected today because of his use of stage blackface. Jolson was not a racist, and blackface as a theatrical convention was used by many performers (both white and black). Some critics argue the make-up acted as a mask for Jolson, giving his performances a greater spontaneity. However, the make-up had roots in minstrelry shows where Jolson got his start, and is today viewed by many as racially offensive.

Jolson was billed as "The World's Greatest Entertainer," which is how many of the greatest stars, including Bing Crosby, Frank Sinatra, Judy Garland, Elvis Presley, Mick Jagger, Jim Morrison, Rod Stewart, and Jackie Wilson referred to him. Charlie Chaplin wrote in his autobiography that Jolson was one of the most electrifying entertainers he had ever seen.

Forty-four years after Jolson's death, the United States Postal Service acknowledged his contribution by issuing a postage stamp in his honor. The 29-cent stamp was unveiled by Erle Jolson Krasna, Jolson's fourth wife, at a ceremony in New York City's Lincoln Center on September 1, 1994. This stamp was one of a series honoring popular American singers, which included Bing Crosby, Nat King Cole, Ethel Merman, and Ethel Waters.

Jolson has three stars on the Hollywood Walk of Fame: For his contributions to the motion picture industry, to the recording industry, and to the radio industry. In August 2006, Al Jolson had a street in New York named in his honor after nine years of attempts by the international Al Jolson Society.