Klein was born in the Bronx, the son of Frieda (née Moskowitz) and Benjamin Klein and was raised in a "prototypical 1950s Bronx Jewish" environment. After graduating from De Witt Clinton high school, Klein planned to study medicine. However, he changed his mind during his studies in university. After graduating from Alfred University, he studied at Yale Drama School when he got wind of an opportunity to audition for The Second City. In a piece he wrote for the improvisational troupe's book, Klein recalled sitting in a room full of other hopefuls including Fred Willard. Klein's audition consisted of an improvisation set with Willard about two guys in a nightclub, which was successful enough to get Klein and Willard hired to join The Second City. Klein had said another young man, Billy Dee Williams, was invited to join the troupe but turned it down.
His first major appearance was as host of the 1970 summer replacement television series Comedy Tonight, on which were introduced many of the routines that in the next few years would be released on record albums. His extensive routines about the Watergate scandal made him highly popular in the 1970s. He was one of the first observational comedians. Klein appeared in several stand-up comedy specials on HBO dating to its very early broadcast days. He has traditionally concluded these shows with his "I can't stop my leg" routine. In 1979, Klein was nominated for a Tony Award for Best Actor in a Musical for his role in They're Playing Our Song.
He has appeared in such movies as The Owl and the Pussycat, Primary Colors, Radioland Murders and The Safety of Objects. He had a recurring role in the TV drama series Sisters. In the 1970s, he hosted Saturday Night Live twice.
Klein is the author of The Amorous Busboy of Decatur Avenue: A Child of the Fifties Looks Back, an autobiography published in 2006.
In March 2007 he starred with Adam Sandler in Reign Over Me, and in September 2007 released a new DVD compilation of his eight live HBO specials titled Robert Klein: The HBO specials 1975-2005
Klein has made several albums, the most successful being his first two. In Child of the Fifties, Klein talks about his life as a child in the 1950s: about air raid drills, Johnny Mathis music, showing off condoms while at the high school dance, the high school lunch ladies, Senator Joseph McCarthy, Governor Averill Harriman (of New York), meeting Yankee stars, the Yankees losing the World Series, and much more. He also goes into other things that he has observed in his life, such as substitute teaching, FM radio disc jockeys (as they were in 1972, when the album came out), late night delis, and two songs that he wrote and sang himself: "Fabulous 50's", and "Middle Class, Educated Blues."
Mind Over Matter, his next album, included extensive discussion of the Watergate scandal and another song, the title track, about a kid who turned to humor to become popular.