Johnson coached under Johnny Majors at Iowa State University, at the University of Oklahoma working alongside future rival Barry Switzer, at Arkansas, and Pittsburgh before being named head coach at Oklahoma State. He coached for five seasons at Oklahoma State University before taking the head coaching job at the University of Miami.

In 1984, Johnson was hired by the University of Miami to replace former coach Howard Schnellenberger who had won Miami's first national championship in 1983 and who had departed for the recently formed United States Football League. Amidst an initial response of "Jimmy Who?" by the fans and media and a shaky 8-5 record his first season, Johnson continued what came to be known as "The Decade of Dominance". In his five years at Miami, Johnson compiled a 52-9 record, appeared in five New Year's Day bowl games, winning one national championship (1987) and playing for a second.

In 1989, Jerry Jones, the new owner of the NFL's Dallas Cowboys, a long-time friend and former University of Arkansas teammate of Johnson's, asked him to be the new head coach, replacing Tom Landry, who had coached the team since its beginning in 1960. Johnson was reunited with former Miami standout Michael Irvin.

Jimmy Johnson served as head coach of the Cowboys from 1989 through 1993. He became only the fourth man in NFL history after Vince Lombardi, Don Shula, and Chuck Noll to coach consecutive Super Bowl winners, winning Super Bowl XXVII in 1992 and Super Bowl XXVIII in 1993.

He left the Cowboys to coach the Miami Dolphins. After leaving the Dolphins, Johnson became a TV studio analyst again for Fox Sports NFL coverage.