Cafego was drafted as the number one overall pick in 1940 NFL draft by the Chicago Cardinals. He eventually played for the Brooklyn Dodgers. After playing one season, his career was interrupted by a brief stint of Army service in World War II. During this time he appeared in several games for the Newport News Builders of the Dixie League. Returning to the Dodgers in 1943, he was traded to the Washington Redskins after five unspectacular games. For the 1944 and 1945 seasons, Cafego played for the Boston Yanks before retiring.
After his playing days were over, Cafego served as an assistant coach at Wyoming, Furman, Arkansas, and 30 years at his alma mater, Tennessee, serving under a total of six different head coaches during his UT coaching career. He was also the head coach of Tennessee Volunteers baseball from 1958 to 1962. He retired from coaching following the 1984 season.