Groat was signed by Pirates general manager Branch Rickey just days after graduating from Duke, where he had been a 2-time All-American in basketball and baseball. Both the St. Louis Cardinals and New York Giants were also interested in him, but he had always hoped to play for the Pirates after growing up a few miles away from Forbes Field. He broke in with the Pirates in June, never playing in the minor leagues, and batted .284 over the rest of the year. Afterwards, he pursued his basketball career before serving two years in the Army. He led Fort Belvoir teams to worldwide Army championships in both sports, the first time a single base had won both titles in the same year, hitting .362 in baseball and averaging 35 points per game in basketball.
Returning to the Pirates in 1955, he batted second for the team, with leadoff hitter Bill Virdon later recalling his particular skill at the hit and run. That year he led the NL in putouts for the first time; pitcher Roy Face has noted that Groat was always in the best position for the various hitters, although he didn't have great speed or a strong arm. He batted .315 (fifth in the league) in 1957, along with a career high of 7 home runs; on September 29 of that year against the Giants, he threw out the final batter ever at the Polo Grounds. In 1958 he again hit .300, and led the NL in putouts and double plays as the Pirates finished in second place, the first time they had placed higher than seventh since 1949. He led the NL in putouts and double plays again in 1959, and made his first of five All-Star teams. In the ensuing offseason he was nearly traded for Roger Maris, but the deal was cancelled by manager Danny Murtaugh.
Groat responded with his best year as team captain, becoming the first Pirate to be named MVP since Paul Waner in their last pennant year of 1927, and also the first right-handed Pirates hitter to win the batting title since Honus Wagner in 1911. He missed a few weeks late in the season after having his wrist broken by a Lew Burdette pitch on September 6. In the 1960 World Series against the New York Yankees, he tied Game 1 at 1-1 with a first-inning double and scored to give Pittsburgh the lead; they stayed in front, winning 6-4, with Groat turning a double play to end the game. In Game 7, he had an RBI single and scored in the eighth inning, in which the Pirates scored five runs to take a 9-7 lead; the Pirates won the Series on Mazeroski's famed home run in the next inning.
He batted .275 in 1961, with both he and Mazeroski leading the league in double plays. In 1962 he batted .294, finishing third in the league in doubles (34), and led the NL in putouts, assists and double plays. In November 1962, in the hope of bolstering the team's pitching, general manager Joe L. Brown traded him to the Cardinals in exchange for Don Cardwell. Groat was deeply hurt by the trade, having hoped to become a coach and eventually manager after retiring, and severed all contact with the team until a 1990 reunion of the 1960 champions. He had another outstanding year in 1963, finishing fourth in the league with a .319 batting average – just seven points behind champion Tommy Davis – and collecting 201 hits. He also led the NL with 43 doubles, and was third with a personal high of 11 triples; he was the runnerup in the MVP voting, behind Sandy Koufax.
In 1964 he batted .292 for the pennant-winning Cardinals, again leading the league in assists and double plays and making his last All-Star team. In the World Series against the Yankees, he reached base on Bobby Richardson's error in the sixth inning of Game 4, and scored on Ken Boyer's grand slam in the 4-3 St. Louis victory. Groat also tagged out Mickey Mantle in the third inning of that game on only the second hidden ball trick in World Series play, Bill Coughlin having executed the other, in the 1907 World Series. He scored in the 3-run tenth inning of Game 5, a 5-2 win, and had an RBI groundout in the final 7-5 win in Game 7. After hitting .254 in 1965, he was traded to the Philadelphia Phillies in a six-player deal. He batted .265 for the 1966 Phillies, and his contract was sold to the Giants (now in San Francisco) in June 1967; he ended his career that season with a .156 average in 44 games.
In a fourteen-season career, Groat compiled a .286 batting average with 2138 hits, 39 home runs, 829 runs, 707 runs batted in, 352 doubles and 14 stolen bases in 1929 games.
Groat played college basketball for Duke University. He was twice (1951 and 1952) an All-American, and was named the Helms Foundation Player of the Year in 1951 and the UPI National Player of the Year in 1952 after setting an NCAA record with 839 points. On May 1 of that year, his #10 was the first jersey to be retired in the rafters of Cameron Indoor Stadium, and it remained the only jersey retired by the school until 1980. During the 1951-52 season, he scored 48 points against North Carolina, the most ever scored against the Tar Heels.
After college, Groat was drafted with the 3rd overall pick in the 1952 NBA Draft as a guard for the Fort Wayne Pistons of the National Basketball Association. He played only one season for 26 games with the number 5, and averaged 11.9 points 3.3 rebounds 2.7 assists with a .368 FG%. His basketball career was cut short by military service though; when his enlistment was up, he returned to the Pirates but not to the Pistons. Groat was inducted into the National Collegiate Basketball Hall of Fame in 2007.