He spent his entire working life at J&J. He joined J&J as a sales representative for the McNeil Pharameutical division in 1971 and eventually became the head of J&J's Ethicon Endo-Surgery business in 1992. He became the head of J&J's pharmaceutical operations in 1998 and then became J&J's CEO in 2002. As CEO, Weldon engineered some of the largest acquisitions in J&J's history including the purchase of Alza and Pfizer's consumer-health product line. In 2009, he earned a total compensation of $22,830,834, which included a base salary of $1,802,500, a cash bonus of $12,831,146, stock awards of $2,762,532, option awards of $5,238,069, and other compensation of $196,587. In 2011, the New York Times named him on its list of "The Worst C.E.O.'s of 2011" for the increased number of Johnson & Johnson product recalls under his leadership. Additionally, as of 2013, his story is a case of study at the Leadership & Corporate Accountability course at Harvard Business School. Weldon retired as chairman of Johnson & Johnson on December 28, 2012 and was reported to receive $143.5 million in retirement pay.
In Weldon's first full year as J&J's CEO, total revenues increased from $32.3bn to $36.3bn and net earnings from $5.7bn to $6.6bn.  In 2011, his last full year as CEO, revenues were $44.7bn and net earnings were $9.7bn.  He also sits on the JPMorgan Chase & Co. Board of Directors and the Quinnipiac University Board of Trustees.
He married his high school sweetheart, Barbara, in 1969. They have two children.