During his thirty-five-year career, Brumfield won 4,573 races in 33,222 rides. He retired from racing in 1989. Brumfield was the "track all-time leading rider in terms of races won (925)" at Churchill Downs, where he won 16 riding titles. His record was later broken by Pat Day, who won more than 2,000 races at Churchill Downs in his career. Brumfield rode Kauai King to victory in the 1966 Kentucky Derby.
He was inducted into the National Museum of Racing and Hall of Fame in 1996.
Following his retirement from the saddle, Brumfield worked as a racing official at various racetracks. He most recently served as a steward at Gulfstream Park in Florida for their 2015 winter-spring meet. Brumfield often appears at Churchill Downs for promotions honoring past Kentucky Derby winning jockeys.