Minnifield attended Henry Clay High School in Lexington, Kentucky.
Considered too small for college football at just 5'9" and 140 pounds, he walked on to the football team at the University of Louisville, earning a scholarship spot for his final three years. As a junior in 1981, he led the nation in average yards per kickoff return (30.4) and led his team in punt returns.
After graduation in 1982, Minnifield joined the Chicago Blitz (later the Arizona Wranglers) of the USFL. He successfully sued the Wranglers for the right to move to the NFL, and signed as a free agent with the Browns in 1984. Minnifield became a fixture at cornerback for Cleveland and was named to the Pro Bowl four straight years from 1986 to 1989. Known for his aggressive bump-and-run coverage and hard hitting style, he was named to the NFL 1980s All-Decade Team as selected by voters of the Pro Football Hall of Fame. Perhaps more importantly to Cleveland fans, he and fellow cornerback Hanford Dixon originated and named the Dawg Pound cheering section at Cleveland Stadium.
Upon retirement, Minnifield founded Minnifield All-Pro Homes, a homebuilding company in Lexington. In 1993, he became the first African American executive named to the Lexington Chamber of Commerce Board, and as of 2000, he was the only African American home builder in Lexington.
Minnifield's son, Chase, played in the NFL as a cornerback for the Washington Redskins.