Born July 10, 1818 near Mount Sterling, Kentucky, Williams attended the common schools and graduated from Miami University in Oxford, Ohio, in 1839. He studied law, was admitted to the bar in 1840, and commenced practice in Paris, Kentucky. He served in the Mexican-American War, first as a captain of an independent company attached to the 6th U.S. Infantry, and afterward as a colonel of the Fourth Regiment of the Kentucky Volunteers. He received the nickname "Cerro Gordo Williams" for his gallantry at that battle.
Williams was a member of the Kentucky House of Representatives in 1851 and 1853. He became known as a leading proponent of states rights. He was initially an anti-secessionist, but abhorred President Abraham Lincoln's policies and cast his lot with the Confederacy.
With the outbreak of hostilities, Williams traveled to Prestonburg in early 1861 and was commissioned colonel of the 5th Kentucky Infantry. He served initially in the Eastern Theater, initially under Humphrey Marshall in southwestern Virginia. He participated in Marshall's ill-fated invasion of eastern Kentucky in 1862. He was promoted to brigadier general in late 1862 and assigned command of the Department of Southwestern Virginia.
He organized a brigade of cavalry and helped resist Ambrose Burnside's invasion of eastern Tennessee in the autumn of 1863, participating in the Battle of Blue Springs. He resigned that command and transferred to Georgia, assuming command of the Kentucky regiments in the cavalry of Joseph Wheeler in June 1864. He received a formal resolution of thanks from the Second Confederate Congress in the fall of 1864 for his actions at the Battle of Saltville. He surrendered in 1865.
Williams returned home following the war and went on to engage in agricultural pursuits, with his residence in Winchester, Kentucky.
He again became a member of the State House in 1873 and 1875. He ran unsuccessfully for Governor of Kentucky in 1875, and was a presidential elector on the Democratic ticket in 1876. He was elected to the United States Senate in 1879 and served from March 4, 1879 to March 3, 1885. He failed in his reelection bid and returned to his agricultural pursuits.
Williams became involved in land development in Florida in the late 1880s. Along with a partner, Louisville businessman Walter N. Haldeman, the publisher of the Louisville Courier-Journal; they founded the town of Naples, Florida.