Williams was best known as a music critic, especially in reference to jazz. He co-founded The Jazz Review and wrote a series of books on the subject. He directed the Jazz and American Culture programs at the Smithsonian Institution for a decade (1971–81).
Williams also wrote on subjects on popular culture. He co-edited, with Bill Blackbeard, The Smithsonian Collection of Newspaper Comics, Volume 1 (1977), and, with Michael Barrier, A Smithsonian Book of Comic-Book Comics (1981).