Herald was born in Manhattan in 1939, to an Armenian born poet father Leon Srabian Herald. Through his father, Herald was exposed to live performances by blues and folk legends Lead Belly and Woody Guthrie. At a summer camp concert when he was 15, he met Pete Seeger and was further influenced by Don Larkin's bluegrass program on a New Jersey radio station while attending Manumit School, a Christian socialist boarding school in Bucks County, Pennsylvania.

In 1958, Herald formed The Greenbriar Boys, along with Bob Yellin (banjo) and Paul Prestopino (mandolin). The following year, Eric Weissberg (mandolin and fiddle), replaced Prestopino, and Weissberg was soon replaced by Ralph Rinzler (mandolin). Herald was lead guitarist and vocalist. The trio often played the Greenwich Village scene and were notable enough to be the first Northern group to win the Union Grove Fiddler's Convention competition, where Yellin also took top honors for banjo. Shortly after backing Joan Baez on her second LP, The Greenbriar Boys were signed to Vanguard Records, for whom they released three records. In 1969, Linda Ronstadt recorded Herald's "High Muddy Water." Two years previously, she had recreated his vocal of Mike Nesmith's "Different Drum," which became a hit for her band the Stone Poneys.

After the trio split up, Herald played sessions for Vanguard. In 1972, he recorded an eponymous solo album for Paramount Records, then went "electric country bluegrass" on a 1978 disc featuring the John Herald Band (a group he'd formed while living in Philadelphia in 1976). Reviewing the solo album in Christgau's Record Guide: Rock Albums of the Seventies (1981), Robert Christgau wrote: "This casually joyous solo debut by the former Greenbriar Boy gives in at times to such folky vices as mere flash, mere lyricism, and mere whimsy. But 'Fire Song,' a casually joyous ditty about how his house burned down, and 'Brother Sam,' unpresumptuous compassion for a returned Vietnam vet, should inspire Paul Simon to work real hard on the follow-up. And his high notes should inspire Art Garfunkel to go back to architecture school.