Byard Lancaster was born on August 6, 1942, in Philadelphia. Lancaster's family is from the South; his mother was born near Gloucester, VA, and his mother's family is registered as having started the first slave uprising in 1743, on September 13th. His father was a very good businessman and brought his mother to Philadelphia. Lancaster has three siblings; his brothers are a businessman and politician and an educator, respectively, and his sister, Dr. Mary Anne Lancaster Tyler, is a noted musicologist who studied with Donald Byrd and Nathan Davis. Byard and his sister were the musicians of the family and played in church starting in 1949. Byard played piano until age five, when his mother bought him his first alto saxophone: “I wanted to play saxophone originally, because there was this junkie across the street who sat on his porch high and played every day.”

Lancaster became an in-demand section saxophonist in school. He was restless to try new things and new approaches. Lancaster attended the Settlement Music School in 1959, the oldest music school in the country and Lola Junior High and Germantown High with pianist Kenny Barron.

Lancaster attended Shaw University, in North Carolina, for his first year. Then, he went to Berklee School of Music and was a part of the class of the Second Wave of free jazz, including pianist Dave Burrell, trumpeter Ted Daniel and drummer Bobby Kapp. He studied music education and, with Burrell, orchestrated loft parties and late-night jam sessions attended by school musicians and jazzmen like Lee Morgan and Elvin Jones.

Following Berklee, Lancaster and Burrell moved to New York and Burrell quickly started another loft space at Bowery and Bond Streets, where Elvin Jones and Archie Shepp came. Archie lived up the street and Rashid Ali lived not too far away, Amiri Baraka, Marzette Watts. Marzette wanted to learn the saxophone, and Archie wouldn't teach him, so I taught him.

In 1969, Lancaster went to Paris and played the Actuel Festival with Sunny Murray, with whom Lancaster made his first session  - Sunny Murray Quintet (ESP, 1966). He returned to Philadelphia after the festival, and returned to France, with Murray, in 1971, and stayed for six months.

Lancaster returned, in 1974, on his own and met with pianist-composer Jef Gilson, producer of Palm Records and a catalyst of the Parisian avant garde jazz scene. He recorded nine sessions for Palm, including important duets with percussionist Keino Speller (Exactement, 1975), a trio with bassist Sylvain Marc and drummer Steve McCall entitled "Us", and a tribute to James Brown.

He went back to New York, in 1978, and to Philadelphia, shortly after, returning to his roots.  “Philadelphia is a tribal city, the spiritual capital of the United States and rivals Mecca. The laws of the country and its culture were born there, and we are the root of all culture in the world because we're running the world culture, now, and the root of America is Philadelphia.”