He was going to turn 78 on July 2.
Mjo, who also played for the African Springboks, was a top actor in Eastern Cape where he featured in various stage dramas and was influential in the formation of The East London Harmony Set musical choir.
He played for Swallows, one of the oldest rugby clubs in South Africa. The former political activist also represented Border province.
Mjo was an outstanding scrumhalf of his era and was selected in an all-time African XV for the book The Badge: A century of The Springbok Emblem. He also championed non-racial rugby structures and tournaments.
He is listed as one of the 20 greatest players of colour in South Africa as well as part of the Dream Team of African rugby in the book.
South African Rugby Union president Oregan Hoskins said they were saddened to learn about the death of Mjo, who was also the father of top sports administrator Onke Mjo, who is attached to the ministry of sport and recreation. Mjo will be buried in Port Elizabeth on June 29.