Zuma made the announcement in Pretoria yesterday, after days of uncertainly over Cele's fate.
Cele was suspended in October last year. He is the second police chief to be fired by Zuma. His predecessor, Jackie Selebi, was sentenced to 15 years in prison for corruption in 2010.
Cele is expected to hold a press briefing to respond to his axing, following damning findings by a board of inquiry which was chaired by Judge Jake Moloi, which led to him losing his job.
Zuma said: "General Cele was appointed as national commissioner in August 2009. He was given a clear brief to take the war to the criminals.
"I would like to extend my personal gratitude to General Cele for the unquestionable commitment to his work as national commissioner.
"Leading from the front, he brought much-needed passion, energy, expertise and focus that boosted the morale of the police, leading to improved productivity and a visible reduction in crime levels."
His successor, Phiyega, is a highly regarded professional with a good reputation among her peers.
She is currently the chairwoman of the presidential review committee on state-owned enterprises and a director at Tsa Rona Investment company.
Zuma sang her praises as a capable person to deal with problems within the South African Police Service.
Zuma said he was "disappointed at the spectacle in which police officers jeopardise state security by placing it in the public domain".
He was referring to former intelligence boss Richard Mdluli, who is fighting for his reinstatement in court.
In ensuring a smooth transition, Phiyega would have to work with Police Minister Nathi Mthethwa to cement loopholes, even in the SAPS' financial systems.
Zuma suspended Cele last year after a report by Public Protector Thuli Madonsela revealed there was misconduct in the awarding of two leases for police office space.