The music mogul - who along with Simon Cowell is one of the original
judges on the US show - has decided not to return to the competition
next year because he wants to focus on his role as chairman of Epic
He told 'Access Hollywood': "I have decided that I will not return to 'The X Factor' next year. It saddens me a little bit, but only a little bit. I have a company to run that I've kind of neglected."
However, L.A. isn't planning to say goodbye to Simon, who he considers a close friend.
He added: "I love Simon. We have a great relationship. We have fun together, we talk, we laugh and do bad things, we're friends.
"This was a nice break but now it's time to get back to work."
Along with L.A's departure, it was recently claimed Britney Spears' future on the panel is also in doubt as producers are under pressure to cut costs.
A source said: "Producers were hoping to pull in between 15-20 million viewers but the show hasn't reached those expectations and in a recent episode that clashed with the World Series only 5.7 million watched.
"Those numbers add up to cost cutting and Britney will be expendable. So, considering Fox has already renewed 'The X Factor' for a third season, producers will be looking to spend less on the production of the show, including the judges' salaries.
"Britney, who was signed on a one-year deal, is the first person who will be expected to have her salary chopped. She is on a $15 million contract, which is absurd given that there's been no huge surge in the ratings."