Sure - if getting in was only based on grades and test scores. And there are so many other students that also wouldn’t get in if not for other factors outside of grades and test scores. Sometimes schools want to let more out of state kids in - or have majors where you’re going to get grades and test scores for incoming freshman that may be lower than other majors (most likely the engineering kids are doing better at math than the English majors.
Schools have decided that athletics is important to their missions - Just like an English department is important. And they allocate a certain number of spots to athletics, the same way they do for the English department. The athletes aren’t “taking” spots at a school from someone else - they’re filling what was allocated to them.
But really, let’s take USC, they let in 6,150 Freshman a year. If someone didn’t get in, they don’t have it. If your rank for applicants is 6,151 or higher, “deserving” is a major stretch. Even if you take ever athlete out, they’re still sitting in the very high 5,000’s. There are 5,300 colleges in the US, there’s plenty of spots for anyone.