People said the same about Gary Patterson whose name was linked to several A-list opportunities including Texas, Texas A&M, LSU, Georgia, and Notre Dame at various points, yet he stayed-put every time. Consider how often Mike Gundy's name has been linked to other jobs, yet he hasn't left either. Once someone has a good job and they are winning and appreciated, it's awfully hard to get them to leave, especially when you know their current school will match any competing offer.
It's been 44 years since USC actually pulled a head coach away from another P5 program (Jim Carlen in 1975) and Carlen was just 6-4-2 at Texas Tech the year USC hired him. None of the 6 hires that USC has made since then (Morrison, Woods, Scott, Holtz, Spurrier, and Muschamp) walked away from success in another P5 HC job to coach South Carolina instead. After all, South Carolina is not exactly an easy place to win. Only one coach, Spurrier, has ever managed to produce a top ten finish at USC and even he couldn't sustain it, falling all the way to 3-9 with a humiliating loss to the Citadel just a couple years later. Meanwhile, USC's only conference championship in program history came in 1969. That was 50 years ago, in the ACC, with a team that went just 7-4.
So ask yourself honestly if it would be a smart career decision to leave an undefeated and #12 ranked Baylor program for a rebuild project at USC. If I were in Matt Rhule's position, I wouldn't leave for anything less than an A-list job. Better to just stay put and keep building at Baylor than to try to pull-off a rebuild at a non-blueblood program in the brutal SEC.