clock menu more-arrow no yes mobile

Filed under:

Florida ranked No. 1 school for student athletes by STACK

It's cool, but not holistic. You can make numbers say whatever you like!

By Andy Hutchins
Kevin Jairaj-USA TODAY Sports

Florida is ranked the No. 1 school for athletes by the latest issue of STACK magazine, which would be a truly prestigious honor — if I had had any idea what STACK prior to this week, or thought the methodology used to arrive at this ranking were in any way rigorous.

I mentioned the ranking, posted to the Internet on Monday (note: obnoxious and unavoidable auto-play video ahoy), in Wednesday's Chomping at Bits, with "yeah, sure" as my entire editorial comment.

Now that Florida is using the ranking to market itself, it's time to expound.

STACK — STACK, the company, was started in 2005 as "a company dedicated to empowering and inspiring athletes through sports performance information and services," and repeatedly touts its almost instant presence in high schools, which no doubt helped make claims like "circulation of 300,000 copies was acquired virtually overnight" true — describes the methodology behind its rankings as such:

To make life easier for student-athletes, STACK partnered with Niche, one of the world’s most popular college ranking services. After examining data and surveying more than 75,000 students, Niche compiled a list of the best schools for student-athletes.

The rankings below reflect averages of Niche's academic and athletic grades for each school. Their academic grade is based on factors such as acceptance rate, SAT/ACT scores, graduation rate and research expenditures. Their athletic grade takes into consideration student opinions, NCAA championships, sports attendance and percentage of students playing varsity sports.

I clicked my way to Niche's methodologies for best academics rankings and best athletics rankings. The methodology for determining academic rankings is suprisingly robust, incorporating 14 distinct variables, a Bayesian method to account for small sample sizes, and formulae that produce standardized z-scores, which are then combined into a more holistic z-score. (On the athletics side, the same process is done with fewer variables.) For the most part, it's solid surveying.

The 2015 Niche rankings slot Florida in at No. 88 in Best Academics, which ranks second behind Miami (No. 84 nationally) among in-state schools, and within the top 20 public schools in the country. I'm not surprised at that ranking, given how important selectivity is to Niche's rankings — acceptance rate is the only factor that is weighted more than 10 percent, of the 14 used, and Florida's hovers around 40 percent, which is comparable with most other flagship schools in big states. Texas is around that mark, too, and Michigan is slightly more selective, while North Carolina, UCLA, and Virginia, the other three public schools that rank ahead of No. 30 Florida in Niche's overall rankings, are all significantly more picky.

But that means, logically, that Florida's Best Athletics score would have to be absurdly high for the school to somehow top a ranking that averages it with Best Academics — so it's no surprise that Florida is No. 1 on that list.

What is at least a little surprising is how that Best Athletics methodology works. The single biggest factor in determining a Best Athletics score is student survey responses — and here's how those are described.

Student opinions about the quality of the athletics at the college they currently or recently attend(ed). Includes 204,756 opinions from 74,957 unique students. Minimum 10 unique students required at each college.

Warning: We're about to do math. And I had to take Stats 2 twice.

As a whole, those numbers are really good — 75,000 unique students is a fantastic sample for evaluating the entire population of students in American colleges and universities, because the margin of error for a sample of even a few hundreds of people, even for a massive population, is relatively small. (Gallup's daily poll of President Obama's approval rating, for instance, has a sample size of about 1,500 respondents, which produces a margin of error of plus or minus three percent — for a population of hundreds of millions of American adults.)

But it's the provision for even a tiny sample size — the "minimum 10 unique students required at each college" bit — that troubles me. The Best Athletics rankings contain 274 schools, which indicates — to me — at least 10 responses from students at those schools. If those 274 schools produced all of the 74,957 unique students surveyed, the average population surveyed — about 275 students per school — is pretty accurately representative of even the largest student body population in America. A 275-person sample produces a plus or minus 5.89 percent margin of error for a population of 50,000 — which isn't fantastic, but further calculation of confidence intervals would suggest relatively good confidence in results for a population that large.

My hunch, however, is that the nearly 75,000 students instead came from the 1,410 schools that got a grade in the Best Athletics ranking, or from the 2,245 four-year schools that Niche includes in its rankings. And that would probably mean that some of the survey response scores are being extrapolated from very small sample sizes — a 10-person sample has a huge margin of error of nearly plus or minus 31 percent for a population of even just 1,000. And even smaller sample sizes are bad in sampling: Even a 100-person sample for a population of 1,000 produces a substantial margin of error of plus or minus 9.3 percent.

That throws the accuracy and reliability of the sampling into some doubt.

Fortunately, Niche does give us some more data, listing the number of respondents at each school: At Florida, that number is 713. Using a rough estimate of 49,000 UF students, slightly less than the Fall 2013 population, that produces a very good margin of error of plus or minus 3.64 percent. But Florida is the only school in the top 10 that had more than 520 respondents; it's more likely that most sample sizes are smaller than that one.

I trust Niche, or any half-competing polling or survey company, to do its best with the data it collects (and Bayesian analysis helps), but the best solution for flawed data is just getting better data.

It's the rest of the variables for the Best Athletics grade, though, that give me more pause. The most valuable among them, with a 15 percent weight, is "NCAA Championship Score", which is defined as:

Number of NCAA Championships won since 2000 across Division I, II, and III. Eight sports are measured, including football, men's and women's basketball, baseball, softball, men's and women's soccer, and hockey. Championships were weighted by division and sport, with Division I getting a 3x multiplier and Division II getting a 2x multiplier. Football and men's basketball received a 3x multiplier, and baseball and hockey received a 2x multiplier. FCS Football was also included but did not receive a multiplier.

Obviously, that weights football and men's basketball heavily, and Florida does fine in that regard: Florida's NCAA Championship Score is 39, with 18 points from football, 18 from men's basketball, and three from softball. But Connecticut's is 51, with 27 from men's basketball — and 24 from eight other national titles in women's basketball and men's soccer. Florida's won 13 other national titles since 2000, but none count in this methodology, and the Gators don't even field teams in two of the sports measured.

I get not counting every single national title — but Stanford, which has a National Championships Score of 3 thanks to a 2011 title in women's soccer, has an astounding 29 national championships since 2000 that don't get tallied in. And that just seems absurd, especially given that this Niche score is being used in a ranking of "the best schools for student-athletes" — not just football and men's basketball players.

And I think the logic behind culling the number of sports measured really impacts Niche's rankings for female athletes, which slate Florida in at a relatively low No. 11, behind Grand Valley State and Messiah College ... while counting just women's basketball, soccer, and softball, and thus not crediting Florida with any of the seven national championships it has won in other women's sports since 2000.

That's not the only bit of trouble I have with weighting, either. Average home football attendance and average home men's basketball attendance count for 14 percent and 11 percent of the formula, and conference averages for those two sports are factored in, too. Those conference averages help SEC schools substantially, as the SEC's the only conference with an average football attendance better than 75,000, and one of just four with an average men's basketball attendance better than 10,000 — and they further bias the rankings toward schools with big football programs.

There's also an NCAA Championship Score for Conference, which is tallied across the same eight sports as NCAA Championship Score. I'd be inclined to believe those numbers only run through the 2012-13 season, because the SEC's is listed at 19 — and I can only get that number when counting of SEC national titles by not counting Vanderbilt's 2014 national title in baseball or Florida's 2014 national title in softball. (And Texas A&M's women's basketball title won in the Big 12 counts for the Big 12.)

But the ACC, thanks mostly to 15 titles won in men's and women's soccer since 2000, tallies an NCAA Championship Score for Conference of 25, best in the survey — and I can only get that number from the ACC national titles list by counting Virginia's 2014 men's soccer title, and Florida State's 2014 women's soccer title.

That's a relatively minor part of the formula, to be fair. Football attendance and student responses — which I will hazard to guess are generally better at football powerhouses, and after good seasons — are much more important. So it's stunning, really, that seven SEC schools — No. 1 Florida, No. 2 Alabama, No. 5 LSU, No. 8 South Carolina, No. 9 Kentucky, No. 11 Tennessee, and No. 15 Auburn — finish in the top 15 of Best Athletics. The SEC helps its schools, and significantly, in these rankings.

But Florida is best positioned to benefit: It also has more points from national championships than any other SEC school (Alabama is second with 30), has great survey scores, and doesn't lag in any category of these rankings.

And so Florida, per these rankings, is better at sports than MIT is at academics.

From the section called The Outcome from the Best Athletics methodology:

The top ranked college was University of Florida, which ranked very highly in most factors analyzed and had a final score that was more than six standard deviations above the mean college, an exceptionally high number. The next three colleges—University of Alabama, University of North Carolina at Chapel Hill, and University of Connecticut, also had exceptionally high scores, all scoring more than five standard deviations above the mean college. All four colleges scored very highly in NCAA championships won, with each winning several championships across multiple sports since 2000.

And from The Outcome of the Best Academics methodology:

The top ranked college was Massachusetts Institute of Technology, which ranked highest in nearly every factor analyzed and had a final score that was more than five standard deviations above the mean college, an exceptionally high number.

How does that translate to Florida being on top for student athletes, despite being No. 88 in Best Academics? The rankings are a pure 50-50 average of the two combined z-scores — which means Florida's extraordinarily high score in Best Athletics is buoying it significantly, while MIT, which does not even qualify for a Best Athletics Ranking, pops up at No. 13 in this list.

That also means Florida's student survey responses are more than twice as important to the calculation of these rankings as MIT's acceptance rate.

Now, you can spin that as either a good or a bad thing: The acceptance rate of a school would seem to matter very little to the average "student-athlete" reading STACK in a high school classroom, while student satisfaction with a school's athletics program would be significant. But student satisfaction with an athletics program as the most important variable in an analysis like this — when most student athletes are somewhat inured from student satisfaction, and never quite get around to asking the coaches who recruit them how their schools scored on Niche's surveys — is a curious choice.

And I can spin Florida being No. 1 in this ranking, despite not having all its trophies counted and a disadvantage it can't really help on the academic side, too. Florida didn't even need to have all those championships factored in to be the only school more than six standard deviations better than the average college ... but the Gators ranking this high is partly thanks to a methodology I don't totally agree with, one that gives them a bump based on conference strength, and relies on survey data more than it perhaps should.

If I had my druthers — and I certainly hope I will, someday, if only to finally learn what druthers are — I'd find a happy medium between streamlining (you don't need 10 variables, I think) and expanding (to measure, you know, every sport) the methodologies used in rankings like this, and I'd absolutely weight them differently. But I'd also need the time and money to conduct surveys like the ones Niche has — and, to be clear, that's valuable data worth including in some way. I can quibble with how it's used, and how other data is used, but the data's worth something, and costs plenty to acquire.

Of course, STACK knows that, too: That's why it used Niche's rankings with no alteration, repackaged them, and publicized it. And now I know what STACK is!

I think that was really the point of this whole exercise.