A quick, unscientific look at how well the recruiting sites predict future success

[ChrisBrown]

Someone (not me ) should see where they rank in average star ranking. That's all that matters really, and doesn't even take into account holes needed to fill.

Okay, so here is a quick, unscientific look at how well the recruiting sites predict future success. There are tons of different ways to look at this, and my method is woefully inadequate, but hopefully we’ll get a general idea. The records at Scout and Rivals go back to 2002. I'll try to take a look at ESPN soon, but their damn 100-point system makes comparisons difficult.

So, basically I just looked at the first-round draft picks in the NFL going back to 2005 and saw how each site rated them. Here are the results:

2005 Draft – There were only seven players from this draft in the two sites’ databases.

Rivals – 4 four-stars, 1 three-star, 2 two-stars
Scout – 1 four star, 4 three stars, two unrated players

The two sites disagreed about every one of these players:

Alex Smith (02) – Rivals 2*, Scout NR
Adam Jones (02) Rivals 2*, Scout NR
Troy Williamson (02) – Rivals 4*, Scout 3*
Mike Williams (02) – Rivals 4*, Scout 3*
Shawne Merriman (02) – Rivals 4*, Scout 3*
Fabian Washington (02) - Rivals 4*, Scout 3*
Aaron Rodgers (03) – Rivals 3*, Scout 4*

2006 Draft – Eight of the players taken in the first round of the 2006 Draft were 2001 recruits, and thus, there was no official ranking for them at any of the sites. Of the 24 remaining players drafted that year:

Rivals – 5 five-stars, 13 four-stars, 4 three-stars, 2 two-stars
Scout – 6 five-stars, 8 four-stars, 3 three-stars, 2 two-stars, 5 unrated players

The two sites agreed on 12 players. The disagreements were as follows:

Reggie Bush (03) – Rivals 5*, Scout 4*
Donte Whitner (03) – Rivals 4*, Scout 5*
Broderick Bunkley (02) – Rivals 4*, Scout NR
Antonio Cromartie (03) – Rivals 4*, Scout 5*
Laurence Maroney (03) – Rivals 4*, Scout 2*
Manny Lawson (02) – Rivals 2*, Scout NR
Davin Joseph (02) – Rivals 4*, Scout 3*
Jonathan Joseph – (04) Rivals 3*, Scout 4*
Santonio Holmes (02) – Rivals 3*, Scout 4*
John McCargo (02) – Rivals 2*, Scout NR
De’Angelo Williams (02) – Rivals 4*, Scout NR
Nick Mangold (02) – Rivals 4*, Scout NR

2007 Draft - The first round of the 2007 draft contained:

Rivals –5 five-stars, 17 four-stars, 7 three-stars, 3 two-stars
Scout – 6 five-stars, 14 four-stars, 3 three-stars, 5 two-stars, 4 unrated players

The two sites agreed on 20 players. The disagreements were as follows:

JaMarcus Russell (03) – Rivals 4*, Scout 5*
Calvin Johnson (04) – Rivals 4*, Scout 5*
Joe Thomas (03) – Rivals 4*, Scout NR
Gaines Adams (02) – Rivals 3*, Scout 2*
Patrick Willis (03) – Rivals 3*, Scout 2*
Adam Carriker (02) – Rivals 3*, Scout NR
Justin Harrell (02) – Rivals 4*, Scout 3*
Jarvis Moss (03) – Rivals 5*, Scout 4*
Michael Griffin (03) – Rivals 4*, Scout 3*
Brandon Meriweather (02) – Rivals 3*, Scout 4*
Anthony Spencer (02) – Rivals 3*, Scout NR
Ben Grubbs (02) – Rivals 3*, Scout NR

2008 Draft - The first round of the 2008 contained:

Rivals – 5 five-stars, 11 four-stars, 6 three-stars, 7 two-stars, 2 unrated players
Scout – 7 five-stars, 7 four-stars, 6 three-stars, 7 two-stars, 4 unrated players

The two sites agreed on 18 players. The disagreements were as follows:

Chris Long (04) – Rivals 4*, Scout 3*
Darren McFadden (05) – Rivals 5*, Scout 4*
Glenn Dorsey (04) – Rivals 4*, Scout 5*
Vernon Gholston (04) – Rivals 4*, Scout 3*
Jerod Mayo (04) – Rivals 4*, Scout 3*
Brandon Albert (04) – Rivals 2*, Scout NR
Gosder Cherilus (03) – Rivals 3*, Scout 4*
Felix Jones (05) – Rivals 4*, Scout 3*
Rashard Mendenhall (05) – Rivals 4*, Scout 5*
Chris Johnson (04) – Rivals 2*, Scout NR*
Antoine Cason (04) – Rivals 3*, Scout 2*
Lawrence Jackson (03) Rivals 4*, Scout 5*
Kentwan Balmer (04) – Rivals 3*, Scout 2*

2009 Draft - The first round of the 2009 contained:

Rivals – 6 five-stars, 12 four-stars, 6 three-stars, 7 two-stars, 1 unrated player
Scout – 8 five-stars, 5 four-stars, 7 three-stars, 11 two-stars, 1 unrated player

The two sites agreed on 20 players. The disagreements were as follows:

Tyson Jackson (04) – Rivals 3*, Scout 2*
Darrius Heyward-Bey (05) – Rivals 4*, Scout 3*
Michael Crabtree (06) – Rivals 4*, Scout 2*
Brian Orakpo (04) – Rivals 4*, Scout 3*
Brian Cushing (05) – Rivals 4*, Scout 5*
Jeremy Maclin (06) – Rivals 4*, Scout 3*
Brandon Pettigrew (04) – Rivals 2*, Scout 3*
Michael Oher (05) – Rivals 4*, Scout 5*
Peria Jerry (04) – Rivals 3*, Scout 2*
Donald Brown (05) – Rivals 3*, Scout 2*
Kenny Britt (06) – Rivals 4*, Scout 3*
Evander Hood (05) – Rivals 3*, Scout 2*

2010 Draft - The first round of the 2010 contained:

Rivals – 5 five-stars, 14 four-stars, 7 three-stars, 6 two-stars
Scout – 6 five-stars, 14 four-stars, 5 three-stars, 6 two-stars, 1 unrated player

The two sites agreed on 26 players. The disagreements were as follows:

Sam Bradford (06) – Rivals 3*, Scout 4*
Russell Okung (06) – Rivals 3*, Scout 4*
Anthony Davis (07) – Rivals 4*, Scout 5*
Ryan Mathews (07) – Rivals 3*, Scout 2*
Jason Pierre-Paul (07) – Rivals 2*, Scout NR
Kareem Jackson (07) – Rivals 4*, Scout 3*

So, that was a lot of names, and numbers, and *s that don't really tell us much. I did one more thing, regarding 5-star players:

2002

Rivals - 38 five-star players - 3 taken in the first round
Scout - 92 five-star players (35 JuCo) - 3 taken in the first round

2003

Rivals - 25 five-star players - 5 taken in the first round
Scout - 52 five-star players (17 JuCo) - 6 taken in the first round

2004

Rivals - 25 five-star players - 4 taken in the first round
Scout - 40 five-star players (9 JuCo) - 6 taken in the first round

2005

Rivals - 28 five-star players - 5 taken in the first round
Scout - 61 five-star players (9 JuCo) - 8 taken in the first round

2006 Incomplete due to 5th year guys coming out this year

Rivals - 28 five-star players - 8 taken in the first round
Scout - 62 five-star players (7 JuCo) - 8 taken in the first round

Overall, if you were ranked a 5-star Rivals, you had a 17% chance of being a first-round pick. If you were rated a 5-star by Scout, you had a 10% chance of being a first round pick. If we throw out that wacky first year, where both sites were apparently still judging tractor pulls and dog shows, Rivals' hit rate moves up to 21%, while Scout's rises to 13%.

Conclusions

Well, we haven't broken a whole lot of ground here. People say that Rivals is "better" for football, and that appears to be the case based on this admittedly anecdotal data. Scout is much more willing to give out 5-star ratings, but they are also much more likely to not know who the eff a kid is (17 NRs, compared to Rivals' 3 NRs). Early on, Scout was also more likely to completely whiff on some fairly prominent kids, but they seem to have corrected that. The sites appear to be getting closer and closer to each other in terms of rankings, but Rivals is better at projecting 5-star players for success, so it stands to reason that they are better at projecting the rest of the kids too.

[pGekko]

Rivals owns Scout in the "disagreements" especially the 4 and 3 star ones, at least concerning 1st round draft picks.

[tommyz]

thanks for the work CB..I made it its own thread as its deserving

[joeking1978]

Great work up. You should forward it to forward it to a few more sites. See if you can generate some buzz and get it picked by yahoo or something.

[Preacher_Boy]

Great work up. You should forward it to forward it to a few more sites. See if you can generate some buzz and get it picked by yahoo or something.

Yahoo has already done something similar before, but it was based on # of all americans and it was only Rivals based because yahoo is associated with Rivals.

[ChrisBrown]

Thanks guys. It's not nearly as complete as I'd like it to be, but without access to the raw data at each site, doing stuff like this becomes sort of an unwieldy flustercluck. I am going to try to convert the ESPN scores into star rankings though.

[RickFromFowler]

This is a great, great post. Thank you for sharing Chris.

Something I would be interested in seeing, and I'm not asking for this, just saying it would be interesting.

Each pick in order as drafted, their two rival rankings, then the bust review factor..

Like

1) Player Selected, QB, R-5, S-4. All-Pro (NFL Starter, NFL Bench, Bust)
2) So On..

It would be interesting to see if the draft bust quotient was higher for guys not rated highly by scout, but then still drafted.

[phillyfan]

pRick seal of approval

[ChrisBrown]

Overall, if you were ranked a 5-star Rivals, you had a 17% chance of being a first-round pick. If you were rated a 5-star by Scout, you had a 10% chance of being a first round pick. If we throw out that wacky first year, where both sites were apparently still judging tractor pulls and dog shows, Rivals' hit rate moves up to 21%, while Scout's rises to 13%.

Update!

So in the process of trying to find a way to convert ESPN's scores into star ratings, I discovered that Rivals hides a handful of 5-star players every year. I was just going off their Top 100 list, but when you search exclusively for 5-star players, you get a bunch of JuCo 5-stars that apparently don't qualify (for the Rivals list). The numbers don't change too much, but here they are:

2002

Rivals - 60 five-star players (24 JuCo) - 3 taken in the first round
Scout - 92 five-star players (35 JuCo) - 3 taken in the first round

2003

Rivals - 31 five-star players (6 JuCo) - 5 taken in the first round
Scout - 52 five-star players (17 JuCo) - 6 taken in the first round

2004

Rivals - 34 five-star players (9 JuCo) - 4 taken in the first round
Scout - 40 five-star players (9 JuCo) - 6 taken in the first round

2005

Rivals - 36 five-star players (8 JuCo) - 5 taken in the first round
Scout - 61 five-star players (9 JuCo) - 8 taken in the first round

2006 Incomplete due to 5th year guys coming out this year

Rivals - 38 five-star players (10 JuCo) - 8 taken in the first round
Scout - 62 five-star players (7 JuCo) - 8 taken in the first round

Overall, if you were ranked a 5-star Rivals, you had a 12.5% chance of being a first-round pick. If you were rated a 5-star by Scout, you had a 10% chance of being a first round pick. If we throw out that wacky first year, where both sites were apparently still judging tractor pulls and dog shows, Rivals' hit rate moves up to 16%, while Scout's rises to 13%. I guess the only other thing to update is that none of these JuCo kids ended up making the NFL, so I guess I understand why Rivals leaves them off the Top 100 list.

On to the ESPN stuff in a minute.

[ChrisBrown]

So, in attempt to compare the ESPN scores to the Scouts and Rivals star rankings, I figured I should first find averages for Scouts and Rivals. So, here they are, going back to 2005:

2005
Rivals – 36 five-stars, 285 four-stars, 811 three-stars
Scout – 61 five-stars, 295 four-stars, 660 three-stars

2006
Rivals – 38 five-stars, 332 four-stars, 871 three-stars
Scout – 62 five-stars, 303 four-stars, 770 three-stars

2007
Rivals – 37 five-stars, 343 four-stars, 989 three-stars
Scout – 59 five-stars, 397 four-stars, 850 three-stars

2008
Rivals – 37 five-stars, 368 four-stars, 993 three-stars
Scout – 59 five-stars, 348 four-stars, 973 three-stars

2009
Rivals – 35 five-stars, 401 four-stars, 1295 three-stars
Scout – 57 five-stars, 294 four-stars, 889 three-stars

2010
Rivals – 27 five-stars, 395 four-stars, 1645 three-stars
Scout – 52 five-stars, 293 four-stars, 1015 three-stars
ESPN - 12 five-stars, 286 four-stars, 1097 three-stars

2011
Rivals – 26 five-stars, 306 four-stars, 1591 three-stars
Scout – 52 five-stars, 274 four-stars, 1210 three-stars
ESPN - 14 five-stars, 280 four-stars, 1168 three-stars

Averages
Rivals – 34 five-stars, 347 four-stars, 1171 three-stars
Scout – 57 five-stars, 314 four-stars, 910 three-stars
ESPN - 13 five-stars, 283 four-stars, 1133 three-stars

Whoopee! ESPN began ranking players in 2006, but they didn't start using a star system until 2010. ESPN is kind of prohibitive (and confusing) in their rankings. In the two years they've been assigning numbers and stars, any player ranked 85 or above is a 5-star, and any player ranked below 75 is a 2-star. I'm not sure how they determine the line between 3- and 4-stars though, because there are a bunch of guys ranked 79, and it seems like half of them are 4-stars, and half are 3-stars. Stupid ESPN. I could use their numbers, but I already did some pointless math below.

Here are the ESPN scores that correspond to the averages from Rivals and Scout for 2006-2009:

2006
R5- 86, R4- 77, R3- 40
S5- 83, S4- 77, S3- 67

2007
R5- 82, R4- 77, R3- 68
S5- 81, S4- 77, S3- 70

2008
R5- 83, R4- 78, R3- 70
S5- 82, S4- 78, S3- 73

2009
R5- 83, R4- 78, R3- 74
S5- 83, S4- 79, S3- 75

Averages
R5- 84, R4- 78, R3- 71
S5- 82, S4- 78, S3- 71

In my system, ESPN players ranked 83 or above = 5-stars, players ranked 78-82 = 4-stars, and players ranked 71-77 = 3-stars.

The only recruiting class where we can make use of this data is 2006, so I guess I should do it:

2006 Still incomplete due to 5th year guys coming out this year

Rivals - 38 five-star players (10 JuCo) - 8 taken in the first round
Scout - 62 five-star players (7 JuCo) - 8 taken in the first round
My ESPN - 58 five-star players - 8 taken in the first round
ESPN85 - 42 five-star players - 7 taken in the first round

So yeah, that's a ****load of numbers and averages for very little payoff, but maybe in a few years it'll actually be useful.

[Blindly Painted Bastard]

This is a great idea. If you want to take it further I'd be willing to help - I have some ideas for statistical analysis. I think it wouldn't take too long to compile the data. This could easily be made scientific.