My frustration with "scouting"

A few years ago, Baseball Prospectus summed up the supposed dichotomy between scouting and objective statistical analysis with the phrase "beer and tacos." If some asks you if you would like beer or tacos, the correct answer is not one or the other, but both. They serve completely different purposes. Similarly, people often try to pit scouting and statistical analysis against each other as if they were opposite ends of the same spectrum. In reality, they are two totally different tools with the same aim: better decision making through better information.

Let me be perfectly clear about this: good scouting is absolutely essential to a well run baseball team. Scouting provides data that raw statistics will struggle to uncover. Scouts are very important and I have no quarrel with them.

Unfortunately, a disturbing trend has developed in the mainstream presentation of scouting data. Too often, analysts that are supposed to be providing the public with a scout's view of players have instead become nothing more than poor statistical analysts, justifying their use of small sample sizes with the vague notion that they are scouting. This type of analysis is not only completely useless, adding nothing to the discussion, but also damaging to scouting as a whole. Scouting should not become a tool for giving undue weight to a small sample of performances. It is supposed to supersede the small sample by providing data that cannot be gleaned from statistics alone.

I suppose an example or two is in order. Keven Goldstein is a writer for Baseball Prospectus. In fact, he's supposed to be their scouting guru. He was brought to BPro with the hopes of expanding their coverage beyond just statistical analysis. I like Kevin's columns and I read almost everything he writes. He writes a column every Monday that offers a small blurb on ten different prospects of note. Here is an example from his latest:

RHP Jake Arrieta, Phoenix Desert Dogs (Orioles)

Arrieta is becoming an offseason Ten Pack regular, as the Orioles keep pitching him an inning at a time, and he keeps putting up zeroes. At this point it’s gone from “nice start” to “downright impressive,” as Arrieta had his best outing yet on Saturday, striking out all three batters he faced. So far, the Orioles fifth-round pick who got first-round money has put together 12 scoreless innings over 10 appearances, while allowing just six hits and striking out 13. It’s a little too early to call him a steal, and his disappointing final college season is still in the back of people’s minds, but his timetable is on the verge of getting accelerated.

It's nice to know what Arrieta is up to, but if you're looking for any useful information here, you should be sorely disappointed. There isn't any. There is not one shred of scouting data in this blurb. The only information presented to the reader is some statistical data from a sample size so absurdly small that it is totally, utterly, meaningless. Arrieta might be a good prospect, but there is 100% no reason from this paragraph to suppose that he is. If Goldstein knows what makes Arrieta great, he hasn't included that information here, and it defeats the purpose of his presence of the BPro staff.

A small sample size is a small sample size. Unless you present compelling evidence above and beyond the data itself that it should be given significance, you cannot glean any useful information from a small sample. Let's look at another example:

RHP Daniel Bard, Honolulu Sharks (Red Sox)

Friday’s Boston prospect rankings, like any prospect list, generated a lot of email. Most of it concerned guys who didn’t make it, like Brandon Moss or Craig Hansen, but nobody asked about Daniel Bard. Twelve months ago, that wouldn’t have been the case, because last year at this time, Bard was a highly regarded first-round pick who could touch 100 mph, although he had some issues when it came to command and secondary stuff. This year, the wheels fell off. Beginning the year at High-A and then spending the majority of the year at Low-A after a demotion, Bard finished the year with a 7.08 ERA and 78 walks in 75 innings. Using the Hawaii Winter League as an opportunity to find the magic once again, the good news is that Bard has a 0.69 ERA in 13 innings while allowing just seven hits. The bad news is that he’s walked 11. It doesn’t matter how hard you throw if you have no idea where it is going.

This paragraph is only slightly better. We hear about why Bard was highly regarded, but then Goldstein uses another small sample size to explain that Bard is in serious trouble as a prospect. There's only one problem: he gives not one single ounce of scouting evidence to suggest that Bard is struggling. Again, it doesn't matter if you say you are a scout, a small sample size is a small sample size and it adds absolutely nothing to the discussion by definition unless it can be supported by extrastatistical evidence. That is what scouting is supposed to do. Goldstein and his scouting sources may know why Bard is struggling, but until that information is presented, Goldstein's paragraph amounts to little more than saying, "Bard is in trouble. Trust me, I know because I talk to scouts." How useful is that?

Scouting analysis can be done. Let's rewrite the Bard paragraph with some fictitious, though plausible, scouting analysis. The italicized part indicates my rewrite.

RHP Daniel Bard, Honolulu Sharks (Red Sox)

Friday’s Boston prospect rankings, like any prospect list, generated a lot of email. Most of it concerned guys who didn’t make it, like Brandon Moss or Craig Hansen, but nobody asked about Daniel Bard. Twelve months ago, that wouldn’t have been the case, because last year at this time, Bard was a highly regarded first-round pick who could touch 100 mph, although he had some issues when it came to command and secondary stuff. This year, the wheels fell off. Beginning the year at High-A and then spending the majority of the year at Low-A after a demotion, Bard's mechanics deteriorated. He began shortening his stride, causing a drop in his velocity. To compensate for this, he began to "muscle up" when throwing ball, exerting greater effort with his upper body. His left shoulder was no longer positioned properly when the ball was released, causing him to lose any semblance of command or control. It doesn’t matter how hard you throw if you have no idea where it is going.

Now, I can't really write like a scout, but that is essentially what scouting information should look like. We aren't relying on any statistics at all. There is no small sample size.

Scouting analysis can work, but it must be disciplined. The moment it deteriorates into a parade of small sample sizes, it loses all of its value. I sincerely hope that Mr. Goldstein recognizes this so that we can reap the full benefit of his scouting connections.

There is hope. The one absolutely essential scouting columnist on the Internet is The Hardball Times' Carlos Gomez. Gomez provides a true scout's view of players, including some excellent takes on Joba Chamberlain versus Phil Hughes and Ian Kennedy versus Clay Buchholz. Gomez's analysis breaks down each player physically, analyzing how they do what they do and how what they do leads to results. Assuming that Mr. Gomez just isn't talking out of his ass, every writer who wants to write about scouting should aspire to his level of work. If that happens, we'll finally start reaping the benefits of beer and tacos.

**EDIT** Fixed minor spelling mistake.

I hope Cashman doesn't think this way

From the mouth of Hank Steinbrenner:

“This game is 70 percent pitching, and even more in the postseason.”

I'm pretty sure that this is not just false, but provably false. In fact, it's so false that Hank is probably off by a factor of two.

Here's how it breaks down. To win a baseball game, one must outscore one's opponent. It doesn't matter what the score is, so winning 12-11 is as good as winning 1-0. From this, we will assume that a run scored is as valuable as a run prevented.*

From this, it follows that run prevention as a whole is as important as run acquisition (doesn't that sound cool) as a whole. Run prevention is divided neatly into two parts: action before a batting event (pitching) and action after a batting event (fielding). Since run prevention and run acquisition are equal with respect to winning ballgames, if we assume that fielding has any importance at all, then pitching must be less than 50% of "the game."

So that's my reasoning. If we assume that preventing a run increases your chances of winning by roughly the same amount as scoring a run, then this logic is nigh infallible. I really hope Hank doesn't have as much influence as his Dad did.**

* Due to the fact that you cannot fall below zero runs scored, this will not be quite true, but it is a safe assumption that the difference is negligible for this exercise. For example, if you have a game that would otherwise be tied 4-4, you can win by either preventing a run or by scoring another run. The events have equal value. Perhaps I shall test this assumption in the near future.

** I will note in Hank's defense that if he means that pitching is more valuable than the other aspects of the game, then he could be correct, if good/great pitching were appreciably more scarce than hitting, base running, and fielding. This is unlikely to be the case, and it's hard to see that this is his meaning from his choice of words.

Wrapping up the McCarver discussion

I wanted to spend another post on the Tim McCarver lead off walk/lead off home run discussion in order to wrap it up.

Let's do this bullet point style!

• I was surprised at how close the two outcomes were with respect to multi-run innings, about a 2-4% difference historically. One of the great things about studying baseball is that you are surprised by the results and you learn things. That's part of why I write this blog. If I wasn't leaning anything, I would just be up here pontificating. In this case, I found that I was guilty of attributing more import to the home run than it deserved. Obviously, the home run is far more valuable because it is one guaranteed run, but with respect to multi-run innings its value is marginalized relative to the walk. I did not recognize this, and thus my initial approach to the problem was incorrect.
  
• We can definitely conclude that Tim McCarver is totally wrong. Whether we examine the problem from the perspective of history or probability, both correct ways of examining the problem, the weight of evidence is greatly against him. Even thought the difference in probabilities is seemingly small, it's over such a large set of samples that it takes on much significance. It's nearly impossible to conceive of a scenario in which the walk is more valuable than the home run, which was McCarver's basic assertion.
• We reaffirmed a basic baseball truth: outs are your most precious commodity. The difference between the two events with respect to multi-run innings has nothing to do with their relative value to each other from a run expectancy point of view. The home run is more valuable with respect to multi-run innings not because it scores a run, but rather because it precludes the possibility of an out.
• Finally, we provided a tad more evidence for the Markov property in baseball. The conjecture about the walk being more likely to lead to big innings is largely based on the assumption that indicates more about the game state than it actually does. Baseball is largely dependent only on the current state of the game, not on how each state was reached.

Screw Peter Abraham

I take back every nice thing I ever said about him and his blog. The following is an ignorant, misinformed, bullshit, throw-away blurb of which he ought to be completely and utterly ashamed (emphasis mine).

3. Think about how much money it could cost the Yankees to retain Jorge Posada, Mariano Rivera and Alex Rodriguez. Over the lifetime of their contracts, you could be talking an investment of at least $250 million. That’s a lot for a 36-year-old catcher, a 38-year-old pitcher and a guy who never played in the World Series.

That's what A-Rod is to you, Pete? Go to hell.

The McCarver Strikes Back

Today, during the Indians-Red Sox game, as I was watching live, McCarver again reiterated the not-so-surprising not-so-revelation that your chances of a multi-run inning are higher after a lead off home run than a lead off walk. Why does this surprise Tim McCarver?

Here's how it works. In the game state with no one on base and no one out (hereafter abbreviated 0-000), you have a probability of scoring exactly zero runs from then on (P0), a probability of scoring exactly one run from then on (P1), and the probability of scoring more than one run from then on (P2+ = 1 - P0 - P1). Conversely, the probability of no multi-run inning is P0 + P1 = 1 - P2+.

After a lead off home run, you return to the 0-000 game state. Only now, in order for there to be a multi-run inning, you only need to score one more run. Therefore, the probability of no multi-run inning is now just P0. The probability of a multi-run inning has become P1 + P2+.

After a lead off walk, you enter the game state 0-100 (man on first, no one out). You still need to score two more runs. As before, you have a probability of scoring zero runs P0', one run P1', and two or more runs P2+'.

In order for the lead off walk to be more valuable, P0 would have to exceed P0' + P1'.

Now, P0 is roughly 0.72. P0' is roughly 0.58. P1' is roughly 0.25. Therefore, P0' + P1' is equal to roughly 0.83. Therefore, the probability of scoring one run or more from state 0-000 is 0.28. The probability of scoring two or more runs from state 0-100 is 0.17. (All of these numbers are based on Keith Woolner's "An Analytical Framework for Win Expectancy" from Baseball Prospectus 2005.)

The difference is roughly 11%. Most (all?) of this will be accounted for by the fact that the man on first can be doubled off and the man who hit the lead off home run can't.

So that's the math. But do you really need it? A home run is one guaranteed whole run that no one can take away. The lead off walk increases your odds of scoring exactly one run by roughly 14%. The lead off home run increases those odds by ONE HUNDRED FREAKING PERCENT.

**EDIT** As pointed out in comment number one, referencing the probability of scoring one run is misleading, as scoring one run and scoring zero runs both count for nothing for the purposes of counting multi-run innings. The ninth-inning analogy is apt: it's the second runner scoring that is important and neither the lead off walk nor the lead off home run will have a great affect on what that second runner does. What is important is that the lead off home run eliminates the probability of the double play and the lead off walk does not. In other words, it is the out that is important, not the run. Point well taken. **END EDIT**

Tim, do us a favor and stop bringing this up as if it's surprising. You will sound a whole lot more intelligent and we I will be a whole lot less aggravated.

No, Todd Helton. You are wrong.

From Mr. Todd Helton, following the Rockies' amazing comeback win over Trevor Hoffman and the San Diego Padres in the National League's Wild Card Elimination game yesterday:

"I can't believe it," Helton said. "Can you believe it? We were down. We battled back. We did it against the best closer of all time."

Umm, Todd? Mariano Rivera says "Hi."

One of these things is not like the other

Buried deep in Buster Olney's most recent blog entry at ESPN is this note:

The Marlins will consider all options this offseason, perhaps even the trade of Miguel Cabrera. Marlins executives should know Cabrera and his habits better than anyone, and they have to ask themselves this question: Do they think that the 24-year-old Cabrera will get a handle on his physical condition and expanding waistline?

If they don't believe he will, they should look to move him ASAP, while his trade value is still extraordinary. If he arrives in spring training appearing heavy and has any kind of physical breakdown in 2008, his trade value will plummet, because rival talent evaluators will attribute his problems to his conditioning.

The Marlins' working model for this situation should be Kevin Mitchell, a staggering talent who hit 47 homers and drove in 125 runs at age 27, and then was basically finished as an everyday player within two years because of his condition. Cabrera could be one of the greatest hitters of his generation, but at some point, he will need to make an adjustment.

Let me begin by saying that I do not mean to quibble with Olney's chief assertion. It is entirely possible, more so than with most Major League Baseball players, that Miguel Cabrera will eat himself out of the Hall of Fame. Furthermore, it may indeed be the right idea for the Marlins to trade him, because he would certainly fetch a ginormous bounty. I don't believe either of these events are likely, but they certainly should be scenarios of which the Marlins should be aware.

No, my quibble is with the astonishing choice of comparison for Mr. Cabrera: Kevin Mitchell. This is a very bizarre and inaccurate selection. Kevin Mitchell is nearly the poster child for an above average baseball player's career path: breaks into the majors in his mid twenties, peaks at age 27, and then slowly declines for 5 or 6 years. I will admit that I do not know the details of Mitchell's career and that the chief reason for his decline is not a decrease in his rate of production, but in his playing time.

However, to compare him to Cabrera is astonishingly foolish. Let's look at Cabrera and Mitchell year by year using WARP3:

Age:        20    21    22    23    24    25    26    27    28
Cabrera:   2.5   6.7   9.4  11.5  10.9   ???   ???   ???   ???
Mitchell:  N/A   N/A   -0.1  N/A   4.1   5.7   6.7  12.8   8.6

            29    30    31    32    33    34    35    36 
Cabrera:   ???   ???   ???   ???   ???   ???   ???   ???
Mitchell:  5.3   3.8   4.8   8.0   0.3   2.2  -0.1  -0.1

Folks, Miguel Cabrera has had one of the best starts to a career ever. He broke in as a rookie for the Marlins' World Series run in 2003, serving as a huge lift and a big threat for the latter half the year. He then made himself a perennial MVP candidate by the age of 22. Simply great baseball players do not do that. Alex Rodriguez does that. Ken Griffey Jr. does that. Barry Bonds does that. Inner circle Hall of Famers do that.

Kevin Mitchell wasn't a regular until he was 24, and even then didn't play in 140 games until he was 26. Miggy has played in 150+ games since his first full season. Heading into his age 25 season, Miggy has roughly 40 WARP3. Mitchell had 4 WARP3 at that point. He had only 61.9 for his whole career.

Essentially, Kevin Mitchell is a bad comp for Miguel Cabrera. In fact, just about every ballplayer ever is a bad comp for Miggy (those named Mantle, Mays, and Rodriguez excepted). Miguel Cabrera is on track to be one of the greatest baseball players who ever lived. Kevin Mitchell was never even close to that. Ever. Busting out at age 27 is just far too common to serve as the basis for comparison to a man who put up MVP numbers at ages 22, 23 and 24.

Miggy may eat his way out of greatness, but the Marlins should absolutely not proceed like Miguel Cabrera is Kevin Mitchell. Indeed, they would be much wiser to treat him as if he were Frank Robinson, Ernie Banks, Eddie Mathews, or Lou Gehrig. Each of these players is more comparable to Cabrera than Mitchell.