Caveat: This may be a cop-out, but I don’t have the time these days to do a lot of meticulous research, so I’m going on my general knowledge.
It is my understanding that the way American sports leagues started was similar to the way European soccer is currently run. The leagues had 10-15 teams, each team played the other teams an equivalent number of times, and the team with the best overall record at the end of the season was crowned champion. Hence, in the olden days of baseball the goal was was to win the pennant. Wikepedia:
The last few weeks of the regular American professional baseball season are known as a pennant race. This is a holdover from the time (pre-1969) when the league championships were determined by the team with the best record at the end of the regular season.
The introduction of a series of games to settle the championship evolved out of desire to see a true champion crowned from the winner of the two separate leagues, the National League and the American League. In theory, the leagues could have done two other actions:
1) merged their regular seasons and continued playing each team an equivalent number of times
2) separated all the baseball teams into hierarchical leagues- that is, to say the best 15 teams will play in the top league, and next best 15 teams will play in league one level below. This is the model followed by soccer around the world (not in the US).
It is my strong hunch that baseball chose to tack on a series at the end of the respective American and National league seasons rather that purse either of the other two options simply because it required the least amount of logistical headache. The season-ending series, henceforth known as the World Series, preserved the leagues as distinct entities during the regular season and didn’t require the coercion that would have been necessary to see some teams dropped into a lower league.
I’m sure much has been written on this topic elsewhere by baseball historians, but what interests me is that the decision was not taken with a narrative perspective in mind. That is, the people in charge of baseball didn’t see the World Series as solving the problem of the baseball season lacking a natural climax. Sometimes there was a thrilling pennant race, but I’m guessing most of the time the Yankees put it to bed early on.
The reason I think the lack of climax didn’t bother anyone is because of the widely held view – and here we connect back to European soccer – that deserving winner of the championship trophy was the best team in the league. How should one determine the best team in the league? Well, if each team played all the other teams the same amount of times, then the best team is the won with the most wins. Simple and fair.
The World Series introduces a different element, because the Best-of-7 series, with its much smaller sample size, can hinge much more on chance. It’s only 4-7 games, compared to the 150 that were played during the regular season. In this case, we now have a championship that is not based a season-long effort, but on being sharp in a short series.
Long story short: drama wins big with the introduction of the World Series. Every game counts, winner take all, loser goes home. And where drama wins, justice loses. A team like the Atlanta Braves in the 1990′s can consistently prove over the course of 162 games that it is a superior team, only to fail in a short series in the playoffs. And American sports fans can draw the conclusion (and I admit, back then I was one of them) that despite winning the large sample size, the fact that the Braves couldn’t win when it counted in the playoffs marks them as relative failures.
Over time, despite the fact that historically things were otherwise, Americans subconsciously accepted the playoffs as opposed to the regular season as being the more legitimate proving ground for a team. Get to the playoffs, peak in the playoffs, and become champions.
I would argue the only reason for having the playoffs stems from their superior ability to offer sporting drama, but not as a mechanism for truly determining the best team. (The NFL is a different animal here. This applies to sports with large sample sizes and equivalent schedules in the regular season MLB, NBA, and NHL.) The playoffs, with their short series, offer the inferior team a better chance of winning than they would have in a large sample size of games like the regular season. The NCAA basketball tournament is the epitome of this rule: Princeton University has a slight chance of upsetting Kentucky in a single game- it has essentially zero chance of defeating them in a 7-game series. (Whatever brilliant plan Princeton concocted to defeat Kentucky one game will be quickly dealt with by the Kentucky coaching staff, and thus the element of surprise is negated.)
Now, just because playoffs are unjust is not a reason to not have them. They have a lot of value in the predictable (in the good sense) drama they provide. But I think one still has to consider whether the value of this drama is worth it in light of the two factors discussed thus far: the siphoning of excitement from the regular season and the increased likelihood that the best team (as determined by the largest sample size of games) will not be crowned champion.