Before World War II, the Key System serviced a huge chunk of the San Francisco East Bay Area (Oakland, Alameda, Berkeley, etc) with an electric streetcar and train network that connected with San Francisco. But, it was discontinued after World War II because trains were the past and cars were the future.
Not very long after they brought the system down, people started to clamor to bring it back. Except new and different.
BART was to be the first "modern" rail system, built to coexist with the automobile. It was to be refined and modern and electronic and space age. All metro rail systems built after BART took great advantage of what people learned building BART, both in what they did right and what they did that was stupid. See, in this urge to make it modern and fancy, the seeds for present problems were planted. Problems that make BART unable to be a useful rail system and that the collected bay area transit agencies have refused to deal with for so long.
The problem with BART, at the core, is that it's more expensive than other similar transit systems, even after you adjust for the properties of the bay area. Because of the cascade of deep-lodged problems, starting with the distance between tracks is, BART will remain this way until a very-expensive retrofit occurs... and the more extensions they try to add to BART, the worse that will be.
Rail cars do not get cranked off assembly lines like cars. They are usually semi-custom creations where a given design is adjusted to match the desired properties of a railroad... how they do electronic signaling, how they want the seats arranged, what other cars they are expected to work with, how high the train platforms are, etc. However, there is some level of standardization and there is a market for used rail cars to be had.
BART cars, however, are so completely unlike all other rail cars that this is not possible. The biggest reason for this is the distance between the rails. For a variety of reasons, BART uses "broad gauge" that was supposed to allow for cars that were harder to topple, quieter, and more optimal. Were other transit systems to have been designed that way, it would have been OK, but even the Washington DC Metro system, which can be considered BART's cousin, doesn't use broad gauge.
If you've ever riden past BART's maintenence yard, you'll see a bunch of weird-looking rail cars. Why are they weird? Well, they took a switching engine or flatbed car designed for normal-gauge railroads and then added longer axles. That works for maintenence vehicles, but you can't do that with anything else.
Mostly standard railcars allows for a lot more flexibility. Santa Clara VTA sold their first set of trains to Sacramento in order to get cars with a lower boarding level. The passenger coaches that CalTrain uses are the same ones that at least twelve other agencies have. Agencies using the same Bombardier BiLevel Coaches as CalTrain have loaned them out to other agencies. CalTrain picked up eight more cars to alleviate crowding. BART can't do this.
This also means that just simple things like BART cars are expensive. BART expects to spend around 3.4 billion dollars for 700 cars, which is about twice as expensive on a per-car basis than CalTrain spent on their last buy. Except that each CalTrain car has around twice as many seats, plus a bathroom on each car. The Acela Express high speed trainsets were a mere 661 million dollars which purchased 20 sets, which is about 50% more per car than BART costs. The New York City Subway is spending 2.4 billion for 1700 cars, which is even cheaper on a per-car basis than CalTrain's cars.
CalTrain and the other commuter rail systems have the option of running on normal trains, shared with freight rail. It's often better to lay dedicated track, of course, but it does provide a way to extend CalTrain service down to Gilroy. When you do run rail lines, the cost per mile is almost two orders of magnitude more expensive... $200 million instead of $4 million... because of the unique requirements for BART.
One of the bottlenecks with BART is the trans-bay tube. Pretty much, at rush hours, it is impossible to safely run any more trains through the tunnel. There's a rail corridor across the Dumbarton bridge that has an estimated $400 million rehab bill. They haven't started estimating what it would cost to add more rail lines to the tube, but given that it cost $180 million in 1970, thus being about a billion in 2009 terms, it's unlikely it would be that cheap.
One problem with putting transit into the bay area is that most of it has fairly low density and the range of journeys is fairly large and wide. One of the biggest CalTrain improvements was to add an extra pair of "passing" tracks and allow trains to pass each other, thus allowing a person to get from San Jose to San Francisco in under an hour. However, the way BART works is incompatible with this and, given past history, it's likely to be difficult to change it even if you had passing tracks because of the system of moving blocks.
Even if they could run express trains, the fastest train designs do not use a third rail like BART. The existing trains can go, at most, 75mph, and the fastest practical third-rail system can go 100mph. Meanwhile, high-speed rail systems with overhead wires can reach 175mph.
BART's long-term goal is to replace CalTrain and ring the bay.
In 2003 a 1.5 billion dollar extension (that's for a mere 8.7 miles of track and four stations, with no new cars) opened between Daly City and the San Francisco Airport. It was projected to serve 50,000 people every day but has, at best, serviced 37,500. There's even a cross-over where you can transition between the CalTrain system and the BART system, intended to suck up CalTrain riders trying to get downtown. The problem is that if you want to get to San Francisco, it's faster and cheaper to get there via CalTrain unless you want to go close to a BART station. The substantial CalTrain bike population isn't able to take their bikes on BART during rush hour. And, starting from the CalTrain service area, it takes far too long to get anywhere in the East Bay to be useful. It's easier to drive directly there or even to just drive directly to a BART station.
We've even got funding on the way to start extending BART down to San Jose.
Most of the problems with BART are not without solutions. If you wanted to change the distance between the rails such that BART could use normal cars, that's not impossible. It's just expensive, because you'd need to have another rail in between the existing two traction rails in the interim. If you wanted to switch the signalling system, you can do so. All of these things cost money.
One mental block in big decisions like this is the concept of Sunk Costs. See, the money already spent and "invested" on BART is gone. You can't get it back and if you do, it's rare you will get all of it back. If BART is paying several times more per car than other transit systems and several times more per mile of rail line than other area transit systems, perhaps it's better to consider what would happen if you didn't make new rail lines use the existing rolling stock.
Remember that the extension to San Jose that has been funded already does not include the cost of new cars, that's a separate and unfunded $3.4 billion task that happens to include the cars that have yet to be purchased. This is so that BART can avoid the question as much as possible of why we don't just string CalTrain up to Fremont.
In my opinion, and I believe the facts bear me out on this, because most people aren't going to think about what it means to extend BART the way it is around the entire bay as opposed as to what it means to have a rail line circling the bay, we are being swindled by our transit planners.
The tragedy here is that we're paying more money for an ineffective system. So, my tax dollars go to fund a system that's hideously inefficient. And my wife has no rail options that will get her to work within a reasonable timeframe, so she has to drive.