Since Silicon Valley consists of mostly suburban areas this is defintely the kind of place you'll need a car. While there are a number of public transit options within the Valley, how practical those are as a primary means of transportation will really depend on where you live and where you work. If you're walking distance to a bus stop or train station on both sides of that equation, then it's workable (though you'll likely still want a car for errands and weekend excursions). The lone exception to this rule is Downtown San Jose, which is a hub of transportation.
Caltrain has extensive coverage, running from San Francisco in the north to Gilroy in the south. It's got stations within most of the cities along the way -- San Mateo, Belmont, San Carlos, Redwood City, Menlo Park, Palo Alto, Mountain View, Sunnyvale, Santa Clara, and San Jose, to name several -- and serves nearly 20 million riders a year.
The area covered by the train is broken up into six zones and the cost of travel is dependent on how many zones you're traveling in. For example, a one way ticket within zone 2 is $3.75, but if you're going from zone 2 to zone 3 then that same ticket is $5.75. Monthly passes are also available, ranging from $84.80 to $349.80 depending on how many zones you'll travel through.
Significantly discounted fares are available to seniors, disabled, youth, and Medicare cardholders.
VTA Light Rail and Bus
The Santa Clara Valley Transportation Authority (VTA for short) operates a system of buses and light rail.
The Light Rail system starts in Mountain View and goes southeast to San Jose before splitting off on other tracks to the east, south, and west.
Bus routes mainly stick to the commercial roads cutting through the valley: the 101 freeway, Middlefield Road, Alma Street, and El Camino Real. You could be walking a mile or more to the nearest stop. The main exception to this is around Downtown San Jose, which has a relatively dense network.
San Mateo County has its own bus service to cover the 16 cities, 4 towns, and other communities that fall within its borders. Called samTrans, this bus service extends north into San Francisco and south into Palo Alto for commuters. However, the network of routes isn't particularly dense, so unless you live near a stop it's not going to be convenient.
Mountain View has a free community shuttle that operates 7 days a week from 10:00am to 6:00pm. It's got 50 stops around town.
Palo Alto also offers free shuttle service with stops at schools, commercial districts, and business parks.
Many of the Silicon Valley tech behemoths operate their own private bus systems for employees. Google alone carries about 4,000 people a day to its offices from around the Bay area. Facebook, Box, Apple, and many other companies also provide private bus service to employees.