What can I do about water for home use?

I hate the taste of my tap water, added to the fact that it leaves a brownish-yellow deposit on whatever it touches. I tried buying bottled water, but it doesn't taste much better than the tap water. I use roughly 2 gallons a day for drinking and food preparation and at $1.79 a gallon, that's over $100 a month to buy water and lug it home. I tried to get a home delivery service of spring water, but they require payment by credit card, which I don't have. Then I considered something like a Britta filtration system, but question if it can filter the copious quantities I require and the hle of changing the filters periodically. Any suggestions what I can do?