Bitcoin is anonymous, at least until you try trading for real money; e.g. Verifying with Mt. Gox. However, even when all of ones' transactions are kept in a bitcoin wallet, there are ways for those willing to go data mining, to find patterns and follow trails as we reported on earlier today. There are ways to defeat this tracking, or at least make it a lot harder.
Thanks to having a public Block Chain, it is possible to search for a specific Bitcoin address. Having paid someone once on Bitcoin, you have their address. You can easily go and put that address into the search box at the Block Chain website. The resulting page will show you items like how many times the address has been used for a transaction, what the total amount of money received is and what is its current balance.
Even worse is that you can view a full breakdown of every transaction of that address has sent and received. Fortunately, because Bitcoin addresses are anonymous, you can't tell where those payments were going, or coming from.
However, you could build up a directory of addresses of everyone you've ever dealt with, and then try cross referencing those addresses, and if several people combine their identifying bitcoin addresses, a whole series of transactions could be reconstructed with real identities.
There are practices that businesses can implement to guard against this kind of attack - Wired has a case study of Foodler's attempts to do so.
The easiest thing to do is to create a new address for every transaction. Even the most widespread Bitcoin-qt client recommends you do this. Addresses cost nothing to create, so there is no impact on your bottom line.
The next suggestion, which comes from Foodler's Christian Dumontet recommends, is to split payments up into random amounts and with random block lengths.
Admittedly, such data gathering attacks would be difficult to implement, but they're not impossible. In the same way, it's unlikely and highly difficult for someone to raid your trash and reconstruct your shredded bills - such things still happen and we need to be ready for them.