Every now and then, we hear the worries that Japanese stores won't accept credit cards at all. This is still true to some extent, specially in the smaller towns outside the main tourist areas. But in central Tokyo, even the small family-owned gift shop in the alley are starting to accept the plastic. Still, for a peace of mind, it is still recommended that you carry some Japanese cash with you during your stay.
You can withdraw cash from the ATMs at Seven-Eleven convenience stores (practically one in every corner, at least in central Tokyo) or at the Post Office.
The transaction will usually require the input of your PIN Number, so make sure you know them.
If you would like to exchange your local currency for Yen, note that there is no Tokyo version of places like Chungking Mansion in Hong Kong, i.e. a place with really good rates. Your best bet is to find a ticket shop called Daikokuya (usually located close to the bigger stations in Tokyo; Shinjuku, Ikebukuro, Ueno etc) that does money exchange as well, and they usually have a better rate than the banks.
This is a sample of what was being offered in the Shinjuku area on May 7th, 2015
Daikokuya - 1 USD → 117.45 yen
Mizuho Bank - 1 USD → 116.67 yen
Travelex - 1 USD → 111.76 yen
*Inter-bank rate 1 USD → 119.4 yen
Just to be fair, Travalex offers a not-so-bad rate when exchanging Yen to Dollars, but the other way around you won't be getting a good deal as all, as you can see from above.
Once you have your Japanese Yen, note that even if you are buying just one bottle of water, the clerk will happily give you back your exact change even if you hand him/her a 10,000 yen bill. The only time when it's better to have the smaller 1,000 yen bill is when you board the bus (many buses don't accept 5,000 yen/10,000 yen bills) or a taxi, when the driver might not have small change available.
Keep in mind an increasing number of vending machines also accept all bills as well, but they will not take your one or five yen coins.