First you need a multi national bank. I use HSBC. I know the Hong Kong branch, but not Australia so this applies to HK. Call the branch where you want to open. Explain you are a foreigner and want to open an account and will certify all your paperwork through the local (your home country) branch. They will fax or email you the documents then you have to take them to a HSBC branch in the US with international banking services and have the paperwork certified. If you live in Kansas or Montana obviously you are hosed on this part. Send the papers back and you have your foreign account. I'm not sure if all the new banking regs in the last 2-3 years has made this process invalid, you would have to ask.
That being said, moving large amounts of money to another currency to try to arbitrage interest rates is a very dangerous game best played by professionals. You need to look at the net interest rate after inflation, then the currency stability. I've seen my local currency move between .55 and .89 more than once in the last 5 years. I keep money offshore for safety more than investment.
Don't forget to file TDF-90-22.1, aka FBAR, by June 30 each year.
Almost no cash in banks in the US just real estate. Most cash in New Zealand at around 3%. Had some in HK but the interest rates have dropped to almost zero so I've moved most out. Don't keep much liquid cash anyway. I'm going to be in Malaysia and Indonesia in July. I will check out banking there while I'm on the ground.
Forget NZ unless you are on the ground. The only banks are local yokels and oz based local yokels. The only multis at all are JP Morgan and Citi. Both only do commercial and barely at that.
This interests me too. If all wealthy people hide their money off shore, there has got to be a good benefit to that.
Pretty amazing how successful the government has been in their 1984 style brainwashing campaign. There are many very good reasons to have offshore banking accounts including perfectly legal tax avoidance ala Mitt Romney. Yet the assumption of most people have is that offshore accounts are used exclusively for hiding money and tax evasion.
The end game of all this isn't tax evasion. It will cost more to administer fatca than the irs will ever recover from prosecuting tax evasion. The end game is unlimited access to snoop into every aspect of every citizens life without oversite. The whole offshore accounts/tax evasion hype is just a smokescreen.