Investment retirement accounts (IRAs) are supposed to be sacrosanct. Because they’re intended to help you save for retirement, the Internal Revenue Service (IRS) doesn’t want you to withdraw any funds from them before you turn 59½—and to enforce that, it charges a 10% penalty on the amount withdrawn, along with income tax. But every rule has its exceptions. It’s possible to use funds from an IRA, penalty free, to buy a house, even if you aren’t six months away from your 60th birthday.

The rules differ depending on which type of IRA you have, though. Here are your options, and if you pursue one, choose one of the best brokers for IRAs.

Who Qualifies for the IRA Exemption?

To use money in your IRA to buy a house, you must be a first-time home buyer. The IRS defines that status rather loosely. You are considered a first-timer if you (or your spouse) haven’t owned a home at any point during the last two years. So even if you possessed a principal residence at some point in the past—say, five years ago—you may well meet the first-time-buyer requirement. The key word, by the way, is “principal.” If you’ve owned a vacation home or taken part in a time-share during the last two years, the exemption can still apply.

Also, you yourself don’t have to be the one shopping around. You can tap into your IRA and qualify for the exemption if the money is to aid your spouse, child, grandchild, or parent buy their home. And that’s even if you are a homeowner now.

The Traditional IRA Exemption

If you qualify as a first-time home buyer, you can withdraw up to $10,000 from your traditional IRA to help cover the costs of buying a home. Your spouse can also withdraw up to $10,000 from their IRA. (Remember, IRAs are individual retirement accounts; you don’t share them with a spouse).

Even though you’ll avoid the 10% early withdrawal penalty on this money, you’ll still owe income tax on any amount you (and your spouse) withdraw. Also, that $10,000 is a lifetime limit. You won’t get to use the first-time home buyer provision again to buy a home, even if you use a different IRA.

The Roth IRA Exemption

The rules are a bit different with a Roth IRA. A factor here is how long you’ve had the account.

First of all, you can withdraw a sum equal to the contributions you’ve made to your Roth IRA tax free and penalty free at any time, for any reason (this is because you’ve already paid taxes on the contributions). Once you’ve exhausted your contributions, you can withdraw up to $10,000 of the account’s earnings or money converted from another account—without paying a 10% penalty—for a first-time home purchase.

If the Roth IRA has been in existence less than five years, you will owe income tax on the earnings (though not on the converted funds). But if you’ve had the Roth IRA for at least five years, the withdrawn earnings are tax free, as well as being penalty free.

Self-Directed IRAs

Another option is to open—or convert your existing IRA into—a self-directed IRA, or SDIRA. These are specialized IRAs that give you complete control over the investments in the account. SDIRAs allow you to invest in a wider variety of investments than standard IRAs—everything from LLCs and franchises to precious metals and real estate. And “real estate” doesn’t refer merely to houses. You can also invest in vacant lots, parking lots, mobile homes, apartments, multifamily buildings, and boat slips.

The big catch: Real estate purchased with funds from an SDIRA must be an arm’s-length transaction. It can’t benefit you or your family (including parents, grandparents, children, spouse, and fiduciaries). In other words, you (and most of your relatives) can’t live in the home, use it as a vacation property or otherwise personally benefit from it. As the SDIRA—not you—owns the home, using personal funds or even your time (sweat equity) to benefit the property is also prohibited.

“There are many ways you can use your self-directed IRA to purchase real estate inside your IRA. You could buy a rental property, use your IRA as a bank and loan money to someone backed by real estate (i.e. a mortgage), you can purchase tax liens, buy farmland, and more. As long as you are investing in real estate [that’s] not for personal use, you can use your IRA to make that purchase,” says Kirk Chisholm, wealth manager at Innovative Advisory Group in Lexington, Mass.

Thus the SDIRA option works mainly for an investment property—a house or an apartment you want to rent out for income. All the money that goes into or comes out of the property has to come from or go back into the SDIRA. However, once you turn 59½, you can start withdrawing assets from your SDIRA. You can then live in the home, as it will have become your personal property after distribution.

Tap Your 401(k) Instead

If you have a 401(k), you might think about taking a loan from that account instead of withdrawing money from your IRA. In general, you can borrow up to 50% of your 401(k) balance—up to a maximum of $50,000—for any reason without incurring taxes or penalties. You’ll pay interest on the loan, typically the prime rate plus one or two percentage points, which will go back into your 401(k) account. In most cases you have to repay the loan within five years, but if you’re using the money for a house, the repayment schedule may be extended to as many as 15 years.

A couple of things to keep in mind: “You will have to include the payments in your monthly budget. Also, the interest you are charged for the 401(k) loan may not be tax deductible (check with your tax advisor) and will probably be higher than current mortgage rates,” says Peter J. Creedon, CFP®, ChFC®, CLU®, CEO, Crystal Brook Advisors, New York, N.Y., adding, “Another minor point is you are paying the retirement loan back with after-tax dollars, so the loan may be more expensive than you may think.”

In most cases you repay the loan through automatic paycheck deductions. This sounds easy enough, but it’s important to understand what happens if you miss payments. If it’s been longer than 90 days since you’ve made a payment, the remaining balance will be considered a distribution and will be taxed as income; if you’re under 59½, you’ll also owe a 10% penalty. Another caveat: If you leave your job (or are let go), you’ll have to repay the entire loan balance within 60 to 90 days. Otherwise, the balance will be taxed, and you’ll owe the 10% early withdrawal penalty (unless you are older than 55 when you leave your job). 

The IRA Rollover

Consider this: Instead of withdrawing the money from your IRA, borrow it. Technically, you can’t take a loan from a traditional or Roth IRA, but you can access money for a 60-day period through what’s called a “tax-free rollover”—as long as you put the money back into the IRA (whether the one you made the withdrawal from or another one) within 60 days. If you don’t, income (including state) taxes and penalties are imposed.

This is mainly a short-term solution to a specific problem. For example, “Some first-time home buyers may want to have a substantial down payment to avoid [having to take out] private mortgage insurance,” says Marguerita M. Cheng, CFP®, CEO of Blue Ocean Global Wealth in Gaithersburg, Md. The tax-free rollover might be “the most efficient way to access funds for the down payment,” qualify for better financing and thus clinch the home purchase.

Plan Ahead

In terms of timing, if you want to take advantage of the IRA first-time home buyer’s provision, plan ahead. Any IRA funds distributed to you must be used within 120 days of your receiving them, and they must be used to pay qualified acquisition costs: to buy the home, build or substantially renovate the home, or pay other financing expenses (closing or settlement costs, mortgage points, etc.). They can’t be used to prepay an existing mortgage or on general furnishings; they have to be used to acquire the property. And the property is considered "acquired" on the date you sign the contract to purchase it, not the date escrow actually closes.