≡ Menu

Why it’s difficult to retire with passive income

Get new articles via Email:
RB40 won't spam you


difficult to retire with passive income

Passive income will keep rolling in even when you’re on vacation.

Generating enough passive income to cover the cost of living is the holy grail of early retirement. If you have enough passive income, then you’ll have a lot more choices. You can invest all your active income and ramp up the passive income even more. You can donate some of your passive income to a good cause. Or you can even quit your day job to pursue your own interests. However, that level of passive income can be extremely difficult to achieve.

What’s Passive Income?

I’m sure most readers are familiar with the concept of passive income, but let’s take an in depth look anyway. Passive income can be somewhat amorphous. According to the IRS, passive income only comes from two sources: rental activity and business activities that you do not actively participate in. For me, anything other than income from your job(s) is passive income. I know that’s probably too inclusive, but that’s how I see it. So that’s the two extremes and your definition is probably somewhere in the middle.

What are some sources of passive income?

Rentals – Acquiring a property and renting it out is a great way to generate passive income. You get the monthly rent check and usually the property will appreciate over the long term. However, being a landlord can be stressful and take up a lot of your time. The best way to go is to use a property management company, but a good one can be difficult to find. Investing in rental properties can also require a lot of cash so it might not be possible for someone just starting out.

Dividend stocks – Many companies return a percentage of their profit to their investors. You can invest in these companies and enjoy regular dividend payments. Chevron, for example, will send you $4.28/year for every stock you own. I love our dividend stock portfolio because it doesn’t require a lot of maintenance. I still need to review it occasionally to make sure the companies are doing well, but that doesn’t take a lot of time. Investors also need to be aware that the stock market is volatile. Stock price and the dividend can decrease during bad economic years.

Peer to peer lending – You can lend your money to borrowers through peer to peer lending companies like Prosper and Lending Club. The risk of default is pretty high, but the ROI is supposed to make up for that. My peer to peer lending portfolio is not doing that great. The ROI is about 7.8% and I’m not sure if it’s worth the level of risk. If the economy heads south, I’m sure the ROI would sink like a rock.

Business income – It’s hard to be completely passive as a business owner. If you don’t pay attention to your business, usually it will go downhill. I do have a passive online business at Midlife Finance. I hire writers, a VA, and an advertising rep to run the whole site. The site takes only a few minutes of my time per month, but it doesn’t make much money either. I’m sure there are many successful businesses owners out there, though.

Royalties – You can earn royalties from a book, music, photography, art, and many other creative products. I don’t know much about royalties, but I heard the income is pretty small unless your product is extremely popular.

Bonds –You can lend your money to the government or private companies by buying bonds. These institutions then pay you back the principle plus interest. Bonds are generally much more stable than stocks because a bond is a payment of debt. Usually, you will get your money back unless the company goes bankrupted. Bonds are pretty boring, but you need it to balance out your portfolio. When the stock market crashes, bonds usually do well.

Certificate of Deposit – You can buy CDs from your banks and credit unions. Basically, you let the bank use your money during a fixed term (3 months, 6 months, or 1 to 5 years.) CDs are very safe, but the interest rate is pretty low. Withdrawals before maturity rate usually have a hefty penalty attached.

Annuities – An annuity is a type of insurance. You pay the insurance company a lump sum up front and they will send you a monthly check for the rest of your life. Actually, I don’t know much about annuities because I’m not the right age for it. Annuities can have high fees and expenses so you need to shop around to get the right product from a reputable company.

Social Security Benefit – If you work in the US, then you’re paying into the social security program through payroll taxes. Social security benefits include retirement income, disability income, Medicare, and Medicaid. Retirement benefit can begin as early as 62 and the amount is dependent on how much you earned over your working years.

Whew, that’s just some of the more common ways to generate passive income. The problem with passive income is that it generally takes a lot of time and effort to buildup. Supporting your retirement with passive income will be difficult because the rate of return can be quite small. For example, safer high yield dividend stocks can return 3-4% of your investment. Let’s say your monthly expense is extremely low at $2,000/month. To generate that much dividend income, you’d need about $700,000 invested. That’s a lot of money and we haven’t even included tax yet.

Financially, I think the most attractive thing on this list is the rental property. You can leverage your investment and borrow money from the bank to buy properties. The rent can be increased to keep up with inflation so that’s another positive. Eventually, the property will be paid off and you’ll get to keep most of the rent checks. You still need a substantial amount of cash for the down payment and good credit to get a mortgage, though.

Anyway, the point is that it’s quite difficult to build up a substantial passive income unless you have a high salary already. A more realistic goal is probably shooting for having enough passive income to pay 50% of your monthly expense and then fill the gap with active income. Let’s continue with the example above. Instead of investing $700,000, you’d need only half that. Then you can work part time on something you like to generate $1,000/month to cover the rest of the bills. A little active income can go a long way in retirement.

What do you think? Is it possible to fund your retirement with passive income? It’ll definitely be easier once social security kicks in, but that’s no longer early retirement.

Yes, it is possible. See how Sam at Financial Samurai plans to generate $200,000/year from passive income.

Get free update via email:
Stay in touch with Joe and see how he handles Retiring by 40 and being a stay at home dad.
We hate spam just as much as you

{ 53 comments… add one }

  • Shane August 13, 2014, 5:11 am

    Articles like this is why I love your website! I appreciate that you provide in detail the different types of passive income and which ones you favor. I agree with you that sufficient amount passive income is hard to obtain and depend on but it is worthy thing to strive for. I have found if you set small goals regarding passive income it can create confidence and momentum. What I mean is this. Suppose you set a goal of creating enough passive income to cover a small expense such as cell phone or utilities for starters. If you can shoot for obtaining these small steps in regards to passive income then they will add up. I like the idea of trying obtain half with passive and the rest with active income. There is nothing wrong with always having active income coming in granted that you are working at something you enjoy. Thanks again for this article and can’t wait to hear other responses about this topic. If anyone has strategies for creating lots of passive income I would love to hear them.

    • peter August 13, 2014, 10:20 am

      Some financial gurus advocate buy stock in companies that you do business with. Phone, bank, grocer and the like.
      Dividends from T could pay cell-phone bills etc.

    • retirebyforty August 13, 2014, 12:13 pm

      That’s a great idea. I also like to chunk my goals. One good goal to shoot for when you’re starting out is covering 10% of your monthly expense with passive income. Once you reach 10%, then you can go for 20%. Eventually, you’ll get to 100%, but it will take a lot of time for most people.

  • EL August 13, 2014, 5:28 am

    That is awesome if you can successfully build up the passive income. Not many can do it, but if you can you will have an easier retirement. I am looking to build mine with dividends, and rental income.

  • Financial Samurai August 13, 2014, 7:55 am

    I think it’s possible, but it takes a long damn time! It’s taken me 15 years to build around $150,000 in passive income per my latest post, and I’ve still got probably 3-5 years left to get to $200,000.

    Curious, if you believe rental property is one of the best ways, why did you sell?


    • retirebyforty August 13, 2014, 12:15 pm

      I meant to link to your post, but it was 2am when I finished this one and I just needed to hit the hay. I linked to it now.
      You are doing extremely well with your passive income, but is that really possible for regular people? I don’t know…

      I sold because I don’t have time to deal with them right now. They were too far away and I couldn’t manage them effectively. Actually, we got a replacement property closer so we’ll see how that turns out in a few years.

      • Financial Samurai August 13, 2014, 4:29 pm

        Very cool, thanks!

        Did you write about the replacement property yet? I don’t think I saw it. I thought the last update was that you were looking to do the 1031 exchange.

        The other problem w/ passive income is that interest rates have come down over the past 30+ years. So generating an income return has become harder.

        • retirebyforty August 14, 2014, 9:52 am

          Not yet. It’s not really a great investment so I’m reluctant to write about it. I’ll try to get an article out sometime soon.

  • Justin August 13, 2014, 8:08 am

    Difficult yes, but not impossible. I went the “save a lot of money and live off the dividends and take occasional capital gains”. If you have a decent income from a STEM/medical/professional job then it’s not too hard to save a large percentage of your wealth over a couple of decades and retire in your 40’s. Using a 3 to 3.5% withdrawal rate means you can pull $35k per year from a million in savings. Pay off the mortgage before you retire early and $35k goes farther than you think. Or if you need more, save more. $2 million = $70k/yr passive income using a 3.5% safe withdrawal rate.

    • Financial Samurai August 13, 2014, 9:21 am

      But we’re talking earning passive income, not drawin down capital.

      More fun and challenging to build a perpetual passive income machine!

      • Robin Munich August 13, 2014, 2:06 pm

        Hey Sam

        Re-read your post again and noticed the bulk of your income comes from real estate, since you worked foe less than 20 years how did you manage to pay off so much mortgage debt so quickly, even with saving 70% of your income it wouldn’t explain how younwere able to aquire so much wealth so fast, unless youndid a post on this and I missed it


        • retirebyforty August 14, 2014, 2:38 pm

          I’m pretty sure he still have mortgages on his properties. He also had a highly paid job in the finance sector and lived frugally, a great combo.

          • Rob in Munich August 14, 2014, 11:27 pm

            Thanks, that explains a lot, I’m in the same boat, we just bought two more rental properties, added nicely to our income but will take a while to pay off the mortgages.

  • Kurt August 13, 2014, 8:14 am

    The ~zero saver interest rate that has prevailed now for about 6 years sure makes a lot more challenging living off passive income, particularly if you’re conservative and you don’t want to dump a lot of money into risky stocks. If you read the book “Your Money or Your Life” (one of my money and lifestyle bibles) and follow its plan, in the end you’d be living off Treasury bond interest. Not so easy to do at today’s 30-year rate anyway!

    • retirebyforty August 13, 2014, 12:17 pm

      The low interest rate is really tough on retirees. Hopefully, we’ll see better interest at some point. But then, we’ll see higher inflation as well, right? It’s tough.

  • Even Steven August 13, 2014, 8:24 am

    I think it is possible, I surely am drinking the kool-aid, that’s why I write about it. I’m heavy rental real estate as I believe the best way to wealth is through this avenue.

  • SavvyFinancialLatina August 13, 2014, 9:37 am

    More motivation to get started early, right?

  • Emily August 13, 2014, 9:38 am

    I think what scares me about relying on passive income is that some of the sources are likely to drop off, especially if you are treating them as truly passive. Businesses can fail, royalty-generating work can diminish in popularity, properties need upgrades or the rents may not keep up with inflation, and of course SS benefits can be cut. We’re talking about over the course of many decades, here! Financial products (index funds) are the most attractive to me because the underlying asset will only erode from inflation, which you can work against through your asset allocation. I guess investing in various asset classes seems more inherently diversified to me than having a small number of properties or businesses or whatever, since while it can drop it is unlikely to go to 0 and is not location-dependent.

    • retirebyforty August 13, 2014, 12:20 pm

      You’re right about dropping off. I think you have to actively manage your passive income for them to grow. You can’t just put it on auto pilot. I like equities too, but it can also be pretty volatile.

  • Merlion August 13, 2014, 9:59 am

    I elected to save between 30-40% of my salary in the past 10 years which allowed me to build up reasonable retirement savings. I have invested in 2 rental properties – the rents (after expenses) provides about 40% of my annual expenses. I am looking to buy another property as well as increase my purchase of dividend paying stocks to meet the rest of the gap.

    • retirebyforty August 13, 2014, 12:21 pm

      Great job with your saving rate. Not many people can do that. It looks like you’re getting there. Do you think the 2nd half of the journey will take as much time or less?

  • Emma @ emmalincoln.com August 13, 2014, 10:32 am

    My question is the same as Sam’s – I know you sold your rental property for a lot of good reasons, but if you think it’s the best source of passive income, are you regretting your decision?

    • retirebyforty August 13, 2014, 12:22 pm

      Not really. They weren’t the right property for me and the price was very good. We’ll probably look at more rentals when the market is down. Actually, we picked up a replacement property which we’ll move into at some point. We’ll see how that turns out.

  • Tawcan August 13, 2014, 10:56 am

    I think it’s definitely possible but it does take a bit of time to build enough passive income to cover your expenses. Take dividend stocks for example, you need to pick stocks that increase dividend on a yearly basis so your yield on cost would increase slowly over time and the dividend amount would keep up with inflation.

    • retirebyforty August 13, 2014, 12:23 pm

      Dividend growth is a great way to go, but you’ll still need to keep adding to the investment. The good thing is you can start small and keep investing. It will take a long time, but if you’re diligent, then you can get there.

  • Dividend Mantra August 13, 2014, 12:02 pm


    Obviously, my preferred method of passive income is dividend growth stocks. Sure, it takes some hard work in the beginning, but the snowball eventually starts to roll itself down the hill.

    Rental properties aren’t for me; I’d rather collect the high dividend from REITs.

    But I agree with you in that a little active income can go a long way. Working, say, 20 hours or less per week while covering all your bills and then some frees up a lot of time to do what you enjoy.

    Best wishes!

    • retirebyforty August 13, 2014, 12:24 pm

      REITs are pretty good too. You don’t have to deal with tenants and the toilet calls…

  • Nicoleandmaggie August 13, 2014, 12:17 pm

    Annuities are kind of odd because you lose your principal to buy into one… much like a less risky 4% rule. And one could argue rentals are too much work to be truly passive.

    • retirebyforty August 14, 2014, 9:50 am

      I’m thinking about buying enough annuity to cover the bare minimal living expense. That way, at least I’ll have the basics covered. We’ll see in 20 years. :)
      Yeah, rental properties can be a lot of work. Passive income aren’t really passive anyway, but rentals probably take up the most time from this list.

  • retirebyforty August 13, 2014, 12:18 pm

    I don’t know much about this insurance product. I’ll need to do some research. Most of what you’re saying is going over my head at the moment.

  • Tommy August 13, 2014, 12:21 pm

    As someone who is living off of my portfolio I can tell you it is very hard to get enough interest and dividends today to pull it off. My passive income covers 66% of my current early retirement lifestyle this year. The rest is funded by way of the bucket approach I have and includes cash and occasional sales of equities to keep things funded. I am currently just under a 4% withdrawal rate. I am on what I call my second retirement and I don’t worry too much about it as I believe in retiring early and often so who knows when something I am passionate about doing for pay comes my way again. I truly see the benefit of rental properties as a passive income stream and I wish I had the temperament to be a landlord but I just can’t bring myself to do it. I think it would be constant drag on my retirement-mellow.

    • retirebyforty August 14, 2014, 9:51 am

      66% is great. I like your philosophy. You don’t need to stay retired. If you find something you like to do, then why not get paid for it. The landlord thing is kind of a crap shoot. You have to vet your tenants really well to avoid future headache. That’s the main thing.

  • Bryan August 13, 2014, 9:30 pm

    I think most of us LIKE to work and be productive so needing 100% passive income to pay all expenses seems extreme. Personally I have enough investments to pay about 30% of my living expenses and the other 70% I pay between two businesses I own which I actually enjoy. It’s the shortcut to a good life. So I can live now at age 35, a fantastic life, rather than needing to save another 15-20 years before I can pay 100% with passive income.

    A friend always told me “live a life you never want to retire from”. This has been my strategy and although I’m not totally financially independent, I have hard money loans, rentals, stocks that pay 30% of living expenses and fun businesses for the other 70%.

    We could certainly trim expenses though as our annual expenses are around $90k… 3 kids, living in an expensive area, etc. So I suppose if the crap hit the fan we’d move to a very cheap place and try living off $30K and then I’d cover us with passive income.

    I think work helps us stay healthy though so pick work you like and take the shortcut to happiness.

    • retirebyforty August 14, 2014, 9:55 am

      Oh wow, great job! That sounds like an ideal way to live. I’m shooting for 50% passive income, but it will take a while to get there. Of course, once pension and social security kick in, then it will be much easier.
      Your philosophy about work is right on too. That’s why I like part time work.

  • Ernie Zelinski August 14, 2014, 12:43 am

    As for me, I turned 65 this year and my Canada Pension is only around $550 a month.
    Although I have dividend stocks and savings in the area of over $750,000, I will not touch whatever new interest and dividends will be generated this year. I don’t need to even though my government pension is only $550 a month.

    The royalties from the print edition from my book “The Joy of Not Working” (written 23 years ago) should generate me around $12,500 for the year. The ebook edition should generate around another $12,500 for a total of $25,000 in pretax profits from this title.

    But the bigger earner will be my “How to Retire Happy, Wild, and Free” (written over 10 years ago.) This book should earn me at least four times as much in pretax profits as what “The Joy of Not Working” earns me.

    In short, my passive income from book royalties will more than cover my expenses.

    As an aside, I recommend that people read “You Can Retire Sooner Than You Think” by Wes Moss. In the book, Moss states that the happiest retirees have several sources of retirement income. You can read my featured review of the book on Amazon at:


    “You Can Retire Sooner Than You Think” is really one of the best retirement books ever written.

    Ernie J. Zelinski
    The Prosperity Guy
    “Helping Adventurous Souls Live Prosperous and Free”
    Author of the Bestseller “How to Retire Happy, Wild, and Free”
    (Over 200,000 copies sold and published in 9 languages)
    and the International Bestseller “The Joy of Not Working”
    (Over 275,000 copies sold and published in 17 languages)

  • Melanie August 14, 2014, 1:22 am

    I am just starting to learn about passive income, so thanks so much for this detailed and specific post! It was so helpful. I would love to make passive income. I feel like I work pretty hard for my money, so it would be nice for it to work for me more.

  • Jon August 14, 2014, 3:55 am

    It definitely takes a lot of time and effort to build up a nice sized stream of income through passive ventures. I take the approach you mention at the end. At first, I was all excited to quit working next week. Then I slowly realized how much money I would need to generate a passive income. So I shifted my focus. Instead of focusing on getting $5,000 a month in passive income, I worked on paying for my car’s monthly gas through passive income. Then the groceries. Then the mortgage. By approaching it this way, I find it much more exciting and interesting.

  • Andrew August 14, 2014, 6:32 am

    I’ve been looking into getting more passive income. The rental property idea really intrigues me but it’ll have to be long distance because properties in NYC are too expensive. I don’t own much dividend paying stocks and have focused more on growth stocks. My lending club ROI is about the same as yours. I have only a small amount of money in it…just for fun, but agree that the risk might not be worth the returns.

    • retirebyforty August 14, 2014, 9:57 am

      I don’t think I can deal with long distance rental. I guess I’m too much of a control freak. I’m not very good at hiring people either….

  • Stefanie August 14, 2014, 6:57 am

    I’m in the process of writing a book, which in an ideal world, would provide some passive income. But I’m not counting on it :)

  • Financial Forager August 14, 2014, 6:57 am

    I am currently trying to build up my passive income stream with a dividend portfolio. I think the key is to cut your expenses, pay off your house and never borrow money. I have seen two other people in my life achieve this goal. So, I try to find out how they do it. I asked them as many questions as I can and try to live my life like them. I also notice that living simple is also key.

    • retirebyforty August 14, 2014, 9:58 am

      I also think that cutting your expense is the key. I’m writing about that today and will post it tomorrow. Paying off your house is optional, but it’s good to have no debt.

  • Brent August 14, 2014, 7:46 am

    Nice article on sources of passive income. It’s usually surprising when most people realize how much they would need invested to supply enough passive income to live on. I’m targeting 5k monthly in passive income from my stock portfolio which would require a portfolio size of $1,714,285 that yields 3.5%. I’m sticking to just dividend growth stocks and real estate for now. I have a small p2p lending account but being in Texas, I can’t invest directly. My biggest worry is a change in tax code that would tax dividend income more. That’s something that you can’t really plan well for.

    • retirebyforty August 14, 2014, 9:59 am

      Dividend stock and real estate are probably the two best options there. You still need bonds and cash to balance out your portfolio, though. Don’t neglect them. 5k/month would be quite nice.

  • davidmichael August 14, 2014, 4:09 pm

    Great article Joe. I have to laugh as I am working at a seasonal job along the beautiful Metolius River this summer as a camp host. All of my earnings are going into Passive Income investments, namely…P2P Lending, and Dividend Paying stocks. With little effort, I am receiving about 12% return with Prosper. Working on building it up to the $50,000 mark asap. At $100,000 the return per month on P2P would be nearly $1000 a month. And I don’t agree with you on the downside of P2P. I am investing in A-B-C and a few D loans with my own set of filters. Out of 500 loans, I have only had one default, losing $21 in nine months but earning $950 so far. The P2P is a good short term (5-10 years) investment in my opinion as it will take about 30 years to get those dividend paying stocks to pay out.

    My most worry free and consistent performer over the past 12 years has been IBonds returning about $400-500 a month (4-5%). Of course, with interest rates so low now, the new IBonds are just not returning enough to make them worthwhile. I wish I had focused on Dividend Paying stocks about 40 years ago. I agree with you as that’s the way to go for kost people. Not sure being a landlord is worth all of the trouble! But just buying one extra house, like you have, can make all of the difference.

  • No Nonsense Landlord August 14, 2014, 7:44 pm

    Great advice. I am paying off another mortgage tomorrow, which adds another $13K annually to my passive income from rentals now well over 6 figures. Using a Property Manager is not the answer, unless you can manage the property manager effectively.

  • Annie Logue August 15, 2014, 12:29 pm

    I always thought of being a landlord as being a small business owner, not as passive income – because it is work to be successful at it (screening tenants, maintaining properties, etc.) In fact, most passive income requires a lot of work upfront. I receive royalties for my books, but I had to write them and now I do a lot of ongoing promotion. It’s not like the money just rolls in while I do nothing, you know?

    The other issue is rates of return. Given increasing longevity, you’d need to have enough savings to last for 60 years if you retired at 40. That’s a huge nut to accumulate after just 20 years of so of working, and a nearly impossible amount when rates of return are so low. And so much can change over 60 years!

    And then, what would you do all day for 60 years? That’s a long time to be inactive.

    I’m thinking about what I want to do for work, so this post is interesting.

    • retirebyforty August 16, 2014, 3:55 pm

      That’s what I heard about books too. You have to keep promoting it. Well, I think working part time on something you like is the way to go. You don’t have to retire from that kind of lifestyle.

  • Carlo August 16, 2014, 11:15 am

    Another source of passive income is a pension. I started work at age 20 and after 25 years “retired” at age 45 with a pension of about 70K plus medical benefits paid by employer for life. I still work and contribute to my next pension as well as dividend stocks that I hope wil create about 25-40K a year. Once that happens I’m done with work! Nice article . . . prosper!

    • retirebyforty August 16, 2014, 4:01 pm

      Pensions are great. Mrs. RB40 can get one if she keeps working for 10-15 more years. It’s becoming rare, though.

  • Dave August 17, 2014, 7:14 pm

    Two more sources of passive income:

    1. Pension
    2. Reduced consumption. “A penny saved is a penny earned.”

    (I thought I posted this before, but it may not have gone through.)

  • JimW August 21, 2014, 5:53 pm

    What about an oil well, specifically a direct participation scheme? Lots of tax write-offs. Technically ‘active income’, although all you’re doing is ponying up the cash ($100k typically). Although you have to be an ‘acreditted investor’ by IRS rules ($1M in savings or $200k in income), you can start one of these whilst earning, if eligible, and then just keep pocketing the oil revenue in retirement until the well dries up or the partnership decides to sell it.


  • J. Doe August 25, 2014, 11:30 am

    Being a landlord isn’t for everybody, of course. I personally like real estate and use it as my main passive income source. I shouldn’t say ‘passive’, thou, as I see it as a side business that takes time, effort, constant care, and many people don’t want the hustle. It’s paying for about 40% of our current living expenses, and we plan on buying more properties in the following years (we have 3 rentals now). We both have full time jobs as well (my spouse and I), but enjoy having this business and we are also dabbling in buying fixer-uppers and reselling them for a profit. Our investments are not really ‘passive’ but I enjoy it a lot more than work and I could do this for a lifetime with excitement. I think investors should definitely think about what they enjoy and go with that. If you really hate being a landlord, maybe it’s better to invest in REIs, or something else other than real estate. I enjoy keeping detailed spread sheets about each of my rentals, folders of documents and notebooks of records of bills,I enjoy crunching the numbers, making projections, and even enjoy the occasional hole in my stomach when my agent buys a house without me having set foot in it.

Leave a Comment