Flying Colour Ltd. | Photodisc | Getty Images

Over the past 10 years, investors have pulled out a third of total assets under management, or $543 billion, from actively run equity funds globally and poured $660 billion into passive ones that don’t require managers to pick stocks, but instead track a benchmark index, HSBC said, citing data from EPFR, a Boston-based firm that tracks fund flows. (Read More. Investors Exit Hedge Funds After Another Bad Month .)

The assets under management (AUM) of exchange-traded funds climbed more than 14 times over the same period, from 2002 to 2012, to $1.5 trillion from $105 billion, according to data from BlackRock.

“Retail investors and regulators have been made very nervous by the big swings in stock prices,” said Garry Evans, HSBC’s global head of equity strategy in a report released late last week. “The high volatility also explains the big flows into passive funds in recent years: volatility makes it hard for active or thematic fund managers to perform well.”

Retail investors have balked at putting their money back into equities after being burnt by wild market swings. For instance, the S&P 500 index plunged 57 percent over 17 months from an all-time high hit in October 2007. It then rebounded from March 2009 and has more than doubled in value since then.

Investors who had their savings in mutual funds and saw these savings evaporate in the bear run have also realized that these funds do not outperform stock indices, HSBC said.

In fact, only 40 percent of large-cap U.S. funds and 38 percent of small-cap funds outperformed their benchmarks, according to data from ratings agency Standard & Poors.

The high fees that active managers command — usually 80 to 150 basis points for an U.S. equity fund compared to 20 to 30 basis points for the equivalent index fund — are harder to justify now, Evans said.

Equities Still in Favor

That does not mean that investors will abandon equities because over the long term, stocks still do better than bond and cash investments, Evans added.

Investors will just put most of their money — as high as 80 percent of their equity allocation — into passive instruments, HSBC forecasts. The rest was still expected to go into actively managed equity funds, particularly dividend funds.

“There have been almost $80 billion of flows into dividend funds (globally) over this time, making it the most popular of the themes tracked by EPFR,” Evans said. “With cash yielding zero and top-quality government bonds little more than 1.5 percent, it is unsurprising that investors are scrambling to pick up yield.”

To make up for lost ground, fund managers need to develop funds that provide high returns with low volatility, or different combinations of risk and return, according to HSBC. There are signs that the industry has started to do just that, with the multi-asset fund, a fast-growing type of product that aims for returns in a range of assets with a targeted level of volatility.

Another big challenge facing the fund industry is the decline of hedge funds. which have underperformed the S&P over the past few years. In the first half of the year, the Bank of America/Merrill Lynch global diversified hedge fund composite index returned just 1.3 percent. well below the S&P 500’s 8.3 percent gain.

“Hedge funds have struggled to perform recently,” Evans said. “Hedge funds tend to do best in absolute terms during economic expansions and equity bull markets, such as 2003-7, and in relative terms during market collapses like the Global Financial Crisis of 2007-9. But they may struggle during the trendless, risk on-risk off type of market we have seen recently.”

They will be hard-pressed in justifying their “2-20” fees, Evans added. This is the flat fee of 2 percent of asset plus 20 percent of the gains over some base return or “hurdle rate.

As returns lagged that of the S&P 500, hedge funds have shrunk. In the third quarter, $15.56 billion flowed out of the $2.52 trillion industry, according to eVestment, a New York-based hedge fund research firm. (Read More. Hedge Fund Returns Worsen—Is ‘Enormous Unraveling’ Near? )

Funds that have shut and returned money to investors in the past couple of years include billionaire investor George Soros’ firm. John Arnold’s Centaurus Energy Master Fund. and Voras Hedge Fund, run by former Morgan Stanley co-president Zoe Cruz.

—By CNBC’s Jean Chua.