The idea of trying to beat the market never loses its allure, any more than dreaming about winning the lottery goes out of fashion.

When does it become a problem? When you start counting too much on it – “I’ll be able to pay off my credit card bills when I win the lottery”. Or when you’ve dropped half of your week’s disposable income on scratch-and-win cards or MegaMillions tickets.

Wall Street works the same way. A surprising number of responsible investment managers are hoping they’ll beat the investment lottery. Instead of going to the 7-Eleven for Quick Picks, they’re betting on glamorous, get-rich-quick hedge funds. What they should be doing is putting more money in slow, deliberate private equity funds.

To see how this works, look at your local pension funds. Pension funds manage money for police, firefighters or teachers, but they’re not cuddly Main Street denizens. Instead, pension funds are run by sophisticated managers who get their pick of Wall Street’s finest wares: pension funds can take that retiree money and put it in stocks, bonds, commodities, farmland, hedge funds or private equity funds.

This is where the lottery mentality comes in.

Hedge funds are the often-glamorous pools of money run by former bank employees who like to take a lot of risk on stocks, bonds, commodities, oil field or anything else you can back with a dollar and a dream. Pension-fund managers agree to pay the hedge-fund managers high fees: 2% of all the assets they manage and 20% of any profits. In return, the pension funds think they’re going to be printing money: returns of 10% or more.

John Paulson, founder of New York-based hedge fund Paulson & Co. Paulson made billions of dollars during the financial crisis, but has not been able to reprise that success since. Photograph: Reuters Photographer / Reuters/REUTERS

So, instead of fat returns, a lot of pension funds are sitting in a sad sea of scratch-off tickets. The Los Angeles Fire and Police Pensions invested 4% of its assets in hedge funds. Its returns? Only 2%. The rich price for that underwhelming result: fees to hedge funds are 17% of all the fees that the pension fund pays.

South Carolina’s state pension fund forked out nearly $1.2 bn in fees over the last five years to hedge funds. The awkward result: Instead of getting a Rolls Royce for its money, the state’s treasurer has raged. “we’re driving off in a (Ford) Pinto.”

The California Public Employees Retirement System, or Calpers, is leading the charge. Calpers was paying $135m in fees to hedge funds with middling performance: only a 7.1% boost in returns every year. To put that in perspective, the S&P 500 index has returned 17% so far this year.

Calpers has joined the Los Angeles fire and police employees’ union to abandon hedge funds.

While Calpers, Los Angeles and South Carolina sour on hedge funds, too many other pension funds still have that lottery mentality.

Which is why they optimistically scatter their assets amongst a bunch of hedge funds, hoping that at least one of them will turn out to be run by a Mr. or Ms. Infallible, who will be worth the 2% annual management fee and the 20% of the profits he charges.

The bad news? There’s no Mr.Infallible, and he probably won’t be worth those high fees.

Put another way: you can’t outsmart the market. You can only respond to it really well. They can only respond to it really well.

One of those ways is private equity funds.

Known as “PE funds” on Wall Street, they’re more familiar to most of us as the stars of the RJR Nabisco buyout and Barbarians at the Gate; the unobtrusive power players who own SeaWorld, or Burger King.

CEO of Kohlberg Kravis Roberts & Co Henry Kravis last month. KKR was the firm at the center of the RJR Nabisco buyout in the 1980s, and Kravis is still seen as the king of the industry. Photograph: CARLO ALLEGRI/REUTERS

Private equity funds also charge high fees, with that same “2 and 20” idea. But private equity has done better for Calpers in terms of returns (18.4% in the most recent fiscal year ) than hedge funds have done for most investors.

There’s a bigger difference at work here, though. What a private equity manager is doing is buying a company and reorganizing and restructuring it – transforming it, with borrowed capital.

Private equity funds are far from perfect. Their companies are often saddled with debt even as the funds make rich profits, and the necessity of cost-cutting, including job cuts, have drawn the ire of many. But for pension-fund managers looking for a way to make a profit, private equity funds are a handy standby.

Another benefit: private equity funds are long-term investors. It’s not unusual for a PE firm to stay invested in a company between three and 10 years – a far cry from the days and months that many hedge funds devote to a single investment.

Why does long-term investment matter? Because pensions are best managed for years. Previous financial booms and busts have shown that what goes up must come down. The hottest lottery ticket now will not pay for anyone’s retirement in the next crash. It’s only surprising that it took pension-fund managers this look to realize that.

And as most have us ordinary investors have long since realized, every dollar that we have to pay in fees is a dollar that isn’t going to around to help produce the investment returns we’ll need to finance that retirement either – or at least, not our retirement.

It’s long past time that the pension fund managers, too, woke up to this fact and started viewing hedge funds as precisely what they are: a deluxe, platinum-lined retirement plan for the Masters of the Universe who run them rather than a surefire way to preserve the retirement income of the plan’s beneficiaries.