A good employee is hard to find - even in the age of Facebook and LinkedIn. Vetting for the top spot can be especially tough. Why reference checks fail, and how they can be fixed.

By Vickie Elmer, contributor

FORTUNE -- A few years ago several senior execs at Fortune 500 companies set up an online business. They hired a major executive-search firm to find a CEO for their startup, and after months of looking, the board chose its man. But the board didn't check his references or talk to his former co-workers. If they had, they would have discovered that at his last job he had had a habit of looking at pornography in the office.

After an internal investigation by his current employer turned up X-rated material on his computer, he was fired; he sued for wrongful termination, and the litigation cost the company time and money (though the suit was ultimately dismissed). An HR manager who was at the company at the time the exec was fired didn't want to be identified when contacted for comment out of fear of reprisal by the company -- a perfect example of why checking references often proves so daunting.

A good reference check is difficult to conduct. Fear of litigation too often keeps everyone overly quiet. Gerald Maatman Jr., partner at the business law practice Seyfarth Shaw, says that an ex-employee may charge his former employer with defamation or retaliation for exposing unseemly habits about him. It's safer to leave no paper trail and send nothing. He recommends that candidates fill out an authorization form that waives any claims based on reference checks.

MORE: How to win the tech talent war

Companies find other ways around tough reference policies by talking to people the potential hire used to work with but may not currently: former direct reports, peers, and supervisors. Susanna Hunter, who has more than 20 years of HR experience and is lead research analyst with McLean & Co., says that she routinely hears about managers hired in a hurry. References aren't called because those in charge trust their gut rather than hard data and research. Krista Canfield, a LinkedIn (LNKD) spokeswoman, has noticed an alternative. She says that she sees more companies that have a policy of contacting at least 10 former colleagues to vet a prospect, even if the potential hire has provided only a few references.

Social media, paradoxically, poses an entirely different sort of dilemma in the search for good hires: It reveals too much. While combing over a candidate's profiles or status updates, an HR manager may inadvertently open the company up to charges of discrimination based on the religion or sexual orientation of a candidate. Despite recent flaps over potential employers' requests for a candidate's Facebook password, most companies would never take that route. Instead, premium users of LinkedIn might search for his or her former colleagues within a narrow band of years to broaden the reference pool. Likewise, the company SkillSurvey uses e-mail and an aggregation of anonymous online comments to collect a broader picture of potential candidates.

Then there is perhaps the best approach, which is also the simplest: Ask great questions. Elena Bajic, chief executive of Ivy Exec, a member-based career site and recruiting firm, always asks references if they would hire a candidate again. She listens for a pause in their responses, which might mean she'll want more references, for still more evaluation.

This story is from the September 24, 2012 issue of Fortune.