This originally appeared in Fast Company back in 2015. Much has changed since its original publication. I hope to write future posts looking back on this project and how it worked out.

Long ago, the business world settled on a conventional hiring process. And while some companies are experimenting with their own approaches, the standard elements – a cover letter, resume, and interview – still prevail in most places. The thing is, they don’t necessarily serve the end goal: hiring the most qualified candidate.

Despite its convenience, the traditional hiring process has its shortcomings. For one thing, resumes and cover letters only convey self-reported data, so candidates can and do bend their experiences to fit the job description. Quirky interview questions can sometimes lighten the mood and get a candidate to give more revealing answers, but they don’t necessarily identify the most qualified candidate. For instance, Airbnb has been said to ask applicants, “What would you do if you were the one survivor in a plane crash?” It’s a cool question, but (hopefully) not a predictor of success at Airbnb.

Here at Compose, we realized that our own hiring process was not only less than completely effective, it was also–like most–susceptible to bias. So we set out to make it more transparent, less subjective, and less biased–all while decreasing the time we spent finding the ideal hire. Here’s how we did it.

Getting Rid of Resumes

We decided to ask new candidates to submit work samples instead of the usual resume and cover letter. We thought it would help us better identify the candidates who were worth interviewing. Since implementing the switch, we’ve never asked for more than two to four hours of coding or writing. If the process was too arduous, we’d need to review boatloads of materials, and it might turn off otherwise qualified candidates.

Soon, though, those of us who began to evaluate work samples realized our methods were haphazard. We scored responses inconsistently, and candidates were still subject to our own biases, despite our best efforts at remaining neutral.

We realized that our own hiring process was not only less than completely effective, it was also susceptible to bias. So we decided we needed to take our hiring process one step further, and we built an extension to our internal project-tracking tool that would help us better manage job candidates. It’s now a standalone app that moves applicants through our hiring pipeline as efficiently as possible. We use it to draft job descriptions, set hiring criteria, create work samples, and invite other Compose staff to grade completed work samples, giving us a complete view of our entire hiring process all within one portal.

The New Pipeline

When a candidate applies for a job, our hiring app generates a unique URL with a series of sample tasks and space to enter responses. We encourage applicants to take their time, answer the questions completely, and spend as much effort as they would on a real-world problem.

Submissions are returned anonymized. Our staff then evaluates completed materials solely on their merit, according to predefined criteria. Each work-sample response has a series of checkboxes describing the attributes we’re looking for in a given position.

For instance, we ask marketing candidates to write a sample blog post, and our criteria include things like:

  • Written in a compelling way that our audience will appreciate
  • Has an authoritative voice
  • Clear and engaging

For a developer role, we ask candidates to submit a code sample, and the criteria includes:

  • Code compiles
  • Can communicate ideas in clean, working code

As submissions roll in and we grade responses, the top performers float to the top of the stack. Then we invite them for in-depth, group interviews. The final scores of work samples are weighted based on the criteria that we created for the role. There’s also a separate, subjective score from one to 10 that evaluators can add for their own reference. They typically use this score when they think they’ve found someone amazing, but the criteria score might say otherwise.

The hiring process can never be perfect. In our case, it’s still imperfect. But by addressing each of these five areas, we’re getting closer to hiring more effectively, efficiently, and without bias:

  1. First impressions and stereotypes: Biases take root just about instantaneously. Name, email address, education, country of origin, gender, and other details can cause hiring managers to make subconscious judgments–which can often lead to making the wrong decision. Before you pass any interview responses to graders, take out all personal identifiers.

  2. Performance bias: Good or bad, resumes will lead you to form opinions about the applicants’ skills and experiences that aren’t grounded in reliable evidence. Don’t ask for resumes. If you do accept them, don’t share them with those grading work samples. Evaluators can have strong reactions to even the most trivial details.

  3. Confirmation bias: This one’s similar to item one, but refers to a hiring manager’s expectations that might not be grounded in stereotypes. If, for whatever reason, you’re really excited about a candidate, that emotion will influence how you judge their abilities. They’ll appear qualified because you want them to be. When you send candidates’ answers to evaluators, don’t just anonymize the answers – present them non-sequentially too. In other words, if a grader reviews five responses, they might be from five different candidates at various stages of the hiring process.

  4. Self-selection: At the application stage, don’t ask questions like, “What superpower would you like to have?” or, “When the zombie apocalypse begins, what would you do first?” Questions that might seem offbeat and interesting are often freighted with regional and cultural associations that can disadvantage certain candidates. It’s one thing to look for a good fit for your company’s culture, and another to let what you think that consists of count for more than other, more meaningful qualifications.

  5. Anchoring and intuition: If a candidate hits a home run on the first question, you’ll be disappointed if their answer to the second is just a base hit. That’s another reason to shuffle responses. The scorer shouldn’t know which responses belong to which applicant.

This early sorting process may seem over the top, but hiring the wrong person can disrupt a well-running team and sap energy from the entire company. When you consider what it costs to hire, train, pay, and then replace the wrong employee, this process is a bargain. And in sidestepping a great deal of bias in its many forms, it has the added benefit of being the right thing to do.