Evaluate AI engineer and researcher portfolios effectively. Screen GitHub projects, papers, demos, and case studies to hire the best talent.
Introduction
Traditional resumes lie. A consultant's title says 'Lead' but they managed 0 people. A researcher's title says 'ML Engineer' but they've never shipped code to production.
In AI hiring, portfolios - GitHub repos, research papers, demos, and case studies - cut through the noise. But most hiring managers don't know how to evaluate them. A beautiful GitHub profile might hide sloppy engineering. A published paper might lack reproducibility. A flashy demo might paper over architectural flaws.
This guide teaches you how to assess AI talent through their actual work, what to look for, and what to ignore. By the end, you'll be able to screen candidates with confidence, even if you're not a PhD yourself.
GitHub Projects: Reading Between the Commits
GitHub is your primary signal for engineers. It shows how they code, collaborate, and ship. But not all GitHub activity is equal.
Published Papers: Evaluating Research Rigor
Research papers signal depth, but they can be misleading. A paper at a top-tier venue (NeurIPS, ICML, ICLR) is impressive; a paper at a marginal conference might not be. Here's how to evaluate.
Demos: Assessing Implementation Quality
A demo is the most revealing portfolio artifact. It shows if someone can take research or ideas and make them usable. But demos can be deceptive, a beautiful interface hides a fragile backend.
Case Studies: Real-World Impact Evidence
Case studies show if a candidate can solve real business problems, not just toy academic problems. They're the hardest to fake.
Weighting: What Matters Most
Different roles prioritize different portfolios:
- ML/Research Engineer hiring: 50% GitHub, 25% papers/blog posts, 15% demos, 10% case studies. Code quality is paramount.
- Full-stack/backend engineer hiring: 60% GitHub, 20% case studies, 15% demos, 5% papers. Production code matters most.
- Research scientist hiring: 40% papers, 30% case studies (applied research), 20% GitHub, 10% demos. Published work signals depth.
- Applied AI/ML engineer (LLM Ops, Evals, RAG): 50% case studies, 30% GitHub, 15% demos, 5% papers. Practical, shipped work matters most.
Common Mistakes in Portfolio Evaluation
- Obsessing over GitHub stars: Popularity ≠ quality. A candidate with 10K stars on a 5-year-old project might not ship anymore. Recent activity and maintenance matter more.
- Over-weighting papers: A PhD with 50 papers might be a poor product builder. Papers show research skill; shipped products show engineering skill. Weight based on role.
- Judging by tools and frameworks: Don't ding candidates for using different tech stacks (TensorFlow vs. PyTorch, AWS vs. GCP). They can learn. Judge by principles: clarity, robustness, and thoughtfulness.
- Assuming flashy demos = strong engineers: A beautiful UI hiding a messy backend is worse than an ugly demo with clean architecture. Always dig into the code.
Conclusion
Portfolios are your window into actual capability. GitHub shows coding discipline, papers show research rigor, demos show product thinking, and case studies show business impact. Don't rely on one signal. A candidate strong in GitHub but weak on case studies might be a good IC but not a lead. A candidate with papers and demos but no shipped product might be a researcher, not an engineer. Evaluate holistically, weight by role, and always dive deep in interviews. The best AI hires have portfolios that tell a coherent story: solving real problems, with thoughtful trade-offs, and shipped to actual users. Look for that.
