We are in a post LeetCode Era

Software ate the world, now AI is eating software. What does this mean for interviews and algorithmic problem solving?

Integrating generative AI into developer IDEs is moving us into a new paradigm of programming. Words alone have a hard time describing how much AI code completion changes the game for LeetCode type programs. So here is a video reenactment of my experience doing a LeetCode problem with Replit’s AI infused IDE on the sub-string problem.

Before we can even start to think about the problem, the AI has provided a solution that beats the majority of users on runtime and memory performance metrics. Also we didn’t give the AI the actual problem statement we were solving, only the name of the function and its signature! So for interviewing purposes LeetCode seems to be dead. Interviews cannot be built on these algorithmic problems like they have in the past.

While some companies ban AI tools in interviews, its use is becoming the new norm. AI pairs will only get more ingrained into developer workflows. Fighting that is like fighting against using IDEs. Developers should be allowed to use the tools they would have in their day to day work in interviews.

So what do interviews look like if we can’t rely on algorithmic problems? Fortunately AI also greatly improves developer productivity. This study by Microsoft and MIT found approximately a 50% increase. (Note Microsoft may be a biased source, However I have been part of private studies that found similar productivity gains. So their numbers are consistent with my experience). Based on these findings we can expect the speed of generating code goes WAY up. Which means we can now think about doing much more complex and interesting problems in interviews. Problems that weren’t reasonable to ask are now fair game, For example write a complete web app from scratch.

Before generative AI, building a web app in an interview meant spending half the time writing boiler plate code. This code was necessary to get a working solution but not interesting from a problem solving perspective. But with AI systems spitting out boiler plate as well as more complex code, candidates can crank out a complete solution within interview time limits.

The implications of this are exciting. For many hiring managers, it becomes apparent whether a new team member will be successful within the first few days on the job. And usually this is evidenced by their first or second ticket. The on the job indication is a lot more predictive of overall job success vs solving LeetCode algorithm problems. From a candidate perspective it is more fulfilling to work on something that could have benefit to end users (vs brain teaser questions that only serve the hiring manager). Lastly when the interviewer is implementing an actual application they have more opportunity to learn something that they can apply elsewhere because the problem space is much wider. In general this move toward full solution interviews driven by AI is better for both the hiring managers and the candidates. It reflects a truer measure of a software engineer’s success, not defined by solving puzzles, but by providing working solutions.