Code reviews, which assist developers improve code quality, are time-consuming. According to one survey, 50% of organizations devote two to five hours per week to them. Without enough personnel, code reviews can become overwhelming and divert developers' attention away from more critical tasks.
Harjot Gill believes that code reviews can be largely automated with artificial intelligence. He is the co-founder and CEO of CodeRabbit, an AI-powered code analysis tool that provides feedback.
Gill worked as the senior director of technology at Nutanix, a datacenter software firm, before launching CodeRabbit. He joined Nutanix when it bought his startup, Netsil, in March 2018. Gur Singh, CodeRabbit's other founder, previously oversaw development teams at the white-label healthcare payments platform Alegeus.
According to Gill, CodeRabbit's technology automates code reviews by applying "advanced AI reasoning" to "understand the intent" behind code and providing "actionable," "human-like" feedback to developers.
"Traditional static analysis tools and linters are rule-based and often generate high false-positive rates, while peer reviews are time-consuming and subjective," says Gill. "CodeRabbit, by contrast, is an AI-first platform."
These are big claims that include a lot of buzzwords. Unfortunately for CodeRabbit, anecdotal data indicates that AI-powered code reviews are inferior to human-in-the-loop ones.
In a blog post, Graphite's Greg Foster discusses internal experiences using OpenAI's GPT-4 for code reviews. While the model did detect some relevant information, such as small logical problems and spelling faults, it also produced a large number of false positives. According to Foster, even fine-tuning failed to significantly diminish them.
These are not revelations. A recent Stanford study discovered that programmers who use code-generation tools are more likely to introduce security flaws in the apps they create. Copyright is also a continuing challenge.
There are also logistical challenges with employing AI for code reviews. As Foster points out, more traditional code reviews need engineers to learn through sessions and interactions with their development colleagues. Offloading reviews jeopardizes this knowledge sharing.
Gill feels differently. "CodeRabbit's AI-first approach improves code quality and significantly reduces the manual effort required in the code review process," he told me.
Some people are purchasing the sales pitch. According to Gill, CodeRabbit's services are currently paid for by approximately 600 organizations, and the company is in pilot with "several" Fortune 500 businesses.
It has also made investments: CodeRabbit announced a $16 million Series A funding round led by CRV, with participation from Flex Capital and Engineering Capital. The additional funding, which brings the company's total investment to just under $20 million, will be used to expand CodeRabbit's 10-person sales and marketing responsibilities as well as product offerings, with an emphasis on improving its security vulnerability analysis capabilities.
“We’ll invest in deeper integrations with platforms like Jira and Slack, as well as AI-driven analytics and reporting tools,” Gill said, adding that Bay Area-based Code Rabbit is in the process of setting up a new office in Bangalore as it roughly doubles the size of the team. “The platform will also introduce advanced AI automation for dependency management, code refactoring, unit test generation and documentation generation.”