As an innovative firm, McCoy Russell has been at the forefront of patent law with its development and use of proprietary AI software via its software arm Ironcrow AI. Recently, Ironcrow has invested substantial efforts to create a specialized AI LLM Sandbox as a key tool for patent professionals.
Ironcrow is excited to announce a groundbreaking achievement in the field of AI/ML for Patent Law professionals – Ironcrow’s specialized AI LLM Sandbox has achieved a score above the 70% threshold required to pass the patent bar exam, using a test set of questions. While other researchers have developed tools to pass a state bar exam, none have attempted to pass the specialized patent bar exam administered by the USPTO.
This remarkable feat showcases the innovation by the Ironcrow and McCoy Russell partnership and the ability of the LLM Sandbox’s “Interrogate” feature to answer questions based on the knowledge of the patent procedure. The Sandbox can provide well-cited answers along with relevant excerpts from the MPEP, etc., to its users. This unique feature sets Ironcrow AI’s LLM Sandbox apart from other systems in the market.
To evaluate the performance of Ironcrow AI’s LLM Sandbox, a comparative study was conducted using the same set of Patent Bar Exam questions. The study compared Ironcrow’s LLM Sandbox against various other AI models. The results of this study shed light on the system’s capabilities and provide insights into its performance when compared to other systems and human performance. In short, no other model comes close to meeting its performance, or obtaining passing level proficiency relative to the USPTO Patent Bar Exam.
The passing score for the Patent Bar Exam is set at 70%. In 2022, the passing rate for humans was approximately 46.27%, assuming a symmetrical distribution of moderate deviation, which would place the average human score around 68%. With this in mind, the study aimed to establish a baseline performance for Ironcrow AI’s LLM Sandbox and other systems.
The findings of the study are as follows:
- Ironcrow AI’s LLM Sandbox emerges as the first AI system capable of achieving a passing score for the patent bar exam on a test set of patent bar exam questions. It achieved scores ranging from 78% to 80%, surpassing the average human performance of 68%. This remarkable achievement demonstrates the system’s deep and factually grounded understanding of the patent law and procedure and its ability to apply this understanding in answering complex questions.
- Ironcrow AI’s LLM Sandbox, even when utilizing the faster and more cost-effective chatGPT (GPT-3.5-turbo), showed impressive performance improvements over baseline chatGPT, almost reaching the level of GPT-4. This highlights the efficacy of Ironcrow AI’s approach to improve expertise in the domain of U.S. patent prosecution, without necessarily resorting to the largest and most expensive models.
- Other systems that claim to have been trained on an extensive database of patent knowledge did not exhibit significant improvements in performance compared to their underlying models. While many companies in the patent software space claim to be “experts in AI”, an unfortunate reality is that many of these companies simply act as middlemen between patent professionals and the underlying models, without adding value. This also underscores the importance of quantitatively substantiating claims made in regards to AI performance, something Ironcrow AI hopes to achieve by making these evaluation results publicly available (to access the test set of patent bar exam questions please inquire at the email below).
Ironcrow AI’s LLM Sandbox’s success in achieving a passing score for the Patent Bar Exam signifies a major milestone in the development of AI systems ability to assist patent professionals. Its ability to provide accurate answers with proper citations and excerpts from the MPEP sets a new standard in the field.
If you want to learn more about Ironcrow AI’s LLM Sandbox or the performance study email [email protected].
