Numerous AI Model Safety Assessments Show Considerable Shortcomings

A recent report suggests that current methodologies for ensuring Artificial Intelligence (AI) safety and reliability may be inadequate amid growing calls for better scrutiny.

The surge in generative AI technologies, capable of producing text, images, music, and videos, has heightened concerns over their error-prone and unpredictable behavior. Consequently, entities ranging from governmental organizations to major technology companies are proposing advanced benchmarks for evaluating the safety of these AI models.

In an effort to address these concerns, Scale AI, a startup, launched a laboratory last year aimed at assessing the alignment of models with established safety protocols. This initiative was followed by the introduction of risk assessment tools by NIST and the U.K. AI Safety Institute this month.

However, questions persist about the effectiveness of these testing techniques and methodologies.

The Ada Lovelace Institute (ALI), based in the UK and focusing on AI research, engaged in a study that involved discussions with experts from academia, civil organizations, and AI model developers. It also reviewed the latest research on AI safety evaluations. The investigations revealed that while existing assessments offer some value, they are not comprehensive, can be manipulated, and fail to predict real-world behavior of the models accurately.

Elliot Jones, a senior researcher with ALI and co-author of the study, emphasized to TechCrunch the expectation that products, ranging from smartphones to cars, undergo rigorous testing for safety before being released. The research aimed to highlight the limitations in current AI safety evaluation approaches, explore their application in policymaking and regulation, and identify potential improvements.

Evaluating Models: Benchmarks and Red Teaming

To grasp the landscape of AI model risks and the effectiveness of evaluation processes, the study’s authors analyzed academic literature and conducted interviews with 16 AI experts, including personnel from tech firms at the forefront of developing generative AI technologies.

This analysis uncovered a divided industry opinion concerning the most suitable evaluation methods and classifications for AI models.

Some methodologies focus solely on how well models perform against laboratory benchmarks without considering their potential real-world impacts. Also, evaluations that are meant for research purposes are sometimes applied to production models by developers, an approach that raises concerns about its appropriateness.

According to the study’s insights, benchmarks might not reliably indicate a model’s capabilities and could even inflate performance evaluations if the model has previously been trained on the same data set used in testing.

Mahi Hardalupas, an ALI researcher and one of the study’s co-authors, shared with TechCrunch that concerns include developers potentially manipulating benchmarks by training models on test datasets or selecting specific evaluations that favor their model, which might not truly reflect its safety or reliability.

The study also critiques the practice of red teaming, where models are deliberately challenged to expose weaknesses. While companies like OpenAI and Anthropic employ red teaming, the lack of standard practices makes its effectiveness questionable.

Seeking Solutions

According to the report, a rush to market and a reluctance to undertake thorough testing are major hurdles to improving AI assessments.

One expert voiced concerns over the rapid pace at which foundation models are being developed, suggesting that this trend hampers efforts to ensure they are adequately evaluated for safety and reliability.

Despite these challenges, Hardalupas remains optimistic that progress can be made through greater involvement by public-sector agencies in the evaluation process.

He advocates for a regulatory framework that clearly defines evaluation objectives, alongside efforts by the evaluation community to be candid about the limitations and possibilities of their methodologies.

Jones proposes developing specific evaluations that not only test model responses but also consider the potential impact on different user demographics and how malicious attacks could bypass existing safeguards.

However, it’s recognized that no model can be deemed entirely safe. Hardalupas notes that “safety” is a multifaceted issue, contingent upon how a model is used, its accessibility, and the robustness of its safeguards. Evaluations can highlight possible risks, but they are not definitive proofs of safety.