Category: Technical

Potential Source of Harm: Model Mismatch

Updated March 14, 2024

 

Nature of Harm

Model mismatch refers to the risks that an AI model (a) demonstrates good results when tested but does not perform as well when used on real-world data or (b) is otherwise used in a context for which it is not appropriate or intended. This is related to the separate risk of model drift.

 

For example, Google DeepMind built an application for diagnosis of diabetic retinopathy (an eye disorder) that was shut down in 2020 because of failure on real-world data.

 

Regulatory and Governance Solutions

China's regulations on various areas of AI -- including algorithmic recommendation services, deepfakes and generative AI more generally -- are the most specific in the world regarding model performance. It is likely that similarly detailed regulation will emerge in other countries.

 

The EU AI Act will impose requirements relevant to model mismatch on "high-risk" AI systems. Among other things, the obligations of the AI Act include (i) risk management throughout the lifecycle of the AI system, (ii) requirements on training, test and validation data sets and (iii) technical documentation. The details of these requirements will become clearer one the AI Act is adopted in 2024.

 

The US Executive Order on AI requires various US government agency to take actions related to model safety.

Practices regarding model quality and appropriateness are also recommended in evolving AI system governance initatives.

 

See our Regulatory & Governance Solutions page for details on the above. These requirements and governance practices will evolve substantially (including legislation in other countries) in the coming months and years. 

 

Technical Solutions

Development of AI models that work properly in a given context is of course at the core of AI research and development. So technical solutions are extensive, constantly evolving, and context-specific. They cannot be summarized effectively here.

 

Government Entities

Chinese regulators, including the Cyberspace Administration of China (CAC), already have significant authority for regulation of AI models, especially under the regulations mentioned above. EU and EU member state regulators will eventually have analogous authority under the EU AI Act; and the US has given authority to various government agencies.

 

Government AI research institutions may also play a significant role in developing solutions for AI model mismatch.

 

Private Entities

Many private companies and other entities are working on improved AI models and applications using them. Detailing this work is beyond the scope of Saihub.info, but we may later add more detailed summaries of this work.