Agile AI Regulations for Risk Mitigation
September 17th, 2024 by Jana Sukkarieh.
A part of a Discussion Salon at World CC AI Contracting Week 2024 on September 17th, 2024 titled:
AI Adventures: Learning from Missteps and “Mastering Success”.
Introduction
The following discussion examines key points of AI regulation in the EU and the US. We also consider guidelines for navigating these regulations and fostering success in solving AI problems.
(to accompany the presentation)
Key Points in AI Regulations
It is important to note that AI algorithms themselves are not inherently harmful. In many cases, negative outcomes could be avoided by educating users and managers; especially in high stake industries such as health, legal or financial.
In general, though, the potential for negative outcomes depends on how an AI system or product is designed, developed or deployed.
In many cases, it is clear what went wrong, for example, training on biased data but as AI is getting more sophisticated, pervasive and integrated into various aspects of society, the concerns about risk of harm, whether concerns from individuals, unions or enterprises, increase. Keep in mind that in some cases, even top AI experts may be uncertain about the potential harm of some AI models.
These concerns predate LLM-based Generative AI. See some concerns listed in tables 3 and 4 on slide 3 of the accompanying presentation.
Consequently, the need for well-designed and effective regulations, legal frameworks or guidelines, worldwide, is essential; hence, we see the development of frameworks such as the EU AI act, and UK’s AI regulatory principles. In the US, there are some rules or acts about banning fake reviews and testimonials, the defiance act punishing the creation of non-consensual deep fake porn. There is also some bills, a proposed piece of legislation, in California, called senate bill 1047 for safe and secure AI models (https://legiscan.com/CA/text/SB1047/id/2919384) or a bill in Colorado for consumer protection while interacting with AI systems (https://leg.colorado.gov/bills/sb24-205) .
There is also a set of principles or guidelines such as the US AI Bill of rights.
The EU AI act is more comprehensive and a legally binding piece of legislation which focuses on regulating AI development and applications based on risk levels. As we saw earlier in the discussion (slide 2 of the presentation), the EU AI act defines 4 levels of risk in ascending order of “harm level”, namely, minimal, limited, high-risk and unacceptable. AI systems with unacceptable risk or harm levels, such as systems that target the elderly or children, are prohibited (full stop). Minimal risk systems, such as weather prediction tools, are ignored while low/limited and high-risk systems are regulated and have different measures of risk mitigation. I should not say different measures but for high-risk systems the requirements are stricter, for example additional types of evaluation for high-risk systems. It is like the set of measures for limited risk is a subset of the set of measures for high-risk systems. The measures include traceability, security, explainability, accuracy and robustness (as you can see in the first table on slide 3).
The US AI bill of rights is a more principle-based framework which focuses on protecting individual rights in the context of AI. The measures suggested to protect the individual rights include equity, accessibility, safety and plain language information, as one can see in Table 2 on slide 3.
We all agree that AI communities and users, in general, have been talking about these principles and measures, even without a particular legal framework or regulations.
In general, to navigate AI regulations, legal frameworks or guidelines and better understand your (regulatory) obligations, consider questions that fall under the following 4 categories.
General AI regulation
Asking about the scope or definition of AI in the regulation or framework is paramount. What is being regulated? Because this has major implications on who is liable when something goes wrong? Is it the developers, researchers, deployers, and/or providers?
Imagine the implications it has, for example, for your hiring process. Imagine you are hiring a developer and asking them to sign that if anything goes wrong then it is their responsibility! No one will accept signing up or working for you.
In the EU AI act, everyone, in one capacity or another of involvement of the limited or high-risk systems is mentioned while the SB 1047 in California, developers, in particular, seem to be most responsible for what the bill calls Frontier AI.
Another question that falls under the general AI regulation is what are the criteria or how do you determine risk level or harm? As we saw in the discussion slide earlier, there might be some disagreement on what constitutes a risk level. Take for example, the weather prediction tools, common sense implies it is of minimal risk level due to the unpredictability of the weather. Can you imagine, in the UK someone suing the meteorology department when the 4 seasons could materialize in the same day? Saying that, some global lawsuits against some meteorology department were successful. In such cases, the judge might have considered the harm of the tool as limited or high-risk, especially if it causes someone’s death and not minimal (making assumptions here – just to give a real-life example).
A 3rd question that falls under general AI regulation is about the role humans must play. Do they oversee everything at every stage? What is their required AI literacy?
Data Governance
Questions under data governance could include how to ensure my data is relevant and representative and not erroneous? Note that in some cases, the EU AI act mention data completeness under general AI regulation.
Another question under data governance that one should ask is how to strike a balance between the need to train AI and privacy compliance / concerns? What about cross border data flows?
Ethical Considerations
This category focuses on the ethical principles and guidelines that must be followed. Or how to address potential biases, discrimination or societal impacts of AI?
Enforcement and Compliance
Under enforcement and compliance, one should ask about what specific regulations apply to my AI system? How can I ensure compliance? And of course, as mentioned earlier and one cannot emphasize this point enough, who or which roles are liable?
Of course, it goes without saying that international cooperation is crucial under any of the above categories, and a recent development underscores this point: a global AI treaty has been opened for signature. As the Council of Europe announced, this is the "first-ever international legally binding treaty aimed at ensuring that the use of AI systems is fully consistent with human rights, democracy, and the rule of law." (
I also think these regulations are designed to be agile and will likely evolve. Therefore, when implementing or deploying an AI system, consider whether you can anticipate these changes to stay ahead of the curve.
Last but not least, it is important to mention that the goals of these frameworks are not to stifle innovation. The intention is to harness the power of AI while mitigating risks or minimizing AI potential harm.
N.B. The EU AI act does not regulate AI systems deployed in wars or conflicts, as military applications are excluded.
Concluding remark for this section
To conclude this section, it is worth saying that while AI legal frameworks, principles, and guidelines are a crucial step forward, there's a pressing need for clearer liability frameworks. The question of 'who is liable?' , I think, remains unresolved. Is it the developer, researcher, provider, or the company that adopted the system? The EU AI Act and SB 1047 offer some guidance, but the issue is complex and likely case-by-case. As a researcher and developer, I'm concerned about the uncertainty surrounding legal compliance. It's unclear when my work might cross a legal boundary. The prospect of needing legal counsel before every project is daunting. At the end of the day, regulations are really contracts that you have no option of not signing. You need to understand it carefully. Interesting enough already several researchers in the UK and US are using AI to help people navigate all these regulations about AI.
Success and Missteps
In AI, particularly when dealing with natural languages and/or reasoning, no task is ever truly solved. There is always room for refinement and advancement. While there is no foolproof recipe for solving any given problem, following some established guidelines and avoiding common missteps/pitfalls can significantly enhance your roadmap towards success in AI and/or in fulfilling your regulatory obligations.
George Pólya's influential work, How to Solve It (1945), while focused on mathematics, provides a framework applicable to a wide range of problems, including those in artificial intelligence. Pólya's four guidelines might seem intuitive. However, it's still worthwhile to examine them.
Understand the Problem
Put a plan
Implement the plan
Revise: Iterative Thinking
A fundamental principle outlined in the book is the importance of fully understanding the problem at hand before attempting a solution.
When approaching a problem in AI, the first crucial step is to determine if it truly necessitates an AI solution. Many challenges can be effectively addressed using traditional computer science algorithms.
In my career, I encountered a complex task involving a system developed by international teams and dealing with data in over 22 languages. Initially, this seemed like a daunting AI problem due to its language and domain-agnostic requirements. However, after careful analysis and experimentation, I discovered that a classic computer science algorithm, proposed in the 1960s, could solve the issue without relying on knowledge bases or training data.
This experience underscores the importance of discerning between AI-specific problems and those that can be resolved using established computer science techniques.
Once you've determined that an AI solution is indeed necessary, the choice of your solution often hinges on a deeper understanding of your specific data.
When working on automatic content scoring for short essays, I initially conceptualized it as a paraphrasing problem. However, a closer examination of the data revealed that the task was one of textual entailment. This distinction is crucial, as paraphrasing is bidirectional, while textual entailment is unidirectional.
Even after identifying the correct task, which in this case is a challenging AI problem, it's essential to conduct a thorough analysis of a subset of your data to gain a comprehensive understanding before proceeding to training or testing.
Remember, data-related challenges can arise in unexpected ways. Not only might you encounter incorrect or incomplete data due to human error, but you may also face the hurdle of having non-digital data in your organization.
You do the above i.e. understanding your problem and a subset of your data to draw a tentative plan. The second step in Polya’s guidelines.
Third, a common mistake is to believe that the initial plan or implementation will yield optimal results. The reality is that problem-solving is an iterative process. As you gain a deeper understanding of your problem and data, revisit your plan and explore ways to enhance its effectiveness.
As Polya suggests, it's essential to 'look back' at your work and ask yourself if there are opportunities for improvement.
In my projects, I adopt an iterative approach, continuously refining my solutions. Even groundbreaking AI models like LLM-based GenAI underwent multiple iterations and benefited from extensive human feedback to reach their current state. Unlike well-defined mathematical systems, the real world is rife with uncertainties and biases, both in data and human cognition. This makes it unlikely that an initial solution will be optimal.
At every stage, critical evaluation is essential. Ask yourself: have I overcomplicated the solution? Have I handled sensitive data appropriately? Have I conducted sufficient testing? Beyond quantitative metrics like accuracy, precision, and recall, error analysis is crucial. Identify where and why your AI system fails, focusing on specific data points that have led to negative outcomes.
The final guideline that I add to Polya’s four guidelines is educate.
Educate: Educate users, management, and companies.
A common misstep occurs when changes are made to the task or domain without involving the AI team. In one of our projects, a model was developed for a specific task and domain. However, domain experts later modified the task without consulting the AI team. This led to a mismatch between the model's original purpose and the revised requirements.
To prevent such issues, it's essential to establish clear processes and guidelines. Similar to software development practices, where requirements are finalized within a specific timeframe, AI projects should also have well-defined boundaries for changes. By adhering to these guidelines, we can avoid situations where problem-solvers are unfairly blamed for not addressing a task that was modified after the fact.
Most importantly, manage expectations. The attitudes of managers, companies, and users toward AI can be summarized in the AI Mindset framework below.
AI Mindset Framework: Manage Expectations
a) Trust in AI: While some individuals or organizations may exhibit complete trust or distrust in AI, a balanced approach is generally recommended. It's essential to maintain a positive attitude while applying critical thinking and acknowledging the limitations of AI.
b) Capability of AI: Expectations regarding AI's capabilities often range from skepticism to exaggerated optimism (like science fiction capabilities). While AI has made significant advancements, it's crucial to ground expectations in reality. Business units may not fully understand the limitations of AI or computer science, often requesting solutions that are beyond the realm of current technology.
c) Anxiety about AI: Concerns about AI can vary widely, from minimal anxiety to outright panic. It's important to remember that AI tools are designed to assist humans, not replace them. AI systems require clear instructions and guidance, as they are fundamentally computer programs and do not possess autonomous agency or the ability to rule the world (this is a controversial issue among researchers and AI experts). As with any technological innovation, artificial intelligence can be deployed for beneficial or detrimental purposes. However, that is a separate consideration (hence, the regulations discussed above).
Promote education and collaboration regarding AI. Encourage open dialogue and knowledge-sharing among users, management, and technical teams to facilitate informed decision-making. Many AI projects fail due to communication breakdowns and differing perspectives among stakeholders. To mitigate these risks, foster a collaborative environment where all parties are willing to understand and accommodate each other's viewpoints.
These principles apply equally to AI system providers when investigating systemic risks and assessing compliance with global regulations. By adopting the guidelines, fostering a culture of education, collaboration, and mutual understanding, organizations can avoid common missteps, and enhance the successful adoption and deployment of AI solutions.