Mind the Traps: What AI Risks to Watch Out For?

Artificial intelligence is becoming central to business strategy, but it also amplifies critical risks such as algorithmic bias, data privacy violations, and GenAI-driven data breaches that can mislead decisions, damage reputations, and trigger legal exposure. At the same time, as regulators, customers, and enterprises react with stricter rules, usage limits, and technical safeguards, it is increasingly clear that only robust AI governance and coordinated global collaboration will let organizations fully capture AI’s value while keeping these risks in check.

Written by

Nguyen Vu Quynh Trang

Published on

April 12, 2024

Mind the Traps: What AI Risks to Watch Out For?

Artificial Intelligence (AI) has undeniably moved to the forefront of many companies' strategies. However, global stakeholders, including business leaders, increasingly recognize the need for robust AI governance to fully harness this technology. Reflecting this awareness, 84% of surveyed executives cite responsible AI as a top management priority [1]. To make this ambition a reality, companies must first identify the potential risks associated with AI.

Why Does AI Bias Occur Despite Expectations of Neutrality?

AI systems are often expected to be neutral. However, they frequently reproduce and even amplify human bias because they inherit both our cognitive biases and the skewed patterns embedded in the training data. Instead of eliminating discrimination, biased algorithms can legitimize and scale it, especially when used to support high-stakes decisions.

For decades, computer programs in general, and AI in particular, have been praised as a counterweight to human bias. However, studies show a more troubling reality: humans tend to simplify information processing based on our own preferences and experiences, often bypassing strict logical reasoning.

When humans develop and train algorithms, these cognitive biases are projected onto AI systems. The problem becomes even more pronounced when the performance of such technology is benchmarked against human capabilities, encouraging designers to mirror existing human decisions instead of questioning them.

This issue is not new. As early as the 1900s, a British medical school was found guilty of discrimination after using a computer program to screen applicants for interviews. The program was designed to match human admissions decisions with up to 95% accuracy, effectively encoding historical judgment patterns. It was later determined that the system discriminated against women and applicants with non-European names [2].

Even if an AI system somehow manages to minimize the cognitive bias inherited from its human developers, biased outcomes remain inevitable when the input data is already skewed. Training data can be biased in several ways that reflect structural inequalities rather than objective truth:

Historical data that encodes existing human biases and social injustice.
Insufficient or unbalanced data that underrepresents certain groups or minorities.

When such information is used as input for AI, the system does not cure human bias; instead, it amplifies it at scale and can mislead decision-makers. A well-known example is Amazon's experimental AI recruiting tool, which was secretly scrapped after being found to discriminate against women. The tool was designed to automatically review and rate candidates' CVs and was trained on ten years of resumes submitted to the company, most of which came from male applicants due to the tech industry's male dominance. As a result, the algorithm learned to favor male candidates. It downgraded resumes from graduates of all-women colleges and penalized CVs that included terms such as "women's"[3].

A similar pattern of bias was discovered in a widely used algorithm in the US health system that predicts which patients need extra healthcare. The model was trained using healthcare spending as a proxy for medical needs. However, historical data show lower spending on black patients than on white patients with similar risk levels, due to unequal access and treatment. Because the algorithm learned from this distorted spending data, it systematically underestimated the needs of black patients. As a consequence, the number of black patients identified for extra care was reduced by more than half [4].

Data Privacy

Data plays an indispensable role in AI development and can ultimately determine how well an AI system performs. However, even as technology has made data collection and storage easier, organizations still struggle with how to access and use that data for AI training without compromising privacy.

These challenges should not be resolved at the expense of consumers' data privacy. Prioritizing AI performance over protecting personal data can severely damage a company's reputation, as most consumers oppose having their data accessed without consent, even if it enables personalization such as targeted marketing. In fact, 67% of US consumers object to such uses of their data. Beyond reputational harm, intruding on consumers' privacy also creates legal exposure, as people increasingly call for stricter safeguards. A survey by Prosper Insights & Analytics shows that over 63% of customers expect new legislation to prevent social media platforms and search engines from selling their data [5].

Failing to comply with data privacy rules has already resulted in lawsuits and operational restrictions for AI companies. In 2023, ChatGPT faced a temporary ban from the Italian data protection authority, which found that the company had unlawfully collected users' data and failed to implement an age-verification system [6]. In other news, Stability AI was sued by Getty Images for misusing over 12 million photos to train its Stable Diffusion AI image-generation system[6]. Similarly, artists worldwide have been taking AI companies to court, with one prominent case involving a group of seven artists filing a copyright lawsuit against Stability, Midjourney, DeviantArt, and Runway AI for misusing their works to train Generative AI systems[7].

Rising concerns about data privacy in AI have drawn the attention of legislative and regulatory authorities worldwide. After the temporary ban of ChatGPT in Italy in March 2023, which was later lifted, the Italian privacy authority continued to warn OpenAI over potential breaches of data protection rules. Data protection regulators in Germany, France, and Spain have launched their own investigations into ChatGPT's data handling practices following complaints about insufficient safeguards. These developments have accelerated research and policymaking around AI governance, with authorities in the European Union, South Korea, China, Canada, and Brazil taking leading roles in shaping AI-related regulations[8].

Data Breach

Driven by the rise of ChatGPT, Generative AI (Gen AI) applications have rapidly gained popularity among business users thanks to their ability to significantly boost productivity. Their potential spans a wide range of tasks, from artifact creation to code generation, and from bug identification to automated fixing.

However, without appropriate security measures, Gen AI applications can expose organizational vulnerabilities, particularly the risk of data breaches. A study on AI-related data breach incidents found that source code is the most frequently exposed category of sensitive data, with an average of 158 incidents per month for every 10,000 users. Other often-compromised data types include regulated data such as financial, healthcare, or other personally identifiable information (18 incidents per month per 10,000 users) and intellectual property (4 incidents per month per 10,000 users)[9].

In response, global companies have introduced various controls governing the use of Gen AI in the workplace. Samsung, for example, decided to prohibit the use of ChatGPT after an employee inadvertently uploaded sensitive code to the platform[10]. Other well-known companies, including Apple, Spotify, and Verizon, have also banned or restricted employees from using Gen AI tools at work, due to concerns about exposing company information and customer data[11].

At the industry level, highly regulated sectors such as financial services and healthcare tend to adopt a more conservative stance, with nearly 1 in 5 organizations completely banning the use of ChatGPT. By contrast, other sectors like technology are more flexible in allowing employees to use Gen AI at work. For instance, 1 in 4 technology companies apply Data Loss Prevention (DLP) measures to identify sensitive data uploaded to ChatGPT.

Another emerging control is real-time user coaching, which regularly reminds employees of company policies and the risks associated with Gen AI tools. This measure is currently implemented by 1 in 5 technology companies[9].

A Global Joint Effort

As AI systems grow increasingly complex, the potential risks associated with them are also expected to rise. This escalation has already prompted several tech leaders to call for a pause in AI development, warning that it could pose "profound risks to society"[12]. In this context, building robust AI governance practices emerges as a critical way to navigate and mitigate these risks.

However, establishing such governance frameworks demands substantial effort and investment, making global collaboration in AI research and development increasingly important. When multiple stakeholders work together toward this goal, as exemplified by the AI Alliance that brings together organizations such as Meta, IBM, and FPT, meaningful progress in AI governance research and development is made possible.

Frequently Asked Questions

How can AI drive business growth while creating new kinds of risks?

AI can streamline operations, create new products, and unlock revenue, but it also introduces novel risks at scale: bias, privacy violations, data leaks, and systemic failures. Leaders need a structured view of these risks and clear governance to capture upside while avoiding reputational, legal, and societal harm.

What is AI bias and why isn’t AI automatically more objective?

AI bias occurs when systems produce systematically unfair outcomes, often mirroring human prejudice. Algorithms learn from biased historical data and from design choices made by people. Instead of removing bias, AI can scale it, as seen in biased hiring, healthcare, and other high-stakes decisions.

Why is data so critical for AI and what privacy risks come with it?

AI performance depends on large volumes of high-quality data, but using sensitive or personal data raises serious privacy and regulatory risks. Misuse or non-consensual collection can trigger lawsuits, fines, bans, and reputational damage, so organizations must balance accuracy with strict data protection.

What data breach and security risks come with using Gen AI at work?

Generative AI tools can inadvertently expose source code, customer data, or IP when employees paste sensitive content into external systems. This creates compliance, competitive, and security risks. Companies are responding with bans, strict use policies, data loss prevention tools, and user training.

Why is global collaboration and governance essential for safe AI?

As AI systems grow more complex and powerful, individual firms or countries cannot manage all risks alone. Coordinated global governance, shared standards, and joint research are needed to address societal harms, align rules, and ensure AI is developed and deployed responsibly worldwide.