Responsible AI: Towards A Fair, Transparent, and Ethical Future of Healthcare

The rapid rise of AI in healthcare brings serious risks such as biased training data and algorithms, opaque “black-box” models, and hallucinated outputs, making Responsible AI essential to ensure safe, equitable, and trustworthy clinical use. In response, initiatives like the U.S. Trustworthy & Responsible AI Network and the EU AI Act are beginning to define shared standards around fairness, transparency, accountability, and the protection of vulnerable groups. Going forward, healthcare organizations must invest in high-quality, representative data, rigorous ethical impact assessments, multi-stakeholder development processes, and compliance with emerging regulations so AI can genuinely advance care without deepening existing health inequities.

Written by

Tuan Minh Tran

Published on

June 27, 2024

The emergence of artificial intelligence has raised a range of ethical concerns. Therefore, Responsible AI has become an approach to developing and deploying artificial intelligence (AI) in a safe, trustworthy, and ethical manner. As 94% of IT leaders believe more attention should be paid to responsible AI development, the healthcare industry needs to devise strategies to alleviate the current AI challenges.

Current challenges of AI in healthcare

As AI becomes more deeply embedded in healthcare, several persistent challenges limit its safe and equitable use. Key issues include biases in training data and algorithms, a lack of explainability in complex models, and misleading or erroneous results that can affect clinical decisions:

Training data bias

Bias can arise from the way researchers and healthcare institutions collect and prepare data to develop AI models. When the training data for an ML-based system suffers from sampling bias, the patient cohort does not accurately represent the target population. For instance, if an ML-based system is trained to recognize skin diseases such as melanoma using images primarily of people with white skin, it may misinterpret images from patients with darker skin tones and fail to diagnose melanoma due to sampling bias (Adamson & Smith, 2018).

Despite accounting for only 1% of skin cancers, melanoma is responsible for over 80% of skin cancer deaths. As a result, ML developers should disclose details about their training data, including patient demographics and baseline characteristics such as age, race, ethnicity, and gender.

Algorithmic bias

Research by Panch et al. (2019) defines algorithmic bias as the use of an algorithm that reproduces or amplifies inequalities related to socioeconomic status, race, ethnic background, religion, gender, disability, or sexual orientation. When the dataset used to train an AI system lacks diversity, the system may learn biased patterns and perform well for certain demographic groups while systematically failing others.

This can aggravate existing health inequities and lead to poor outcomes for underrepresented groups. In 2023, UnitedHealth faced a class action lawsuit for allegedly using an AI algorithm to deny rehabilitation care to seriously ill patients, even though the companies were aware that the algorithm had a high error rate.

Lack of explainability

Responsible AI implementation in healthcare requires explainable AI, which means embedding fairness, model transparency, and accountability into the AI methods that real-world organizations use. In many cases, AI is used to predict conditions such as sepsis or heart failure by analyzing extensive patient data, including vital signs and laboratory results.

However, these models often rely on complex calculations, especially when they use neural networks in deep learning, which can be difficult for humans to interpret. This difficulty is known as the "black box" problem and refers to the challenge of understanding how an algorithm reaches specific conclusions. The opaque nature of these models can make it hard for clinicians to rely on their predictions, particularly in life-or-death situations where understanding the rationale behind a recommendation is crucial.

Misleading healthcare results

AI hallucinations are incorrect or misleading results generated by AI models. These errors can stem from factors such as insufficient training data, flawed assumptions within the model, or biases in the data used to train it. Hallucinations pose a serious risk when AI systems are used to support high-stakes decisions, including medical diagnoses or treatment planning.

Amid the rapid adoption of AI, one group of researchers conducted a study on ChatGPT to explore how it responds to questions about health and medication plans. They found that ChatGPT often generated inaccurate and even dangerous responses. In one case, the researchers asked whether the COVID-19 antiviral Paxlovid and the blood-pressure-lowering medication verapamil would interact in the body. ChatGPT responded that taking the two medications together would lead to no adverse effects. In reality, people who take both medications may experience a large drop in blood pressure, which can cause dizziness and fainting.

Current initiatives for responsible AI in healthcare

Promoting the responsible use of AI across institutions that adopt this technology is essential, especially in healthcare where new applications are rapidly emerging. As these uses expand, the need for clear development and evaluation standards becomes increasingly important to ensure ethical deployment.

In May 2024, healthcare leaders in the U.S. came together to introduce the Trustworthy & Responsible AI Network (TRAIN). This consortium was created to explore and establish standards for the safe application of AI in healthcare. TRAIN members can potentially help in several ways:

Improving the quality and trustworthiness of AI by sharing best practices related to its use in healthcare settings;
Enabling registration of AI systems used for clinical care or clinical operations through a secure online portal; and
Providing tools to measure outcomes associated with AI implementation.

Beyond these efforts in the U.S., the European Union has introduced the AI Act, a regulation focused on the responsible use of AI. The regulation emphasizes that diversity, non-discrimination, and fairness are critical. AI systems should therefore be developed and used in ways that involve diverse stakeholders and promote equal access, gender equality, and cultural diversity, while avoiding discriminatory impacts and unfair biases.

The AI Act also highlights the importance of evaluating and preventing negative impacts of AI on vulnerable individuals, including people with disabilities, and of actively promoting gender equality.

What still needs to be done?

Beyond setting stringent policies on AI, a genuinely responsible AI ecosystem requires a coordinated, multi-stakeholder approach. AI development in healthcare needs the active involvement of diverse actors — including developers, ethicists, policymakers and patients — to ensure a comprehensive perspective and to foster ethical, trustworthy implementations.

Responsible AI also depends on sustained efforts from healthcare researchers to tackle biased algorithms. A study by Obermeyer et al. (2019) uncovered racial bias in a widely used risk prediction algorithm. This algorithm used insurance claims data to forecast patients’ future healthcare needs based on their recent healthcare expenditures. However, its designers did not account for the fact that healthcare spending for Black Americans is typically lower than for white Americans with similar health conditions, not because of better health, but due to barriers to accessing care, inadequate care or lack of insurance.

After being alerted, the manufacturer tested the algorithm with its own data, confirmed the issue and worked with the researchers to mitigate the bias. By developing new predictive models and adjusting how outcomes were labeled, the team reduced the number of excess active chronic conditions among Black patients and achieved an impressive 84% reduction in sampling bias. These findings indicate that label biases can be corrected, and that healthcare researchers must carefully reassess and refine the data labels used to train algorithms.

At the same time, technology organizations are collaborating to advance responsible AI in real-world business operations. For example, FPT Software has renewed its strategic partnership with the Mila Institute to conduct joint research on large language models (LLMs) and natural language processing (NLP) while promoting Responsible AI. The company has also partnered with NVIDIA and Landing AI to strengthen Vietnamese AI skills and expertise, position Vietnam as a global AI hub and accelerate AI adoption across industries worldwide.

Empowering healthcare with ethics and transparency

With the rapid growth of AI, it is crucial for healthcare and other sectors to recognize the associated challenges. Healthcare institutions, in particular, need to prioritize access to high-quality data and thoroughly analyze training datasets to ensure they are representative and suitable for building unbiased AI models.

Before launching AI software development projects, organizations should conduct ethical impact assessments to identify potential ethical implications in advance. It is also essential to establish transparency in AI systems so that users can understand how decisions are made and on what basis.

Equally important, AI developers should adopt a user-centric approach to AI software development ethics by involving a wide range of stakeholders in the design and implementation of AI systems. This collaborative involvement helps ensure that diverse needs, expectations, and risks are properly addressed.

According to a prediction by Gartner, 50% of governments globally will enforce responsible AI through policies by 2026. In this context, adhering to AI regulations is essential to secure a safer and more trustworthy future for AI in healthcare.

Frequently Asked Questions

What does responsible AI in healthcare actually mean in practice?

Responsible AI in healthcare means designing, training, and using AI systems in ways that are safe, fair, transparent, and accountable. It focuses on protecting patients, reducing harm and bias, making AI decisions explainable to clinicians, and aligning AI use with medical ethics, regulation, and public trust.

How does biased training data put patients at risk in AI care?

Biased training data occurs when the patients or cases used to build an AI model do not reflect the real population. In healthcare, this can cause models to work well for some groups but fail for others, leading to missed or wrong diagnoses and widening existing health inequities.

How can we build fair, transparent AI using better health data?

Fair, transparent AI starts with carefully curated data. Healthcare teams must ensure datasets reflect real patient diversity, analyze them for gaps and biases, adjust labels when they encode inequity, and document demographics and data sources. This underpins explainable models and ethical impact assessments.

What more must healthcare leaders do to ensure ethical AI use?

Beyond high level principles, healthcare leaders need stronger AI governance, robust policies, and inclusive decision making. That means involving clinicians, patients, ethicists, and policymakers, continuously auditing models for bias and harm, and correcting problems in algorithms and labels when inequities are detected.

What networks and regulations guide responsible AI in healthcare?

Multiple initiatives now promote safe AI in healthcare. The Trustworthy & Responsible AI Network helps providers share best practices, register clinical AI tools, and track outcomes. Regulations like the EU AI Act set legal requirements around fairness, diversity, and protection of vulnerable patients in AI design and deployment.