Managing AI Deployment Risks: Key Considerations for Negotiating With AI Developers

Insurers are increasingly relying on artificial intelligence (“AI”) and automated decision-making technology in underwriting, pricing, and claims handling. These tools can increase operational efficiencies, reduce costs, and lead to positive customer experiences. Some insurers are developing certain tools in-house. Others are integrating AI tools developed by third-parties into their day-to-day operations. The use of AI technology, whether developed internally or externally, carries risks associated with errors or outcomes that could be viewed as biased. This post focuses on mitigating these risks when leveraging third-party technologies. How should insurers approach the procurement of third-party tools?

Insurers are also considering these issues in the context of claims made against insureds. In evaluating a claim, a significant consideration may be the extent to which an insured can shift its potential liability for AI-related risk to the AI developer. Will contractual indemnity provisions be enforceable, particularly with regard to claims of AI-based discrimination?

First, it helps to look at the litigation and risk landscape. What steps can insurers take early on to avoid finding themselves in litigation later? In Huskey v. State Farm Fire & Cas. Co., 2023 WL 5848164 (N.D. Ill. Sept. 11, 2023), State Farm faces allegations that its algorithmic decision-making tool evaluated insurance claims in a racially discriminatory manner by subjecting Black homeowners to more burdensome claims processing requirements, which ultimately resulted in less coverage for claims similar to those submitted by white homeowners.

Additionally, multiple discrimination claims have been made regarding the use of AI technology in hiring. In a pending California action, the court has preliminarily certified a nationwide class of individuals over 40 who applied for positions using an embedded third-party party platform and were denied employment recommendations. In addition to age discrimination, this action also involves claims of racial discrimination and discrimination based on disability. In New York, the U.S. Equal Employment Opportunity Commission (“EEOC”) asserted that an employer’s AI tool discriminated against applicants based on age, including by automatically rejecting more than 200 applicants aged 55 or older.

These cases demonstrate that a top-of-mind risk for insurers is the risk of discrimination claims by customers, including the risk of class actions. The Huskey and employment cases are just two early examples of putative class actions alleging discriminatory conduct arising out of an insurer’s use of platforms developed by third-parties. Thus far, plaintiffs are successfully advancing their cases.

Second, with the AI litigation landscape in mind, it is helpful to look at how courts have viewed contractual risk transfer provisions for potential liability under anti-discrimination statutory regimes. The scenario is this: A business enters into a contract for services with a third-party vendor. The contract contains broadly worded indemnity and limitations of liability provisions in favor of the business, whereby the business’ risks of liability arising out of the vendor’s services are significantly or entirely transferred to the vendor. The business is sued by a customer or end user, and seeks indemnity from the vendor under the contract’s risk shifting provisions. Indemnification “involves shifting the entire loss from one wrongdoer to another; contribution requires each wrongdoer to pay his proportionate—or pro rata—share of the adverse judgment.” George v. Overall Creek Apartments, LLC, 2024 WL 1539848, at *7 (M.D. Tenn. Apr. 9, 2024).

Many courts have held that the duty not to discriminate cannot be delegated in its entirety. See George v. Overall Creek Apartments, LLC, 2024 WL 1539848, at *13 (M.D. Tenn. Apr. 9, 2024) (“The consensus among federal courts is that breach-of-contract claims that are de facto claims for indemnification are, in fact, preempted under the FHA.”). The Fourth Circuit Court of Appeals, in Equal Rts. Ctr. v. Niles Bolton Assocs., held that the American with Disabilities Act (“ADA”) preempted a developer’s crossclaims for implied and express total contractual indemnification against its architect who allegedly did not comply with the ADA’s disability accessibility requirements in designing the developer’s housing units. 602 F.3d 597, 599-602 (4th Cir. 2010). The court emphasized that the purpose of the ADA is “regulatory rather than compensatory,” and thus, “denying indemnification encourages the reasonable care required by the [federal statute].” Id. at 601-02. In other words, the developer could not insulate itself from responsibility via its contract with the architect. See also E.E.O.C. v. Blockbuster Inc., 2010 WL 290952, at *3-4 (D. Md. Jan. 14, 2010) (holding that contractual claim for indemnification was preempted by Title VII because if the defendant could “contract around its obligations and shift its entire responsibility for complying with Title VII,” then the purpose would be “thwarted.”); George v. Overall Creek Apartments, LLC, 2024 WL 1539848, at *14 (M.D. Tenn. Apr. 9, 2024) (finding that to the extent that developers were “asking for the entire loss to be allocated to [their contracted architect], as a result of [the architect’s] alleged breach of contract, the court construes their breach-of-contract claim as a de facto claim for indemnification, which is preempted under the FHA.”); Baker, Watts & Co. v. Miles & Stockbridge, 876 F.2d 1101, 1108 (4th Cir. 1989) (“[D]e facto claims for indemnification . . . are preempted.”).

Although a less settled landscape, some courts have found that claims for contribution, where a party seeks to hold another liable for its pro rata share of liability, are not preempted by federal anti-discrimination laws like the FHA and ADA. For example, in Cnty. of Livingston v. PSA-Dewberry, Inc., 691 F. Supp. 3d 907 (C.D. Ill. 2023) the County and its board brought suit against its architectural firm for negligently designing a courthouse in accordance with the requirements under the ADA. The architectural firm moved to dismiss and the United States District Court for the Central District of Illinois denied the motion. The court reasoned that because plaintiffs were not seeking to hold the architectural firm liable for the total amount of damages, their claims were better characterized as de facto contribution claims, which the  ADA does not preempt. Id. at 915; see also Overall Creek Apartments, LLC, 2024 WL 1539848, at *14 (“To the extent . . . that [the developers] are only asking [their architect] to pay its ‘proportion—or pro rata—share of the adverse judgment’ arising out of [the architect’s] alleged breach of contract, the court construes their breach-of-contract claim as a de facto claim for contribution, which is not preempted under the FHA.”).

Alternatively, if the contractual claim is sufficiently distinct from an indemnification or contributions claim premised on federal anti-discriminatory laws, it may proceed. Courts “considering whether claims such as breach of contract . . . are preempted by the FHA consider whether the claims seek to vindicate an obligation owed independently of the FHA or whether the claims are merely reiterations of an indemnification or contribution claim seeking to vindicate FHA violations.” Clover Communities Beavercreek, LLC v. Mussachio Architects P.C., 676 F. Supp. 3d 82, 93 (N.D.N.Y. 2023). Stated differently, parties may sustain a breach of contract claim that is premised on an “independent obligation to . . .  fulfill the terms of [the] contract.” United States v. Quality Built Const., Inc., 309 F. Supp. 2d 767, 779 (E.D.N.C. 2003) (allowing contribution claim to proceed because “Defendants hired Hite for its architectural expertise to perform design services for the interiors of the units,” and thus “Hite had an independent obligation to perform competently and fulfill the terms of its contract.”); Clover Communities Beavercreek, 676 F. Supp. 3d at 94 (finding that breach of contract claim was not preempted as a de facto indemnification claim because plaintiffs alleged that the architectural firm “breached its contractual duty to inform the owner of each project ‘of any deviations, defects or deficiencies.’”).

In attempting to mitigate litigation risks associated with alleged discrimination as a result of deploying third-party AI tools, insurers should negotiate contracts with developers strategically. Rather than relying solely on sweeping, broadly worded Limitation of Liability or Indemnification provisions, insurers should consider contractual provisions demonstrating efforts to avoid unintended outcomes. Such provisions may require regular testing and retesting of AI tools, grant audit rights, and otherwise provide transparency regarding training data and functionality. A robust risk management framework also means ensuring meaningful human oversight and engagement with the developer. Courts are likely to view such provisions more favorably than attempts to shift all risk associated with potential discriminatory outcomes to the vendor. AI is redefining the insurance industry, and the ability to negotiate an AI vendor agreement that prioritizes transparency will be crucial to assess and mitigate these emerging risks.

Meet the Authors

Natalie Limber

Partner, Dentons

Natalie Limber is a Partner in Dentons’ Los Angeles and Chicago offices. She has a broad range of combined law firm and in-house experience with a focus on serving the legal and regulatory needs of the insurance industry. Prior to joining Dentons, she held roles in claims and corporate legal with a large national P&C carrier, including leading the corporate internal investigation team and serving as general counsel to the company’s Insurtech business unit. She was previously a partner at a law firm in Chicago representing insurers in matters around the country with a focus on excess, professional liability, management liability and commercial lines. Her current practice is focused on defending insurers in coverage and bad faith litigation across policy lines in jurisdictions throughout the country.

Katya Keklikian

Associate, Dentons

Katya Keklikian is an Associate in Dentons’ Los Angeles office. Her practice focuses on representing insurers, financial institutions and other businesses in complex commercial litigation matters, including coverage, bad faith, trade secrets, and contract disputes.