Ep.33 | The role of Data Protection Officers in the A.I. era

Show Transcript

[00:00:00] Hello, and welcome to episode 33 of Privacy Lawls, where I, Donata Stroink-Skillrud, speak with amazing privacy professionals, and we have some laughs along the way as well. Today, I’ll be speaking with Lena Kempe about the evolving role of the data protection officer in the AI era. Lena is an AI, IP, and privacy attorney and founder of LK Law Firm, one of only two New York law firms ranked in Chambers Spotlight 2026 for privacy and a go-to firm for major companies.

With over 20 years spanning an AM Law 100 firm, senior in-house counsel roles at a Fortune 500 company, and a multi-million, multi-billion dollar, uh, global corporation, and general counsel positions at global e-commerce and healthcare tech companies, she advises multi-million dollar enterprises on high-stakes AI, data privacy, and IP matters.

Lena publishes extensively in ABA’s Business Law Today and Bloomberg Law, [00:01:00] speaks on ABA AI panels, and shares practical privacy and AI inse- insights on her blog. Uh, Lena, thank you so much for joining me today. Absolutely. Thank you so much for your invite, and it’s great to be here today. Awesome. So our main topic today is the evolving role of the data protection officer in the age of AI.

To start us off, how would you describe the traditional role of the DPO? Yeah. Traditionally, the DPO function was heavily documentation-driven. While the role became widespread with GDPR in 2018, um, in those e- early years, it was primarily about building and maintaining a compliance framework, records of processing activities, privacy impact assessments, handling data subject requests, and serving as the point of contact with supervisory authority.

It was, in many ways, reactive by design. [00:02:00] Something would happen, a breach, a complaint, a regulatory inquiry, and the DPO would step in to respond. The role was also intentionally positioned at arm’s length from the business. GDPR mandates independence. The DPO cannot be instructed on how to carry out their tasks and must avoid conflicts of interest.

In practice, that independence often created a degree of organizational distance. The DPO was frequently brought in after decisions had already been made, rather than being embedded in the decision-making process itself. And critically, and this matters for everything we’re gonna discuss today, the DPO role is distinct from legal counsel.

It requires a deep understanding of GDPR, CCPA, and other applicable, applicable [00:03:00] data privacy laws, but it’s not intended to serve as the organization’s ultimate decision maker on legal risk. When there is meaningful legal exposure, the DPO’s role is to identify and articulate that risk and to ensure the appropriate legal stakeholders are engaged rather than to make the final legal determination.

That’s great. Yeah, thank you so much for, for that rundown. And today we’re gonna talk about how that role is evolving in the age of AI. Um, and we’re actually doing things a little bit differently in this episode in that we’re gonna discuss some scenarios and examples, um, to show how the DPO role has adapted to AI developments over the past two to three years.

So let’s take a moment to, to look at our first scenario. Um, so in this scenario, a health and benefits company based in Boston offers an AI-driven corporate wellness app to its employer clients. [00:04:00] Uh, employees use the app to track stress, sleep, and nutrition. And the product team recently launched a personalized wellness recommender.

Um, it uses behavioral data, step counts, sleep scores, and mood check-ins to suggest customized mental health interventions. The company markets it as a general wellness tool, not a medical product. The DPO is sitting in a quarterly business review when the chief product officer mentions it in passing.

She goes very quiet, realizing that the system may be processing sensitive mental health data without proper authorization. So in this case, what does the DPO do after that business review, and how does she make the case for potentially pausing a revenue-generated product, um, pending a proper legal review?

Yeah, great questions. The first thing the DPO needs to do is to understand the product because she can’t brief anyone, including attorneys, without knowing what the system is actually doing. [00:05:00] She needs a technical walkthrough, what data is collected, how it is stored, who has access, and critically, whether any of it flows to third parties.

That last question is enormous. A lot of mobile apps have analytics, SDKs, or advertising libraries embedded in them that are quietly receiving health-adjacent data, and that’s precisely what has drawn FTC enforcement attention in recent cases. She also needs to pull the privacy notice and whatever was disclosed to employees when they enrolled.

The gap between what was disclosed and what actually happens is almost always where the legal exposure lives. Then, before she goes back to leadership with findings, she needs to consult with outside counsel. The DPO has [00:06:00] identified a potential problem and gathered the facts. The attorney needs to assess the legal exposure and advise on how to proceed.

Getting their counsel engaged early also means the subsequent internal analysis can be covered by attorney-client privilege, which matters enormously if this ever becomes a regulatory or litigation matter. On making the case for a pause, this is a leadership communication challenge as much as anything else.

She should not walk in– she should not walk into that room as the person who wants to kill a product. She should walk in as the person who is trying to protect the product from being killed by someone else, specifically by a federal regulator or a state attorney general. And she should propose a bounded [00:07:00] pause, not an indefinite suspension.

“I need 15 days for outside counsel to conduct a proper legal review,” is a very different ask from, “We need to stop this product.” A 15-day review has an endpoint, a scope, and a deliverable. A green light with documentation, a remediation roadmap, or a recommendation to restructure the data model. Coming in with options rather than just a problem, problem signals she’s trying to find a path forward, not block the business That’s great.

So let’s talk about our second scenario. So I, I think this one will hit close to home to a lot of people because I feel like AI is being increasingly used in the hiring process or hiring [00:08:00] decision-making, so this one’s very relevant. So in this scenario, a fintech company based in Frankfurt uses an AI-powered applicant tracking system to screen CVs.

The system automatically rejects any candidate scoring below 60 out of 100 without a human ever reviewing the application. And I think a lot of us, um, a lot of people who are looking for jobs have seen that where, you know, now it’s, uh, looking for a job is AI versus AI. You have AI write your cover letter and your resume, and then another AI reviews it.

Um, so a rejected candidate invokes her GDPR Article 22 rights, um, requesting an explanation of the automated decision and asks for human review. HR forwards the request to the DPO, and the DPO has never heard of this tool. So when the DPO received the request from HR, what is their immediate legal exposure?

This one’s stuck, and the DPO needs to get attorney on the phone the same day [00:09:00] HR forwards that request. Article 22 of GDPR prohibits fully automated decision-making that produces legal or similarly significant effects on individuals unless specific conditions are met. Rejecting a job application almost certainly qualifies as a significant effect.

The conditions that permit automated decisions of this kind require either explicit consent, contractual necessity, or an EU or member state law authorizing it. And even when those conditions are met, the data subject has the right to obtain human review, to express their point of view, and to contest that decision.

But, um, the bigger issue is that the DPO has never heard of this tool. That means it wasn’t [00:10:00] registered in the records of processing activities. That means no data protection impact assessment was conducted. And for systematic automated processing with significant effects on individuals, a DPIA is not optional.

It’s mandatory under Article 35 of the GDPR, and it almost certainly means there’s no lawful basis documented for processing. So you have layered violations, failure to maintain adequate records, failure to conduct a DPIA, likely failure to establish a lawful basis And a potential Article 22 breach in the form of this specific candidate’s rejected application.

The supervisory authority in Frankfurt has the power to impose fines of up to 20 million euros or [00:11:00] 4% of global annual turnover for serious GDPR violations. That is the top line exposure. But the more immediate risk is the individual candidate. She has invoked her rights. There’s now a clock. Mm-hmm. Yeah.

So I, I think if the, the DPOs… uh, any DPO that saw this would probably start off by having a heart attack. Um, but after the heart attack, what does the DPO need to do in the next 48 hours? Yeah. Um, within 48 hours, a DPO needs to do several things in parallel. First, get an attorney engaged immediately. The candidate’s request triggers a response obligation, and the response needs to be attorney-directed.

GDPR gives one month to respond to data subject requests with a possible extension by two additional months for complex [00:12:00] cases. Second, gather the facts. The DPO needs to understand this tool, who procured it, when it was deployed, what data it process, where that data goes, whether the da- vendor is acting as a data processor under contract, and whether there is a processing agreement in place.

The odds that there is a proper data processing agreement are low, given that the DPO didn’t know the system existed, but that needs to be confirmed. Third, the DPO needs to brief leadership on the exposure in plain language, not to alarm, but because the decision on whether to suspend the tool is a business decision that leadership needs to make with full awareness.

Yeah. So, you know, that’s the first 48 hours. What about longer term? What does the DPO need to do [00:13:00] after the first 48 hours to fix this? Yeah, that’s a, that’s excellent question. The governance failure here is actually more fundamental than the specific GDPR violation. The company deploy a tool that makes consequential decisions about real people, and nobody in the privacy or legal function knew it existed.

That can’t be fixed by updating a policy document. It requires structure change. The fix is a mandatory pre-deployment review process, what is sometimes called an AI governance gate or AI intake process Before any AI tools is deployed anywhere in a company, it goes through a cross-functional review that includes the DPO, legal counsel, technical team, HR in this case, and other relevant teams.[00:14:00]

That review assesses whether a DPIA is required, establish lawful basis for the processing, determines whether the tool involves automated decision-making with significant effects, reviews the vendor contract for data processing adequacy, and documents all of it. That process doesn’t need to be slow. For low-risk tools, it can be a short checklist.

The point is that nothing goes live without a documented review. The DPO’s job is to design that process and make sure it has organizational teeth, meaning leadership has signed off on it, and there are real consequences for bypassing it. Yeah. I, I absolutely agree. I think in all three of our scenarios, the stop would’ve prevented so many issues, you know?

Mm-hmm. Mm-hmm. Um, if you just have a stop in your… [00:15:00] Let’s say you have a product launch checklist and, um, or product development checklist or software development life cycle- Yeah … adding that review in there can really help prevent a lot of these types of issues, because if you have the review, then you know, okay, the DPA has already been completed.

Mm. There’s been a risk assessment completed. The policies have been updated. People have been informed, you know, and, and that really helps take care of so many issues. And I know that, um, you know, people on the development team don’t necessarily love those reviews- … um, but it can really help prevent, uh, issues in the future.

Yeah, exactly. Yeah. So you discussed, um, GDPR, but in this scenario, what additional obligations does the EU AI Act create for the deployment company beyond, uh, what’s required by GDPR? Yeah. This is where the attorneys really need to be in the room because the EU AI Act creates a distinct compliance framework that [00:16:00] runs alongside, alongside GDPR rather than inside it.

An AI-powered CV screening s- system that makes or significantly influence employment decisions is classified as a high-risk AI system under the EU AI Act. That classification triggers a substantial set of obligations for the deploying company Deployers of high-risk AI systems are required to conduct fundamental rights impact assessments, implement human oversight mechanisms, ensure the system is used in accordance with the provider’s instructions, monitor performance, and maintain logs of the system operation.

The automatic rejection below a score threshold with no human review almost certainly violates the human oversight requirements on its [00:17:00] face. There are also transparency obligations. Affected individuals, in this case job candidates, have rights to certain information about the use of AI in disclo– uh, in decisions affecting them, and there are record-keeping requirements that are distinct from GDPR’s documentation obligations.

That brings up a practical question of ownership. In some companies, EU AI compliance might fall to a dedicated AI governance function. In others, the responsibility is distributed across different departments, including legal, technology, and HR. The DPO needs to work with counsel and leadership to clarify who owns AI Act compliance and how it connects to the existing privacy governance [00:18:00] structure.

Yes. It, it’s very important to reduce or eliminate these silos because- Mm-hmm … you know, if you just assign this to the AI team, then the DPO didn’t know about this still. Yeah. Um, you know, and then there’s privacy issues with this scenario with GDPR, and then you have the EU AI Act issues as well. So- Mm-hmm

yeah, very important to make sure that those teams actually work together. Exactly. That’s right. Do you find that, you know, as more and more people apply for jobs, you know, sometimes you’ll post a job and it has 500 applicants. Half the resumes are, you know, written by AI or- … you know, a lot of them are, are scam.

Um, do you think that more and more companies are gonna be leaning on AI tools to perform this type of screening on applicants when, when it comes to hiring, or do you think that because of these regulatory requirements that we won’t actually see that many companies use these types of tools? [00:19:00] I do see the trend of, uh, many companies using, you know, more and more company using this kind of tool because it’s very efficient.

Um, on the other hand, the, the, you know, e-even in the US there’s already a lawsuit there, uh, claiming that, uh, the, this kind of tool discriminate against, uh, uh, job applicants. So it’s- Mm … hard to say whether, um, what the future trend look like. Yeah. It, it’s a tough one for sure. Mm-hmm. So let’s move into our third scenario here.

So in this third scenario, there’s a SaaS company based in Austin, Texas. Uh, they serve B2B clients across the US, including many in California. A product engineer acting entirely on their own initiative integrates a third-party large language model into the company’s customer support chat. Um, full chat transcripts, including customer names, email addresses, account details, and partial payment information are sent to the LLM provider’s API.

Three months [00:20:00] pass before anyone notices, and the DPO finds out when a San Francisco customer asks them, “Where does my data go when I chat with your support bot?” Um, so to start us off on this scenario, under CPRA, could sending customer data to an LLM provider constitute a sale or sharing of personal information?

And what are the consequences if it does? Yeah. Yeah. This is one of the most consequential questions in US privacy law right now. Under CPRA, a sale is a disclosure of personal information for monetary or other valuable, uh, consideration. A sharing is a disclosure for cross-context behavior advertising.

At first glance, if the LLM provider is purely acting as a service provider, processing the data only to provide the API service back to the company under contract with no right to use data for its own purposes, [00:21:00] it might not be a sale or sharing. It might be a service provider relationship. But here’s the problem with this specific situation.

There’s almost certainly no contract. A product engineer integrated this tool on his own. There was no vendor review, no data processing agreement, and no service provider contract. Without a qualifying contract that meets CPRA’s specific requirements for s-service provider relationships, the LLM provider arguably cannot be classified as a service provid-provider at all.

That potentially means every transmission of customer data to that a-API was a disclosure to a third party, which under CPRA could constitute either sale or sharing or, at minimum, a disclosure requiring notice that was never given. If a [00:22:00] California resident’s data was sold or shared without notice, without an opportunity to opt out, and without the required contractual protections, that is a CPRA violation.

The California Privacy Protection Agency has enforcement authority There is also a private right of action under CPRA for certain types of breaches of unredacted personal information, and partial payment information, which these transcripts contained, is exactly the kind of data that attracts that private right of action.

The DPO needs to get a privacy attorney engaged the same day she finds out about this. Yeah. I mean, when it comes to customer chats, even if you tell people not to tell the chat something- Mm-hmm … th- there will still be customers who are saying, “Oh, I can’t log in. Here’s my email and [00:23:00] password. Why doesn’t it work?”

You know- Yeah … in the text. Or they’ll say, “You know, I’m trying to make a payment with this card, and here’s the full card number.” You know? People put all kinds of things into these chats. Um, and I really think that when it comes to these LLM providers, you know, any provider really, unless you’re checking the contract, um, and unless you’re negotiating the contract, a lot of times these contracts just say all kinds of stuff like, “We can use any of this information for our own marketing and sales and advertising.”

Or, you know, “We’re using this information to train the LLM,” or- Mm-hmm … “We’re sharing this information with, like, 30 different providers.” Mm-hmm. Um, so you know, y- I think you’re completely right. There’s no contract here probably, or- Mm-hmm … you know, maybe the product engineer accepted the standard terms of service when signing up.

Yeah. And that’s about it. So- Mm-hmm. Yeah. I mean, it could have any kind of, any kind of stuff in there. So, you know, if we didn’t– weren’t in this situation with a product engineer and [00:24:00] the DPO was involved earlier, what does a compliant vendor contract with an LLM provider actually need to include? Yeah.

That’s a great question. This is an area where the law is still catching up to the technology, and the attorneys drafting these agreements are working through novel questions in real time. But there are core elements that any contract with an LLM provider that receives personal data needs to address.

The contract need to restrict the provider from using the data for its own purposes, specifically from using customer data to train its models without explicit permission. This has become a flashpoint in LLM contracts. Many providers’ default terms include broad rights to use input data for model improvement.

That is incompatible with most privacy obligations, and it [00:25:00] needs to be explicitly excluded or limited. Under CPRA A service provider contract needs to restrict the provider to processing personal information only for the specified business purpose, prohibit selling or sharing the data, prohibit compo- combining the data with data from other clients, require the provider to comply with CPRA, and gives the company the right to audit compliance.

The contract also needs to address data retention and deletion, what happens to data after the API call, how long it is retained, and what the provider does when the contract ends. And increasingly, these contracts need to address the AI-specific questions of whether inputs could be reproduced in outputs or other uses.

If [00:26:00] a customer’s account details go into a prompt, like what you said, and could theoretically appear in a response to someone else, that is a confidentiality and privacy risk that the contract needs to address. The DPO’s role is to work with the attorney drafting these agreements and flag any gaps before it is executed.

One thing that makes me ver- very nervous with integrating AI tools, obviously you have all the privacy and the compliance issues. Mm-hmm. But, you know, whenever somebody in our team brings up this idea, I’m like, “What happens if the customer’s talking to the AI and the AI gives them an answer that’s, like, completely inappropriate?”

Like they say something really nasty or… You, you know exactly what I mean, right? Right. Right. How do we address that in a contract with an LLM provider of saying, [00:27:00] you know, is the LLM responsible for these types of responses? Because, you know, if a customer gets that type of response, they’re gonna get extremely offended, they’re gonna leave, they’re not gonna be a customer anymore.

How do these LLM contracts handle a situation like that? Yeah. That’s a, that’s a great question. Um, that, to be honest, uh, it’s down to the level who has negotiation power because, um, if there are default terms, I guarantee you they will have terms about limit of their liability, right? So usually- Yes … it will say, “My liability is limited to 12 months of service fee,” right?

So- Mm-hmm … on the other hand, if you pay enterprise license where, uh, you potentially can push, um, you want the higher liability cap, or you could, uh, also add… Well, one of the things you can definitely add is requiring the, the LL- LLM provider to [00:28:00] comply with industry standard. In other words, they have to do anything that, uh, their peers would do so that to reduce the chance of hallucination, right?

What you just talked about is called AI hallucination or AI going wild. And, and sometimes they, they would usually have a disclaimer and say, “No, I– we can do what we can do, but when AI goes wrong, it’s a- actually beyond our reasonable, uh, control.” So, uh, you know, in that case, you can try your best to protect your own interests, but, um, it really depends on how much power you have to, to push back.

Absolutely. Yeah. What, what happened to this world? We used to have girls gone wild, now we have AI gone wild. So we kinda touched on this earlier a little bit, but I did wanna see if you had any additional tips for this. What governance process should the DPO put in place so that no AI tool ever gets deployed without [00:29:00] review again?

Yeah. The core problem in the scenario we just discussed is that a technical decision integrating an external API that process customer data was treated as purely a product decision with no privacy, legal, or security review. The governance fix has to make that structurally impossible or at least structurally difficult.

The DPO needs to work with engineering leadership, legal, and chief information security officer to put in place what is sometimes called an AI or technology intake process. The basic principle is simple. Before any new tool integration or API that process personal data goes into production environment, it goes through a document review– documented review.

Not an endless review, a right-sized review. For low-risk tools, it [00:30:00] might be a short questionnaire and a one-week turnaround. For tools that process sensitive data or make consequential decisions, it is a full privacy impact assessment with legal sign-off That intake process needs to be embedded in the engineering workflow, ideally as part of the existing change management or architectural review process.

So it’s not a separate, uh, bureaucratic hurdle, but a natural step in how new tools are evaluated. Security reviews should trigger privacy reviews automatically. Um, the DPO needs to make sure that connection, that connection is explicit. There also needs to be a vendor management piece. A list of approved AI vendors with pre-negotiated co- data processing agreements removes a huge amount of friction.

If an engineer wants to integrate an LLM, then there are [00:31:00] three pre-approved providers with compliant contracts already in place. The path of least, least resistance is compliance. Right now, the path of least resistance, what happened here, sign up with a credit card and start sending data. And finally, the DPO needs an inventory, a register of all AI tools currently in use across the company with the data flows, the vendors, and the basis for processing docu- documented.

Without that inventory, the DPO is always flying behind, as this scenario illustrate perfectly. Yeah. I really like the idea of having a list of pre-approved vendors already on hand- Mm-hmm … um, so that people don’t have to wait for you to do vendor due diligence. Um, you know, I mean, you could even set up rules, um, like for development, we use Jira as the task management software.

Mm-hmm. And you have different rules of how a project goes from start to launch. [00:32:00] Um, and you know, you could have a rule like, you know, tested and then ready to launch, but before you can actually launch it, it has to go through a review, and there’s only a certain person that can approve that. You know, it can be as simple as some- having something like that set up.

So for my final question for you, um, for DPOs, what skills are becoming essential that weren’t before? Yeah. The, the traditional DPO skill set was essentially legal literacy plus organizational influence, understand what law requires, and persuade people to comply. That’s still the foundation, but it’s nowhere near sufficient anymore.

Um- Mm-hmm … the first new essential skill is technical literacy in AI and machine learning. Not the ability to write code, but the ability to sit in a conversation with product engineer or a data scientist and understand what they are describing well enough to ask the right questions. So when someone tells you an AI system is making decisions [00:33:00] based on a score, you need to know to ask What is in the training data?

How is the model dev- uh, validated? Where are the outputs going, and is there any human review? Those are, those are not legal questions. They are technical questions, and the DPO who can ask them will always be working from incomplete information. And the second- Yeah. Yeah. And then the, the second skill is what we call regulatory, um, landscape monitoring.

The pace of change is really unprecedented. CPRA is implementing regulations, new state privacy laws, um, coming into effect, FTC enforcement guidance evolving. The DPO needs systems to– for tracking all of that and understanding the implications for the organization. That is not just reading newsletters, it’s building relationships with [00:34:00] outside counsel across jurisdictions and having regular briefings built into the calendar.

Absolutely. Yeah, I totally agree with that. I think having that technical knowledge will also make sure that those, um, meetings are not frustrating- Mm-hmm … um, to the development team. You know, if they have to explain things over and over again instead of the DPO coming in with a general technical understanding.

Mm-hmm. Um, I think that definitely helps those conversations go smoother and be more productive too. Yeah. Yeah. Um, so Lena, thanks so much for, for sharing your insights with me today. Thank you so much for having me today. Absolutely. And to our listeners, um, make sure to subscribe to Privacy Lawls so that you don’t miss our next episode.