How Do Practice Group Heads Audit AI Tool Use Across Associates?

The practice group head told me he was confident his associates were following the firm's AI policy. I asked how he knew. He said he had not heard any complaints. That is when we discovered that one of his fourth-year associates had been pasting deposition excerpts into ChatGPT consumer for two months because the firm's enterprise CoCounsel seat went to a different practice group and nobody filed the IT ticket to expand the license.

This is the gap most mid-size firms have right now. The AI policy exists. Everyone said they read it. Almost nobody is being asked, on a recurring schedule, what they actually do day-to-day. The privilege exposure compounds quietly until something surfaces in discovery or a deposition.

This guide walks through the quarterly audit a practice group head runs to catch shadow AI use before it becomes a privilege problem. It covers the self-reporting questionnaire, the IT and vendor-log review, the remediation conversation, and the executive committee report. Read this if you supervise associates and have not personally verified what tools they use this quarter.

Why this matters for practice group heads specifically

Practice group heads in mid-size firms sit in a unique supervision spot. They are senior enough to set workflow standards, junior enough to know what associates do day-to-day, and accountable enough that a privilege issue in their group lands at their feet. The managing partner does not run this audit. The IT director cannot, because IT does not see what happens on personal devices. The practice group head is the right person.

What changes when a practice group head runs the audit well: shadow AI use drops, associates get clearer guidance, the firm's AI policy gets refined to match real workflow needs, and malpractice exposure quietly contracts. Firms doing this poorly find out about AI use the same way they find out about expense fraud: in deposition, in a client complaint, or in a state bar inquiry.

What the AI use audit actually does

The audit is a structured quarterly review that combines three data sources to produce a complete picture of AI use across a practice group:

• Enterprise AI vendor audit logs. CoCounsel, Lexis+ AI, Harvey, and Westlaw Precision AI all produce per-user, per-matter audit logs showing every query and output. This data is the firm's primary visibility into authorized AI use.
• IT and SaaS monitoring data. The firm's IT department can see network-level access to known AI domains (OpenAI, Anthropic, Google AI Studio, Microsoft Copilot) and SaaS-level access patterns. This data catches some unauthorized use on firm devices but misses everything on personal devices.
• Associate self-reporting. A structured questionnaire administered quarterly to every attorney and paralegal in the practice group, asking which tools they used, on which matter types, and whether the use complied with the firm policy. This catches the personal-device and personal-account use that IT cannot see.

Three things make this audit different from a generic compliance review:

• It is calibrated for AI-specific risks. Hallucinated citations, privilege waiver through training data exposure, work product disclosure to third-party vendors, and supervision documentation under the state bar opinions.
• It produces remediation, not just findings. Every gap identified gets a remediation action with an owner and a deadline. The audit is not a report that sits in a folder; it is a forcing function for policy refinement and individual coaching.
• It runs on a quarterly cadence with amnesty for self-reported issues. The cadence keeps the firm current on tool evolution. The amnesty drives candor.

Think of it as the firm's continuous-improvement layer for AI governance. The policy says what should happen. The audit confirms what does happen. The gap between the two is where you do the work.

Before you start

You need:

• The firm's current written AI policy. If it does not exist, the audit's first output is recommending one.
• Access to the audit logs from the firm's enterprise AI vendors. Coordinate with IT or with the vendor account manager.
• A relationship with the IT director willing to share network and SaaS monitoring data on AI domain access.
• A structured questionnaire for associates and paralegals. We will build the template below.
• Approval from the firm's general counsel or risk partner to run the audit under privileged supervision. This is a one-time conversation that protects the audit work product.
• About 30 minutes for an initial audit on a small practice group, more for a larger group.

One thing to settle before you start: the privilege framing. The audit must be conducted under the supervision of the firm's general counsel or risk partner so that the audit work product itself is protected from disclosure. We have a dedicated section below on the privilege and malpractice non-negotiables. It is non-negotiable.

Task 1: The quarterly self-reporting questionnaire

The failure pattern: a practice group head asks associates 'Are you following the AI policy?' at a Tuesday morning practice group meeting, every associate says yes, and the head moves on. No documentation, no specifics, no real visibility.

What to send instead, as a structured questionnaire:

AI Use Self-Report, Q2 2026, [Practice Group Name]

1. Which AI tools did you use for any work-related task in the past quarter? Select all that apply: CoCounsel, Lexis+ AI, Westlaw Precision AI, Harvey, ChatGPT (consumer), ChatGPT (enterprise), Claude (consumer), Claude (enterprise), Microsoft Copilot, Google Gemini, Other (specify).

2. For each tool you selected, describe the use cases. Was the work on client matters or internal firm work?

3. For any client matter use, was the tool accessed under the firm's enterprise license? If yes, was the work associated with the correct matter folder?

4. For any client matter use, what was the verification step? Who reviewed the output before it was used?

5. Did you use any AI tool on a personal device or personal account for any work-related task? If yes, describe the use case and the data involved.

6. Have you encountered a situation where you wanted to use an AI tool but the firm policy did not clearly cover it? If yes, describe the situation.

7. Are there workflow changes that would make AI use easier to do correctly? If yes, describe.

The questionnaire is doing several things. It maps the actual tool landscape against the policy. It asks about personal-device use directly, which is where most exposure hides. It surfaces policy gaps from the bottom up, which is more useful than top-down enforcement. It creates a documentary record of supervision under the rules of professional conduct.

Distribute it through the practice group head's email with a 14-day response window. Make clear that voluntary disclosure during this window is amnesty-protected. Anything discovered later through IT or vendor-log review that was not self-reported gets handled differently.

For large practice groups (30+ attorneys), batch the responses through a structured intake form rather than free-text email replies. The structure makes pattern analysis tractable.

Task 2: The vendor audit log review

The failure pattern: the firm signed an enterprise CoCounsel agreement 18 months ago, the audit log feature has been available the whole time, and no one has ever pulled a report.

What to do instead, as a structured monthly pull:

Pull the CoCounsel audit log for the practice group, filtered by user and matter, for the past 90 days. For each user, identify: total queries, total matters touched, average queries per matter, distribution of query types (research, summarization, drafting, review), any flagged content (queries the system identified as potentially out of scope), and any matter touched by users who are not assigned to that matter.

The audit log review surfaces three things that matter:

• Heavy users vs light users. The pattern usually splits the practice group into 'AI-native' associates and 'AI-skeptical' associates. Both populations need attention. The heavy users need verification that their volume is matched by appropriate supervision. The light users need to know what they are missing and whether their workflow is leaving billable hours on the table.
• Cross-matter access patterns. An associate touching matters they are not formally assigned to is a flag worth investigating. Sometimes it is legitimate (a partner asked for cross-matter research). Sometimes it is a conflicts or scope issue.
• Volume anomalies. A spike in queries on a single matter often correlates with a deadline or a complex issue, but it can also indicate over-reliance on AI without sufficient verification. The audit log shows the volume; the conversation with the associate explains it.

Do this in parallel with the self-reporting questionnaire. The triangulation between self-report and audit log catches inconsistencies. An associate who reports 'minimal AI use' but whose audit log shows 200 queries last month gets a different conversation than an associate whose self-report and log are consistent.

Task 3: The IT and SaaS monitoring review

The failure pattern: the practice group head assumes IT is monitoring AI tool access and will surface anomalies. IT assumes the practice group is responsible for use compliance. Nobody actually pulls the data.

What to do instead, in coordination with the IT director:

Pull SaaS access logs for AI-related domains for the practice group's user accounts, past 90 days. Domains to include: openai.com, chat.openai.com, anthropic.com, claude.ai, gemini.google.com, copilot.microsoft.com, harvey.ai, casetext.com (legacy CoCounsel), lexisnexis.com, thomsonreuters.com, plus any other AI tools the firm has detected in network logs. Identify any access from firm devices to consumer-tier AI tools (chat.openai.com without enterprise SSO, claude.ai personal, etc.).

The IT review surfaces consumer-tier access from firm devices, which is the most clearly policy-violating pattern. It does not surface personal-device or personal-account use, which is why the self-reporting questionnaire is necessary alongside it.

The coordination move: the IT director provides the data, the practice group head interprets it. The IT director should not be making policy compliance calls about specific associates. That is the practice group head's responsibility under the supervision rules.

For firms with strict device management (managed Chromebooks, locked-down Windows endpoints), the IT review is more comprehensive. For firms with BYOD policies and laptop discretion, the IT review is limited and the self-reporting becomes proportionally more important.

Task 4: The remediation conversation

The failure pattern: the audit identifies an associate who used ChatGPT consumer on a client matter. The practice group head sends a stern email, the associate apologizes, and the conversation ends without addressing why the associate did it or what changes would prevent recurrence.

What to do instead, in a structured 1-on-1 conversation:

Three-question remediation conversation, scheduled within 5 business days of audit findings:

1. Walk me through the workflow that led to using [tool] on [matter]. What were you trying to accomplish, what were the time pressures, and what alternative did you have under the firm policy?

2. Looking at the same task today, what would the right workflow look like under the policy? What support would you need to do it that way?

3. What is the remediation action for the work that was already done? Do we need to verify the output, re-run the analysis under the enterprise tool, or notify the supervising partner?

The conversation is doing three things. It treats the associate as a professional with reasoning, which produces useful information about why the policy gap occurred. It surfaces the systemic issue (workflow gap, license shortage, training gap) that probably affects other associates too. It produces a concrete remediation plan, not a vague promise to do better.

Document the conversation in a brief memo to the file (under privilege, supervised by the firm's general counsel). The memo identifies the issue, the discussion, the remediation plan, and the deadline for completion. Aggregate findings across multiple remediation conversations show patterns the executive committee report can address at the firm level.

Task 5: The policy refinement output

The failure pattern: every quarter's audit identifies the same policy gaps, but the policy never gets updated because policy revision is treated as a separate project nobody owns.

What to do instead, as a deliverable from each audit cycle:

At the end of each quarterly audit, produce a memo to the firm's general counsel and risk committee identifying:

1. Policy gaps identified during this cycle (situations where the policy did not clearly cover the use case)
2. Tool landscape changes (new AI tools associates encountered, new features in existing tools)
3. Recommended policy amendments with proposed language
4. Recommended training updates (CLE topics, internal training sessions, written guidance)
5. Recommended tool licensing changes (additional seats, additional tools, deprecated tools)

The policy refinement output is what makes the audit a continuous-improvement function rather than a static enforcement exercise. Without it, the policy ages out of relevance within 12 months of any audit cycle.

For mid-size firms, the practice group head delivers the recommendations, the general counsel or risk committee evaluates and approves, the executive committee ratifies, and the revised policy gets distributed firm-wide. The cycle takes 30 to 45 days from audit completion to policy update.

For firms with multiple practice groups running parallel audits, the recommendations are aggregated quarterly and the policy gets refined once or twice per year rather than every quarter. The cadence depends on tool evolution; in a fast-moving year, more frequent updates make sense.

Task 6: The executive committee dashboard

The failure pattern: the practice group head runs a thorough audit, produces a 12-page report, sends it to the executive committee, and nobody reads it.

What to do instead, as a one-page dashboard:

Quarterly AI Use Dashboard, [Practice Group], Q2 2026

Total AI use volume: [X] queries across [Y] matters Compliance rate: [Z]% of use was under enterprise license with proper matter association Top three policy gaps identified: [list] Top three remediation actions taken: [list] Recommended policy changes: [list, with link to detailed memo] Forward-looking risks: [the two or three things to watch for next quarter]

The dashboard is doing several things. It gives the executive committee the systemic view in 60 seconds. It avoids the temptation to micromanage individual associate behavior. It preserves the privileged work product (detailed findings) in a separate memo that goes only to the general counsel.

For firms with multiple practice groups, the dashboards aggregate into a firm-wide view that the managing partner uses to set firm-level priorities. The aggregation also surfaces cross-practice patterns: maybe litigation has a research-AI issue while corporate has a contract-drafting issue, and both want different kinds of policy refinement.

The audit-specific prompts that actually work

After watching mid-size firms run AI use audits for the better part of a year, the difference between an audit that improves the firm's AI posture and one that creates resentment without changing behavior comes down to four moves.

Specify the time period and the data sources. Three months, four sources (questionnaire, vendor logs, IT logs, manager observations). Vague audits produce vague findings. Specific audits produce actionable remediation.

Specify the amnesty rule and the consequences for non-disclosure. Voluntary disclosure during the self-reporting window is amnesty-protected. Discovery during vendor-log or IT review of unreported use gets handled differently. The asymmetric incentive drives candid reporting.

Specify the privilege framing of the audit itself. The audit work product is privileged when conducted under general counsel or risk partner supervision. Document this at the start of the audit. The protection covers the detailed findings; the dashboard summary can be shared more broadly without waiver concerns.

Specify the remediation owner for every finding. Every issue identified gets an owner (practice group head, individual associate, IT director, general counsel) and a deadline. Findings without owners stay unresolved.

The privilege and malpractice non-negotiables

This section is short because the rule is simple, but it is the most important section in this guide.

Do not include any of the following in the audit report or in the audit process itself in a way that creates discoverable records:

• Privileged client communications discussed during associate interviews
• Specific work product details from active matters
• Settlement positions or negotiation strategy
• Witness identities or substantive testimony
• Any client matter substance that does not relate directly to the AI tool use being audited

Conduct the audit under the supervision of the firm's general counsel or risk partner so the audit work product is privileged. Document the supervision relationship at the start of each audit cycle. Keep the detailed findings in a privileged memo. Distribute only aggregate findings to the executive committee.

The state bar opinions on AI use (the 2024 New York opinion, the 2024 California guidance, the 2024 Florida advisory, the 2024 Illinois opinion, and the 2025 Texas opinion) all treat reasonable supervision as a positive factor when AI-related issues arise. Running the audit and documenting the corrective actions is a defense, not a liability. The firms that get into trouble are the ones who never audited and discovered the problem at a deposition, in a discovery review, or during a client complaint.

The Mata v. Avianca case remains the canonical lesson. Two attorneys filed a brief citing six fictional cases. The supervision failure was not catching it. A firm that runs quarterly audits would catch the verification gap before it becomes a sanctions hearing. The audit is the supervision documentation.

Malpractice insurance carriers as of 2026 increasingly ask about AI governance practices in annual applications. ALPS, ProAssurance, and the major specialty malpractice carriers want to see the firm's written AI policy, evidence of supervision, and evidence of regular review. The audit produces all three. Carriers reward firms with documented governance. Some offer premium discounts for verified AI governance programs.

The practical workflow that respects these rules: conduct the audit quarterly under general counsel supervision, document the audit at each step, keep detailed findings in privileged memos, distribute aggregate dashboards to the executive committee, and maintain the audit records for the firm's standard records retention period.

If your firm has signed an enterprise vendor agreement that includes audit log access, the audit can be run more efficiently. Confirm with your IT director and vendor account manager what audit features your contract includes. Do not assume.

When NOT to use the audit process

The audit is a powerful supervision tool but it is not universal. It is the wrong answer for:

• Active matters under deposition or discovery review. Audits during active discovery on a matter can complicate privilege analysis. Run the audit on completed matters and on the firm's overall AI use, not on matters where AI use is a current discovery issue.
• Performance review contexts. The audit is a supervision tool, not an associate performance evaluation. Mixing the two creates incentives for associates to hide AI use rather than report it.
• Pre-employment screening. Some firms have tried using AI use questionnaires during lateral interviews. This produces mostly noise. Lateral candidates will say what they think the firm wants to hear.
• Vendor evaluation. The audit reviews how associates use existing approved tools. It is not the right framework for evaluating new vendors. Use a separate vendor evaluation process for new tool decisions.

A simple rule: the audit is the firm's continuous-improvement layer for AI governance. Trust it for that. Use other processes for performance, hiring, and vendor selection.

The quick-start template

Here is the audit scaffold a practice group head can run for a first cycle. Copy it, adapt to your firm, run it quarterly.

Quarterly AI Use Audit, [Practice Group], [Quarter]

Step 1: Distribute self-reporting questionnaire to all attorneys and paralegals (14-day response window with amnesty for voluntary disclosure).

Step 2: Pull enterprise AI vendor audit logs for the practice group, past 90 days. Identify volume, matter coverage, cross-matter access, and any flagged content.

Step 3: Coordinate with IT director for SaaS access logs covering AI domains, past 90 days.

Step 4: Triangulate the three sources to identify gaps between policy, self-reported use, and observed use.

Step 5: Conduct remediation conversations within 5 business days of identifying gaps, using the three-question structure.

Step 6: Produce policy refinement memo and one-page executive dashboard. Distribute appropriately.

Step 7: Schedule the next quarterly audit.

That is the whole pattern. For 80 percent of mid-size firm practice groups, this is enough. For larger firms, layer the practice group audits into a firm-wide aggregate view managed by the general counsel.

Bigger wins beyond compliance

Once a practice group has the audit running cleanly, the next layer of value shows up in places that are not strict compliance.

Workflow optimization across associates. The audit data shows which associates are extracting the most value from AI tools and which are leaving hours on the table. Pair the heavy users with the light users for informal coaching. The light users learn faster than they would from formal training. The heavy users develop articulation skills useful for partner-track development.

Practice group billable-hour analysis. Cross-reference the audit data with billable hours per matter. Matters where AI was used heavily and supervised well typically show shorter cycle times and higher realization rates. The data builds the case for additional AI tool licenses, more training, and matter-pricing adjustments. The practice group head uses this in budget conversations with the executive committee.

Policy refinement that matches reality. The audit findings drive policy changes that match how associates actually work. Stale policies create more compliance gaps. Living policies, refreshed quarterly based on audit data, stay relevant and get followed. The firms doing this well report policy violations dropping by half within two cycles.

Lateral hiring intelligence. When a lateral associate joins the firm, the audit's questions become part of onboarding. The lateral describes their AI workflow at their prior firm, the practice group head identifies any gaps with the new firm's policy, and the onboarding conversation includes specific guidance. Laterals who join firms with strong AI governance report better adjustment and faster integration.

The law firm AI consulting connection

This is one supervision practice in one operational area. The bigger AI question for mid-size firms is structural. Firms that figure out where AI fits, where it does not, how to deploy it with the right privilege architecture, and how to audit and refine the deployment over time end up with better realization rates, faster matter turns, and a competitive position against the BigLaw firms that previously won every cross-jurisdiction pitch on resources alone. Firms that wait usually end up either banning AI awkwardly, allowing it under the table without supervision, or both.

If your firm is wrestling with the bigger AI question, the AI Consulting for Law Firms page covers the full scope: where AI actually fits in mid-size firm operations, what the common failure modes look like, how the privilege architecture works under current state bar opinions, and what an engagement looks like when it works.

Closing

The goal is not for practice group heads to become AI auditors. It is for the firm to maintain the supervision posture the rules of professional conduct require, on a cadence that catches issues before they become privilege exposures or malpractice claims. The audit is the operational layer that makes the firm's AI policy real. Without it, the policy is a memo on a server. With it, the policy is a living governance practice that produces better work product and tighter risk management.

Pick the practice group most willing to run a first cycle. Schedule the questionnaire for the next quarter. Coordinate with your general counsel and IT director. Run the audit. Then make it a recurring calendar event.

If you want to talk about how AI fits into your firm at the program level, the AI Consulting for Law Firms page lays out the full picture and how an engagement works.