Challenges and Results

Challenges

  1. The Contract Office's checks reportedly involve as many as 100 to 200 points of judgment, including consistency with laws and internal regulations, but no written criteria existed, so judgments depended on the individual
  2. With a team of six (two veterans, two mid-level staff, and two new hires), the accuracy and level of detail of checks varied
  3. Of the 1,000 to 1,200 applications received each year, identical content could be flagged in one year and not in another, causing unnecessary rework

Results

  1. Defined an approach for training AI on check patterns using 2,000 to 4,000 past documents with reviewer comments
  2. Defined and proposed an approach for AI-based first-pass checks using Copilot Studio and Claude

Before

Check accuracy that depended on who did the checking

The Contract Office is responsible for checking submitted specifications and approval requests for errors or omissions. The points to check reportedly include consistency with laws and internal regulations, but they had never been fully put into words, and only broad, conceptual rules existed. As a result, the accuracy of checks varied from person to person. Veterans could review documents in detail in line with laws and rules, while new staff had no choice but to interpret unwritten criteria by feel, which led to oversights and less thorough checks.

Another challenge was unnecessary rework caused by inconsistent checks. Even for applications whose content was nearly identical apart from the fiscal year, there were several cases each year where an application that had passed smoothly the year before was flagged for problems this year. Checks are handled by a team of six: two veterans, two mid-level staff, and two new hires. These six people share the processing of about 1,000 to 1,200 application documents a year (about 100 a month).

Figure: The checking team and the volume it handles

Redesign

The goal: standardizing check criteria, not introducing AI

Once the existing approval app was running stably and maintenance required less time, ITSO interviewed staff on the ground to align on the challenges, and began exploring how AI could be used.

A guiding principle throughout was not to make introducing AI a goal in itself. The primary goal is to standardize (put into words) the check criteria. With 100 to 200 check criteria, turning them all into a paper checklist would make simply going through it a heavy burden. ITSO therefore defined an approach centered on AI, which excels at handling large volumes of data like this.

Specifically, Contract Office staff have long written correction instructions by hand using the comment feature in Word (10 to 20 per document), and 2,000 to 4,000 of these past records have accumulated. ITSO outlined a concept in which this data is fed into an LLM so that it can automatically identify and learn check patterns, such as “this kind of wording gets this kind of comment,” and summarized it as an approach that envisions Copilot Studio as the front end and Claude as the back end.

The ultimate vision is for AI to take over the correction instructions that Contract Office staff have been writing with Word’s comment feature. After reading an application document, the AI automatically detects passages likely to contain problems and writes draft correction comments directly into the document. Final review and decisions remain with Contract Office staff, but by having AI perform a comprehensive first-pass check, the aim is both to eliminate missed issues and to close the gap in check quality that separates veterans from new staff.

Figure: Overview of check pattern learning

Starting with requirements, not with AI

In defining the approach, ITSO first investigated the environment and organized the data, then clarified requirements by asking whether automated AI checks were truly the right measure, or whether simply clarifying and standardizing the check procedures would be enough. Conducting this review before development keeps AI adoption from becoming a goal in itself and helps identify the measures that lead directly to solving the actual problem.

After

Improving through both standardization and AI

The client has a strong desire to build up AI know-how in house and to extend it to other areas of work in the future. Rather than simply building a system, the engagement aims to support the client side by side while systematizing how to proceed with AI and what timeline to expect. Work to reduce dependence on individuals is moving forward on two fronts: standardizing check criteria and using AI.

If your organization’s checks depend on individuals, and judgments vary with the year or the person in charge, we hope this case offers useful insights.