Troubleshooting Generative AI: A Practical Checklist

Hello! I am Noriaki Asato, representative attorney at Legal Agent.

Generative AI sometimes produces an answer that looks polished but misses the point. A contract review may stay at the level of general principles. A proposed revision to an NDA may protect the disclosing party even though your company is the recipient. A summary may be accurate but unusable for the manager who needs to make a decision in five minutes.

These results do not always mean that the model is unsuitable. Often, the request is missing information about the purpose, the company's position, the desired output, or the commercial context. This article provides a practical checklist for diagnosing those gaps and improving the next answer.

What this article covers

The checklist below helps identify why an answer is off target, improve a request without starting from scratch, and distinguish a well-formatted answer from a legally reliable one. The examples focus on corporate legal work, but the same approach applies to many business tasks.

The first checklist to run

When an answer is not useful, review these points in order:

• Purpose: Did you explain who will use the answer and what decision it should support?
• Position: Did you identify whether your company is the customer or supplier, principal or contractor, discloser or recipient?
• Format: Did you specify a table, bullet points, a word limit, or a maximum number of issues?
• Context: Did you provide the transaction value, importance, timing, relationship with the counterparty, and risk tolerance that matter?
• Scope: Did you ask for too many different tasks in one request?
• Example: Did you show one example of the structure or tone you want?

In an NDA review, for example, simply stating that your company will receive information changes the focus. The answer should then pay closer attention to the definition of confidential information, permitted use, disclosure to affiliates and advisers, security obligations, return or deletion, and the survival period.

Purpose and position matter most

In legal work, purpose and position often have the greatest effect on the usefulness of an answer.

Purpose means who needs the output and what they will do with it. A summary for a business-unit head should not look like a research note for legal counsel. A prompt such as “Identify the three issues most likely to affect the commercial decision, in order of importance, for a five-minute briefing to the business-unit head” gives the model a much clearer target.

Position means which side of the transaction the company occupies. The risks in a services agreement differ depending on whether the company is the customer or service provider. The same limitation of liability, intellectual property, acceptance, service level, or termination clause can have a very different effect on each side.

If the position is not stated, the model may default to a neutral explanation. Neutrality can be useful for learning, but it is rarely enough for negotiation or internal decision-making.

How to correct an answer that still misses the mark

If the initial checklist does not solve the problem, try a focused follow-up:

• Narrow the task: “Limit the answer to risks for the supplier.”
• Change the audience: “Rewrite this for a non-lawyer executive.”
• Change the perspective: “How would the counterparty justify this clause?”
• Ask for a comparison: “Show the current wording, the proposed wording, and the reason for the change in three columns.”
• Restart with a clean context: If a long conversation has accumulated conflicting assumptions, begin a new conversation with the confirmed facts.
• Test another approved tool: Models can differ in document handling, reasoning style, and output quality, but the same security and confidentiality rules should continue to apply.

Exploring the counterparty's perspective can be particularly useful. For example, if a damages clause appears one-sided, asking how the other party would defend it may reveal an operational dependency, insurance constraint, or pricing assumption that should be addressed in the negotiation.

Basic prompt for restarting the task

The following structure combines purpose, position, format, and context:

You are assisting a corporate legal team.
Review the following contract clause.

Context:
- Our position: service provider
- Audience: the head of the business unit
- Purpose: decide whether the issue must be negotiated before signing
- Output: no more than three bullet points, ordered by risk
- For each point: explain the risk and the preferred revision in one sentence each

[Paste the clause here]

This structure does not guarantee that the legal analysis is correct. It does, however, reduce the chance that the answer will drift into a generic explanation that does not support the intended decision.

A follow-up prompt for revising an existing answer

When an answer is broadly correct but too long or technical, a short correction may be enough:

Revise the previous answer for a non-lawyer internal reader.

- Use plain language and briefly explain any necessary legal term.
- Keep only the three most important points.
- Use bullet points of no more than two lines each.
- Do not add facts, statutory provisions, or cases that have not been verified.

The accuracy of all legal authorities will be checked separately by a person.

Explaining how the answer missed the target is more effective than simply saying that it was wrong. The correction should identify the audience, the required scope, and the feature that must change.

Good presentation is not legal accuracy

A better prompt can improve structure and relevance. It cannot, by itself, make the answer legally reliable.

A response may contain clear headings and plausible statutory citations while referring to a provision that does not exist, has been amended, or applies to a different situation. Dates, thresholds, party names, case citations, and defined terms are especially easy to overlook when the writing appears confident.

Important facts and legal authorities should therefore be checked against primary sources. For Japanese statutes, this commonly means confirming the current text through an official source such as the e-Gov Laws and Regulations Search. Contract analysis also requires checking the full document, its definitions, schedules, related agreements, and the actual transaction background.

The more polished an answer looks, the more important it is to maintain an independent verification step.

Common failure patterns

One common mistake is repeating the same request after an unsatisfactory answer without adding any new conditions. Unless the missing information is supplied, the model may reproduce the same type of answer.

Another is combining too many tasks. “Summarize the agreement, identify every risk, draft all revisions, and prepare the negotiation email” asks the model to optimize several different outputs at once. Dividing the work into review, prioritization, revision, and communication usually produces a clearer result and makes human checking easier.

A third is treating presentation as proof. A table, risk rating, or redline can still contain a false assumption. The format should make review easier, not replace it.

Points to remember in business use

• Do not enter confidential information, personal data, or unpublished transaction terms unless the tool and the use have been approved.
• Confirm what data the provider retains, whether it is used for training, and which workspace settings apply.
• Keep a person responsible for checking legal authorities, important facts, and the final external communication.
• Record the assumptions that materially affect the answer.
• When an output will be sent outside the company, review both substance and tone before sending it.