A Checklist for When Generative AI Isn't Working: An Introduction
Hello, I'm Noriaki Asato, Representative Attorney at LegalAgent.
When trying out generative AI in legal work, you may come across situations where you feel that "it stays at a general explanation and can't be used in practice" or "it's slightly off from what I asked for." For example, you ask it to check a service agreement and get back only an innocuous, textbook-style explanation, or you ask for proposed revisions to a non-disclosure agreement and, even though your company is the recipient (the party receiving the information), it proposes wording favorable to the discloser.
When such gaps arise, several factors may be intertwined: not only insufficient sharing of premises in the instructions, but also the model's own strengths and weaknesses, information being cut off because of limits on reading long text, or how the output format was specified. The way the user gives instructions is not necessarily the only cause, and giving the same instructions will not always produce the same result.
Even so, there are many situations in which reviewing how instructions are put together can improve the accuracy and usefulness of the answers you can draw from AI. Having a set of points you can check at hand helps when you are unsure in your day-to-day work.
When the Answer Doesn't Fit Your Company's Position
When you are not getting the answer you intended, it is worth first checking whether the context you are giving the AI is sufficient. This is because, especially in legal work, the points to focus on change greatly depending on "who the text is for" and "which side your company is on."
For example, in reviewing a non-disclosure agreement, the priority of the clauses that need attention differs depending on whether your company is the discloser or the recipient. If your company is the recipient, the main items to check are whether the definition of confidential information is too broad, whether restrictions on use for other purposes will interfere with your business, whether the range of related parties to whom disclosure is permitted is secured, and whether the survival period is too long. If your position is not conveyed, the answer may stay at general principles or take the side opposite to what you intended.
That said, conveying your position and purpose in detail does not guarantee legal correctness. An answer obtained by specifying your position is merely a proposal of material for consideration from that viewpoint, and human checking remains essential for judging whether it is appropriate in practice.
Checking Purpose, Position, Format, and Premises
When an answer feels off, it helps to check in turn whether the instructions include the following elements.
- Statement of purpose: specifying to whom and for what use the text will be submitted
- Position in the transaction: filling in your company's position, such as whether it is the client or the contractor, the discloser or the recipient
- Specification of output format: narrowing down the desired form, such as bullet points or a table, number of characters or number of items
- Sharing of premise facts: supplementing confirmed facts, such as the importance of the transaction and background circumstances
When conveying the purpose, showing a concrete use, such as "please narrow these down in order of greatest concern so that I can explain the key points to the head of the business unit," makes it easier to get a practical answer. Also, putting a summary, issue extraction, drafting of revised clauses and more all into one instruction tends to blur the focus, so asking in separate stages is also useful.
Another method is to ask "How would the other party argue about this clause?" and anticipate counterarguments. However, the circumstances raised there are hypotheses, and you should not confuse them with the other party's actual intentions or with confirmed facts.
Additional Instructions for Parts That Are Hard to Understand
If the first answer differs from what you expected, it can go more smoothly to add instructions focused on the points you want to improve rather than starting over from scratch. For example, give specific revision instructions such as "rephrase technical terms in plain language" or "compare only the clauses that impose a heavy burden on the contractor." When the exchange becomes long and you see confusion with earlier instructions, starting a new chat and re-entering an organized set of premises leads to better results. However, opening a new chat does not mean the data on the server disappears. Depending on the service, shared memory features and the like may also be at work, so it is not certain that none of the previous context carries over.
When trying a different tool, check whether that service and account have been approved internally. For the materials you move over, also confirm that they are within the scope of what may be input.
Next is an instruction for reporting to the head of a business unit from the contractor's position. The instruction to narrow it down to three points is meant to set priorities for the report.
You are an assistant supporting corporate legal work. Please list review points for the following contract clause.
The premises regarding our position and purpose are as follows. Changing just these will change the emphasis of the answer you return.
- Our position: contractor (the party receiving the work)
- Purpose: for a report to the head of the business unit
- Output format: up to 3 points, as bullet points, in order of greatest risk
- For each point, one sentence on why it is a problem and one sentence on how we would like to fix it
[Paste the clause here]
If the answer is too difficult, tell the AI for which readers it is hard to understand. The next instruction adjusts the wording and length for non-specialists within the company.
The current answer is a little too difficult for non-specialists in our company, and it is also too long. Please revise it as follows.
- Avoid technical terms, and add a short paraphrase where needed
- Narrow it down to the 3 most important points
- Use bullet points, with each point within 2 lines
Note that a person will separately check the accuracy of the facts and statutory provisions. This step cannot replace the final check in practice.
[If needed, note here the parts that were particularly hard to understand]
Even after revision, check that shortening the answer has not dropped conditions or exceptions. Keep requests to polish the wording separate from verifying the content.
Checking the Original Is Still Necessary for Shortened Answers
Even when its text is well formed or neatly bulleted, generative AI may cite article numbers that do not exist or use old laws that do not apply to the transaction in question. In addition, even if the answer lists the "top 3 risks," that does not mean the entire contract has been checked. It requires conscious attention that the more polished the format, the harder errors are to notice.
Check article numbers and their content in e-Gov Law Search, and for case law, check the actual judgment text in the courts' case information or a reliable case law database. It is important to maintain a system in which individual facts, such as transaction amounts and the history of negotiations with the other party, are checked by a person against primary sources such as your company's contracts and internal records. In particular, for draft contracts to be submitted outside the company and external communications, a person needs to check the content before making the final decision.
Services Supporting Practical Operation
If you would like individual support on how to use generative AI in contract review or on building instruction templates tailored to your company, the following services are available.
Services connected to this topic