What Is AI Writing? Generation, Revision, and Governance

Published Jun 26, 202612 min readOrthotropy Team

AI writing is the use of a generative language system to draft, rewrite, summarize, or edit written content from human instructions and source material. Modern systems process context as tokens and generate a continuation one token at a time. They can produce fluent prose quickly, but they do not supply factual verification, authorship, editorial judgment, or publication accountability; those remain part of the human-controlled workflow.

AI writing is a workflow, not an author category

AI writing covers any writing process in which a generative system materially proposes or transforms language. The input may be a one-line request, a detailed brief, an existing draft, interview notes, or a collection of approved sources. The output may be an outline, email, summary, translation, rewrite, product description, or long-form draft.

The neighboring terms are useful only if they describe what the system actually does. Generative AI is the broad category that can create text and other media; AI writing is its text-specific application. An AI text generator usually starts new prose from a prompt. An AI writer is often the same capability packaged around a writing use case. An AI writing assistant usually spans both generation and transformation: it can draft, add to a document, rewrite selected text, change tone, summarize, or use supplied files as context. Microsoft documents that combined pattern in Copilot for Word, while also telling users to verify and modify the result.

One deployed writing assistant supports starting a draft, grounding a new document in selected files or communications, adding to existing content, regenerating a response, and refining a draft for tone or concision. Its documentation explicitly says the generated draft still needs verification and modification.

These labels do not replace the business purpose of the prose. Copywriting is written to move a reader toward an action; content writing is usually written to explain, educate, or build understanding. AI can assist with either. Likewise, AI-assisted and AI-generated describe how work was divided, not whether the result is good, original, lawful, or ready to publish. Automated low-value content is a publication failure; it is not a synonym for every use of AI.

There is no formal formula for AI writing because it is a workflow and tool category, not a calculated metric. The underlying model is mathematical, but no equation turns a prompt into a universal score for “good writing.” There is no accepted benchmark for that score either. Accuracy, source fidelity, revision time, brand fit, acceptance rate, and reader outcome are different measurements, and the right combination depends on the artifact.

One bounded experiment shows why headline productivity numbers need context. Noy and Zhang studied 453 professionals completing short, occupation-specific writing tasks. ChatGPT access reduced average completion time by 40 percent and raised evaluator-rated quality by 18 percent, but the assignments did not require precise factual accuracy or organization-specific context. That is evidence that assistance can improve some writing tasks—not a promise that any tool will make an editorial system 40 percent faster.

In the reported experiment, professionals with ChatGPT completed the assigned writing tasks faster and received higher average quality ratings. The study’s bounded tasks, system, participants, and limited factual requirements constrain any transfer of those percentages to another workflow.

How a generative system produces a draft

A modern language model first learns statistical relationships in large collections of tokenized text. Tokens are pieces of text: sometimes a word, sometimes part of a word, punctuation, or another recurring unit. During pretraining, an autoregressive model repeatedly learns to estimate what token could follow the preceding context.

The Transformer architecture made that process more capable by using self-attention to relate positions in a sequence. Attention helps the model weigh which earlier tokens matter to the current position; masking prevents the decoder from looking ahead at output tokens it has not generated yet. The model is not retrieving a completed paragraph hidden in its parameters. It constructs a sequence by selecting a next token, appending it to the context, and repeating.

The foundational Transformer decoder uses masked self-attention for autoregressive sequence generation. GPT-3 is an autoregressive language model that demonstrated how a pretrained model could respond to instructions and examples supplied in text across multiple language tasks.

Instruction tuning then makes a base model more responsive to requests. In the InstructGPT work, researchers collected human-written demonstrations, trained a model on those examples, collected human rankings of alternative outputs, and used that preference signal for further training. This changes which continuations the model tends to produce. It does not turn statistical generation into a truth database, and the researchers still reported simple mistakes.

Human demonstrations and rankings can shape an instruction-following model toward outputs evaluators prefer, but the reported training process did not eliminate untruthful, unhelpful, toxic, or simply mistaken responses.

At use time, the system combines its trained parameters with the material available in the current context: system rules, the user’s prompt, prior conversation, examples, tool results, and attached or retrieved text. Decoding settings influence whether it repeatedly chooses highly probable continuations or samples more broadly. That is why the same prompt can yield different wording and why precise context usually matters more than adjectives such as “excellent” or “professional.”

Consider an illustrative request: “Turn these release notes into a 120-word customer update. Preserve every date, do not add benefits that the notes do not state, and use this approved example for tone.” At the output layer, the model generates a continuation under those constraints rather than retrieving a completed 120-word document. The notes give it facts, the example supplies a pattern, and the instruction defines the transformation. A reviewer must still check every preserved date and benefit claim against the notes.

Grounding changes the evidence available, not the model’s accountability

A model that relies only on its parameters may produce a plausible answer without a recoverable source. Retrieval-augmented generation adds a search step: the system retrieves passages from an external collection and conditions the generator on them. The original RAG research framed this as a combination of parametric memory and inspectable, updateable non-parametric memory.

In the reported experiments, retrieval-augmented models combined a pretrained generator with retrieved documents and produced more factual language than the paper’s parametric-only baseline on the evaluated tasks.

Grounding is useful, but it is not a guarantee. A retrieval layer can return the wrong document, an outdated version, a low-authority source, or a passage that does not support the generated claim. The generator can also blend supported text with an unsupported inference or attach a citation to the wrong sentence. The operational unit to review is therefore not “a cited draft.” It is a chain: source selected, passage retrieved, claim generated, citation attached, and human decision to release.

This explains a central AI-writing failure mode. NIST uses confabulation for confidently presented false or internally inconsistent content. Because a language model is generating a statistically plausible continuation, fluency and factual support are separate properties. A polished sentence can be wrong; a real link can fail to support the clause beside it.

NIST describes confabulation as a natural risk of generative design: next-token prediction can produce accurate and consistent text, but it can also produce false, internally inconsistent, or fabricated content and citations.

Revision is another generation pass

When an AI assistant “edits” a paragraph, it receives the draft plus a new instruction and generates a replacement or continuation. It does not possess a stable editorial intention outside the supplied context. A request to make prose shorter can remove a qualification. A tone change can strengthen a tentative claim. A request for examples can introduce unsupported specifics.

Use revision prompts to name both the desired change and what must remain invariant. “Cut 25 percent while preserving every number, source, limitation, and decision condition” is more operational than “make it punchier.” Then inspect a diff against the source draft. For factual content, recheck the changed claims rather than assuming a second model pass verified the first.

The main AI-writing modes create different review obligations:

ModeWhat the system is asked to doPrimary review question
Blank-page draftingPropose structure and language from a briefDid it introduce claims or assumptions absent from the brief?
TransformationShorten, expand, translate, reformat, or change toneDid the meaning, scope, or level of certainty drift?
SummarizationCompress supplied materialDoes every material statement remain faithful to the source?
Grounded synthesisCombine claims from retrieved or attached sourcesDoes each claim have the right source, version, and limitation?
CritiqueIdentify gaps, objections, or possible revisionsAre the criticisms valid, material, and independent of the model’s preferred style?

The safest use is often transformation because the human can specify the source and compare before and after. Blank-page generation carries more hidden assumptions. Grounded synthesis can be powerful, but its evidence chain needs the most explicit inspection.

Evaluate the writing task, not “the AI” in the abstract

A general model leaderboard cannot tell a team whether an onboarding email is accurate, whether a case study preserves consent, or whether a support article reflects the current product. NIST’s text-to-text pilot uses multiple measures even for the narrower problem of summary quality, including syntactic, semantic, and reference-overlap approaches. Business publishing adds requirements those measures do not capture.

NIST’s text-to-text evaluation does not reduce generated-summary quality or AI-text discrimination to one number; it reports several metrics and describes multiple ways to compare generated summaries with source documents.

Build a small test set from the work you will actually release. Include ordinary cases, difficult cases, missing-source cases, contradictory sources, and instructions the system should refuse or escalate. Score the artifact on separate dimensions:

DimensionA testable questionEvidence to retain
Task fitDid the output satisfy the brief, audience, format, and length constraints?Rubric score and failure notes
Factual supportIs every material claim supported by the approved source set?Claim-to-source check
Meaning preservationDid a rewrite or summary retain qualifications, numbers, and uncertainty?Source-to-output diff
Editorial effortHow much human time was spent prompting, checking, rewriting, and correcting?End-to-end time, not generation time alone
Risk complianceDid the workflow protect restricted inputs and trigger required review or disclosure?Control log and exceptions
Reader outcomeDid the released content help the intended reader complete the real task?Appropriate behavioral or research evidence

Track false acceptance as well as false rejection. A draft the system produced quickly but an editor should have rejected is more important than a harmless style preference. Compare against the existing human workflow, not an imaginary zero-cost baseline. Re-run the set after a model, retrieval corpus, prompt, or policy changes.

Govern the path to release by consequence

A useful AI-writing policy does not begin with “AI allowed” or “AI banned.” It begins with the consequence of an error and the sensitivity of the inputs.

Low-consequence internal work—brainstorming headings, reformatting approved notes, or creating a private first-pass summary—may need a light review, provided confidential data stays within approved systems. Public marketing, documentation, and customer communications need claim verification, brand and accessibility review, and a named owner. Legal, medical, financial, employment, safety, or other consequential content needs qualified domain review and may be unsuitable for automated release at all.

That is a policy choice, not a universal three-tier standard. The durable control is that review depth rises with potential harm. NIST’s generative-AI profile points organizations toward inventories, data provenance, known-issue records, human oversight roles, context-specific testing, monitoring, and incident response rather than reliance on one generic quality claim.

NIST recommends documenting proposed use, limitations, data provenance, evaluation information, human oversight, and legal or regulatory considerations; it also recommends testing in conditions similar to deployment and monitoring real-world behavior.

A practical source-to-release path has five gates:

Define the artifact and owner

State the reader task, permitted use of AI, prohibited claims, release channel, and the human who can approve it.

Control the inputs

Use an approved service and source set. Exclude secrets, personal data, client material, or licensed text unless policy and contractual terms allow that processing.

Ground material claims

Separate supplied facts from generated transitions and inferences. Preserve source URLs, versions, access dates, and relevant limitations.

Review the output

Check factual support, meaning, originality, tone, accessibility, and domain requirements. Use a qualified reviewer when the consequence demands one.

Record and release

Keep the approved version, material corrections, reviewer, and proportionate information about automation or model use. Add a disclosure where readers, policy, contract, or law reasonably requires it.

The record should be light enough to maintain and strong enough to reconstruct a failure. For a low-risk internal rewrite, a document history may be enough. For a public, evidence-heavy article, preserve the brief, source set, claim review, approval, and correction path.

Three shortcuts fail as governance

“AI writing is plagiarism” is too broad. AI-generated text is not automatically copied from a specific source, but generation does not make text original, attributable, accurate, or licensed. Check for close reproduction, identify the ideas and evidence that require attribution, and follow the applicable rights and editorial rules. Copyrightability is a different question from plagiarism. In the United States, the Copyright Office says AI assistance does not bar protection for human-authored expression, while entirely AI-generated material is not protected; mixed works require a fact-specific assessment of human contribution.

The U.S. Copyright Office distinguishes AI assistance from entirely AI-generated expression and evaluates whether a human supplied sufficient expressive authorship. That copyrightability analysis does not decide whether a particular use is plagiarism, infringement, or lawful in another jurisdiction.

An AI detector is not proof of authorship. A detector estimates whether text resembles examples in its training or test setup. It does not reconstruct the writing process. In a 2023 study of 14 systems, the tested detectors were neither fully accurate nor reliable, and translation or obfuscation worsened performance. Use detection, if at all, as a weak review signal. Edit history, source records, declared workflow, and direct examination of the work provide more useful governance evidence.

The evaluated AI-text detectors produced errors and materially different results after translation or obfuscation, so their scores could not serve as definitive evidence of how a text was produced.

“Google penalizes all AI writing” is also false.

Google says appropriate use of AI or automation is not inherently against its guidelines. Using automation primarily to manipulate rankings is against its spam policies. The useful publishing question is whether the page is original, accurate, helpful, and made for the reader—not whether the first draft began in a text box.

Google’s stated distinction is based on purpose and quality: appropriate automation can be used to create helpful content, while automation used primarily to manipulate search rankings violates its spam policies.

Let the system propose prose; make a person own the claim

AI writing is most useful when the task is bounded, the source material is available, the expected transformation is clear, and a reviewer can compare the result with evidence. It is least defensible when the workflow asks fluent output to substitute for missing research, domain judgment, permission, or accountability.

The decision
The operating rule is simple: let the system propose prose, but require a named person to own every released claim. That owner decides which sources are acceptable, which changes preserve meaning, which risks require escalation, and whether the final artifact deserves publication. Generation can be automated. Responsibility cannot.

Sources

  1. Advances in Neural Information Processing Systems, “Attention Is All You Need”Supports: The Transformer architecture uses attention rather than recurrence or convolution as its primary sequence-processing mechanism; Masked self-attention in the decoder prevents a position from using later output positions during autoregressive generation. Checked 2026-06-26.Limitation: This foundational paper describes the Transformer architecture and machine-translation experiments; it does not describe every component, training stage, or control in a modern AI writing product.
  2. Advances in Neural Information Processing Systems, “Language Models are Few-Shot Learners”Supports: GPT-3 is an autoregressive language model trained to model token sequences; A pretrained language model can perform varied language tasks from instructions and examples supplied as text without task-specific parameter updates; The paper documents meaningful capability limits and methodological concerns alongside performance results. Checked 2026-06-26.Limitation: The paper studies GPT-3 as released in 2020; its scale and benchmark results are not representative of every later model or deployed writing system.
  3. Advances in Neural Information Processing Systems, “Training language models to follow instructions with human feedback”Supports: Instruction-following behavior can be shaped with human-written demonstrations and rankings of model outputs; Fine-tuning with human feedback improved preference, truthfulness, and toxicity results in the reported evaluation while leaving known mistakes. Checked 2026-06-26.Limitation: The results concern the InstructGPT training and evaluation setup; they do not prove that human-feedback tuning makes later systems factual, safe, or aligned in every use.
  4. Advances in Neural Information Processing Systems, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”Supports: Retrieval-augmented generation combines a pretrained generator with retrieved external documents; External retrieval can make knowledge easier to update and inspect than relying only on information represented in model parameters; The reported RAG models produced more factual language than the paper's parametric-only baseline on the evaluated tasks. Checked 2026-06-26.Limitation: The paper reports particular models, corpora, and tasks; retrieval quality, source authority, citation faithfulness, and later product behavior still require direct evaluation.
  5. National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”Supports: Generative systems can produce confident false or internally inconsistent content as a consequence of statistical generation; Relevant risk areas include confabulation, data privacy, information integrity, intellectual property, harmful bias, and human-AI configuration; Suggested controls include inventories, provenance records, human oversight roles, context-specific testing, monitoring, and incident handling. Checked 2026-06-26.Limitation: The profile is voluntary, cross-sector risk guidance rather than a certification, product endorsement, or substitute for applicable law and domain-specific controls.
  6. National Institute of Standards and Technology, “2024 NIST GenAI Pilot Study: Text-to-Text Evaluation Overview and Results”Supports: Text-to-text generation and discrimination are evaluated with multiple metrics rather than one universal writing-quality score; Summary quality can be assessed against source documents through several syntactic, semantic, and overlap-oriented measures. Checked 2026-06-26.Limitation: The pilot focuses on specific generator, summarization, and discriminator tasks; its metrics are not a complete editorial rubric for business content.
  7. Science, “Experimental evidence on the productivity effects of generative artificial intelligence”Supports: In an experiment with 453 college-educated professionals, access to ChatGPT reduced completion time for the assigned writing tasks by 40 percent and increased evaluator-rated output quality by 18 percent; The tasks covered bounded professional writing assignments and did not require precise factual accuracy or organization-specific context. Checked 2026-06-26.Limitation: The experiment used particular short writing tasks and a particular system; its average effects do not establish expected savings, factual reliability, or business outcomes for another workflow.
  8. U.S. Copyright Office, “Copyright and Artificial Intelligence, Part 2: Copyrightability”Supports: Under the Office's U.S. analysis, AI assistance does not by itself prevent copyright protection for human-authored expression; Entirely AI-generated content is not protected, while human selection, arrangement, or modification may be protectable when it supplies sufficient authorship; Copyrightability requires a fact-specific assessment of the work and the circumstances of its creation. Checked 2026-06-26.Limitation: This is U.S. Copyright Office guidance about copyrightability, not legal advice, a plagiarism test, or a statement of law in every jurisdiction.
  9. International Journal for Educational Integrity, “Testing of detection tools for AI-generated text”Supports: The study tested 14 AI-text detection systems and found that the evaluated tools were neither fully accurate nor reliable; Machine translation and content-obfuscation techniques materially affected detector performance. Checked 2026-06-26.Limitation: The study evaluates tools and text available in 2023, primarily for academic-integrity use; it does not establish the accuracy of every later detector or every content type.
  10. Google Search Central, “Google Search's guidance about AI-generated content”Supports: Appropriate use of AI or automation is not inherently against Google Search guidelines; Using automation primarily to manipulate search rankings violates Google's spam policies; Google advises creators to focus on original, high-quality, people-first content and to provide creation context when readers would reasonably expect it. Checked 2026-06-26.Limitation: This is Google's own search guidance, not a ranking guarantee, traffic forecast, or general editorial standard for other distribution channels.
  11. Microsoft Support, “Draft and add content with Copilot in Word”Supports: A deployed writing assistant can start drafts, use provided files or communications as context, add to existing text, regenerate output, and revise tone or concision; Microsoft instructs users to verify and modify generated details for accuracy, tone, and style. Checked 2026-06-26.Limitation: This source documents one commercial product and is used only as a concrete example of assistant functions, not as a category standard or recommendation.

Continue the evidence path

Related reading

Run your growth team from one screen.

Invite only