Align to Structure: Aligning Large Language Models with Structural Information

Source: converted from arXiv LaTeX source papers_md/2504.03622_align_to_structure/source/anonymous-submission-latex-2026.tex.

Abstract

Generating long, coherent text remains a challenge for large language models (LLMs), as they lack hierarchical planning and structured organization in discourse generation. We introduce Structural Alignment, a novel method that aligns LLMs with human-like discourse structures to enhance long-form text generation. By integrating linguistically grounded discourse frameworks into reinforcement learning, our approach guides models to produce coherent and well-organized outputs. We employ a dense reward scheme within a Proximal Policy Optimization framework, assigning fine-grained, token-level rewards based on the discourse distinctiveness relative to human writing. Two complementary reward models are evaluated: the first improves readability by scoring surface-level textual features to provide explicit structuring, while the second reinforces deeper coherence and rhetorical sophistication by analyzing global discourse patterns through hierarchical discourse motifs, outperforming both standard and RLHF-enhanced models in tasks such as essay generation and long-document summarization.

links Codehttps://github.com/minnesotanlp/StructAlign Datasetshttps://huggingface.co/datasets/zaemyung/writing_prompts_collection Extended versionhttps://arxiv.org/abs/2504.03622 links

Introduction

The rapid advances in large language models (LLMs) have sparked significant interest in aligning these models with desired behaviors and outputs [casper2023open,kaufmann2024surveyreinforcementlearninghuman]. While alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) [Christiano2017deep,ouyang2022training], have effectively improved the helpfulness'' andharmlessness'' of generated text, most such work focuses on what humans prefer, rather than how humans structure an eloquent and coherent discourse.

Human writers naturally employ established text structures such as problem-solution, cause-effect, or descriptive schemes [meyer1975organization], to maintain local coherence and logical flow. Theoretical frameworks [mann1987rhetorical] further illuminate how arguments, explanations, and narratives link together to ensure global consistency. Incorporating these structural principles into LLM training could enhance text generation, enabling models to produce more human-like, coherent discourse and better align with human writing conventions.

Figure fig:struct_align_concept: Structural alignment extends conventional LLM alignment by incorporating well-defined textual and discourse-level structures, guiding models toward more coherent and human-like text generation.

To this end, we introduce structural alignment, a novel alignment objective that extends beyond aligning models with human values to incorporating linguistically-driven document structures. We explore two avenues of research. First, we align LLMs with established surface-level text structures (cf. Figure fig:struct_align_concept) by leveraging an external LLM to guide or evaluate generation within a Proximal Policy Optimization (PPO) framework [schulman2017proximalpolicyoptimizationalgorithms] under a Reinforcement Learning from AI Feedback (RLAIF) paradigm [Bai2022ConstitutionalAH].

Second, we parse documents using the RST discourse framework to construct hierarchical discourse trees and derive discourse motifs--characteristic substructures that highlight patterns distinguishing human-authored from LLM-generated texts. By integrating an authorship classifier that leverages these motifs to discriminate between human and AI texts, we obtain a (soft) binary label serving as a reward in the PPO framework, effectively guiding the generation process toward more structurally coherent and human-like discourse.

A key motivation for structural alignment lies in long-form text generation, where ensuring logical progression and thematic consistency poses major challenges. Although existing alignment efforts optimize user preferences, they often overlook the ``rhetorical scaffolding'' essential for a comprehensive and logically consistent composition [Mann2006-MANRST]. In contrast, structural alignment focuses on organizing content according to recognized discourse relations, thereby facilitating continuity and depth over extended passages. This emphasis proves particularly beneficial for longer texts---such as multi-paragraph essays or creative narratives---where abrupt topic shifts, undergeneration, or superficial cohesion can profoundly affect the overall reading experience [10.1145/3544548.3581225]. To enhance stability in long-form generation, we introduce a dense reward scheme for PPO that allocates rewards to words based on their relative distinctiveness in human-written and LLM-generated text. Our contributions are as follows: enumerate 0em Introduce a novel alignment framework by systematically incorporating linguistically grounded structures into the alignment process, enabling LLMs to generate more well-structured, logically consistent text. Propose a dense reward scheme based on linguistic structures to enhance the stability of RLHF training, particularly for long-form generation (more than 1,000 tokens). Both quantitative and qualitative evaluations show that structurally aligned LLMs generate discourse structures that more closely mirror human writing, leading to essays with improved clarity, coherence, and overall quality. enumerate

Related Work

Aligning Large Language Models

Various alignment frameworks have emerged recently, such as reinforcement learning from human feedback (RLHF) [Christiano2017deep, ouyang2022training] or from AI Feedback (RLAIF) [Bai2022ConstitutionalAH,ankner2024critiqueoutloudrewardmodels,ye2024improvingrewardmodelssynthetic], using different optimization techniques such as Direct Preference Optimization (DPO) [dpo], Group Relative Policy Optimization (GRPO) [shao2024grpo], among others. Inspired by RLAIF that leverages LLM judges as reward models (RMs) [ankner2024critiqueoutloudrewardmodels, ye2024improvingrewardmodelssynthetic], we adopt a similar approach by utilizing an off-the-shelf LLM as the RM, eliminating the need for additional fine-tuning. Although these alignment techniques differ in their underlying mechanisms, their primary objective remains very similar: shaping model outputs to align with user preferences, social norms, and ethical constraints, ensuring they are perceived as appropriate and acceptable by humans. In contrast, our work proposes an alternative alignment objective—training LLMs to generate text with well-structured, human-like discourse. Our concept of structural alignment defines an alignment goal rather than prescribing a specific training technique, allowing for flexible implementation across different learning paradigms.

Challenges in Long-Form Text Generation

Recent advances in memory-efficient architectures [hu2021loralowrankadaptationlarge,dao2023flashattention2fasterattentionbetter] and extended context handling [press2022trainshorttestlong,su2023roformerenhancedtransformerrotary] have improved LLMs’ ability to process long-form inputs. However, producing high-quality long-form text remains challenging [deng-etal-2022-model,10.1145/3544548.3581225], due to the scarcity of large-scale human-annotated datasets [köksal2024longformeffectiveinstructiontuning], the inherent subjectivity of extended text evaluation [tan2024proxyqaalternativeframeworkevaluating,wu2025longgenbenchbenchmarkinglongformgeneration], and the complexities of RL-based alignment [greenberg2021detecting].

To mitigate the dataset gap, we leverage linguistically grounded discourse structures and LLM-based evaluation, building on the premise that verifying the correctness of long-form text is easier than generating it [zhang2024generative]. Prior work has addressed RL training instability by employing dense reward shaping [cao-etal-2024-enhancing,chan2024denserewardfreereinforcement]. In a similar vein, we introduce a discourse-aware dense reward scheme (:app_reward_shaping) that assigns rewards based on the structural alignment of generated text with human-like discourse patterns [kim-etal-2024-threads], precomputed from a large corpus.

Structural Organization in Texts

Text structures---such as problem-solution, cause-effect, comparison-contrast, and description---are foundational in shaping coherent, reader-friendly discourse [meyer1975organization]. They guide the flow of information, highlight relationships between ideas, and ensure logical progression. Meanwhile, hierarchical frameworks like Rhetorical Structure Theory (RST) [mann1987rhetorical] and Segmented Discourse Representation Theory (SDRT) [lascarides2007segmented] add deeper layers of structure by explicitly defining the roles and connections among discourse segments.

Figure fig:main_approach: An illustration of our approach to structural alignment. We explore two types of reward modeling: (1) scoring surface-level text structures, and (2) scoring human-like discourse via authorship classifier. Components highlighted in blue remain frozen during RL training, while those in orange are trained.

In this work, we adopt RST [mann1987rhetorical] to define a hierarchical discourse tree, wherein Elementary Discourse Units (EDUs) represent basic phrases, and higher-level nodes merge these units into increasingly complex structures. Each connection (edge) carries a discourse label, such as elaboration, and contrast, revealing how ideas fit together. Crucially, these labeled nodes indicate the text's main ideas (nuclei) and supporting details (satellites), illustrating the text's overall rhetorical organization. Such higher-level structures function as meta-level signals that convey speaker intent, pragmatic cues, and rhetorical strategies [haugh2012speaker,degen2023rational]. By aligning LLMs with these discourse structures, we enable them to maintain thematic continuity, manage contextual shifts, and generate text that is more consistent, context-sensitive, and human-like.

Structural Alignment

The goal of structural alignment is to enable LLMs generate more logically coherent and human-like text. We emphasize that we are introducing a new alignment ``task,'' not a new algorithm, as methods like DPO or GRPO can be readily integrated into our framework. In this study, we achieve this objective through the RLAIF approach within the PPO framework (:rlaif_prelim, :rlaif_struct), exploring two research avenues: (1) employing an evaluator LLM to score surface-level text structures (:app_text_structure), and (2) evaluating human-like discourse via an authorship classifier grounded in hierarchical discourse motifs (:app_graph_structure). Additionally, we propose a reward-shaping scheme that leverages dense discourse motifs to enhance training stability (:app_reward_shaping).

Preliminaries for RLAIF

Reinforcement Learning from AI Feedback (RLAIF) extends the paradigm of using human feedback by employing AI systems to provide evaluative signals for training a policy model [Bai2022ConstitutionalAH]. Under the Proximal Policy Optimization (PPO) framework [schulman2017proximalpolicyoptimizationalgorithms], RLAIF optimizes a policy \(\pi_\theta(a_t | s_t)\) , parameterized by \(\theta\) , to produce actions \(a_t\) (such as generated stories) given states \(s_t\) (such as writing prompts). The AI-generated feedback is translated into scalar reward signals \(r_t\) that quantify the quality of the actions.

The optimization objective in PPO is designed to maximize expected rewards while ensuring that policy updates are stable and do not deviate excessively from the previous policy. This is achieved by maximizing a clipped surrogate objective function:

\[ \begin{aligned} L^{\text{PPO}}(\theta) = \mathbb{E}_t\left[ \min \left( r_t(\theta) \hat{A}_t, \right. \right. \\ \text{clip}\left. \left. \left( r_t(\theta), 1 - \epsilon, 1 + \epsilon \right) \hat{A}_t \right) \right] \end{aligned} \]

where \(r_t() = _(a_t | s_t)__old(a_t | s_t)\) represents the probability ratio between the new and old policies, and () is a hyperparameter that limits the extent of policy updates. The advantage function \(A_t\) estimates the relative benefit of an action compared to the average, and can be computed as:

\[ \hat{A}_t = r_t + \gamma V_{\theta_{\text{old}}}(s_{t+1}) - V_{\theta_{\text{old}}}(s_t) \]

with \(V__old(s_t)\) being the value function of the old policy and () the discount factor. By iteratively performing gradient descent on \(L^PPO()\), the policy parameters () are updated to maximize the expected rewards:

\[ \theta_{\text{new}} = \theta_{\text{old}} + \alpha \nabla_\theta L^{\text{PPO}}(\theta) \]

where () is the learning rate. This iterative process allows the policy to learn from AI feedback, progressively improving its performance while maintaining stability through the clipping mechanism in the PPO objective. Consequently, the policy becomes adept at generating higher-quality outputs as evaluated by AI feedback, reducing the reliance on human annotations and enabling scalable training.

RLAIF for Aligning to Structure

Directly applying RLAIF to long-form text generation poses unique challenges, primarily due to the difficulty of measuring coherence and organization at scale. Longer outputs require complex thematic progression and consistent rhetorical flow, making it hard to distill quality into a single, easily computed reward. Even powerful off-the-shelf LLMs can struggle to provide reliable ``one-shot'' scores for coherence when dealing with multi-paragraph or multi-page content.

Consequently, we adopt two distinct yet complementary scoring approaches, as illustrated in Figure fig:main_approach. The first, introduced in Section sec:app_text_structure, uses an evaluator LLM as a Text Structure Evaluator that assigns a 1–5 rating based on surface-level text structures—though an imperfect proxy, it still captures key signals such as logical transitions and topical balance. The second, described in Section sec:app_graph_structure, offers a more explicit framework by segmenting extended text and extracting theory-grounded ``discourse motifs'' via RST, which are then fed into an Authorship Classifier for binary classification.

Surface-Level Text Scoring

Our first approach to reward modeling employs a powerful off-the-shelf LLM to provide three surface-level text scores grounded in pragmatics and discourse analysis: Logical Flow and Structure; Hierarchical Organization; and Balance and Emphasis. These scores guide the LLM to generate text that is not only grammatically correct but also pragmatically effective in conveying its intended message within the discourse context.

itemize 0em Logical Flow and Structure assesses whether ideas progress logically and the overall organization is coherent, ensuring that the discourse is easy to follow. Hierarchical Organization evaluates how effectively content is structured, transitioning from general concepts to specific details, with each section building upon the previous one to support the main argument. Balance and Emphasis examines whether key ideas are appropriately highlighted and whether different points receive balanced coverage, aligning with pragmatic principles of relevance and informativeness. itemize Each score is assigned an integer value from 0 (lowest quality) to 5 (highest quality), and the final reward is computed as their average. The full prompt instruction for this scoring mechanism is provided in supplementary materials.

Graph-Level Discourse Scoring

Our approach to aligning LLMs with human-like text structure builds on [kim-etal-2024-threads], who developed a methodology for distinguishing human-written from machine-generated text using hierarchical discourse analysis. Their approach, illustrated in Figure fig:graph_level_align employs the RST framework to parse texts into discourse trees, converts these trees into recursive hypergraphs, and extracts discourse motifs---recurring structural patterns---as distinguishing features. These motif distributions serve as inputs for classifiers that effectively detect machine-generated text, even in out-of-distribution and paraphrased cases.

Figure fig:graph_level_align: The process of extracting discourse motif vectors from generated text, as illustrated in [kim-etal-2024-threads]. We extend the original approach by introducing a segmentation step for texts exceeding the input length of the RST parser. Motif vectors are then aggregated across multiple segments.

Building on this framework, we extend the RST discourse parser [liu-etal-2021-dmrst] to accommodate longer contexts by segmenting texts at the paragraph level, with each chunk containing approximately 400–512 tokens---a technical limitation that future work can address. Discourse parsing is applied to each segment to compute distributions of distinctive discourse motifs, capturing global discourse patterns beyond surface-level text. To prevent over-representation of motifs, segments are non-overlapping. These aggregated motif distributions are fed into a Longformer-based [beltagy2020longformerlongdocumenttransformer] authorship classifier [li-etal-2024-mage], where they are concatenated with the model's [CLS] embedding to integrate textual content with explicit structural signals and produce binary classification scores.

Aside from segment-level adaptations, minimal architectural changes are made. The classifier is trained on existing datasets supplemented by curated human-written essays described in Section sec:dataset. By capturing how human-authored texts typically display richer and more varied structural patterns, while machine-generated texts often rely on uniform, sequential prediction strategies, this method remains effective even against paraphrasing attacks that obscure surface cues. The classifier's binary output—differentiating human from machine-authored text—becomes the reward signal in our structural alignment task. This setup allows the model to harness hierarchical discourse structures, thereby retaining global coherence patterns and enhancing authorship classification for long-form text generation.

Reward Shaping with Discourse Motifs

In the standard RLHF framework with PPO, the episodic reward is assigned only at the end of an episode, specifically, to the final token of the model's generation. This sparsity of rewards limits the training signal's granularity, providing no direct feedback on the quality of individual actions (i.e., token selections) throughout the sequence.

Consequently, as RL episode lengths increase, training stability declines due to the delayed credit assignment, higher exploration complexity, and compounding execution errors [pignatelli2024a]. Reward variance also grows, making policy updates noisier and less effective, particularly in policy gradient methods. While techniques like curiosity-driven exploration [pathak2017curiositydrivenexplorationselfsupervisedprediction], imitation learning [reddy2019sqilimitationlearningreinforcement], or curriculum learning [NEURIPS2021_503e7dbb] can help, we instead employ reward shaping to provide more fine-grained signals—taking advantage of the inherently granular structure of language, in particular, discourse trees.

Figure fig:dense_rewards: In addition to the episodic scalar reward, tokens contributing to human-distinctive discourse motifs receive a reward of \(12 num_tokens\) .

We identify specific tokens within each sequence that should receive non-zero token-level rewards, enabling more fine-grained guidance for the policy model. Specifically, in addition to this episodic scalar reward, we assign \(\frac{1}{2 \cdot \text{num\_tokens}}\) reward scores to tokens that contribute to a discourse motif that is more distinctively characteristic of human writing compared to those frequently observed in LLM-generated text (cf. Figure fig:dense_rewards). These distinctive discourse motifs are identified across entire corpora by computing Motif Frequency-Inverse Document Frequency (MF-IDF) scores and selecting motifs that exceed a threshold of at least one standard deviation, following the approach of [kim-etal-2024-threads]. To preserve finer granularity, we compute discourse motifs over EDUs rather than full sentences; each EDU contains, on average, 20 subword tokens.

While the overall PPO training objective remains unchanged, the implementation involves adding these token-level rewards to the corresponding positions in the reward tensor for each sequence. This design ensures gradient updates capture the localized contribution of each token, improving credit assignment throughout generation. As a result, policy improvements typically occur more quickly and with greater stability than when relying on a single end-of-sequence reward [chan2024denserewardfreereinforcement].

Length-Penalty Normalization of Rewards

Noting that existing LLMs often fail to adhere to the desired response length specified in prompts [wu2025longgenbenchbenchmarkinglongformgeneration], we adjust the original score by applying a penalty proportional to the shortfall in response length, but only if the response is shorter than the desired length.

The normalized score, \(S_{\text{n}}\) , based on the response length can be represented as:

\[ S_{\text{n}} = S_{\text{o}} \times \left[1 - \alpha \times \max\left(0, \dfrac{L_{\text{d}} - L_{\text{r}}}{L_{\text{d}}}\right)\right] \]

where \(S_{\text{o}}\) is the original score; \(L_{\text{r}}\) is the length of the generated response; \(L_{\text{d}}\) is the desired length of the response; and \(\alpha\) is the length penalty factor.

Experiments

As shown in Figure fig:main_approach, we investigate two reward modeling approaches within the RLAIF framework using PPO: (1) surface-level text structure scores and (2) graph-level discourse structure scores. To support this, we construct a dataset of prompts by collecting and refining writing prompts for essay generation (:dataset) and validate the quality of the LLM-based Text Structure Evaluator (:verify_rm). Finally, we evaluate structure-aligned LLMs against baseline models trained with standard RLHF using an off-the-shelf general-purpose RM (:struct_align_exp).

Throughout our experiments, we use Qwen2-72B-Instruct-AWQ as the Text Structure Evaluator and Qwen2-1.5B-Instruct as the initial policy model for structural alignment [yang2024qwen2technicalreport]. Due to resource constraints, we selected a 1.5B parameter model as it strikes a balance between being sufficiently powerful and small enough to enable efficient PPO training in data parallelism mode. At the time of our experiments, Qwen 2 models offered state-of-the-art performance in the LLM landscape. For hierarchical discourse analysis, we adopt the DMRST parser [liu-etal-2021-dmrst] as our Hierarchical Discourse Parser, while the Discourse Motif Extractor and Authorship Classifier are based on [kim-etal-2024-threads]. Details on implementation are provided in Appendix app:exp_details.

Constructing an Essay Prompt Dataset

Our work focuses on formal essay writing, as its well-defined rhetorical conventions provide a strong foundation for evaluating structural alignment. To ensure diversity in formal essays, we collected prompts from various domains emphasizing clear logical structures and rhetorical coherence.

Specifically, our PPO training dataset is constructed by refining and integrating multiple existing datasets containing essay prompts, including those from an English proficiency exam, persuasion corpus [crossley2023large], and an argumentation source, namely, the Change My View (CMV) subreddit [10.1145/2872427.2883081].

For CMV-derived samples, we employ an LLMQwen/Qwen2-72B-Instruct to generate essay instructions that require students (or LLMs) to either support or oppose the original poster's opinion. Each CMV discussion yields two types of instructions: one prompting the student to argue in favor of the given claim and the other requiring them to argue against it. Given the nature of CMV discussions, many of these prompts may be inherently controversial.

To ensure an effective evaluation of model performance, 30% of the dataset is randomly sampled as the test set, except for CMV, where a predefined test split is already provided. The final dataset consists of 26,013 samples for training and 4,096 samples for testing.Included in the supplementary material.

Verification of Text Structure Evaluator

To validate the quality of the text scores generated by our LLM-based RM, Text Structure Evaluator, we examined their correlation with human scores using the ``Hewlett Foundation: Automated Essay Scoring'' dataset [asap-aes].We exclude verification of the graph-level discourse scoring approach, as it follows the validated method proposed by [kim-etal-2024-threads]. This dataset contains high school student essays along with scores assigned by expert human graders. For each essay, students responded to a writing prompt, and human annotators assessed their quality.

When analyzing the Pearson correlation between the discourse scores from the surface-level RM and human-assigned scores, we observe moderate positive correlations across key dimensions: coherence (0.47), organization (0.44), balance (0.39), purpose (0.37), and variability (0.39). These values indicate the discourse scores align reasonably well with human judgments in these areas. Additionally, using linear regression, we measured the mean squared error of the predicted scores to be 0.79 within the 0-4 range, translating to an average deviation of approximately 0.88 points from human scores. We determined the correlation results were strong enough to justify using the LLM-based discourse scoring as the RM and proceeded with PPO training.

Structural Alignment

We evaluate the effectiveness of structural alignment by assessing its impact on the quality of generated essays and the similarity of generated summaries to human-written long summaries.

We use ``Base'' to denote the base model, Qwen2-1.5B-Instruct. Models \(\text{SA}_{\text{S}}\) and \(\text{SA}_{\text{G}}\) represent structurally aligned variants of the base model, where alignment is assessed using surface-level textual scores and graph-level discourse structures, respectively. Lastly, \(\text{RLHF}_{\text{OA}}\) denotes the base model aligned with an off-the-shelf RM from OpenAssistantOpenAssistant/reward-model-deberta-v3-large-v2, which was trained to predict human-preferred responses given a question.

Exp #1: Assessment of Generated Essay Quality

Setup.

We configure the models to generate sequences between 400 and 2K tokens. As training progresses within each epoch, the required generation length is gradually increased using a length penalty reward shaping mechanism (:length_pen). For this experiment, we introduce two additional models by further training \(\text{SA}_{\text{S}}\) and \(\text{SA}_{\text{G}}\) using the alternate method in a two-step process, resulting in $SA_SG $ and $ SA_GS$. We evaluated generation quality by randomly sampling 1,000 writing prompts from the test set and generating 700-token essays with our models. Because no gold-standard reference essays exist for these prompts, reference-based evaluation is not feasible. Additionally, assessing such lengthy text manually can be both time-consuming and prone to inconsistencies at longer contexts. In response, we adopted an LLM-as-judge approach, which is increasingly common in the field [chiang-lee-2023-large]. Specifically, we employed GPT-4o-2024-08-06 to assess overall essay quality, presenting all candidate generations in a randomized order to reduce potential bias in ordering.

Figure fig:eval_llm_judge_base: Pairwise evaluation results for six models. $ Base $ is the unaligned Qwen2-1.5B-Instruct model. $ SA_S $ and $ SA_G $ are Base aligned with surface-level and graph-level structural rewards, respectively. $ RLHF_OA $ is Base aligned with an OpenAssistant RLHF reward model. $ SA_S G $ and $ SA_G S $ are the two-stage variants, applying the opposite structural reward in a second alignment step.

Results.

Figure fig:eval_llm_judge_base presents evaluation results comparing generation quality across the various alignment settings. Overall, results show that models trained with structural alignment consistently outperform their baselines, with graph-level alignment providing the most substantial improvements. The two-step alignment process yields slightly higher performance; however, the improvement appears marginal relative to the additional computational cost. Since the two structural alignment methods exhibit low correlation (:corr_s_g), we believe their effects are complementary. Developing more effective multi-reward alignment strategies could further improve performance, making this a promising avenue for future research. We provide two samples of generated essays with target lengths of 350 and 1K tokens in supplementary materials. Overall, the learned policy models effectively utilize explicit discourse connectives and frequently attempt to structure the text by dividing it into sections.

Exp #2: Long-Document Summarization

Setup.

We evaluate the aligned models on the downstream task of summarizing a long document. Specifically, we utilize the GovReport dataset [huang-etal-2021-efficient] which is a large-scale dataset comprising around 19.5K US government reports, each paired with expert-written abstractive summaries. The dataset contains significantly longer documents (averaging 9.4K words) and summaries (553 words) compared to datasets like PubMed and arXiv. We randomly selected 5K reports from the U.S. Government Accountability Office (GAO), ensuring that each report selected for summarization does not exceed 14K tokens in length. To guide the LLM via prompting, we identify key aspects to summarize by analyzing the section titles of the gold summaries (``Highlights''), which typically focus on two main elements: why the report was conducted and what it found.

Table tab:eval_summ: Evaluations on long document summarization (GovReport). R-1 and R-2 measure unigram and bigram overlap, respectively. R-L leverages the longest common subsequence to assess sentence-level structural similarity.

\begin{table}[t!]
\centering\small
\resizebox{0.67\columnwidth}{!}{
\begin{tabular}{l|cccc}
\toprule
Model                                       & R-1   & R-2   & R-L   \\
\midrule
Base                                        & 53.21 & 20.13 & 50.39 \\
Base+$\text{RLHF}_{\text{OA}}$              & 53.25 & 20.25 & 50.47 \\
Base+$\text{SA}_{\text{S}}$           & 55.45 & 21.43 & 52.30 \\
Base+$\text{SA}_{\text{G}}$             & 55.86 & 21.72 & 52.81 \\
\bottomrule
\end{tabular}
}
\caption{Evaluations on long document summarization (\textsc{GovReport}). R-1 and R-2 measure unigram and bigram overlap, respectively. R-L leverages the longest common subsequence to assess sentence-level structural similarity.}

\label{tab:eval_summ}
\end{table}

Results.

Table tab:eval_summ presents the ROUGE scores [lin-2004-rouge] for the generated summaries, demonstrating that structural alignment significantly enhances long-document summarization. Among the approaches, graph-based alignment once again delivers the best overall performance.

Analyses

By incorporating structured reward signals, we observe significant improvements in discourse organization and human-like text structuring.

Effects of Surface-Level Structural Alignment

We qualitatively observe that aligning LLMs with surface-level text structure scores results in (1) increased use of discourse connectives (e.g., therefore, however, in contrast, and moreover), (2) improved logical progression and argumentative flow, and (3) more frequent inclusion of section headings (e.g., Introduction, Claim 1, Claim 2, and Conclusion). A similar pattern is indirectly observed in Figure fig:before_and_after, where surface-level text structure alignment leads to a significant increase in discourse motifs that exemplify hierarchical structures, such as Joint'' andHyperedges.''

Effects of Graph-Level Structural Alignment

Unlike surface-level alignment, graph-level discourse structure alignment focuses on global discourse organization rather than local discourse markers (e.g., connectives). Figure fig:comp_avg_motif_freq tracks the proportion of human-distinctive discourse motifs identified in each training batch, illustrating how these motifs evolve throughout the generation process. The plot shows that (i) their proportion steadily increases during training, and (ii) the trend plateaus after approximately 50 batch steps, suggesting discourse structuring reaches an optimal threshold. Notably, standard RLHF training without structural alignment shows a slow decline in such motif proportions, suggesting that conventional RMs do not reinforce human-like discourse organization as effectively.

Figure fig:comp_avg_motif_freq: Comparison of human-distinctive discourse motif frequencies across batch updates. The blue line (our method) shows an increasing trend, while the red dashed line (standard PPO) fluctuates downward.

Figure fig:before_and_after: Difference in discourse motif distributions before (red) and after (green) surface-level text structure alignment. Notably, we observe a significant increase in the usage of hierarchical relations such as Joint'' andHyperedges (Hyp.)''. Full version is in supplementary materials.

Correlation between Surface and Graph Scores

Figure fig:struct_align_concept visualizes the relationship between surface and graph-level logits for essays generated by the two structurally aligned models and the standard RLHF-aligned model. While the structurally aligned models tend to achieve higher scores within their respective evaluation criteria, there appears to be little to no correlation between surface-level and graph-level scores. Additionally, we observe that essays generated by the standard RLHF-aligned model exhibit wide variance in surface scores and are predominantly classified as machine-generated, as indicated by negative graph logits.

In summary, our findings suggest that (1) Surface-level alignment enhances readability and clarity via explicit structuring techniques; (2) Graph-level alignment fosters deeper coherence and rhetorical sophistication by reinforcing global discourse structure; and (3) These two levels of alignment appear to be complementary, with the two-step process improving overall quality.

Conclusion

We present an RL approach that aligns LLMs with human-like discourse by utilizing two RMs---one for surface fluency, another for global discourse structure---resulting in clearer, more coherent long-form text that outperforms RLHF baselines. Future work includes scaling to larger models and diverse datasets for greater robustness, and incorporating other linguistic frameworks to refine reward signals and enhance performance.

Experiment Details

RL Pipeline Setup

Our Text Structure Evaluator is a 72B-parameter language model, which made an efficient serving framework essential. We used SGLang [zheng2024sglangefficientexecutionstructured] to load the Qwen2-72B-Instruct-AWQ model, where AWQ stands for Activation-aware Weight Quantization [lin2024awqactivationawareweightquantization]. This quantized model can be loaded onto a single NVIDIA A100 80GB GPU. We ran eight separate evaluator instances, each on its own A100 GPU, and communicated with these evaluators through HTTP requests. SGLang handled these batch requests efficiently, allowing us to process multiple queries in parallel.

For training, we used the TRL library [vonwerra2022trl] to perform Proximal Policy Optimization (PPO). Although TRL provides a strong base for reinforcement learning from human or AI feedback, we made extensive modifications to support our online reward mechanism, which involved querying the Text Structure Evaluator in real-time. The modified code, which enables discourse alignment via remote evaluation, will be made publicly available for further research and development.

The actual PPO-based reinforcement learning (RL) training took place on another server with eight NVIDIA A100 80GB GPUs. We set the per-device train batch size to 2, gradient accumulation steps to 4, local rollout forward batch size to 12, and KL coefficient to 0.03. Despite having large VRAM, we could not increase these batch sizes further due to the large output generation lengths (up to 2K tokens) which places substantial memory demands on the system.

Graph-Level Discourse Structure Scoring

We closely followed the approach outlined by [kim-etal-2024-threads] to implement our Graph-Level Discourse Structure Scoring pipeline. Using their curated list of distinctive discourse motifs, we constructed a Discourse Motif Extractor and trained a Longformer-based [beltagy2020longformerlongdocumenttransformer] authorship classifier as described in the original paper. However, we extended the training set for the classification by adding 5K essay generations from Qwen2-1.5B-Instruct as negative samples, in addition to the datasets originally used in [kim-etal-2024-threads].

Training Plots

Figure fig:surface_reward_curves: The curves represent the average reward scores provided by the RM over multiple steps or episodes, with each curve corresponding to models targeting different output lengths.

Figure fig:surface_reward_curves shows the mean RLHF reward for training via the surface-level text structure scoring. The mean reward is the average feedback score over multiple episodes, reflecting how well the model aligns with desired outcomes. We note that different curves represent separate training of the same baseline models with different target lengths of generation. We can see that the policy models learned to optimize its generation after around 7K episodes.

Surface-Level Discourse Motifs Pre- and Post Alignment

Figure fig:discourse_dist: Difference of distribution of size-3 discourse motifs.

Prompt Instruction for LLM as Text Structure Evaluator

lstlisting[backgroundcolor=lightgray] You will act as an English instructor and evaluate the quality of an essay or story written by a student in response to given instructions. When grading, consider the following discourse aspects of the text. - Logical Flow and Structure (flow): Assess the logical progression of ideas and the overall organization of the text, ensuring that it is easy to follow and well-structured. - Hierarchical Organization (organization): Examine the organization of ideas in a hierarchical manner, from general to specific, ensuring that each section supports the main argument or narrative. - Balance and Emphasis (balance): Ensure that important ideas are appropriately emphasized and that there is a balance in the coverage of different points or sections of the text.

For each aspect, you need to assign an integer score from 0 (worst quality) to 5 (best quality). When assigning the score, carefully consider which specific parts of the text relate to each aspect.

Assign lower scores when: - The text is poorly structured and do not conform to the standard of an English essay or a story. - The text contains a lot of non-sensical words such as special tokens or programming code. - The text contains a lot of non-English words. - The text does not fully answer the writing instruction with full content, and therefore, is unfinished.

Your evaluation output should conform to the following JSON format: "flow": int, "organization": int, "balance": int

Write after outputting the JSON result. lstlisting

Prompt Instructions for LLM as Discourse Annotator

lstlisting[language=python,backgroundcolor=lightgray] intro_prompt = """You will act as an English instructor and analyze how a specific target_type is crafted to serve a discourse-related purpose in the overall text. The specific target_type is: target

It is used and can be found in the following scene: scene """

summary_prompt = """First, summarize the target_type in one sentence using no more than 70 words. When finished, append at the end. """

purpose_prompt = """What is the discourse-related purpose of the target_type? Answer as a concise sentence using no more than 50 words without repeating the target_type. When finished, append at the end. """

discourse_label_prompt = """Based on the purpose you found, determine which one of the following discourse labels is most applicable. When finished, append at the end. - Temporal: Describes a relationship where events or propositions are sequenced in time, where one follows another chronologically. - Causation: Refers to a cause-effect relationship where one proposition leads to or results in the other. - Conditional: Represents a relationship where one proposition is contingent on the fulfillment of another. - Purpose: One proposition explains the intention or goal behind another. - Concession: A relationship where one proposition presents an opposing viewpoint or circumstance, but the other still holds. - Contrast: Highlights a difference between two propositions. - Similarity: Describes a relationship where two propositions highlight similar characteristics or situations. - Conjunction: Involves the simple addition of information, where two or more propositions are connected logically. - Disjunction: Represents alternatives between propositions, where one excludes the other. Selected label: """ lstlisting

Generated Samples for Different Target Lengths

A Writing Prompt Example

lstlisting[backgroundcolor=lightgray] Write an essay that argues against the claim that there is no universal right or wrong, and instead, there are just perspectives. In your essay, consider discussing the existence of objective moral truths and the role of reason in discerning right from wrong. Be sure to provide evidence and examples to support your argument. lstlisting

Target Length of 350 Tokens

lstlisting[backgroundcolor=lightgray] The notion that there is no universal right or wrong perspective is a misguided belief that ignores the profound impact of reason and objective moral truths on human behavior and societal development. Historical and contemporary evidence consistently demonstrate the existence of objective moral truths that guide societies towards progress and harmony.

For instance, the Universal Declaration of Human Rights, adopted by the United Nations in 1948, articulates a set of fundamental rights that transcend cultural, economic, and political differences. This document serves as a beacon of hope, guiding nations towards equitable treatment and respect for human dignity. Numerous examples from around the world illustrate how adherence to these principles has led to significant improvements in human rights, economic development, and social cohesion.

Moreover, the role of reason in discerning right from wrong is evident in the scientific method, which has revolutionized fields such as medicine, engineering, and environmental conservation. Empirical evidence and logical reasoning have led to groundbreaking discoveries that have improved human health and quality of life.

However, it is crucial to acknowledge that while objective moral truths exist, they are not absolute. Cultural, historical, and individual perspectives can influence moral judgments and practices. For instance, the widespread acceptance of slavery in the Western world during the 19th century was largely due to prevailing cultural and economic factors rather than objective moral truths.

In conclusion, while there may be no universal right or wrong perspective, objective moral truths continue to guide societies towards progress and human flourishing. Evidence from history and contemporary examples demonstrates the transformative power of reason and collective action in promoting ethical behavior and social harmony. lstlisting

Target Length of 1K Tokens

lstlisting[backgroundcolor=lightgray] The claim that there is no universal right or wrong is a common one, often expressed by those who believe that moral judgments are subjective and vary from person to person. This belief is often referred to as relativism, and it has been a topic of debate for centuries. While some argue that there are objective moral truths that are universally applicable, others contend that morality is a matter of personal opinion and varies from culture to culture.

One of the most prominent proponents of relativism is philosopher John Stuart Mill, who argued that moral judgments are based on individual preferences and that there is no objective standard for what is right or wrong. Mill believed that morality is a matter of personal choice and that individuals should be free to make their own judgments about what is right and wrong based on their own beliefs and values.

However, this view of morality has been criticized for its lack of clarity and for its inability to provide a consistent framework for ethical decision-making. Many argue that morality is not simply a matter of personal preference, but rather it is based on objective moral truths that are shared by all humans.

One example of this is the concept of universal moral principles, such as the prohibition of murder, theft, and lying. These principles are believed to be universally applicable and are considered to be objective moral truths. For example, the prohibition of murder is considered to be a universal moral principle because it is a fundamental aspect of human nature and is recognized by all cultures and societies.

Another example of objective moral truths is the concept of justice. Justice is considered to be a universal moral principle because it is a fundamental aspect of human society and is recognized by all cultures and societies. The idea of justice is based on the belief that everyone should be treated equally and that fairness and equality are essential components of a just society.

Despite the existence of objective moral truths, some argue that morality is still a matter of personal opinion and that there is no objective standard for what is right or wrong. This view is often referred to as moral relativism, and it is based on the idea that moral judgments are based on individual preferences and cultural differences. One of the main criticisms of moral relativism is that it fails to provide a consistent framework for ethical decision-making. Many argue that moral relativism leads to a lack of consistency and inconsistency in moral judgments, as different cultures and societies may have different moral standards and values.

Another criticism of moral relativism is that it undermines the importance of moral education and the development of moral reasoning skills. Without a clear understanding of objective moral truths, individuals may struggle to make ethical decisions and may be more likely to make mistakes or to act in ways that are harmful to others.

In conclusion, the claim that there is no universal right or wrong is a common one, and it is often expressed by those who believe that moral judgments are subjective and vary from person to person.

lstlisting

Acknowledgments

We are grateful to the researchers at Amazon and members of the intern cohort for their valuable feedback during weekly research discussions. We also thank members of Minnesota NLP for their insightful input during group meetings. ZMK is generously supported by the 3M Science and Technology Fellowship and the Doctoral Dissertation Fellowship at the University of Minnesota.