[{"content":"","date":"30 September 2026","externalUrl":null,"permalink":"/blogs/","section":"Blog","summary":"","title":"Blog","type":"blogs"},{"content":" LLM Decoding and Reranking # LLM decoding methods fall into two families [1]:\nDesign a function fff that acts on the per-step token distribution P(yi∣X,y\u0026lt;i)P(y_i \\mid X, y_{\u0026lt;i})P(yi​∣X,y\u0026lt;i​), and pick tokens one at a time according to f(P(yi∣X,y\u0026lt;i))f\\big(P(y_i \\mid X, y_{\u0026lt;i})\\big)f(P(yi​∣X,y\u0026lt;i​)). Greedy decoding, beam search, and top-kkk/top-ppp sampling all belong to this family. Design a function ggg that acts on the complete output yyy, e.g. g(P(y∣X))g\\big(P(y \\mid X)\\big)g(P(y∣X)), where P(y∣X)=∏iP(yi∣X,y\u0026lt;i)P(y \\mid X) = \\prod_i P(y_i \\mid X, y_{\u0026lt;i})P(y∣X)=∏i​P(yi​∣X,y\u0026lt;i​). This is a two-step process: first let the LLM generate many candidates, then use ggg to select the best one. For this reason, this kind of decoding is often called reranking. Application to Machine Translation # In machine translation, we denote the source sentence by sss, the model translation by yyy, the reference translation by yrefy^{\\mathrm{ref}}yref, and the translation quality metric by q(y,yref)q(y, y^{\\mathrm{ref}})q(y,yref).\nReranking in machine translation rests on a simple intuition: if we generate several candidates instead of just one, we can always pick a better one. We denote the candidate set by C\\mathcal{C}C. The two most widely used reranking methods in machine translation are MBR decoding and QE reranking.\nIn the WMT General Machine Translation Shared Task, both reranking methods have repeatedly been adopted by winning systems [2].\nMBR Decoding # This method is based on Bayesian decision theory: it selects the candidate that \u0026ldquo;agrees most, on average\u0026rdquo; with the other candidates. The function that measures this agreement is called the utility, written u(h,hs)u(h, h_s)u(h,hs​), which scores how good hhh is when hsh_shs​ is treated as the reference. The decision rule is:\ny^MBR=argmaxh∈C1∣C∣∑hs∈Cu(h,hs). \\hat{y}_{\\mathrm{MBR}} = \\mathop{\\mathrm{argmax}}\\limits_{h\\in\\mathcal{C}} \\frac{1}{|\\mathcal{C}|} \\sum_{h_s\\in\\mathcal{C}} u(h,h_s). y^​MBR​=h∈Cargmax​∣C∣1​hs​∈C∑​u(h,hs​).The true reference yrefy^{\\mathrm{ref}}yref is not available at decoding time, so the other candidates hsh_shs​ are used as pseudo-references. The utility is therefore usually a reference-based metric (e.g. COMET, BLEURT, chrF++): we simply replace the reference in q(y,yref)q(y, y^{\\mathrm{ref}})q(y,yref) with a pseudo-reference, u(h,hs)=q(h,hs)u(h, h_s) = q(h, h_s)u(h,hs​)=q(h,hs​). The average in the formula is the expected utility of hhh; choosing the candidate with the highest expected utility caters to the preferences of the metric and yields a better output.\nIn the rest of this post, whenever MBR is involved, \u0026ldquo;utility\u0026rdquo; refers to this reference-based metric, and a \u0026ldquo;utility call\u0026rdquo; means one call to that metric.\nQE Reranking # This method directly scores each candidate with a reference-free model (e.g. CometKiwi) and selects:\ny^QE=argmaxh∈CrQE(h;s). \\hat{y}_{\\mathrm{QE}} = \\mathop{\\mathrm{argmax}}\\limits_{h\\in\\mathcal{C}} r_{\\mathrm{QE}}(h;s). y^​QE​=h∈Cargmax​rQE​(h;s). Computational Cost # From an end-to-end perspective, the computation required for each source is:\nCandidate generation: ∣C∣|\\mathcal{C}|∣C∣ LLM generations With MBR decoding: ∣C∣2|\\mathcal{C}|^2∣C∣2 utility (reference-based metric) calls With QE reranking: ∣C∣|\\mathcal{C}|∣C∣ QE model (reference-free metric) calls Why Does Acceleration Matter? # Take ∣C∣=256|\\mathcal{C}|=256∣C∣=256: translating a single sentence means generating 256 candidates, and MBR then needs 2562=65,536256^2 = 65{,}5362562=65,536 utility calls. On top of that, today\u0026rsquo;s utilities are usually neural models, so calling them this many times costs a lot of compute. The quality gains of reranking are well established, but at this cost it fits competitions and offline evaluation far better than real systems with latency and compute budgets.\nI care less about squeezing out another fraction of a point on a leaderboard than about whether a method can actually be used. Reranking already works well; what holds it back is mostly cost, so this post collects several ways to bring that cost down.\nThere are two directions for acceleration: reduce the number of candidate generations, or reduce the number of utility / QE model calls.\nAcceleration Methods # Method Applies to Computation reduced Key idea Quit [3] MBR / QE Candidate generation + scoring Generate incrementally; stop early once the best score stabilizes PMBR [4] MBR Utility calls Compute only part of the score matrix; recover the rest via low-rank matrix completion PruneMBR [5] MBR Utility calls Turn \u0026ldquo;pick the highest score\u0026rdquo; into \u0026ldquo;estimate each candidate\u0026rsquo;s probability of winning\u0026rdquo;; estimate it with bootstrap and prune CBMBR [6] MBR Utility calls Cluster the pseudo-references and use cluster centroids instead of all of them; equally weighted centroids also reduce multimodal bias BayesOpt [7] QE QE model calls Model scores with a Gaussian process; only score candidates worth scoring A Unified Method: Quit # Most existing acceleration methods target only the scoring stage of MBR and QE, leaving candidate generation itself largely untouched. Yet in today\u0026rsquo;s LLM-based translation systems, candidate generation is often more expensive than reranking. Quit (Quantifying Uncertainty for Incremental Termination) [3] reduces the cost of both candidate generation and reranking by treating the whole \u0026ldquo;generate-then-rerank\u0026rdquo; pipeline as a sequence of decisions.\nIncremental generation. At each step, generate kkk new candidates (k=8k=8k=8 in the paper); after step iii the candidate set is Ci\\mathcal{C}_iCi​ with ∣Ci∣=ik|\\mathcal{C}_i| = ik∣Ci​∣=ik (for MBR, the pseudo-reference set grows along with it). Let y^i\\hat{y}_iy^​i​ be the output selected by the reranker at step iii and Ri(s)R_i(s)Ri​(s) its score, i.e. the current best score; the full budget Nmax⁡N_{\\max}Nmax​ corresponds to Nmax⁡/kN_{\\max}/kNmax​/k steps.\nIdeal objective: the early-stopping risk. Stopping at step iii costs the true quality gap between its output and the full-budget output:\nLi(s)=∣ q(y^i,yref)−q(y^Nmax⁡/k,yref) ∣. \\mathcal{L}_i(s) = \\big|\\, q(\\hat{y}_i, y^{\\mathrm{ref}}) - q(\\hat{y}_{N_{\\max}/k}, y^{\\mathrm{ref}}) \\,\\big|. Li​(s)=​q(y^​i​,yref)−q(y^​Nmax​/k​,yref)​.The smaller Li(s)\\mathcal{L}_i(s)Li​(s), the less quality is lost by stopping early. But at inference time neither the reference nor the full-budget output is available, so it cannot be computed.\nApproximation 1: replace true quality with the reranker score. This removes the dependence on the reference:\nLiproxy(s)=∣ Ri(s)−RNmax⁡/k(s) ∣. \\mathcal{L}^{\\mathrm{proxy}}_i(s) = \\big|\\, R_i(s) - R_{N_{\\max}/k}(s) \\,\\big|. Liproxy​(s)=​Ri​(s)−RNmax​/k​(s)​.But RNmax⁡/k(s)R_{N_{\\max}/k}(s)RNmax​/k​(s) is only known once the full budget is used, so this is still not computable.\nApproximation 2: replace the gap to the end with recent fluctuation. Partition the steps into non-overlapping windows Bb\\mathcal{B}_bBb​ of size www and look only at the range of the best score within a window:\nΔbR(s)=max⁡j∈BbRj(s)−min⁡j∈BbRj(s). \\Delta_b R(s) = \\max_{j\\in\\mathcal{B}_b} R_j(s) - \\min_{j\\in\\mathcal{B}_b} R_j(s). Δb​R(s)=j∈Bb​max​Rj​(s)−j∈Bb​min​Rj​(s).A large ΔbR(s)\\Delta_b R(s)Δb​R(s) means the best score is still improving noticeably; a small one means it has locally stabilized and further generation has limited benefit. The paper verifies with PRR that it correlates with the true risk Li(s)\\mathcal{L}_i(s)Li​(s).\nStopping rule. Stop once the fluctuation falls below a threshold α\\alphaα (10−310^{-3}10−3 in the paper):\nT=min⁡{ b⋅w:ΔbR(s)≤α }. T = \\min\\{\\, b\\cdot w : \\Delta_b R(s) \\le \\alpha \\,\\}. T=min{b⋅w:Δb​R(s)≤α}.Across 3 NMT models and 19 language pairs, Quit achieves end-to-end speedups of 1.47–2.66× for MBR and 3.43–4.12× for QE reranking while preserving translation quality.\nMBR Decoding: Reducing Utility Calls # The bottleneck of MBR is computing a ∣C∣×∣C∣|\\mathcal{C}| \\times |\\mathcal{C}|∣C∣×∣C∣ score matrix MMM with Mij=u(hi,hj)M_{ij} = u(h_i, h_j)Mij​=u(hi​,hj​). The three methods below reduce this cost from three angles: computing fewer matrix entries, evaluating fewer candidates, and using fewer pseudo-references.\nPMBR # PMBR [4] observes, and verifies experimentally, that the score matrix MMM has a low-rank structure, so MBR can be cast as a matrix completion problem:\nCompute only a random subset of the entries of MMM; Use Alternating Least Squares (ALS) to factorize M≈UV⊤M \\approx UV^\\topM≈UV⊤ and fill in the missing entries; Average each row of the completed matrix and select the MBR output. On WMT22 (en↔de, en↔ru), it matches the COMET22 score of vanilla MBR with only 1/161/161/16 of the utility computations, and outperforms other approximation baselines.\nPruneMBR # The core idea of PruneMBR [5] (EMNLP 2023 Best Short Paper): we don\u0026rsquo;t need each candidate\u0026rsquo;s exact expected utility, only whether it can still win.\nWrite the candidate set as HHH and the expected utility as U(y,Y)=Ey^∼Y[u(y,y^)]U(y, \\mathcal{Y}) = \\mathbb{E}_{\\hat{y}\\sim\\mathcal{Y}}[u(y,\\hat{y})]U(y,Y)=Ey^​∼Y​[u(y,y^​)]. The true MBR objective is argmaxy∈HU(y,pθ(⋅∣x))\\mathop{\\mathrm{argmax}}_{y\\in H} U\\big(y, p_\\theta(\\cdot\\mid x)\\big)argmaxy∈H​U(y,pθ​(⋅∣x)); standard MBR approximates pθp_\\thetapθ​ with pseudo-references RRR, costing ∣H∣∣R∣|H||R|∣H∣∣R∣ utility calls. PruneMBR replaces \u0026ldquo;pick the highest score\u0026rdquo; with estimating each candidate\u0026rsquo;s probability of being the true winner:\np(⋀yˉ∈HU(y,pθ(⋅∣x))≥U(yˉ,pθ(⋅∣x))). p\\Big(\\bigwedge_{\\bar{y}\\in H} U\\big(y, p_\\theta(\\cdot\\mid x)\\big) \\ge U\\big(\\bar{y}, p_\\theta(\\cdot\\mid x)\\big)\\Big). p(yˉ​∈H⋀​U(y,pθ​(⋅∣x))≥U(yˉ​,pθ​(⋅∣x))).Approximation 1: bootstrap. Since pθp_\\thetapθ​ is unknown, resample the current pseudo-references RtR_tRt​ with replacement and approximate it by \u0026ldquo;winning on a bootstrap sample\u0026rdquo;:\nER^t∼boot(Rt) 1(⋀yˉ∈HtU(y,R^t)≥U(yˉ,R^t)). \\mathbb{E}_{\\hat{R}_t\\sim\\mathrm{boot}(R_t)}\\, \\mathbb{1}\\Big(\\bigwedge_{\\bar{y}\\in H_t} U(y,\\hat{R}_t) \\ge U(\\bar{y},\\hat{R}_t)\\Big). ER^t​∼boot(Rt​)​1(yˉ​∈Ht​⋀​U(y,R^t​)≥U(yˉ​,R^t​)).Approximation 2: compare only with the current winner. When HtH_tHt​ is large this has high variance, so compare only with the current winner yˉˉ=argmaxyˉ∈HtU(yˉ,Rt)\\bar{\\bar{y}} = \\mathop{\\mathrm{argmax}}_{\\bar{y}\\in H_t} U(\\bar{y}, R_t)yˉ​ˉ​=argmaxyˉ​∈Ht​​U(yˉ​,Rt​):\nER^t∼boot(Rt) 1(U(y,R^t)≥U(yˉˉ,R^t)). \\mathbb{E}_{\\hat{R}_t\\sim\\mathrm{boot}(R_t)}\\, \\mathbb{1}\\big(U(y,\\hat{R}_t) \\ge U(\\bar{\\bar{y}},\\hat{R}_t)\\big). ER^t​∼boot(Rt​)​1(U(y,R^t​)≥U(yˉ​ˉ​,R^t​)).Beating everyone implies beating yˉˉ\\bar{\\bar{y}}yˉ​ˉ​, so this is an upper bound of the previous quantity and independent of ∣Ht∣|H_t|∣Ht​∣. If it falls below 1−α1-\\alpha1−α, yyy is pruned. Bootstrap only recombines utilities already computed, so it needs no extra calls.\nAlgorithm. The number of pseudo-references doubles according to a schedule (from 8 for COMET, up to 256). Each step adds pseudo-references, computes new utilities only for the surviving candidates, and prunes via bootstrap; it stops when one candidate remains or the last step is reached. Too few initial pseudo-references bias the bootstrap and risk pruning the winner, so r1r_1r1​ sets the speed–accuracy trade-off.\nResults: statistically indistinguishable from standard MBR, with at least 7× fewer utility calls for chrF++ and 15× for COMET.\nCBMBR # CBMBR [6] rests on an elegant idea: pseudo-references are just vectors in feature space, so nearby ones can be merged into a single representative point.\nCOMET encodes the source, candidate, and reference separately with an encoder fff, then scores them with an output layer ϕ\\phiϕ: u(h,hs)=ϕ(f(s),f(h),f(hs))u(h, h_s) = \\phi\\big(f(s), f(h), f(h_s)\\big)u(h,hs​)=ϕ(f(s),f(h),f(hs​)). Encoding takes only ∣C∣|\\mathcal{C}|∣C∣ passes; the bottleneck is the ∣C∣2|\\mathcal{C}|^2∣C∣2 evaluations of ϕ\\phiϕ. The method relies on two assumptions:\nSemantically similar sentences have nearby vectors. On STS-B, COMET\u0026rsquo;s sentence vectors reach a Pearson correlation of 73.6 with human similarity, even above LaBSE (72.7). ϕ\\phiϕ is approximately linear. Within a cluster Ki\\mathcal{K}_iKi​, \u0026ldquo;score, then average\u0026rdquo; can then be swapped for \u0026ldquo;average, then score\u0026rdquo;: 1∣Ki∣∑hs∈Kiϕ(f(s),f(h),f(hs))≈ϕ(f(s),f(h),ci),ci=1∣Ki∣∑hs∈Kif(hs). \\frac{1}{|\\mathcal{K}_i|}\\sum_{h_s\\in\\mathcal{K}_i}\\phi\\big(f(s), f(h), f(h_s)\\big) \\approx \\phi\\big(f(s), f(h), c_i\\big),\\quad c_i = \\frac{1}{|\\mathcal{K}_i|}\\sum_{h_s\\in\\mathcal{K}_i} f(h_s). ∣Ki​∣1​hs​∈Ki​∑​ϕ(f(s),f(h),f(hs​))≈ϕ(f(s),f(h),ci​),ci​=∣Ki​∣1​hs​∈Ki​∑​f(hs​).Concretely, the pseudo-reference vectors are clustered into kkk clusters with kmeans++ and k-means (k=64k=64k=64 in the paper), and\ny^CBMBR=argmaxh∈C1k∑i=1kϕ(f(s),f(h),ci), \\hat{y}_{\\mathrm{CBMBR}} = \\mathop{\\mathrm{argmax}}\\limits_{h\\in\\mathcal{C}} \\frac{1}{k}\\sum_{i=1}^{k} \\phi\\big(f(s), f(h), c_i\\big), y^​CBMBR​=h∈Cargmax​k1​i=1∑k​ϕ(f(s),f(h),ci​),reducing the complexity from O(∣C∣2)O(|\\mathcal{C}|^2)O(∣C∣2) to O(∣C∣ k)O(|\\mathcal{C}|\\,k)O(∣C∣k).\nWhy can it beat standard MBR? The formula averages the centroids with equal weights. Standard MBR treats every pseudo-reference equally, so when candidates are multimodal (e.g. from several systems), the translation with the most samples dominates. CBMBR gives each cluster one vote and is more robust to this bias. An ablation supports this: CBMBRcnt_{\\mathrm{cnt}}cnt​, which weights centroids by cluster size and thus approximates standard MBR more closely, scores almost the same as standard MBR (86.6 vs 86.7) and below CBMBR (87.0).\nResults. With diverse candidates, it stays within 0.1 COMET of standard MBR while speeding up the expected-utility computation 5.7× (1.8× end to end). With multi-system candidates, it beats standard MBR by up to 0.5 COMET. Its limitation is that it only applies to metrics that encode each sentence independently.\nQE Reranking: Reducing QE Model Calls # BayesOpt # QE reranking needs only ∣C∣|\\mathcal{C}|∣C∣ calls, but QE models (e.g. CometKiwi) are themselves expensive. Cheng et al. [7] cast reranking as a Bayesian optimization problem: the goal is to find argmaxh∈CrQE(h;s)\\mathop{\\mathrm{argmax}}_{h\\in\\mathcal{C}} r_{\\mathrm{QE}}(h;s)argmaxh∈C​rQE​(h;s) without scoring every candidate.\nModel the scores of unscored candidates with a Gaussian process, using an RBF kernel over candidate sentence vectors (mean-pooled representations obtained as a by-product of generation, at almost no extra cost); At each step, use an acquisition function (Expected Improvement) to balance exploration (candidates that differ from those already scored) and exploitation (candidates similar to high-scoring ones), and pick the next candidate to score; When the scoring budget is exhausted, return the highest-scoring candidate among those scored. With 200 candidates, it needs only about 70 CometKiwi calls on average to match the result of scoring a random subset of 180 candidates. The paper also proposes a multi-fidelity variant that uses a cheaper, smaller QE model as a proxy to further improve efficiency.\nReferences # CMU Advanced NLP Spring 2025 (7): Decoding Algorithms. YouTube Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, et al. Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets. WMT 2025. ACL Anthology Guangyu Chen, Boxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura. Quit While You\u0026rsquo;re Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking. arXiv:2609.00588, 2026. arXiv Firas Trabelsi, David Vilar, Mara Finkelstein, Markus Freitag. Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms. arXiv:2406.02832, 2024. arXiv Julius Cheng, Andreas Vlachos. Faster Minimum Bayes Risk Decoding with Confidence-based Pruning. EMNLP 2023. ACL Anthology Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe, Hideki Tanaka, Masao Utiyama. Centroid-Based Efficient Minimum Bayes Risk Decoding. Findings of ACL 2024. ACL Anthology Julius Cheng, Maike Züfle, Vilém Zouhar, Andreas Vlachos. A Bayesian Optimization Approach to Machine Translation Reranking. NAACL 2025. ACL Anthology ","date":"30 September 2026","externalUrl":null,"permalink":"/blogs/efficient-mt-reranking/","section":"Blog","summary":"A summarization of efficient machine translation reranking methods.","title":"Efficient Machine Translation Reranking","type":"blogs"},{"content":" About # I\u0026rsquo;m passionate about applied NLP, including machine translation, LLM agents, and RAG pipelines. In research, I focus on making machine translation more efficient and usable, currently by reducing its computational cost; in my previous work, I aimed to translate real-world problems into technical ones and turn ideas into reliable, practical LLM agent systems that create real value.\nUpdates # 2026.09 — New blog post: Efficient Machine Translation Reranking. 2026.09 — I launched my personal page. ","date":"30 September 2026","externalUrl":null,"permalink":"/","section":"Home","summary":"","title":"Home","type":"page"},{"content":"","date":"30 September 2026","externalUrl":null,"permalink":"/tags/llm-decoding/","section":"Tags","summary":"","title":"LLM Decoding","type":"tags"},{"content":"","date":"30 September 2026","externalUrl":null,"permalink":"/tags/machine-translation/","section":"Tags","summary":"","title":"Machine Translation","type":"tags"},{"content":"","date":"30 September 2026","externalUrl":null,"permalink":"/tags/reranking/","section":"Tags","summary":"","title":"Reranking","type":"tags"},{"content":"","date":"30 September 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":" Education # Master\u0026rsquo;s, Information and Communications Engineering, School of Engineering, Institute of Science Tokyo · Okumura–Funakoshi Lab — 2026.04 – Present Bachelor\u0026rsquo;s, Artificial Intelligence, College of Computer and Data Science, Fuzhou University — 2021.09 – 2025.06 Experience # AI Application Engineer, CATL — 2025.08 – 2026.03 Research # Preprints # Quit While You\u0026rsquo;re Ahead: Quit for Efficient Candidate Generation in Machine Translation Reranking\nGuangyu Chen, Boxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura\narXiv preprint, 2026 · arXiv · PDF Projects # mllm-prompt-template — A Python template library for multimodal LLMs that extends string.Template to support image substitution in prompts.\nGitHub · PyPI · Download stats · ","externalUrl":null,"permalink":"/profile/","section":"Home","summary":"","title":"Profile","type":"page"}]