<div class="markdown_content"><ul>
<li><strong>status</strong>: open --> closed-fixed</li>
<li><strong>Comment</strong>:</li>
</ul>
<p>fixed in commit 1373</p>
<hr/>
<p><strong> <a class="alink strikethrough" href="https://sourceforge.net/p/cwb/bugs/70/">[bugs:#70]</a> cwb-encode breaks UTF-8 when truncating long tokens</strong></p>
<p><strong>Status:</strong> closed-fixed<br/>
<strong>Group:</strong> TODO-3.5<br/>
<strong>Created:</strong> Tue Dec 03, 2019 02:03 PM UTC by Stefan Evert<br/>
<strong>Last Updated:</strong> Tue Dec 03, 2019 02:03 PM UTC<br/>
<strong>Owner:</strong> Andrew Hardie</p>
<p>cwb-encode truncates tokens exceeding the length limit CL_MAX_LINE_LENGTH, but does so by cutting of at the last possible byte position and adding a "$" marker (around line #1520 in the source code). This can break UTF-8 codepoints, leading to invalid tokens. Note that UTF-8 validation is carried out for the entire input line and does not detect broken UTF-8 introduced later.</p>
<p>Proposed solution: Truncation should find the last complete UTF-8 codepoint within the length limit and truncate there.</p>
<hr/>
<p>Sent from sourceforge.net because cwb@sslmit.unibo.it is subscribed to <a href="https://sourceforge.net/p/cwb/bugs/">https://sourceforge.net/p/cwb/bugs/</a></p>
<p>To unsubscribe from further messages, a project admin can change settings at <a href="https://sourceforge.net/p/cwb/admin/bugs/options.">https://sourceforge.net/p/cwb/admin/bugs/options.</a> Or, if this is a mailing list, you can unsubscribe from the mailing list.</p></div>