<div dir="ltr"><div dir="ltr">On Mon, May 27, 2019 at 3:37 PM Hardie, Andrew <<a href="mailto:a.hardie@lancaster.ac.uk">a.hardie@lancaster.ac.uk</a>> wrote:<br></div><div dir="ltr"><br></div><div>Hi Andrew,</div><div><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div class="gmail-m_332910913174634553WordSection1">
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">There is a known issue in frequency tables using CI collations, which is that although all diacritics are folded together, the version
that </span><i style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">appears</i><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt"> is the first version that is seen. Using English, if you have [...] </span><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">then what will appear is naive with no
</span><i style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">diaresis</i><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">. (same happens with case).</span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">This issue would seem to be behind the cases you report like “mi” and “mí”. The frequency list rolls them together, so it is just luck
of the draw which shows up. It’s expected behaviour.</span></p></div></div></blockquote><div><br></div><div>I understand that behavior occurring when the various forms are treated as equivalents (e.g. "naive" and "naïve" with <font face="courier new, monospace">%cd</font>). But in my case, "mi" and "mí" actually correspond to different lemmas ("mi" and "mí", respectively). Would this <i>still</i> be the expected behavior in this case?</div><div><br></div><div>If so, does this mean that lemmas that differ only by diacritical marks are treated as one single lemma by one or more parts of CWB/CQPweb? </div><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-GB"><div class="gmail-m_332910913174634553WordSection1"><p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">I think the best thing would be, for one word, to use CQP queries to see exactly what is in the index. IE run the following,</span><br></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span lang="FR" style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">[lemma="que"]<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span lang="FR" style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">[lemma="qúe"]<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span lang="FR" style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">[lemma="qùe"]</span></p></div></div></blockquote><div><br></div><div>Okay. Here are the results:</div><div><ol><li>Your query “[lemma="que"]” returned 314,658 matches in 4,701 different texts<br></li><li>Your query “[lemma="qúe"]” returned 4 matches in 4 different texts<br></li><li>Your query “[lemma="qué"]” returned 12,924 matches in 2,723 different texts<br></li><li>(“[lemma="qùe"]” has 0 results)</li></ol></div><div>1 and 3 are both actual words in Spanish, so both forms are expected. 2 shows that I've got four of these typos in the corpus, which isn't unexpected since this is a speech corpus transcribed by fallible humans. 4 wasn't actually a word I saw in frequency lists, so 0 hits would also be expected here.</div><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-GB"><div class="gmail-m_332910913174634553WordSection1"><p class="MsoNormal" style="margin-left:36pt"><span lang="FR" style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:Verdana,sans-serif;font-size:10pt">That will narrow down the problem IE determine whether it is a CWB issue or a MySQL issue.</span></p></div></div></blockquote><div><br></div><div>Hope this provides enough info to figure that out. If not, just let me know.</div><div><br></div><div>Best wishes,</div><div>Scott</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-GB"><div class="gmail-m_332910913174634553WordSection1"><p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> <a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a> <<a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a>>
<b>On Behalf Of </b>Scott Sadowsky<br>
<b>Sent:</b> 25 May 2019 21:39<br>
<b>To:</b> Open source development of the Corpus WorkBench <<a href="mailto:cwb@sslmit.unibo.it" target="_blank">cwb@sslmit.unibo.it</a>><br>
<b>Cc:</b> Open source development of the Corpus WorkBench <<a href="mailto:CWB@liste.sslmit.unibo.it" target="_blank">CWB@liste.sslmit.unibo.it</a>><br>
<b>Subject:</b> Re: [CWB] Strange issue with character encoding (?) in frequency lists<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sat, May 25, 2019 at 2:20 PM Hardie, Andrew <<a href="mailto:a.hardie@lancaster.ac.uk" target="_blank">a.hardie@lancaster.ac.uk</a>> wrote:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Hi Andrew,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:4.8pt">
<span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">One possibility is that the wrong charset/collation is being activated for the frequency tables. Could you check this?</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:4.8pt">
<span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">If you run create table freq_corpus_<b>nameofyrcorpus</b>_word; the mysql command prompt, then the character set / collation should be stated either for the table as a whole, or
for the “item” column.</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">That shows "<span style="font-family:"Courier New";color:black">ENGINE=InnoDB DEFAULT CHARSET=utf8</span>". All my source texts are UTF8, and the database is created as that too, by the way.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Scott<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> <a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a> <<a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a>>
<b>On Behalf Of </b>Scott Sadowsky<br>
<b>Sent:</b> 25 May 2019 13:45<br>
<b>To:</b> Open source development of the Corpus WorkBench <<a href="mailto:CWB@liste.sslmit.unibo.it" target="_blank">CWB@liste.sslmit.unibo.it</a>><br>
<b>Subject:</b> [CWB] Strange issue with character encoding (?) in frequency lists<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I've run into a strange issue that might have to do with character encoding (or it might not). <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">When I go to Corpus Queries > Frequency lists, select my full corpus, choose to view a list based on lemmas, and then hit Show Frequency List, I get a list of lemmas in which quite a few have phantom accent marks and other diacriticals,
e.g. "sì", "còmo", "èn", "sú", "ïgual", "cúando" (obviously, this is a Spanish corpus).<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">However, when I click on the links for these words and go to the concordance, not a single word has these marks. When I further click through and go to the source texts, the marks also aren't there.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I've grepped through my tagger's dictionary files (FreeLing), and none of these forms exist as lemmas or lexemes. I've also grepped through the *.vrt files that the corpus was compiled from, and none of these forms are present.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I've run into an additional strange issue that is probably related. When I make a subcorpus that is an exact copy of the source corpus, the same problem occurs,but most of the spurious accents and such are
<u>different</u> (e.g. "qúe", "nó", "á", "sì", "én"). <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I'm attaching an edited screenshot that shows the top of the frequency list based on the full corpus on the left and the subcorpus that contains the full corpus on the right, with errors in red boxes. <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><img border="0" width="174" height="502" style="width: 1.8083in; height: 5.2333in;" id="gmail-m_332910913174634553_x0000_i1025" src="cid:16afc97b68f4cff311" alt="Lemmas.png"><u></u><u></u></p>
<p class="MsoNormal">Of course, in some cases both of the highlighted forms exist in Spanish (e.g. #28, "mi" and "mí"), but in spite of being different they have the same frequencies in the corpus and the subcorpus, which further suggests that it's not the
underlying data that's causing this.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Scott<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
CWB mailing list<br>
<a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br>
<a href="http://liste.sslmit.unibo.it/mailman/listinfo/cwb" rel="noreferrer" target="_blank">http://liste.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
</blockquote></div><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.7273px">Dr. Scott Sadowsky<br>Profesor Asistente de Lingüística</div><div dir="ltr" style="font-size:12.7273px">Pontificia Universidad Católica de Chile<br></div><div dir="ltr" style="font-size:12.7273px"><br></div><div dir="ltr" style="font-size:12.7273px">ssadowsky gmail com</div><div dir="ltr" style="font-size:12.7273px">scsadowsky uc cl<br><a href="http://sadowsky.cl/" target="_blank">http://sadowsky.cl/</a></div><div dir="ltr" style="font-size:12.7273px"> </div></div></div></div></div></div></div></div></div></div></div></div>