[CWB] Strange issue with character encoding (?) in frequency lists

[Hardie, Andrew]

Hi Scott,

One possibility is that the wrong charset/collation is being activated for the frequency tables. Could you check this?

If you run

          show create table freq_corpus_nameofyrcorpus_word;

… at the mysql command prompt, then the character set / collation should be stated either for the table as a whole, or for the “item” column.

I have been fiddling with the code for collations lately but trying to do so in such a way as to not disturb current functions … it’s possible something got past me though.

best

Andrew.

From: cwb-bounces at sslmit.unibo.it <cwb-bounces at sslmit.unibo.it> On Behalf Of Scott Sadowsky
Sent: 25 May 2019 13:45
To: Open source development of the Corpus WorkBench <CWB at liste.sslmit.unibo.it>
Subject: [CWB] Strange issue with character encoding (?) in frequency lists

I've run into a strange issue that might have to do with character encoding (or it might not).

When I go to Corpus Queries > Frequency lists, select my full corpus, choose to view a list based on lemmas, and then hit Show Frequency List, I get a list of lemmas in which quite a few have phantom accent marks and other diacriticals, e.g. "sì", "còmo", "èn", "sú", "ïgual", "cúando" (obviously, this is a Spanish corpus).

However, when I click on the links for these words and go to the concordance, not a single word has these marks. When I further click through and go to the source texts, the marks also aren't there.

I've grepped through my tagger's dictionary files (FreeLing), and none of these forms exist as lemmas or lexemes. I've also grepped through the *.vrt files that the corpus was compiled from, and none of these forms are present.

I've run into an additional strange issue that is probably related. When I make a subcorpus that is an exact copy of the source corpus, the same problem occurs,but most of the spurious accents and such are different (e.g. "qúe", "nó", "á", "sì", "én").

I'm attaching an edited screenshot that shows the top of the frequency list based on the full corpus on the left and the subcorpus that contains the full corpus on the right, with errors in red boxes.

[Lemmas.png]
Of course, in some cases both of the highlighted forms exist in Spanish (e.g. #28, "mi" and "mí"), but in spite of being different they have the same frequencies in the corpus and the subcorpus, which further suggests that it's not the underlying data that's causing this.

Best,
Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://liste.sslmit.unibo.it/pipermail/cwb/attachments/20190525/3bda53f0/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 102890 bytes
Desc: image001.png
URL: <http://liste.sslmit.unibo.it/pipermail/cwb/attachments/20190525/3bda53f0/attachment-0003.png>