<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial">Hi Andrew,<br><span style="background-color: rgb(255, 255, 255);"></span><span style="color: rgb(0, 0, 0);"><span style="background-color: rgb(255, 255, 255); color: rgb(0, 0, 0);">Thanks for the new commit. I recompiled v 3.4.4 and </span></span><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><span style="color: rgb(0, 0, 0);"><span style="background-color: rgb(255, 255, 255); color: rgb(0, 0, 0);">cwb-scan-corpus complains no more. </span><br style="background-color: rgb(255, 255, 255);"><span style="background-color: rgb(255, 255, 255); color: rgb(0, 0, 0);">The result is however mixed, using my corpus posted earlier. Good news first.<br><br>Success 1:Standard Query->Start Query (OK)<br>Success 2: Restricted query (CQP syntax) (OK)<br><br>The bad news is that the frequency list reads like gibberish to me (definitely not Chinese).<br>Issue 1:Standard query->Collocation->Create collocation database->Collocation controls.<br><br>NO. word<br>1 。<br>2 ä¹ æƒ¯<br>...<br>Issue 2: Frequency lists->Show frequency list<br>No. Word Frequency<br>1 çš„ 3<br>2 了 2<br>...<br><br>Also , no word in the Frequency list page can be linked back to its concordance view.<br><br>After checking the freq_corpus_test_word table, I can see the item column contains just gibberish there. That might be able to explain something.<br><br>Meanwhile, I also want to know how sorting is done for languages other than English. For Chinese, there are usually two types of sorting: PINYIN(bopomofa) and character strokes. Is it possible to do that kind of thing in CQPweb? If not present in CQPweb yet, is there an interface (even just envisaged)to do so?<br><br>BTW: I will check your update for CQPweb a moment later and will post my findings in that thread. Thanks.<br><br>Best,<br>Ray<br><br><br></span></span><span style="background-color: rgb(255, 255, 255);"></span><br></span><div id="divNeteaseMailCard"></div><br>At 2012-05-27 19:13:56,"Hardie, Andrew" <a.hardie@lancaster.ac.uk> wrote:<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<style></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">Hi Ray,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">The segfault had nothing to do with your data, it was an internal structure used by cwb-scan-corpus for pattern matching. There is a fix in SVN now; recompile
v 3.4.4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">best<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">Andrew.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> <a href="mailto:cwb-bounces@sslmit.unibo.it">cwb-bounces@sslmit.unibo.it</a> [mailto:<a href="mailto:cwb-bounces@sslmit.unibo.it">cwb-bounces@sslmit.unibo.it</a>]
<b>On Behalf Of </b>Ray Wu<br>
<b>Sent:</b> 26 May 2012 13:49<br>
<b>To:</b> Open source development of the Corpus WorkBench<br>
<b>Subject:</b> [CWB] CQPweb 3.0.7 on CWB 3.4.3 cwb-scan-corpus error! Segmentation fault<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Hi all,<br>
I want to make CQPweb to process Chinese (my native tongue) on my Ubuntu 8.04, so I updated to CWB 3.4.3. The compiling process was successful and I could query a tiny Chinese text via cqp from the terminal.<br>
<br>
I could also load the Chinese text into CQPweb and finished part of the metadata page. But when I wanted to "Manage metadata->Create frequency tables", CQPweb complained and says it encountered an error and could not continue. Here is the error message:<br>
cwb-scan-corpus error! Segmentation fault<br>
... in file /usr/local/apache2/htdocs/cqp/lib/freqtable.inc.php line 100. <br>
<br>
This sounds strange to me as I have browsed the entire archived mailing list and get to know that error message is mostly likely to happen when a token is too long. But my toy corpus is just a few lines long. I tried it on an small English text and the same
situation occurs.<br>
<br>
To make the picture clearer, I will try to illustrate my experiment by listing what I have done.<br>
<br>
My compiling context for CWB 3.4.3: CWB from svn: 3.4.3; PCRE: 7.4; glib-2.0; gcc: Ubuntu 4.2.4-1ubuntu3.
<br>
<br>
The compiling process seemed normal and I could build a tiny Chinese corpus using the following text (See the end of the post). Hopefully it can make it through the wild net to your computer remaining intelligible):<br>
<br>
I then ran the following to index it:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-encode -c utf8 -d /home/ray/cqputf8 -f cqpweb_chinese_test_utf8.txt -R /usr/local/share/cwb/registry/test -P pos -S
text -S s -S text_id</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Annotations of s-attribute <text> not stored (file cqpweb_chinese_test_utf8.txt, line #1, warning issued only once).</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-makeall -V TEST (everyting says OK)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-huffcode -A TEST (fine, nothing wrong)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-compress-rdx -A TEST (fine again)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">I queried the new corpus and nothing broken:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cqp -eC</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">[no corpus]> TEST</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="DE">TEST> "</span><span style="font-family:SimSun;color:black">了</span><span style="color:black" lang="DE">";</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="DE"> 7:
</span><span style="font-family:SimSun;color:black">们</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">的</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">行为</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">也</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">引来</span><span style="color:black" lang="DE"> <</span><span style="font-family:SimSun;color:black">了</span><span style="color:black" lang="DE">>
</span><span style="font-family:SimSun;color:black">不少</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">公园</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">游客</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">的</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="DE"> 29:
</span><span style="font-family:SimSun;color:black">，</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">他们</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">早已</span><span style="color:black">
</span><span style="font-family:SimSun;color:black">习惯</span><span style="color:black" lang="DE"> <</span><span style="font-family:SimSun;color:black">了</span><span style="color:black" lang="DE">>
</span><span style="font-family:SimSun;color:black">。</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">TEST> <s> []* "</span><span style="font-family:SimSun;color:black">了</span><span style="color:black" lang="EN-US">" []* </s>; (query is OK)
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US"><br>
Finally, I resorted to run </span><span style="color:black" lang="EN-US"> cwb-scan-corpus manually and did find something usual:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-scan-corpus -C TEST pos (fully OK)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-scan-corpus TEST pos+0 pos+1 (segmentation fault)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">ray@ray-laptop:~$ cwb-scan-corpus TEST pos+0 pos+1 pos+2</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Scanning corpus TEST for 3-tuples ...
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Scan complete. </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Printing frequency table on stdout ...
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="PL">...</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:SimSun;color:black">段错误</span><span style="color:black" lang="PL"> ("segmentation fault" in English)</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">I have very little knowledge in C so I cannot go further to investigate more.
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Does anyone know where the problem is? Thanks for any input.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Best,
<br>
Ray</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Hunan University of Commerce, China</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">PS: My computer parameters:</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">System: Ubuntu 8.04<br>
Apache: 2.0.63<br>
MySQL: 5.0.88<br>
PHP: 5.2.12 (lower than expected 5.3.0)<br>
Perl: 5.8.8<br>
CWB: 3.4.3 (compiled from svn source)<br>
Linux utilites: awk, tar, gzip, iconv</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">LANG=zh_CN.UTF-8</span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-bottom:double windowtext 2.25pt;padding:0cm 0cm 1.0pt 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">GDM_LANG=zh_CN.UTF-8</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US">Inside cqpweb_chinese_test_utf8.txt:
</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black" lang="EN-US"><text id="test"><br>
<s><br>
</span><span style="font-family:SimSun;color:black">这些</span><span style="color:black" lang="EN-US"> r<br>
</span><span style="font-family:SimSun;color:black">网友</span><span style="color:black" lang="EN-US"> n<br>
</span><span style="font-family:SimSun;color:black">们</span><span style="color:black" lang="EN-US"> k<br>
</span><span style="font-family:SimSun;color:black">的</span><span style="color:black" lang="EN-US"> u<br>
</span><span style="font-family:SimSun;color:black">行为</span><span style="color:black" lang="EN-US"> n<br>
</span><span style="font-family:SimSun;color:black">也</span><span style="color:black" lang="EN-US"> d<br>
</span><span style="font-family:SimSun;color:black">引来</span><span style="color:black" lang="EN-US"> v<br>
</span><span style="font-family:SimSun;color:black">了</span><span style="color:black" lang="EN-US"> u<br>
</span><span style="font-family:SimSun;color:black">不少</span><span style="color:black" lang="EN-US"> m<br>
</span><span style="font-family:SimSun;color:black">公园</span><span style="color:black" lang="EN-US"> n<br>
</span><span style="font-family:SimSun;color:black">游客</span><span style="color:black" lang="EN-US"> n<br>
</span><span style="font-family:SimSun;color:black">的</span><span style="color:black" lang="EN-US"> u<br>
</span><span style="font-family:SimSun;color:black">围观</span><span style="color:black" lang="EN-US"> v<br>
</span><span style="font-family:SimSun;color:black">。</span><span style="color:black" lang="EN-US"> w<br>
</s><br>
<s><br>
</span><span style="font-family:SimSun;color:black">而</span><span style="color:black" lang="EN-US"> c<br>
</span><span style="font-family:SimSun;color:black">对于</span><span style="color:black" lang="EN-US"> p<br>
</span><span style="font-family:SimSun;color:black">人们</span><span style="color:black" lang="EN-US"> n<br>
</span><span style="font-family:SimSun;color:black">的</span><span style="color:black" lang="EN-US"> u<br>
</span><span style="font-family:SimSun;color:black">议论</span><span style="color:black" lang="EN-US"> v<br>
</span><span style="font-family:SimSun;color:black">，</span><span style="color:black" lang="EN-US"> w<br>
</span><span style="font-family:SimSun;color:black">这些</span><span style="color:black" lang="EN-US"> r<br>
</span><span style="font-family:SimSun;color:black">汉</span><span style="color:black" lang="EN-US"> t<br>
</span><span style="font-family:SimSun;color:black">服</span><span style="color:black" lang="EN-US"> v<br>
</span><span style="font-family:SimSun;color:black">爱好者</span><span style="color:black" lang="EN-US"> n<br>
</span><span style="font-family:SimSun;color:black">表示</span><span style="color:black" lang="EN-US"> v<br>
</span><span style="font-family:SimSun;color:black">，</span><span style="color:black" lang="EN-US"> w<br>
</span><span style="font-family:SimSun;color:black">他们</span><span style="color:black" lang="EN-US"> r<br>
</span><span style="font-family:SimSun;color:black">早已</span><span style="color:black" lang="EN-US"> d<br>
</span><span style="font-family:SimSun;color:black">习惯</span><span style="color:black" lang="EN-US"> v<br>
</span><span style="font-family:SimSun;color:black">了</span><span style="color:black" lang="EN-US"> y<br>
</span><span style="font-family:SimSun;color:black">。</span><span style="color:black" lang="EN-US"> w<br>
</s><br>
</text></span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</blockquote></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>