<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000">Hello Andrew and Mansur,<br><br>Thank you very much for all of your help.<br><br>It was indeed a memory problem. We have recently added new positional attributes and as a consequence memory consumption increased significantly.<br>We could 'fix' the problem by excluding a few attributes but it's already good to know what caused the problem as the error message was not that helpful.<br>I will also try again with more swap memory.<br><br>Best,<br>Stefan<br><br><hr id="zwchr" data-marker=""><div data-marker=""><b>Von: </b>"mansur" <6688000@gmail.com><br><b>An: </b>"cwb" <cwb@sslmit.unibo.it><br><b>Gesendet: </b>Montag, 3. Dezember 2018 07:46:34<br><b>Betreff: </b>Re: [CWB] Failure of offline-freqlists.php<br></div><br><div data-marker=""><div dir="ltr"><div dir="ltr"><div>Hello!</div><br><div>Stefan, you can try to increase temporarily your swap just to check if this issue is RAM related. For example, to set additional 10Gb of swap use:<br></div><div>dd if=/dev/zero of=/home/USER/swap bs=1G count=10<br>mkswap /home/USER/swap<br>swapon /home/USER/swap</div><br><div>Best,</div><div>Mansur<br></div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Am Mo., 3. Dez. 2018 um 07:25 Uhr schrieb Hardie, Andrew <<a href="mailto:a.hardie@lancaster.ac.uk" target="_blank">a.hardie@lancaster.ac.uk</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm afraid it is next to impossible to diagnose this at a distance. The most likely culprit is the OOM killer, if you're on Linux. There might be something in an error log (or via dmesg?) <br>
<br>
However, there is a decent chance it will just tell you that the system was out of memory. Why this would be, I don't know. Corpus size is not the issue, rather, the number of distinct type tuples is (where a type tuple is every distinct combination of word form plus forms of all the other attributes at any given corpus position....)<br>
<br>
best<br>
<br>
Andrew.<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a> <<a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a>> On Behalf Of Stefan Fischer<br>
Sent: 28 November 2018 18:17<br>
To: <a href="mailto:cwb@sslmit.unibo.it" target="_blank">cwb@sslmit.unibo.it</a><br>
Subject: [CWB] Failure of offline-freqlists.php<br>
<br>
Hello everyone,<br>
<br>
I would like to import a corpus (300M words) into CQPweb. The corpus is already indexed in CWB and the import into CQPweb worked well. As the corpus is rather large, I ran "php offline-freqlists.php my_corpus" in the terminal. Unfortunately, the script fails after several hours and I get the following error message:<br>
<br>
----<br>
<br>
cwb-scan-corpus error!<br>
Killed<br>
<br>
<br>
<br>
PHP debugging backtrace<br>
=======================<br>
array(2) {<br>
[1]=><br>
array(4) {<br>
["file"]=><br>
string(42) "/var/www/html/cqpweb/lib/freqtable.inc.php"<br>
["line"]=><br>
int(99)<br>
["function"]=><br>
string(17) "exiterror_general"<br>
["args"]=><br>
array(1) {<br>
[0]=><br>
&string(29) "cwb-scan-corpus error!<br>
Killed"<br>
}<br>
}<br>
[2]=><br>
array(4) {<br>
["file"]=><br>
string(46) "/var/www/html/cqpweb/bin/offline-freqlists.php"<br>
["line"]=><br>
int(136)<br>
["function"]=><br>
string(22) "corpus_make_freqtables"<br>
["args"]=><br>
array(1) {<br>
[0]=><br>
&string(10) "test_corpus"<br>
}<br>
}<br>
}<br>
<br>
----<br>
<br>
I have already imported corpora larger than this one. So I guess corpus size is not the issue. What else could cause a failure of cwb-scan-corpus?<br>
<br>
Best,<br>
Stefan<br>
_______________________________________________<br>
CWB mailing list<br>
<a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br>
<a href="http://liste.sslmit.unibo.it/mailman/listinfo/cwb" rel="noreferrer" target="_blank">http://liste.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
_______________________________________________<br>
CWB mailing list<br>
<a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br>
<a href="http://liste.sslmit.unibo.it/mailman/listinfo/cwb" rel="noreferrer" target="_blank">http://liste.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
</blockquote></div>
<br>_______________________________________________<br>CWB mailing list<br>CWB@sslmit.unibo.it<br>http://liste.sslmit.unibo.it/mailman/listinfo/cwb<br></div></div></body></html>