<div dir="ltr">This is fantastic, Andrew! <br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 16, 2019 at 6:52 PM Hardie, Andrew <<a href="mailto:a.hardie@lancaster.ac.uk">a.hardie@lancaster.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div class="gmail-m_-6711453741705748738WordSection1">
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">PS, Scott, if you can get your templates working, you might find check-in 1250 of interest.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">cd bin<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">php install-corpus.php --help<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Hardie, Andrew
<br>
<b>Sent:</b> 16 June 2019 22:05<br>
<b>To:</b> Open source development of the Corpus WorkBench <<a href="mailto:cwb@sslmit.unibo.it" target="_blank">cwb@sslmit.unibo.it</a>><br>
<b>Subject:</b> RE: [CWB] Expanding existing corpora<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">cd bin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">>></span> creating and assigning corpus and frequency list permissions<span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb add_corpus_to_privilege_scope PRIVILEGE-INTEGER-ID CORPUS-HANDLE<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb remove_corpus_from_privilege_scope PRIVILEGE-INTEGER-ID CORPUS-HANDLE<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb create_corpus_default_privileges CORPUS-HANDLE<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb add_new_privilege 1 "" "Permission to use at retricted level (initially has scope over no corpora, they can be added later)"<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb add_new_privilege 2 "" "Permission to use at normal level "<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb add_new_privilege 3 "" "Permission to use at full level "<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb add_new_privilege 4 5000000 "Permission to create freq lists up to 500 K tokens"<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb grant_privilege_to_user USERNAME PRIVILEGE-INTEGER-ID<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb grant_privilege_to_group GROUP-NAME PRIVILEGE-INTEGER-ID<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb remove_grant_from_user USERNAME PRIVILEGE-INTEGER-ID<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb remove_grant_from_group GROUP-NAME PRIVILEGE-INTEGER-ID<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)">>>
</span>; setting things like inter-linear gloss views, corpus title and corpus-level metadata<span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb update_corpus_visualisation_gloss CORPUS-HANDLE 1-OR-0-FOR-SHOw-IN-CONCORDANCE 1-OR-0-FOR-SHOW-IN-CONTEXT P-ATTRIBUTE-HANDLE<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb update_corpus_visualisation_translate CORPUS-HANDLE 1-OR-0-FOR-SHOw-IN-CONCORDANCE 1-OR-0-FOR-SHOW-IN-CONTEXT S-ATTRIBUTE-HANDLE<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb add_variable_corpus_metadata CORPUS-HANDLE ATTRIBUTE-DESCRITPION VALUE-CONTENT<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New";color:rgb(31,73,125)">./cqpweb update_corpus_title CORPUS-HANDLE "new title goes here"<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Verdana,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> <a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">
cwb-bounces@sslmit.unibo.it</a> <<a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a>>
<b>On Behalf Of </b>Scott Sadowsky<br>
<b>Sent:</b> 16 June 2019 16:21<br>
<b>To:</b> Open source development of the Corpus WorkBench <<a href="mailto:cwb@sslmit.unibo.it" target="_blank">cwb@sslmit.unibo.it</a>><br>
<b>Subject:</b> Re: [CWB] Expanding existing corpora<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sat, Jun 15, 2019 at 6:44 AM Maarten Janssen <<a href="mailto:maartenpt@gmail.com" target="_blank">maartenpt@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks very much for answering, Maarten. It's not so much the tagging, compiling, uploading and doing frequency counts that I'm trying to not repeat, since I script all that. It's everything afterwards you have to do in CQPweb -- creating
and assigning corpus and frequency list permissions; setting things like inter-linear gloss views, corpus title and corpus-level metadata; generating subcorpora; and so on. And it doesn't help that I haven't yet gotten the XML or metadata templates to work.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In theory this, too, could all be scripted, since you can pass MySQL any command you want from Bash, Perl or whatever. But it would take a rather deep understanding of CQPweb, its database structures, etc., which I don't have. Hence my
search for other ways to streamline things. But as Andrew made clear, copying one corpus version on top of another and then updating is definitely not the way to go!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Best wishes,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Scott<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<p class="MsoNormal" style="margin-bottom:12pt">Updating a CQP corpus (not necessarily the MySQL tables, those I know little of but the raw CQP files) is not really possible - there are various attempts out there to do things in parts, but in the end, due
to the set-up of the files, there is no secure way of updating files - files have an index of values, in corpus order, and a list linking corpus positions to the numbers in that index. Theoretically, you could not care about the corpus order and just change
a corpus position index number, but unless you know beforehand which to change, it would not save time since you still have to go through the entire corpus; and you would have to be completely certain no extra values have appeared (or values that were possible
actually became used). And even if you would manage, there are other files that count the number of values and such, and those would have to be recompiled in any case. So unless you have gigaword corpora, the best way is just to recompile; you might be able
to get away with just recompiling the CQP corpus itself (running just cwb-encode), which is relatively fast; in my experience, a 500M corpus takes about half an hour to compile, and that is using my own encoder directly from XML, so the native cwb-encode is
likely to be even faster than that….<u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
_______________________________________________<br>
CWB mailing list<br>
<a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br>
<a href="http://liste.sslmit.unibo.it/mailman/listinfo/cwb" rel="noreferrer" target="_blank">http://liste.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.7273px">Dr. Scott Sadowsky<br>Profesor Asistente de Lingüística</div><div dir="ltr" style="font-size:12.7273px">Pontificia Universidad Católica de Chile<br></div><div dir="ltr" style="font-size:12.7273px"><br></div><div dir="ltr" style="font-size:12.7273px">ssadowsky gmail com</div><div dir="ltr" style="font-size:12.7273px">scsadowsky uc cl<br><a href="http://sadowsky.cl/" target="_blank">http://sadowsky.cl/</a></div><div dir="ltr" style="font-size:12.7273px"> </div></div></div></div></div></div></div></div></div></div></div>