<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Verdana","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">This is … interesting behaviour.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">I see you’re using 3.0.0 but I’ve reproduced it with 3.4.10 and a 1.2 billion token corpus:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">EEBOV3> L = <text>[]*</text><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">EEBOV3> size L;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">761<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">EEBOV3> LL = <text>[]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">EEBOV3> size LL;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">44421<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">EEBOV3> LLL = <text>[] expand to text;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">EEBOV3> size LLL;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">44421<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">I am not really sure what’s going on here. It probably involves the parts of the CQP code that I’m a bit scared of…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D">Andrew.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cwb-bounces@sslmit.unibo.it [mailto:cwb-bounces@sslmit.unibo.it]
<b>On Behalf Of </b>Simone Ueberwasser<br>
<b>Sent:</b> 29 April 2017 07:50<br>
<b>To:</b> cwb@sslmit.unibo.it<br>
<b>Subject:</b> [CWB] Too many tokens in an attribute?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="bloop_customfont">
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Hi every body<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">I try to create a CQP corpus with XML attributes from 618 vrt-files. They look as follows:<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><text encrypted_msg="0" contains_fra="true" content_msg="1838" user_msg="1873" no_consent_msg="0" consent_speakers="2" lang_100_and_more="fra" speakers="2" empty_msg="0"
media_msg="35" system_msg="0" total_msg="1873"><o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><msg msg_id="162577"><o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">token1 pos1<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">token2 pos2<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""></msg><o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""></text><o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">An example with two messages is available here:
<a href="http://www.ueberwasser.eu/chat105_original.vrt">www.ueberwasser.eu/chat105_original.vrt</a><o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">In this example, the first message contains 3 tokens. If I run the following query, the message is found:<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><msg>[]*</msg>:: match.msg_msg_id = “162577"<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">The second message is 95 tokens long. The same query shows no results:<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><msg>[]*</msg>:: match.msg_msg_id = "162578"<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">If I remove any 5 tokens from this message, the query is fine for this message, too. Is this a normal behaviour? Is there a limit to the number of tokens within an attribute?
I could not find any information in the documentation.<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Many thanks for any help<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Simone <o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">*********************************<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Setup: <o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Xubuntu 16.04<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">cwb-3.0.0-linux-x86_64<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">CWB Perl-CWB-3.0<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">CWB-CL Perl-CWB-CL-3.0<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">CWB-Web Perl-CWB-Web-3.0<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">CWB-CQI Perl-CWB-CQI-3.0<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">But I had the same problem with CWB 3.0 and Perl scripts 2.2 on a Mac<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">I create the corpus with:<o:p></o:p></span></p>
</div>
<div id="bloop_customfont">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">sudo -H cwb-encode -c utf8 -x -s -B -d PathToData -f /pathtofile.vrt -R PathToRegisty -P pos -S text:0+contains_fra+no_consent_msg+content_msg+empty_msg+total_msg+speakers+media_msg+system_msg+user_msg+encrypted_msg+consent_speakers+lang_100_and_more+demographics+lang_less_than_100+contains_gsw+contains_eng+contains_spa+contains_deu+contains_ita+contains_sla+contains_roh
-S msg:0+msg_id<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
<div id="bloop_sign_1493448526811790080">
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial","sans-serif";color:#222222">===========================================================<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a href="http://www.ueberwasser.eu/" target="_blank"><span style="font-size:9.5pt;font-family:"Arial","sans-serif"">www.ueberwasser.eu</span></a></span><span style="font-size:9.5pt;font-family:"Arial","sans-serif";color:#222222"><br>
===========================================================</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p></o:p></span></p>
</div>
</div>
</body>
</html>