<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Dear Stéphie,<div class=""><br class=""></div><div class="">I've struggled quite a bit with similar errors while preparing a corpus generated from online news articles. 1. is clearly a sign that there is a non-unique (or possibly empty) entry in one of the <text id=...> entries. (ID is the primary key for text identification so that's what "Duplicate entry '' for key 'PRIMARY'" is about. In my case there were a few empty string or 0 values from errornous html to xml conversions.) </div><div class=""><br class=""></div><div class="">I've also encountered 2. without being able to find the supposedly non-classification entry. I remember I suspected some characters in meta tags to cause some trouble (e.g. '_' or '-' in category values?) and special/invalid characters in the text, but did not systematically check. Is it possible that the xmllint errors you find point to unclosed tags or "<" ">" characters in the text that may cause problems? </div><div class=""><br class=""></div><div class="">My solution was to rebuild the xml from scratch and run each string and element through checks and conditions that make absolutely sure that no empty, invalid or, in case of the text ids, duplicate values get written into the xml files. For the feature wishlist: for debugging a corpus file it would be very helpful if CQPWeb gave hints as to where in a file it encountered errors – e.g. the first duplicate text id, the encountered non-category-handle value or an error position when building metadata from xml. But I should also say that I have no clue as to the amount of work this would cause or the challenges involved...</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Thilo</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">Am 21.02.2017 um 14:40 schrieb Lehner Stéphanie (lehs) <<a href="mailto:lehs@zhaw.ch" class="">lehs@zhaw.ch</a>>:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Dear CWB community<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">I am currently attempting to set up a corpus of about 50’000’000 token in CQPweb. Unfortunately, the following 2 commands both fail; as a CWB newbie, I am a bit at loss as to how I can find and tackle the root cause of these issues.<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">1. I cannot create frequency lists: ‘Generate CWB text-position records’ fails with the following error message:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-GB" style="color: rgb(85, 142, 213);" class="">A MySQL query did not run successfully!<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-GB" style="color: rgb(85, 142, 213);" class="">Original query: insert into ___temp_cqp_text_positions_for_bge_1875_2015_de (text_id, cqp_begin, cqp_end) VALUES ('BGE08965', 38695514, 38706960),('BGE08966', 38706961, 38707670),('BGE08967', 38707671, 38709579),('BGE08968', 38709580, 38711971),('BGE08969', 38711972, 38715398), [shortened, it keeps on listing every single following BGE i.e. text ID in the corpus] /* from User: lehs_admin | Function: populate_corpus_cqp_positions() | 2017-Feb-13 15:06:54 */<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-GB" style="color: rgb(85, 142, 213);" class="">Error # 1062: Duplicate entry '' for key 'PRIMARY'<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">2. I would like to change the datatype of some XML to classification. I am aware that they need to meet handle criteria; also, I don’t think I have any empty<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><i class=""><span lang="EN-GB" style="color: rgb(85, 142, 213);" class="">The datatype of text_year cannot be changed to [classification], because there are non-category-handle values in the CWB index; the first non-handle value found in the index is [] .<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Infos:<span class="Apple-converted-space"> </span><o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">CWB Version: Release 3.5 (Alpha)<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">CQPweb code: 3.2.26, Revision 924<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Database: 3.2.25<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Ubuntu X64 16.04<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">PHP 7 , apache2<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">(VirtualBox)<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Steps undertaken so far:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">- checked similar posts in mailing list (Scott Sadowsky (Sep 2016) had a similar problem Nr. 2. but no definite solution was given)<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 35.4pt;" class=""><span lang="EN-GB" class="">- I wanted to check my database in a similar way<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 35.4pt;" class=""><span lang="EN-GB" class="">$ <text_year="">[];<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 35.4pt;" class=""><i class=""><span lang="EN-GB" style="color: rgb(79, 129, 189);" class="">ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '<text_year="">[]' at line 1<o:p class=""></o:p></span></i></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 35.4pt;" class=""><span lang="EN-GB" class="">- mysqlcheck –c says OK to database<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: 35.4pt;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">- checked XML validity of input .vrt-file: xmllint shows 2 type of errors which should not be an issue (error ‘huge text node’ and multiple errors for …&lt/…&gt not followed by ;)<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">- Also, I think it is strange that the XML elements <pb n=”NUMBER”></pb> (element between <text> … </text>) is classified by CWB as a ‘free text’ element while all other elements (e.g. footnote) are correctly not classified as such. Is this a further sign that something is messed up in my data file?<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Thank you kindly for your support!<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Best<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">Stéphie<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">***<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" class="">output of cwb-describe-corpus:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">============================================================<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">Corpus: BGE_1875_2015_DE<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">============================================================<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">description: <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">registry file: /usr/local/cwb-3.4.10/share/cwb/registry/bge_1875_2015_de<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">home directory: /usr/local/corpora/bge_1875_2015_de/<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">info file: /usr/local/corpora/bge_1875_2015_de/.info<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">encoding: utf8<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">size (tokens): 49086350<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> 3 positional attributes:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> word pos lemma <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class="">22 structural attributes:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> body p pb pb_n <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> head footnote text text_id <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> text_author text_title text_source text_page <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> text_topics text_subtopics text_language text_date <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> text_description text_type text_file <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> text_year text_decade text_url <o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""> 0 alignment attributes:<o:p class=""></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span lang="EN-GB" style="font-size: 8pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></span></div></div><span style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">CWB mailing list</span><br style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:CWB@sslmit.unibo.it" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">CWB@sslmit.unibo.it</a><br style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://liste.sslmit.unibo.it/mailman/listinfo/cwb" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://liste.sslmit.unibo.it/mailman/listinfo/cwb</a></div></blockquote></div><br class=""></div></body></html>