<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Hi Scott,<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
CQPweb is built on the same assumption as CWB generally:that corpora don't change once created.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Overwriting the indexes with an expanded version, while trying to keep the CQPweb extra bits intact , therefore implies undefined behaviour.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Ie, you're not meant to do it; having done it, all kinds of things might break; the low number of things that have actually broken is pure good luck!<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
In particular I have no idea what kinds of inconsistencies might arise in things like user's saved data. As I say, undefined : anything could happen.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
But to answer the question: the safe way to add new metadata category values is to nuke the whole metadata table and rebuild from scratch. You might find an unsafe way by poking around in metadata lib php and script a series of calls... but that would be a
pretty precarious way to do it.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
And in future I am going to shift the underlying SQL datatype of category metadata from strings to enums for efficiency... at which point the above will be super-precarious.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
-------<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
<b>Is there any way to update a corpus so that it rescans metadata like p- and s-attributes and their values?</b> My goal is to avoid having to recreate the corpus from scratch over and over.<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
-------<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Fair enough, but note what you're already doing:<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Offline rebuild of the CWB data <br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Rebuild of all the freq tables <br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Reinstallation of metadata table (As per above)<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
.... which is nearly ALL the work of installing a corpus from scratch already. <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
My recommendation: set up highly specific templates, index the data offline, and use "new corpus from existing cwb" (can't remember the exact wording, sorry; I'm writing at a wedding with no laptop!). That saves you specifying p/s atts as they are taken from
the registry. The rest is what you are doing anyway.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
This also has the benefit of making it possible to preserve the earlier versions -- your users might appreciate this if ever they need to do an analysis on the exact data as an analysis they did last year, for instance... on my server I do this by means of
dates or version nos. on the end of corpus handles.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
best <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Andrew.<br>
<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
From: Scott Sadowsky<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Sent: Saturday 15 June, 10:20 am<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Subject: [CWB] [CQPweb] Expanding existing corpora<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
To: Open source development of the Corpus WorkBench<br>
<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
I have a situation which is probably not the norm for most users here. I have a corpus which I will be putting online gradually, in 20 or 30 installments over the next two years or so, as texts can be reviewed a second time for personally identifying or sensitive
information, and such things can be redacted (it's a speech corpus).<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
When a new batch of texts is ready I process, tag and compile all the files that are fit for public consumption into a CQP corpus, upload the new set of CQP files to the server (replacing the old ones), and then re-run the frequency and STTR calculation scripts
on the server. This updates the frequencies shown everywhere I've looked (test query results, corpus metadata, etc.) -- so far, so good.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
The one thing I haven't been able to get to update, however, are the values of the text metadata and word-level annotation variables (as seen in the selection boxes of restricted queries and subcorpus creation). <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Thus, if the first version of the corpus only had four of six socioeconomic statuses (say 1, 2, 3, 6) and a new version includes one or more speakers of SES 4, this new SES doesn't show up anywhere.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
<b>Is there any way to update a corpus so that it rescans metadata like p- and s-attributes and their values?</b> My goal is to avoid having to recreate the corpus from scratch over and over.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Thanks in advance,<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Scott<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
NOTE Unless I've misunderstood something, I'm <i>not</i> adding new p- or s-attributes, but rather new
<i>values</i> for existing p-attributes.<br>
<br>
<br>
</div>
</body>
</html>