[CWB] cwb testing
Stefan Evert
stefan.evert at uos.de
Sun Jul 30 08:39:20 CEST 2006
On 29 Jul 2006, at 22:04, lars.nygaard at iln.uio.no wrote:
>
>>
>> I attach the Perl source code of a CQP test suite ...
>
> Well, a lot of it was in German, and my mastery of German is rather
> weak,
> so that will limit the usefulness. I think I'll just start over (in
> fact,
> I just did).
>
Very probably a good idea. In my opinion the whole system of "test
records" - each of which consists of a query, an optional subset
expression, followed by optional "group" and "sort" commands, which
would be exectued in various combinations - is far too complicated to
be useful in the long term.
A large part of the code dealt with running CQP and collecting
results (which could now been done through the CQP/Perl interface,
since CQP knows how to act like a well-behaved backend), and figuring
out if CQP had crashed or was hanging. This shouldn't been necessary
any more, since almost all crashes are out-of-memory problems (which
print a clear message on STDERR) and there's only one known case
where CQP will actually get stuck (of course, it would be quite nice
if the test suite wouldn't hang there and wait indefinitely for an
answer from CQP ...).
> What would be useful for me would be a list of functions/features that
> need to be supported.
I think the basic feature is: run a CQP command - or any set of CQP
commands - and compare the output to "gold standard". The old test
suite implemented some basic reformatting of the output so that pure
white-space differences could be ignored automatically. It should be
possible to automatically create a "gold standard" from the output of
a given version of CQP, ather than having to code/verify everything
manually. In this way, we could at least make sure that new CQP
versions don't introduce any new bugs, which is one of the most
important uses of the test suite. Ideally, it should also be
possible to specify Perl code (or external programs?) that generates
items in the gold standard. This makes sense, in particular, for
algorithmically generated corpora, but in principle one could also
take, say, DICKENS in one-word-per-line format and write a (slow)
Perl script that identifies the same tokens or token sequences as a
CQP query (the assumption being that any errors in the Perl script
would be independent from those in CQP and thus produce a different
output for the gold standard).
A lot of testing can be done incrementally, re-using results from
previous tests. While this would make the test suite much faster, in
the interest of a simple and robust implementation it might be most
sensible to repeat all preparatory commands for each test (perhaps
aided by something like macros or a "setup" function in the test
scripts?). Perhaps you can come up with a clever solution here ...
>
>> The test suite also includes a number of test queries and expected
>> output (if you have the right corpus ...).
>
> I could not find any ...
>
Oops, sorry, what I had sent you there is a cleaned-up version of the
test suite from when I was planning to start a partial rewrite ...
I've found the old version with the examples in the meantime. Let me
know if you want to look at them, but I don't think these queries are
particularly useful after all. A better strategy would be to use the
examples from the CQP tutorial as a first test suite.
>
> It is indeed feasible, and I think we need it for queries that are not
> supported by the Perl interface (unless I can figure out a way to send
> several commands), for example dump/undump.
>
You can just concatenate the commands and send them as a single
string (separated by semicolons), but this behaviour is not supported
officially. Why don't you just send multiple commands and either
accumulate the output or just look at the output of the last command?
Let me know if you need some help with, or insights into the CWB/Perl
interface!
> Even simpler: If the encoding/indexing is incorrect, the test
> queries will
> reveal it, so we can give the encode/decode procedure lower priority.
But only if we have a manually coded gold standard (or one that was
created by Perl scripts or so). Still, the test suite for encoding/
decoding is very simple, so I'd rather you focussed on the CQP test
suite.
All the best & thanks again for working on the test suite,
Stefan
More information about the CWB
mailing list