[CWB] cwb testing

[Lars Nygaard]

Stefan Evert wrote:

> On 29 Jul 2006, at 22:04, lars.nygaard at iln.uio.no wrote:

> I think the basic feature is: run a CQP command - or any set of CQP  
> commands - and compare the output to "gold standard".  

Implemented in the attached script (ironically not extensively tested). 
(Use perldoc for usage).

> The old test  
> suite implemented some basic reformatting of the output so that pure  
> white-space differences could be ignored automatically.  

We'll see if that will be needed.

> It should be  
> possible to automatically create a "gold standard" from the output of  a 
> given version of CQP, ather than having to code/verify everything  
> manually. 

Implemented.

> In this way, we could at least make sure that new CQP  
> versions don't introduce any new bugs, which is one of the most  
> important uses of the test suite.  

Yes, and it will also be useful for controlling the development of 
complex corpora.

> Ideally, it should also be  possible 
> to specify Perl code (or external programs?) that generates  items in 
> the gold standard. 

I guess that could be useful, but I don't feel its a very high priority.

> A lot of testing can be done incrementally, re-using results from  
> previous tests.  While this would make the test suite much faster, in  
> the interest of a simple and robust implementation it might be most  
> sensible to repeat all preparatory commands for each test (perhaps  
> aided by something like macros or a "setup" function in the test  
> scripts?). 

In principle, this should not be too hard, but again: it seems premature 
to implement it right now.

>> It is indeed feasible, and I think we need it for queries that are not
>> supported by the Perl interface (unless I can figure out a way to send
>> several commands), for example dump/undump.
> 
> You can just concatenate the commands and send them as a single  string 
> (separated by semicolons), but this behaviour is not supported  
> officially.  Why don't you just send multiple commands and either  
> accumulate the output or just look at the output of the last command?

Well, I tried that, and consistently get the following error message: 
"WARNING: query lock violation attempted".

Could you send an example of code that does this?


cheers,
lars
-------------- next part --------------
#!/usr/bin/perl

use strict;
use Getopt::Mixed qw( nextOption );
use WebCqp::Query;
use Data::Dumper;

Getopt::Mixed::init('binary=s b>binary testfile=s t>testfile create c>create corpus=s k>corpus registry=s r>registry help h>help verbose v>verbose');

my %opts;

$opts{'corpus'}="TEST";
$opts{'binary'}="/usr/local/bin/cqp";

while (my ($option,$value)= nextOption()) {
    unless ($value) { $value=1 }
    $opts{$option}=$value;
}

if ($opts{'help'}) {
    die(`perldoc $0`);
}

unless (-e $opts{'binary'}) {
    die('Binary not found. Aborting. \n');
}


open (TESTS, $opts{'testfile'}) or die ('Testfile not found. Aborting. \n');

my $not_ok = 0;
my $all = 0;

while (<TESTS>) {

    chomp;
    my ($q,$file,$p,$args)=split(/\t/);

    if ($p) {
	search_perl($q,$args);
    }
    else {
	$q =~ s/\"/\\\"/g;
	`echo \"$opts{'corpus'}; $q \"| $opts{'binary'} -P cat > /tmp/cwb-test-result`;	
    }



    if ($opts{'create'}) {
	`cp /tmp/cwb-test-result $file`;
    }
    else {
	
	# diff
	my $diff = `diff /tmp/cwb-test-result $file`;

	if ($diff) {
	    print "NOT ok: TEST $q \n";
	    if ($opts{'verbose'}) {
		print $diff;
	    }

	    $not_ok++;
	}
	else {
	    print "ok: $q \n";
	}
	$all++;
	
    }

}

unless ($opts{'create'}) {
    print "\n\n:: $not_ok of $all failed.\n";
}


sub search_perl {

    my $q = shift;
    my $args = shift;

    my $query = new WebCqp::Query $opts{'corpus'};
    $query->on_error(sub{grep {print "<2>$_</h2>\n"} @_});

    $query->cut(2);
    $query->context('1 s', '1 s');

    my $res = Dumper($query->query($q));

    open (OUT, ">/tmp/cwb-test-result");
    print OUT $res;
    close OUT;
    


}

__END__


=head1 NAME

cqp-test

=head1 VERSION

Version 0.1

=head1 SYNOPSIS

        > cqp-test [options]

E.g.

        > cqp-test --testfile=test.txt --binary=build/cqp

=head1 DESCRIPTION

Run a list of test through cqp, and see if the results remain constant.

=head1 FILE FORMAT

The list of tests (one per line) consist of the following, <tab>-separated
fields:

=over

=item B<query>

=item B<file>

=item B<'p'> (i.e. this query should be run throuht the perl interface)

=item B<args> space-separated list of arguments for the perl interface

=back

i.e.:

 "another";	res/1
 [word="afternoon"];	res/2
 [word="afternoon"];	res/3	p	cut=4 context=5

=head1 OPTIONS

=over

=item B<-b>,    B<--binary>

The binary to use for testing. [default: cqp]

=item B<-c>,    B<--create>

Create gold standard.

=item B<-t>,    B<--testfile>

File containing tests.

=item B<-k>,    B<--corpus>

Corpus to run test on. [default: TEST]

=item B<-h>,    B<--help>

Display usage information.

=item B<-v>,    B<--verbose>

Be verbose.

=back

=head1 AUTHOR

Lars Nygaard, C<< <lars.nygaard at inl.uio.no> >>

=head1 COPYRIGHT & LICENSE

Copyright 2006 Lars Nygaard, all rights reserved.

This program is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.

=cut

-------------- next part --------------
"another";	res/1
[word="afternoon"];	res/2
[word="afternoon"];	res/3	p	cut=4 context=5