[CWB] CWB4 and Ziggurat
Yannick Versley
yversley at gmail.com
Fri Oct 9 14:36:46 CEST 2015
Hi,
just adding my 2 cents - obviously these are just opinions of someone who
has
built things on top of CQP's API
1) horizontal stability: I've seen time and again that this is necessary
for any kind of fast access.
2) developing your own storage engine: My gut feeling says that it may be a
good thing to rely
on existing work (e.g. Apache Parquet, MonetDB, WiredTiger etc.) if the
goal is to support more complex
data structures than currently possible.
See e.g. this overview on techniques on columnar databases
http://www.cs.yale.edu/homes/dna/papers/abadi-column-stores.pdf
WiredTiger is a high-performance embeddable database
http://source.wiredtiger.com/2.6.1/index.html
There are also freely available implementations of query compilers,
such as this one,
if you want to design the storage layer yourself.
https://github.com/uwescience/raco
3) scriptability: I'm not sure whether everyone really wants to learn Lua.
What I would like to see, though, is to have some public API that both
allows
access to the query compiler/execution engine and to raw data columns
(or
lists/vectors of structures), and that can be wrapped using the FFI of
your
favorite language.
Best wishes,
Yannick
On Fri, Oct 9, 2015 at 12:35 PM, Hardie, Andrew <a.hardie at lancaster.ac.uk>
wrote:
> Hi everyone,
>
>
>
> I’d like to draw your attention to a recent addition to the website,
> namely:
>
>
>
> http://cwb.sourceforge.net/cwb4.php
>
>
>
> We’ve added this page as a point of public information for the work toward
> CWB v 4.
>
>
>
> As of now we would like to invite comments, questions and suggestions on
> the plans we have outlined for the future.
>
>
>
> The first proposal is for the new Ziggurat database engine. At the link
> above is posted an outline for how we intend this to work, including an
> explanation of the data model and summary of the file format that
> implements it.
>
>
>
> If you’d to chip in on any aspect of the plan, please do so to this list
> (or, if you’d rather not speak publicly, to mine & Stefan’s email
> off-list). Plus any other questions about where all this is going are
> welcome too.
>
>
>
> There is going to be a period of (probably) some months before serious
> work begins, due to the need to get all components to a stable point that
> we can branch before we break things in the main line of development, so
> there is ample time for discussion!
>
>
>
> best
>
>
>
> Andrew.
>
> _______________________________________________
> CWB mailing list
> CWB at sslmit.unibo.it
> http://devel.sslmit.unibo.it/mailman/listinfo/cwb
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://devel.sslmit.unibo.it/pipermail/cwb/attachments/20151009/b99cc3af/attachment.html>
More information about the CWB
mailing list