Quickwit is very similar to what is described here.
Unfortunately, the files are not in Parquet so even though Quickwit is opensource, it is difficult to tap into the file format.
We did not pick Parquet because we want to actually be able to search and do analysis efficiently, so we ship an inverted index, a row-oriented store, and a columnar format that allows for random access.
We are planning to eventually add ways to tap into the file and get data in the Apache arrow format.
For a quick skim through the docs, it wasn’t clear to me: can I run a stateless Quickwit instance or even a library to run queries, such that the only data accessed is in the underlying object store? Or do I need a long-running search instance or cluster?
Unfortunately, the files are not in Parquet so even though Quickwit is opensource, it is difficult to tap into the file format.
We did not pick Parquet because we want to actually be able to search and do analysis efficiently, so we ship an inverted index, a row-oriented store, and a columnar format that allows for random access.
We are planning to eventually add ways to tap into the file and get data in the Apache arrow format.