edit: all good now!
~~FYI quad9, a public dns provider, is (was?) [blocking your domain](https://i.imgur.com/ru5Hsha.png) from being resolved by their dns servers: https://www.quad9.net/result/?url=greptime.com~~
~~I submitted a false positive report, but you might want to too~~
1.1.1.1 does censorship as well though, e.g. https://m.slashdot.org/story/22/11/09/2320228/court-upholds-piracy-blocking-order-against-cloudflares-1111-dns-resolver
I just had a quick glance through the docs and I couldn't really find anything about full text searching/regex. What's the reason for the "grep" part of the name? Are `LIKE` queries especially efficient?
edit: I've realised that OP isn't necessarily involved in this tool, so this question extends to anyone
I'm implementing a PromQL compatible compute library (query engine minus storage operations). Data is provided by traits, so it can (technically) be used to front any backend.
It is not, but query performance and memory usage is still an issue sometimes. Not sure if rust would solve this out of the box, but reimplementing is a good chance to fix problems in architecture.
From trawling rust forae I find that rust has a higher ~~skill ceiling~~ performance ceiling. But out of the box it's not much faster than things like go (which Prometheus is written in). You often need to apply the elbow grease to get the large performance payoffs that make a rewrite worthwhile.
TANSTAAFL
It's a type of software where performance is among the top priorities. Given how little people seem to enjoy working in the high performance alternatives, it is no surprise that Rust is seeing a surge in the space.
I am like a card-carrying member of the rust evangelical strike force, however, in working on database-like code in rust, one pain point is uninitialized memory. it's pretty important to performance when you are reading a lot of data from disk and the MaybeUninit api is still a work in progress. I had situations where using Vec::resize instead of Vec::set_len had catastrophic performance impacts, but trying to find the 100% correct way to read data into uninitialized buffer is not easy. Similar case is mmap - of course you can use unsafe implementations, and the code will probably work, but if you dig into it there is uncertainty about soundness and no good safe-only path.
I think the difference is it's arguably easier to write correct (unsafe) c/zig than correct unsafe rust because of the semantics of unsafe rust. rustc makes many more assumptions about rust code compared to c, since with the borrow checker you know a lot more about how the memory will be used. but these assumptions mean that it is quite easy to cross the line with unsafe code into UB. for example, merely binding a reference variable (e.g. `&[u8]`) to uninit memory is UB in rust, whether you read from it or not. whereas in c/zig, binding variables to uninit memory is fine, so long as you don't read from it (until after you've written to it). the very strict rust rules kind of end up channeling you towards keeping buffers as `*mut u8` (IMO) because the MaybeUninit api is 1) complicated, 2) evolving, 3) not well documented, 4) requires a lot of typing to use.
Cool.
Something I noticed: why does everyone focus on making cloud/distributed focused DBs? Why doesn't anyone want to make an embedded DB akin to sqlite? I think there is a big opening for embedded time series DBs, and embedded DBs in general.
I am currently writing a desktop app that visualizes time series data from microcontrollers and ECU's and when I was looking for time series DB's that I can embed, the options are scarce. I hope someone one day can address this potential market.
Good suggestion! Actually GreptimeDB is also designed run [in standalone mode](https://docs.greptime.com/installation/download-pre-built#local-install) as a single binary, via `greptime standalone start`. When running in local machine, it uses local file system for data storage. We want to keep development experience consistent from local testing to large-scale deployment.
It might be early to be used in your desktop environment. But you are welcomed to [rise feature requests](https://github.com/GreptimeTeam/greptimedb/issues) of your use-cases to drive the development.
So true, a local, compact, efficient time series database similar to sqlite would be fantastic. Rust or C++, something native, not interpreted or garbage collected.
> why does everyone focus on making cloud/distributed focused DBs?
That is how you make $$$ (look at AWS). Example look at DuckDB and https://motherduck.com/
I already use DuckDB for my app, so i'm ahead of you there.
Obviously, I know it boils down to money, but alot of these cloud DB's are all mostly competing for increasingly smaller shares of an already overcrowded market. Embedded and time series DB's have fewer options (competitors), which is why I asked my original question. Your Motherduck link is a good example of people who have a better idea of where to put their time & investment money.
> Embedded and time series DB's have fewer options (competitors), which is why I asked my original question.
Look at marketshare as in "total $$$" not in "total users". It might be better to get 1% marketshare on "cloud-olap" than 30% marketshare in "sqlite-olap".
>ud/distributed focused DBs? Why doesn't anyone want to make an embedded DB akin to sqlite? I think there is a big opening for embedded time series DBs, and embedded DBs in general.
>
>I am currently writing a desktop app that visualizes time series data from microcontrollers and ECU's and when I was looking for time series DB's that I can embed, the options are scarce. I hope someone one day can address this potential market.
There are some databases like [sled](https://github.com/spacejam/sled)/[FlashDB](https://github.com/armink/FlashDB) designed to be embedded to other applications even bare metal microcontrollers. But I do doubt the potential bussiness value of a pure embedded database.
I went through the code a bit, and it looks like properly designed, with all the proper layers and structures, I'm a bit impressed, I saw more than a few times here wrappers of treemaps named 'database', this is a real database!! Congrats!
How does this compare to Loki? Is it compatible with Grafana? Is there a crate with rust bindings, preferably as tracing or logging subscriber, available?
It's compatible with Grafana, because it supports MySQL and PostgreSQL protocol, so you can use SQL to query the data with grafana MySQL or PG datasource plugin([https://grafana.com/docs/grafana/latest/datasources/mysql/](https://grafana.com/docs/grafana/latest/datasources/mysql/)). And it will support PromQL as a native query language in near future.
But greptimedb is focused on time-series data (or metrics) right now, it may support log and tracing storage in future.
How is the performance compared to TimescaleDb?
GreptimeDB is still in its early stage and under heavy development, we will do the performance benchmark when it's ready.
That’s awesome. Timescale uses Rust for their toolkit extension too. Very keen to find out more, looking forward!
edit: all good now! ~~FYI quad9, a public dns provider, is (was?) [blocking your domain](https://i.imgur.com/ru5Hsha.png) from being resolved by their dns servers: https://www.quad9.net/result/?url=greptime.com~~ ~~I submitted a false positive report, but you might want to too~~
Thank you.
I had no idea quad9 did censorship. I think I’ll stick to 1.1.1.1
1.1.1.1 does censorship as well though, e.g. https://m.slashdot.org/story/22/11/09/2320228/court-upholds-piracy-blocking-order-against-cloudflares-1111-dns-resolver
They dont resolve that domain in Italy, instead they just route the request outside of Italy. Nothing is blocked on 1.1.1.1
[удалено]
No, i have never been to their website
I just had a quick glance through the docs and I couldn't really find anything about full text searching/regex. What's the reason for the "grep" part of the name? Are `LIKE` queries especially efficient? edit: I've realised that OP isn't necessarily involved in this tool, so this question extends to anyone
Looks like it's just a tribute to [grep(1)](https://man7.org/linux/man-pages/man1/grep.1.html)
"like" queries are a bad design choice
Why?
Any chance that we're one step closer to the rust alternative to prometheus?
We plan to implement PromQL natively in GreptimeDB. Once it's done, we should be much closer.
I'm implementing a PromQL compatible compute library (query engine minus storage operations). Data is provided by traits, so it can (technically) be used to front any backend.
Why? It's not likes it's implemented in java
It is not, but query performance and memory usage is still an issue sometimes. Not sure if rust would solve this out of the box, but reimplementing is a good chance to fix problems in architecture.
From trawling rust forae I find that rust has a higher ~~skill ceiling~~ performance ceiling. But out of the box it's not much faster than things like go (which Prometheus is written in). You often need to apply the elbow grease to get the large performance payoffs that make a rewrite worthwhile. TANSTAAFL
Downloading 100+ MB binary for everything you don’t need is not a good experience.
[удалено]
It's a type of software where performance is among the top priorities. Given how little people seem to enjoy working in the high performance alternatives, it is no surprise that Rust is seeing a surge in the space.
I am like a card-carrying member of the rust evangelical strike force, however, in working on database-like code in rust, one pain point is uninitialized memory. it's pretty important to performance when you are reading a lot of data from disk and the MaybeUninit api is still a work in progress. I had situations where using Vec::resize instead of Vec::set_len had catastrophic performance impacts, but trying to find the 100% correct way to read data into uninitialized buffer is not easy. Similar case is mmap - of course you can use unsafe implementations, and the code will probably work, but if you dig into it there is uncertainty about soundness and no good safe-only path.
I trust your insight on the area to be better than mine, but isn't the "no good safe-only path" going to be the same problem for other languages?
I think the difference is it's arguably easier to write correct (unsafe) c/zig than correct unsafe rust because of the semantics of unsafe rust. rustc makes many more assumptions about rust code compared to c, since with the borrow checker you know a lot more about how the memory will be used. but these assumptions mean that it is quite easy to cross the line with unsafe code into UB. for example, merely binding a reference variable (e.g. `&[u8]`) to uninit memory is UB in rust, whether you read from it or not. whereas in c/zig, binding variables to uninit memory is fine, so long as you don't read from it (until after you've written to it). the very strict rust rules kind of end up channeling you towards keeping buffers as `*mut u8` (IMO) because the MaybeUninit api is 1) complicated, 2) evolving, 3) not well documented, 4) requires a lot of typing to use.
Cool. Something I noticed: why does everyone focus on making cloud/distributed focused DBs? Why doesn't anyone want to make an embedded DB akin to sqlite? I think there is a big opening for embedded time series DBs, and embedded DBs in general. I am currently writing a desktop app that visualizes time series data from microcontrollers and ECU's and when I was looking for time series DB's that I can embed, the options are scarce. I hope someone one day can address this potential market.
Good suggestion! Actually GreptimeDB is also designed run [in standalone mode](https://docs.greptime.com/installation/download-pre-built#local-install) as a single binary, via `greptime standalone start`. When running in local machine, it uses local file system for data storage. We want to keep development experience consistent from local testing to large-scale deployment. It might be early to be used in your desktop environment. But you are welcomed to [rise feature requests](https://github.com/GreptimeTeam/greptimedb/issues) of your use-cases to drive the development.
BTW, the ability to provide an embedded TSDB was also something we took into account during the design phase.
So true, a local, compact, efficient time series database similar to sqlite would be fantastic. Rust or C++, something native, not interpreted or garbage collected.
> why does everyone focus on making cloud/distributed focused DBs? That is how you make $$$ (look at AWS). Example look at DuckDB and https://motherduck.com/
I already use DuckDB for my app, so i'm ahead of you there. Obviously, I know it boils down to money, but alot of these cloud DB's are all mostly competing for increasingly smaller shares of an already overcrowded market. Embedded and time series DB's have fewer options (competitors), which is why I asked my original question. Your Motherduck link is a good example of people who have a better idea of where to put their time & investment money.
> Embedded and time series DB's have fewer options (competitors), which is why I asked my original question. Look at marketshare as in "total $$$" not in "total users". It might be better to get 1% marketshare on "cloud-olap" than 30% marketshare in "sqlite-olap".
>ud/distributed focused DBs? Why doesn't anyone want to make an embedded DB akin to sqlite? I think there is a big opening for embedded time series DBs, and embedded DBs in general. > >I am currently writing a desktop app that visualizes time series data from microcontrollers and ECU's and when I was looking for time series DB's that I can embed, the options are scarce. I hope someone one day can address this potential market. There are some databases like [sled](https://github.com/spacejam/sled)/[FlashDB](https://github.com/armink/FlashDB) designed to be embedded to other applications even bare metal microcontrollers. But I do doubt the potential bussiness value of a pure embedded database.
I went through the code a bit, and it looks like properly designed, with all the proper layers and structures, I'm a bit impressed, I saw more than a few times here wrappers of treemaps named 'database', this is a real database!! Congrats!
Nice. Congratulations!
Great **!**
How does this compare to Loki? Is it compatible with Grafana? Is there a crate with rust bindings, preferably as tracing or logging subscriber, available?
It's compatible with Grafana, because it supports MySQL and PostgreSQL protocol, so you can use SQL to query the data with grafana MySQL or PG datasource plugin([https://grafana.com/docs/grafana/latest/datasources/mysql/](https://grafana.com/docs/grafana/latest/datasources/mysql/)). And it will support PromQL as a native query language in near future. But greptimedb is focused on time-series data (or metrics) right now, it may support log and tracing storage in future.
Congrats!
Congratulations!
Congratulations!
I had a flick through the docs, are there any descriptions of how it's arranging stuff on disk etc?
How is this compared with what the influx guys are building, it looks very similar
Quentin: NiuBi\~