Shameless plug:
Did something similar but with on disk storage as an experiment ( https://github.com/yannick/marekvs ). Mostly geared towards testing interesting datastructures.
I don't care about AI being used to build it, or to write the site, but I do care about why its so much faster than redis. Antirez is pretty performance-obsessed. But outside of that, this looks like a fun project.
that will be part of the next benchmark run, my previous run showed kividb was head to head with Garnet, it was not better but also wasn't worse either. But I haven't tested it with kividb 1.1.0 yet
I did run a benchmark against Valkey 9.1.2 with --io-threads 8 earlier, I will rerun it soon, but these were the results from the previous run, I haven’t been able to find someone who can independently verify these so if you would like then please proceed:
At 100 B GET and pipeline 16, valkey is at 1.61M against 19.71M ops/s for kividb. For 1 KB GET, pipeline 16 it was 1.44M against 8.95M and at 100 B GET, no pipelining: 0.99M against 3.11M, but at pipeline 1 the load generator was the limit for the fast engines, so that row undersells all of them
Values were 100 B and 1 KB, so the small case is covered but not 2 KB. That run only did strings and MSET and MGET. lists and hashes aren't in it yet. Both are a fair ask, and I'll add 2 KB values and LPUSH/LRANGE and HSET/HGETALL scenarios to the harness for the next run.
This benchmark was run on an earlier version of kividb, which still had several memory management issues, for example, in this test, the memory per key was around 250B as compared to 178B for redis, so probably that's why, I have not been able to publish these numbers yet, but since several people have raised this, I will be prioritizing it
Nice. I am curious if you would seek to maintain this with some kind of a foundation(apache)
based funding or keep it internal to your work?
> I used AI a lot, but I was the one making the design decisions based on the tests that claude picked from my history of troubleshooting sessions as part of my job.
Cool, this seemed like a good learning experience, learning from testing it in a real world. I wonder if you plan to write up some of the experience in a blog or something?
I haven't given much thought to it, but sounds like a good idea, I do have a blog section on my website, but I have kept it only for announcements and benchmark comparisons, and to boost the website for search appearances, But this seems a good idea, I will post the blog. Just to make it clear, it did start it as a side project, but currently I am persuing it full time. New contributors are always welcome
Just adding my benchmarking repo, that contains terraform code and scripts using which I performed the benchmarks, helpful for re-evaluating and verifying the numbers published on our website. And possibly to learn where we might have gone wrong.
I don't have that yet, but this week we will running a similar workflow like this one shown https://www.youtube.com/watch?v=DgcBFb4L0dI. We were able to secure enough credits recently to run this, I will be posting a video on youtube as well. But if you would be open to collaborate, let me know.
I thought this project was interesting so I went to the blog and noticed I was reading AI. I am sure the same amount of effort was put into this project.
I am sure that is an impression for many, that is true we did use AI for our blogs, most of our website is built with AI, that was never our focus though, our focus was the database engine itself. But in order for others to find this project, we had to post the blogs. However I can assure you, none of them were auto published, I reviewed each post, verified each fact ourselves before publishing, but I may have mistakes, would you able to share a blog that you might think looks misleading?
I haven’t tested against it, that will be next, i will run the benchmarks on the same method and publish the results, or if you want to, you can as well. Even if it performs poorly, I will report that too like we did with dragonfly.
That's true, however the intention to run the benchmark would be against most of the widely used commands and see if there is any command or use case where kividb might be a better fit than kvrocks. strictly speaking from hardware performance, it would certainly beat kvrocks in terms of sheer throughput, but in terms of volume where does it stand would be something interesting to experiment.
Pipelined throughput that high usually means the loadgen is the thing you're measuring. Past ~5M ops/sec we needed 3 separate memtier hosts to keep the client off the critical path. Any p99 numbers alongside it?
Yes, for building test cases, for code refactoring, and commenting, because it writes cleaner code, but yeah design decisions were mine. The benchmarks I ran were what dragonfly advertises, so scripts for those were written with AI. For redis compatibility, we run redis’ own unit test and e2e test suite with every change to make sure I am not breaking anything each change and you can review it in the repo.
And impressive benchmarks.will have to try it.
Shameless plug: Did something similar but with on disk storage as an experiment ( https://github.com/yannick/marekvs ). Mostly geared towards testing interesting datastructures.
I haven't run into problems with Dragonfly yet, was it only OOM you encountered with it? Garnet I had GC pauses causing problems, so I dropped that.
At 100 B GET and pipeline 16, valkey is at 1.61M against 19.71M ops/s for kividb. For 1 KB GET, pipeline 16 it was 1.44M against 8.95M and at 100 B GET, no pipelining: 0.99M against 3.11M, but at pipeline 1 the load generator was the limit for the fast engines, so that row undersells all of them
Values were 100 B and 1 KB, so the small case is covered but not 2 KB. That run only did strings and MSET and MGET. lists and hashes aren't in it yet. Both are a fair ask, and I'll add 2 KB values and LPUSH/LRANGE and HSET/HGETALL scenarios to the harness for the next run.
> I used AI a lot, but I was the one making the design decisions based on the tests that claude picked from my history of troubleshooting sessions as part of my job.
Cool, this seemed like a good learning experience, learning from testing it in a real world. I wonder if you plan to write up some of the experience in a blog or something?
https://github.com/kividbio/kivi-benchmark