About protocols in this vicinity, I've been noticing a missing piece in OSS around transport as well. In Python, you often need incompatible dependency sets in one app, and the usual choices are either ad-hoc subprocess RPC that gets messy over time or HTTP / containers that are overkill and make you change deployment strategy.
I ended up building a protocol for my own use around a very strict subprocess boundary for Python (initially at least, protocol is meant to be universal). It has explicit payload shape, timeout and error semantics. I already went a little too far beyond my usecase with deterministic canonicalization for some common pitfall data types (I think pickle users would understand, though). It still needs some documentation polish, but if anyone would actually use it, I can document it properly and publish it.
I have been on a similar odyssey making a 'zero copy' Java library that supports protobuf, parquet, thrift (compact) and (schema'd) json. It does allocate a long[] and break out the structure for O(1) access but doesn't create a big clump of object wrappers and strings and things; internally it just references a big pool buffer or the original byte[].
I also find this misleading, and could be solved so easily by just explaining that of course varints need resolving and things will just happen lazily (presumably, I didn’t read the code) when they are requested to be read rather than eagerly.
Is this still true? New versions of protobuf allow codegen of `std::string_view` rather than `const std::string&` (which forces a copy) of `string` and `repeated byte` fields.
I've been running into _a lot_ of issues with Hyper/Tonic. Like literal H2 spec violations. Try hosting a tonic server behind nginx or ALB. It will literally just not work as it can't handle GOAWAY retries in a H2 spec-compliant way.
If this fixes that I might consider switching.
However, Google is also working in a new grpc-rust implementation and I have faith in them getting it right so holding tight a little bit longer.
Google really dropped the ball with protobuf when they took so long to make them zero-copy. There are 3rd party implementations popping up now and a real risk of future wire-level incompatibilities across languages.
"zero copy" in this context just means that the contents of the input buffer are aliased to string fields in the decoded representation. This is a language-level feature and has nothing to do with the wire format.
On the other hands, having half the packages depend on packages such as serde, syn, procmacro2 might not be such a good idea. First of all it is annoying when creating new projects to have to move over table stakes. Second, it is a security nightmare. most of rust could be vulnerable if dtolnay decided to go rogue.
It is not that everything should go into the stdlib, but having syn, procmacro and serde would be a good start imo. And like golang having a native http stack would be really awesome, every time you have to do any HTTP, you end up pulling in some c-based crypto lib, which can really mess up your day when you want to cross-compile. With golang it mostly just works.
It isn't really in the flavor of rust to do, so I don't think it is going to happen, but it is nice when building services, that you can avoid most dependencies.
I would always go to the official docs page for the needs I have, and use their HTTP library (or any other). It removes decision making, having to ensure good quality practices from lesser known libraries, and risks of supply chain attacks (assuming the root stdlib of a language would have more attention to detail and security than any random 3rd-party library thrown into github by a small group of unpaid devs)
Only when it falls short on my needs, I would drop the stdlib and go in dearch of a good quality, reputable, and reliable 3rd-party lib (which is easier said than done).
Has worked me well with Go and Python. I would enjoy the same with Rust. Or at a minimum, a list of libraries officialy curated and directly pointed at by the lang docs.
I ended up building a protocol for my own use around a very strict subprocess boundary for Python (initially at least, protocol is meant to be universal). It has explicit payload shape, timeout and error semantics. I already went a little too far beyond my usecase with deterministic canonicalization for some common pitfall data types (I think pickle users would understand, though). It still needs some documentation polish, but if anyone would actually use it, I can document it properly and publish it.
I have been on a similar odyssey making a 'zero copy' Java library that supports protobuf, parquet, thrift (compact) and (schema'd) json. It does allocate a long[] and break out the structure for O(1) access but doesn't create a big clump of object wrappers and strings and things; internally it just references a big pool buffer or the original byte[].
The speed demons use tail calls on rust and c++ to eat protobuf https://blog.reverberate.org/2021/04/21/musttail-efficient-i... at 2+GB/sec. In java I'm super pleased to be getting 4 cycles per touched byte and 500MB/sec.
Currently looking at how to merge a fast footer parser like this into the Apache Parquet Java project.
https://protobuf.dev/reference/cpp/string-view/
If this fixes that I might consider switching.
However, Google is also working in a new grpc-rust implementation and I have faith in them getting it right so holding tight a little bit longer.
If anything, there should be “less than blessed” “*-awesome” libraries
Didn't we learn this with python?
How many python http client libraries are in the dumping ground that is the python "batteries included" standard library?
And yet people always reach for the one that is outside stdlib.
It is not that everything should go into the stdlib, but having syn, procmacro and serde would be a good start imo. And like golang having a native http stack would be really awesome, every time you have to do any HTTP, you end up pulling in some c-based crypto lib, which can really mess up your day when you want to cross-compile. With golang it mostly just works.
It isn't really in the flavor of rust to do, so I don't think it is going to happen, but it is nice when building services, that you can avoid most dependencies.
Only when it falls short on my needs, I would drop the stdlib and go in dearch of a good quality, reputable, and reliable 3rd-party lib (which is easier said than done).
Has worked me well with Go and Python. I would enjoy the same with Rust. Or at a minimum, a list of libraries officialy curated and directly pointed at by the lang docs.