Comments | zunger: Protocol buffer tips

zunger

Protocol buffer tips

Jul 08, 2008 21:18

(FYI: The following entry is going to be much more technical than most of what I post. Anyone who doesn't care about code or data serialization can pretty much hit "next" right now.)

A few days ago, Google open sourced one of its key data serialization formats, protocol buffers. There's already been some chat on how they're similar to or different ( Read more... )

computer science

Comments 10

jephly July 9 2008, 05:41:31 UTC

Don't you get a single byte header for tag numbers 1 through 15 (not 7)? The docs say that the 3 least significant bits are the wire type, so you have 4 left for the tag (the MSB is the "varint continue" bit).

zunger July 9 2008, 06:30:26 UTC

Um. Yes. I can count, really.

zorbathut July 9 2008, 17:59:33 UTC

I'm really happy you guys finally opened this one up, I've been half-considering reimplementing it.

One more thing anonymous July 13 2008, 02:55:50 UTC

I have to disagree with your "One more thing", or at least add a warning. The reason reusing a protocol buffer object is faster than creating a new one is that the Clear() method zeroes out existing storage but doesn't actually free any memory, so a reused protocol message will grow but never shrink. This gets especially bad with complex structures (i.e. nested repeated groups) as each repeated subfield gets stuck at its maximum size. I've seen real-world cases where a reused protocol message grew to hundreds of megs while no individual message was bigger than 100K.

Reusing protocol message objects can be a useful optimization, but it should be done only as needed rather than by default.

-Ben

Re: One more thing zunger July 13 2008, 20:07:38 UTC

True, one needs to balance between the two. There's a little code in there to reclaim memory on the fly during reuse (e.g., clearing a string field will often release memory) but if there's a risk of it jumping, that's bad.

Reflection anonymous August 13 2008, 02:19:25 UTC

Nitpick: Point 5 should probably suggest using reflection rather than writing code that is coupled with the protocol buffer wire format.

-Kenton

Need your advise anonymous January 24 2009, 05:32:24 UTC

Hi,
I was googling for examples and came across your post. I am exploring the possibility of replacing our existing Java web application based on WSDL XML with Google's Protocol Buffers. So in essense the Java app remains the same but the transport is now Proto Buffers. My first task is to build a prototype. I have no idea how I will make Proto Buffers run on the server side. I mean there is no ready server out there as there is for XML RPC. Could you please share your experience and knowledge on how to go about it?
You can email me directly at sunitkatkar at gmail.

Thanks,
Sunit

Re: Need your advise zunger January 26 2009, 17:37:07 UTC

Google hasn't at this point open sourced any standard server code. However, using protocol buffers on the server side is fairly easy -- their serialized form can be the payload of an HTTP POST, and you can simply open them up and use them. It's a good fairly compact (in terms of byte usage) transport type.