Protobuf notes - Revision history

Ted: Add section "Google protoc compiler".

2025-01-09T17:55:00Z

Add section "Google protoc compiler".

Ted: Create sub-section "Handling of values of zero".

2024-10-29T23:12:52Z

Create sub-section "Handling of values of zero".

Ted: /* ^ Nanopb */ create sub-section "Set Up Errors".

2024-10-20T07:21:24Z

^ Nanopb: create sub-section "Set Up Errors".

Ted: /* ^ To Predetermine Encoded Data Size */ - noting answer by Kenton Varda regarding max encoded message size.

2024-10-16T19:28:48Z

^ To Predetermine Encoded Data Size: - noting answer by Kenton Varda regarding max encoded message size.

Ted: Add section "To Determine Encoded Data Size at Build Time".

2024-10-16T18:42:27Z

Add section "To Determine Encoded Data Size at Build Time".

Ted: Correct a typo, update section title.

2024-10-16T18:37:52Z

Correct a typo, update section title.

Ted: Add section "Other Protobuf Libraries".

2024-10-16T18:17:32Z

Add section "Other Protobuf Libraries".

Ted: /* ^ Nanopb */ - link to University of Hannover reference.

2024-10-15T19:23:43Z

^ Nanopb: - link to University of Hannover reference.

Ted: /* ^ Encoding Submessages */ - How to encode strings.

2024-10-15T18:16:34Z

^ Encoding Submessages: - How to encode strings.

Ted: /* ^ Encoding Submessages */ protobuf callbacks for decoding are not supported for `oneof` types.

2024-10-14T22:59:21Z

^ Encoding Submessages: protobuf callbacks for decoding are not supported for `oneof` types.

← Older revision		Revision as of 17:55, 9 January 2025
Line 299:		Line 299:

	* https://pigweed.dev/pw_protobuf/#comparison-with-other-protobuf-libraries		* https://pigweed.dev/pw_protobuf/#comparison-with-other-protobuf-libraries
		+
		+	<!-- odne komentar -->
		+
		+	== [[#top\|^]] Google protoc compiler ==
		+
		+	How to pass options via cmake to protoc . . .
		+
		+	* https://github.com/nanopb/nanopb/issues/432

	<!-- odne komentar -->		<!-- odne komentar -->

@@ Line 87: / Line 87: @@
 </pre>
-=== Set Up Errors ===
+=== [[#top|^]] Handling of values of zero ===
 *  https://github.com/zephyrproject-rtos/zephyr/issues/70065

@@ Line 86: / Line 86: @@
 }
 </pre>
 == [[#top|^]] Protobuf C Code Examples ==

← Older revision		Revision as of 19:28, 16 October 2024
Line 207:		Line 207:

	* https://stackoverflow.com/questions/30915704/maximum-serialized-protobuf-message-size		* https://stackoverflow.com/questions/30915704/maximum-serialized-protobuf-message-size
		+
		+	<!--
		+	Stack Overflow discussion and helpful answer by Kenton Varda:
		+
		+	"
		+	33
		+
		+	In general, any Protobuf message can be any length due to the possibility of unknown fields.
		+
		+	If you are receiving a message, you cannot make any assumptions about the length.
		+
		+	If you are sending a message that you built yourself, then you can perhaps assume that it only contains fields you know about -- but then again, you can also easily compute the exact message size in this case.
		+
		+	Thus it's usually not useful to ask what the maximum size is.
		+
		+	With that said, you could write code that uses the Descriptor interfaces to iterate over the FieldDescriptors for a message type (MyMessageType::descriptor()).
		+
		+	See: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor
		+
		+	Similar interfaces exist in Java, Python, and probably others.
		+
		+	Here's the rules to implement:
		+
		+	Each field is composed of a tag followed by some data.
		+
		+	For the tag:
		+
		+	Field numbers 1-15 have a 1-byte tag.
		+	Field numbers 16 and up have 2-byte tags.
		+
		+	For the data:
		+
		+	bool is always one byte.
		+	int32, int64, uint64, and sint64 have a maximum data length of 10 bytes (yes, int32 can be 10 bytes if it is negative, unfortunately).
		+	sint32 and uint32 have a maximum data length of 5 bytes.
		+	fixed32, sfixed32, and float are always exactly 4 bytes.
		+	fixed64, sfixed64, and double are always exactly 8 bytes.
		+	Enum-typed fields' maximum length depends on the maximum enum value:
		+	0-127: 1 byte
		+	128-16384: 2 bytes
		+	... it's 7 bits per byte, but hopefully your enum isn't THAT big!
		+	Also note that negative values will be encoded as 10 bytes, but hopefully there aren't any.
		+	Message-typed fields' maximum length is the maximum length of the message type plus bytes for the length prefix. The length prefix is, again, one byte per 7 bits of integer data.
		+	Groups (which you shouldn't be using; they're a decrepit old feature deprecated before protobuf was even released publicly) have a maximum size equal to the maximum size of the contents plus a second field tag (see above).
		+
		+	If your message contains any of the following, then its maximum length is unbounded:
		+
		+	Any field of type string or bytes. (Unless you know their max length, in which case, it's that max length plus a length prefix, like with sub-messages.)
		+	Any repeated field. (Unless you know its max length, in which case, each element of the list has a max length as if it were a free-standing field, including tag. There is NO overall length prefix here. Unless you are using [packed=true], in which case you'll have to look up the details.)
		+	Extensions.
		+	"
		+	-->

	== [[#top\|^]] Encoding Submessages ==		== [[#top\|^]] Encoding Submessages ==

← Older revision		Revision as of 18:42, 16 October 2024
Line 199:		Line 199:

	In this kitsune project see also `kitsune/kitsune/protobuf/provision.pb.h`.		In this kitsune project see also `kitsune/kitsune/protobuf/provision.pb.h`.
		+
		+	== [[#top\|^]] To Predetermine Encoded Data Size ==
		+
		+	This section might also be titled "To Determine Encoded Data Size at Build Time".
		+
		+	Often not possible with messages of variable and unknown length, the question of determining maximum possible encoded message size at build time can be useful for projects where messages have fixed field counts in a given message. This section collects what tools and online public discussions cover this protobuf analysis topic.
		+
		+	* https://stackoverflow.com/questions/30915704/maximum-serialized-protobuf-message-size

	== [[#top\|^]] Encoding Submessages ==		== [[#top\|^]] Encoding Submessages ==

@@ Line 29: / Line 29: @@
 *  https://protobuf.dev/programming-guides/proto3/#assigning
-== [[#top|^]] Factoring Protobug Message Definitions ==
+== [[#top|^]] To Factor Protobuf Message Definitions ==
 The following online guide speaks to defining multiple related protobuf messages in a single .proto file:

← Older revision		Revision as of 18:17, 16 October 2024
Line 223:		Line 223:

	* https://eli.thegreenplace.net/2011/08/02/length-prefix-framing-for-protocol-buffers		* https://eli.thegreenplace.net/2011/08/02/length-prefix-framing-for-protocol-buffers
		+
		+	== [[#top\|^]] Other Protobuf Libraries ==
		+
		+	Pigweed protobuf library . . .
		+
		+	* https://pigweed.dev/pw_protobuf/#comparison-with-other-protobuf-libraries
		+
		+	<!-- odne komentar -->

	== [[#top\|^]] References To Sort ==		== [[#top\|^]] References To Sort ==

← Older revision		Revision as of 19:23, 15 October 2024
Line 56:		Line 56:

	* https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_encode_submessage		* https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_encode_submessage
		+
		+	A further reference from University of Hannover:
		+
		+	* https://gitlab.uni-hannover.de/tci-gateway-module/grpc/-/blob/47a06ace92d0db299e6fa9ecc9a9d26db8d85c62/third_party/nanopb/docs/reference.rst#pb-encode
		+

	pb_encode.c has an interesting function . . .		pb_encode.c has an interesting function . . .

← Older revision		Revision as of 18:16, 15 October 2024
Line 208:		Line 208:

	* https://stackoverflow.com/questions/39854434/nanopb-correctly-encoding-and-decoding-repeated-construct-fields-in-submessage		* https://stackoverflow.com/questions/39854434/nanopb-correctly-encoding-and-decoding-repeated-construct-fields-in-submessage
		+
		+	How to encode strings, hint: use the .arg message structure member:
		+
		+	* https://stackoverflow.com/questions/57569586/how-to-encode-a-string-when-it-is-a-pb-callback-t-type

	== [[#top\|^]] Length Prefixing ==		== [[#top\|^]] Length Prefixing ==

← Older revision		Revision as of 22:59, 14 October 2024
Line 204:		Line 204:

	* https://github.com/nanopb/nanopb/issues/331		* https://github.com/nanopb/nanopb/issues/331
		+
		+	Evidently protobuf callbacks for decoding are not supported for `oneof` types:
		+
		+	* https://stackoverflow.com/questions/39854434/nanopb-correctly-encoding-and-decoding-repeated-construct-fields-in-submessage

	== [[#top\|^]] Length Prefixing ==		== [[#top\|^]] Length Prefixing ==