Difference between revisions of "Protobuf notes"
m (Add section "Length Prefixing".) |
m (Add reference to protobuf.dev document section "Adding More Message Types".) |
||
Line 17: | Line 17: | ||
* https://protobuf.dev/programming-guides/encoding/ | * https://protobuf.dev/programming-guides/encoding/ | ||
+ | == [[#top|^]] Factoring Protobug Message Definitions == | ||
+ | The following online guide speaks to defining multiple related protobuf messages in a single .proto file: | ||
+ | |||
+ | * https://protobuf.dev/programming-guides/proto3/#adding-types | ||
+ | |||
+ | == [[#top|^]] Nanopb == | ||
+ | |||
+ | 2022-01-08 Saturday | ||
+ | |||
+ | * https://github.com/nanopb/nanopb/blob/master/generator/proto/nanopb.proto | ||
+ | * https://jpa.kapsi.fi/nanopb/docs/whats_new.html | ||
+ | * https://jpa.kapsi.fi/nanopb/docs/ | ||
+ | |||
+ | * https://docs.python.org/3/tutorial/modules.html | ||
+ | |||
+ | Cmake script to locate Nanopb headers and sources: | ||
+ | |||
+ | * https://chromium.googlesource.com/external/github.com/nanopb/nanopb/+/nanopb-0.2.9.1/extra/FindNanopb.cmake | ||
+ | |||
+ | == [[#top|^]] Protobuf C Code Examples == | ||
When compiling nanopb Protobuf library as part of C language programs, nested Protobuf messages require use of nanopb defined function type `pb_callback_t` in order to encode and to decode those nested messages. Some examples of this on github: | When compiling nanopb Protobuf library as part of C language programs, nested Protobuf messages require use of nanopb defined function type `pb_callback_t` in order to encode and to decode those nested messages. Some examples of this on github: | ||
Line 166: | Line 186: | ||
. . . It appears that the integer values which message elements are assigned as tantamount to key names in JSON. | . . . It appears that the integer values which message elements are assigned as tantamount to key names in JSON. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
<!-- comentario --> | <!-- comentario --> |
Revision as of 03:54, 10 October 2024
Introduction:
Also a good more detailed introduction:
Another reference with possible good detail:
Encoding details of protobuf "on the wire":
Contents
^ Factoring Protobug Message Definitions
The following online guide speaks to defining multiple related protobuf messages in a single .proto file:
^ Nanopb
2022-01-08 Saturday
- https://github.com/nanopb/nanopb/blob/master/generator/proto/nanopb.proto
- https://jpa.kapsi.fi/nanopb/docs/whats_new.html
- https://jpa.kapsi.fi/nanopb/docs/
Cmake script to locate Nanopb headers and sources:
^ Protobuf C Code Examples
When compiling nanopb Protobuf library as part of C language programs, nested Protobuf messages require use of nanopb defined function type `pb_callback_t` in order to encode and to decode those nested messages. Some examples of this on github:
In the first example an early on file instance of `pb_callback_t` occurs on line 56. Looking further this project has a few dozen protoc generated files . . . switching to a possible smaller project:
In kitsune project, looking at:
(1) file kitsune/kitsune/audio_features_upload_task.c function setup_protbuf( . . . )
(2) file audio_features_upload_task_helpers.c function encode_repeated_streaming_bytes_and_mark_done(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
(3) in same file reviewing function write_streams(pb_ostream_t *stream, const pb_field_t *field,hlo_stream_t * hlo_stream)
Here is an excerpt from proto_utils.c which appears to contain a pb_callback_t definition:
147 bool encode_device_id_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg) { 148 //char are twice the size, extra 1 for null terminator 149 char hex_device_id[2*DEVICE_ID_SZ+1] = {0}; 150 if(!get_device_id(hex_device_id, sizeof(hex_device_id))) 151 { 152 return false; 153 } 154 155 return pb_encode_tag_for_field(stream, field) && pb_encode_string(stream, (uint8_t*)hex_device_id, strlen(hex_device_id)); 156 } Same routine no line numbers, plus following routine which references first routine in function point assignment: bool encode_device_id_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg) { //char are twice the size, extra 1 for null terminator char hex_device_id[2*DEVICE_ID_SZ+1] = {0}; if(!get_device_id(hex_device_id, sizeof(hex_device_id))) { return false; } return pb_encode_tag_for_field(stream, field) && pb_encode_string(stream, (uint8_t*)hex_device_id, strlen(hex_device_id)); } void pack_batched_periodic_data(batched_periodic_data* batched, periodic_data_to_encode* encode_wrapper) { if(NULL == batched || NULL == encode_wrapper) { LOGE("null param\n"); return; } batched->data.funcs.encode = encode_all_periodic_data; // This is smart :D batched->data.arg = encode_wrapper; batched->firmware_version = KIT_VER; batched->device_id.funcs.encode = encode_device_id_string; } A search for calls to `pack_batched_periodic_data()`: $ grep -nr pack_batched_periodic_data ./* ./commands.c:1069: pack_batched_periodic_data(&data_batched, &periodicdata); ./proto_utils.c:158:void pack_batched_periodic_data(batched_periodic_data* batched, periodic_data_to_encode* encode_wrapper) ./proto_utils.h:29:void pack_batched_periodic_data(batched_periodic_data* batched, periodic_data_to_encode* encode_wrapper);
Tracing yet further back kitsune project commands.c has following routine which declares and uses a `periodic_data` type:
1038 void thread_tx(void* unused) { 1039 batched_periodic_data data_batched = {0}; 1040 #ifdef UPLOAD_AP_INFO 1041 batched_periodic_data_wifi_access_point ap; 1042 #endif 1043 periodic_data forced_data; 1044 bool got_forced_data = false; 1045 1046 LOGI(" Start polling \n"); 1047 while (1) { 1048 if (uxQueueMessagesWaiting(data_queue) >= data_queue_batch_size 1049 || got_forced_data ) { 1050 LOGI( "sending data\n" ); 1051 1052 periodic_data_to_encode periodicdata; 1053 periodicdata.num_data = 0; 1054 periodicdata.data = (periodic_data*)pvPortMalloc(MAX_BATCH_SIZE*sizeof(periodic_data)); 1055 1056 if( !periodicdata.data ) { 1057 LOGI( "failed to alloc periodicdata\n" ); 1058 vTaskDelay(1000); 1059 continue; 1060 } 1061 if( got_forced_data ) { 1062 memcpy( &periodicdata.data[periodicdata.num_data], &forced_data, sizeof(forced_data) ); 1063 ++periodicdata.num_data; 1064 } 1065 while( periodicdata.num_data < MAX_BATCH_SIZE && xQueueReceive(data_queue, &periodicdata.data[periodicdata.num_ data], 1 ) ) { 1066 ++periodicdata.num_data; 1067 } 1068 1069 pack_batched_periodic_data(&data_batched, &periodicdata); 1070 1071 data_batched.has_uptime_in_second = true; 1072 data_batched.uptime_in_second = xTaskGetTickCount() / configTICK_RATE_HZ; 1073 1074 if( !is_test_boot() && provisioning_mode ) { . . .
^ Length Prefixing
One way to send large data sets via protobuf is to break them into smaller pieces, and apply protobuf definition to give these pieces a meaning both sender and receiver can understand. See one Mr. Eli's article on this strategy:
^ References To Sort
Protobuf references, somewhat arbitrary starting point yet introduces some key topics of Protobuf standard and use cases:
- https://www.crankuptheamps.com/blog/posts/2017/10/12/protobuf-battle-of-the-syntaxes/
- https://www.educative.io/edpresso/what-is-the-difference-between-protocol-buffers-and-json
JSON supported data types:
First Protobuf .proto file, compiles using `protoc-c`, part of a package available with Ubuntu 20.04:
// syntax = "proto3"; syntax = "proto2"; // Notes: // $ protoc-c --c_out=. ./first.proto message sensorUpdates { required int32 message_id = 1; optional float vrms = 2; }
. . . It appears that the integer values which message elements are assigned as tantamount to key names in JSON.