Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.
How does protocol buffer work?
By defining the protocol buffer message types in .proto files, we can specify how our information to be serialized and how they are structured. Here is a basic example of a .proto file that defines a message containing information about a person:
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
Each message type has one or more uniquely numbered fields, and each field has a name and a value type, where values types can be numbers (integer or floating point), booleans, strings, raw bytes, or even other protocol buffer message types, allowing us to structure our data hierarchically. The field can be optional or required.
Once we have the message defined, we can run the protocal buffer compiler for our chosen language on the .proto file to generate data access classes Person. We can then use this class in our application to populate, serialize, and retrieve Person protocol buffer messages.
In Java we can write codes to utilize the Person class like this.
Person john = Person.newBuilder()
.setId(1234)
.setName("John Doe")
.setEmail("jdoe@example.com")
.build();
output = new FileOutputStream(args[0]);
john.writeTo(output);
And similar codes in C++.
Person john;
fstream input(argv[1],
ios::in | ios::binary);
john.ParseFromIstream(&input);
id = john.id();
name = john.name();
email = john.email();
With Protocol Buffer, we can add new fields to our message formats without breaking backwards-compatibility; old binaries simply ignore the new field when parsing. Therefore, we can extend our protocol without worrying about breaking existing code.
Basic Steps
- Define message formats in a
.protofile. - Use the protocol buffer compiler.
- Use the Java/C++/Python protocol buffer API to write and read messages.
Define message formats in a .proto file
The definitions in a .proto file are simple: we add a message for each data structure we want to serialize, then specify a name and a type for each field in the message. Let's use the an address book application as an example. To define our messages, we start with the addressbook.proto.
package tutorial;
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
message AddressBook {
repeated Person person = 1;
}
package & java_package
The .proto file starts with a package declaration package tutorial, which helps to prevent name conflicting. It will by default used as the Java package unless we explicitly specify the java_package as shown above. It is recommended to always specify a package to avoid name collisions in Protocol Buffer name spaces even in non-Java languages.
java_outer_classname
The java_outer_classname option defines the class name which should contain all of the classes in this file. If we don't give a java_outer_classname explicitly, it will be generated by converting the file name to camel case. For example, "my_proto.proto" would, by default, use "MyProto" as the outer class name.
Message
Next, we have the message definitions. A message is just an aggregate containing a set of typed fields. Several basic types are available: bool, int32, float, double and string.
Complex message type
A more complicated structure is supported. As shown above, Person message contains PhoneNumber messages, while the AddressBook message contains Person messages. We can even define message types nested inside other messages – as we can see, the PhoneNumber type is defined inside Person. Enum is also supported– here we have a phone number that can be one of MOBILE, HOME, or WORK.
Number tag
The " = 1", " = 2" markers on each element identify the unique "tag" that field uses in the binary encoding. Tag numbers 1-15 require one less byte to encode than higher numbers, so as an optimization we can decide to use those tags for the commonly used or repeated elements, leaving tags 16 and higher for less-commonly used optional elements. Each element in a repeated field requires re-encoding the tag number, so repeated fields are particularly good candidates for this optimization.
Required Optional Repreated
Each field must be annotated with one of the following modifiers:
-
required: a value for the field must be provided, otherwise the message will be considered "uninitialized". Trying to build an uninitialized message will throw a
RuntimeException. Parsing an uninitialized message will throw anIOException. - optional: the field may or may not be set. If an optional field value isn't set, a default value is used. For simple types, we can specify our own default value, as we've done for the phone number type in the example. Otherwise, a system default is used: zero for numeric types, the empty string for strings, false for bools.
- repeated: the field may be repeated any number of times (including zero). The order of the repeated values will be preserved in the protocol buffer.
More
We'll find a complete guide to writing .proto files – including all the possible field types – in the Protocol Buffer Language Guide.
Compiling Protocol Buffers
Now that we have a .proto, the next thing we need to do is generate the classes we'll need to read and write AddressBook messages. To do this, we need to run the protocol buffer compiler protoc on our .proto:
- If we haven't installed the compiler, download the package.
- Now run the compiler, specifying the source directory (where our application's source code lives with the current directory as default), the destination directory (where we want the generated code to go; often the same as the source directory), and the path to our
.proto.
protoc -I=$SRC_DIR --java_out=$DST_DIR $SRC_DIR/addressbook.proto
Assuming we want Java classes, we use the --java_out option here. Similar options are provided for other supported languages.
This generates com/example/tutorial/AddressBookProtos.java in our specified destination directory.
The Protocol Buffer API - Java
If we look in AddressBookProtos.java, we can see that it defines a class called AddressBookProtos, nested within which is a class for each message we specified in addressbook.proto. Each class has its own Builder class that we use to create instances of that class.
Here are some of the accessors for the Person class.
// required string name = 1;
public boolean hasName();
public String getName();
// required int32 id = 2;
public boolean hasId();
public int getId();
// optional string email = 3;
public boolean hasEmail();
public String getEmail();
// repeated .tutorial.Person.PhoneNumber phone = 4;
public List<PhoneNumber> getPhoneList();
public int getPhoneCount();
public PhoneNumber getPhone(int index);
Meanwhile, Person.Builder has the same getters plus setters:
// required string name = 1;
public boolean hasName();
public java.lang.String getName();
public Builder setName(String value);
public Builder clearName();
// required int32 id = 2;
public boolean hasId();
public int getId();
public Builder setId(int value);
public Builder clearId();
// optional string email = 3;
public boolean hasEmail();
public String getEmail();
public Builder setEmail(String value);
public Builder clearEmail();
// repeated .tutorial.Person.PhoneNumber phone = 4;
public List<PhoneNumber> getPhoneList();
public int getPhoneCount();
public PhoneNumber getPhone(int index);
public Builder setPhone(int index, PhoneNumber value);
public Builder addPhone(PhoneNumber value);
public Builder addAllPhone(Iterable<PhoneNumber> value);
public Builder clearPhone();
As we can see, there are simple JavaBeans-style getters and setters for each field.
Create instances
Here is an example about how to create an instance of Person.
Person john = Person.newBuilder()
.setId(1234)
.setName("John Doe")
.setEmail("jdoe@example.com")
.addPhone(
Person.PhoneNumber.newBuilder()
.setNumber("555-4321")
.setType(Person.PhoneType.HOME))
.build();
Serialization and Deserialization
To persist the data, we can simply run the following codes.
Person john = Person.newBuilder()
.setId(1234)
.setName("John Doe")
.setEmail("jdoe@example.com")
.addPhone(
Person.PhoneNumber.newBuilder()
.setNumber("555-4321")
.setType(Person.PhoneType.HOME))
.build();
// Write to file
FileOutputStream output = new FileOutputStream("target/person.ser");
john.writeTo(output);
output.close();
Once persisted, we can read the data as such.
// Read from file
Person person = Person.parseFrom(new FileInputStream("target/person.ser");
The Protocol Buffer API - C++
For example in C++, we can write codes like:
Person person;
person.set_name("John Doe");
person.set_id(1234);
person.set_email("jdoe@example.com");
fstream output("myfile", ios::out | ios::binary);
person.SerializeToOstream(&output);
Then later on we can read the message back like:
fstream input("myfile", ios::in | ios::binary);
Person person;
person.ParseFromIstream(&input);
cout << "Name: " << person.name() << endl;
cout << "E-mail: " << person.email() << endl;
Check out this post for more information:)
