PB.TextFormat (PB v0.2.0)

Copy Markdown View Source

Protobuf text format for PB.

Text format is the human-readable syntax used for configuration files, test fixtures, .textproto/.txtpb files and debugging output. decode/4 parses it into the same value PB.decode/3 would produce from the equivalent wire bytes, and encode/4 prints a value as text. Both honor the schema's configured projections.

Decoding

PB.TextFormat.decode(~S'''
name: "Ada"   # comments run to the end of the line
phones { number: "555-0100" type: MOBILE }
scores: [90, 85]
''', schema, :"example.Person")

Parsing follows the protobuf C++ implementation, which is what the conformance suite checks and what most text format in the wild is written against. That includes its leniencies: <...> as well as {...} around messages, an optional , or ; after each field, an optional : before a message value, [a, b] lists for repeated fields, adjacent string literals concatenated, t/True/f/False and 0/1 for booleans, an f suffix on floats, and hex or octal integers. Out-of-range float literals become infinity or zero rather than errors.

Extensions are written with their full name in brackets, [pkg.ext]: 1, and decode under :__extensions__ as they do from the wire. A group field (or an editions delimited field shaped like one) also answers to its message type's name, so Data { ... } sets the field data. Fields named after one of the message's reserved names are skipped.

Inside a google.protobuf.Any, the expanded form [type.googleapis.com/pkg.Message] { ... } sets type_url and packs the message into value, honoring the packed type's projections. As in C++, the URL prefix must be type.googleapis.com/ or type.googleprod.com/.

PB is stricter than C++ in two places. A non-repeated field given twice is always rejected (C++ lets an implicit-presence field repeat after a default value). Unicode escapes may not encode surrogates, even as a valid pair, as the conformance suite requires.

Encoding

encode/4 validates the value exactly as PB.encode/4 does and prints what the wire would carry, in the layout the C++ printer uses:

optional_int32: 5
optional_string: "héllo"
optional_nested_message {
  a: 1
}
repeated_int32: 1
repeated_int32: 2
map_string_string {
  key: "a"
  value: "b"
}

Fields print in field-number order and map entries sorted by key, so the same value always prints the same text. Group-like fields print their type name, extensions [full.name], and a google.protobuf.Any whose type resolves prints expanded as [type.googleapis.com/pkg.Message] { ... }. Floats print the shortest text that reads back as the same value (0.1, not 0.10000000149011612, for a float field).

Two choices differ from the C++ defaults for readability: string values keep their UTF-8 characters rather than escaping every non-ASCII byte, and empty messages print as name {}. Bytes outside printable ASCII print as \xNN escapes.

Unknown fields print by number (5: 123), as in C++. PB's parser only reads field numbers back with allow_field_numbers: true, and then only for fields the schema knows, so pass print_unknown_fields: false for output that must decode again.

The output parses back to the same value with decode/4 and with the C++ parser. Text format output is not byte-for-byte identical across protobuf implementations, so compare decoded values rather than text.

Errors

Text that is not well-formed returns a PB.DecodeError with kind: :malformed_text. So does a value that does not fit its field's syntax, such as a scalar field without a colon or a list for a non-repeated field. A non-repeated field given twice returns kind: :duplicate_field. Values that are well-formed but invalid for the field (an unknown field, out of range, unknown enum value, invalid UTF-8 in a string) return a PB.ValueError, as PB.JSON does. Every error locates the field by :path and the text by its 1-based :line and :column, except a missing required field, which has no position in the text.

Message nesting is limited to a depth of 100, matching the wire and JSON decoders.

Encoding returns the errors PB.encode/4 would, with operation: :text_encode.

Options

Decode (decode/4) accepts:

  • :unknown_fields — :error (the default) returns {:error, %PB.ValueError{kind: :unknown_field}} for a field the message does not define; :ignore skips it.
  • :allow_field_numbers — when true, a field may be named by its number (3: "x"), as the C++ parser's AllowFieldNumber permits. A number naming an extension sets that extension, and a reserved number is skipped. Without it a number is treated as an unknown field name. Defaults to false.

Encode (encode/4, encode_iodata/4) accepts:

  • :unknown_fields — how map keys the message does not define are handled, as for PB.encode/4: :error (the default) or :ignore.
  • :single_line — when true, prints everything on one line with fields separated by a space: a: 1 b { c: 2 }. Defaults to false.
  • :short_repeated — when true, prints repeated numeric, boolean and enum fields as one list, values: [1, 2, 3], instead of one line per element. Defaults to false.
  • :print_unknown_fields — when false, leaves unknown fields out. Defaults to true.

An unknown option key, an option meant for the other direction, or a value the option does not take returns {:error, %PB.OptionError{}}.

Summary

Functions

Decodes text format into the value PB.decode/3 would produce.

Decodes text format into the value PB.decode/3 would produce, raising PB.Error on failure.

Encodes data as text format. Returns {:ok, binary} or {:error, error} (see PB.Error).

Encodes data as text format, raising PB.Error on failure.

Encodes data as text format iodata. Returns {:ok, iodata} or {:error, error} (see PB.Error).

Encodes data as text format iodata, raising PB.Error on failure.

Types

opts()

@type opts() :: [
  unknown_fields: :error | :ignore,
  allow_field_numbers: boolean(),
  single_line: boolean(),
  short_repeated: boolean(),
  print_unknown_fields: boolean()
]

Functions

decode(text, schema, message_name, opts \\ [])

@spec decode(binary(), PB.schema_source(), PB.message_name(), opts()) ::
  {:ok, term()} | {:error, PB.Error.t()}

Decodes text format into the value PB.decode/3 would produce.

Returns {:ok, value} or {:error, error} (see PB.Error).

decode!(text, schema, message_name, opts \\ [])

@spec decode!(binary(), PB.schema_source(), PB.message_name(), opts()) :: term()

Decodes text format into the value PB.decode/3 would produce, raising PB.Error on failure.

This is the bang variant of decode/4.

encode(data, schema, message_name, opts \\ [])

@spec encode(term(), PB.schema_source(), PB.message_name(), opts()) ::
  {:ok, binary()} | {:error, PB.Error.t()}

Encodes data as text format. Returns {:ok, binary} or {:error, error} (see PB.Error).

encode!(data, schema, message_name, opts \\ [])

@spec encode!(term(), PB.schema_source(), PB.message_name(), opts()) :: binary()

Encodes data as text format, raising PB.Error on failure.

This is the bang variant of encode/4.

encode_iodata(data, schema, message_name, opts \\ [])

@spec encode_iodata(term(), PB.schema_source(), PB.message_name(), opts()) ::
  {:ok, iodata()} | {:error, PB.Error.t()}

Encodes data as text format iodata. Returns {:ok, iodata} or {:error, error} (see PB.Error).

encode_iodata!(data, schema, message_name, opts \\ [])

@spec encode_iodata!(term(), PB.schema_source(), PB.message_name(), opts()) ::
  iodata()

Encodes data as text format iodata, raising PB.Error on failure.

This is the bang variant of encode_iodata/4.