Class: ElasticGraph::ProtoIngestion::SchemaDefinition::FieldNumberMappings

Inherits:
Object
  • Object
show all
Defined in:
elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb

Overview

Registry of the protobuf field and enum value numbers assigned to an ElasticGraph schema. Parses and validates the numbers stored in the proto_field_numbers.yaml artifact, hands out the next available numbers for new fields and enum values, and serializes the updated mappings for the next artifact dump so that numbers stay stable over time.

Constant Summary collapse

MAX_FIELD_NUMBER =

The largest field number protobuf allows (2^29 - 1), per https://protobuf.dev/programming-guides/proto3/#assigning.

536_870_911
RESERVED_FIELD_NUMBER_RANGE =

Field numbers protobuf reserves for its own implementation; they may not be used as field tags, per https://protobuf.dev/programming-guides/proto3/#assigning.

19_000..19_999
MAX_ENUM_VALUE_NUMBER =

The largest enum value number protobuf allows (the int32 maximum), per https://protobuf.dev/programming-guides/proto3/#enum.

2_147_483_647
JSON_SCHEMA =

JSON schema for the proto_field_numbers.yaml artifact.

{
  "$schema" => "http://json-schema.org/draft-07/schema#",
  "definitions" => {
    "field_number" => field_number_schema,
    "next_field_number" => field_number_schema.merge({"maximum" => MAX_FIELD_NUMBER + 1}),
    "enum_value_number" => enum_value_number_schema,
    "next_enum_value_number" => enum_value_number_schema.merge({"maximum" => MAX_ENUM_VALUE_NUMBER + 1})
  },
  "type" => "object",
  "properties" => {
    "messages" => {
      "type" => "object",
      "additionalProperties" => {
        "type" => "object",
        "properties" => {
          "fields" => {
            "type" => "object",
            "additionalProperties" => {"$ref" => "#/definitions/field_number"}
          },
          "next_number" => {"$ref" => "#/definitions/next_field_number"}
        },
        "required" => ["fields", "next_number"],
        "additionalProperties" => false
      }
    },
    "enums" => {
      "type" => "object",
      "additionalProperties" => {
        "type" => "object",
        "properties" => {
          "values" => {
            "type" => "object",
            "additionalProperties" => {"$ref" => "#/definitions/enum_value_number"}
          },
          "next_number" => {"$ref" => "#/definitions/next_enum_value_number"}
        },
        "required" => ["values", "next_number"],
        "additionalProperties" => false
      }
    }
  },
  "additionalProperties" => false
}

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(message_mappings_by_name:, enum_mappings_by_name:) ⇒ FieldNumberMappings

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of FieldNumberMappings.

Parameters:

  • message_mappings_by_name (Hash<String, MessageMapping>) —

    validated message mappings

  • enum_mappings_by_name (Hash<String, EnumMapping>) —

    validated enum mappings



144
145
146
147
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 144

def initialize(message_mappings_by_name:, enum_mappings_by_name:)
  @message_mappings_by_name = message_mappings_by_name
  @enum_mappings_by_name = enum_mappings_by_name
end

Class Method Details

.from_parsed_yaml(parsed_yaml) ⇒ FieldNumberMappings

Builds an instance from parsed proto_field_numbers.yaml, validating its structure and every mapped number.

Parameters:

  • parsed_yaml (Hash, nil) —

    parsed contents of the artifact (or a hash in the same format)

Returns:

Raises:

  • (Errors::SchemaError) —

    if the mappings deviate from the artifact format or contain invalid numbers



127
128
129
130
131
132
133
134
135
136
137
138
139
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 127

def self.from_parsed_yaml(parsed_yaml)
  parsed_yaml ||= {} # : ::Hash[::String, untyped]
  if (validation_error = VALIDATOR.validate_with_error_message(parsed_yaml))
    raise Errors::SchemaError, "Invalid protobuf field-number mappings:\n\n#{validation_error}"
  end

  empty_section = {} # : ::Hash[::String, untyped]

  new(
    message_mappings_by_name: parse_messages(parsed_yaml.fetch("messages", empty_section)),
    enum_mappings_by_name: parse_enums(parsed_yaml.fetch("enums", empty_section))
  )
end

Instance Method Details

#enum_value_numbers_for(enum_name, value_names) ⇒ Hash<String, Integer>

Returns the stable protobuf numbers for an enum's values, assigning the next available numbers to values that have no stored mapping.

Parameters:

  • enum_name (String)
  • value_names (Array<String>)

Returns:

  • (Hash<String, Integer>) 


211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 211

def enum_value_numbers_for(enum_name, value_names)
  enum_mapping = enum_mapping_for(enum_name)
  value_numbers = enum_mapping.value_numbers_by_name
  new_value_names = value_names - value_numbers.keys
  next_available_number = enum_mapping.next_number
  next_available_number_after_allocations = next_available_number + new_value_names.size

  if next_available_number_after_allocations > MAX_ENUM_VALUE_NUMBER + 1
    raise Errors::SchemaError, "Cannot allocate another protobuf enum value number for enum `#{enum_name}`: " \
      "the maximum enum value number (#{MAX_ENUM_VALUE_NUMBER}) has been reached."
  end

  new_value_numbers = new_value_names.each_with_index.to_h do |value_name, index|
    [value_name, next_available_number + index]
  end
  updated_value_numbers = value_numbers.merge(new_value_numbers)
  updated_mapping = enum_mapping.with(
    value_numbers_by_name: updated_value_numbers,
    next_number: next_available_number_after_allocations
  )
  @enum_mappings_by_name = @enum_mappings_by_name.merge(enum_name => updated_mapping)

  value_names.to_h do |value_name|
    [value_name, updated_value_numbers.fetch(value_name)]
  end
end

#field_number_for(message_name:, public_field_name:, previous_field_names:) ⇒ Integer

Returns the stable protobuf number for a message field, assigning the message's stored next_number if the field has no mapping. When the field was renamed, the mapping stored under one of its previous_field_names (and its number) carries over.

Parameters:

  • message_name (String)
  • public_field_name (String)
  • previous_field_names (Array<String>) —

    old public names of the field, if renamed

Returns:

  • (Integer) 


157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 157

def field_number_for(message_name:, public_field_name:, previous_field_names:)
  message_mapping = message_mapping_for(message_name)
  field_numbers = message_mapping.field_numbers_by_name

  return field_numbers.fetch(public_field_name) if field_numbers.key?(public_field_name)

  old_field_names = previous_field_names.intersection(field_numbers.keys)
  if old_field_names.size > 1
    formatted_old_field_names = old_field_names.sort.map { |name| "`#{name}`" }.join(" and ")
    raise Errors::SchemaError, "Cannot preserve a protobuf field number for `#{message_name}.#{public_field_name}`: " \
      "multiple previous field names have mappings (#{formatted_old_field_names}). A field can preserve only one " \
      "protobuf number; use `renamed_from` for the name whose number should carry over and `deleted_field` for the others."
  end

  old_field_name = old_field_names.first
  updated_mapping =
    if old_field_name
      message_mapping.with(
        field_numbers_by_name: field_numbers
          .except(old_field_name)
          .merge(public_field_name => field_numbers.fetch(old_field_name))
      )
    else
      allocate_field_number(message_name, public_field_name, message_mapping)
    end

  @message_mappings_by_name = @message_mappings_by_name.merge(message_name => updated_mapping)
  updated_mapping.field_numbers_by_name.fetch(public_field_name)
end

#next_enum_value_number_for(enum_name) ⇒ Integer

Returns the next value number that will be assigned for the given enum.

Parameters:

  • enum_name (String)

Returns:

  • (Integer) 


242
243
244
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 242

def next_enum_value_number_for(enum_name)
  enum_mapping_for(enum_name).next_number
end

#next_field_number_for(message_name) ⇒ Integer

Returns the next field number that will be assigned for the given message.

Parameters:

  • message_name (String)

Returns:

  • (Integer) 


191
192
193
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 191

def next_field_number_for(message_name)
  message_mapping_for(message_name).next_number
end

#reserved_enum_value_numbers_for(enum_name, active_value_names) ⇒ Hash<String, Integer>

Returns value names and numbers retained in the mappings but absent from the enum.

Parameters:

  • enum_name (String)
  • active_value_names (Array<String>)

Returns:

  • (Hash<String, Integer>) 


251
252
253
254
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 251

def reserved_enum_value_numbers_for(enum_name, active_value_names)
  value_numbers = enum_mapping_for(enum_name).value_numbers_by_name
  reserved_numbers_by_name(value_numbers, active_value_names)
end

#reserved_field_numbers_for(message_name, active_field_names) ⇒ Hash<String, Integer>

Returns field names and numbers retained in the mappings but absent from the message.

Parameters:

  • message_name (String)
  • active_field_names (Array<String>)

Returns:

  • (Hash<String, Integer>) 


200
201
202
203
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 200

def reserved_field_numbers_for(message_name, active_field_names)
  field_numbers = message_mapping_for(message_name).field_numbers_by_name
  reserved_numbers_by_name(field_numbers, active_field_names)
end

#to_dumpable_hash ⇒ Hash<String, Object>

Serializes the mappings back to the proto_field_numbers.yaml artifact format, with messages and enums sorted by name and their fields and values sorted by number.

Returns:

  • (Hash<String, Object>) 


260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
 
# File 'elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/field_number_mappings.rb', line 260

def to_dumpable_hash
  {
    "messages" => @message_mappings_by_name
      .sort_by(&:first)
      .to_h do |message_name, message_mapping|
        [message_name, {
          "fields" => message_mapping.field_numbers_by_name.sort_by { |field_name, number| [number, field_name] }.to_h,
          "next_number" => message_mapping.next_number
        }]
      end,
    "enums" => @enum_mappings_by_name
      .sort_by(&:first)
      .to_h do |enum_name, enum_mapping|
        [enum_name, {
          "values" => enum_mapping.value_numbers_by_name.sort_by { |value_name, number| [number, value_name] }.to_h,
          "next_number" => enum_mapping.next_number
        }]
      end
  }
end