ElasticGraph::ProtoIngestion

An ElasticGraph extension that supports ingesting Protocol Buffer data into ElasticGraph. Currently it generates proto3 Protocol Buffers schema artifacts from ElasticGraph schemas.

Dependency Diagram

graph LR; classDef targetGemStyle fill:#FADBD8,stroke:#EC7063,color:#000,stroke-width:2px; classDef otherEgGemStyle fill:#A9DFBF,stroke:#2ECC71,color:#000; classDef externalGemStyle fill:#E0EFFF,stroke:#70A1D7,color:#2980B9; elasticgraph-proto_ingestion["elasticgraph-proto_ingestion"]; class elasticgraph-proto_ingestion targetGemStyle; elasticgraph-support["elasticgraph-support"]; elasticgraph-proto_ingestion --> elasticgraph-support; class elasticgraph-support otherEgGemStyle;

Usage

First, add elasticgraph-proto_ingestion to your Gemfile, alongside the other ElasticGraph gems:

diff --git a/Gemfile b/Gemfile
index 4a5ef1e..5c16c2b 100644
--- a/Gemfile
+++ b/Gemfile
@@ -8,6 +8,7 @@ gem "elasticgraph-query_registry", *elasticgraph_details

 # Can be elasticgraph-elasticsearch or elasticgraph-opensearch based on the datastore you want to use.
 gem "elasticgraph-opensearch", *elasticgraph_details
+gem "elasticgraph-proto_ingestion", *elasticgraph_details

 gem "httpx", "~> 1.3"

Next, update your Rakefile so that ElasticGraph::ProtoIngestion::SchemaDefinition::APIExtension is included in the schema-definition extension modules:

diff --git a/Rakefile b/Rakefile
index 2943335..26633c3 100644
--- a/Rakefile
+++ b/Rakefile
@@ -3,5 +3,6 @@
 require "elastic_graph/json_ingestion/schema_definition/api_extension"
 require "elastic_graph/local/rake_tasks"
+require "elastic_graph/proto_ingestion/schema_definition/api_extension"
 require "elastic_graph/query_registry/rake_tasks"
 require "rspec/core/rake_task"
 require "standard/rake"
@@ -16,6 +17,7 @@ ElasticGraph::Local::RakeTasks.new(
   # Determines casing of field names. Can be either `:camelCase` or `:snake_case`.
   tasks.schema_element_name_form = :camelCase
   tasks.schema_definition_extension_modules << ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension
+  tasks.schema_definition_extension_modules << ElasticGraph::ProtoIngestion::SchemaDefinition::APIExtension

   # Customizes the names of fields generated by ElasticGraph.
   tasks.schema_element_name_overrides = {

Adding the schema definition extension automatically enables schema.proto generation with the default elasticgraph package. Optionally, configure a custom package name from your schema definition:

# in config/schema/protobuf.rb

ElasticGraph.define_schema do |schema|
  schema.proto_schema_artifacts package_name: "myapp.events.v1"
end

After running bundle exec rake schema_artifacts:dump, ElasticGraph will generate schema.proto.

Schema Definition API

Custom Scalar Types

Built-in ElasticGraph scalar types are automatically mapped to proto scalar types. For custom scalar types, use protobuf to define the proto scalar type:

# in config/schema/email.rb

ElasticGraph.define_schema do |schema|
  schema.scalar_type "Email" do |t|
    t.mapping type: "keyword"
    t.json_schema type: "string", format: "email"
    t.protobuf type: "string"
  end
end

Type Mappings

The generated schema.proto uses these built-in scalar mappings:

ElasticGraph Type	Protobuf Type
Boolean	bool
Cursor	string
Date	string
DateTime	string
Float	double
ID	string
Int	int32
JsonSafeLong	int64
LocalTime	string
LongString	int64
String	string
TimeZone	string
Untyped	string

Additionally:

• List types become repeated fields.
• Lists of lists (e.g. [[Float!]!]!) are not supported because Protocol Buffers cannot represent them directly. Schema artifact generation raises an error identifying the unsupported field.
• Enum types generate enum definitions whose values are prefixed with the enum type name in UPPER_SNAKE_CASE, including a zero-valued *_UNSPECIFIED entry.