When working with distributed Erlang applications at Logflare/Supabase, we encountered an interesting issue where the :global_name_server would become overwhelmed with messages, leading to a boot loop situation. This issue is particularly relevant when dealing with the prevent_overlapping_partitions feature.
The :syn library provides a distributed process registry for Elixir applications, offering an alternative to :global for name registration across clusters. It allows you to define custom event handler callbacks to handle process conflicts and registration scenarios.
The prevent_overlapping_partitions option in Erlang is a configuration parameter that affects how the :global module handles network partitions in distributed Erlang systems.
When working with large distributed Elixir clusters at Logflare/Supabase, we encountered situations where the syn_gen_scope gen_server process would become overwhelmed with messages during cross-cluster synchronization. This happens particularly when thousands of processes register under a single scope, causing the message queue to grow significantly and impacting cluster discovery and synchronization performance.
This week, I'm hunting down a few things that are plaguing Logflare, namely: