When working with distributed Erlang applications at Logflare/Supabase, we encountered an interesting issue where the :global_name_server would become overwhelmed with messages, leading to a boot loop situation. This issue is particularly relevant when dealing with the prevent_overlapping_partitions feature.
The :syn library provides a distributed process registry for Elixir applications, offering an alternative to :global for name registration across clusters. It allows you to define custom event handler callbacks to handle process conflicts and registration scenarios.
When working with large distributed Elixir clusters at Logflare/Supabase, we encountered situations where the syn_gen_scope gen_server process would become overwhelmed with messages during cross-cluster synchronization. This happens particularly when thousands of processes register under a single scope, causing the message queue to grow significantly and impacting cluster discovery and synchronization performance.
This week, I'm hunting down a few things that are plaguing Logflare, namely:
Quite surprisingly, Supervisors do not have an exposed option for taking a spawn_opt. spawn_opt are process options that are used to control the process behaviour when it comes to memory management, and can be incredibly useful when hunting down garbage build-up in processes.