When I first deployed a RabbitMQ cluster on Kubernetes (K8s) with only pure IPv6 addresses, I ran into an unexpected problem: RabbitMQ couldn’t resolve hostnames over IPv6, even though my cluster was fully IPv6-enabled.
The error I saw was:
{failed_connect,
[{to_address,{"kubernetes.default.svc.cluster.local",443}},
{inet,[inet],nxdomain}]}
At first glance, this looked like a simple DNS issue. But after digging deeper, I realized the problem was that Erlang’s HTTP client (httpc
) was trying to resolve the hostname using IPv4 (inet
) only, while my services only had IPv6 (inet6
) available.
First Code Setup (with the Problem)
My initial RabbitMQ setup included the following environment variables:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
And my erl_inetrc
file looked like this:
{inet6, true}.
Despite this, the rabbit_peer_discovery_k8s
plugin failed when invoking the Kubernetes API. Erlang’s httpc:request
still attempted IPv4 resolution, which led to:
{inet,[inet],nxdomain}
Error Explanation
This happens because:
- Erlang’s resolver (
inet
) defaults to IPv4 unless explicitly told to use IPv6. httpc
does not automatically switch to IPv6 mode unless the environment forces it.- RabbitMQ itself doesn’t control DNS resolution—the Erlang runtime does.
So, even though I had {inet6, true}
in erl_inetrc
, Erlang modules like httpc
still defaulted to IPv4-first resolution.
How I Fix It
To fix this, I needed to force Erlang to prefer IPv6 resolution everywhere.
Update erl_inetrc
I expanded the file to look like this:
{inet6, true}.
{hosts_file, "/etc/hosts"}.
{lookup, [dns, file]}.
This explicitly prioritizes IPv6 and ensures Erlang uses DNS lookups.
Verify Container DNS Setup
I made sure that /etc/resolv.conf
pointed to an IPv6-capable DNS server and that /etc/hosts
included the proper entries.
Start RabbitMQ with IPv6 Parameters
I restarted RabbitMQ with:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp -ssl_dist inet6_tcp"
This guaranteed that both Erlang distribution and SSL traffic used IPv6.
Test Resolver Inside the Pod
I ran:
kubectl exec -ti <rabbit-pod> -- \
rabbitmqctl eval 'inet:gethostbyname("kubernetes.default.svc.cluster.local").'
And I finally got the expected result:
{ok,{hostent,"kubernetes.default.svc.cluster.local",[],inet6,16,
[{64769,43981,0,0,0,0,0,1}]}}
Success! Erlang could now resolve IPv6 addresses correctly.
Extra Debugging & Practice Functionality
To better understand the gap between inet:gethostbyname/1
and httpc:request/1
, I played around with Erlang directly:
% Test with inet (IPv4)
inet:gethostbyname("kubernetes.default.svc.cluster.local").
% Test with inet6 (IPv6)
inet:gethostbyname("kubernetes.default.svc.cluster.local", inet6).
% Simulate httpc request (forcing IPv6 transport)
httpc:request(get, {"https://[fd01:abcd::1]:443/api", []}, [], []).
Why this helps:
inet:gethostbyname/1
→ shows what Erlang’s DNS resolver does by default.inet:gethostbyname/2
→ lets me force and confirm IPv6 resolution.httpc:request/4
→ verifies how Erlang’s HTTP client behaves with IPv6 addresses.
Best Practices for RabbitMQ + IPv6 on K8s
Here are the practices I now follow whenever I deploy RabbitMQ in an IPv6-only cluster:
- Force IPv6 in
{inet6, true}.
- Verify DNS Config:
Make sure/etc/resolv.conf
points to an IPv6-capable DNS server. - Test Resolution Before Plugins:
Runrabbitmqctl eval 'inet:gethostbyname(...).'
to confirm hostnames resolve. - Use Bracketed IPv6 Addresses in Configs:
For example:[fd01:abcd::1]
. - Enable Verbose Debugging:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp -sasl sasl_error_logger tty"
Final Thought
At first, I thought my issue was with RabbitMQ itself. But in reality, it was about how Erlang handles DNS resolution under IPv6. Once I understood that httpc
and inet
behave differently, the fix was straightforward: configure erl_inetrc
properly, verify the container DNS, and always test with rabbitmqctl eval
. Now, my RabbitMQ cluster can resolve IPv6-only Kubernetes services without crashing, and the peer discovery plugin works seamlessly.