Exadata X6, fixing the martian source errors, static routing, asymetric network, route-eth3, reverse path filtering and more

This blog post will be about martian source errors which may be recorded in the syslog(/var/log/messages) of Exadata compute nodes. I want to give a place to this subject, as I have faced this problem in one of my customer environments.

The platform was an Exadata X6 and it was newly deployed. I was there to migrate EBS database to this new platform and seen the following errors in /var/log/messages when I was doing my very first health check.

martian source someip from someip, on dev some_interface

ll header: ff:ff:ff:ff:ff:ff:00:18:f8:0e:..........

When I interpreted this and spoked with the clients, realized that some clients could not ping or connect via ssh to Exadata compute nodes.


The martian erros was reported because of a network configuration problem. That is, for some reason, Linux Operating System running on Exadata compute nodes did not like the network interface from where some packets were coming from.

If I need to be more specific , Linux OS did not like the packets that are coming to bondeth0(client connection interface) ,as it thought that those packets had to come from the eth3 interface(the backup interface).

At this point; I want to remind you something important for the Exadata network:

"If the backup network is on a tagged VLAN network, the client network must also be on a separate tagged VLAN network."

So this phrase clearly states that, if your backup network is on a vlan (suppose it is 10.56.23.0) then your client network must be on another vlan . So the client network should not be on the same VLAN as the backup network. So if the backup network is 10.56.23.0, there should not be any clients on 10.56.23.0. (client term can be thought as anything that connects to the Exadata via client interface or through the management interface.) So, the clients that are there for managing the Exadata machine via the management network should be on another VLAN  too.

Let me explain this with an example scenario;

Suppose;

the backup interface (eth3 for Exadata X6) is configured to run on 10.56.23.35 

the management interface is configured to run on 10.10.10.100

the bonded client network is configured to run on the ip addresses like 192.168.0.80 192.168.0.81 and so on.

suppose there is a client with ip address 10.56.23.81. (same network with the backup interface 10.56.23.0)

So now imagine the client with ip address 10.56.23.81 wants to ping the Exadata management interface. (ping 10.10.10.100)

When he pings; the OS sees the packet and sees that packet is coming from the same network as the eth3(backup interface) 10.56.23.0, however what OS also sees that the packet is coming to the management interface which is on 10.10.10.0 network.

So at that time, OS thinks that this packet should have come from the backup interface(eth3), as its network is the same as the sender of the packet. Well, at this point; the reverse path filter of OS comes into play and filters that packet and prevent the network connection between that client and the management interface. so ping doesn't work.  (any connection won't work actually)

This is a problem right? Yes , it is . This is a problem of the deployment actually, but as we may already deployed the Exadata machine, let's see what we can do to fix/or workaround it.

Well, this behaviour can be changed by a workaround like disabling the packet filter in Exadata Compute node Operating Systems by modifying the /etc/sysctl.conf file and running sysctl -p to make it effect. (it is on by default, so we can make it off)

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
and so on..

After disabling the reverse path filter, now when we ping the management interface (ping 10.10.10.100), we see that it works.

This move may not be supported by Oracle, but it solves the problem.
So this is an asymetric routing problem, and turning of rp filter does the job.

After disabling the filter , the client can ping. However it still can not telnet to Database listener port using the client network.

That is, the client with ip 10.56.23.81 is okay with the ping command after disabling the rp filter but now it wants to test the database connection. 

Now , it uses telnet to reach the listener port running on the client interface of one of the Exadata compute nodes (telnet 192.168.0.80 1523), however, this test fails.
The client cannot telnet to the client network using port 1523. (listener is up and running, no firewall is running).

So after disabling the rp filter, the problematic client can reach the management network via ping, but can not reach the client network via port 1523 using telnet.

Why?

ping uses icmp (level 3). On the other hand, telnet is upper level. When the client sends ping; the client first sends a ICMP Echo Request Packet to host and then, the host will reply with an ICMP Echo Reply to client.

However, as for the telnet, the tcp packets are dropped.
That is, as there is an asymetric routing happening out there. The network components visited all the way down from client to server and server to client such as routers/firewalls(any network components) block  the SYN-ACK  (the first server response in 3 way connection establishment of TCP) packets.  Remember, we disabled the reverse path filtering but , still we have a wrong route defined in our exadata compute nodes for the problematic client's network.
 That is , client still tries to connect to the management network, but the route for the server replies are still configured to be through the backup network.

Info:  Asymetric routing can be a problem for TCP which has strict state tracking but often does not affect "stateless" protocols such as ICMP or UDP.

In Exadata, we see some static routing are in place. When we look at the /etc/sysconfig/network-scripts directory of a compute node we see the rule and route files.

An example route for eth3 (backup interface):

10.56.23.0/24 dev eth3 table 223
default via 10.56.23.254 dev eth3 table 223

An example rule for eth3 (backup interface):

from 10.56.23.0/24 table 223
to 10.56.23.0/24 table 223

So these route and rule files simply say: use eth3 if you want to go to 10.56.23.0 (remember our problematic client was in the same network as the backup interface and its ip address was 10.56.23.81)

So when these files are in place, the routing table of the compute nodes have a static routing for eth3, so no matter what we do , we can't disable the path that packets will follow for going out from the server to the 10.56.23.* network.

But there is one thing that we can do ;

That is; we can simply change the backup network interface IP and take it to another network where there is no Exadata clients exists and lastly remove those static route and rule files from the /etc/sysconfig/network-scripts directory!

So as for disabling the static route(for backup interface) that comes with the deployment of Exadata, we simply remove the route-eth3 and rule-eth3 files and restart network services.

After this change, our compute node that we are dealing with does not contain any static route for the eth3 device, and then we can telnet to our listener using the client network. ( next move may be adding new route and rule files according to the new ip/network addresses of the backup interface)

This is what I did in our recent project. I did the exact same approach for working around the exact same problem and it did work.

I find this problem and its fix quite interesting and that's why shared this arcticle with you. Thanks for reading.