This scenario has come up a couple of times with customers I have been working with and it is centered around the following situation/question .

I have deployed 2 Log Insight platforms, one for each of my datacenters (assume one vCenter per DC).  How do I get all the events in both Log Insight clusters so I can use either for log analysis of the entire environment?

Note: The reasons behind why you would want to achieve the above scenario are outside the scope of this article.

There are several solutions that could be implemented to perform log analysis for both sites from either location including:

• Configuring all syslog/log insight agents to send events to both locations simultaneously AND configure each cluster to pull logs from both vCenter servers via the vSphere Integration
• Each DC sends logs to its local Log Insight platform with the platforms forwarding events between themselves

The problem with the first solution is that you have much more manual configuration to add and maintain (depending on the number of agents in the environment).  Every agent has to be multi-homed which has an additional management overhead together with an increased liklihood of potential comms issues running logs/agents out of buffer space, resulting in dropped events.

The second solution allows for simplified syslog/agent configuration and places the control of event movement within Log Insight itself which can be controlled globally across all agents that are logging to Log Insight.  The issue that results from this solution though is how to stop events being trapped within infinite loops between the Log Insight platforms.  We will use this blog article to address this issue.

Creating the Forwarder

To start building our forwarder on the first Log Insight cluster we go to “Administration” > “Management” > “Event Forwarding” and add a new destination.  To identify all events being forwarded by this cluster, we add a custom tag with a value that represents cluster 1.

At this stage everything that is being received on cluster 1 will now be forwarded to cluster 2.  On cluster 2 we now add the same forwarder except this time the tag has cluster 2’s name and we add a filter to prevent events that have already been forwarded into cluster 2 being forwarded back to cluster 1.

Note that the tag is only available to be selected in the filter once an event has been received by the cluster with the tag in it.

Once an event has been sent from cluster 2 to cluster 1 (and therefore tagged) we can go back and edit the event forwarding configuration on cluster 1 to add the filtering information.

Now we have both forwarders configured with tags and filters to prevent infinite forwarding loops.