dropped nodes and chunks looping

we’ve been having trouble with dropped nodes, as well. I’m using Muster 5.3 – are you saying that I should increase the length of the heartbeat and the keep alive, or decrease it? I have about 230 nodes and having a lot of extra traffic between explorer and the render nodes seems to cause the dispatcher’s cpu usage to max out, making Muster unresponsive. Sometimes the nodes drop out before the render clients have completely loaded the scene – they come back, but thier chunks will have to be requed. Is there something I can do to my dispatcher network settings to stabilize this?

An additional note about this topic just to clarify how Muster works and some hints about high network usage scenarious:

As most of you know, Muster works with TCP connections. TCP connections are stream of bytes sent across the network that guarantee message delivering and bytes ordering.

The amount of data that Muster streams across the network has nothing to do with your renders, it’s related to controlling/synching messages used to keep track of render status, update the queue status and so on.

Unfortunately for us, the TCP protocol doesn’t provide a reliable way to detect broken connections (test it, if you cut your network cable O_O) your connections will stay alive unless the OS drop down all the active connections (like in happens on Windows).

That’s why we implemented the heartbeat feature in Muster, it’s just a pulse message that’s sent by the Renderclients to notify the Dispatcher they are still alive.

Unfortunately, even this method can raise problems. During a heavy network usage scenario, your entire network is flooded by packets belonging to your scene files and under direct control of your batch renders. This means there’s no way to tune the network usage and in such situation, a control/update message generated by Muster may take ages to be delivered. This includes the hb “pulse” messages as well as updates to the explorer’s queue and controlling messages (pause/reinit/kill)…..

Raising the heartbeat setting is a solution but this not improve your system usage, let’s look at a typical scenario:

You’ve 100 hosts currently rendering, due to the network usage, a controlling message may take up to 5 minutes to reach the Dispatcher. When a render is completed, an host sends a “render done” message to the Dispatcher and wait for further commands. This 5 minutes delay means that you’ll end with a render host idle for at least 5 minutes waiting for further commands, result: 5 minutes of wasted rendering times….

Improving the network bandwidth doesn’t solve the problems, if your jobs are big enough, they may be able to saturate even the fastest fiber connection.

There’re two hints users running big environments may follow:

1) Tune your network using QoS (Quality of services) tools. Those tools let you specify which kind of traffic should have priority. Giving full priority to the Muster TCP packets running on the Muster ports (7680, 7681, 7683, 7690) will make sure controlling messages will be delivering almost realtime. Do not worry about the network performances, the Muster network usage is really low. On a 100 hosts environment, It may reach a maximum of an average 1Kb/s on a standard idle usage.

2) If you own layer 4 switchs, the priority may be set at switch level, this will give you the same results of point 1.

3) Use a dual network setup. By using 2 network cards on each host, you can route the Muster controlling messages on a network, and the file streaming on the second. This may be done easily assigning different network masks like:

controlling network: 192.168.1.x
file-stream network 192.168.2.x

If you tell Muster to connect to the Dispatcher, i.e. to 192.168.1.1 and access the files on \192.168.2.1myreposity, you’re forcing a network routing that splits the messages….

Alternatively, you can setup the routing tables on your hosts, this may be a little more complicated….

Hope this helps the ones experiencing issues.

Timeouts are handled by the heartbeat, double check your config, you shouldn’t get any kind of timeout if hb is disabled…