Leonardo Bernardini

An additional note about this topic just to clarify how Muster works and some hints about high network usage scenarious:

As most of you know, Muster works with TCP connections. TCP connections are stream of bytes sent across the network that guarantee message delivering and bytes ordering.

The amount of data that Muster streams across the network has nothing to do with your renders, it’s related to controlling/synching messages used to keep track of render status, update the queue status and so on.

Unfortunately for us, the TCP protocol doesn’t provide a reliable way to detect broken connections (test it, if you cut your network cable O_O) your connections will stay alive unless the OS drop down all the active connections (like in happens on Windows).

That’s why we implemented the heartbeat feature in Muster, it’s just a pulse message that’s sent by the Renderclients to notify the Dispatcher they are still alive.

Unfortunately, even this method can raise problems. During a heavy network usage scenario, your entire network is flooded by packets belonging to your scene files and under direct control of your batch renders. This means there’s no way to tune the network usage and in such situation, a control/update message generated by Muster may take ages to be delivered. This includes the hb “pulse” messages as well as updates to the explorer’s queue and controlling messages (pause/reinit/kill)…..

Raising the heartbeat setting is a solution but this not improve your system usage, let’s look at a typical scenario:

You’ve 100 hosts currently rendering, due to the network usage, a controlling message may take up to 5 minutes to reach the Dispatcher. When a render is completed, an host sends a “render done” message to the Dispatcher and wait for further commands. This 5 minutes delay means that you’ll end with a render host idle for at least 5 minutes waiting for further commands, result: 5 minutes of wasted rendering times….

Improving the network bandwidth doesn’t solve the problems, if your jobs are big enough, they may be able to saturate even the fastest fiber connection.

There’re two hints users running big environments may follow:

1) Tune your network using QoS (Quality of services) tools. Those tools let you specify which kind of traffic should have priority. Giving full priority to the Muster TCP packets running on the Muster ports (7680, 7681, 7683, 7690) will make sure controlling messages will be delivering almost realtime. Do not worry about the network performances, the Muster network usage is really low. On a 100 hosts environment, It may reach a maximum of an average 1Kb/s on a standard idle usage.

2) If you own layer 4 switchs, the priority may be set at switch level, this will give you the same results of point 1.

3) Use a dual network setup. By using 2 network cards on each host, you can route the Muster controlling messages on a network, and the file streaming on the second. This may be done easily assigning different network masks like:

controlling network: 192.168.1.x
file-stream network 192.168.2.x

If you tell Muster to connect to the Dispatcher, i.e. to 192.168.1.1 and access the files on \192.168.2.1myreposity, you’re forcing a network routing that splits the messages….

Alternatively, you can setup the routing tables on your hosts, this may be a little more complicated….

Hope this helps the ones experiencing issues.

The keepalive/heartbeat feature is required by Muster to be warn if a node becames unresponsive. It’s a simple pulse that’s sent from the client to the server each X seconds, then the server sends an ACK back.

If networks are overloaded, it may be possible that this “pulse” gets a big amount of time to get back. It that case, there are two solutions:

– Raise the settings both on the Dispatcher and the renderclients (the Dispatcher value should be the double of the client value)

– Disable the heartbeat function both on the dispatcher and on the clients.

Update to this topic:

Current installers may have problems in including the Dispatcher service in the firewall rules. If that happens on your host, install the Dispatcher service as a console application, then include it in the firewall exception.

Open a dos prompt, navigate to the installation folder and type:

dispatcher -install

Then configure the user account from the service control panel applet.

Stop the renderclient service. Also skip the user account creation during the installation.
Create manually an user on the domain or on every host if you’re on a workgroup (assign the same password). Check the read/write policies of this user then:

Logon with this user and start the renderclient as a console application. If it still reports that kind of error, the problem is in the permissions. Double check your environment.

If it works, assign the user manually to the service using the Services control panel applet , then start it . It should work.

The latest release of Muster (5.21) includes a custom template for Mayaman. If you have no time for a full update, just install 5.21 on a PC where you’re not running Muster, and locate the mayaman template in the template folder. drop the template in your dispatcher installation and issue a soft-restart on the Dispatcher and the renderclients.

Regards.

To move a job out of a folder, you should drag to the bottom. Dragging multiple jobs is actually unsupported.

We will support it in future releases of version 5.

Regards.

Maybe a configuration issue. What kind of engine/template are you using ? And on what platform ? (Mac or linux ? )

If you get an exit code that differs from 0, it means that the mayabatch executable is failing to render.

You can find the log of each batch render session inside the “log” folder of the renderclient installation folder.

Check the reason of the failure and move on the service installation when you find the reason.

You can install the service manually from the command line using:

dispatcher -i
renderclient -i

After installing the services, locate them in the control panel and change the user according to your network permission settings.

Regards.