Max licenses combined with instances

Viewing 9 posts - 1 through 9 (of 9 total)
  • 2nd October 2019 at 1:08 pm #22695

    We are currently using the max licenses option for our nuke template and it works great, however we are also splitting our render nodes into multiple instances, 4 in this case, in order to better utilize them as nuke is not really good with that many cores. Our problem arises when using the max licenses in combination with multiple instances as there’s currently no way for Muster to distinguish between four nodes and one node split into four. Since license wise you can run infinite instances of nuke on the same machine and only use one license it would be great if there was a way to specify max licenses per node vs max licenses per instance as we’re currently running 4 jobs on the same node maxing out the “max licenses”, ie 1 node x 4 instances/license, instead of rendering on 4 nodes with 4 instances each, ie 4×4.
    So if there was a way/option for the max licenses to treat all instances of a node as one license that would be awesome.

    4th October 2019 at 9:12 am #22784

    Hi Alex,

    That sounds a reasonable request, we’ll see if we can push this change into 9.0.14 LTS for the upcoming patch release.

    Thanks

    10th October 2019 at 9:39 am #23073

    Hi Alex,

    Muster 9.0.14-11511 will implement a new model for license counting configurable into the MTemplate object. Further info in the release note, it will be released between today and tomorrow, if you get in touch with me on my email, I can send you the direct download info, the license is already into your account.

    Cheers!

    11th October 2019 at 9:31 am #23124

    you can download 11511, and use into a template:

    self.setLicensesMode(MTemplateAPI.MTemplate.kTemplateLicenseCountNodes

    17th October 2019 at 11:42 am #23421

    So I’ve installed the latest version and modified my template so it’s
    self.setMaximumLicenses(4)
    self.setLicensesMode(MTemplateAPI.MTemplate.kTemplateLicenseCountNodes)

    I now get another funky issue.

    All our nodes are split into 4 instances. I have a job submitted and the 1 by 1 I unpause the nodes.
    So when I have 1×4 nodes unpause all instances picks up the job, same for 2×4 and 3×4. So while I only have 3 nodes unpaused all 12 instances run the job. But as soon as I unpause the 4th nodes instances ie trying to run 4×4 in accordance with the self.setMaximumLicenses(4)
    each node only allows 1 instance to run the job, so I get 4 nodes running the job on 1 instance each. If I pause the 4th node again all 12 instances pick up the job just fine. Something fishy seems to be going on here

    17th October 2019 at 11:48 am #23422

    Hi Alex,

    This kind of check runs in a multithreaded environment, so it is possible that some instances are skipped because others are being selected in that moment. By pausing/clicking into Muster, you’re forcing the “selection logic” to be run again and further instances are picked. The selection logic has its own timeout and will re-run on its own after a couple of seconds. I suggest you to wait to see if the instances start on their own, while running in a production scenario this should not be a problem.

    17th October 2019 at 11:55 am #23425

    I’ve tried having a job running for 10+ minutes on 3 nodes vs 4 nodes enabled.
    With 3 nodes enabled we always get 3×4, 12 instances through the whole job.
    With 4 nodes enabled we always get 4×1, 4 instances through the whole job.

    So it doesn’t seem like selection logic is working properly and it’s not refreshing/rerunning after a few seconds.
    Almost seems like the setLicenseMode is active as long as the amount of enabled nodes are below self.setMaximumLicenses(4) and then shuts off or changes if it’s more or equal.

    17th October 2019 at 11:56 am #23426

    we will run some additional tests and let you know asap

    18th October 2019 at 11:07 am #23471

    We found an issue in the logic, please get in touch with me so I can send you a pre-release that address the issue, it will be part of the next service release.

    Thanks

Viewing 9 posts - 1 through 9 (of 9 total)

You must be logged in to reply to this topic.