Dynamic per job environment variables

Viewing 2 posts - 1 through 2 (of 2 total)
  • Back to forum
    Posted in: Python template system
  • Alex Oddbratt
    23rd May 2019 at 5:27 pm #17503

    We are looking at jumping over to Muster from Royal render and I’m currently investigating how to best implement dynamic environmental variables on a per job basis.

    So in our case we use a bunch of env vars and they are different per project/shot/job, 30+ vars currently.
    These are used mainly for specifying which plugins and versions are needed for this specific render as well as sending along information about shot/user/assets aso used for publishing.

    With our current setup using Royal Render we can generate a custom env file when submitting a job that containing all needed environmental variables and this env file is then injected in the commandline submission and is “added” on the fly to any machine before running a job. For example job 1 needs maya2017, arnold 3.0.1 and yeti 2.8 while job 2 needs other versions of those same plugins as well as maybe some other plugin that the first job doesn’t need. So those jobs will need different env vars set before kicking of the render.

    I’ve looked at the templates in Muster 9, last I used it we were still on Muster 8, and the onBuildEnvironment function seems like it could be a likely candidate but it seems to be limited per template.

    How would we go about “injecting” our custom env vars on a per job/submission basis?
    Basically have a job read an env file on disc and load all those env vars before launching the application and doing the render.
    Is it possible to send along a variable pointing to the env file that then get’s parsed in the onBuildEnvironment function?

    Leonardo Bernardini
    23rd May 2019 at 7:55 pm #17504

    Hi Alex,

    You have a pletora of different ways to handle your scenario:

    1) If you still want to submit a job using a command line tool (our mrtool cmd) and keep the current work flow as similar as the previous, you can directly pass a list of environmental variables using the mrtool flag (-attr) that let you pass a custom attribute to a job. Custom attributes are metadata that do not fit the standard jobs settings (like name, type, template etc etc). In that situation, the environment metadata is called “environmental_variables”. So the flag for mrtool may be something like -attr “environmental_variables” “value=1;value2=2” 0 . The last 0 means if you want to content of the attribute to be passed through the paths substitution engine, but I think that’s not your case so you can keep it to 0. Be aware also that inside environmental variables, you can use python inline evaluation (more on this on our wiki), so you can basically generate the environmental dynamically depending on the content of the MChunk or MJob object. If you want further info on this , ask in your reply.

    2) If you want to really stay with your current workflow, I assume that when you submit a job, you create an xml, json or whatever else file side by side with the submission. You can still take this path, just be sure to first create the environmental file and then submit the job. In that case, I suggest you to create a custom template derived from the one you need, and as you already guess, create the environment inside the template onBuildEnvironment. That function pass you also the MJob object, so you can read the variables and/or attributes to get your original file location, calculate the location of your environmental variable file, parse it and change the environment according.

    3) For a future proof solution by the way, I suggest you to submit the jobs using the REST APIs, that’s actually simpler than the tons of flags required by mrtool , or use a command line that triggers our Python interpreter (MPython inside muster installation folder) that in turns call a Python script that uses the MClientAPI python module, you can find into the SDK folder examples on how to submit a job. Attaching an environment is like with mrtool, you just add an environmental_variables attribute

    So pick the way you think is more comfortable for you and let us know how it goes and if you have further questions

    Have a nice day!

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic.