Differences

This shows you the differences between two versions of the page.

Link to this comparison view

muster:9.0:databases_structure [2018/01/31 12:58] (current)
Line 1: Line 1:
 +====== Databases structure ======
 +
 +Muster stores the queue and internal configurations in a database that may be on a local file (using sqlite) or external (Mysql or SQL Server).
 +
 +An internal abstraction layer is used for certain data types, the field types you’ll find in the descriptions may appear differently depending on the destination database.
 +
 +The following section lists the internal tables and their purpose. **You are just allowed to read data from the Muster database. You are not allowed to write data neither change the data structure of the tables: Muster will check the structure on any service start-up and rebuild them according.**
 +
 +Queue database tables:
 +
 +  * db_jobs: Table holding the jobs general data either for folders and real jobs
 +  * db_jobs_data:​ Table holding attributes for real jobs as well as custom attributes
 +  * db_chunks: Table holding the chunks data
 +  * db_logs: Table holding the internal Muster log
 +  * db_mails: Table holding distribution lists
 +  * db_pools: Table holding pools configuration
 +  * db_users: Table holding users configuration
 +  * db_users_groups:​ Table holding groups configuration
 +  * db_paths: Table holding global substitution paths
 +
 +History database tables:
 +
 +  * db_history_chunks:​ Table holding the Muster processing history ​
 +  * db_history_stats:​ Table holding the Muster loads statistics
 +
 +Backup database tables:
 +
 +  * backup_db_jobs:​ Table holding the jobs general data either for folders and real jobs
 +  * backup_db_jobs_data:​ Table holding attributes for real jobs as well as custom attributes
 +  * backup_db_chunks:​ Table holding the chunks data
 +
 +===== Queue database =====
 +
 +==== db_jobs ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|parent_id|int|ID of the parent folder|
 +|indent|int|Level of indentation|
 +|status|int|Job status|
 +|tasks_mask|string|Job tasks mask|
 +|chunks_priority|int|The chunks priority type|
 +|chunks_interleave|int|The chunks priority interleave value|
 +|packet_type|int|Type of job packets|
 +|packet_size|int|Size of each chunk|
 +|requeued|int|Number of times job has been requeued|
 +|engine|int|Template uid|
 +|engine_version|string|The template version|
 +|max_cpu|int|Maximum number of concurrent instances|
 +|max_cpu_working|int|Number of current working instances|
 +|depend_id|string|Depend from ids|
 +|depend_mode|int|Depend mode used|
 +|depend_link_mode|int|Depend link mode|
 +|job_type|int|Job type, Folder or real job|
 +|job_name|string|Name of the job|
 +|priority|int|Priority for the job|
 +|is_paused|bool|Pause status|
 +|is_locked|bool|Locked status|
 +|job_pool|string|Target pools comma separated|
 +|Job_excluded_pool|string|Excluded pools comma separated|
 +|submission_time|int time_t|Time of submission|
 +|start_time|int time_t|Time of job execution|
 +|progress|int|Job progress|
 +|end_time|int time_t|Time of job termination|
 +|start_on|int time_t|Specify the scheduled starting time for the job|
 +|resume_on|int time_t|Specify the scheduled resume time for the job|
 +|pause_on|int time_t|Specify the scheduled paused time for the job|
 +|submitter|string|Username of the job owner|
 +|exitcodes_err_check_type|int|Specify the type of exit codes check|
 +|logs_err_check_type|int|Specify the type of logs texts error check|
 +|emergency_queue|bool|Specifies if the job is into the emergency queue|
 +|logs_parsing_rules|string|Encoded overriden logs parsing rules|
 +|override_notificator_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_notificator_notifications_at_job_level|int|Overrides of notifications|
 +|notificator_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|notificator_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_mobile_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_mobile_notifications_at_job_level|int|Overrides of notifications|
 +|mobile_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|mobile_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_mail_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_mail_notifications_at_job_level|int|Overrides of notifications|
 +|mail_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|mail_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_start_notificator_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_start_notificator_notifications_at_job_level|int|Overrides of notifications|
 +|start_notificator_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|start_notificator_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_start_mobile_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_start_mobile_notifications_at_job_level|int|Overrides of notifications|
 +|start_mobile_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|start_mobile_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_start_mail_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_start_mail_notifications_at_job_level|int|Overrides of notifications|
 +|start_mail_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|start_mail_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_chunks_timeout|boolean|Overrides chunks timeout state|
 +|override_chunks_timeout_value|int|Overrides chunks timeout value|
 +|is_archived|boolean|Job is archived|
 +|job_project|string|Job'​s project|
 +|job_department|string|Job'​s department|
 +|job_camera|string|Job'​s camera|
 +|job_shot|string|Job'​s shot|
 +|job_sequence|string|Job'​s sequence|
 +|override_valid_exit_codes|string|Overrides of exit codes|
 +|override_warning_exit_codes|string|Overrides of exit codes|
 +|override_error_exit_codes|string|Overrides of exit codes|
 +|override_minimum_threads|bool|Overrides minimum threads|
 +|override_minimum_cores|bool|Overrides minium cores|
 +|override_minimum_physical|bool|Overrides physical cores|
 +|override_minimum_speed|bool|Overrides minimum speed|
 +|override_minimum_ram|bool|Overrides minimum RAM|
 +|override_minimum_disk_space|bool|Overrides minimum disk space|
 +|override_maximum_chunks_requeue|bool|Overrides maximum chunks requeue|
 +|override_maximum_chunks_requeue_value|int|Overrides maximums chunks requeue value|
 +|override_valid_exit_codes_value|string|Overrides of exit codes value|
 +|override_valid_warning_codes_value|string|Overrides of exit codes value|
 +|override_error_exit_codes_value|string|Overrides of exit codes value|
 +|override_minimum_threads_value|int|Overrides minimum threads value|
 +|override_minimum_cores_value|int|Overrides minimum cores value|
 +|override_minimum_physical_value|int|Overrides minimum physical value|
 +|override_minimum_speed_value| floating |Overrides mimumum speed value|
 +|override_minimum_ram_value| floating |Overrides minimum RAM value|
 +|override_minimum_disk_space_value|floating|Overrides minimum disk space value|
 +
 +==== db_jobs_data ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|job_id|int|Job ID reference|
 +|key_name|string|Attribute name|
 +|key_value|string|Attribute value|
 +|key_state|bool|Attribute activation status|
 +|key_subst|bool|Attribute substitution status|
 +
 +==== db_chunks ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|chunk_id|int|Sequential chunk id|
 +|job_id|int|Job ID reference|
 +|priority|int|The chunk priority|
 +|status|int|Chunk status|
 +|packet_type|int|The chunk packet type|
 +|requeued|int|Number of times chunk has been requeued|
 +|start_frame|floating|Starting frame for the chunk|
 +|end_frame|floating|Ending frame for the chunk|
 +|by_frame|floating|By framing for the chunk|
 +|starting_time|int time_c|Chunk starting time|
 +|ending time|int time_c|Chunk ending time|
 +|rendering_host|string|Instance name that processed the chunk|
 +|result|string|Error string returned by the process|
 +|starting_number|int|Starting numbering for the chunk|
 +|is_assembler|bool|Flagged for assembler chunks|
 +|task|string|The chunk'​s job task|
 +|depend_from_chunks|string|Depends from chunks IDs|
 +|depend_from_chunks_error_levels|int|Depends error level|
 +|warning_markers|string|Warnings markers|
 +|error_markers|string|Error markers|
 +|silenced_warning_markers|string|Warning silenced markers|
 +|silenced_error_markers|string|Error silenced markers|
 +
 +==== db_logs ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|host_ip|string|IP of the host that generated the log|
 +|log_time|string|Formatted time of the log|
 +|host_name|string|Name of the instance that generated the log|
 +|event_user|string|Username of the user|
 +|log_entry|string|Text of the log entry|
 +|log_type|int|Log type|
 +|job_id|int|The job ID the log entry reefer too|
 +|chunk_id|int|The chunk ID the log entry reefer too|
 +|instance_id|int|The instance ID the log entry reefer too|
 +|history_chunk_id|int|The history chunk ID the log entry reefer too|
 +
 +==== db_paths ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|name|string|Repository name|
 +|local_path|string|Dispatcher side path|
 +|win|string|Windows side path|
 +|linux|string|Linux side path|
 +|mac|string|Mac side path|
 +|scope|int|Path scope|
 +
 +==== db_pools ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|name|string|The name of the pool or the node entry|
 +|nodes_priority|int|The nodes priority overrides|
 +|jobs_priority|int|The jobs priority overrides|
 +|enable_nodes_priority|bool|Enables nodes priority override|
 +|enable_jobs_priority|bool|Enables jobs priority override|
 +|autoPool_ip_filter|string|Filter to create an autopool|
 +|autoPool_hostname_filter|string|Hostname to create an autopool|
 +|time_rules|string|Time rules for the pool|
 +|pool_id|int|Parent pool id for a node entry|
 +|host_name|string|Host name for a node entry|
 +|host_ip|string|Host ip for a node entry|
 +
 +==== db_users ====
 +^Field name^Field type^Description^
 +|id|int|Primary key ID|
 +|account_name|string|Account name|
 +|groups|string|Account list of groups|
 +|encoded_password|string|Encoded password|
 +|email_address|string|Email address|
 +|allowed_rights|string|Allowed rights bit mask|
 +|forbidden_rights|string|Forbidden rights bit mask|
 +|allowed_mail_notifications|string|Notifications bit mask|
 +|blocked_mail_notifications|string|Notifications bit mask|
 +|allowed_notificator_notifications|string|Notifications bit mask|
 +|blocked_notificator_notifications|string|Notifications bit mask|
 +|allowed_mobile_notifications|string|Notifications bit mask|
 +|blocked_mobile_notifications|string|Notifications bit mask|
 +|allowed_templates_state|bool|Enable state|
 +|forbidden_templates_state|bool|Enable state|
 +|allowed_pools_state|bool|Enable state|
 +|forbidden_pools_state|bool|Enable state|
 +|maximum_instances_state|bool|Enable state|
 +|maximum_priority_state|bool|Enable state|
 +|enable_home_folder_state|bool|Enable state|
 +|constrain_to_home_folder_state|bool|Enable state|
 +|repositories_state|bool|Enable state|
 +|limited_pools_skip_on_idle_state|bool|Enable state|
 +|limited_instances_skip_on_idle_state|bool|Enable state|
 +|allowed_pools|string|Allowed pools|
 +|forbidden_pools|string|Forbidden pools|
 +|allowed_templates|string|Allowed templates|
 +|forbidden_templates|string|Forbidden templates|
 +|maximum_instances|int|Maximum instances|
 +|maximum_priority|int|Maximum priority|
 +|repositories|string|Repositories IDs list|
 +|home_folder_id|int|Home folder ID|
 +|is_ldap|bool|ldap imported object|
 +
 +==== db_users_groups ====
 +^Field name^Field type^Description^
 +|id|int|Primary key ID|
 +|group_name|string|Group name|
 +|allowed_rights|string|Allowed rights bit mask|
 +|allowed_mail_notifications|string|Notifications bit mask|
 +|allowed_notificator_notifications|string|Notifications bit mask|
 +|allowed_mobile_notifications|string|Notifications bit mask|
 +|allowed_templates_state|bool|Enable state|
 +|forbidden_templates_state|bool|Enable state|
 +|allowed_pools_state|bool|Enable state|
 +|forbidden_pools_state|bool|Enable state|
 +|maximum_instances_state|bool|Enable state|
 +|maximum_priority_state|bool|Enable state|
 +|enable_home_folder_state|bool|Enable state|
 +|constrain_to_home_folder_state|bool|Enable state|
 +|repositories_state|bool|Enable state|
 +|limited_pools_skip_on_idle_state|bool|Enable state|
 +|limited_instances_skip_on_idle_state|bool|Enable state|
 +|allowed_pools|string|Allowed pools|
 +|forbidden_pools|string|Forbidden pools|
 +|allowed_templates|string|Allowed templates|
 +|forbidden_templates|string|Forbidden templates|
 +|maximum_instances|int|Maximum instances|
 +|maximum_priority|int|Maximum priority|
 +|repositories|string|Repositories IDs list|
 +|is_ldap|bool|ldap imported object|
 +
 +==== db_sequences ====
 +^Field name^Field type^Description^
 +|table_name|string|Name of the table|
 +|next_id|int|Next valid table ID|
 +
 +==== db_nodes ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|instance_name|string|Name of the instance|
 +|instance_num|int|Instance number|
 +|host_ip|string|Ip address of the instance|
 +|host_name|string|Host name of the instance|
 +|instance_notes|string|Notes of the instance|
 +|platform|int|Platform of the instance|
 +|mac_addresses|string|List of instance NICs Mac addresses|
 +|wakeup_fullload|bool|Wakeup when at full load|
 +|wakeup_bypool|bool|Wakeup when required by pool|
 +|wakeup_policies|int|Wakeup policies|
 +|wakeup_match_availability|bool|Match availability status|
 +|availability_policies|string|Availability policies (encoded)|
 +|availableByDefault|bool|Availability default flag|
 +|status|int|Instance status|
 +
 +===== History database =====
 +
 +==== db_history_chunks ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|chunk_id|int|Chunk reference id|
 +|job_id|int|Job reference id|
 +|job_name|string|Job name|
 +|job_project|string|The job project|
 +|job_group|string|The job group|
 +|job_department|string|The job department|
 +|job_file|string|The job main file|
 +|task|string|The chunk task|
 +|status|int|Chunk final status|
 +|start_frame|floating|Chunk starting frame|
 +|end_frame|floating|Chunk ending frame|
 +|by_frame|floating|Chunk ending frame|
 +|starting_time|int time_t|Chunk starting time|
 +|ending_time|int time_t|Chunk ending time|
 +|rendering_host|string|Chunk assigned instance name|
 +|result|string|Chunk result text|
 +|starting_number|int|Chunk starting numbering|
 +|step_by|int|Chunk numbering step by|
 +|packet_size|int|Chunk packet size|
 +|packet_type|int|Chunk packet type|
 +|is_assembler|bool|Chunk assembler flag|
 +|warning_markers|string|Chunk warnings markers|
 +|error_markers|string|Chunk errors markers|
 +|silenced_warning_markers|string|Chunk silenced warning markers|
 +|silenced_error_markers|string|Chunk silenced error markers|
 +
 +==== db_history_stats ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|sample_time|int time_t|Time of the samples snapshot
 +|connected_rc|int|Number of connected clients|
 +|connected_ex|int|Number of connected consoles/​mrtool|
 +|connected_not|int|Number of connected notificators|
 +|connected_web_sessions|int|Number of active web sessions|
 +|connected_web|int|Number of active HTTP connections|
 +|connected_https|int|Number of active HTTPS connections|
 +|rc_used|int|Number of working nodes|
 +|rc_paused|int|Number of paused nodes|
 +|rc_idle|int|Number of idle nodes|
 +|rc_unavailable|int|Number of disconnected nodes|
 +|total_in_kbs|floating|Total amount of incoming KBytes|
 +|total_out_kbs|floating|Total amount of outcoming KBytes|
 +|control_out_kbs|floating|Console/​mrtool outcoming KBytes|
 +|control_in_kbs|floating|Console/​mrtool incoming KBytes|
 +|rc_in_kbs|floating|Nodes incoming KBytes|
 +|rc_out_kbs|floating|Nodes outcoming KBytes|
 +|not_in_kbs|floating|Notificators incoming KBytes|
 +|not_out_kbs|floating|Notificators outcoming KBytes|
 +|web_in_kbs|floating|Web incoming KBytes|
 +|web_out_kbs|floating|Web outcoming KBytes|
 +|connected_web_sessions|int|Number of concurrent HTTP(S) sessions|
 +
 +===== Backup database =====
 +
 +
 +==== backup_db_jobs ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|job_id|int|The ID of the original job this backup refer to|
 +|backup_time|int time_t|The time when the job has been backed up|
 +|parent_id|int|ID of the parent folder|
 +|indent|int|Level of indentation|
 +|status|int|Job status|
 +|tasks_mask|string|Job tasks mask|
 +|chunks_priority|int|The chunks priority type|
 +|chunks_interleave|int|The chunks priority interleave value|
 +|packet_type|int|Type of job packets|
 +|packet_size|int|Size of each chunk|
 +|requeued|int|Number of times job has been requeued|
 +|engine|int|Template uid|
 +|engine_version|string|The template version|
 +|max_cpu|int|Maximum number of concurrent instances|
 +|max_cpu_working|int|Number of current working instances|
 +|depend_id|string|Depend from ids|
 +|depend_mode|int|Depend mode used|
 +|depend_link_mode|int|Depend link mode|
 +|job_type|int|Job type, Folder or real job|
 +|job_name|string|Name of the job|
 +|priority|int|Priority for the job|
 +|is_paused|bool|Pause status|
 +|is_locked|bool|Locked status|
 +|job_pool|string|Target pools comma separated|
 +|Job_excluded_pool|string|Excluded pools comma separated|
 +|submission_time|int time_t|Time of submission|
 +|start_time|int time_t|Time of job execution|
 +|progress|int|Job progress|
 +|end_time|int time_t|Time of job termination|
 +|start_on|int time_t|Specify the scheduled starting time for the job|
 +|resume_on|int time_t|Specify the scheduled resume time for the job|
 +|pause_on|int time_t|Specify the scheduled paused time for the job|
 +|submitter|string|Username of the job owner|
 +|exitcodes_err_check_type|int|Specify the type of exit codes check|
 +|logs_err_check_type|int|Specify the type of logs texts error check|
 +|emergency_queue|bool|Specifies if the job is into the emergency queue|
 +|logs_parsing_rules|string|Encoded overriden logs parsing rules|
 +|override_notificator_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_notificator_notifications_at_job_level|int|Overrides of notifications|
 +|notificator_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|notificator_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_mobile_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_mobile_notifications_at_job_level|int|Overrides of notifications|
 +|mobile_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|mobile_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_mail_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_mail_notifications_at_job_level|int|Overrides of notifications|
 +|mail_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|mail_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_start_notificator_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_start_notificator_notifications_at_job_level|int|Overrides of notifications|
 +|start_notificator_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|start_notificator_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_start_mobile_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_start_mobile_notifications_at_job_level|int|Overrides of notifications|
 +|start_mobile_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|start_mobile_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_start_mail_notifications_at_chunk_level|int|Overrides of notifications|
 +|override_start_mail_notifications_at_job_level|int|Overrides of notifications|
 +|start_mail_notifications_at_chunk_level_type|int|Overrides of notifications|
 +|start_mail_notifications_at_job_level_type|int|Overrides of notifications|
 +|override_chunks_timeout|boolean|Overrides chunks timeout state|
 +|override_chunks_timeout_value|int|Overrides chunks timeout value|
 +|is_archived|boolean|Job is archived|
 +|job_project|string|Job'​s project|
 +|job_department|string|Job'​s department|
 +|job_camera|string|Job'​s camera|
 +|job_shot|string|Job'​s shot|
 +|job_sequence|string|Job'​s sequence|
 +|override_valid_exit_codes|string|Overrides of exit codes|
 +|override_warning_exit_codes|string|Overrides of exit codes|
 +|override_error_exit_codes|string|Overrides of exit codes|
 +|override_minimum_threads|bool|Overrides minimum threads|
 +|override_minimum_cores|bool|Overrides minium cores|
 +|override_minimum_physical|bool|Overrides physical cores|
 +|override_minimum_speed|bool|Overrides minimum speed|
 +|override_minimum_ram|bool|Overrides minimum RAM|
 +|override_minimum_disk_space|bool|Overrides minimum disk space|
 +|override_maximum_chunks_requeue|bool|Overrides maximum chunks requeue|
 +|override_maximum_chunks_requeue_value|int|Overrides maximums chunks requeue value|
 +|override_valid_exit_codes_value|string|Overrides of exit codes value|
 +|override_valid_warning_codes_value|string|Overrides of exit codes value|
 +|override_error_exit_codes_value|string|Overrides of exit codes value|
 +|override_minimum_threads_value|int|Overrides minimum threads value|
 +|override_minimum_cores_value|int|Overrides minimum cores value|
 +|override_minimum_physical_value|int|Overrides minimum physical value|
 +|override_minimum_speed_value| floating |Overrides mimumum speed value|
 +|override_minimum_ram_value| floating |Overrides minimum RAM value|
 +|override_minimum_disk_space_value|floating|Overrides minimum disk space value|
 +==== backup_db_jobs_data ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|job_id|int|Job ID reference|
 +|key_name|string|Attribute name|
 +|key_value|string|Attribute value|
 +|key_state|bool|Attribute activation status|
 +|key_subst|bool|Attribute substitution status|
 +
 +
 +==== backup_db_chunks ====
 +^Field name^Field type^Description^
 +|id|int|Primary key|
 +|chunk_id|int|Sequential chunk id|
 +|job_id|int|Job ID reference|
 +|priority|int|The chunk priority|
 +|status|int|Chunk status|
 +|packet_type|int|The chunk packet type|
 +|requeued|int|Number of times chunk has been requeued|
 +|start_frame|floating|Starting frame for the chunk|
 +|end_frame|floating|Ending frame for the chunk|
 +|by_frame|floating|By framing for the chunk|
 +|starting_time|int time_c|Chunk starting time|
 +|ending time|int time_c|Chunk ending time|
 +|rendering_host|string|Instance name that processed the chunk|
 +|result|string|Error string returned by the process|
 +|starting_number|int|Starting numbering for the chunk|
 +|is_assembler|bool|Flagged for assembler chunks|
 +|task|string|The chunk'​s job task|
 +|depend_from_chunks|string|Depends from chunks IDs|
 +|depend_from_chunks_error_levels|int|Depends error level|
 +|warning_markers|string|Warnings markers|
 +|error_markers|string|Error markers|
 +|silenced_warning_markers|string|Warning silenced markers|
 +|silenced_error_markers|string|Error silenced markers|