Recommended model parameters

CosmicAC recommends these values for each supported model's model master.

Qwen3-VL-235B-A22B-Thinking-FP8

Field	Value
job_type	`INFERENCE_VLLM`
base_os_image	`Ubuntu 22.04 + CUDA 13.0`
disk_gb	500
cuda_driver_version	`CUDA 13.0`
model_name	`Qwen/Qwen3-VL-235B-A22B-Thinking-FP8`
runtime_image	`vllm/vllm-openai:v0.15.1`
data_type	`Auto`
quantisation	`null`
tensor_parallel	8
per_replica_gpu_count	8
gpu_memory_utilisation	0.9
max_model_length	131072
max_concurrent_sequences	64
reasoning_parser	`deepseek_r1`
multimodal	`true`
replica	1
require_auth_header	`true`
min	`["base_os_image"]`
max	`["gpu_memory_utilisation"]`
inference_param_overrides.root_disk_size_gb	500
inference_param_overrides.env.TRUST_REMOTE_CODE	`true`
inference_param_overrides.env.SWAP_SPACE	`0`
inference_param_overrides.env.ENABLE_EXPERT_PARALLEL	`true`
inference_param_overrides.env.ENFORCE_EAGER	`false`

Field	Value
job_type	`INFERENCE_VLLM`
base_os_image	`Ubuntu 22.04 + CUDA 13.0`
disk_gb	500
cuda_driver_version	`CUDA 13.0`
model_name	`Qwen/Qwen3.5-122B-A10B`
runtime_image	`vllm/vllm-openai:v0.17.1`
data_type	`Auto`
quantisation	`FP8`
tensor_parallel	8
per_replica_gpu_count	8
gpu_memory_utilisation	0.9
max_model_length	32768
max_concurrent_sequences	32
reasoning_parser	`deepseek_r1`
multimodal	`true`
replica	1
require_auth_header	`true`
min	`["base_os_image"]`
max	`["gpu_memory_utilisation"]`
inference_param_overrides.root_disk_size_gb	500
inference_param_overrides.env.TRUST_REMOTE_CODE	`true`
inference_param_overrides.env.SWAP_SPACE	`0`
inference_param_overrides.env.ENABLE_EXPERT_PARALLEL	`true`
inference_param_overrides.env.ENFORCE_EAGER	`false`

Field	Value
job_type	`INFERENCE_VLLM`
base_os_image	`Ubuntu 22.04 + CUDA 13.0`
disk_gb	500
cuda_driver_version	`CUDA 13.0`
model_name	`MiniMaxAI/MiniMax-M2.5`
runtime_image	`vllm/vllm-openai:v0.15.1`
data_type	`Auto`
quantisation	`null`
tensor_parallel	4
per_replica_gpu_count	4
gpu_memory_utilisation	0.85
max_model_length	27000
max_concurrent_sequences	256
reasoning_parser	`deepseek_r1`
multimodal	`true`
replica	1
require_auth_header	`true`
min	`["base_os_image"]`
max	`["gpu_memory_utilisation"]`
inference_param_overrides.root_disk_size_gb	500
inference_param_overrides.env.TRUST_REMOTE_CODE	`true`
inference_param_overrides.env.SWAP_SPACE	`0`
inference_param_overrides.env.ENABLE_EXPERT_PARALLEL	`true`
inference_param_overrides.env.ENFORCE_EAGER	`false`

Field	Value
job_type	`INFERENCE_VLLM`
base_os_image	`Ubuntu 22.04 + CUDA 13.0`
disk_gb	150
cuda_driver_version	`CUDA 13.0`
ssh_public_key	`null`
model_name	`Qwen/Qwen2-VL-2B-Instruct`
runtime_image	`vllm/vllm-openai:v0.15.1`
data_type	`Auto`
quantisation	`null`
tensor_parallel	1
gpu_memory_utilisation	0.9
max_model_length	32768
max_concurrent_sequences	64
reasoning_parser	`default`
multimodal	`true`
replica	1
per_replica_gpu_count	1
require_auth_header	`true`
inference_param_overrides.root_disk_size_gb	150
inference_param_overrides.env.TRUST_REMOTE_CODE	`true`
inference_param_overrides.env.SWAP_SPACE	`0`
inference_param_overrides.env.ENFORCE_EAGER	`false`
min	`["base_os_image"]`
max	`["gpu_memory_utilisation"]`

The Parakeet model master stores a different set of fields. Parakeet Managed Inference parameters defines each one.

Field	Value
job_type	`INFERENCE_PARAKEET`
model_name	`nvidia/parakeet-tdt-0.6b-v3`
require_auth_header	`false`
chunk_duration	30
chunk_overlap	5
max_file_size_mb	1024
disk_gb	250
replica	1
inference_param_overrides.root_disk_size_gb	150
inference_param_overrides.env.MODEL_SOURCE	`huggingface`
inference_param_overrides.env.HF_HOME	`/mnt/external-disk/huggingface`
inference_param_overrides.env.NEMO_CACHE_DIR	`/mnt/external-disk/nemo`
inference_param_overrides.env.HOST	`0.0.0.0`
inference_param_overrides.env.INFERENCE_BACKEND	`parakeet`
inference_param_overrides.env.PORT	`8777`
inference_param_overrides.env.PARAKEET_STARTUP_TIMEOUT_MS	`600000`