{"id":2300,"date":"2025-02-11T14:44:18","date_gmt":"2025-02-11T05:44:18","guid":{"rendered":"https:\/\/blog.peddals.com\/?p=2300"},"modified":"2025-11-15T22:39:20","modified_gmt":"2025-11-15T13:39:20","slug":"fine-tune-vram-size-of-mac-for-llm","status":"publish","type":"post","link":"https:\/\/blog.peddals.com\/en\/fine-tune-vram-size-of-mac-for-llm\/","title":{"rendered":"Optimizing VRAM Settings for Using Local LLM on macOS (Fine-tuning: 1)"},"content":{"rendered":"\n

When using a large language model (LLM) locally, the key point to pay attention to is how to run it at 100% GPU usage, that is, how to fit everything into VRAM (GPU memory). If the model overflows from VRAM, it can cause a decrease in response speed, make the entire OS laggy, and in the worst case, crash the OS.<\/p>\n\n\n\n

When using a local LLM, the combination of the parameter size and quantization size of the model that can be run, as well as the context length available for use, is generally determined by the capacity of the Unified Memory installed on an Apple Silicon Mac. This article will share methods to exceed the “set” limitations through some deeper settings, optimizing the processing speed and usable context length of local LLMs. If your Mac has a larger amount of Unified Memory installed, it becomes possible to run multiple LLMs or even larger models (= with higher performance) that were previously difficult to execute.<\/p>\n\n\n\n

Fine-tuning a generative AI model is not something amateurs can easily undertake, but since “environmental fine-tuning” is involved, you can easily try it out and see results right away. This covers the basics, so even if you’re a beginner you should give it a read if interested.<\/p>\n\n\n\n

First, let’s find out the model size that works on your Mac<\/h2>\n\n\n\n

Mac’s Unified Memory can be accessed by both the CPU and GPU, but there is a set proportion that the GPU can use. Based on some posts I’ve seen on forums, if no settings have been changed, for Unified Memory of 64GB or more<\/strong>, it seems that up to 3\/4 (75%)<\/strong> can be used by the GPU; for less than 64GB, about 2\/3 (approximately 66%)<\/strong> can be utilized. Since my Mac has 32GB RAM installed, this means the GPU can use up to 21.33GB of it. If LM Studio<\/a> is installed, you can check the hardware resources (Command + Shift + H), where VRAM will show something like the below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

When you see “Likely too large” in red while downloading a model in LM Studio, it is telling you that the model is too big for your VRAM capacity. The following screenshot shows that the DeepSeek R1<\/a> parameter size of 70B, with an 8-bit quantized MLX format model taking up 74.98GB, so it’s letting you know that it may not work on your environment.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

In Ollama<\/a>, similar value is output as recommendedMaxWorkingSetSize<\/code> in the log file. Below are the outputs from my environment (server2.log was the latest log file):<\/p>\n\n\n\n

<\/circle><\/circle><\/circle><\/g><\/svg><\/span>
%<\/span> <\/span>grep<\/span> <\/span>recommendedMaxWorkingSetSize<\/span> <\/span>~\/.ollama\/logs\/server2.log<\/span>|<\/span>tail<\/span> <\/span>-5<\/span><\/span>\nggml_metal_init:<\/span> <\/span>recommendedMaxWorkingSetSize<\/span>  <\/span>=<\/span> <\/span>22906.50<\/span> <\/span>MB<\/span><\/span>\nggml_metal_init:<\/span> <\/span>recommendedMaxWorkingSetSize<\/span>  <\/span>=<\/span> <\/span>22906.50<\/span> <\/span>MB<\/span><\/span>\nggml_metal_init:<\/span> <\/span>recommendedMaxWorkingSetSize<\/span>  <\/span>=<\/span> <\/span>22906.50<\/span> <\/span>MB<\/span><\/span>\nggml_metal_init:<\/span> <\/span>recommendedMaxWorkingSetSize<\/span>  <\/span>=<\/span> <\/span>22906.50<\/span> <\/span>MB<\/span><\/span>\nggml_metal_init:<\/span> <\/span>recommendedMaxWorkingSetSize<\/span>  <\/span>=<\/span> <\/span>22906.50<\/span> <\/span>MB<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Looking for a model of usable size (for beginners)<\/h2>\n\n\n\n
Just because a model you want to use is actually smaller than your VRAM does not mean it will be usable. The prompts you input and the text output by the LLM also use VRAM. Therefore, even if a model itself is 21GB in size, it won’t run smoothly. To find a model that fits within your actual VRAM, you would try models of plausible sizes in sequence based on the following information.<\/p>\n\n\n\n
    \n
  1. Look to models with fewer parameters<\/strong> (if 140B or 70B are not feasible, consider 32B \u2192 14B \u2192 7B, etc.)<\/li>\n\n\n\n
  2. Search for quantized<\/strong> models (such as 8bit, 4bit for MLX or Q8, Q4_K_M, etc. for GGUF format models)<\/li>\n<\/ol>\n\n\n\n
    Models with a smaller number of parameters tend to be created by distilling the original larger model or training them on less data. The goal is to reduce the amount of knowledge while minimizing degradation in features and performance. Depending on the capabilities and use cases of the models themselves, many popular ones these days are usable at around 10 to 30 billion parameters. With fewer parameters, the computation (inference) time also becomes shorter.<\/p>\n\n\n\n
    The other factor “quantization” is a method to reduce the size of a model using a different approach. Although this expression may not be common and might not be entirely accurate, it can be interpreted similarly to reducing resolution or decreasing color depth in images. While it’s not exactly the same upon closer inspection, it’s a technique that reduces the size to an extent where performance degradation is barely noticeable. Quantization also increases processing speed. Generally, it is said that with 8-bit or Q8 quantization, the benefits of faster processing and smaller size outweigh the percentage of performance loss. The model size decreases as the number gets smaller, but so does performance; therefore, around 4-bit or Q4_K_M would be considered the minimum threshold to maintain decent performance (the last letters S\/M\/L in GGUF format stand for Small\/Medium\/Large sizes).<\/p>\n\n\n\n
    After trying out several downloads, you see the maximum model size you can use on your Mac. In my case, with models that offer multiple parameter sizes, I try downloading one that pushes it to the limit at 32B Q4_K_M, and also download either F16 or Q8 of a smaller parameter like 14B.<\/p>\n\n\n\n
    Please note, when choosing a vision model, VLM, or so-called multimodal models, it is better to select ones that are even smaller in size compared to language models (LLM). This is because processing tasks such as reading images and determining what is depicted often requires more VRAM, given that images tend to be larger in size than text.<\/p>\n\n\n\n
    Download and use LLMs<\/h2>\n\n\n\n
    LM Studio allows you to download directly via the Download button and conduct chats through its GUI. For Ollama, after selecting a model on the Models<\/a> page, if there are options for parameter counts or quantization, choose them from the dropdown menu, then download and run using the Terminal.app (ollama run modelname<\/code>). Both applications can function as API servers, allowing you to use downloaded models from other apps. I often use Dify<\/a>, which makes it easy to create AI applications. For methods on how to use the APIs of Ollama and LM Studio via Dify, please check my posts below. (Japanese only for now. I’ll translate in the near future.)<\/p>\n\n\n\n
    \n
    Dify \u3068 Ollama \u3092\u5225\u3005\u306e Mac \u3067\u52d5\u304b\u3059\u30ed\u30fc\u30ab\u30eb LLM \u74b0\u5883<\/a><\/blockquote>