Confirmed Environment<\/h2>\n\n\n\nOS and app versions:<\/h3>\n\n\n\nmacOS: 14.5\nOllama: 0.3.8\nDify: 0.6.15\nVisual Studio Code - Insiders: 1.93.0-insider\nContinue: 0.8.47<\/code><\/pre><\/div>\n\n\n\nLLM and size<\/h3>\n\n\n\nModel name<\/td> Model size<\/td> Context length<\/td> Ollama download command<\/td><\/tr> llama3.1:8b-instruct-fp16<\/td> 16 GB<\/td> 131072<\/td> ollama pull llama3.1:8b-instruct-fp16<\/code><\/td><\/tr>deepseek-coder-v2:16b-lite-instruct-q8_0<\/td> 16 GB<\/td> 163840<\/td> ollama run deepseek-coder-v2:16b-lite-instruct-q8_0<\/code><\/td><\/tr>deepseek-coder-v2:16b-lite-instruct-q6_K<\/td> 14 GB<\/td> 163840<\/td> ollama pull deepseek-coder-v2:16b-lite-instruct-q6_K<\/code><\/td><\/tr><\/tbody><\/table>Mac with 32GB RAM is capable of running them on memory.<\/figcaption><\/figure>\n\n\n\nConclusion<\/h2>\n\n\n\n
Check the context length and lower it.<\/p>\n\n\n\n
By setting “Size of context window” in Dify or Continue to a sufficiently small value, you can solve this problem. Don’t set a number just because the model supports it or for future use; instead, use the default value (2048) or 4096 and test chatting with a small number of words. If you get a response as you expect, congrats, the issue is resolved.<\/p>\n\n\n\n
Context size: It is also called \"context window\" or \"context length.\" It represents the total number of tokens that an LLM can process in one interaction. Token count is approximately equal to word count in English and other supported languages. In the table above, Llama 3.1 has a context size of 131072, so it can handle approximately 65,536 words text as input and output.<\/pre>\n\n\n\nChanging Context Length<\/h2>\n\n\n\nDify <\/h3>\n\n\n\n\n- Open the LLM block in the studio app and click on the model name to access detailed settings.<\/li>\n\n\n\n
- Scroll down to find “Size of cont…” (Size of content window) and uncheck it or enter 4096.<\/li>\n\n\n\n
- The default value is 2048 when unchecked.<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/blog.peddals.com\/wp-content\/uploads\/2024\/08\/image-13-1024x525.jpg\")
<\/figure>\n\n\n\nContinue (VS Code LLM extension)<\/h3>\n\n\n\n\n- Open the config.json file in the Continue pane’s gear icon.<\/li>\n\n\n\n
- Change the
contextLength<\/code> and maxTokens<\/code> values to 4096<\/code> and 2048<\/code>, respectively. Note that maxTokens<\/code> is the maximum number of tokens generated by the LLM, so we set it half.<\/li>\n<\/ul>\n\n\n\n![\"\"](\"https:\/\/blog.peddals.com\/wp-content\/uploads\/2024\/08\/image-13.png\")
<\/figure>\n\n\n\n {\n "title": "Chat: llama3.1:8b-instruct-fp16",\n "provider": "ollama",\n "model": "llama3.1:8b-instruct-fp16",\n "apiBase": "http:\/\/localhost:11434",\n "contextLength": 4096,\n "completionOptions": {\n "temperature": 0.5,\n "top_p": "0.5",\n "top_k": "40",\n "maxTokens": 2048,\n "keepAlive": 3600\n }\n }<\/code><\/pre><\/div>\n\n\n\nChecking Context Length of LLM<\/h2>\n\n\n\n
The easiest way is to use the Ollama’s command ollama show <modelname><\/code> to display the context length<\/code>. Example:<\/p>\n\n\n\n% ollama show llama3.1:8b-instruct-fp16\n Model \n \tarch \tllama \t \n \tparameters \t8.0B \t \n \tquantization \tF16 \t \n \tcontext length \t131072\t \n \tembedding length\t4096 \t \n \t \n Parameters \n \tstop\t"<|start_header_id|>"\t \n \tstop\t"<|end_header_id|>" \t \n \tstop\t"<|eot_id|>" \t \n \t \n License \n \tLLAMA 3.1 COMMUNITY LICENSE AGREEMENT \t \n \tLlama 3.1 Version Release Date: July 23, 2024<\/code><\/pre><\/div>\n\n\n\nContext Length in App Settings<\/h2>\n\n\n\nDify > Model Provider > Ollama<\/h3>\n\n\n\n
When adding an Ollama model to Dify, you can override the default value of 4096 for Model context length and Upper bound for max tokens. Since setting a upper limit may make debugging difficult if issues arise, it’s better to set both values to the model’s context length and adjust the Size of content window in individual AI apps.<\/p>\n\n\n\n![\"\"](\"https:\/\/blog.peddals.com\/wp-content\/uploads\/2024\/08\/image-14-740x1024.png\")
<\/figure>\n\n\n\nContinue > “models”<\/h3>\n\n\n\n
In the “models<\/code>” section of the config.json, you can add multiple settings for different context length by including a description like “Fastest Max Size<\/code>” or “4096<\/code>“. For example, I set the title to “Chat: llama3.1:8b-instruct-fp16 (Fastest Max Size)” and changed the contextLength<\/code> value to 24576<\/code> and maxTokens<\/code> value to 12288<\/code>. This combination was the highest that I confirmed working perfectly on my Mac with 32 GB RAM.<\/p>\n\n\n\n {\n "title": "Chat: llama3.1:8b-instruct-fp16 (Fastest Max Size)",\n "provider": "ollama",\n "model": "llama3.1:8b-instruct-fp16",\n "apiBase": "http:\/\/localhost:11434",\n "contextLength": 24576,\n "completionOptions": {\n "temperature": 0.5,\n "top_p": "0.5",\n "top_k": "40",\n "maxTokens": 12288,\n "keepAlive": 3600\n }\n }<\/code><\/pre><\/div>\n\n\n\nWhat’s happening when LLM processing is slow (based on what I see)<\/h2>\n\n\n\n
When using\u00a0ollama run<\/code>, LLM runs quickly, but when using Ollama through Dify or Continue, it becomes slow due to large size of context length. Let’s check the process with\u00a0ollama ps<\/code>. Below are examples – first one had the max context length 131072 and the second one had 24576:\u00a0<\/p>\n\n\n\n% ollama ps\nNAME \tID \tSIZE \tPROCESSOR \tUNTIL \nllama3.1:8b-instruct-fp16\ta8f4d8643bb2\t49 GB\t54%\/46% CPU\/GPU\t59 minutes from now\t\n\n% ollama ps\nNAME \tID \tSIZE \tPROCESSOR\tUNTIL \nllama3.1:8b-instruct-fp16\ta8f4d8643bb2\t17 GB\t100% GPU \t4 minutes from now<\/code><\/pre><\/div>\n\n\n\nIn the slow case, SIZE is much larger than the actual model size (16 GB), and processing occurs on CPU at 54% and GPU at 46%. It seems that Ollama processes LLM as a larger size model when a large size context length is passed via API regardless of the actual number of tokens being processed. This is only my assumption, but the above tells.<\/p>\n\n\n\n
Finding a suitable size of context length<\/h2>\n\n\n\n
After understanding the situation, let’s take countermeasures. If you can live with 4096 tokens, it’s fine, but I want to process as many tokens as possible. Unfortunately, I couldn’t find Ollama’s specifications, so I tried adjusting the context length by hand and found that a value of 24576 (4096*6) works for Llama 3.1 8B F16 and DeepSeek-Coder-V2-Lite-Instruct Q6_K.<\/p>\n\n\n\n
Note that using non-multiple-of-4096 values may cause character corruption, so be careful. Also, when using Dify, the SIZE value will be smaller than in Continue.<\/p>\n\n\n\n
Ollama, I’m sorry (you can skip this)<\/h2>\n\n\n\n
I thought Ollama’s server processing was malfunctioning because LLM ran quickly when running on CLI but became slow when used through API. However, after trying an advice “Try setting context length to 4096” from an issue discussion about Windows + GPU, I found that it actually solved the problem.<\/p>\n\n\n\n
Ollama, I’m sorry for doubting you!<\/p>\n\n\n\n
macOS: 14.5\nOllama: 0.3.8\nDify: 0.6.15\nVisual Studio Code - Insiders: 1.93.0-insider\nContinue: 0.8.47<\/code><\/pre><\/div>\n\n\n\nLLM and size<\/h3>\n\n\n\nModel name<\/td> Model size<\/td> Context length<\/td> Ollama download command<\/td><\/tr> llama3.1:8b-instruct-fp16<\/td> 16 GB<\/td> 131072<\/td> ollama pull llama3.1:8b-instruct-fp16<\/code><\/td><\/tr>deepseek-coder-v2:16b-lite-instruct-q8_0<\/td> 16 GB<\/td> 163840<\/td> ollama run deepseek-coder-v2:16b-lite-instruct-q8_0<\/code><\/td><\/tr>deepseek-coder-v2:16b-lite-instruct-q6_K<\/td> 14 GB<\/td> 163840<\/td> ollama pull deepseek-coder-v2:16b-lite-instruct-q6_K<\/code><\/td><\/tr><\/tbody><\/table>Mac with 32GB RAM is capable of running them on memory.<\/figcaption><\/figure>\n\n\n\nConclusion<\/h2>\n\n\n\n
Check the context length and lower it.<\/p>\n\n\n\n
By setting “Size of context window” in Dify or Continue to a sufficiently small value, you can solve this problem. Don’t set a number just because the model supports it or for future use; instead, use the default value (2048) or 4096 and test chatting with a small number of words. If you get a response as you expect, congrats, the issue is resolved.<\/p>\n\n\n\n
Context size: It is also called \"context window\" or \"context length.\" It represents the total number of tokens that an LLM can process in one interaction. Token count is approximately equal to word count in English and other supported languages. In the table above, Llama 3.1 has a context size of 131072, so it can handle approximately 65,536 words text as input and output.<\/pre>\n\n\n\nChanging Context Length<\/h2>\n\n\n\nDify <\/h3>\n\n\n\n\n- Open the LLM block in the studio app and click on the model name to access detailed settings.<\/li>\n\n\n\n
- Scroll down to find “Size of cont…” (Size of content window) and uncheck it or enter 4096.<\/li>\n\n\n\n
- The default value is 2048 when unchecked.<\/li>\n<\/ul>\n\n\n\n
<\/figure>\n\n\n\nContinue (VS Code LLM extension)<\/h3>\n\n\n\n\n- Open the config.json file in the Continue pane’s gear icon.<\/li>\n\n\n\n
- Change the
contextLength<\/code> and maxTokens<\/code> values to 4096<\/code> and 2048<\/code>, respectively. Note that maxTokens<\/code> is the maximum number of tokens generated by the LLM, so we set it half.<\/li>\n<\/ul>\n\n\n\n<\/figure>\n\n\n\n {\n "title": "Chat: llama3.1:8b-instruct-fp16",\n "provider": "ollama",\n "model": "llama3.1:8b-instruct-fp16",\n "apiBase": "http:\/\/localhost:11434",\n "contextLength": 4096,\n "completionOptions": {\n "temperature": 0.5,\n "top_p": "0.5",\n "top_k": "40",\n "maxTokens": 2048,\n "keepAlive": 3600\n }\n }<\/code><\/pre><\/div>\n\n\n\nChecking Context Length of LLM<\/h2>\n\n\n\n
The easiest way is to use the Ollama’s command ollama show <modelname><\/code> to display the context length<\/code>. Example:<\/p>\n\n\n\n% ollama show llama3.1:8b-instruct-fp16\n Model \n \tarch \tllama \t \n \tparameters \t8.0B \t \n \tquantization \tF16 \t \n \tcontext length \t131072\t \n \tembedding length\t4096 \t \n \t \n Parameters \n \tstop\t"<|start_header_id|>"\t \n \tstop\t"<|end_header_id|>" \t \n \tstop\t"<|eot_id|>" \t \n \t \n License \n \tLLAMA 3.1 COMMUNITY LICENSE AGREEMENT \t \n \tLlama 3.1 Version Release Date: July 23, 2024<\/code><\/pre><\/div>\n\n\n\nContext Length in App Settings<\/h2>\n\n\n\nDify > Model Provider > Ollama<\/h3>\n\n\n\n
When adding an Ollama model to Dify, you can override the default value of 4096 for Model context length and Upper bound for max tokens. Since setting a upper limit may make debugging difficult if issues arise, it’s better to set both values to the model’s context length and adjust the Size of content window in individual AI apps.<\/p>\n\n\n\n<\/figure>\n\n\n\nContinue > “models”<\/h3>\n\n\n\n
In the “models<\/code>” section of the config.json, you can add multiple settings for different context length by including a description like “Fastest Max Size<\/code>” or “4096<\/code>“. For example, I set the title to “Chat: llama3.1:8b-instruct-fp16 (Fastest Max Size)” and changed the contextLength<\/code> value to 24576<\/code> and maxTokens<\/code> value to 12288<\/code>. This combination was the highest that I confirmed working perfectly on my Mac with 32 GB RAM.<\/p>\n\n\n\n {\n "title": "Chat: llama3.1:8b-instruct-fp16 (Fastest Max Size)",\n "provider": "ollama",\n "model": "llama3.1:8b-instruct-fp16",\n "apiBase": "http:\/\/localhost:11434",\n "contextLength": 24576,\n "completionOptions": {\n "temperature": 0.5,\n "top_p": "0.5",\n "top_k": "40",\n "maxTokens": 12288,\n "keepAlive": 3600\n }\n }<\/code><\/pre><\/div>\n\n\n\nWhat’s happening when LLM processing is slow (based on what I see)<\/h2>\n\n\n\n
When using\u00a0ollama run<\/code>, LLM runs quickly, but when using Ollama through Dify or Continue, it becomes slow due to large size of context length. Let’s check the process with\u00a0ollama ps<\/code>. Below are examples – first one had the max context length 131072 and the second one had 24576:\u00a0<\/p>\n\n\n\n% ollama ps\nNAME \tID \tSIZE \tPROCESSOR \tUNTIL \nllama3.1:8b-instruct-fp16\ta8f4d8643bb2\t49 GB\t54%\/46% CPU\/GPU\t59 minutes from now\t\n\n% ollama ps\nNAME \tID \tSIZE \tPROCESSOR\tUNTIL \nllama3.1:8b-instruct-fp16\ta8f4d8643bb2\t17 GB\t100% GPU \t4 minutes from now<\/code><\/pre><\/div>\n\n\n\nIn the slow case, SIZE is much larger than the actual model size (16 GB), and processing occurs on CPU at 54% and GPU at 46%. It seems that Ollama processes LLM as a larger size model when a large size context length is passed via API regardless of the actual number of tokens being processed. This is only my assumption, but the above tells.<\/p>\n\n\n\n
Finding a suitable size of context length<\/h2>\n\n\n\n
After understanding the situation, let’s take countermeasures. If you can live with 4096 tokens, it’s fine, but I want to process as many tokens as possible. Unfortunately, I couldn’t find Ollama’s specifications, so I tried adjusting the context length by hand and found that a value of 24576 (4096*6) works for Llama 3.1 8B F16 and DeepSeek-Coder-V2-Lite-Instruct Q6_K.<\/p>\n\n\n\n
Note that using non-multiple-of-4096 values may cause character corruption, so be careful. Also, when using Dify, the SIZE value will be smaller than in Continue.<\/p>\n\n\n\n
Ollama, I’m sorry (you can skip this)<\/h2>\n\n\n\n
I thought Ollama’s server processing was malfunctioning because LLM ran quickly when running on CLI but became slow when used through API. However, after trying an advice “Try setting context length to 4096” from an issue discussion about Windows + GPU, I found that it actually solved the problem.<\/p>\n\n\n\n
Ollama, I’m sorry for doubting you!<\/p>\n\n\n\n