Best Local LLM for German Language?

[vibe-code-repo]

Which LLM do you think is the best for performing classifications locally with Ollama? (It should run with a maximum of 24GB VRAM).

[icereed]

I found many to perform quite badly on German, especially on tagging. But there seems to be some movement towards optimized models for German: https://ollama.com/cas/discolm-mfto-german
Would be happy to see what has worked for others :)

[WildEchoWanderer]

Wow this discussion is what is exactly what i was wondering. I have a 3070 8GB VRAM available. Was wondering if i need more VRAM to be more precise🫠

this ones are kind of okay:

• qwen2.5:7b
• mistral-nemo:12b-instruct-2407-q4_0

I tried all of these: (German, translation below)

• 'qwen2.5:7b' # Testmodell 7b gutes resultat, bei 2. test korrespondent schwierig/auf chinesisch
• 'deepseek-r1:7b' #Test von deepseek->ungeeignet 2x getestet
• 'deepseek-r1:8b-llama-distill-q4_K_M' #Test von q4-deepseek->ungeeignet
• 'phi3.5:3.8b-mini-instruct-q3_K_S' #Test von minimodell bis 4GB VRAM->hmm weiss nicht, braucht vlt genauere prompts
• 'mistral-nemo:12b-instruct-2407-q4_0' #Test 2 minimodell bis 4g VRAM->hat relativ gut funktioniert, ist aber grösser
• 'phi3.5:3.8b' # 3.8B modell #sehr schnell, trotz bitcrack nebenbei, zu viele tags, sehr genau für 3.8b- mach immer "input" tags
• 'gemma:7b' #antwortet auf englisch...
• 'huggingface.co/TheBloke/em_german_mistral_v01-GGUF:latest' #schreibt käse, muss der prompt auch Deutsch sein?
• 'https://huggingface.co/caretech-owl/leo-hessionai-7B-quotes-gguf' #ist glaube ich zu gross for 8bg vram
• 'https://huggingface.co/deepfile/flan-t5-xl-gguf'#gibt error in ollama

_Translated with chatgpt:

• 'qwen2.5:7b' # Test model 7B, good result; in the second test, responses were difficult/came in Chinese
• 'deepseek-r1:7b' # Tested DeepSeek -> unsuitable, tested twice
• 'deepseek-r1:8b-llama-distill-q4_K_M' # Tested Q4 DeepSeek -> unsuitable
• 'phi3.5:3.8b-mini-instruct-q3_K_S' # Tested mini model up to 4GB VRAM -> not sure, might need more precise prompts
• 'mistral-nemo:12b-instruct-2407-q4_0' # Second mini model test up to 4GB VRAM -> worked relatively well but is larger
• 'phi3.5:3.8b' # 3.8B model -> very fast, even while running BitCrack; too many tags, very accurate for 3.8B – always uses "input" tags
• 'gemma:7b' # Responds in English...
• 'huggingface.co/TheBloke/em_german_mistral_v01-GGUF:latest' # Produces nonsense; does the prompt need to be in German?
• 'https://huggingface.co/caretech-owl/leo-hessionai-7B-quotes-gguf' # Probably too large for 8GB VRAM
• 'https://huggingface.co/deepfile/flan-t5-xl-gguf' # Causes an error in Ollama_

[WildEchoWanderer]

I now made some more tests and the qwen2.5-14b in k5 quant is realtively good. this uses more than only the gpu and is relatively slow, but okay with 32 gb ram ( ihave 64gb installed)

i tested these all:

• LLM_MODEL: 'cas/discolm-mfto-german' #zu gross
• LLM_MODEL: 'bengt0/em_german_leo_mistral' # zu Gross?
• LLM_MODEL: 'marco/em_german_mistral_v01' #grösse gut, schnell, prompts auf deutsch-> relativ gute antworten
• LLM_MODEL: 'hf.co/TheBloke/em_german_7b_v01-GGUF:Q5_K_M' #schnell, ergebnisse unbrauchbar bei prompts auf de, bei englischen prompts auch
• LLM_MODEL: 'hf.co/TheBloke/em_german_leo_mistral-GGUF:Q5_K_M' #zu gross?
• LLM_MODEL: 'hf.co/TheBloke/em_german_leo_mistral-GGUF:Q5_K_S' #zu gross?
• LLM_MODEL: 'hf.co/TheBloke/em_german_leo_mistral-GGUF:Q4_K_M' #geht irgendwie nicht?
• LLM_MODEL: 'hf.co/TheBloke/em_german_mistral_v01-GGUF:Q5_K_M' #geht irgendwie nicht?
• LLM_MODEL: 'hf.co/TheBloke/SauerkrautLM-7B-v1-mistral-GGUF:Q5_K_M' #irrelevante & keine antworten
• LLM_MODEL: 'hf.co/eramax/SauerkrautLM-7b-HerO-q8_0-gguf' #schlecht
• LLM_MODEL: 'qwen2.5:14b-instruct-q5_K_M' #sehr gutes ergebniss
• LLM_MODEL: 'qwen2.5:14b-instruct-q4_K_M' #
• LLM_MODEL: 'qwen2.5:32b-instruct-q5_K_M'#sehr langsam, ergebniss okay, aber mit klammern in title

[WildEchoWanderer]

I'm using qwen2.5-instruct:32b with custom promts engineered towards it with my RTX3090.

It has a near 100% success rate (or basically suggestions), it is super fast and with proper instructions even able to work on really gibberish OCR.

Using 178 documents about 80% of them were quite good in results - by that i mean non-quantitive jugement.

Its the only model that put out these results for me. Especially looking at dates ...

That given, maybe we can improve the prompting, say allowing different models per task, or including things like date extraction and allow more then a single prompt - I have quiet a few ideas here to improve results with other models too.

Seems like the rtx3070 has just not enough vram to get good results. I am tempted to get a rtx3090...

Can you share your custom prompts?

My Server has 8gb vram and 64 gb ram. 32b models can be loaded but they are nearly cpu only and way to slow...
14b was the biggest i had success

[thiscantbeserious]

I'm using qwen2.5-instruct:32b with custom promts engineered towards it with my RTX3090.
It has a near 100% success rate (or basically suggestions), it is super fast and with proper instructions even able to work on really gibberish OCR.
Using 178 documents about 80% of them were quite good in results - by that i mean non-quantitive jugement.
Its the only model that put out these results for me. Especially looking at dates ...
That given, maybe we can improve the prompting, say allowing different models per task, or including things like date extraction and allow more then a single prompt - I have quiet a few ideas here to improve results with other models too.

Seems like the rtx3070 has just not enough vram to get good results. I am tempted to get a rtx3090...

Can you share your custom prompts?

My Server has 8gb vram and 64 gb ram. 32b models can be loaded but they are nearly cpu only and way to slow... 14b was the biggest i had success

It works quite good with 14b too ...

#327

[thiscantbeserious]

So with CPU-only this is sadly not feasible. Maybe some boost comes in the rise of TPU processors. I have a rented server (Intel Xeon E3-1270V3; 4x RAM 8192 MB DDR3) that has not nearly the power to operate a local LLM for setting up AI apps/tools like document classification.

Any reason you are not getting something like that and simply add some DDR4 ram and a RTX 3060 with 12GB VRam instead?

https://www.ebay.de/itm/135709318642

[netvienna]

I had better results with qwen2.5 14B with strickt rules

[xlxl]

I had better results with qwen2.5 14B with strickt rules

Could you share the rules you used and how you enforce them?

[crgnntb7tt-source]

I am also courious about that topic. I have a Mac Mini M4 pro 24Gb - and qwen2.5:14b as LLM. Paperless NGX works fine, but PaperlessGPT just drops errors when trying to make doc-related suggestions.