WordCrafter Pro - Local Models testing

🔥

I just thought I would share my experiences using local models with WordCrafter Pro. First a bit of my setup I run a Mac Studio M4 with 128GB RAM. So overkill for most things but large enough to run some larger size LLMs.

With WordCrafter Pro I originally tried Ollama. I was able to see my local models without a problem however if I tried to connect with any of them, even setting context length, I kept getting 504 errors (if I remember correctly). So I moved over to LM Studio and tried with Gemma 4 31B with 128K context window. Again I could see my models but in case I could actually connect.

I started with BrainStorm and threw it a silly throw away idea. Working through the BrainStorm process it was able to take this more or less random prompt and help craft it into something actual workable. I was able to take this and go through Story Dev then Character Creator. It seemed to more or less keep the conversation coherent and not drift too much from what we worked out. I did cause some direction changes, but much less than I normally would when working through a project, and it seemed to handle these changes fine. In this case I simply wanted evidence that it could work with the model then a finished project I would work on. Finally, I had it create the first chapter. The prose was more or less the standard AI generated I would expect but it generally kept to the story I wanted to create.

In the process I did have a few cases where the prompt would process up to 100% on my side within LM Studio (i.e. finished) but then get disconnected and nothing showed up within WordCrafter Pro. Some of these might be some of the errors

@Michael Culp

was mentioning about in his post. Not sure who was at fault. However, asking it to 'continue' within the application seemed to work and allow it to finish the process as expected. Though I was starting to get the impression that the more context included within a prompt to the local model, the more likely it would run into problems and/or not be able to process at all.

Therefore, overall impression is that with a powerful enough model I think creating a short story with a local model is very viable. I am less certain how it would do with a complete novel. Also, frontier models will no doubt give better results but if you are willing to do a bit of work this might be a viable solution to save a little on token expenses.

2 comments

WordCrafter Pro - Local Models testing