AI and LLM Tools

Current experiences with LLM-powered coding agents, self-hosted AI infrastructure, and automation workflows.

Kimi 2.5 and Self-Hosting Open WebUI

Been poking around with the Kimi 2.5 LLM and also started self-hosting Open WebUI on my server (a self-hosted ChatGPT-style web frontend for LLM APIs).

Kimi probably isn’t the best model on the market, but Kimi 2.5 is the first time I’ve used a truly open source model that feels to be vaguely in the same category of performance as ChatGPT, etc. And I don’t really feel much of a penalty using it vs ChatGPT.

Of course, running it directly is way beyond what any device I have can do reasonably well.

But there are already API providers around offering it with very favorable privacy and data retention policies, so I’m probably going to switch to using it over ChatGPT.

I wouldn’t recommend using the chat/API offered by the model’s creator–I don’t really trust that company.

If I self-host the front end, all of the actually sensitive data like chat logs etc are stored on my server.

Open WebUI is pretty cool. It works almost as well as ChatGPT does. I’ve run into some issues with the model occasionally freezing during processing, but I’ve occasionally seen that type of thing with other LLM providers.

It has a search integration that works with the model so it can web search etc. It’s pretty customizable.

I quickly created a custom tool that the model can use which queries the OpenAlex API to find open access academic articles. The code for that can be found here https://git.selfhosted.onl/theo/openwebui-tools-skills/src/branch/main

Experience Using Opencode on the Latest Models

I’ve been experimenting more with the latest LLM models for coding. And it’s pretty impressive how far things have come, and how these tools are pretty impressive.

I’ve mostly been using the Kimi 2.5 model with Opencode as the coding agent. I still find that mix pretty great. I think the whole vibe coding/AI assisted programming workflow that Opencode and similar encourage might not be the best for quality code, but it is pretty addictive seeing that type of rapid progress. Until you get to the very highest (expensive) tier of Claude/Anthropic and OpenAI models, Kimi performs basically on par or better than what the biggest companies offer.

And these coding agents can take care of a lot of the boring drudgework of programming. They are good enough right now that I don’t have to spend too much time manually intervening and fixing what the LLM did — these tools are getting pretty accurate.

I’ve spent many hours working on a project, tweaking it back and forth, with the main thing stopping me from spending even more time is the fact that I have to pay for the credits to run inference for the models. Once you spend the time working through all of quirks these tools have gotten pretty smooth as far as workflow goes. It’s genuinely fun to do this on the recent LLM models that have come out.

Cost right now is the only real problem, these things will burn through tokens by the millions. It’s pretty clear that the $20/mo for coding agents tier from OpenAI etc (even though the limits are being tightened) are being subsidized pretty aggressively. When you compare the amount of time you get on a coding agent from such a plan vs what open source alternatives cost, OpenAI etc can’t be making money from the coding agent offerings. On the other hand, it’s probably cheaper to use a self hosted frontend (OpenWebUI etc with a hosted inference API) than it is to pay for a paid tier of ChatGPT.

I’ve also noticed that Opencode and other open source agents/frontends are very sensitive to token output performance. Using a somewhat more expensive inference provider that provides fast performance will improve the experience quite a bit. Switching API providers basically fixed some of the issues I was having with the model freezing etc.

The project I’ve been working on as part of my testing is this https://git.selfhosted.onl/theo/marginleaf

It’s a personal blogging CMS. It can do the typical blogging engine things, but instead of a frontend editing interface, I created an API, and I built some tools that allow me to fully manage it from Open WebUI, which opens some pretty neat possibilities. It feels like a somewhat interesting possibility to have some of these chat tools getting good enough that they can be the main interface with an application — instead of a more traditional web UI.

In particular, the Open WebUI tools can be found here https://git.selfhosted.onl/theo/marginleaf/src/branch/main/openwebui_tools

But mostly I created it because it’s fun to work on that type of thing.

ChatGPT Makes Automation Symmetrical with Doing

One of the clearest implications of ChatGPT for systems administrators is that it makes automating a task almost symmetrical with doing a task.

On the new file server I use for personal projects (dedicated server with a NVMe SSD boot drive and four hard drives as secondary file storage drives, I recently did a reinstall of Debian. I set this server up with the hard drives in a BTRFS raid 5. I installed Docker on it, and I set up the Apache web server to make the files on that server public. Cloudflare Tunnel was used to put that Apache server behind SSL.

I took quick and rough notes on what commands were used and what may vary between servers and had ChatGPT create an automation script in Python.

The notes can be found here https://gist.github.com/theopjones/a7f2b6ba17f3de23826f688f0a87d01d

The prompt I used is

Create a python script to automate the server setup task in the following notes/log of a manual setup. Assume that the python script is running as root. In the case of commands which require manual intervention, wait for the user to conduct the manual intervention, the command should be started as part of the script.

The result of ChatGPT was the following, which is good enough to make this setup easily reproducible across servers or to document with code what the setup was so that it can be easily reproduced.

https://gist.github.com/theopjones/6147770b550356e55d209e67549fb948

See also: ai-legacy-experiments for earlier GPT-3 era work.