Homo-Adminus Blog

Earlier today, Nick Rempel mentioned in a tweet how important it is to set up tight feedback loops for AI agents when you’re working on something:

I could definitely relate to his sentiment. In all of my projects over the past few weeks I have now switched to a TDD approach where I explicitly ask an agent to write a set of tests describing the feature they are going to build (step 1) and then ask them to ensure that all of the tests fail in exactly the way they expected them to fail (step 2).

This essentially sets up the first loop of writing tests: spell out your assumptions, execute them, see them fail, repeat until your model of the world aligns with the reality you’re going to be changing. After the tests are done I ask the model to commit them and start fixing them until they pass. That is the second loop.

It is an extremely effective way of keeping the agent from making wildly wrong assumptions about the system that usually lead to poor results.

But that was me working on code in isolated environments. And then today I had a really cool related experience while working on real infrastructure.

I had a problem with some software on a personal server that we have deployed yesterday as a part of the my secret “Jarvis home assistant” project. Before attempting a fix myself, I decided to see if the agent could figure it out.

Setting up the “test”

First, I told it that the app was not opening in Chrome and gave it the error I was seeing (connection refused error). This set up the goal for the agent – find and fix the problem.

Then I asked the agent to quickly create me a comprehensive health check script that would verify every thing the agent can imagine breaking within that specific component and display health information to the console in a clean and easy to understand form.

If you ever worked with an agent on building tests, you know how weird they are in coming up with insane edge cases and generally in covering the possibility space. So, that’s exactly what happened here: The agent came up with an insane thousand line script that checked more than a dozen different aspects of the system end-to-end. From DNS resolution to TCP to SSL cert validity and expiration, remote SSH checks for disk space and and memory, firewall checks, docker open port verification, errors or warnings in logs, etc.

When it ran the script for the first time, it was pretty clear that the service was very broken, almost all the checks came back red.

The tight fixing/implementation loop

The agent quickly noticed that the service was dead (TCP connection failure), saw the error captured from the logs and immediately knew where to go next. After addressing the configuration issue, it re-deployed the software, noticed other checks fail (unexpected HTTP redirect), fixed them… and within two minutes the agent had a full solution working and verified because it could use a tight feedback loop with the script and quickly fix all the issues end to end.

And what’s even cooler is that now it has that script documented in cursor rules and I can see that it started using it every time it needs to check if the service is OK. So the script has turned into a permanent safety check the agent can rely on before and after any important change in the system. We didn’t just fix a bug; we permanently upgraded the agent’s ability to manage this system.

New important behaviour

This experience wasn’t just a cool one-off. It immediately created a new default behavior for how I work with the agent. The rule is now simple: before you try to fix something based on incomplete data, you must first write a script that defines what ‘correct’ looks like. Then, you run it to make sure it fails in the way you expect. Only then, with a clear target and a reliable check, do you start making changes and use that script to guide you to the goal.

Seeing the Future

At this stage, I cannot help but feel that this will become a pattern for our AI use and, in general, an important aspect of AI behavior in complex systems going forward.

The agent spends time building a narrow-use tool to gain a new capability (like reliably performing a 20-point check on a remote service) and then uses the tool to get better at performing its job. And, of course, nothing prevents them from potentially combining the tools into more and more complex and powerful capabilities that reduce the risk from guessing and codify reliable patterns of operating a complex system.

Exciting times!

If you have any suggestions for improvement or comments on the described approach, let me know! If you are interested in content like this, feel free to join my free Telegram channel where I share my thoughts on AI-related topics and relevant content I find interesting.

This is going to be a really short post, but for someone it could save an hour of life.

So, you’ve nothing to do and you’ve decided to play around with IPv6 or maybe you’re happened to be an administrator of a web service that needs to support IPv6 connectivity and you need to make your nginx server work nicely with this protocol.

First thing you need to do is to enable IPv6 in nginx by recompiling it with --with-ipv6 configure option and reinstalling it. If you use some pre-built package, check if your nginx already has this key enabled by running nginx -V.

Read the rest of this entry →

It’s been a few months since we‘ve started actively using ActiveMQ queue server in our project. For some time we had pretty weird problems with it and even started thinking about switching to something else or even writing our own queue server which would comply with our requirements. The most annoying problem was the following: some time after activemq restart everything worked really well and then activemq started lagging, queue started growing and all producer processes were stalling on push() operations. We rewrote our producers from Ruby to JRuby, then to Java and still – after some time everything was in a bad shape until we restarted the queue server.

Read the rest of this entry →

[lang_en]

Looks like MySQL Camp 2006 was really interesting and useful for its attendees and for entire MySQL community. As the result of this meeting of many MySQL-related professionals we’ve got lots of interesting publications and I want to refer one of them in this post. Very interesting list of 84 MySQL performance tips has beed created on first day of this year MySQL Camp at Google headquarters:

1. Index stuff.
2. Don’t Index Everything
3. Use benchmarking
4. Minimize traffic by fetching only what you need.
   • Paging/chunked data retrieval to limit
   • Don’t use SELECT *
   • Be wary of lots of small quick queries if a longer query can be more efficient
5. Use EXPLAIN to profile the query execution plan
6. Use Slow Query Log (always have it on!)
7. Don’t use DISTINCT when you have or could use GROUP BY
8. Use proper data partitions (For Cluster. Start thinking about Cluster *before* you need them)
9. Insert performance
   • Batch INSERT and REPLACE
   • Use LOAD DATA instead of INSERT
10. LIMIT m,n may not be as fast as it sounds
11. …

So, I think this list can be really useful for all developers/DBAs working with MySQL and want to say “Thanks” to its authors.

[/lang_en]

[lang_ru]

Похоже, что MySQL Camp 2006 был действительно очень интересным и полезным событием как для тех, кто его посетил, так и для всего сообщества MySQL. Как результат этой встречи большого количества профессионалов, связанных с MySQL, появилось множество интересных публикаций. Именно одну из таких публикаций я и хотел бы “прорекламировать” сегодня. Очень интересный список 84 Лучших советов по производительности MySQL был создан в первый день работы MySQL Camp в штабквартире Google:

1. Index stuff.
2. Don’t Index Everything
3. Use benchmarking
4. Minimize traffic by fetching only what you need.
   • Paging/chunked data retrieval to limit
   • Don’t use SELECT *
   • Be wary of lots of small quick queries if a longer query can be more efficient
5. Use EXPLAIN to profile the query execution plan
6. Use Slow Query Log (always have it on!)
7. Don’t use DISTINCT when you have or could use GROUP BY
8. Use proper data partitions (For Cluster. Start thinking about Cluster *before* you need them)
9. Insert performance
   • Batch INSERT and REPLACE
   • Use LOAD DATA instead of INSERT
10. LIMIT m,n may not be as fast as it sounds
11. …

Я думаю, что этот список советов может быть очень полезен всем разработчикам и администраторам, работающим с MySQL и потому хочу сказать “Большое Спасибо” его авторам.

[/lang_ru]