Notes from using AI while developing a new feature for TreeTags
These are some notes from using Claude Code with Opus 4.8 when working on adding many-to-many mapping between file extensions and language parsers for Treetags. It appears to me to be a complex enough change to begin forming an opinion on AI’s performance for programming1. These two pull requests are the result of the development efforts treetags#43 and treetags#44.
What went well
- The required functionality was added to treetags
- The agent was able to come up with representative tests during many stages of development for verifying new changes. This helped find and fix a few bugs in the first attempts of the implementation.
- When asked whether the newly added
glob_matchfunction had common logic with the pre-existingshell_to_regexfunction that could be reused. It made a sensible case against it. What was surprising is that while evaluating this, it identified an existing bug inshell_to_regexall on its own. - The development time was definitely shorter with the AI agent than with what it would have been working on my own.
What did not go so well
- It missed adding the many-to-many mapping between file extensions and parsers provided by WASM plugins altogether.
- It messed up the matching of query-based parsers to source files in its initial attempt by somehow mixing up between the language name assigned to a parser and file extensions associated with that parser.
- The agent often skipped generating integration tests and instead preferred to
rely on verifying functionality by invoking
cargo runand observing the output. - It can generate an overwhelming amount of code in a short time. I often found myself either with too much code to review from a few prompts or a pace of iteration for small steps that was too much to keep up with, and I had to take a break now and then. This is despite using another agent to review the first one’s output.
Conclusion
Overall, this proved to be a pleasant and worthwhile experience. The AI agent was definitely far from perfect, but the parts that it manages to get right are quite astonishing. Astonishing, but not perfect.