Lessons Learned about API Calls and Automated Testing
Last week, I made a post about needing to find someone to pay me for my work or I was going to have to sell plasma to cover the token usage I was going through. @David Vogel suggest I make a post about it in a bit more detail. When I posted it, I had no idea what was actually going on. Here is the story and the lesson I learned the hard way. I was working on Week 7's competition agent. I had tested it manually six ways to Sunday, made some fixes, tested again, made more fixes. Before I submitted it, I wanted to really put it through its paces with an automated testing script. The plan was simple. It would randomly assign one of six project documents as the default and then run the agent through a number of scenarios to catch any logic issues or edge cases, then switch to another random project and try again. So far so good, right? I decided that eusing code from another project was going to be the best option here. I used it three times already and it was working. All I would need to do is change the details for this project and viola. I copied that test file into my folder. I gave Claude a list of things I wanted to test for and how it should be done. I also told it to modify the existing script to work for this situation and let it rip. A few minutes later, I had a ready to run test script. It was a big and complex sucker. It would randomly pick a company project, set that as the default, then run 25 tests with multiple steps. It came back and said it would take about 30 min to run and cost about $10 in API calls. I thought to myself, I put $50 in a couple of weeks ago and hardly used any, so that isn't an issue, and I can go grab some food and when I come back, it should be done. While waiting in line, I get an email that I am out of credits for my API key. I just reauthorized for another $10 to get it going again. I get home and before I can check my computer, I get another email that I am out of credits. This is when I made the post in the forum, in jest, confusion and frustration.