Would you all be interested in a course where I “interview” five different chatbots for the job as my assistant for this community? I’m thinking a video recording of me asking the same set of job interview questions to 5 different chatbots and then rating their responses and discussing what we learned about how the chatbots responded differently to the same questions. Here are some thoughts about why I think this could be valuable: Just because an AI is really good at some things doesn’t mean it will be good at what you need or want it to do. Here’s three ideas on how you can conduct a vibe test job interview to compare various AI tools (ex. Ask ChatGPT, Gemini, and Claude these same questions and rate the responses). 1. Ask it about something in which you already have expertise. Ex. “What’s the best way to get from Maple Grove, MN to downtown St. Paul and arrive at 7:15am on a weekday?” (That’s my daily commute, Claude said drive, chatGPT recommended some convoluted public transit plan, clearly never been to Minnesota where public transit for the suburbs is not great especially in the winter ❄️❄️❄️) 2. Ask an entertaining question to get a sense of the general personality. Ex. “Write a crossover episode of HBO’s Silicon Valley and the American Version of the office.” 3. Ask it to rate a business idea you know is a bad idea. This one is especially important when bringing an AI into your organization to serve as an advisor for your team. Ex. AI researcher Ethan Mollick asks AIs to provide feedback on his “Guaco Drone”business idea, a delivery service that only delivers guacamole. What questions would you want included in the interview?