Test Whether an AI Skill Actually Improves Results
What you will learn
In this playbook, you will learn how to test whether an AI Skill genuinely improves an AI model’s performance.
You will compare the same task in two conditions:
Test A: The AI completes the task without the Skill.
Test B: The AI completes the task with the Skill.
You will then evaluate both results using clear criteria rather than relying on which answer simply “looks better”.
Who this playbook is for
This playbook is suitable for beginners who:
Use ChatGPT, Claude, Gemini or another AI assistant.
Create reusable prompts, Custom GPT instructions, Gems or Agent Skills.
Want more consistent AI results.
Need a simple way to test whether their instructions are working.
Want to improve an AI workflow without guessing.
You do not need coding experience.
Estimated time: Approximately 35 to 50 minutes.
Level: Beginner
Tools you will need:
Choose one AI assistant:
ChatGPT
Claude
Gemini
Microsoft Copilot
Another text-based AI assistant
You will also need somewhere to record your results:
A Word or Google document
A spreadsheet
A notes application
The evaluation table included in this playbook
For the beginner activity, you do not need to install any software.
An optional advanced activity later in the playbook introduces the open-source agent-skills-eval project.
1. What is an AI Skill?
An AI Skill is a reusable set of instructions, examples, rules and resources that helps an AI perform a specific task.
A Skill might teach an AI how to:
Review a CV.
Analyse a spreadsheet.
Write a lesson plan.
Assess learner work.
Research a topic.
Create a presentation.
Review another AI agent’s code.
Produce a report using a required format.
A Skill may be stored in different ways.
For example:
Custom instructions
A Custom GPT
A Gemini Gem
A saved master prompt
A SKILL.md file
A reusable workflow inside an AI application
The important idea is that the Skill should provide more than a one-time prompt.
It should give the AI a consistent method to follow.
2. Why Skill evaluation matters
It is easy to assume that a longer prompt or more detailed Skill automatically produces a better result.
That is not always true.
A Skill may improve:
Accuracy
Structure
Consistency
Completeness
Safety
Usefulness
However, it may also:
Add unnecessary detail.
Make the output too long.
Increase token usage.
Introduce incorrect assumptions.
Force the AI to use the wrong process.
Work well for one example but fail on another.
Cause the AI to follow a poor procedure more consistently.
This is why Skills should be tested rather than trusted automatically.
The open-source project agent-skills-eval follows this principle by running the same task both with and without a Skill, then comparing the outputs. It supports baseline tests, judge-model grading, tool-call checks and structured reports.
Original repository:
3. The evaluation method
You will use a simple four-stage process:
Stage 1: Choose a real task
Select one task you already use AI to complete.
Stage 2: Create a baseline
Ask the AI to complete the task without your Skill.
Stage 3: Apply the Skill
Run the same task again with the Skill included.
Stage 4: Compare the results
Evaluate both outputs using the same scoring criteria.
The key rule is:
Change only one important variable: whether the Skill is used.
Try to keep the following the same:
The AI model
The original task
The input information
The expected audience
The requested output
The chat environment, where possible
4. Choose your test task
Pick a task that has a clear purpose and a result you can evaluate.
Good beginner tasks
Turn rough notes into a professional email.
Summarise a short article.
Create a weekly study plan.
Review a CV against a job description.
Convert meeting notes into actions.
Explain a difficult idea to a beginner.
Create a social media post from source material.
Check a report for missing information.
Turn a task list into a priority plan.
Produce a short lesson activity.
Avoid tasks that are too vague
For example:
Tell me something interesting about AI.
This is difficult to evaluate because there is no clear definition of success.
A stronger task would be:
Explain the difference between an AI agent and a chatbot to a beginner in no more than 200 words. Include one example of each and avoid technical jargon.
5. Define success before testing
Before asking the AI to complete the task, decide what a good result should contain.
This prevents you from changing your expectations after seeing the answers.
Example task
Convert messy meeting notes into a clear action tracker.
Success criteria
A good result should:
Identify every confirmed action.
Separate decisions from suggestions.
Include an owner only when one is stated.
Include a deadline only when one is stated.
Mark missing information clearly.
Avoid inventing details.
Use a readable table.
These criteria will become your evaluation checklist.
6. Create your baseline test
The baseline is the AI’s result without the Skill.
Open a new chat and use a clear but ordinary prompt.
Baseline prompt template
Complete the following task.
Task:
[DESCRIBE THE TASK]
Input:
[PASTE THE MATERIAL]
Audience:
[DESCRIBE THE AUDIENCE]
Required output:
[DESCRIBE THE OUTPUT]
Do not use any previously saved workflow or specialised instructions.
Example baseline prompt
Convert the meeting notes below into an action tracker.
Use these columns:
Action
Owner
Deadline
Status
Meeting notes:
The website launch is planned for next month. Ahmed will check the images.
The pricing page needs confirmation before Friday.
The contact form may still be sending messages to the old email address.
Sarah suggested adding testimonials later.
The team will review progress next Tuesday.
Save the baseline result as:
Output A: Without Skill
Do not correct it yet.
7. Create a simple AI Skill
Now create a short Skill that teaches the AI how to complete the task properly.
Your Skill should contain:
The role
The objective
The method
Important rules
The output structure
Quality checks
AI Skill template
Skill name:
[NAME OF THE SKILL]
Purpose:
This Skill helps the AI to [DESCRIBE THE RESULT].
When this Skill is used, follow this process:
1. [FIRST STEP]
2. [SECOND STEP]
3. [THIRD STEP]
4. [FOURTH STEP]
Rules:
- [IMPORTANT RULE]
- [IMPORTANT RULE]
- [IMPORTANT RULE]
- Do not invent information.
- Mark missing information clearly.
- Ask for clarification when a decision cannot be made safely.
Required output:
[DESCRIBE THE FORMAT]
Before finalising the answer, check:
- Is every claim supported by the input?
- Has any information been invented?
- Does the output follow the requested structure?
- Is the response suitable for the intended audience?
8. Worked example Skill
Skill name
Meeting Notes to Action Tracker
Skill instructions
Purpose:
This Skill converts unstructured meeting notes into a verified action tracker.
Process:
1. Read all notes before extracting actions.
2. Separate confirmed decisions, suggested ideas, actions and unanswered questions.
3. Identify an owner only when the notes explicitly name one.
4. Identify a deadline only when the notes explicitly state one.
5. Mark missing owners or deadlines as “Not stated”.
6. Create a final action table.
Rules:
- Do not invent an owner.
- Do not invent a deadline.
- Do not present a suggestion as a confirmed decision.
- Preserve uncertainty.
- Highlight any risk or missing information.
- Use British English.
Required output:
Section 1: Confirmed decisions
Section 2: Action tracker with these columns:
Action
Owner
Deadline
Status
Evidence from notes
Section 3: Questions requiring clarification
Final quality check:
- Every owner must be supported by the notes.
- Every deadline must be supported by the notes.
- Every action must include evidence.
- Suggestions must not be presented as decisions.
9. Run the Skill test
Open another new chat with the same AI model where possible.
Paste the Skill first.
Then paste the exact same task and source material you used in the baseline.
Skill test prompt
Use the Skill below to complete the task.
[PASTE THE SKILL]
Task:
[PASTE THE EXACT SAME TASK USED IN THE BASELINE]
Input:
[PASTE THE SAME INPUT]
Do not change the original task. Follow the Skill exactly.
Save the result as:
Output B: With Skill
10. Compare the two results
Use the evaluation table below.
Score each category from 1 to 5.
Criterion Output A: Without Skill Output B: With Skill
Accuracy
Completeness
Correct structure
Clarity
Follows instructions
Avoids invented details
Practical usefulness
Appropriate length
Total out of 40
Scoring guide
1 — Poor
The output seriously fails the requirement.
2 — Weak
The output partially completes the task but has important problems.
3 — Acceptable
The output is usable but needs correction.
4 — Strong
The output meets most requirements with only minor issues.
5 — Excellent
The output fully meets the requirement and needs little or no correction.
11. Record evidence, not impressions
Do not write only:
Output B was better.
Write specific observations.
Strong evidence examples
Output A invented a deadline for two tasks.
Output B correctly marked both deadlines as “Not stated”.
Output A mixed suggestions with confirmed decisions.
Output B separated them into different sections.
Output B followed the required table structure.
Output B was more accurate but 40% longer.
Output A was shorter but missed two confirmed actions.
The Skill improved consistency but added unnecessary explanation.
This helps you understand exactly what the Skill changed.
12. Ask an AI judge to compare the outputs
You can ask another AI chat to act as a second reviewer.
Do not rely on this score alone. Use it as additional evidence.
AI judge prompt
You are evaluating two AI-generated outputs for the same task.
Use only the criteria provided.
Task:
[PASTE THE ORIGINAL TASK]
Evaluation criteria:
1. Accuracy
2. Completeness
3. Correct structure
4. Clarity
5. Instruction-following
6. Avoidance of invented information
7. Practical usefulness
8. Appropriate length
Score each output from 1 to 5 for every criterion.
For each score:
- Give a brief reason.
- Cite specific evidence from the output.
- Do not favour the longer answer automatically.
- Do not assume that confidence means correctness.
- Identify any unsupported claims.
Output A, without Skill:
[PASTE OUTPUT A]
Output B, with Skill:
[PASTE OUTPUT B]
Finish with:
- Total score for Output A
- Total score for Output B
- The three most important differences
- Whether the Skill produced a meaningful improvement
- One recommendation for improving the Skill
13. Compare human and AI evaluation
Now compare your score with the AI judge’s score.
Ask:
Did the judge notice the same problems?
Did it reward length rather than quality?
Did it overlook invented information?
Did it score its own preferred writing style more highly?
Did it explain its judgement with evidence?
Do you agree with its conclusion?
A judge model can help, but it can also be wrong.
For important workflows, combine:
Human judgement
AI evaluation
Deterministic checks
Real-world testing
14. Improve the Skill
Do not rewrite the whole Skill immediately.
Identify the weakest area and make one focused improvement.
Example findings
Problem
The Skill improved structure but created very long outputs.
Improvement
Add:
Keep the final response under 300 words unless the user requests more detail.
Problem
The Skill still confused suggestions with decisions.
Improvement
Add:
A suggestion must remain labelled as a suggestion unless the notes include explicit approval.
Problem
The Skill omitted evidence from the source.
Improvement
Add:
Include the exact phrase or note that supports every action.
15. Run a second iteration
Repeat the test with the improved Skill.
Keep the original task the same.
Label the results:
Output A: No Skill
Output B: Skill version 1
Output C: Skill version 2
Compare the three scores.
Iteration table
Version Score Main strength Main weakness
No Skill
Skill version 1
Skill version 2
Your goal is not to create the longest Skill.
Your goal is to create the smallest reliable instruction set that produces a meaningful improvement.
16. Test the Skill on a second example
A Skill that succeeds on one example may still fail elsewhere.
Use a second input with the same type of task.
For example, when testing a meeting-notes Skill:
Test it first on project notes.
Test it again on training notes.
Test it again on a client update.
The Skill should work across several examples without becoming too rigid.
Second-example test prompt
Use the same Skill on this new example.
Do not change the Skill.
After completing the task, identify:
1. Which parts of the Skill worked well.
2. Which instructions were unclear.
3. Whether the Skill needs modification.
4. Whether the result remained accurate and useful.
17. Look for unintended side effects
A Skill can improve one area while making another worse.
Check whether the Skill:
Increased the response length too much.
Made the language sound unnatural.
Added unnecessary sections.
Reduced creativity.
Repeated instructions in the final answer.
Increased the number of assumptions.
Made the process slower.
Used more tokens without producing better results.
Forced one structure onto every situation.
This is why evaluation should include both quality and efficiency.
18. Optional technical extension
Learners who are comfortable with command-line tools can explore agent-skills-eval.
The project allows users to define a Skill, create evaluation cases, run the task with and without the Skill, and generate a side-by-side report.
A minimal Skill folder can include:
my-skill/
├── SKILL.md
├── references/
├── scripts/
└── evals/
└── evals.json
The tool can then compare:
with_skill
against:
without_skill
It supports structured assertions, model-based grading, tool-call checks and static HTML reports.
This optional extension is useful for teams that want repeatable testing or CI integration.
Common mistakes
Mistake 1: Changing the prompt between tests
If the original task changes, you cannot tell whether the Skill caused the difference.
Mistake 2: Using different models
Different models can produce very different results. Use the same model for both tests where possible.
Mistake 3: Judging by style alone
A polished answer may still be inaccurate.
Mistake 4: Rewarding longer answers
More detail is not always more useful.
Mistake 5: Using only one test case
One successful example does not prove that the Skill is reliable.
Mistake 6: Asking the same AI to judge itself without evidence
An AI judge should explain every score and cite examples.
Mistake 7: Adding too many instructions
Conflicting rules can reduce performance.
Mistake 8: Ignoring efficiency
A Skill that improves the result slightly but doubles the time or token cost may not be worthwhile.
Your practical project
Project title
Evaluate and Improve One AI Skill
Your task
Choose one real task from your work, learning or personal life.
Examples:
Prioritising emails
Reviewing a CV
Creating lesson activities
Converting notes into actions
Writing LinkedIn posts
Analysing feedback
Planning a project
Checking a report
Create a simple Skill for that task.
Then:
Define what a successful result should contain.
Run the task without the Skill.
Run the same task with the Skill.
Score both outputs.
Record specific differences.
Improve one part of the Skill.
Run the test again.
Decide whether the Skill created a meaningful improvement.
Required learner deliverables
Save the following:
The original task
The success criteria
The Skill version 1
Output A, without Skill
Output B, with Skill
The evaluation table
The improved Skill version 2
Output C, using Skill version 2
A short conclusion
Conclusion template
The Skill was designed to:
[DESCRIBE THE PURPOSE]
Without the Skill, the AI:
[DESCRIBE THE BASELINE]
With Skill version 1, the AI:
[DESCRIBE THE CHANGE]
The most important improvement was:
[DESCRIBE THE IMPROVEMENT]
The main remaining weakness was:
[DESCRIBE THE WEAKNESS]
I changed the Skill by:
[DESCRIBE THE CHANGE]
After the second test:
[DESCRIBE THE RESULT]
My final decision is:
The Skill produced a meaningful improvement / a small improvement / no clear improvement.
My evidence is:
[PROVIDE THE EVIDENCE]
Completion checklist
I selected one real AI task.
I defined success before running the test.
I created a baseline without the Skill.
I ran the same task with the Skill.
I kept the task and model consistent.
I scored both outputs using the same criteria.
I recorded evidence rather than impressions.
I identified one weakness in the Skill.
I created an improved version.
I tested the improved Skill.
I decided whether the Skill produced a meaningful improvement.
I removed confidential information before sharing.
Community sharing task
Share the following in the Skool community:
1. The task you tested
Describe the task in one sentence.
2. One improvement produced by the Skill
For example:
The Skill stopped the AI from inventing deadlines.
3. One weakness you discovered
For example:
The Skill made the output too long.
4. One change you made
For example:
I added a maximum length and required evidence for each action.
5. Your final verdict
Choose one:
Meaningful improvement
Small improvement
No clear improvement
The Skill made the result worse
Do not share confidential documents, personal information, client data or private workplace material.
Skool poll options
Because the poll accepts selectable statements rather than a separate question, use these options:
My AI Skill produced a meaningful improvement
My AI Skill produced only a small improvement
My AI Skill improved one area but weakened another
My AI Skill made no clear difference
My AI Skill made the result worse
I have not tested an AI Skill yet
Today’s practical task
Choose one repetitive task you already complete with AI.
Create a short Skill that teaches the AI how to perform it.
Run the same task once without the Skill and once with it.
Score both results out of 40 using the evaluation table, then improve one instruction and run the test again.
Share your final verdict and one lesson from the experiment in the community.
My AI Skill produced a meaningful improvement
My AI Skill produced a small improvement
My AI Skill improved one area but weakened another
My AI Skill made no clear difference
My AI Skill made the result worse
0 votes
0
0 comments
Alaaeldin Mostafa
1
Test Whether an AI Skill Actually Improves Results
powered by
CareerReady AI | AI Skills
skool.com/ai-career-accelerator-2265
Practical AI and data capability for professionals and organisations: career projects, custom workforce training and business demo projects.
Build your own community
Bring people together around your passion and get paid.
Powered by