README

Gemma4 vs Qwen Code Generation Comparison

This repository contains a comparison of code generation capabilities between Gemma4 and Qwen models. The focus is on evaluating their ability to generate correct and functional code in C++ and Pascal for various algorithms and logic problems.

Repository Structure

Directories

• cpp_algorithms_actual/: Contains C++ algorithm implementations generated by Gemma4 and Qwen, along with comparison reports and evaluation results.
  • gemma4/: C++ algorithms generated by Gemma4.
  • qwen/: C++ algorithms generated by Qwen.
  • comparison.md: Detailed comparison of the generated C++ algorithms.
  • final_scores.txt: Final scores for the C++ algorithm evaluations.
  • final_verdict.txt: Summary verdict on the C++ algorithm comparison.
• pascal_algorithms_actual/: Contains Pascal algorithm implementations generated by Gemma4 and Qwen, along with comparison reports and evaluation results.
  • gemma4/: Pascal algorithms generated by Gemma4.
  • qwen/: Pascal algorithms generated by Qwen.
  • comparison.md: Detailed comparison of the generated Pascal algorithms.
  • final_scores.txt: Final scores for the Pascal algorithm evaluations.
  • final_verdict.txt: Summary verdict on the Pascal algorithm comparison.
• logic_problems/: Contains logic problems and their solutions generated by Gemma4 and Qwen.
  • gemma4_solutions.txt: Solutions generated by Gemma4.
  • qwen3_solutions.txt: Solutions generated by Qwen.
  • evaluation_report.txt: Evaluation report for the logic problems.
  • final_scores.txt: Final scores for the logic problem evaluations.
  • final_verdict.txt: Summary verdict on the logic problem comparison.

ChatDSL Scripts

• cpp_algorithms.chatdsl: ChatDSL script used to generate and evaluate C++ algorithms.
• pascal_algorithms.chatdsl: ChatDSL script used to generate and evaluate Pascal algorithms.
• logic_problems.chatdsl: ChatDSL script used to generate and evaluate logic problems.

ChatDSL Scripts

The ChatDSL scripts in this repository are designed to automate the generation and evaluation of code and logic problem solutions using Gemma4 and Qwen models. These scripts facilitate:

1. Code Generation: Automatically generate code for specified algorithms in C++ and Pascal.
2. Evaluation: Compare the generated code against expected outputs and criteria.
3. Reporting: Generate detailed comparison reports and final verdicts.

Relation to ChatyBot

While this repository focuses on evaluating Gemma4 and Qwen models, the ChatDSL scripts are inspired by the automation and evaluation capabilities found in projects like ChatyBot. ChatyBot aims to provide a framework for testing and comparing AI models, and these scripts serve a similar purpose by automating the evaluation of code generation tasks.

For more information on ChatyBot, you can refer to the ChatyBot documentation (if available).

Usage

To use the ChatDSL scripts:

1. Generate Code: Run the ChatDSL scripts to generate code for the specified algorithms.
2. Evaluate: Use the evaluation scripts to compare the generated code against expected outputs.
3. Review Reports: Check the comparison reports and final verdicts to understand the performance of Gemma4 and Qwen.

Contributing

Contributions are welcome! If you have suggestions for improving the evaluation criteria, adding more algorithms, or enhancing the ChatDSL scripts, please open an issue or submit a pull request.

License

This repository is licensed under the MIT License. See the LICENSE file for more details.