Can metagpt show more comparison with AgileCoder

[CoderYiFei]

AgileCoder:
https://github.com/FSoft-AI4Code/AgileCoder

[stellaHSR]

Hi CoderYiFei,

Thank you for your suggestion regarding a comparison with AgileCoder. I believe Agile(https://en.wikipedia.org/wiki/Agile_software_development) is a great development methodology, particularly emphasizing feedback-based iteration and incremental improvement, which can significantly enhance software quality. It would be valuable to test and verify how well this approach works within an LLM-based multi-agent framework.

From both the paper and code, it seems that AgileCoder builds on ChatDev by incorporating roles like Testers and Scrum Masters, as well as prompts to manage the development flow. These additions are likely helpful for software development tasks which we haven't integrated right now. However, it seems they are still static and involve one-off code generation. I haven't yet seen a complete workflow for finer-grained development and continuous iterative improvement. If you could point out the relevant parts of their code in this regard, that would be very helpful.

I'm open to further discussing a comparison or benchmark focused on incremental development scenarios specific to software features, or adding related capabilities to MetaGPT. If you're interested, we could discuss here about how to define and set up an evaluation process for these comparisons.

Looking forward to your thoughts!

[CoderYiFei]

For comparing metrics,
I recommend https://github.com/ActivityWatch/activitywatch/tree/master
we can assign some general hard software project to a programmer,
And let him use different multi-agent tools, to find the difference of time cost , and the times human engaged.

I am hoping 1.0 's version released soon, so that , we can discuss evaluation process and more adding to Metagpt, more effectively

For AgileCoder's paper , I haven't yet seen a complete workflow for finer-grained development and continuous iterative improvement either , and will do more research this weekend...

[stellaHSR]

I think this is a great idea.
Regarding the observation of time consumption across different tools, are you referring to metrics similar to Jira, such as task hours, meeting times, and usage proportions of related development tools? Tracking these tools and their usage duration can indeed reflect differences in the software development process, but it might not be straightforward to compare (e.g., across different stages or frameworks), as these differences may not imply an absolute good or bad outcome. I believe the more important aspect here is to define how these differences relate to software quality and outcomes.