gabyhelp feedback

[rsc]

I am experimenting with running a new bot @gabyhelp (which I am writing) in the Go issue tracker to try to help automate various mundane things that a machine can do reasonably well. Some aspects of the work may involve AI/LLMs; some will not.

Please thumbs up or thumbs down individual comments from the bot depending on whether they are correct/helpful or not. If the bot is getting in the way, we obviously want to know and fix it.

When the bot posts a comment, it will link back to this discussion page, which is a good place to leave non-boolean feedback. Thanks!

This discussion is also a good place to post ideas for things you wish a bot would do for you in the issue tracker.

To snoop around in all the issues where the bot has commented, you can use this search.

[rsc]

Right now, @gabyhelp is editing recently modified/posted comments and issues. Anywhere the plain text "CL nnn" appears, it changes the text to be a link to go.dev/cl/nnn. I've meant to do this for many years now, so that's the first chore.

[qiulaidongfeng]

See https://go.dev/doc/contribute#commit_messages , The rest of the description elaborates and should provide context for the change and explain what it does. Write in complete sentences with correct punctuation, just like for your comments in Go. Don't use HTML, Markdown, or any other markup language.
Does this mean that the Contribution Guide need to be revised？

[thepudds]

Hi @qiulaidongfeng, I think that section of the Contribution Guide is talking about avoiding HTML and Markdown in commit messages, vs. Russ above is talking about automatically editing GitHub issues.

[rsc]

@qiulaidongfeng The bot accidentally edited your PR description, because GitHub pretends that every PR is an Issue. The bot is only meant to be editing Issues, and I changed it to explicitly ignore PRs. Apologies!

[qiulaidongfeng]

For example :
@gabyhelp rewrite

CL 512155

to

[CL 512155](https://go.dev/cl/512155)

in #67902.

[CL 512155](https://go.dev/cl/512155)

isn't Markdown?

[rsc]

Yes, that's the PR that it accidentally edited. Again, my apologies.

[rsc]

@gabyhelp has also started posting to new issues (plus the issues created in the last two days, to get started) with lists of similar issues, if it finds any that are similar enough. If not, it will stay quiet.

Here is one success story: #67887 (comment). And here is another: #67897 (comment). This one is only a partial success because @thepudds got there first, but only because the bot gave him a head start. 😄

[dolmen]

This post by @gabyhelp is definitely useful: #67943 (comment).

[dmitshur]

In #68028 (comment), it's visible that gabyhelp currently doesn't filter out PRs in the list of potential "similar issues", which it probably should. GitHub API treats issues and PRs similarly, but PRs are a subset of CLs.

[rsc]

I think the only issue is the title (Similar Issues). In general if something is very related I think we should surface it. If an issue gets filed and gabyhelp posts hey there is a PR that seems to be about this issue, that's very helpful. The plan is to include all Gerrit CLs eventually too. (At that point we'd probably want to deduplicate GitHub PRs against Gerrit CLs.)

[dolmen]

@gabyhelp 's success is more about GitHub search being a big fail.

As an example I have just filled #68102 because a search of GOBIN didn't show anything relevant. See https://github.com/golang/go/issues?q=is%3Aissue+GOBIN+-author%3Adolmen

[magical]

I imagine this is on your list already, but in case not: another thing that would be nice would be for the bot to rewrite google-internal urls to their public counterparts.

[rsc]

Indeed. Thanks for the reminder. Done! And apologies again for having the internal-vs-external split at all. I'm just as annoyed by it as all of you. (It's literally just the same site with a different URL to force a different, stricter authentication path.)

[randall77]

It might be nice to have gabyhelp not generate emails when posting to issues. I see its updates, particularly related issues posts, as not needing a "ping". It's just additional information put there when you otherwise navigate to an issue.

[rsc]

@prattmic This should be fixed now.

[hyangah]

On the other hand I miss the github's formatting that shows whether the issues are closed or not clearly, and gets updated. Is there a way to mix both somehow?

[rsc]

Sure. Once I get better tests written I will add "(closed)" markers.

[cespare]

@rsc but that doesn't cover "and gets updated" part. I guess the "(closed)" markers will just be a snapshot of the state at whatever time the bot processed that comment?

IMO all these options are worse than going back to the initial state where we let GitHub dynamically format the issue links and sacrifice a bit of readability in the emails.

[AlexanderYastrebov]

Just found this thread and would like to +1 the idea to unfurl github links in comments.

[seankhliao]

I wonder if can be smart enough to recognize misformatted text, especially code blocks. We often see issues with no fences around code blocks or using single backticks (I assume this is from jira's formatting).

[deefdragon]

Related, I would love if it could add syntax highlighting to existing code-blocks that don't have it. It makes it so much easier for me to read when the highlighting is applied.

[rsc]

@seankhliao Can you point to specific issues with this problem? Thanks.

@deefdragon Different people have different but very strong opinions about syntax highlighting. I'm going to focus on clearer wins.

[seankhliao]

• unfenced code: Bug Report: JSON Deserialization Case Sensitivity Issue in Go #67689 , runtime: time.Sleep problem on Windows 10, but not Windows 7 or Linux #61042 , go v1.17.x breaks the (os.exec) Cmd.Start() functionality on Windows #49044
• unfenced stacktrace: net/http: Transport memory leak #43966 , Panic when trying to serve http with embedded assets #43761
• jira style(?) code blocks: crypto/rsa: can generate digital signature when hash algorithm (digest method) is set to MD5 in FIPS mode #45565

[seankhliao]

you'll need to look at the edit history for these, since I found them with a google search of "edited by seankhliao"

[ianlancetaylor]

Just a note that the similar issues list in #67861 was not helpful other than the reference to #33818. The issue #67861 was about changing sort.Sort (which we can't do), and the related issues are about other changes to sorting. We don't necessarily have to do anything here, the other issues weren't outrageously wrong, they just weren't helpful.

[cespare]

In #67948 (comment), the similar issues were not helpful.

The first listed issue, #56471, is very related -- as I wrote in the description, I created #67948 as a spin-off from #56471. Perhaps gabyhelp could be updated not to list issues which are already mentioned in the description?

The other four listed issues are superficially similar but not actually related in a useful way.

[jfrech]

I would feel alienated were I nudged to interact with an LLM by merely browsing Go development. I hope you will confine your bot to deterministic, comprehensible actions only.

[rsc]

I probably share your skepticism and concerns, but I don't fully understand the specific scenario you are concerned about. Can you say more?

[Merovius]

I think one thing I don't want to happen is a bot ask me clarifying questions, for example.

Another example of something I've seen LLMs used for which I consider insulting would be a bot re-phrasing comments.

I guess personally I'm mostly fine with AI use-cases I'd call "classification" but would feel patronized by any use-case that tries to emulate (or interlope in) human interaction.

[jfrech]

@rsc I don't envision a scenario. For me, it is a purely aesthetic matter: I have personally sworn not to interact with AI in my life. So when it becomes prevalent on this forum, I will have to leave. In this sense I feel getting alienated.

[mvdan]

Would this bot eventually replace the rather basic "Gopher Robot" at https://go-review.googlesource.com/q/owner:[email protected]? It currently seems to rely on a few heuristics to tell PR/CL authors about common mistakes, such as in https://go-review.googlesource.com/c/build/+/591455. It seems rather similar to the bot discussed here.

[rsc]

I think in the long term yes we'd want it to be able to do everything the current Gopherbot does. Right now we're focusing on what it can do that's new.

[ianlancetaylor]

In #67934 gabyhelp correctly reported that the issue was similar to #67933. That is because they are both backport requests for the same original issue. Perhaps gabyhelp can skip backport issues.

[findleyr]

Thanks @seankhliao, you beat me to it. It would be nice if gabyhelp could recognize issues that are 'process'.

[rsc]

I already disabled it on watchflakes issues, and I will disable it on backports issues too.

I am happy to disable it on gopls release issues too. Is that any issue with title prefix "x/tools/gopls: release version v"?

[findleyr]

@rsc yes all gopls release issues follow that format.

[rsc]

Thanks. Updated to skip both backports and gopls releases:

	if strings.Contains(issue.Body, `— [watchflakes](https://go.dev/wiki/Watchflakes)`) ||
		strings.HasSuffix(issue.Title, ` backport]`) ||
		strings.HasPrefix(issue.Title, `x/tools/gopls: release version v`) {
		continue
	}

Eventually we will hopefully be able to tell the bot to do this kind of thing. 😄

[seankhliao]

also package removal requests?
#68003 (comment)

[dolmen]

The Go project has a redirector for permalinks.
Examples: https://golang.org/s/generatedcode https://go.dev/s/generatedcode (I don't know which one should be preferred nowadays).

It would be helpful if @gabyhelp could replace links to targets of the redirector (and hopefully, any previous targets if the short link has an history) by the short permalink.

[thanm]

For my newly filed issue #67953, the related issues that @gabyhelp annotated it with (in #67953 (comment)) are not really all that helpful. Yes, they are all problems with coverage, but don't really relate to the very specific problem at hand.

[rsc]

The related issues for my issue #67961 (filed using gopls bug) are all empty instances of the gopls bug template. They are 90% similar to mine. Maybe I was too concise in my report!

Probably we need to look for empty templates anyway for auto-closing and then exclude them. (It's easy to tell, the empty templates are all 99% similar to each other.)

[rsc]

Another potential thing to explore is trying to help deduplicate the automated crashes in https://github.com/golang/vscode-go/issues.

[MikeMitchellWebDev]

As a relative newcomer to filing issues, I found the Gaby bot helpful for re-categorizing the issue I am filing (i.e editing my own title after the Gaby bot makes its post) (#67984)

[findleyr]

In #67995 (comment), Gaby provided a remarkably accurate summary of the timeline of our feelings about file watching in gopls: the initial issue where we discussed the philosophy of client-side file watching, and then the most relevent issues where we struggled to work around client-side issues, culminating in the current https://go.dev/issue/67995 which I filed to explore server-side file watching.

I experienced delight :).

[Jorropo]

I prefer how the bot used to link issues, for example: #67854 (comment)
Versus now: #68499 (comment)

The markdown different is:

- https://github.com/golang/go/issues/68293

versus:

- [encoding/xml: accepts various ill-formed XML](https://github.com/golang/go/issues/68293)

The first one is parsed by the github UI, it replaces the link text with the issue name (just like the bot manually does) and a helpful SVG based on the issue status (which the bot does not do) :

The github UI version is also updated automagically in case the issue status change or it is renamed.

[dmitshur]

It used to do that, but that renders less nicely in email notifications that GitHub mails. It was changed to the current format to make the email notifications more readable, but it has the downside you mention for viewing the comments on github.com itself.

I would also prefer the previous format since I don’t use GitHub’s email notifications, but each one has trade-offs. (Ideally GitHub should change the email notifications to render more similarly to the web version.)

[Jorropo]

Rahhhh tradeoffs :'(
Would you happen to know of an existing bug report to github for the better kind to notifications to render "properly" in emails ?

Also I wouldn't mind if the bot used to emojies to emulate the same behavior but I guess the code would be more involved as it would need to update existing comments once the status of an issue change.

[fvbommel]

Does GitHub send email notifications on comment edits?
If it does not, the bot could initially post its comments with the "nice for email" markup, then shortly after edit them with the bare links that look better on the site itself.

[zpavlinovic]

x/vuln: In #68152, @gabyhelp suggestions were not helpful. This is an issue with a brand new feature so there won't be many, if any, similar issues.

x/vuln: In #68100, a first few suggestions are ok. The next several suggestions are not relevant and belong to a different repo x/net. Would it make sense to restrict suggestions to the same repo, at least for specialized repos such as x/vuln?

[jfrech]

I think you should hard-code your bot to ignore "x/pkgsite" issues which are package removal requests.

[jfrech]

Is label:gabywins collecting successful work employing this tool? The label doesn't have a description.

[gaby]

@gabyhelp Hello 👋🏻

[rolandshoemaker]

It would be perhaps useful to filter out related issues/documentation if they are explicitly mentioned in the parent comment. In #69025 I reference #19413 as a previous/similar issue, which makes @gabyhelp referencing it as related/similar in the follow-up comment somewhat redundant.

[AlekSi]

@gabyhelp thinks that all CVEs are similar: #69138 (comment)

[andig]

@rsc is gaby specific to Go or could it be used for other languages/repos? I haven't been able to find much on Github Marketplace that seems to provide similar functionality.

[dmitshur]

@andig You may find https://github.com/golang/oscar#prototype and the rest of https://github.com/golang/oscar#readme relevant.

[FiloSottile]

It'd be nice if @gabyhelp skipped issues already mentioned in the body of the issue (and omitted commenting if those were all the results).

[cespare]

(Also mentioned here and here.)

[mateusz834]

Why does @gabyhelp link to https://go.dev/pkg/go/ast/?m=old#SortImports instead of https://pkg.go.dev/go/ast#SortImports ?

See #69694 (comment)

[iand]

Also @gabyhelp linked to #36140 which was closed in favour of #20818 which is still open. Probably 20818 should also be linked.

[nsrip-dd]

@gabyhelp left two identical comments on this issue: #69747 (comment) and #69747 (comment)

[jba]

@nsrip-dd That's the result of a change in the way we check for done-ness. The new way should have incorporated the old check, but didn't. We'll fix that.

[mvdan]

Can @gabyhelp learn to close empty issues? This should also include issues which copied the template but did not fill it out. For example: #69795

[jba]

Filed golang/oscar#37.

[Jorropo]

Some kind of issues are not helpful to give recommendations #70376 (comment) altho it's mostly harmless
could be very smart like by predicting click through rate and not posting if it would be too low or as simple as hardcoding access:

[prattmic]

While I agree that linking to other approver issues isn't helpful, it actually is helpful that it linked to your prior may-start-trybots issue. (Though just having the issue title isn't great since it isn't obvious that it is your issue).

[Jorropo]

Fair enough

Though just having the issue title isn't great since it isn't obvious that it is your issue

Yes got me

[millerresearch]

@gabyhelp not helpful on issue #70484 reporting on a bug which was introduced by a specific CL. Maybe if the title includes "after CL ..." or "since CL ..." it's not worth looking for other issues?