Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to train a model before evaluating it #4

Open
dirkgr opened this issue Feb 25, 2022 · 0 comments
Open

Add a way to train a model before evaluating it #4

dirkgr opened this issue Feb 25, 2022 · 0 comments

Comments

@dirkgr
Copy link
Member

dirkgr commented Feb 25, 2022

Motivation: Full fine-tuning is a baseline, or rather an upper bound, in many zero-shot and few-shot experiments. @pdasigi has explicitly asked for this.

As part of this work, we'll add a new Tango step to Catwalk that trains a model on a given task/dataset, or on multiple tasks/datasets at the same time. It should call into Tango's training functions to do so. We'll also need to add a method or two to Catwalk's Model class to make this happen. Then we'll do a full evaluation on all reasonable tasks and all reasonable models, to establish good baselines across the board. This might make for a good blog post, too.

As a stretch goal, we should also try to train adaptation methods like prompt tuning, prefix tuning, or even IA3. There are some very nice implementations of some methods at https://github.com/r-three/t-few/tree/master/src.

@dirkgr dirkgr self-assigned this Feb 25, 2022
This was referenced Feb 25, 2022
@dirkgr dirkgr removed their assignment Apr 20, 2022
OyvindTafjord added a commit that referenced this issue May 19, 2023
Fix missing file for num_model_inputs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant