TorchText Vocab TypeError: Vocab.__init__() got an unexpected keyword argument 'min_freq'

Question

I am working on a CNN Sentiment analysis machine learning model which uses the IMDb dataset provided by the Torchtext library. On one of my lines of code

vocab = Vocab(counter, min_freq = 1, specials=('\<unk\>', '\<BOS\>', '\<EOS\>', '\<PAD\>'))

I am getting a TypeError for the min_freq argument even though I am certain that it is one of the accepted arguments for the function. I am also getting UserWarning Lambda function is not supported for pickle, please use regular python function or functools partial instead. Full code

from torchtext.datasets import IMDB
from collections import Counter
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import Vocab
tokenizer = get_tokenizer('basic_english')  
train_iter = IMDB(split='train')
test_iter = IMDB(split='test')
counter = Counter()
for (label, line) in train_iter:
    counter.update(tokenizer(line))
vocab = Vocab(counter, min_freq = 1, specials=('\<unk\>', '\<BOS\>', '\<EOS\>', '\<PAD\>'))

Source Links towardsdatascience github Legacy to new

I have tried removing the min_freq argument and use the functions default as follows

vocab = Vocab(counter, specials=('\<unk\>', '\<BOS\>', '\<EOS\>', '\<PAD\>'))

however I end up getting the same type error but for the specials argument rather than min_freq.

Any help will be much appreciated

Thank you.

Answer 1

正如https://github.com/pytorch/text/issues/1445提到的，您应该将“Vocab”更改为“vocab”。我认为他们错过了从旧笔记本到新笔记本的打字。

正确代码：

from torchtext.datasets import IMDB
from collections import Counter
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import vocab
tokenizer = get_tokenizer('basic_english')  
train_iter = IMDB(split='train')
test_iter = IMDB(split='test')
counter = Counter()
for (label, line) in train_iter:
    counter.update(tokenizer(line))
vocab = vocab(counter, min_freq = 1, specials=('\<unk\>', '\<BOS\>', '\<EOS\>', '\<PAD\>'))

我的环境：

• 蟒蛇3.9.12
• 火炬文本 0.12.0
• 火炬1.11.0