如何保存分类器textblob NaiveBayesClassifier的结果？

Question

我NaiveBayesclassifier根据我选择的给定主题使用TextBlob 进行文本分析.

数据量巨大(约3000个条目).

虽然我能够得到一个结果,但是如果没有再次调用该函数并等待数小时直到处理完成,我将无法将其保存以备将来使用.

我试着通过以下方法进行酸洗

ab = NaiveBayesClassifier(data)

import pickle

object = ab
file = open('f.obj','w') #tried to use 'a' in place of 'w' ie. append
pickle.dump(object,file)

我收到一个错误,如下:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\pickle.py", line 1370, in dump
    Pickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python27\lib\pickle.py", line 615, in _batch_appends
    save(x)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 662, in _batch_setitems
    save(k)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 501, in save_unicode
    self.memoize(obj)
  File "C:\Python27\lib\pickle.py", line 247, in memoize
    self.memo[id(obj)] = memo_len, obj
MemoryError

我也试过sPickle但它也导致了错误,例如:

#saving object with function sPickle.s_dump
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\sPickle.py", line 22, in s_dump
    for elt in iterable_to_pickle:
TypeError: 'NaiveBayesClassifier' object is not iterable

#saving object with function sPickle.s_dump_elt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\sPickle.py", line 28, in s_dump_elt
    pickled_elt_str = dumps(elt_to_pickle)
MemoryError: out of memory

任何人都可以告诉我我要做什么来保存对象？

或者无论如何,保存分类器的结果以备将来使用？

Answer 1

我自己解决了这个问题.

首先使用64位版本的Python(适用于2.6到3.4的所有版本)

64位版本解决了所有内存问题

使用cPickle

import cPickle as pickle

其次打开你的文件为

file = open('file_name.pickle','wb') #same as what Robert said in the above post

在文件上写入对象

pickle.dump(object,file)

你的对象将被转储到一个文件中.但你必须检查你的对象使用了什么内存.pickleing也占用了内存空间,因此至少有25%的内存可用于被腌制的对象

对我来说,我的笔记本电脑有一个8 GB的RAM,所以内存足以只有一个对象.

(我的分类器很重,有3000个字符串实例,每个字符串包含大约15-30个单词的句子.情绪/主题的数量为22个.)

因此,如果您的笔记本电脑死锁(或者,一般而言,停止工作),那么您可能需要关闭它并重新开始并尝试使用较少的号码.实例或更少的没有.情绪/主题.

在这里,cPickle是非常有用的bcz它比任何其他pickleing模块快得多,我wud建议使用tht.