tiktoken is a fast BPE tokeniser for use with OpenAI’s models.
>>> import pandas
>>> import tiktoken
Traceback (most recent call last):
File “”, line 1, in
ModuleNotFoundError: No module named ‘tiktoken’
tiktoken 사용하려면 먼저 설치 이후 사용할 수 있습니다.
(py382) > pip install tiktoken
>>> import pandas
>>> import tiktoken