tiktoken is a fast BPE tokeniser for use with OpenAI’s models.

>>> import pandas
>>> import tiktoken

Traceback (most recent call last):
File “”, line 1, in
ModuleNotFoundError: No module named ‘tiktoken’

tiktoken 사용하려면 먼저 설치 이후 사용할 수 있습니다.

(py382) > pip install tiktoken

>>> import pandas
>>> import tiktoken