Kuzushiji-MNIST

• by user1
• 05 March, 2022

Classify handwritten characters in ancient Japanese manuscripts

LicenseCC BY-SA 4.0

Tagscomputer science, image data, classification, literature, multiclass classificationand 2 more

Background

Recorded historical documents give us a peek into the past. We are able to glimpse the world before our time; and see its culture, norms, and values to reflect on our own. Japan has very unique historical pathway. Historically, Japan and its culture was relatively isolated from the West, until the Meiji restoration in 1868 where Japanese leaders reformed its education system to modernize its culture. This caused drastic changes in the Japanese language, writing and printing systems. Due to the modernization of Japanese language in this era, cursive Kuzushiji (くずし字) script is no longer taught in the official school curriculum. Even though Kuzushiji had been used for over 1000 years, most Japanese natives today cannot read books written or published over 150 years ago.

The result is that there are hundreds of thousands of Kuzushiji texts that have been digitised but have never been transcribed, and can only currently be read by a few experts. We’ve built Kuzushiji-MNIST and sister datasets by taking handwritten characters from these texts and preprocessing them in a format similar to the MNIST dataset, to create easy to use benchmark datasets that are more modern and difficult to classify than the original MNIST dataset.

By releasing these datasets, we’re also hoping to bring together the fields of Japanese literature and ML 😊

📚 Read the paper to learn more about Kuzushiji, the datasets and our motivations for making them!

The datasets

1. Kuzushiji-MNIST is a drop-in replacement for the MNIST dataset (28×28 grayscale, 70,000 images), provided in the original MNIST format as well as a NumPy format. Since MNIST restricts us to 10 classes, we chose one character to represent each of the 10 rows of Hiragana when creating Kuzushiji-MNIST.

• kmnist-[train/test]-[images/labels].npz: These files contain the
  Kuzushiji-MNIST as compressed numpy arrays, and can be read with:
  arr = np.load(filename)['arr_0']. We recommend using these files to
  load the dataset.
• [train/t10k]-[images/labels]-idx[1/3]-ubyte.gz:
  These files contain Kuzushiji-MNIST with the same file format and
  filenames as the original MNIST for drop-in compatibility.
• kmnist-classmap.csv: A mapping from class IDs to unicode characters
  for Kuzushiji-MNIST

Some examples of Kuzushiji-MNIST with the first column being the modern Hiragana counterpart:

1. Kuzushiji-49, as the name suggests, has 49 classes (28×28 grayscale, 270,912 images), is a much larger, but imbalanced dataset containing 48 Hiragana characters and one Hiragana iteration mark.

• k49-[train/test]-[images/labels].npz: These files contain the Kuzushiji-49 as compressed numpy arrays
• k49-classmap.csv: A mapping from class IDs to unicode characters for Kuzushiji-49

1. Kuzushiji-Kanji is an imbalanced dataset of total 3832 Kanji characters (64×64 grayscale, 140,426 images), ranging from 1,766 examples to only a single example per class. We’re also planning to release a train-test split for this dataset soon.

• kkanji2/: This folder contains subfolders of images, with each subfolder corresponding to one Kanji character class.

License

The dataset is licensed under a permissive CC BY-SA 4.0 license, which requires attribution, and we would suggest to use the following attribution to the KMNIST dataset:

“KMNIST Dataset” (created by CODH), adapted from “Kuzushiji Dataset” (created by NIJL and others), doi:10.20676/00000341

Size: 584510 KB Price: Free Author: anokas and 1 collaborator Data source: kaggle.com