Intl Explorer is an interactive tool for experimenting and trying out the ECMAScript Internationalization API.

Transport stream web based HLS segmenter.

Linux HLS Server including uploader, segmenter, chunks dealer and media manager

Tokenization, sentence segmentation, POS tagging and dependency parsing for biomedical texts (BMC Bioinformatics 2019)

A high-performance wrapper around Intl.Segmenter for efficient text segmentation. This class resolves memory handling issues seen with large strings and "maximum call stack exceeded" exceptions that occur when strings exceed 40-50k characters. Enhances performance by 50-500x. Only ~70 loc (with comments) and no dependencies.

Chinese Tokenizer module for Python

Text segmenter and tokeniser for Danish, English and other languages. Reads an RTF or flat text file and outputs the text, one line per sentence & optionally tokenized.

Segmenter and Orthography Standardazier (SOS) for Classical Arabic (CA)

This is a tool that allows you to create an DASH manifest from any fmp4 stream file. There are online and CLI versions, also we provide file and stream versions

Data for layout analysis and HTR.

Dataset and models for catalogs' Layout analysis and HTR

A lightweight, intuitive wrapper around Intl.Segmenter for seamless segment-aware string operations in TypeScript and JavaScript

A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic features

Graph-Guided Token Merging (G2TM) is a lightweight one-shot module designed to eliminate redundant tokens early in the ViT architecture. It performs a single merging step after a shallow attention block, enabling all subsequent layers to operate on a compact token set. It leverages graph theory to identify groups of semantically redundant patches.

An extension of Intl.Segmenter that supports ANSI escape codes – with zero dependencies.

jieba中文分词库jieba.NET的.NET Standard版本

Text tokenizer for linguistic purposes, such as text matching. Supports more than 40 languages, including English, French, Russian, Japanese, Thai etc.

A native macOS SwiftUI app to split long text into clean, copy-ready segments — with smart logic and multilingual support.

Tools for segmenting natural language words into morphs / morphemes.

Sequence tagging(Word Segmenter), POS tagging and NER with Tensorflow