Abstract
This document outlines the design, methodology, and implementation of a large-scale English word collection, enhancement, and cleaning system. The system leverages prefix-based word collection, AI-driven semantic enhancement, and strict validation/cleaning rules to construct a massive English–Korean dictionary with metadata useful for downstream AI and NLP applications.
1. Introduction
Building a comprehensive wordlist is essential for natural language processing (NLP), educational tools, and AI-driven language models. Traditional dictionaries are limited by scope and update cycles. Our system introduces a dynamic, automated pipeline capable of:
- Collecting English words using adaptive prefix strategies.
- Enhancing words with Korean translations, part-of-speech tags, age-level classifications, and example sentences.
- Cleaning and validating entries with configurable rules.
2. System Architecture
2.1 Components
- Word Collector (
011.py): Gathers candidate English words using prefix expansion and an Ollama-backed model. - Word Enhancer: Enriches words with Korean meaning, age-group classification, part-of-speech, and examples.
- Data Cleaner (
data_cleaner.py): Applies validation, normalization, and deduplication to produce a high-quality dataset.
2.2 Workflow Sequence
- Collection Phase
- Generate prefixes (adaptive, 2-letter, or 3-letter).
- Query the model for words matching each prefix.
- Store results with timestamps and metadata.
- Enhancement Phase
- Process words in chunks.
- Request Korean translations, part-of-speech, and usage examples.
- Merge enhanced data with base word entries.
- Cleaning Phase
- Validate words with configurable rules.
- Remove duplicates, invalid forms, and undesired patterns.
- Generate distribution statistics (letter frequencies, age groups, etc.).
- Output
- Store datasets in JSON format with metadata.
- Provide CLI tools for analyzing patterns and cleaning data.
3. Validation Rules
Words are accepted or rejected based on the following constraints:
- Length: Must be within 2–50 characters (configurable).
- Character set: Alphabet-only; optional allowance for hyphen and apostrophe.
- Repetition: Rejects words with repeated character runs ≥ N (default 3).
- Blacklist: Regex-based exclusion of offensive or unwanted terms.
- Edge rules: No leading/trailing hyphens or apostrophes.
- Special rules: No double specials (e.g.,
--,'').
4. Metadata and Statistics
Each processed dataset includes metadata:
- Counts: Original, cleaned, removed, duplicates.
- Removal statistics: Breakdown by reason (e.g., too short, repeated chars).
- Letter distribution: Frequency of first letters A–Z.
- Age group distribution: For enhanced dictionaries.
- Timestamps: Collection, enhancement, and cleaning dates.
5. Command Line Interface (CLI)
Data Cleaner
# Analyze problematic patterns
python3 data_cleaner.py analyze words.json
# Clean dataset
python3 data_cleaner.py clean words.json -o cleaned.json
Dictionary Generator
# Stage 1: Collect words
python3 011.py --stage 1 --mode adaptive --batch 30
# Stage 2: Enhance words
python3 011.py --stage 2 --file massive_wordlist_xxx.txt --chunk 50
6. Applications
- NLP Research: Provides a scalable dataset for language model training.
- Education: Supplies graded vocabulary lists by age group.
- AI Assistants: Enables contextual translation and simplified explanations.
- Lexicography: Supports automated dictionary building and updates.
7. Conclusion
This system creates a bridge between raw, large-scale word collection and refined, structured dictionaries. By combining automated collection, semantic enrichment, and rigorous cleaning, it produces a dataset suitable for high-quality AI-driven applications.
Future work includes:
- Expansion with multi-language support.
- Integration with vector embeddings for semantic similarity.
- Continuous updates with crowdsourced validation.