Text Encoding Explained: UTF-8, ASCII, and Beyond
Text encoding determines how characters are stored as bytes. Understanding UTF-8, ASCII, and other encodings prevents garbled text, mojibake, and data corruption in your applications and documents.
Key Takeaways
- Computers store everything as numbers.
- ASCII (American Standard Code for Information Interchange) maps 128 characters to numbers 0-127.
- UTF-8 is the dominant encoding on the web (used by 98%+ of websites).
- When text encoded in one system is decoded in another, you get mojibake — garbled characters like `é` instead of `e` or `???` instead of Chinese characters.
- Unless you have a specific reason to use another encoding, always choose UTF-8.
Word Counter
Count words, characters, sentences, and paragraphs.
What Is Text Encoding?
Computers store everything as numbers. Text encoding is the mapping between characters (letters, symbols, emoji) and the numbers that represent them. When sender and receiver use different encodings, text appears garbled.
ASCII: The Foundation
ASCII (American Standard Code for Information Interchange) maps 128 characters to numbers 0-127. It covers the English alphabet, digits, punctuation, and control characters. ASCII is a subset of virtually every modern encoding.
UTF-8: The Universal Standard
UTF-8 is the dominant encoding on the web (used by 98%+ of websites). It can represent every character in the Unicode standard — over 149,000 characters from all writing systems.
Key properties of UTF-8:
- ASCII-compatible: The first 128 characters use identical byte values.
- Variable-width: Characters use 1-4 bytes depending on their code point.
- Self-synchronizing: You can find character boundaries from any position.
Common Encoding Issues
Mojibake
When text encoded in one system is decoded in another, you get mojibake — garbled characters like é instead of e or ??? instead of Chinese characters.
BOM (Byte Order Mark)
Some editors add a BOM (EF BB BF) at the start of UTF-8 files. While harmless in most contexts, it can cause issues in scripts, CSV files, and configuration files.
Best Practice: Always Use UTF-8
Unless you have a specific reason to use another encoding, always choose UTF-8. It supports every language, is backward-compatible with ASCII, and is the expected default on modern systems.
Herramientas relacionadas
Formatos relacionados
Guías relacionadas
Regular Expressions: A Practical Guide for Text Processing
Regular expressions are powerful patterns for searching, matching, and transforming text. This guide covers the most useful regex patterns with real-world examples for common text processing tasks.
Markdown vs Rich Text vs Plain Text: When to Use Each
Choosing between Markdown, rich text, and plain text affects portability, readability, and editing workflow. This comparison helps you select the right text format for documentation, notes, and content creation.
How to Convert Case and Clean Up Messy Text
Messy text with inconsistent capitalization, extra whitespace, and mixed formatting is a common problem. This guide covers tools and techniques for cleaning, transforming, and standardizing text efficiently.
Troubleshooting Character Encoding Problems
Garbled text, question marks, and missing characters are symptoms of encoding mismatches. This guide helps you diagnose and fix the most common character encoding problems in web pages, files, and databases.
Best Practices for Counting Words, Characters, and Lines
Accurate text counting is important for meeting length requirements, estimating reading time, and analyzing content. This guide covers the nuances of counting words across different languages and contexts.
How to Find and Replace Text Using Regular Expressions
Regular expressions enable powerful pattern-based find and replace operations. Learn practical regex patterns for common text transformation tasks.
How to Remove Duplicate Lines From Text
Duplicate lines in data files, logs, and lists waste space and cause errors. Learn efficient methods to deduplicate text while preserving order.
How to Sort Text Lines Alphabetically and Numerically
Sorting text lines helps organize data, find duplicates, and prepare content for processing. Learn alphabetical, numerical, and custom sorting techniques.
How to Extract Emails, URLs, and Phone Numbers From Text
Extracting structured data from unstructured text saves hours of manual copying. Learn pattern-based extraction for common data types.
Plain Text vs Rich Text vs Markdown: Format Comparison
Each text format serves different purposes. Plain text is universal, rich text supports formatting, and Markdown balances readability with structure.
Troubleshooting Line Ending Issues (CRLF vs LF)
Different operating systems use different line endings, causing text files to display incorrectly or break scripts. Learn how to detect and fix the issue.
Text Diff and Comparison: Finding Changes Between Versions
Comparing text versions reveals exactly what changed. Learn how diff algorithms work and how to use them for code review, document comparison, and data validation.
Best Practices for Cleaning Messy Data in Text Files
Messy text data — extra spaces, inconsistent formatting, mixed encodings — creates problems for processing. Learn systematic approaches to text cleanup.
How to Clean and Normalize Text Data
Remove invisible characters, normalize whitespace, fix encoding issues, and standardize text for data processing.
How to Extract and Transform Structured Data from Text
Parse emails, addresses, phone numbers, dates, and URLs from unstructured text using regex and pattern matching.
Text Comparison and Diff Algorithms Explained
Understand how diff algorithms work, compare line-level vs word-level diffs, and choose the right approach for your use case.
How to Validate and Format Data Formats
Validate JSON, XML, YAML, and CSV data for structural correctness and format them for readability.
Text Transformation for Data Migration Projects
Data migrations often require bulk text transformations — changing delimiters, reformatting dates, normalizing encodings, and restructuring flat files.
Advanced Regex Patterns for Log File Analysis
Log files contain critical diagnostic information buried in semi-structured text. Master regex patterns to extract timestamps, error codes, IP addresses, and stack traces.
Markdown Editors and Renderers Compared
Compare Markdown editors, live preview tools, and rendering differences across platforms.
Solving Text Encoding Issues Across Platforms
Fix character corruption (mojibake) when text appears garbled due to encoding mismatches.
Unicode Normalization: NFC, NFD, NFKC, NFKD Explained
The same visible character can have multiple Unicode representations. Learn when and how to normalize text to prevent comparison failures and search issues.
Text Diff and Merge Tools Compared
Compare text comparison tools for code review, document editing, and content management.
Troubleshooting Text Truncation and Overflow
Text that displays correctly in English often breaks in other languages due to word length, character width, and directional differences. Learn how to identify and fix these issues.
CSV Data Cleaning: Common Pitfalls and Solutions
CSV files are deceptively simple. Embedded commas, inconsistent quoting, mixed encodings, and trailing whitespace cause silent data corruption during processing.
Regular Expressions: Pattern Matching Essentials
Master fundamental regex patterns for text search, validation, and transformation tasks.
Markdown for Technical Documentation
Markdown has become the standard for technical documentation. Learn the extended syntax, tooling, and best practices for writing clear, maintainable technical docs.