✖


Jump to content

Кодировка символов

(Перенаправлено из модели кодировки Unicode )

Перфолента со словом «Arc.Ask3.Ru», закодированным в ASCII . Наличие и отсутствие отверстия соответствует 1 и 0 соответственно; например, «W» кодируется как «1010111».

Кодирование символов — это процесс присвоения чисел графическим символам , особенно письменным символам человеческого языка , позволяющий их хранить , передавать и преобразовывать с помощью цифровых компьютеров . [1] Числовые значения, составляющие кодировку символов, известны как « кодовые точки » и вместе составляют «кодовое пространство», « кодовую страницу » или « карту символов ».

Ранние коды символов, связанные с оптическим или электрическим телеграфом, могли представлять собой только подмножество символов, используемых в письменных языках , иногда ограничиваясь только прописными буквами , цифрами и некоторыми знаками препинания . Низкая стоимость цифрового представления данных в современных компьютерных системах позволяет использовать более сложные коды символов (например, Unicode ), которые представляют большинство символов, используемых во многих письменных языках. Кодирование символов с использованием международно признанных стандартов позволяет осуществлять обмен текстом в электронной форме по всему миру.

Наиболее используемой кодировкой символов в Интернете является UTF-8 , которая по состоянию на май 2024 года используется на 98,2% опрошенных веб-сайтов. [2] В прикладных программах и операционной системы задачах популярными вариантами являются UTF-8 и UTF-16 . [3] [4]

History

[edit]

The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance, using once-novel electrical means. The earliest codes were based upon manual and hand-written encoding and cyphering systems, such as Bacon's cipher, Braille, international maritime signal flags, and the 4-digit encoding of Chinese characters for a Chinese telegraph code (Hans Schjellerup, 1869). With the adoption of electrical and electro-mechanical techniques these earliest codes were adapted to the new capabilities and limitations of the early machines. The earliest well-known electrically transmitted character code, Morse code, introduced in the 1840s, used a system of four "symbols" (short signal, long signal, short space, long space) to generate codes of variable length. Though some commercial use of Morse code was via machinery, it was often used as a manual code, generated by hand on a telegraph key and decipherable by ear, and persists in amateur radio and aeronautical use. Most codes are of fixed per-character length or variable-length sequences of fixed-length codes (e.g. Unicode).[5]

Common examples of character encoding systems include Morse code, the Baudot code, the American Standard Code for Information Interchange (ASCII) and Unicode. Unicode, a well-defined and extensible encoding system, has supplanted most earlier character encodings, but the path of code development to the present is fairly well known.

The Baudot code, a five-bit encoding, was created by Émile Baudot in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. 2 (ITA2) in 1930. The name baudot has been erroneously applied to ITA2 and its many variants. ITA2 suffered from many shortcomings and was often improved by many equipment manufacturers, sometimes creating compatibility issues. In 1959 the U.S. military defined its Fieldata code, a six-or seven-bit code, introduced by the U.S. Army Signal Corps. While Fieldata addressed many of the then-modern issues (e.g. letter and digit codes arranged for machine collation), it fell short of its goals and was short-lived. In 1963 the first ASCII code was released (X3.4-1963) by the ASCII committee (which contained at least one member of the Fieldata committee, W. F. Leubbert), which addressed most of the shortcomings of Fieldata, using a simpler code. Many of the changes were subtle, such as collatable character sets within certain numeric ranges. ASCII63 was a success, widely adopted by industry, and with the follow-up issue of the 1967 ASCII code (which added lower-case letters and fixed some "control code" issues) ASCII67 was adopted fairly widely. ASCII67's American-centric nature was somewhat addressed in the European ECMA-6 standard.[6]

Hollerith 80-column punch card with EBCDIC character set

Herman Hollerith invented punch card data encoding in the late 19th century to analyze census data. Initially, each hole position represented a different data element, but later, numeric information was encoded by numbering the lower rows 0 to 9, with a punch in a column representing its row number. Later alphabetic data was encoded by allowing more than one punch per column. Electromechanical tabulating machines represented date internally by the timing of pulses relative to the motion of the cards through the machine. When IBM went to electronic processing, starting with the IBM 603 Electronic Multiplier, it used a variety of binary encoding schemes that were tied to the punch card code.

IBM used several Binary Coded Decimal (BCD) six-bit character encoding schemes, starting as early as 1953 in its 702[7] and 704 computers, and in its later 7000 Series and 1400 series, as well as in associated peripherals. Since the punched card code then in use only allowed digits, upper-case English letters and a few special characters, six bits were sufficient. These BCD encodings extended existing simple four-bit numeric encoding to include alphabetic and special characters, mapping them easily to punch-card encoding which was already in widespread use. IBM's codes were used primarily with IBM equipment; other computer vendors of the era had their own character codes, often six-bit, but usually had the ability to read tapes produced on IBM equipment. These BCD encodings were the precursors of IBM's Extended Binary-Coded Decimal Interchange Code (usually abbreviated as EBCDIC), an eight-bit encoding scheme developed in 1963 for the IBM System/360 that featured a larger character set, including lower case letters.

In trying to develop universally interchangeable character encodings, researchers in the 1980s faced the dilemma that, on the one hand, it seemed necessary to add more bits to accommodate additional characters, but on the other hand, for the users of the relatively small character set of the Latin alphabet (who still constituted the majority of computer users), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users). In 1985, the average personal computer user's hard disk drive could store only about 10 megabytes, and it cost approximately US$250 on the wholesale market (and much higher if purchased separately at retail),[8] so it was very important at the time to make every bit count.

The compromise solution that was eventually found and developed into Unicode[vague] was to break the assumption (dating back to telegraph codes) that each character should always directly correspond to a particular sequence of bits. Instead, characters would first be mapped to a universal intermediate representation in the form of abstract numbers called code points. Code points would then be represented in a variety of ways and with various default numbers of bits per character (code units) depending on context. To encode code points higher than the length of the code unit, such as above 256 for eight-bit units, the solution was to implement variable-length encodings where an escape sequence would signal that subsequent bits should be parsed as a higher code point.

Terminology

[edit]

Informally, the terms "character encoding", "character map", "character set" and "code page" are often used interchangeably.[9] Historically, the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units — usually with a single character per code unit. However, due to the emergence of more sophisticated character encodings, the distinction between these terms has become important.

  • A character is a minimal unit of text that has semantic value.[9][10]
  • A character set is a collection of elements used to represent text.[9][10] For example, the Latin alphabet and Greek alphabet are both character sets.
  • A coded character set is a character set mapped to set of unique numbers.[10] For historical reasons, this is also often referred to as a code page.[9]
  • A character repertoire is the set of characters that can be represented by a particular coded character set.[10][11] The repertoire may be closed, meaning that no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series); or it may be open, allowing additions (as is the case with Unicode and to a limited extent Windows code pages).[11]
  • A code point is a value or position of a character in a coded character set.[10]
  • A code space is the range of numerical values spanned by a coded character set.[10][12]
  • A code unit is the minimum bit combination that can represent a character in a character encoding (in computer science terms, it is the word size of the character encoding).[10][12] For example, common code units include 7-bit, 8-bit, 16-bit, and 32-bit. In some encodings, some characters are encoded using multiple code units; such an encoding is referred to as a variable-width encoding.

Code pages

[edit]
Main article: Code page

"Code page" is a historical name for a coded character set.

Originally, a code page referred to a specific page number in the IBM standard character set manual, which would define a particular character encoding.[13] Other vendors, including Microsoft, SAP, and Oracle Corporation, also published their own sets of code pages; the most well-known code page suites are "Windows" (based on Windows-1252) and "IBM"/"DOS" (based on code page 437).

Despite no longer referring to specific page numbers in a standard, many character encodings are still referred to by their code page number; likewise, the term "code page" is often still used to refer to character encodings in general.

The term "code page" is not used in Unix or Linux, where "charmap" is preferred, usually in the larger context of locales. IBM's Character Data Representation Architecture (CDRA) designates entities with coded character set identifiers (CCSIDs), each of which is variously called a "charset", "character set", "code page", or "CHARMAP".[12]

Code units

[edit]

The code unit size is equivalent to the bit measurement for the particular encoding:

  • A code unit in ASCII consists of 7 bits;
  • A code unit in UTF-8, EBCDIC and GB 18030 consists of 8 bits;
  • A code unit in UTF-16 consists of 16 bits;
  • A code unit in UTF-32 consists of 32 bits.

Code points

[edit]

A code point is represented by a sequence of code units. The mapping is defined by the encoding. Thus, the number of code units required to represent a code point depends on the encoding:

  • UTF-8: code points map to a sequence of one, two, three or four code units.
  • UTF-16: code units are twice as long as 8-bit code units. Therefore, any code point with a scalar value less than U+10000 is encoded with a single code unit. Code points with a value U+10000 or higher require two code units each. These pairs of code units have a unique term in UTF-16: "Unicode surrogate pairs".
  • UTF-32: the 32-bit code unit is large enough that every code point is represented as a single code unit.
  • GB 18030: multiple code units per code point are common, because of the small code units. Code points are mapped to one, two, or four code units.[14]

Characters

[edit]
Main article: Character (computing)

Exactly what constitutes a character varies between character encodings.

For example, for letters with diacritics, there are two distinct approaches that can be taken to encode them: they can be encoded either as a single unified character (known as a precomposed character), or as separate characters that combine into a single glyph. The former simplifies the text handling system, but the latter allows any letter/diacritic combination to be used in text. Ligatures pose similar problems.

Exactly how to handle glyph variants is a choice that must be made when constructing a particular character encoding. Some writing systems, such as Arabic and Hebrew, need to accommodate things like graphemes that are joined in different ways in different contexts, but represent the same semantic character.

Unicode encoding model

[edit]

Unicode and its parallel standard, the ISO/IEC 10646 Universal Character Set, together constitute a unified standard for character encoding. Rather than mapping characters directly to bytes, Unicode separately defines a coded character set that maps characters to unique natural numbers (code points), how those code points are mapped to a series of fixed-size natural numbers (code units), and finally how those units are encoded as a stream of octets (bytes). The purpose of this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. To describe this model precisely, Unicode uses its own set of terminology to describe its process:[12]

An abstract character repertoire (ACR) is the full set of abstract characters that a system supports. Unicode has an open repertoire, meaning that new characters will be added to the repertoire over time.

A coded character set (CCS) is a function that maps characters to code points (each code point represents one character). For example, in a given repertoire, the capital letter "A" in the Latin alphabet might be represented by the code point 65, the character "B" by 66, and so on. Multiple coded character sets may share the same character repertoire; for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different code points.

A character encoding form (CEF) is the mapping of code points to code units to facilitate storage in a system that represents numbers as bit sequences of fixed length (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units can only directly represent code points 0 to 65,535 in each unit, but larger code points (say, 65,536 to 1.4 million) could be represented by using multiple 16-bit units. This correspondence is defined by a CEF.

A character encoding scheme (CES) is the mapping of code units to a sequence of octets to facilitate storage on an octet-based file system or transmission over an octet-based network. Simple character encoding schemes include UTF-8, UTF-16BE, UTF-32BE, UTF-16LE, and UTF-32LE; compound character encoding schemes, such as UTF-16, UTF-32 and ISO/IEC 2022, switch between several simple schemes by using a byte order mark or escape sequences; compressing schemes try to minimize the number of bytes used per code unit (such as SCSU and BOCU).

Although UTF-32BE and UTF-32LE are simpler CESes, most systems working with Unicode use either UTF-8, which is backward compatible with fixed-length ASCII and maps Unicode code points to variable-length sequences of octets, or UTF-16BE,[citation needed] which is backward compatible with fixed-length UCS-2BE and maps Unicode code points to variable-length sequences of 16-bit words. See comparison of Unicode encodings for a detailed discussion.

Finally, there may be a higher-level protocol which supplies additional information to select the particular variant of a Unicode character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the XML attribute xml:lang.

The Unicode model uses the term "character map" for other systems which directly assign a sequence of characters to a sequence of bytes, covering all of the CCS, CEF and CES layers.[12]

Unicode code points

[edit]

In Unicode, a character can be referred to as 'U+' followed by its codepoint value in hexadecimal. The range of valid code points (the codespace) for the Unicode standard is U+0000 to U+10FFFF, inclusive, divided in 17 planes, identified by the numbers 0 to 16. Characters in the range U+0000 to U+FFFF are in plane 0, called the Basic Multilingual Plane (BMP). This plane contains most commonly-used characters. Characters in the range U+10000 to U+10FFFF in the other planes are called supplementary characters.

The following table shows examples of code point values:

CharacterUnicode code pointGlyph
Latin AU+0041Α
Latin sharp SU+00DFß
Han for EastU+6771東
AmpersandU+0026&
Inverted exclamation markU+00A1¡
Section signU+00A7§

Example

[edit]

Рассмотрим строку букв «ab̲c𐐀», то есть строку, содержащую объединяющий символ Юникода ( U+0332 ̲ COMBINING LOW LINE) as well a supplementary character (U+10400 𐐀 DESERET CAPITAL LETTER LONG I). This string has several Unicode representations which are logically equivalent, yet while each is suited to a diverse set of circumstances or range of requirements:

  • Четыре составленных персонажа :
    a, b̲, c, 𐐀
  • Пять графем :
    a, b, _, c, 𐐀
  • Пять кодовых точек Unicode :
    U+0061, U+0062, U+0332, U+0063, U+10400
  • Пять кодовых единиц UTF-32 (32-битные целочисленные значения):
    0x00000061, 0x00000062, 0x00000332, 0x00000063, 0x00010400
  • Шесть кодовых единиц UTF-16 (16-битные целые числа)
    0x0061, 0x0062, 0x0332, 0x0063, 0xD801, 0xDC00
  • Девять кодовых единиц UTF-8 (8-битные значения или байты ).
    0x61, 0x62, 0xCC, 0xB2, 0x63, 0xF0, 0x90, 0x90, 0x80

Обратите внимание, в частности, что 𐐀 представлен либо одним 32-битным значением (UTF-32), двумя 16-битными значениями (UTF-16) или четырьмя 8-битными значениями (UTF-8). Хотя каждая из этих форм использует одинаковое общее количество битов (32) для представления глифа, неясно, как связаны фактические числовые значения байтов.

Транскодирование

[ редактировать ]

В результате использования множества методов кодирования символов (и необходимости обратной совместимости с архивными данными) было разработано множество компьютерных программ для перевода данных между схемами кодирования символов - процесс, известный как транскодирование . Некоторые из них приведены ниже.

Кроссплатформенность :

  • Веб-браузеры . Большинство современных веб-браузеров имеют функцию автоматического определения кодировки символов . Например, в Firefox 3 см. подменю «Вид/Кодировка символов».
  • iconv — программа и стандартизированный API для преобразования кодировок
  • luit — программа, которая преобразует кодировку ввода и вывода в программы, работающие в интерактивном режиме.
  • Международные компоненты для Unicode — набор библиотек C и Java для преобразования кодировок. uconv можно использовать из ICU4C.

Окна :

  • Encoding.Convert – .NET API [15]
  • MultiByteToWideChar/WideCharToMultiByte — для преобразования из ANSI в Unicode и из Unicode в ANSI. [16] [17]

Общие кодировки символов

[ редактировать ]
Основная статья: Популярность кодировок текста

Наиболее используемой кодировкой символов в Интернете является UTF-8 , которая по состоянию на май 2024 года используется на 98,2% опрошенных веб-сайтов. [2] В прикладных программах и операционной системы задачах популярными вариантами являются UTF-8 и UTF-16 . [3] [4]

См. также

[ редактировать ]

Ссылки

[ редактировать ]
  1. ^ «Определение кодировки символов» . Словарь технических терминов . 24 сентября 2010 г.
  2. ^ Перейти обратно: а б «Обзор использования кодировок символов с разбивкой по рейтингу» . W3Techs . Проверено 29 апреля 2024 г.
  3. ^ Перейти обратно: а б «Чарсет» . Android-разработчики . Проверено 2 января 2021 г. Примечание для Android. Платформа Android по умолчанию всегда использует кодировку UTF-8.
  4. ^ Перейти обратно: а б Галлоуэй, Мэтт (9 октября 2012 г.). «Кодировка символов для разработчиков iOS. Или UTF-8, что теперь?» . www.galloway.me.uk . Проверено 2 января 2021 г. на самом деле вы обычно просто предполагаете UTF-8, поскольку это, безусловно, самая распространенная кодировка.
  5. ^ Том Хендерсон (17 апреля 2014 г.). «Таблицы кодов символов древних компьютеров – и почему они все еще актуальны» . Умный медведь . Проверено 29 апреля 2014 г.
  6. ^ Том Дженнингс (1 марта 2010 г.). «Аннотированная история некоторых кодов символов» . Проверено 1 ноября 2018 г.
  7. ^ «Предварительное информационное руководство по электронным машинам обработки данных IBM типа 702» (PDF) . 1954. с. 80. 22-6173-1. Архивировано (PDF) из оригинала 9 октября 2022 года.
  8. ^ Стрелхо, Кевин (15 апреля 1985 г.). «IBM переводит жесткие диски на новые стандарты» . Инфомир . Popular Computing Inc., стр. 29–33 . Проверено 10 ноября 2020 г.
  9. ^ Перейти обратно: а б с д Шон Стил (15 марта 2005 г.). «В чем разница между кодировкой, кодовой страницей, набором символов и Юникодом?» . Документы Майкрософт .
  10. ^ Перейти обратно: а б с д и ж г «Глоссарий терминов Юникода» . Консорциум Юникод.
  11. ^ Перейти обратно: а б «Глава 3: Соответствие». Стандарт Unicode версии 15.0 – Основная спецификация (PDF) . Консорциум Юникод. Сентябрь 2022 г. ISBN  978-1-936213-32-0 .
  12. ^ Перейти обратно: а б с д и Уистлер, Кен; Фрейтаг, Асмус (11 ноября 2022 г.). «UTR № 17: Модель кодировки символов Юникода» . Консорциум Юникод . Проверено 12 августа 2023 г.
  13. ^ «Информация о программаторе видеотерминала VT510» . Корпорация цифрового оборудования (DEC). 7.1. Наборы символов — Обзор. Архивировано из оригинала 26 января 2016 года . Проверено 15 февраля 2017 г. В дополнение к традиционным наборам символов DEC и ISO , которые соответствуют структуре и правилам ISO 2022 , VT510 поддерживает ряд кодовых страниц IBM PC ( номера страниц в руководстве по стандартному набору символов IBM) в режиме PCTerm для эмуляции терминала консольного ПК отраслевого стандарта.
  14. ^ «Терминология (Уроки Java)» . Оракул . Проверено 25 марта 2018 г.
  15. ^ «Метод Encoding.Convert» . Библиотека классов Microsoft .NET Framework .
  16. ^ «Функция MultiByteToWideChar (stringapiset.h)» . Документы Майкрософт . 13 октября 2021 г.
  17. ^ «Функция WideCharToMultiByte (stringapiset.h)» . Документы Майкрософт . 9 августа 2022 г.

Дальнейшее чтение

[ редактировать ]
[ редактировать ]
Викискладе есть медиафайлы, связанные с кодировками .
В Викиверситете есть учебные ресурсы по кодировке символов.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Character_encoding&oldid=1237576733#Unicode_encoding_model"
Arc.Ask3.Ru: конец переведенного документа.
Arc.Ask3.Ru
Номер скриншота №: 82d9535015be4bf368c5885cf63ef5ef__1722335040
URL1:https://arc.ask3.ru/arc/aa/82/ef/82d9535015be4bf368c5885cf63ef5ef.html
Заголовок, (Title) документа по адресу, URL1:
Character encoding - Wikipedia
Данный printscreen веб страницы (снимок веб страницы, скриншот веб страницы), визуально-программная копия документа расположенного по адресу URL1 и сохраненная в файл, имеет: квалифицированную, усовершенствованную (подтверждены: метки времени, валидность сертификата), открепленную ЭЦП (приложена к данному файлу), что может быть использовано для подтверждения содержания и факта существования документа в этот момент времени. Права на данный скриншот принадлежат администрации Ask3.ru, использование в качестве доказательства только с письменного разрешения правообладателя скриншота. Администрация Ask3.ru не несет ответственности за информацию размещенную на данном скриншоте. Права на прочие зарегистрированные элементы любого права, изображенные на снимках принадлежат их владельцам. Качество перевода предоставляется как есть. Любые претензии, иски не могут быть предъявлены. Если вы не согласны с любым пунктом перечисленным выше, вы не можете использовать данный сайт и информация размещенную на нем (сайте/странице), немедленно покиньте данный сайт. В случае нарушения любого пункта перечисленного выше, штраф 55! (Пятьдесят пять факториал, Денежную единицу (имеющую самостоятельную стоимость) можете выбрать самостоятельно, выплаичвается товарами в течение 7 дней с момента нарушения.)