那些遇到过的问题

Ruby 编码 ASCII_8BIT 和扩展 ASCII

nor*_*raj 5 ruby encoding extended-ascii

关于ASCII_8BIT

Encoding::ASCII_8BIT 是一种特殊的编码，通常用于字节字符串，而不是字符串。但顾名思义，其 ASCII 范围内的字符被视为 ASCII 字符。当您将 ASCII-8BIT 字符与其他 ASCII 兼容字符一起使用时，这非常有用。

资料来源：ruby-doc.org/core-2.6.4

语境

我想使用 ASCII_8BIT，因为我需要对 0x00 (0d00) 和 0xff (0d255) 之间的所有字符进行编码，因此 ASCII (0-127) 加上扩展 ASCII (128-255)。ASCII（编码，US-ASCII）是一种 7 位编码，仅识别 ASCII（字符集）字符 (0-127)。正如名称所示，我预计 ASCII_8BIT 会将其扩展到 8 位以添加对 128-255 的支持。

问题

当我使用 chr 时，编码会自动设置为 ASCII_8BIT，但是当我放置时，我将 128-255 (0x80-0xff) 之间的字符直接放入字符串中，然后询问编码是什么，我得到的是 UTF-8，如果我尝试将其转换为 ASCII_8BIT 会出错。

irb(main):014:0> 0x8f.chr
=> "\x8F"
irb(main):015:0> 0x8f.chr.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):016:0> "\x8f".encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
        5: from /usr/bin/irb:23:in `<main>'
        4: from /usr/bin/irb:23:in `load'
        3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        2: from (irb):16
        1: from (irb):16:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main):021:0> "\x8F".encoding
=> #<Encoding:UTF-8>
Run Code Online (Sandbox Code Playgroud)

ruby 核心有错误吗？我需要能够对 8 之间的所有内容进行编码

ASCII 8BIT 的另一个名称是 BINARY，因为正如前面的引用所述，它应该能够对任何字节进行编码。

irb(main):035:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]
Run Code Online (Sandbox Code Playgroud)

其他编码

请告诉我使用另一种编码并不是问题的答案，除非它是真正映射所有 255 个扩展 ASCII 字符的编码。

  • 我不想使用UTF-8，因为编码是多字节而不是单字节。
  • ISO/IEC 8859-1（Latin1，8 位）仅包含 191 个字符（ASCII + 63 个字符）

    ISO 字符集与代码页的一个显着区别是，字符位置 128 到 159（对应于具有高阶位集的 ASCII 控制字符）在 ISO 标准中专门未使用和未定义，尽管它们经常被使用对于专有代码页中的可打印字符，这违反了几乎普遍存在的 ISO 标准。参考号 扩展 ASCII-ISO 8859 和专有改编

  • Windows-1252（CP-1252，8 位）不包含所有 255 个字符，并且作为扩展 ASCII 的不同映射

ruby 中可用的编码：

irb(main):036:0> Encoding.name_list
=> ["ASCII-8BIT", "UTF-8", "US-ASCII", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "UTF-16", "UTF-32", "UTF8-MAC", "EUC-JP", "Windows-31J", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "Windows-1250", "Windows-1251", "Windows-1252", "Windows-1253", "Windows-1254", "Windows-1257", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "IBM037", "ebcdic-cp-us", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "EUC-JIS-2004", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "CP1253", "CP1254", "CP1257", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal"]
Run Code Online (Sandbox Code Playgroud)

用于比较 python 编码https://docs.python.org/3/library/codecs.html#standard-encodings

注意事项

通过阅读扩展 ASCII - 多字节字符编码，似乎唯一真正的扩展 ASCII 编码是 UTF-8 但是 Multi-byte 。似乎也不存在真正的扩展 ASCII 单字节编码。

从字节的角度来看，我可以使用任何 8 位（单字节）编码，如此处扩展 ASCII - 在计算机可读语言中的使用

所有 ASCII 字节（0x00 到 0x7F）在扩展 ASCII 的所有变体中都具有相同的含义，

但问题是像 ISO-8859-1 这样的实现专门未定义某些字符范围，因此会导致错误。

irb(main):009:0> (0..255).map { |c| c.chr}.join.encode(Encoding::ISO_8859_1)
Traceback (most recent call last):
        6: from /usr/bin/irb:23:in `<main>'
        5: from /usr/bin/irb:23:in `load'
        4: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        3: from (irb):9
        2: from (irb):9:in `rescue in irb_binding'
        1: from (irb):9:in `encode'
Encoding::UndefinedConversionError ("\x80" to UTF-8 in conversion from ASCII-8BIT to UTF-8 to ISO-8859-1)
Run Code Online (Sandbox Code Playgroud)

更新-force_encoding

我找到了字符串方法force_encoding。

irb(main)> a = "\x8f"
=> "\x8F"
irb(main)> a.encoding
=> #<Encoding:UTF-8>
irb(main)> a.encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
        5: from /usr/bin/irb:23:in `<main>'
        4: from /usr/bin/irb:23:in `load'
        3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        2: from (irb):42
        1: from (irb):42:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main)> a.force_encoding(Encoding::ASCII_8BIT)
=> "\x8F"
irb(main):040:0> a.encoding
=> #<Encoding:ASCII-8BIT>
Run Code Online (Sandbox Code Playgroud)

force_encoding使用而不是有什么危险encode？难道只是如果我不小心传递了一个多字节字符，它会被转换为多个单字节字符吗？因此，如果确保传递给应用程序的所有字符都在扩展 ASCII 范围（单字节）内，则不会有危险，但不安全，并且如果将某些 UTF-8 字符传递给应用程序，则会导致静默转换问题。

irb(main):044:0> "\ud087".force_encoding(Encoding::ASCII_8BIT)
=> "\xED\x82\x87"
irb(main):045:0> "\ud087".bytes
=> [237, 130, 135]
Run Code Online (Sandbox Code Playgroud)

更新 - 答案

@mu-is-too-short 的答案和 @ForeverZer0 评论建议的是我应该使用pack并unpack处理原始字节。

因此，不要使用编码和解决方法

irb(main):014:0> 0x8f.chr
=> "\x8F"
irb(main):015:0> 0x8f.chr.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):016:0> "\x8f".encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
        5: from /usr/bin/irb:23:in `<main>'
        4: from /usr/bin/irb:23:in `load'
        3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        2: from (irb):16
        1: from (irb):16:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main):021:0> "\x8F".encoding
=> #<Encoding:UTF-8>
Run Code Online (Sandbox Code Playgroud)

我应该直接使用字节

irb(main):035:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]
Run Code Online (Sandbox Code Playgroud)

或者这种更容易阅读的语法

irb(main):036:0> Encoding.name_list
=> ["ASCII-8BIT", "UTF-8", "US-ASCII", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "UTF-16", "UTF-32", "UTF8-MAC", "EUC-JP", "Windows-31J", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "Windows-1250", "Windows-1251", "Windows-1252", "Windows-1253", "Windows-1254", "Windows-1257", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "IBM037", "ebcdic-cp-us", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "EUC-JIS-2004", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "CP1253", "CP1254", "CP1257", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal"]
Run Code Online (Sandbox Code Playgroud)

mu *_*ort 5

字符串文字（通常）是 UTF-8 编码的，无论字节是否是有效的 UTF-8。因此：

"\x8f".encoding
Run Code Online (Sandbox Code Playgroud)

即使字符串不是有效的 UTF-8，也显示 UTF-8。您应该安全使用String#force_encoding，但如果您确实想使用原始字节，那么最好使用整数数组并将Array#pack它们混搭成字符串：

[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*')
# "\x8F\x11\x06#\xFF\x00" 
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').encoding
# #<Encoding:ASCII-8BIT> 
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').bytes
# [143, 17, 6, 35, 255, 0]
Run Code Online (Sandbox Code Playgroud)

结果应该是相同的，但是，IMO，这是明确使用二进制数据（即原始字节），使您的意图清晰，并且应该避免任何编码问题。

还有一种情况是String#unpack，您正在读取的字节是否有已知的结构，并且您想将其破解。

  • 这主要是一个品味问题（但为了明确起见，我会说“force_encoding('binary')”），但如果我使用字节，那么我会使用“pack”。 (3认同)

归档时间：

查看次数：

7963 次

最近记录：

6 年，7 月 前
相关归档
如何修改Homebrew的PATH？ 147
如何在Rails 3迁移中描述枚举列？ 64
如何在Windows 7上安装RVM 34
用Homebrew安装Ruby 33
Ruby(和Rails)嵌套模块语法 29
如何在rails中缓存计算列？ 14
Node.JS刮编码？ 12
to_json没有将特殊字符转换为unicode样式 10
如何在Ruby中将整数舍入到<最接近的大数>？ 9
我如何让python将瑞典语字母（åäö）写入html文件？ 5
难疑归档
不同浏览器中URL的最大长度是多少？ 4676
403 Forbidden vs 401 Unauthorized HTTP响应 2544
如何将命令行参数传递给Node.js程序？ 2280
是否有标准函数来检查JavaScript中的null,undefined或blank变量？ 2088
找到已安装的npm软件包的版本 2037
如何基于通配符匹配以递归方式查找当前和子文件夹中的所有文件？ 1695
从Git提交中删除文件 1484
在jQuery中序列化为JSON 1189
为什么使用Redux而不是Facebook Flux？ 1126
如何将参数传递给批处理文件？ 1100