415 lines
15 KiB
Plaintext
415 lines
15 KiB
Plaintext
-*-mode: text; coding: utf-8;-*-
|
||
|
||
Copyright (C) 2002-2024 Free Software Foundation, Inc.
|
||
See the end of the file for license conditions.
|
||
|
||
Importing a new Unicode Standard version into Emacs
|
||
-------------------------------------------------------------
|
||
|
||
Emacs uses the following files from the Unicode Character Database
|
||
(a.k.a. "UCD):
|
||
|
||
. UnicodeData.txt
|
||
. Blocks.txt
|
||
. BidiBrackets.txt
|
||
. BidiMirroring.txt
|
||
. IVD_Sequences.txt
|
||
. NormalizationTest.txt
|
||
. PropertyValueAliases.txt
|
||
. ScriptExtensions.txt
|
||
. Scripts.txt
|
||
. SpecialCasing.txt
|
||
. confusables.txt
|
||
. emoji-data.txt
|
||
. emoji-zwj-sequences.txt
|
||
. emoji-variation-sequences.txt
|
||
. emoji-sequences.txt
|
||
. BidiCharacterTest.txt
|
||
|
||
Emacs also uses the file emoji-test.txt which should be imported from
|
||
the Unicode's Public/emoji/ directory, and IdnaMappingTable.txt from
|
||
the Public/idna/ directory.
|
||
|
||
"admin/unidata/README" contains the URLs that can be used to download
|
||
these files.
|
||
|
||
First, the first 15 files, emoji-test.txt and IdnaMappingTable.txt
|
||
need to be copied into admin/unidata/, and the file
|
||
https://www.unicode.org/copyright.html should be copied over
|
||
copyright.html in admin/unidata (some of them might need trailing
|
||
whitespace removed before they can be committed to the Emacs
|
||
repository).
|
||
|
||
Next, review the assignment of default values of the Bidi Class
|
||
property to blocks in the file extracted/DerivedBidiClass.txt from the
|
||
UCD (search for "unassigned" and "@missing" in that file). Any
|
||
changes should be reflected in the unidata-gen.el file, where it sets
|
||
up the default values around line 210.
|
||
|
||
Then Emacs should be rebuilt for them to take effect. Rebuilding
|
||
Emacs updates several derived files elsewhere in the Emacs source
|
||
tree, mainly in lisp/international/.
|
||
|
||
When Emacs is rebuilt for the first time after importing the new
|
||
files, pay attention to any warning or error messages. In particular,
|
||
admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
|
||
new bidirectional attributes of characters, because unidata-gen.el,
|
||
bidi.c and dispextern.h need to be updated in that case; failure to do
|
||
so will cause aborts in redisplay. unidata-gen.el will also complain
|
||
if the format of the Unicode Copyright notice in copyright.html
|
||
changed in significant ways; in that case, update the regular
|
||
expression in unidata-gen-file used to extract the copyright string.
|
||
|
||
Next, review the changes in UnicodeData.txt vs the previous version
|
||
used by Emacs. Any changes, be it introduction of new scripts or
|
||
addition of codepoints to existing scripts, might need corresponding
|
||
changes in the data used for filling the category-table, case-table,
|
||
and char-width-table in characters.el. The additional scripts should
|
||
cause automatic updates in charscript.el, but it is a good idea to
|
||
look at the results and see if any changes in admin/unidata/blocks.awk
|
||
are required.
|
||
|
||
The setting of char-width-table around line 1200 of characters.el
|
||
should be checked against the latest version of the Unicode file
|
||
EastAsianWidth.txt, and any discrepancies fixed: double-width
|
||
characters are those marked with W or F in that file. Zero-width
|
||
characters are not taken from EastAsianWidth.txt, they are those whose
|
||
Unicode General Category property is one of Mn, Me, or Cf, and also
|
||
Hangul jungseong and jongseong characters (a.k.a. "Jamo medial vowels"
|
||
and "Jamo final consonants"). The list of "ambiguous-width
|
||
characters" recorded in the ambiguous-width-chars table (around line
|
||
1400 of characters.el) should also be checked against the latest
|
||
Unicode data in EastAsianWidth.txt.
|
||
|
||
Any new scripts added by UnicodeData.txt will also need updates to
|
||
script-representative-chars defined in fontset.el, and also the list
|
||
of OTF script tags in otf-script-alist, whose source is on this page:
|
||
|
||
https://docs.microsoft.com/en-us/typography/opentype/spec/scripttags
|
||
|
||
Other databases in fontset.el might also need to be updated as needed.
|
||
One notable place to check is the function setup-default-fontset,
|
||
where new scripts will generally need some addition, most probably to
|
||
the list of "simple" scripts (search for "Simple").
|
||
|
||
The function 'ucs-names', defined in lisp/international/mule-cmds.el,
|
||
might need to be updated because it knows about used and unused ranges
|
||
of Unicode codepoints, which a new release of the Unicode Standard
|
||
could change.
|
||
|
||
The data used by ucs-normalize.el might need to be updated.
|
||
Specifically, the values of 'ucs-normalize-composition-exclusions' and
|
||
'check-range", defined at the beginning of ucs-normalize.el, should be
|
||
verified against the latest Unicode data files.
|
||
|
||
Next, test normalization functions against NormalizationTests.txt,
|
||
in the top-level directory run:
|
||
|
||
make -C test lisp/international/ucs-normalize-tests
|
||
|
||
See commentary in test/lisp/international/ucs-normalize-tests.el
|
||
regarding failing lines.
|
||
|
||
Changes in UTS#46 and other relevant Unicode data might break our
|
||
features that detect confusable and suspicious text, so run
|
||
test/lisp-international/textsec-tests to check that:
|
||
|
||
make -C test lisp/international/textsec-tests
|
||
|
||
The file BidiCharacterTest.txt should be copied to the test suite, and
|
||
if its format has changed, the file biditest.el there should be
|
||
modified to follow suit. If there's trailing whitespace in
|
||
BidiCharacterTest.txt, it should be removed before committing the new
|
||
version.
|
||
|
||
src/macuvs.h is a generated file, but if it has changed as a result
|
||
of the updates, please commit it as well (see
|
||
admin/unidata/Makefile.in for an explanation).
|
||
|
||
Visit "emoji-data.txt" with the rebuilt Emacs, and check that an
|
||
appropriate font is being used for the emoji (by default Emacs uses
|
||
"Noto Color Emoji"). Running the following command in that buffer
|
||
will give you an idea of which codepoints are not supported by
|
||
whichever font Emacs is using.
|
||
|
||
(defun check-emoji-coverage (font-name-regexp)
|
||
"Display a buffer containing emoji codepoints for which FONT-NAME is not used.
|
||
This must be run from a buffer in the format of emoji-data.txt.
|
||
FONT-NAME-REGEXP is checked using `string-match'."
|
||
(interactive "MFont Name: ")
|
||
(save-excursion
|
||
(goto-char (point-min))
|
||
(let (res char name ifont)
|
||
(while (re-search-forward "; Emoji_Presentation [^(]+(\\(.\\)[).]" nil t)
|
||
(setq char (aref (match-string 1) 0))
|
||
(setq ifont (car (internal-char-font nil char)))
|
||
(when ifont
|
||
(setq name (font-xlfd-name ifont)))
|
||
(if (or (not ifont) (not (string-match font-name-regexp name)))
|
||
(setq res (concat (string char) res))))
|
||
(when res
|
||
(with-output-to-temp-buffer "*Check-Emoji-Coverage*"
|
||
(princ (format "Font not matching '%s' was used for the following characters:\n%s"
|
||
font-name-regexp (reverse res))))))))
|
||
|
||
Visit "emoji-zwj-sequences.txt" and "emoji-sequences.txt" with the
|
||
rebuilt Emacs, and check that the sample sequences are composed
|
||
properly. Also check the Unicode style chart file available at
|
||
https://unicode.org/emoji/charts/emoji-style.txt for any issues
|
||
involving VS-15 and VS-16, if so you may need to update the value
|
||
generated for auto-composition-emoji-eligible-codepoints by
|
||
admin/unidata/emoji-zwj.awk. Note that your emoji font might not have
|
||
glyphs for the newest codepoints yet.
|
||
|
||
Visit "emoji-variation-sequences.txt", and run the following lisp
|
||
fragment to actually insert the described codepoints, then check that
|
||
all the text in parentheses displays correctly (it can be helpful to
|
||
have `glyphless-char-display-control' customized to show hex codes for
|
||
variation selectors).
|
||
|
||
(save-excursion
|
||
(goto-char (point-min))
|
||
(while (re-search-forward "^\\([0-9A-Z]+\\) \\([0-9A-Z]+\\).*(\\([^)]+\\))" nil t)
|
||
(let ((ch (string-to-number (match-string 1) 16))
|
||
(sel (string-to-number (match-string 2) 16))
|
||
(sp (match-string 3)))
|
||
(replace-match (format "%s %c%c " sp ch sel) nil nil nil 3))))
|
||
|
||
Finally, etc/NEWS should be updated to announce the support for the
|
||
new Unicode version.
|
||
|
||
Problems, fixmes and other unicode-related issues
|
||
-------------------------------------------------------------
|
||
|
||
Notes by fx to record various things of variable importance. Handa
|
||
needs to check them -- don't take too seriously, especially with
|
||
regard to completeness.
|
||
|
||
* SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
|
||
undesirable effects. E.g.:
|
||
(multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
|
||
(multibyte-string-p (concat [?£])) => nil
|
||
(text-char-description ?£) => "M-#"
|
||
|
||
These examples are all fixed by the change of 2002-10-14, but
|
||
there still exist questionable SINGLE_BYTE_CHAR_P in the
|
||
code (keymap.c and print.c).
|
||
|
||
* Rationalize character syntax and its relationship to the Unicode
|
||
database. (Applies mainly to symbol and punctuation syntax.)
|
||
|
||
* Fontset handling and customization needs work. We want to relate
|
||
fonts to scripts, probably based on the Unicode blocks. The
|
||
presence of small-repertoire 10646-encoded fonts in XFree 4 is a
|
||
pain, not currently worked round.
|
||
|
||
With the change on 2002-07-26, multiple fonts can be
|
||
specified in a fontset for a specific range of characters.
|
||
Each range can also be specified by script. Before using
|
||
ISO10646 fonts, Emacs checks their repertories to avoid such
|
||
fonts that don't have a glyph for a specific character.
|
||
|
||
fx has worked on fontset customization, but was stymied by
|
||
basic problems with the way the default face is dealt with
|
||
(and something else, I think). This needs revisiting.
|
||
|
||
* Work is also needed on charset and coding system priorities.
|
||
|
||
* The relevant bits of latin1-disp.el need porting (and probably
|
||
re-naming/updating). See also cyril-util.el.
|
||
|
||
* Quail files need more work now the encoding is largely irrelevant.
|
||
|
||
* What to do with the old coding categories stuff?
|
||
|
||
* The preferred-coding-system property of charsets should probably be
|
||
junked unless it can be made more useful now.
|
||
|
||
* find-multibyte-characters needs looking at.
|
||
|
||
* Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
|
||
charsets.
|
||
|
||
* Lazy-load tables for unify-charset somehow?
|
||
|
||
Actually, Emacs clears out all charset maps and unify-map just
|
||
before dumping, and they are loaded again on demand by the
|
||
dumped emacs. But, those maps (char tables) generated while
|
||
temacs is running can't be removed from the dumped emacs.
|
||
|
||
* iso-2022 charsets get unified on i/o.
|
||
|
||
With the change on 2003-01-06, decoding routines put the 'charset'
|
||
property onto decoded text, and iso-2022 encoder pay attention
|
||
to it. Thus, for instance, reading and writing by
|
||
iso-2022-7bit preserve the original designation sequences.
|
||
The property name 'preferred-charset' may be better?
|
||
|
||
We may have to utilize this property to decide a font.
|
||
|
||
* Revisit locale processing: look at treating the language and
|
||
charset parts separately. (Language should affect things like
|
||
spelling and calendar, but that's not a Unicode issue.)
|
||
|
||
* Handle Unicode combining characters usefully, e.g. diacritics, and
|
||
handle more scripts specifically (à la Devanagari). There are
|
||
issues with canonicalization.
|
||
|
||
* We need tabular input methods, e.g. for maths symbols. (Not
|
||
specific to Unicode.)
|
||
|
||
* Need multibyte text in menus, e.g. for the above. (Not specific to
|
||
Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
|
||
|
||
* Populate char-width-table correctly for Unicode characters and
|
||
worry about what happens when double-width charsets covering
|
||
non-CJK characters are unified.
|
||
|
||
* There are type errors lurking, e.g. in
|
||
Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them.
|
||
|
||
* Old auto-save files, and similar files, such as Gnus drafts,
|
||
containing non-ASCII characters probably won't be re-read correctly.
|
||
|
||
|
||
Source file encoding
|
||
--------------------
|
||
|
||
Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
|
||
subset), but there are a few exceptions, listed below. Perhaps
|
||
someday many of these files will be converted to UTF-8, for
|
||
convenience when using tools like 'grep -r', but this might need
|
||
nontrivial changes to the build process.
|
||
|
||
* chinese-big5
|
||
|
||
These are verbatim copies of files taken from external sources.
|
||
They haven't been converted to UTF-8.
|
||
|
||
leim/CXTERM-DIC/4Corner.tit
|
||
leim/CXTERM-DIC/ARRAY30.tit
|
||
leim/CXTERM-DIC/ECDICT.tit
|
||
leim/CXTERM-DIC/ETZY.tit
|
||
leim/CXTERM-DIC/PY-b5.tit
|
||
leim/CXTERM-DIC/Punct-b5.tit
|
||
leim/CXTERM-DIC/QJ-b5.tit
|
||
leim/CXTERM-DIC/ZOZY.tit
|
||
leim/MISC-DIC/CTLau-b5.html
|
||
leim/MISC-DIC/cangjie-table.b5
|
||
|
||
* chinese-iso-8bit
|
||
|
||
These are verbatim copies of files taken from external sources.
|
||
They haven't been converted to UTF-8.
|
||
|
||
leim/CXTERM-DIC/CCDOSPY.tit
|
||
leim/CXTERM-DIC/Punct.tit
|
||
leim/CXTERM-DIC/QJ.tit
|
||
leim/CXTERM-DIC/SW.tit
|
||
leim/CXTERM-DIC/TONEPY.tit
|
||
leim/MISC-DIC/CTLau.html
|
||
leim/MISC-DIC/pinyin.map
|
||
leim/MISC-DIC/ziranma.cin
|
||
|
||
* cp850
|
||
|
||
This file contains non-ASCII characters in unibyte strings. When
|
||
editing a keyboard layout it's more convenient to see 'é' than
|
||
'\202', and the MS-DOS compiler requires the single byte if a
|
||
backslash escape is not being used.
|
||
|
||
src/msdos.c
|
||
|
||
* iso-latin-1
|
||
|
||
This file is used to test Emacs encoding.
|
||
|
||
test/lisp/gnus/mm-decode-resources/win1252-multipart.bin
|
||
|
||
* iso-2022-cn-ext
|
||
|
||
This file is externally generated from leim/MISC-DIC/cangjie-table.b5
|
||
by Big5->CNS converter. It hasn't been converted to UTF-8.
|
||
|
||
leim/MISC-DIC/cangjie-table.cns
|
||
|
||
* japanese-iso-8bit
|
||
|
||
SKK-JISYO.L is a verbatim copy of a file taken from an external source.
|
||
It hasn't been converted to UTF-8.
|
||
|
||
leim/SKK-DIC/SKK-JISYO.L
|
||
|
||
* japanese-shift-jis
|
||
|
||
This is a verbatim copy of a file taken from an external source.
|
||
It hasn't been converted to UTF-8.
|
||
|
||
admin/charsets/mapfiles/cns2ucsdkw.txt
|
||
|
||
* iso-2022-jp
|
||
|
||
This contains just one CJK charset, but Emacs currently has no
|
||
easy way to specify set-charset-priority on a per-file basis, so
|
||
converting this file to UTF-8 might change the file's appearance
|
||
when viewed by an Emacs that is operating in some other language
|
||
environment.
|
||
|
||
etc/tutorials/TUTORIAL.ja
|
||
|
||
* utf-8-emacs
|
||
|
||
These files contain characters that cannot be encoded in UTF-8.
|
||
|
||
lisp/language/ethio-util.el
|
||
lisp/language/ethiopic.el
|
||
lisp/language/ind-util.el
|
||
lisp/language/tibet-util.el
|
||
lisp/language/tibetan.el
|
||
lisp/leim/quail/ethiopic.el
|
||
lisp/leim/quail/tibetan.el
|
||
lisp/international/titdic-cnv.el
|
||
|
||
* binary files
|
||
|
||
These files contain binary data, and are not text files.
|
||
Some of the entries in this list are patterns, and stand for any
|
||
files with the listed extension.
|
||
|
||
*.bmp
|
||
*.cur
|
||
*.gif
|
||
*.gpg
|
||
*.gz
|
||
*.icns
|
||
*.ico
|
||
*.jpg
|
||
*.kbx
|
||
*.key
|
||
*.pbm
|
||
*.pdf
|
||
*.pif
|
||
*.png
|
||
*.sig
|
||
*.tiff
|
||
*.webp
|
||
*.zip
|
||
etc/e/eterm-color
|
||
etc/e/eterm-direct
|
||
java/emacs.keystore
|
||
|
||
This file is part of GNU Emacs.
|
||
|
||
GNU Emacs is free software: you can redistribute it and/or modify
|
||
it under the terms of the GNU General Public License as published by
|
||
the Free Software Foundation, either version 3 of the License, or
|
||
(at your option) any later version.
|
||
|
||
GNU Emacs is distributed in the hope that it will be useful,
|
||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||
GNU General Public License for more details.
|
||
|
||
You should have received a copy of the GNU General Public License
|
||
along with GNU Emacs. If not, see <https://www.gnu.org/licenses/>.
|