Support Cyrillic text in Notes with UTF-8-safe editing

runtime/SimpleGraphic/Fonts/Liberation-Cyrillic.md

@@ -0,0 +1,26 @@
+# Liberation Sans Cyrillic extension
+
+These assets retain the original ASCII glyph metadata and atlas pixels, and
+append Cyrillic glyphs from Liberation Sans Regular 2.1.5 (U+0400–U+04FF).
+The accompanying `Liberation-LICENSE.txt` contains the font's license.
+
+Font source: [Liberation Fonts 2.1.5 release](https://github.com/liberationfonts/liberation-fonts/releases/tag/2.1.5),
+using the release-linked `liberation-fonts-ttf-2.1.5.tar.gz` archive.
+
+Generation uses the [extended FontGenerator](https://github.com/PathOfBuildingCommunity/PathOfBuilding-FontGenerator/pull/3)
+on Windows, with the Liberation Sans family, normal weight (400), variable pitch,
+default GDI quality, and sizes 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 32, 36,
+40, 48, 56 and 64. Append the generated glyphs to pristine original PoB assets
+(baseline `a69391d6daaaea1aaf8054668a094aad88dd7eaa`):
+
+```text
+python extend_font.py "original/Liberation Sans.tgf" "Generated Fonts/Liberation Sans.tgf" "extended/Liberation Sans.tgf"
+```
+
+Copy the resulting TGF and 16 TGA files together. Do not use an already extended
+atlas as the original input. All 2,048 ASCII metadata entries and original pixels
+were compared with the baseline and are unchanged.
+
+Displaying the new entries requires the corresponding
+[SimpleGraphic Unicode glyph reader](https://github.com/PathOfBuildingCommunity/PathOfBuilding-SimpleGraphic/pull/122).
+The older engine ignores the added metadata and continues using the ASCII glyphs.

runtime/SimpleGraphic/Fonts/Liberation-LICENSE.txt

@@ -0,0 +1,101 @@
+Digitized data copyright (c) 2010 Google Corporation
+	with Reserved Font Arimo, Tinos and Cousine.
+Copyright (c) 2012 Red Hat, Inc.
+	with Reserved Font Name Liberation.
+
+This Font Software is licensed under the SIL Open Font License,
+Version 1.1.
+
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+
+PREAMBLE The goals of the Open Font License (OFL) are to stimulate
+worldwide development of collaborative font projects, to support the font
+creation efforts of academic and linguistic communities, and to provide
+a free and open framework in which fonts may be shared and improved in
+partnership with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves.
+The fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works.  The fonts and derivatives,
+however, cannot be released under any other type of license.  The
+requirement for fonts to remain under this license does not apply to
+any document created using the fonts or their derivatives.
+
+
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such.
+This may include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components
+as distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting ? in part or in whole ?
+any of the components of the Original Version, by changing formats or
+by porting the Font Software to a new environment.
+
+"Author" refers to any designer, engineer, programmer, technical writer
+or other person who contributed to the Font Software.
+
+
+PERMISSION & CONDITIONS
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,in
+   Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+   redistributed and/or sold with any software, provided that each copy
+   contains the above copyright notice and this license. These can be
+   included either as stand-alone text files, human-readable headers or
+   in the appropriate machine-readable metadata fields within text or
+   binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+   Name(s) unless explicit written permission is granted by the
+   corresponding Copyright Holder. This restriction only applies to the
+   primary font name as presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+   Software shall not be used to promote, endorse or advertise any
+   Modified Version, except to acknowledge the contribution(s) of the
+   Copyright Holder(s) and the Author(s) or with their explicit written
+   permission.
+
+5) The Font Software, modified or unmodified, in part or in whole, must
+   be distributed entirely under this license, and must not be distributed
+   under any other license. The requirement for fonts to remain under
+   this license does not apply to any document created using the Font
+   Software.
+
+
+
+TERMINATION
+This license becomes null and void if any of the above conditions are not met.
+
+
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT.  IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER
+DEALINGS IN THE FONT SOFTWARE.

spec/System/TestEditControl_spec.lua

@@ -0,0 +1,110 @@
+-- Cyrillic Notes spec clauses 1 and 4.
+describe("UTF-8 notes editing", function()
+	local utf8 = require("lua-utf8")
+	local aura = utf8.char(0x410, 0x443, 0x440, 0x430)
+	local helmet = utf8.char(0x428, 0x43b, 0x435, 0x43c)
+	local function notes(text)
+		local tab = new("NotesTab"):NotesTab({})
+		tab.width, tab.height = 800, 600
+		tab.controls.edit:SetText(text)
+		return tab
+	end
+
+	local function press(edit, key, ctrl, clipboard, doubleClick)
+		local originalKey, originalPaste = _G.IsKeyDown, _G.Paste
+		local originalCursor = _G.GetCursorPos
+		_G.IsKeyDown = function(value) return value == "CTRL" and ctrl or false end
+		_G.Paste = function() return clipboard end
+		_G.GetCursorPos = function() return 20, 140 end
+		local success, message = pcall(edit.OnKeyDown, edit, key, doubleClick)
+		_G.IsKeyDown, _G.Paste = originalKey, originalPaste
+		_G.GetCursorPos = originalCursor
+		assert(success, message)
+	end
+
+	it("pastes Russian notes with colors, English names and links", function()
+		local edit = notes("").controls.edit
+		press(edit, "v", true, "^xFFFF77Ёж^7\tSmite\nhttps://pobb.in/example")
+		assert.are.equal("^xFFFF77Ёж^7\tSmite\nhttps://pobb.in/example", edit.buf)
+	end)
+
+	it("replaces selected notes with pasted UTF-8", function()
+		local edit = notes("old").controls.edit
+		edit.sel = 1
+		press(edit, "v", true, aura .. " Determination")
+		assert.are.equal(aura .. " Determination", edit.buf)
+	end)
+
+	for _, case in ipairs({
+		{ "BACK", "Aё", 4, "A", 2, false },
+		{ "DELETE", "ёA", 1, "A", 1, false },
+		{ "BACK", helmet .. " Smite", 15, helmet .. " Smit", 14, false },
+		{ "BACK", aura .. " ёж", 14, aura .. " ", 10, true },
+		{ "DELETE", "ёж Smite", 1, " Smite", 1, true },
+		{ "BACK", "А\nё", 6, "А\n", 4, true },
+		{ "DELETE", "ё\nА", 1, "\nА", 1, true },
+		{ "BACK", "A ёж  ", 9, "A ", 3, true },
+		{ "DELETE", "  ёж A", 1, " A", 1, true },
+		{ "BACK", "A ", 3, "", 1, true },
+		{ "DELETE", " A", 1, "", 1, true },
+	}) do
+		it(string.format("deletes complete characters with %s at byte %d in %q (ctrl=%s)", case[1], case[3], case[2], tostring(case[6])), function()
+			local edit = notes(case[2]).controls.edit
+			edit.caret = case[3]
+			press(edit, case[1], case[6])
+			assert.are.same({ case[4], case[5] }, { edit.buf, edit.caret })
+		end)
+	end
+
+	it("undoes and redoes deletion of a complete Russian character", function()
+		local edit = notes("Smite ё").controls.edit
+		press(edit, "BACK", false)
+		local deleted = edit.buf
+		edit:Undo()
+		local restored = edit.buf
+		edit:Redo()
+		assert.are.same({ "Smite ", "Smite ё", "Smite " }, { deleted, restored, edit.buf })
+	end)
+
+	it("moves the caret across complete Russian characters", function()
+		local edit = notes("Aё").controls.edit
+		press(edit, "LEFT", false)
+		local left = edit.caret
+		press(edit, "RIGHT", false)
+		assert.are.same({ 2, 4 }, { left, edit.caret })
+	end)
+
+	it("preserves Russian notes through XML save and load", function()
+		local original = notes("^7Ёж & Smite\n" .. aura .. " <Determination>")
+		local xml = { elem = "Notes" }
+		original:Save(xml)
+		local loaded = notes("")
+		local parsed = common.xml.ParseXML(common.xml.ComposeXML(xml))
+		loaded:Load(parsed[1], "fixture.xml")
+		assert.are.equal("^7Ёж & Smite\n" .. aura .. " <Determination>", loaded.controls.edit.buf)
+	end)
+
+	it("selects a complete Russian word on double click", function()
+		local edit = notes("Smite ёж " .. aura).controls.edit
+		edit.caret = 9
+		press(edit, "LEFTBUTTON", false, nil, true)
+		assert.are.same({ 7, 11 }, { edit.sel, edit.caret })
+	end)
+
+	-- Notes repair clause 1: whitespace must not move the caret into a UTF-8 character.
+	for _, case in ipairs({
+		{ "ёж ", 5, "ёж" },
+		{ "ёж  ", 6, "ёж" },
+		{ "ёж\t", 5, "ёж" },
+		{ "ёж\nSmite", 5, "ёж" },
+		{ "— ", 4, "—" },
+		{ "™\nSmite", 4, "™" },
+	}) do
+		it(string.format("selects a complete word before whitespace at byte %d in %q", case[2], case[1]), function()
+			local edit = notes(case[1]).controls.edit
+			edit.caret = case[2]
+			press(edit, "LEFTBUTTON", false, nil, true)
+			assert.are.equal(case[3], edit:GetSelText())
+		end)
+	end
+end)

src/Classes/EditControl.lua

@@ -9,6 +9,10 @@ local m_floor = math.floor
 local protected_replace = "*"
 local utf8 = require('lua-utf8')
 
+local function character(str, index)
+	return str:sub(index, (utf8.next(str, index, 1) or #str + 1) - 1)
+end
+
 local function lastLine(str)
 	local lastLineIndex = 1
 	while true do
@@ -460,27 +464,27 @@ function EditClass:OnKeyDown(key, doubleClick)
 		if doubleClick then
 			if self.lineHeight then
 				if self.buf:sub(self.caret - 1, self.caret):match("^%C\n$") then
-					self.caret = self.caret - 1
+					self.caret = utf8.next(self.buf, self.caret, -1)
 				end
 				while self.buf:sub(self.caret - 1, self.caret):match("[^\n][ \t]") do
-					self.caret = self.caret - 1
+					self.caret = utf8.next(self.buf, self.caret, -1)
 				end
-				local caretChar = self.buf:sub(self.caret, self.caret)
-				if caretChar:match("%w") then
+				local caretChar = character(self.buf, self.caret)
+				if utf8.match(caretChar, "%w") then
 					self.sel = self.caret
-					while self.buf:sub(self.sel - 1, self.sel - 1):match("%w") do
-						self.sel = self.sel - 1
+					while self.sel > 1 and utf8.match(character(self.buf, utf8.next(self.buf, self.sel, -1)), "%w") do
+						self.sel = utf8.next(self.buf, self.sel, -1)
 					end
-					while self.buf:sub(self.caret, self.caret):match("%w") do
-						self.caret = self.caret + 1
+					while utf8.match(character(self.buf, self.caret), "%w") do
+						self.caret = utf8.next(self.buf, self.caret, 1) or #self.buf + 1
 					end
 				elseif caretChar:match("%S") then
 					self.sel = self.caret
-					while self.buf:sub(self.sel - 1, self.sel - 1) == caretChar do
-						self.sel = self.sel - 1
+					while self.sel > 1 and character(self.buf, utf8.next(self.buf, self.sel, -1)) == caretChar do
+						self.sel = utf8.next(self.buf, self.sel, -1)
 					end
-					while self.buf:sub(self.caret, self.caret) == caretChar do
-						self.caret = self.caret + 1
+					while character(self.buf, self.caret) == caretChar do
+						self.caret = utf8.next(self.buf, self.caret, 1) or #self.buf + 1
 					end
 				end
 			else
@@ -536,7 +540,9 @@ function EditClass:OnKeyDown(key, doubleClick)
 			if self.pasteFilter then
 				text = self.pasteFilter(text)
 			end
-			text = text:gsub("[\128-\255]","?")
+			if not self.allowUnicode then
+				text = text:gsub("[\128-\255]","?")
+			end
 			if self.sel and self.sel ~= self.caret then
 				self:ReplaceSel(text)
 			else
@@ -613,19 +619,19 @@ function EditClass:OnKeyDown(key, doubleClick)
 		if self.sel and self.sel ~= self.caret then
 			self:ReplaceSel("")
 		elseif self.caret > 1 then
-			local len = 1
+			local left = utf8.next(self.buf, self.caret, -1)
 			if IsKeyDown("CTRL") then
-				while self.caret - len > 1 and self.buf:sub(self.caret - len, self.caret - len):match("%s") and not self.buf:sub(self.caret - len - 1, self.caret - len - 1):match("\n") do
-					len = len + 1
+				while left > 1 and utf8.match(character(self.buf, left), "%s") and self.buf:sub(left - 1, left - 1) ~= "\n" do
+					left = utf8.next(self.buf, left, -1)
 				end
-				if self.buf:sub(self.caret - len, self.caret - len):match("%w") then
-					while self.caret - len > 1 and self.buf:sub(self.caret - len - 1, self.caret - len - 1):match("%w") do
-						len = len + 1
+				if utf8.match(character(self.buf, left), "%w") then
+					while left > 1 and utf8.match(character(self.buf, utf8.next(self.buf, left, -1)), "%w") do
+						left = utf8.next(self.buf, left, -1)
 					end
 				end
 			end
-			self.buf = self.buf:sub(1, self.caret - 1 - len) .. self.buf:sub(self.caret)
-			self.caret = self.caret - len
+			self.buf = self.buf:sub(1, left - 1) .. self.buf:sub(self.caret)
+			self.caret = left
 			self.sel = nil
 			self:ScrollCaretIntoView()
 			self.blinkStart = GetTime()
@@ -638,18 +644,20 @@ function EditClass:OnKeyDown(key, doubleClick)
 		if self.sel and self.sel ~= self.caret then
 			self:ReplaceSel("")
 		elseif self.caret <= #self.buf then
-			local len = 1
+			local right = utf8.next(self.buf, self.caret, 1) or #self.buf + 1
+			local current = self.caret
 			if IsKeyDown("CTRL") then
-				while self.caret + len <= #self.buf and self.buf:sub(self.caret + len - 1, self.caret + len - 1):match("%s") and not self.buf:sub(self.caret + len, self.caret + len):match("\n") do
-					len = len + 1
+				while right <= #self.buf and utf8.match(character(self.buf, current), "%s") and character(self.buf, right) ~= "\n" do
+					current = right
+					right = utf8.next(self.buf, right, 1) or #self.buf + 1
 				end
-				if self.buf:sub(self.caret + len - 1, self.caret + len - 1):match("%w") then
-					while self.caret + len <= #self.buf and self.buf:sub(self.caret + len, self.caret + len):match("%w") do
-						len = len + 1
+				if utf8.match(character(self.buf, current), "%w") then
+					while right <= #self.buf and utf8.match(character(self.buf, right), "%w") do
+						right = utf8.next(self.buf, right, 1) or #self.buf + 1
 					end
 				end
 			end
-			self.buf = self.buf:sub(1, self.caret - 1) .. self.buf:sub(self.caret + len)
+			self.buf = self.buf:sub(1, self.caret - 1) .. self.buf:sub(right)
 			self.sel = nil
 			self.blinkStart = GetTime()
 			if self.changeFunc then

src/Classes/NotesTab.lua

@@ -36,6 +36,7 @@ Below are some common color codes PoB uses:	]]
 	self.controls.default = new("ButtonControl"):ButtonControl({"TOPLEFT",self.controls.intelligence,"TOPLEFT"}, {120, 0, 100, 18}, "^7DEFAULT", function() self:SetColor("^7") end)
 
 	self.controls.edit = new("EditControl"):EditControl({"TOPLEFT",self.controls.fire,"TOPLEFT"}, {0, 48, 0, 0}, "", nil, "^%C\t\n", nil, nil, 16, true)
+	self.controls.edit.allowUnicode = true
 	self.controls.edit.width = function()
 		return self.width - 16
 	end