SYSTEM WARNING: Creating default object from empty value

SYSTEM WARNING: Creating default object from empty value

SYSTEM WARNING: Creating default object from empty value

SYSTEM WARNING: Creating default object from empty value

Mantis - Squeak
Viewing Issue Advanced Details
1650 Graphics major always 08-11-05 00:32 09-30-13 23:27
Eddie Cottongim  
closed 3.8  
none 4.4  
0001650: [BUG] CharacterScanner primitive is broken; char scanning generally in a mess
CharacterScanner >> basicScanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: rightX stopConditions: stops kern: kernDelta

calls primitive 103, and its broken (always fails). Seems to fail because it wants an iVar "map" which was removed in 3.8. Causes very slow editing of large paragraphs.

I do not know a trivial fix. Seems to be related to installation of StrikeFontSet and since there is zero documentation on this I don't know how to fix it.
print it: String new:10000 withAll: $. then insert some text near the beginning. extremely choppy in 3.8, works smooth in 3.7.

(Later - prim has to be replaced by a new one in later images; we have to keep the old one around for old images that actually make use of it. The assorted CharacterScanners and MultiCharacterScanners need cleaning out and probably merging)
related to 0007589closed tim text composer and character scanner disagree on the width of a paragraph of antialiased strikefonts 
related to 0001372assigned tim Fonts have invisible characters 
related to 0001781closed tim BitBlt>>primDisplayString: is failing often 
related to 0007338closed tim A little font rendering speedup 
related to 0007789closed tim CharacterScanner improvements to handle fonts and byte/wide Strings more cleanly 

08-17-05 20:29   
I don't know if the primitive is the culprit. Back in Squeak 2.3, 2.4 era, the same primitive had been failing all the time because of the parameter change, but hardly nobody had noticed. (I only noticed it when I ran an image in the InterpreterSimulator and on MI-506 Zaurus PDA.)

  I ran a MessageTally on 'String new:10000 withAll: $.' test, and got something like below. The 10%-ish number is more believeable.

24.9% {4914ms} Delay>>wait
14.9% {2940ms} StrikeFont>>widthOf:
13.2% {2605ms} CharacterBlockScanner(CharacterScanner)>>basicScanCharactersFr...stopConditions:kern:
11.4% {2250ms} StrikeFontSet>>fontOf:ifAbsent:
7.5% {1480ms} StrikeFontSet>>widthOf:
6.7% {1322ms} Character>>leadingChar
3.1% {612ms} StrikeFontSet>>fontOf:
2.3% {454ms} Character>>charCode
Eddie Cottongim   
08-30-05 02:54   
I did the test, held down a key and inserted 'g's for about 10 seconds and copied my message tallies here from clean 3.7 and 3.8 images.

 - 11339 tallies, 11367 msec.

100.0% {11367ms} PasteUpMorph>>doOneCycle
  100.0% {11367ms} WorldState>>doOneCycleFor:
    66.0% {7502ms} WorldState>>doOneCycleNowFor:
      |48.3% {5490ms} WorldState>>displayWorldSafely:
      | |48.3% {5490ms} PasteUpMorph>>displayWorld
      | | 48.3% {5490ms} PasteUpMorph>>privateOuterDisplayWorld
      | | 48.3% {5490ms} WorldState>>displayWorld:submorphs:
      | | 45.8% {5206ms} WorldState>>drawWorld:submorphs:invalidAreasOn:
      | | |44.6% {5070ms} FormCanvas(Canvas)>>fullDrawMorph:
      | | | 44.6% {5070ms} FormCanvas(Canvas)>>fullDraw:
      | | | 44.6% {5070ms} SystemWindow(Morph)>>fullDrawOn:
      | | | 41.2% {4683ms} SystemWindow(Morph)>>drawSubmorphsOn:
      | | | |41.2% {4683ms} FormCanvas(Canvas)>>fullDrawMorph:
      | | | | 41.1% {4672ms} FormCanvas(Canvas)>>fullDraw:
      | | | | 41.1% {4672ms} PluggableTextMorph(Morph)>>fullDrawOn:
      | | | | 40.2% {4570ms} PluggableTextMorph(Morph)>>drawSubmorphsOn:
      | | | | 40.2% {4570ms} FormCanvas(Canvas)>>fullDrawMorph:
[40.2% {4570ms} FormCanvas(Canvas)>>fullDraw:
[ 40.2% {4570ms} TransformMorph(Morph)>>fullDrawOn:
[ 36.6% {4160ms} TransformMorph>>drawSubmorphsOn:
[ |36.5% {4149ms} FormCanvas(Canvas)>>fullDrawMorph:
[ | 36.5% {4149ms} FormCanvas(Canvas)>>fullDraw:
[ | 36.5% {4149ms} TextMorphForEditView(Morph)>>fullDrawOn:
[ | 36.4% {4138ms} FormCanvas(Canvas)>>drawMorph:
[ | 36.4% {4138ms} FormCanvas(Canvas)>>draw:
[ | 36.4% {4138ms} TextMorphForEditView(TextMorph)>>drawOn:
[ | 36.2% {4115ms} FormCanvas>>paragraph:bounds:color:
[ | 36.1% {4103ms} NewParagraph>>displayOn:using:at:
[ | 35.4% {4024ms} DisplayScanner>>displayLine:offset:leftInRun:
[ | 33.5% {3808ms} StrikeFont>>displayString:on:from:to:at:kern:
[ | 33.2% {3774ms} GrafPort(BitBlt)>>displayString:from:to:at:strikeFont:kern:
[ | 33.2% {3774ms} primitives
[ 3.2% {364ms} ScrollBar(Morph)>>drawSubmorphsOn:
[ 3.2% {364ms} FormCanvas(Canvas)>>fullDrawMorph:
[ 3.2% {364ms} FormCanvas(Canvas)>>fullDraw:
[ 3.2% {364ms} RectangleMorph(Morph)>>fullDrawOn:
[ 2.5% {284ms} FormCanvas(Canvas)>>drawMorph:
[ 2.5% {284ms} FormCanvas(Canvas)>>draw:
[ 2.5% {284ms} RectangleMorph(Morph)>>drawOn:
[ 2.3% {261ms} FormCanvas(Canvas)>>fillRectangle:fillStyle:borderStyle:
      | | | 2.5% {284ms} FormCanvas(Canvas)>>drawMorph:
      | | | 2.5% {284ms} FormCanvas(Canvas)>>draw:
      | | | 2.5% {284ms} SystemWindow(Morph)>>drawOn:
      | | | 2.4% {273ms} FormCanvas(Canvas)>>fillRectangle:fillStyle:borderStyle:
      | | 2.4% {273ms} WorldState>>forceDamageToScreen:
      | | 2.4% {273ms} DisplayScreen>>forceDamageToScreen:
      | | 2.4% {273ms} OrderedCollection>>do:
      |17.6% {2001ms} HandMorph>>processEvents
      | 16.9% {1921ms} HandMorph>>handleEvent:
      | 14.0% {1591ms} HandMorph>>sendKeyboardEvent:
      | |14.0% {1591ms} HandMorph>>sendEvent:focus:clear:
      | | 14.0% {1591ms} HandMorph>>sendFocusEvent:to:clear:
      | | 13.9% {1580ms} TextMorphForEditView(Morph)>>handleFocusEvent:
      | | 13.9% {1580ms} TextMorphForEditView(Morph)>>handleEvent:
      | | 13.9% {1580ms} KeyboardEvent>>sentTo:
      | | 13.9% {1580ms} TextMorphForEditView(TextMorph)>>handleKeystroke:
      | | 13.9% {1580ms} TextMorphForEditView>>keyStroke:
      | | 13.7% {1557ms} TextMorphForEditView(TextMorph)>>keyStroke:
      | | 8.5% {966ms} TextMorphEditor>>readKeyboard
      | | |8.5% {966ms} TextMorphEditor(ParagraphEditor)>>readKeyboard
      | | | 8.2% {932ms} TextMorphEditor>>zapSelectionWith:
[7.8% {887ms} NewParagraph>>replaceFrom:to:with:displaying:
[ 7.7% {875ms} NewParagraph>>recomposeFrom:to:delta:
[ 7.7% {875ms} NewParagraph>>composeLinesFrom:to:delta:into:priorLines:atY:
[ 7.7% {875ms} TextComposer>>composeLinesFrom:to:del...ner:wantsColumnBreaks:
[ 7.5% {853ms} TextComposer>>composeAllLines
[ 7.3% {830ms} TextComposer>>composeOneLine
[ 6.1% {693ms} TextComposer>>composeAllRectangles:
[ 5.8% {659ms} TextComposer>>composeEachRectangleIn:
[ 4.9% {557ms} CompositionScanner>>composeFrom:inRectan...leftSide:rightSide:
[ 2.3% {261ms} CompositionScanner>>setStopConditions
[ 2.3% {261ms} CompositionScanner>>setFont
      | | 2.7% {307ms} TextMorphForEditView>>handleInteraction:fromEvent:
      | | |2.7% {307ms} TextMorphForEditView(TextMorph)>>handleInteraction:fromEvent:
      | | | 2.1% {239ms} TextMorphForEditView>>updateFromParagraph
      | | 2.5% {284ms} TextMorphForEditView>>updateFromParagraph
      | | 2.1% {239ms} PluggableTextMorph(ScrollPane)>>setScrollDeltas
      | 2.6% {296ms} MouseOverHandler>>processMouseOver:
      | 2.4% {273ms} HandMorph>>handleEvent:
      | 2.3% {261ms} HandMorph>>sendMouseEvent:
      | 2.3% {261ms} HandMorph>>sendEvent:focus:clear:
      | 2.3% {261ms} PasteUpMorph(Morph)>>processEvent:
      | 2.3% {261ms} PasteUpMorph>>processEvent:using:
      | 2.3% {261ms} PasteUpMorph(Morph)>>processEvent:using:
      | 2.3% {261ms} MorphicEventDispatcher>>dispatchEvent:with:
      | 2.2% {250ms} MorphicEventDispatcher>>dispatchDefault:with:
    33.9% {3853ms} WorldState>>interCyclePause:
      33.8% {3842ms} Delay>>wait
        33.8% {3842ms} primitives

33.8% {3842ms} Delay>>wait
33.2% {3774ms} GrafPort(BitBlt)>>displayString:from:to:at:strikeFont:kern:
2.4% {273ms} OrderedCollection>>do:
2.3% {261ms} GrafPort>>copyBits

    old +286,028 bytes
    young -47,800 bytes
    used +238,228 bytes
    free -238,228 bytes

    full 0 totalling 0ms (0.0% uptime)
    incr 784 totalling 307ms (3.0% uptime), avg 0.0ms
    tenures 4 (avg 196 GCs/tenure)
    root table 0 overflows


 - 13125 tallies, 13138 msec.

100.0% {13138ms} PasteUpMorph>>doOneCycle
  100.0% {13138ms} WorldState>>doOneCycleFor:
    90.5% {11890ms} WorldState>>doOneCycleNowFor:
      |81.7% {10734ms} HandMorph>>processEvents
      | |81.3% {10681ms} HandMorph>>handleEvent:
      | | 78.6% {10326ms} HandMorph>>sendKeyboardEvent:
      | | |78.6% {10326ms} HandMorph>>sendEvent:focus:clear:
      | | | 78.5% {10313ms} HandMorph>>sendFocusEvent:to:clear:
      | | | 78.5% {10313ms} TextMorphForEditView(Morph)>>handleFocusEvent:
      | | | 78.5% {10313ms} TextMorphForEditView(Morph)>>handleEvent:
      | | | 78.5% {10313ms} KeyboardEvent>>sentTo:
      | | | 78.5% {10313ms} TextMorphForEditView(TextMorph)>>handleKeystroke:
      | | | 78.4% {10300ms} TextMorphForEditView>>keyStroke:
      | | | 78.2% {10274ms} TextMorphForEditView(TextMorph)>>keyStroke:
      | | | 75.2% {9880ms} TextMorphEditor>>readKeyboard
      | | | |75.2% {9880ms} TextMorphEditor(ParagraphEditor)>>readKeyboard
      | | | | 75.0% {9854ms} TextMorphEditor>>zapSelectionWith:
[74.1% {9735ms} MultiNewParagraph(NewParagraph)>>replaceFrom:to:with:displaying:
[ 73.9% {9709ms} MultiNewParagraph(NewParagraph)>>recomposeFrom:to:delta:
[ 73.9% {9709ms} MultiNewParagraph(NewParagraph)>>composeLinesFrom:to:delta:into:priorLines:atY:
[ 73.9% {9709ms} TextComposer>>composeLinesFrom:to:del...ner:wantsColumnBreaks:
[ 73.8% {9696ms} TextComposer>>composeAllLines
[ 73.6% {9670ms} TextComposer>>composeOneLine
[ 72.6% {9538ms} TextComposer>>composeAllRectangles:
[ 72.2% {9486ms} TextComposer>>composeEachRectangleIn:
[ 71.5% {9394ms} CompositionScanner>>composeFrom:inRectan...leftSide:rightSide:
[ 66.3% {8710ms} CompositionScanner(CharacterScanner)>>scanCharactersFrom:to...stopConditions:kern:
[ |66.2% {8697ms} CompositionScanner(CharacterScanner)>>basicScanCharactersFr...stopConditions:kern:
[ | 50.8% {6674ms} StrikeFontSet>>widthOf:
[ | |23.1% {3035ms} StrikeFontSet>>fontOf:
[ | | |19.9% {2614ms} StrikeFontSet>>fontOf:ifAbsent:
[ | | | |12.5% {1642ms} primitives
[ | | | |7.4% {972ms} Character>>leadingChar
[ | | |3.2% {420ms} primitives
[ | |19.5% {2562ms} StrikeFont>>widthOf:
[ | | |16.4% {2155ms} primitives
[ | | |3.1% {407ms} Character>>charCode
[ | |8.2% {1077ms} primitives
[ | 15.5% {2036ms} primitives
[ 3.0% {394ms} CompositionScanner>>setStopConditions
[ 3.0% {394ms} CompositionScanner>>setFont
[ 2.8% {368ms} CompositionScanner(CharacterScanner)>>setFont
      | | | 3.0% {394ms} TextMorphForEditView>>handleInteraction:fromEvent:
      | | | 3.0% {394ms} TextMorphForEditView(TextMorph)>>handleInteraction:fromEvent:
      | | | 2.0% {263ms} TextMorphForEditView>>updateFromParagraph
      | | 2.5% {328ms} MouseOverHandler>>processMouseOver:
      | | 2.3% {302ms} HandMorph>>handleEvent:
      | | 2.2% {289ms} HandMorph>>sendMouseEvent:
      | | 2.2% {289ms} HandMorph>>sendEvent:focus:clear:
      | | 2.2% {289ms} PasteUpMorph(Morph)>>processEvent:
      | | 2.2% {289ms} PasteUpMorph>>processEvent:using:
      | | 2.2% {289ms} PasteUpMorph(Morph)>>processEvent:using:
      | | 2.2% {289ms} MorphicEventDispatcher>>dispatchEvent:with:
      | | 2.1% {276ms} MorphicEventDispatcher>>dispatchDefault:with:
      |8.8% {1156ms} WorldState>>displayWorldSafely:
      | 8.8% {1156ms} PasteUpMorph>>displayWorld
      | 8.8% {1156ms} PasteUpMorph>>privateOuterDisplayWorld
      | 8.8% {1156ms} WorldState>>displayWorld:submorphs:
      | 8.6% {1130ms} WorldState>>drawWorld:submorphs:invalidAreasOn:
      | 8.5% {1117ms} FormCanvas(Canvas)>>fullDrawMorph:
      | 8.5% {1117ms} FormCanvas(Canvas)>>fullDraw:
      | 8.5% {1117ms} SystemWindow(Morph)>>fullDrawOn:
      | 8.4% {1104ms} SystemWindow(Morph)>>drawSubmorphsOn:
      | 8.4% {1104ms} FormCanvas(Canvas)>>fullDrawMorph:
      | 8.4% {1104ms} FormCanvas(Canvas)>>fullDraw:
      | 8.4% {1104ms} PluggableTextMorph(Morph)>>fullDrawOn:
      | 8.3% {1090ms} PluggableTextMorph(Morph)>>drawSubmorphsOn:
      | 8.3% {1090ms} FormCanvas(Canvas)>>fullDrawMorph:
[8.3% {1090ms} FormCanvas(Canvas)>>fullDraw:
[ 8.3% {1090ms} TransformMorph(Morph)>>fullDrawOn:
[ 8.0% {1051ms} TransformMorph>>drawSubmorphsOn:
[ 8.0% {1051ms} FormCanvas(Canvas)>>fullDrawMorph:
[ 8.0% {1051ms} FormCanvas(Canvas)>>fullDraw:
[ 8.0% {1051ms} TextMorphForEditView(Morph)>>fullDrawOn:
[ 8.0% {1051ms} FormCanvas(Canvas)>>drawMorph:
[ 8.0% {1051ms} FormCanvas(Canvas)>>draw:
[ 8.0% {1051ms} TextMorphForEditView(TextMorph)>>drawOn:
[ 8.0% {1051ms} FormCanvas>>paragraph:bounds:color:
[ 8.0% {1051ms} MultiNewParagraph>>displayOn:using:at:
[ 7.9% {1038ms} MultiDisplayScanner>>displayLine:offset:leftInRun:
[ 6.5% {854ms} StrikeFontSet>>displayString:on:from:to:at:kern:baselineY:
[ 2.2% {289ms} GrafPort>>copyBits
    9.5% {1248ms} WorldState>>interCyclePause:
      9.4% {1235ms} Delay>>wait
        9.4% {1235ms} primitives

17.0% {2233ms} StrikeFont>>widthOf:
15.6% {2050ms} CharacterBlockScanner(CharacterScanner)>>basicScanCharactersFr...stopConditions:kern:
13.0% {1708ms} StrikeFontSet>>fontOf:ifAbsent:
9.4% {1235ms} Delay>>wait
8.5% {1117ms} StrikeFontSet>>widthOf:
7.7% {1012ms} Character>>leadingChar
3.5% {460ms} Character>>charCode
3.2% {420ms} StrikeFontSet>>fontOf:
2.3% {302ms} GrafPort>>copyBits

    old +170,048 bytes
    young -32,136 bytes
    used +137,912 bytes
    free -137,912 bytes

    full 0 totalling 0ms (0.0% uptime)
    incr 3101 totalling 1,220ms (9.0% uptime), avg 0.0ms
    tenures 2 (avg 1550 GCs/tenure)
    root table 0 overflows
Eddie Cottongim   
08-30-05 03:13   
Analysis of the above: (the call tree is a bit hard to read because spaces got compressed, sorry)

     33.8% into Delay,
     Note: 0000005% into CharacterScanner basicScanCharactersFr...stopConditions:kern: (hard to tell exactly since it doesn't take enough time to show up, i'm estimating from its callers)
     Most remaining time going into printing

     9.4% into Delay
     66.2% into CharacterScanner basicScanCharactersFr...stopConditions:kern:

This is why I think basicScan... performance has gone way downhill and the most likely cause I can see is the failing primitive which is not failing (in this case at least) in 3.7.
08-30-05 04:12   
Clearly, this is the culprit (from the 3.8 trace):

  50.8% {6674ms} StrikeFontSet>>widthOf:

StrikeFont>>widthOf: is (or at least used to be) an EXTREMELY fast method but the implementation in StrikeFontSet first computes the font for the character

  23.1% {3035ms} StrikeFontSet>>fontOf:

accounting for about half of the time for widthOf: and then computing the actual width via

  19.5% {2562ms} StrikeFont>>widthOf:

which is roughly the other half of the time.
08-30-05 04:34   
I added some confusion (and have misread the tally result). The first tally I took was in Nihongo7 image which is similar to Squeak3.8gamma. In that image the scanner wasn't that dominant. And you can see it in 3.8g-6527 image.

In 3.8 release, the choppy behavior (bunch of characters get inserted after several seconds) go away when I move the mouse cursor while I press a keyboard key. That lead me think that some priority issue was related.
Eddie Cottongim   
08-30-05 14:46   
widthOf: would not get called at all if the primitive was working. Its called from the primitive fallback code.

I agree though that in some other places its not very fast and causes problems.

I saw the choppy updates too Ohshima, but I didn't think about them too much, figuring they were just deferred updates or something like that. That may be a separate problem.
07-21-13 18:14   
During the change from 3.7 to 3.8 somebody decided to change the class shape of a crucial VM-known class and not tell anybody. Since then primitive 103, nominally the core string measuring primitive, has not been actually able to work. That's 9 years…
I'm not sure how on earth this managed to happen. At least by this stage prim 103 was not still the main character displaying code or we would really have been screwed!

Currently the assorted scanners use the prim method (and thus waste time trying to use the prim) "basicScanCharactersFrom: startIndex to: stopIndex in: sourceString rightX: rightX stopConditions: stops kern: kernDelta" to measure and detect stop conditions and then the split string is sent to the bitbltplugin method "basicDisplayString: aString from: startIndex to: stopIndex at: aPoint strikeFont: font kern: kernDelta" for actual rendering - unless of course other rendering plugins get involved.

So, what to do? We have a split in image format support right now, with the plain interpreter able to still run old images at least back to 2.8. Up to 3.7 the prim will work as expected and thereafter fail. Post-closure image format split, we know the prim cannot work and so could change it to repair the situation in the stack/cog vm code without penalty. The plain interp would then have an issue about whether to keep old-style, or go new-style, or even to try to detect the right thing to do.
08-08-13 20:08   
To further confuse the issue, the prim 103 backup code in Squeak 4.4 sends #widthAndKernedWidthOfLeft:right:into: which purports to fill the last argument (a 2 element array) with the normal width of the character in the first arg and the width if kerned with the character in the second arg.
As part of that, kerningLeft:right: is used to work out the difference that the kerning makes - and the only implementation always returns 0. Sigh.
09-01-13 17:33   
Fonts and scanning are astonishingly fucked up.

WRT kerning via the widthAndKernedWidthOfLeft:right:into: we have a whole raft of font classes to play with; several of which seem pretty much defunct.
AbstractFont>widthAndKernedWidthOfLeft:right:into: sends #kerningLeft:right: - the default implementation is 0, i.e. no kerning. This is actually used by
AbstractFont (duh)
FixedFaceFont (which I would have expected to pass it on to its baseFont like so many other messages?)
StrikeFont (and FormSetFont & HostFont)
TTCFont (which surprised me)

Only LogicalFont & FreeTypeFont actually do anything with the kerning values. FT2Face uses #kerningLeft:right: but it isn't a subclass of AbstractFont, has no implementation of widthAndKernedWidthOfLeft:right:into: and I have no idea what it is up to.

I propose to make the default widthAndKernedWidthOfLeft:right:into: just stick the width of the character into both slots of the passed in array and merge the kerning calculation into the version for FreeTypeFont. That will of itself slightly sped up life for StrikeFont etc, which is a small improvement. (NB, code change works ok)
 Clearly, the ideal would be to completely avoid the whole kerning thing for fonts that don't support pair-kerning and then repair the primitive to handle that case efficiently. Fonts that can do pair-kerning would avoid the prim and do whatever they need.

How to discriminate the cases? Well we could use liberal applications of #isKindOf: of course, just to keep the ludicrous overuse of that obscenity up to scratch. We could ask the installed font #doesPairKerning and split cases based on that. We could delegate the actual scanning to the font class(es).

(NB just realised that MultiCharacterScanner uses identical code, including trying to call the broken primitive. Is it plausible that it would have a 'sourceString that is a single-byte String? Surely that would break things?)
Ugh and just look at the assorted implementations of the messages under #scanSelector, which is how widestring scanning gets mangled. #isKindOf: in the middle of scanning text? Sigh.
09-04-13 19:18   
Current opinion on prim 103 - leave it in the VM build, stop even bothering to call it in images but allow older images (like the fairly widely used Scratch image) to continue to rely upon it.

When a hopefully successful refactoring of scanning/displaying/measuring is complete, consider options for a new primitive or two to support the new code.
09-05-13 02:24   
Progress report:
by testing the font for #isPairKerningCapable (purely as a stopgap until better factoring is made) we can split between scanning for the two variants of simple and pair-kerning scans. The simple case can use exactly the old code that used to backup the defunct prim 103. The kerning code is a slightly better commented version of the recent 'backup code'.
Both cases had to be implemented in both CharacterScanner and MultiCharacterScanner.
09-05-13 17:15   
Comparing scanner classes -

CharacterScanner vs MultiCharacterScanner
MCS adds several iVars, most of which ought to be moved down to MultiCompositionScanner and several could be dropped altogether.

MCS>addCharToPresentation: - seems unused, two senders but neither used. Not in CS.
CS>isBreakableAtIndex: - unused, MCS version is isBreakableAt:in:in:
CS>measureString:inFont:from:to: slightly different to MCS - changed baseline & kern.
MCS>registerBreakableIndex absent in CS, used for japanese etc scanning
MCS>removeLastCharFromPresentation seems unused as per addCharToPresentation.
MCS>scanJapaneseCharactersFrom:to:in:rightX:stopConditions:kern: differs only slightly from CS; should they be identical (would add isBreakableAt:in:in: etc)
MCS>scanMultiCharactersCombiningFrom:to:in:rightX:stopConditions:kern: unused?
MCS>scanMultiCharactersFrom:to:in:rightX:stopConditions:kern: differs slightly from CS; adds pair kerning support and some HostFont (unused?) support.
MCS>setConditionArray: is different. Only actually used for Justified->paddedSpace or nil->NilCondition. What is intent?
MCS>widthOf:in: seems *almost* unused. Is class CombinedChar still needed? Only plausible usage is in UnicodeCompositionStream>nextPut: which is a small part of readKeyboard handling.

CanvasCharacterScanner vs MultiCanvasCharacterScanner
MCCS drops iVar defaultTextColor

CCS>convertToCurrentVersion:refStream: missing in MCCS
CCS>defaultTextColor[:] missing in MCCS
MCCS>setFont slightly different, uses added baselineY iVar

CharacterBlockScanner vs MultiCharacterBlockScanner
characterBlockAtPoint:index:in: is a good example of how having two parallel trees causes problems; CBS got 'fixed' differently to MCBS
MCBS adds scanMultiCharactersCombiningFrom:to:in: rightX:stopConditions:kern: which appears unused

(Updated 20130925 to reflect changes made)
CompositionScanner vs MultiCompositionScanner
MCompS adds addCharToPresentation: which is only sent by MCS>scanMultiCharactersCombiningFrom:to:in:rightX:stopConditions:kern: (unused?) and itself

CompS has canComputDefaultLineHeight (duh) and canComputeDefaultLineHeight

MCompS columnBreak differs
(now identical)

MCompS added 'baselineY' ivar, unused now. (removed by nice)

MCompS composeFrom:inRectangle:firstLine:leftSide:rightSide: is more complex, adds presentationLine usage
(presentation/Line stuff removed)
- adds 'firstDestX' ivar setting; that is only used in scanJapanese../scanMultiCharactersCombining.../scanMultiCharactersFrom… as part of a simple test of some sort.
- minor difference in ordering of section 'lineHeight := … spaceCount :=..' that might implicate setStopConditions in changing lastIndex ?
- tiny change in final loop exit style. Seems to be functionally identical.

#composeLine:fromCharacterIndex:inParagraph: now identical (removed spurious baselineY)

cr/crossedX/endOfRun adds presentationLine stuff (no longer)
#crossedX is a bit different but seems functionally equivalent

MCompS adds getPresentation/getPresentationLine/presentation/presentationLine/removeLastCharFromPresentation (no longer)

isBreakableAt:in:/registerBreakableIndex - quite different but seem to do ordinary things. Uses text environment slightly to test whether we can break at current char, could be cleaned up somewhat. Actually looks like it could be removed except in scanJapaneseCharacters…. (done - nice managed to add in a decent #space method and fix up the stopCOnditions stuff)
Oddness - see accesses of breakableIndex/breakAtSpace - looks like we currently can't ever get into any of the interesting code in crossedX, which is an issue.

placeEmbeddableObject: adds presentationLine stuff (no longer, now identical)

rightX/setFont adds breakAtSpace stuff
MCompS loses space (but is handled by that nasty isBreakableAt… code)

DisplayScanner vs MultiDisplayScanner
displayLine:offset:leftInRun: - DS defers displayString: to font MDS directs to bitblt, which seems wrong (changed, no problems so far)
also has to sea leith emphasisCode details directly.
displayLines:in:clippedBy: almost identical except for tiny ordering changes? Both display via font…
MDS initializeFromParagraph: clippedBy: loses fix for 1bpp Display; both assume 1bpp font glyphs
MDS adds isBreakableAt:in:in:/presentationText:/scanMultiCharactersCombiningFrom:to:in:rightX:topConditions:kern: (unsent)
MDS placeEmbeddedObject: adds change to baselineY ?
fillBlt ivar is only ever set by text:textStyle:foreground:background:fillBlt:ignoreColorChanges: (where it is set to the GrafPort that is the only sender of the message) and in displayLines:in:clippedBy: (where it is set to a copy of the bitBLt ivar )

-- No more uses of DisplayScanner, no apparent problems.

SegmentScanner has no Multi equivalent

09-23-13 19:38   
{from email Re: [squeak-dev] fonts, characterscanners and dead primitive 103 tim rowledge 2013-09-20 8:08pm pst}
After side-by-side comparing the old scanners with the new 'Multi' scanners, my conclusion is that there is *very* little difference and we really ought to be able to go back to a single set of classes. Which, I claim, would be nice, since we've already visibly suffered from the obvious side-effect of having two trees as bug fixes get added to only one part.

So far as I could tell the only substantive difference relates to the use of the presentation/presentationLine ivars which seems to be not very important (ref Yoshiki's message 8 sept) and even seems to be mostly inactive (ref nice's message regarding Multilingual-nice.116 same date). It would be really nice to get a solid decision on whether it is still wanted, or should be removed, or if it needs some fixes that can be provided.

I'm puzzled by quite a few things I've discovered.

a) There are {language}environment classes and encoding classes. There is #isBreakableAt:in: implemented in both but seemingly unused in the encoding classes because it is just plain broken there. Should it be removed from the encoders? In the language environment classes it is implemented to return true for space and cr by default, but space, cr & lf in Latin1 and Latin2. Is that as expected?

b) MultiCharacterScanner>setConditionArray: cuts out the handling of #space - any ideas why?

c) as previously mentioned MultiCharacterScanner>addCharToPresentation: is currently unused, apparently because of issues with Unicode & #scanMultiCharactersCombiningFrom:to:in:rightX:stopConditions:kern: - do we have any decent hope of a fix?

d) MultiCanvasCharacterScanner>setFont uses baselineY differently to its CanvasCharacterScanner equivalent; why?

e) TextComposer>composeLinesFrom:to:delta:into:…. differs minimally from MultiTextComposer>multiComposeLinesFrom:to:delta:into:…. and is the only sender of #canComputeDefaultLineHeight. What is the intent? Is this just a bug fix added in one place and not the other?

f) one of the oddest - DisplayScanner>displayLine:offset:leftInRun: passes the displayString:.. to the font (which I see as good) but MultiDisplayScanner uses the bitlblt instead which seems quite wrong.
09-23-13 23:42   
No direct users of (Multi)CharacterScanner

- Canvas>paragraph2:bounds:colour: - no senders
- RemoteCanvas>paragraph:bounds:color: - should it be using MultiCanvasCharacterScanner ?
MultiCanvasCharacterScanner - no users

- NewParagraph>characterBlockAtPoint: & characterBlockForIndex: both use MCBS & CBS depending upon whether string is wide or not.
- Paragraph>characterBlockAtPoint: & characterBlockForIndex: both use just CharacterBlockScanner
- TextOnCurve>characterBlockAtPoint:

- Paragraph>replaceFrom:to:with:displaying:
- TextComposer> composeLinesFrom:to:delta:into:priorLines:atY:textStyle:text:container:wantsColumnBreak:
- MultiTextComposer > multiComposeLinesFrom:to:delta:into:priorLines:atY:textStyle:text:container:wantsColumnBreak:
- Paragraph>composeAll (why? seems like a mixup with NewParagraph)

- GrafPort>displayScannerFor:foreground:background: ignoreColorChanges: (only sent by FormCanvas>paragraph:bounds:color:)
- GrafPort>displayScannerForMulti:foreground:background: ignoreColorChanges: ( only sent by FormCanvas>paragraph3:bounds:colour: which is only sent by TextMorph>drawOnTest: which is unsent.)
Both choose DisplayScanner or MultiDisplayScanner depending on paragraph class and string details.

also used in Paragraph>displayLines:affectedRectangle:
09-24-13 01:14   
adds presentationText ivar but this is only used in
- displayOnTest:using:at: which is only sent in FormCanvas>paragraph3:… which is already up for removal
- multiComposeLinesFrom:to:…. which sets it from the composer but does not use it
- presentationText which simply returns it and is sent only by GrafPort>displayScannerForMulti… which is also only sent by FormCanvas>paragraph3:…
adds presentationLines ivar which has a similar story.
The only 'remaining' method in the class is identical to the superclass version.
We can remove this class.
09-28-13 00:53   
MultiTextComposer & MultiNewParagraph are now removed, hurrah!

It appears CompositionScanner & MultiCompositionScanner are interchangeable; a quick hack to remove use of CompositionScanner in Paragraph causes no problems.

Improvements to MCS have also been made to reduce differences to CS. See revised notes above .
09-30-13 22:26   
As of today (30 Sept 2013) we have a quite reasonably cleaned up set of scanner classes in a single class tree. Thus we can avoid future clashes of a find in one place and not the other, which is nice. (and substantially due to 'nice' ;-) )

We're not wasting time trying to use a primitive that can't succeed. We have a moderately acceptable routine for deciding which variant of scanning (byte string, widestring, japanese etc), which may allow a new prim for the common Latin1 languages.

Next steps will be in a new Mantis entry.
09-30-13 23:27   
Look at it and tell me it isn't an improvement...