3DICC / SqueakNim

Commit 70473583735e260a43dec1ea4fd47cacf0fdf306

Authored by Göran Krampe
1 parent 5d233627

First sample and instructions

Inline Side-by-side
Showing 7 changed files with 258 additions and 0 deletions
README.md
  View file @7047358
  1 +# NimSqueak - Squeak and Nim interop via FFI
  2 +
  3 +Nim can produce dynamically loaded libraries that follow the C conventions. Squeak/Pharo has over the years developed several FFI mechanisms, and we can also make VM plugins, but for now this code uses the old "true and tested" Squeak FFI that Terf uses today for all its OpenGL calls.
  4 +
  5 +I also wrote an article when I experimented: http://goran.krampe.se/2014/11/03/squeak-to-nim/
  6 +
  7 +In the testlib directory we have the files from that article. See README.md there for a concrete rundown on how to generate the .st file.
  8 +
  9 +## Squeak FFI
  10 +
  11 +What follows is a text written by Sean DeNigris describing the FFI. You can also see the code in the client image of course, with tests and class comments etc. And then we have the actual SqueakFFIPrims plugin (VM level code) that you can find in the terf-client-vm repo.
  12 +
  13 +## How does FFI work?
  14 +
  15 +Technically what happens is:
  16 +
  17 +* you define what the interface is - the parameters, types etc.
  18 +* when you make the call, the FFI logic assembles the data from the Squeak
  19 +Objects into the proper structures according to the routine calling
  20 +conventions for your architecture, and of course manages the return values.
  21 +So no magic but perhaps just a little assembler in the plugin to properly
  22 +deal with all the registers and condition flags.
  23 +
  24 +How do I use it?
  25 +
  26 +1. make a method (whose structure is similar to a named primitive method)
  27 +
  28 +Example:
  29 +
  30 +```
  31 + system: aString
  32 + "Some kind of comment"
  33 +
  34 + <apicall: long 'system' (char*) module: 'libSystem.dylib'>
  35 + ^self externalCallFailed.
  36 +```
  37 +
  38 +`system: aString` above is the Smalltalk signature. Each named parameter is separated with a keyword. Smalltalk messages are normally camelCase, so if a nim proc was called `drink(bottles: int, brand: string)` then we could typically turn that into `drinkBottles: bottles brand: brand`. The unique (per class) so called `selector` (=name) of such a method in Smalltalk would be `drinkBottles:brand:`. Thus Nim overloading would not work (but we don't need that here).
  39 +
  40 +Then comes the so called pragma that describes the C function to call:
  41 +
  42 +```
  43 + <apicall: long 'system' (char*) module: 'libSystem.dylib'>
  44 +```
  45 +Function specification:
  46 + - should be the first line in the method
  47 + - enclosed in angle brackets: < > containing:
  48 + 1. Calling Convention, either apicall: (Pascal convention) or cdecl: (C convention)
  49 + - Mac - use either one
  50 + - Unix - use cdecl
  51 + - Windows - use apical
  52 + 2. Return Type (see types)
  53 + 3. External Function Name (literal string)
  54 + 4. Argument Types (a literal array)
  55 + 5. Module - "module: " + [filename of the external library (literal string)] (see below).
  56 +
  57 +```
  58 + ^self externalCallFailed.
  59 +```
  60 +
  61 +Failure handler
  62 + - normal smalltalk code
  63 + - executed if the linking to or calling the external function fails
  64 + - API calls don't know how to communicate failure like Squeak primitives do, so:
  65 + - it does not tell you whether the external function succeeded
  66 + - the most common code is simply '^self externalCallFailed.'
  67 +
  68 +Argument Types
  69 + - must be names of ExternalTypes, either:
  70 + - atomic types (see ExternalType class>>initializeFFIConstants and ExternalType class>>initializeAtomicTypes):
  71 + void
  72 + bool
  73 + byte (unsigned)
  74 + sbyte (signed)
  75 + ushort (16-bit unsigned)
  76 + short (16-bit signed)
  77 + ulong (32-bit unsigned)
  78 + long (32-bit signed)
  79 + ulonglong (64-bit unsigned)
  80 + longlong (64-bit signed)
  81 + char (unsigned)
  82 + schar (signed)
  83 + float (single-precision float)
  84 + double (double-precision float)
  85 +
  86 +Structure Types [4]
  87 + - subclass of ExternalStructure
  88 + - class>>fields that returns an array of field descriptions (see below)
  89 + - Example:
  90 + fields
  91 + ^#((red 'byte')(green 'byte')(blue 'byte'))
  92 + - class>>initialize which includes "self defineFields" (which must be
  93 +called before using the class)
  94 + - refer to as MyExternalStructure* (to indicate that the argument or return is a pointer to that structure)
  95 +
  96 +Field description [4]
  97 + - 2-element array (or three but that does something else, I'm not sure what):
  98 + - first element is the field name
  99 + - second is the type
  100 +
  101 +Module Name
  102 +- depends on the platform
  103 + - Mac
  104 + - pre Snow Leopard: flexible, can eliminate leading lib or extension e.g. 'libc.dylib' becomes 'libc', 'c.dylib', or 'c'
  105 + - Snow Leopard
  106 + - file name must be exact including extension (unless Info.plist is altered as in 'Library Location' below)
  107 + - With pre-mach-o VMs
  108 + - For Classic applications, use 'InterfaceLib'
  109 + - For Carbon libs, use 'CarbonLib'
  110 +
  111 +Module Location - where the external library file lives
  112 + - depends on the platform
  113 + - Mac
  114 + - pre Snow Leopard
  115 + - checks VM path and common library paths
  116 + - Snow Leopard
  117 + - only looks in VM bundle's Resources file, you must either [5]:
  118 + - store all external libraries there
  119 + - ln -s path/to/library path/to/VM/Resources/library_name
  120 + - Change the VM's Info.plist "SqueakPluginsBuiltInOrLocalOnly" key from "true" to "false."
  121 +Caveats
  122 + - security
  123 + - malicious users could call arbitrary functions in the OS e.g. "format c:" from "system.dll" [7]
  124 + - VMs do not protect against buffer overflow from bad parameters [8]:
  125 + "this would require an attacker to execute arbitrary Smalltalk
  126 + code on your server. Of course if they can do that they own you
  127 + anyway, especially if you allow FFi or use the OSProcess plugin" - John McIntosh
  128 +
  129 +* difficulty
  130 + - if you make a mistake you'll not drop into the debugger but Squeak will
  131 +just crash [2]
  132 + - If you crash Squeak when it is running the garbage collector, then you
  133 +know your FFI code is leaking bits into object memory [2]
  134 +
  135 +What do I need to use FFI with Squeak?
  136 +
  137 +You need the FFI plugin, which is included with most VM's as of Squeak 3.6
  138 +or so.
  139 +
  140 +You can also build the plugin yourself. See VMMaker.
  141 +
  142 +References:
  143 +[1] http://wiki.squeak.org/squeak/1414
  144 +[2] http://wiki.squeak.org/squeak/2424
  145 +[3] http://wiki.squeak.org/squeak/5716
  146 +[4] http://wiki.squeak.org/squeak/2426
  147 +[5]
  148 +http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td85608.html
  149 +[6] http://wiki.squeak.org/squeak/5846
  150 +[7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073
  151 +[8] http://forum.world.st/Security-td99624.html#a99635:
  152 +
... ...
lib/libnimrtl.so 0 → 100755
View file @7047358
No preview for this file type
testlib/Nim-Tests.st 0 → 100644
  View file @7047358
  1 +TestCase subclass: #FFINimTest instanceVariableNames: 'nim' classVariableNames: '' poolDictionaries: '' category: 'Nim-Tests'! !FFINimTest methodsFor: 'as yet unclassified' stamp: 'gk 11/2/2014 15:08'! setUp nim := FFINimTestLibrary ! ! !FFINimTest methodsFor: 'as yet unclassified' stamp: 'gk 11/3/2014 01:11'! testBasics "getting an int back" self assert: (nim ffiHello = 42). "sending ints and getting back too" self assert: ((nim ffiAdd: 3 with: 4) = 7). "getting a string" self assert: (nim ffiFoo = 'hey'). "sending a string and getting an int back" self assert: ((nim ffiLength: 'abc') = 3) . ! ! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! FFINimTest class instanceVariableNames: ''! !FFINimTest class methodsFor: 'as yet unclassified' stamp: 'gk 11/3/2014 01:12'! stress "self stress" | rand | rand := Random new. ^[10000000 timesRepeat: [ FFINimTestLibrary ffiHello; ffiFoo; ffiAdd: rand next with: rand next; ffiConcat: 'abc' with: '123'; ffiLength: 'abcde']] timeToRun! ! ExternalLibrary subclass: #FFINimTestLibrary instanceVariableNames: 'nim' classVariableNames: '' poolDictionaries: '' category: 'Nim-Tests'! "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "! FFINimTestLibrary class instanceVariableNames: ''! !FFINimTestLibrary class methodsFor: 'primitives' stamp: 'gk 11/2/2014 13:05'! ffiAdd: a with: b "self ffiAdd: 13 with: 29" <cdecl: long 'add' (long long) module: 'testlib'> ^self externalCallFailed! ! !FFINimTestLibrary class methodsFor: 'primitives' stamp: 'gk 11/2/2014 22:49'! ffiConcat: a with: b "self ffiConcat: 'a' with: 'b' " <cdecl: char* 'concat' (char* char*) module: 'testlib'> ^self externalCallFailed! ! !FFINimTestLibrary class methodsFor: 'primitives' stamp: 'gk 11/3/2014 01:10'! ffiFoo "self ffiFoo" <cdecl: char* 'foo' () module: 'testlib'> ^self externalCallFailed! ! !FFINimTestLibrary class methodsFor: 'primitives' stamp: 'gk 11/2/2014 13:06'! ffiHello "self ffiHello" "long is int in Nim" <cdecl: long 'hello' () module: 'testlib'> ^self externalCallFailed! ! !FFINimTestLibrary class methodsFor: 'primitives' stamp: 'gk 11/3/2014 01:10'! ffiLength: x "self ffiLen: 'hey' " <cdecl: long 'length' (char*) module: 'testlib'> ^self externalCallFailed! ! !FFINimTestLibrary class methodsFor: 'accessing' stamp: 'gk 10/31/2014 21:45'! moduleName "Use the fully qualified VM name so we ensure testing loading a library" ^'testlib'! !
0 2 \ No newline at end of file
... ...
testlib/README.md 0 → 100644
  View file @7047358
  1 +The file to generate given testlib.nim would be `Testlib.st`. This is the class old "chunk format" in which the content is separated by `!` marks, in a rather oddish way, but anyway.
  2 +
  3 +Let's go through it:
  4 +
  5 +```
  6 +'From Nim on 18 February 2015 at 11:15:50 pm'!
  7 +```
  8 +Above first chunk is just a String literal with a timestamp, that chunk can be omitted.
  9 +
  10 +```
  11 +ExternalLibrary subclass: #Testlib
  12 + instanceVariableNames: ''
  13 + classVariableNames: ''
  14 + poolDictionaries: ''
  15 + category: 'Nim'!
  16 +```
  17 +Then follows the class declaration. Obviously we need to make sure it says `Testlib` and the category will equal the package in Squeak that this class will end up in. For now, let's hard code it to `Nim`.
  18 +
  19 +Then we can see that there is a "whitespace only" chunk, since the next part starts with `!`.
  20 +
  21 +```
  22 +!Testlib class methodsFor: 'primitives' stamp: 'gk 11/2/2014 13:05'!
  23 +```
  24 +This chunk says here follows methods for "Testlib class" (=class side) in method category "primitives" (we can hardcode it to that).
  25 +
  26 +```
  27 +ffiAdd: a with: b
  28 + "self ffiAdd: 13 with: 29"
  29 +
  30 + <cdecl: long 'add' (long long) module: 'testlib'>
  31 + ^self externalCallFailed!
  32 +
  33 +ffiConcat: a with: b
  34 + "self ffiConcat: 'a' with: 'b' "
  35 +
  36 + <cdecl: char* 'concat' (char* char*) module: 'testlib'>
  37 + ^self externalCallFailed!
  38 +
  39 +ffiFoo
  40 + "self ffiFoo"
  41 +
  42 + <cdecl: char* 'foo' () module: 'testlib'>
  43 + ^self externalCallFailed!
  44 +
  45 +ffiHello
  46 + "self ffiHello"
  47 +
  48 + "long is int in Nim"
  49 +
  50 + <cdecl: long 'hello' () module: 'testlib'>
  51 + ^self externalCallFailed!
  52 +
  53 +ffiLength: x
  54 + "self ffiLen: 'hey' "
  55 +
  56 + <cdecl: long 'length' (char*) module: 'testlib'>
  57 + ^self externalCallFailed! !
  58 +```
  59 +The above is one chunk per method, then an extra empty chunk on the last line to finish the method category "primitives".
  60 +
  61 +```
  62 +!Testlib class methodsFor: 'accessing' stamp: 'gk 2/18/2015 23:13'!
  63 +moduleName
  64 + ^'testlib'! !
  65 +```
  66 +Finally a chunk declaring another method category, and then a single method with the name of the shared library to load.
  67 +
... ...
testlib/Testlib.st 0 → 100644
  View file @7047358
  1 +ExternalLibrary subclass: #Testlib instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Nim'! !Testlib class methodsFor: 'primitives' stamp: 'gk 11/2/2014 13:05'! ffiAdd: a with: b "self ffiAdd: 13 with: 29" <cdecl: long 'add' (long long) module: 'testlib'> ^self externalCallFailed! ffiConcat: a with: b "self ffiConcat: 'a' with: 'b' " <cdecl: char* 'concat' (char* char*) module: 'testlib'> ^self externalCallFailed! ffiFoo "self ffiFoo" <cdecl: char* 'foo' () module: 'testlib'> ^self externalCallFailed! ffiHello "self ffiHello" "long is int in Nim" <cdecl: long 'hello' () module: 'testlib'> ^self externalCallFailed! ffiLength: x "self ffiLen: 'hey' " <cdecl: long 'length' (char*) module: 'testlib'> ^self externalCallFailed! ! !Testlib class methodsFor: 'accessing' stamp: 'gk 2/18/2015 23:13'! moduleName ^'testlib'! !
0 2 \ No newline at end of file
... ...
testlib/testlib.nim 0 → 100644
  View file @7047358
  1 +# We exercise some trivial types and how they map.
  2 +#
  3 +# SqueakFFI Nim
  4 +# ========= ======
  5 +# long => int
  6 +# char* => cstring
  7 +
  8 +import math
  9 +
  10 +# A single proc, returns an int. Since we are on 32 bits
  11 +# an int is 4 bytes (same size as pointer) and in Squeak FFI
  12 +# this is a long. The exportc pragma ensures that the exported
  13 +# name for this proc is exactly "hello" and not mangled.
  14 +proc hello*(): int {.exportc.} =
  15 + 42
  16 +
  17 +# Trivial, a Nim string can be sent as a cstring because they are
  18 +# automatically 0-terminated.
  19 +proc foo*(): cstring {.exportc.} =
  20 + "hey"
  21 +
  22 +# Not a problem taking int arguments, they are "long" in Squeak FFI.
  23 +proc add*(x, y:int): int {.exportc.} =
  24 + x + y
  25 +
  26 +# Just return the length of a cstring
  27 +proc length*(x: cstring): int {.exportc.} =
  28 + len(x)
  29 +
  30 +# Here we convert cstrings to Nim strings, concatenate and return.
  31 +proc concat*(x, y: cstring): cstring {.exportc.} =
  32 + $x & $y
0 33 \ No newline at end of file
... ...
testlib/testlib.nim.cfg 0 → 100644
  View file @7047358
  1 +--app:lib
  2 +--cpu:i386
  3 +--passC:"-m32"
  4 +--passL:"-m32"
  5 +-d:useNimRtl
... ...