jyapayne/nimgenEx
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial version

Browse source
This commit is contained in:
Ganesh Viswanathan 2017-11-07 18:14:33 -06:00
commit 777ae910c8
4 changed files with 535 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
nimcache
*.exe

91
README.md Normal file
View file

@ -0,0 +1,91 @@
Nimgen is a helper for [c2nim](https://github.com/nim-lang/c2nim/) to simpilfy and automate the wrapping of C libraries.
Nimgen can be used to automate the process of manipulating C files so that c2nim can be run on them without issues. This includes adding/removing code snippets, removal of complex preprocessor definitions that c2nim doesn't yet comprehend and recursively running on #include files.
__Installation__
Nimgen can be installed via [Nimble](https://github.com/nim-lang/nimble):
```> nimble install https://github.com/genotrance/nimgen```
This will download, build and install nimgen in the standard Nimble package location, typically ~/.nimble. Once installed, it can be run just like c2nim.
__Usage__
Nimgen is driven by a simple .cfg file that is read using the Nim [parsecfg](https://nim-lang.org/docs/parsecfg.html) module. The sections of the file are described further below.
```> nimgen package.cfg```
A Nimble package for a library that is wrapped with nimgen will have the following:-
* The .cfg file that tells nimgen what exactly to do
* Nimgen defined as a dependency defined in the .nimble file
* Steps within the .nimble file to download the source code that is being wrapped
This way, the library source code doesn't need to get checked into the Nimble package and can evolve independently.
Nimble already requires Git so those commands can be assumed to be present to download source from a repository. Mercurial is also suggested but depends on the user. Downloading arbitrary files depends on the OS. For Linux, wget/curl can be assumed. On Windows, [powershell](https://superuser.com/questions/362152/native-alternative-to-wget-in-windows-powershell) can be used.
__Capabilities & Limitations__
Nimgen supports compiling in C/C++ sources as well as loading in dynamic libraries at this time. Support for static libraries (.a, .lib) are still to come.
To see examples of nimgen in action, the nimssl library is a good example of C code getting compiled in whereas nimbass is an example of linking with a dynamic library.
Nimgen only supports the ```gcc``` preprocessor at this time. Support for detecting and using other preprocessors is TBD.
__Config file__
_[n.global]_
output = name of the Nimble project once installed, also location to place generated .nim files
filter = string to identify and recurse into library .h files in #include statements and exclude standard headers
_[n.include]_
List of all directories, one per line, to include in the search path. This is used by:-
* The preprocessor for #include files
* Nimgen to find #include files that are recursively processed
Nimgen also adds {.passC.} declarations into the generated .nim files for these include paths if compiling source files directly.
_[n.exclude]_
List of all directories or files to exclude from all parsing. If an entry here matches any portion of a file, it is excluded from recursive processing.
_[sourcefile]_
The following keys apply to library source code and help with generating the .nim files.
```recurse``` = find #include files and process them [default: false]
```preprocess``` = run preprocessor (gcc -E) on file to remove #defines, etc. [default: false] - this is especially useful when c2nim doesn't support complex preprocessor usage
```ctags``` = run ctags on file to filter out function definitions [default: false] - this requires the ctags executable and is an alternative to filter out preprocessor complexity
```defines``` = pulls out simple #defines of ints, floats and hex values for separate conversion [default: false] - works only when preprocess or ctags is used and helps include useful definitions in generated .nim file
```flags``` = flags to pass to the c2nim process in "quotes" [default: --stdcall]. --cdecl, --assumedef, --assumendef may be useful
Multiple entries for the all following keys are possible by appending any .string to the key. E.g. dynlib.win, compile.dir
```compile``` = file or dir of files of source code to {.compile.} into generated .nim
```dynlib``` = dynamic library to load at runtime for generated .nim procs
The following keys apply to library source code (before processing) and generated .nim files (after processing) and allow manipulating the files as required to enable successful wrapping.
```create``` = create a file at exact location with contents specified
```search``` = search string providing context for following prepend/append/replace directives
```prepend``` = string value to prepend into file at beginning or before search
```append``` = string value to append into file at the end or after search
```replace``` = string value to replace search string in file
__Feedback__
Nimgen is a work in progress and any feedback or suggestions are welcome. It is hosted on [GitHub](https://github.com/genotrance/nimgen) with an MIT license so issues, forks and PRs are most appreciated.

430
nimgen.nim Normal file
View file

@ -0,0 +1,430 @@
import nre
import os
import ospaths
import osproc
import parsecfg
import ropes
import sequtils
import streams
import strutils
import tables
var FILES: TableRef[string, string] = newTable[string, string]()
var DONE: seq[string] = @[]
var CONFIG: Config
var FILTER = ""
var OUTPUT = ""
var INCLUDES: seq[string] = @[]
var EXCLUDES: seq[string] = @[]
# ###
# Helpers
proc execProc(cmd: string): string =
var p = startProcess(cmd, options = {poStdErrToStdOut, poUsePath, poEvalCommand})
result = ""
var outp = outputStream(p)
var line = newStringOfCap(120).TaintedString
while true:
if outp.readLine(line):
result.add(line)
result.add("\n")
elif not running(p): break
var x = p.peekExitCode()
if x != 0:
echo "Command failed: " & $x
echo cmd
echo result
quit()
# ###
# File loction
proc getnimout(file: string): string =
var nimout = file.splitFile().name & ".nim"
if OUTPUT != "":
nimout = OUTPUT/nimout
removeFile(nimout)
return nimout
proc exclude(file: string): bool =
for excl in EXCLUDES:
if excl in file:
return true
return false
proc search(file: string): string =
if exclude(file):
return ""
result = file
if file.splitFile().ext == ".nim":
result = getnimout(file)
elif not fileExists(result):
var found = false
for inc in INCLUDES:
result = inc/file
if fileExists(result):
found = true
break
if not found:
echo "File doesn't exist: " & file
quit()
return result.replace(re"[\\/]", $DirSep)
# ###
# Loading / unloading
proc loadfile(file: string) =
if FILES.hasKey(file):
return
FILES[file] = readFile(file)
proc savefile(file: string) =
try:
if FILES.hasKey(file):
writeFile(file, FILES[file])
FILES.del(file)
except:
echo "Failed to save " & file
proc savefiles() =
for file in FILES.keys():
savefile(file)
# ###
# Manipulating content
proc prepend(file: string, data: string, search="") =
loadfile(file)
if search == "":
FILES[file] = data & FILES[file]
else:
let idx = FILES[file].find(search)
if idx != -1:
FILES[file] = FILES[file][0..<idx] & data & FILES[file][idx..<FILES[file].len()]
proc append(file: string, data: string, search="") =
loadfile(file)
if search == "":
FILES[file] &= data
else:
let idx = FILES[file].find(search)
let idy = idx + search.len()
if idx != -1:
FILES[file] = FILES[file][0..<idy] & data & FILES[file][idy..<FILES[file].len()]
proc freplace(file: string, pattern: string, repl="") =
loadfile(file)
if pattern in FILES[file]:
FILES[file] = FILES[file].replace(pattern, repl)
proc freplace(file: string, pattern: Regex, repl="") =
loadfile(file)
if FILES[file].find(pattern).isSome():
if "$#" in repl:
for m in FILES[file].findIter(pattern):
FILES[file] = FILES[file].replace(m.match, repl % m.captures[0])
else:
FILES[file] = FILES[file].replace(pattern, repl)
proc compile(dir="", file=""): string =
proc fcompile(file: string): string =
return "{.compile: \"$#\".}" % file.replace("\\", "/")
var data = ""
if dir != "" and dirExists(dir):
for f in walkFiles(dir / "*.c"):
data &= fcompile(f) & "\n"
if file != "" and fileExists(file):
data &= fcompile(file) & "\n"
return data
proc fixfuncprotos(file: string) =
var data = readFile(file)
var edit = false
for fp in data.findIter(re"(?m)(^.*?)[ ]*\(\*(.*?)\((.*?)\)\)[ \r\n]*\((.*?[\r\n]*.*?)\);"):
var tdout = "typedef $# (*type_$#)($#);\n" % [fp.captures[0], fp.captures[1], fp.captures[3]] &
"type_$# $#($#);" % [fp.captures[1], fp.captures[1], fp.captures[2]]
data = data.replace(fp.match, tdout)
edit = true
if edit:
writeFile(file, data)
# ###
# Convert to Nim
proc getincls(file: string): seq[string] =
loadfile(file)
result = @[]
for f in FILES[file].findIter(re"(?m)^\s*#\s*include\s+(.*?)$"):
var inc = f.captures[0].replace(re"""[<>"]""", "").strip()
if FILTER in inc and (not exclude(inc)):
result.add(inc)
result = result.deduplicate()
proc getdefines(file: string): string =
loadfile(file)
result = ""
for def in FILES[file].findIter(re"(?m)^(\s*#\s*define\s+[\w\d_]+\s+[\d.x]+)(?:\r|//|/*).*?$"):
result &= def.captures[0] & "\n"
proc preprocess(file: string): string =
var cmd = "gcc -E " & file
for inc in INCLUDES:
cmd &= " -I " & inc
# Run preprocessor
var data = execProc(cmd)
# Include content only from file
var rdata: Rope
var start = false
let sfile = file.replace("\\", "/")
for line in data.splitLines():
if line.strip() != "":
if line[0] == '#':
start = false
if sfile in line.replace("\\", "/").replace("//", "/"):
start = true
else:
if start:
rdata.add(
line.replace("_Noreturn", "")
.replace("WINAPI", "")
.replace("__attribute__", "")
.replace(re"\(\([_a-z]+?\)\)", "")
.replace(re"\(\(__format__\(__printf__, \d, \d\)\)\);", ";") & "\n"
)
return $rdata
proc ctags(file: string): string =
var cmd = "ctags -o - --fields=+S+K --c-kinds=p --file-scope=no " & file
var fps = execProc(cmd)
var fdata = ""
for line in fps.splitLines():
var spl = line.split(re"\t")
if spl.len() > 4:
if spl[0] != "main":
var fn = ""
var match = spl[2].find(re"/\^(.*?)\(")
if match.isSome():
fn = match.get().captures[0]
fn &= spl[4].replace("signature:", "") & ";"
fdata &= fn & "\n"
return fdata
proc c2nim(fl, outfile, flags: string, recurse, preproc, ctag, define: bool, compile, dynlib: seq[string] = @[]) =
var file = search(fl)
if file == "":
return
if file in DONE:
return
echo "Processing " & file
DONE.add(file)
fixfuncprotos(file)
var incout = ""
if recurse:
var incls = getincls(file)
for inc in incls:
incout &= "import " & inc.splitFile().name & "\n"
c2nim(inc, getnimout(inc), flags, recurse, preproc, ctag, define)
var cfile = file
if preproc:
cfile = "temp.c"
writeFile(cfile, preprocess(file))
elif ctag:
cfile = "temp.c"
writeFile(cfile, ctags(file))
if define and (preproc or ctag):
prepend(cfile, getdefines(file))
savefile(cfile)
var extflags = ""
var passC = ""
var outlib = ""
if compile.len() != 0:
passC = "import strutils\n"
for inc in INCLUDES:
passC &= ("""{.passC: "-I\"" & gorge("nimble path $#").strip() & "/$#\"".}""" % [OUTPUT, inc]) & "\n"
passC &= "{.push importc.}\n{.push header: \"$#\".}\n" % fl
#extflags = "--header:\"$#\"" % fl
if dynlib.len() != 0:
let win = "when defined(Windows):\n"
let lin = "when defined(Linux):\n"
let osx = "when defined(MacOSX):\n"
var winlib, linlib, osxlib: string = ""
for dl in dynlib:
if dl.splitFile().ext == ".dll":
winlib &= " const dynlib$# = \"$#\"\n" % [OUTPUT, dl]
if dl.splitFile().ext == ".so":
linlib &= " const dynlib$# = \"$#\"\n" % [OUTPUT, dl]
if dl.splitFile().ext == ".dylib":
osxlib &= " const dynlib$# = \"$#\"\n" % [OUTPUT, dl]
if winlib != "":
outlib &= win & winlib & "\n"
if linlib != "":
outlib &= lin & linlib & "\n"
if winlib != "":
outlib &= osx & osxlib & "\n"
if outlib != "":
extflags &= " --dynlib:dynlib$#" % OUTPUT
# Run c2nim on generated file
var cmd = "c2nim $# $# --out:$# $#" % [flags, extflags, outfile, cfile]
when defined(windows):
cmd = "cmd /c " & cmd
discard execProc(cmd)
if preproc:
try:
removeFile("temp.c")
except:
discard
# Import nim modules
if recurse:
prepend(outfile, incout)
# Nim doesn't like {.cdecl.} for type proc()
freplace(outfile, re"(?m)(.*? = proc.*?){.cdecl.}", "$#")
freplace(outfile, " {.cdecl.})", ")")
# Include {.compile.} directives
for cpl in compile:
if getFileInfo(cpl).kind == pcFile:
prepend(outfile, compile(file=cpl))
else:
prepend(outfile, compile(dir=cpl))
# Add header file and include paths
if passC != "":
prepend(outfile, passC)
append(outfile, "\n{.pop.}\n{.pop.}\n")
# Add dynamic library
if outlib != "":
prepend(outfile, outlib)
# ###
# Processor
proc runcfg(cfg: string) =
if not fileExists(cfg):
echo "Config doesn't exist: " & cfg
quit()
CONFIG = loadConfig(cfg)
if CONFIG.hasKey("n.global"):
if CONFIG["n.global"].hasKey("output"):
OUTPUT = CONFIG["n.global"]["output"]
if CONFIG["n.global"].hasKey("filter"):
FILTER = CONFIG["n.global"]["filter"]
if CONFIG.hasKey("n.include"):
for inc in CONFIG["n.include"].keys():
INCLUDES.add(inc)
if CONFIG.hasKey("n.exclude"):
for excl in CONFIG["n.exclude"].keys():
EXCLUDES.add(excl)
for file in CONFIG.keys():
if file in @["n.global", "n.include", "n.exclude"]:
continue
var sfile = search(file)
var srch = ""
var action = ""
var compile: seq[string] = @[]
var dynlib: seq[string] = @[]
for act in CONFIG[file].keys():
echo act
echo "A" & CONFIG[file][act] & "A"
action = act.replace(re"\..*", "")
if action == "create":
writeFile(file, CONFIG[file][act])
elif action in @["prepend", "append", "replace", "compile", "dynlib"] and sfile != "":
if action == "prepend":
if srch != "":
prepend(sfile, CONFIG[file][act], CONFIG[file][srch])
else:
prepend(sfile, CONFIG[file][act])
elif action == "append":
if srch != "":
append(sfile, CONFIG[file][act], CONFIG[file][srch])
else:
append(sfile, CONFIG[file][act])
elif action == "replace":
if srch != "":
freplace(sfile, CONFIG[file][srch], CONFIG[file][act])
elif action == "compile":
compile.add(CONFIG[file][act])
elif action == "dynlib":
dynlib.add(CONFIG[file][act])
srch = ""
elif action == "search":
srch = act
if file.splitFile().ext != ".nim":
var recurse = false
var preproc = false
var ctag = false
var define = false
var flags = "--stdcall"
# Save C files in case they have changed
savefile(sfile)
for act in CONFIG[file].keys():
if CONFIG[file][act] == "true":
if act == "recurse":
recurse = true
elif act == "preprocess":
preproc = true
elif act == "ctags":
ctag = true
elif act == "defines":
define = true
elif act == "flags":
flags = CONFIG[file][act]
c2nim(file, getnimout(file), flags, recurse, preproc, ctag, define, compile, dynlib)
# ###
# Main loop
if paramCount() == 0:
echo "nimgen file.cfg"
quit()
for i in 1..paramCount():
runcfg(paramStr(i))
savefiles()

12
nimgen.nimble Normal file
View file

@ -0,0 +1,12 @@
# Package
version = "0.1.0"
author = "genotrance"
description = "c2nim helper to simpilfy and automate the wrapping of C libraries"
license = "MIT"
# Dependencies
requires "nim >= 0.16.0", "c2nim >= 0.9.13"
bin = @["nimgen"]