Simple Python template engine • 5 Jan 2011
UPDATE 2012: A more recent version of the source code can be found at github.com/.../ovotemplate.py
UPDATE 2014: The most recent version of the source code is below. I recently revamped the template parsing code.
An idea for a simple templating engine
I want to do basic template expansion, like this:
Hello {=name}, thanks for your order.
And then feed a 'name' to it, in this way:
vars = { "name": "Joe"}
So that it expands to this:
Hello Joe, thanks for you order.
If "{=name}" occurs multiple times in the template, each instance should be substituted with 'Joe'; and not only the first one encountered.
Reinventing the wheel?
Why did I write this, you ask? Perfectly valid question, because there are already many templating engines out there. I even use a few of them ;-) But for some situations I want to have a template solution which doesn't use embedded Python statements, nor an XML-ish syntax, but does support repeating and conditional blocks. I didn't find an existing one, so I wrote it myself.
In my opinion templates should not contain statements: code belongs in the application. Intermingling code with literal page fragments makes me think of how things were done in the classic ASP days, and stirs up forgotten memories of horrible mashed-together PHP pages. It becomes an unmaintainable mess. Some people object: "So? Why not have code in the template? In the end you are the one looking at them anyway!" But that's not true. I regularly have my templates translated by translators who don't speak any programming language, but understand that something like {=product_id} will be filled in later and shouldn't be translated.
On with the feature list! ...
Conditionals
I also want to be able to mark certain parts of the template so they will only appear in the output when a variable is True:
Hello {=name}, thanks for your order.
{?ccpayment
The sum of {=grandtotal} will be deducted from your creditcard.
}
...combined with this data...
vars = {
"name": "Joe",
"ccpayment": True,
"grandtotal": "23.99",
}
Result:
Hello Joe, thanks for your order.
The sum of 23.99 will be deducted from your creditcard.
If ccpayment would be False, the entire line "The sum of..." would not be displayed.
Repeating groups
And it should do repeating groups, too! For example, to generate the following output:
Hello Joe, thanks for your order.
The sum of 23.99 will be deducted from your creditcard.
You bought:
2x 'Coders at work' by Peter Seibel = 15.10
1x 'Hackers and painters' by Paul Graham = 3.12
1x 'More Joel on Software' by Joel Spolsky = 7.62
I'd like to use this template:
Hello {=name}, thanks for your order.
{?ccpayment
The sum of {=grandtotal} will be deducted from your creditcard.
}
{#purchases
{=count}x '{=title}' by {=author} = {=total}
}
And I'd have to feed it with this data:
vars = {
"name": "Joe",
"ccpayment": True,
"purchases": (
{"count": 2, "title": "Coders at work", "author": "Peter Seibel", "total": 15.10},
{"count": 1, "title": "Hackers and painters", "author": "Paul Graham", "total": 3.12},
{"count": 1, "title": "More Joel on Software", "author": "Joel Spolsky", "total": 7.62},
)
"grandtotal": "23.99",
}
Nesting
Nesting would be a great feature. For example, to make a list with sublist like the one below, a nested repeat can be used:
1. Introduction
1.1. Features
1.2. FAQ
1.3. Getting started
2. Usage
2.1. Daily usage
2.2. Exceptional cases
The template would look like this:
{#chapters
{=nr}. {=title}
{#subchapters
{=subnr}. {=subtitle}
}
}
and the data structure like this:
vars = {
"chapters": (
{"nr": "1", "title": "Introduction", "subchapters": (
{"subnr": "1.1", "subtitle": "Features"},
{"subnr": "1.2", "subtitle": "FAQ"},
{"subnr": "1.3", "subtitle": "Getting started"},
)
},
{"nr": "2", "title": "Usage", "subchapters":
{"subnr": "2.1", "subtitle": "Daily usage"},
{"subnr": "2.2", "subtitle": "Exceptional cases"},
},
)
}
Source code
# -*- coding: utf8 -*-
"""
Simple templating class, software by Michiel Overtoom, motoom@xs4all.nl.
"""
import os
import pprint
import re
import unittest
import codecs
import functools
whitechars = re.compile("\s")
verbose = False
exceptionless = True # False: throw exceptions when something is wrong with the template or rendering it; True: insert an error in the output text instead.
def indent(level):
return "| " + " " * level
class Container(list):
"Generic container."
def __init__(self, name=""):
self.name = name
def __repr__(self):
tag = "%s %s" % (self.__class__.__name__, self.name)
return "%s: %s" % (tag.strip(), super(Container, self).__repr__())
def render(self, vars, last=False, level=0):
if verbose:
print "%sContainer.render(vars=%s,last=%s) type(vars)=%s, self.name=%s" % (indent(level), vars, last, type(vars), self.name)
output = ""
for child in self:
if verbose:
print "%sContainer.render child %s" % (indent(level), child)
value = child.render(vars, last, level + 1)
if value:
output += value
return output
class Lit(Container):
"Container for literal content."
def __init__(self, contents=""):
super(Lit, self).__init__()
self.append(contents)
def __repr__(self):
return super(Lit, self).__repr__()
def render(self, vars, last=None, level=None):
if verbose:
print "%sLit.render(vars=%s,last=%s) type(vars)=%s, self=%s" % (indent(level), vars, last, type(vars), self)
return self[0]
class Sep(Container):
"Container for separator. Same as Lit, but doesn't result in output in the last iteration of a Rep."
def __init__(self, name):
super(Sep, self).__init__(name)
def __repr__(self):
return super(Sep, self).__repr__()
def render(self, vars, last, level):
if verbose:
print "%sSep.render(vars=%s) type(vars)=%s, self=%s" % (indent(level), vars, type(vars), self)
if last:
if verbose:
print "%sSep.render last is True, empty string returned" % indent(level)
return ""
output = ""
for child in self:
if verbose:
print "%sSep.render child %s" % (indent(level), child)
value = child.render(vars, last, level + 1)
if value:
output += value
return output
class Sub(Container):
"Container for a variable substitution."
def __init__(self, name):
super(Sub, self).__init__(name)
def __repr__(self):
return super(Sub, self).__repr__()
def render(self, vars, last, level):
if verbose:
print "%sSub.render(vars=%s) type(vars)=%s, self.name=%s" % (indent(level), vars, type(vars), self.name)
try:
value = vars[self.name]
except KeyError:
return '<span class="ovotemplate_error" style="background-color: red; color: white;">Template error in Sub: unknown variable "%s"</span>' % self.name
if isinstance(value, (int, float, long)):
value = str(value)
return value
class Cond(Container):
"Container for conditional content."
def __init__(self, name, inverting=False):
super(Cond, self).__init__(name)
self.inverting = inverting
def __repr__(self):
return super(Cond, self).__repr__()
def render(self, vars, last, level):
if verbose:
print "%sCond.render(vars=%s) type(vars)=%s, self.name=%s, self.inverting=%s" % (indent(level), vars, type(vars), self.name, self.inverting)
ok = vars.get(self.name) # Assume missing template variable is False.
if self.inverting:
ok = not ok
if not ok:
if verbose:
print "%sCond.render cond is False, empty string returned" % indent(level)
return ""
output = ""
for child in self:
if verbose:
print "%sCond.render child %s" % (indent(level), child)
value = child.render(vars, last, level + 1)
if value:
output += value
return output
class Rep(Container):
"Container for repeating content."
def __init__(self, name):
super(Rep, self).__init__(name)
self.verbose = False
def __repr__(self):
return super(Rep, self).__repr__()
def render(self, vars, last, level):
if verbose or self.verbose:
print "%sRep.render(vars=%s) type(vars)=%s, self.name=%s" % (indent(level), vars, type(vars), self.name)
output = ""
# TODO: Dit kan nuttige debug info opleveren: if not self.name in vars: raise NameNotFound("A required variable name '%s' was not present in '%r'" % (self.name, vars))
try:
subvars = vars[self.name] # A KeyError here means that a required variable wasn't present.
except KeyError:
return '<span class="ovotemplate_error" style="background-color: red; color: white;">Template error in Rep: unknown variable "%s"</span>' % self.name
for nr, subvar in enumerate(subvars):
if verbose or self.verbose:
print "%sRep.render subvar=%s, type(subvar)=%s" % (indent(level), subvar, type(subvar))
for child in self:
last = nr == len(subvars) - 1
if verbose or self.verbose:
print "%sRep.render child %s, last=%s" % (indent(level), child, last)
value = child.render(subvar, last, level + 1)
if value or self.verbose:
output += value
return output
def splitfirst(s):
"Split a string into a first special word, and the rest."
if not s:
return "", ""
if s[0] in createinfo:
parts = whitechars.split(s, 1)
if len(parts) < 2:
return s, ""
else:
return tuple(parts)
else:
return "", s
def feed(seq):
for item in seq:
yield item
def lexer(it):
"""Split input into tokens. A token is either an open curly brace, a closing curly brace, or a string without curly braces."""
tokens = []
token = ""
for c in it:
if c == "{":
if token:
tokens.append(token)
token = ""
tokens.append(c)
elif c == "}":
if token:
tokens.append(token)
token = ""
tokens.append(c)
else:
token += c
if token:
tokens.append(token)
return tokens
def parse(it, node, nesting=0):
"""Build a (recursive) nested list from the tokens."""
for token in it:
if token == "{":
subnode = []
node.append(subnode)
parse(it, subnode, nesting + 1)
elif token == "}":
if nesting == 0:
raise Exception("Unbalanced }")
return
else:
node.append(token)
CHOPNAME = 1
CHOPITEM = 2
createinfo = {
"?": (Cond, CHOPNAME),
"!": (functools.partial(Cond, inverting=True), CHOPNAME),
"#": (Rep, CHOPNAME),
"=": (Sub, CHOPITEM),
"/": (Sep, CHOPNAME),
}
def compile(node, into, level=0):
if verbose:
print "%s compile: " % indent(level), node
for pos, item in enumerate(node):
if isinstance(item, list):
if verbose:
print "%s #%d list: %r" % (indent(level), pos, item)
head = item[0]
if not head[0] in createinfo:
if exceptionless:
return Lit("<span style=\"background-color: red; color: white;\">Template error: '{' without a following valid metachar</span>")
else:
raise ValueError("'{' without a following valid metachar")
first, rest = splitfirst(head)
operator, name = first[0], first[1:]
if verbose:
print "%s operator %s, name %s, rest %r" % (indent(level), operator, name, rest)
# Create correct container
factoryfunc, options = createinfo[operator]
ob = factoryfunc(name)
if options == CHOPNAME:
item[0] = rest
elif options == CHOPITEM:
item = item[1:]
into.append(compile(item, ob, level + 1))
else:
if verbose:
print "%s #%d item: %s" % (indent(level), pos, item)
into.append(Lit(item))
return into
def process(sourcetext):
if verbose:
print "\n\n\nCompile phase"
tokens = lexer(feed(sourcetext))
root = []
parse(feed(tokens), root)
result = compile(root, Container())
if verbose:
print "Compile result:", result
return result
class Ovotemplate(object):
"""Simple templating class."""
def __init__(self, s=None, name=None):
"""Initialize a template, optionally from a template string."""
if s:
self.root = process(s)
elif s is not None:
self.root = Container()
self.root.append(Lit(""))
else:
self.root = None
self.name = name
def fromfile(self, fn):
"""Load a template from a file.
Allows: tem = Ovotemplate().fromfile("hello.tpl")
The template file should contain UTF-8 encoded unicode text
"""
self.root = process(codecs.open(fn, "r", "utf8").read())
self.name = fn.replace(" ", "_")
return self
def pprint(self):
"""Pretty-print the template structure."""
pprint.pprint(self.root)
def render(self, vars):
"""Renders the template to a string, using the supplied variables."""
if verbose:
print "\nRender phase"
if not self.root:
raise Exception("You should either pass a template as a string in the constructor, or use 'fromfile' to read the template from file")
result = self.root.render(vars)
if verbose:
print "Render result:", result
return result
class Test(unittest.TestCase):
"""Unittest for Ovotemplate."""
def test_naming(self):
"""Test the naming; every template instance can have a name (usually the filename where it was loaded from).
This name is used in error reporting."""
tems = "{=name}"
tem = Ovotemplate(tems, "nametest")
self.assertEquals(tem.name, "nametest")
def test_badmetachar(self):
tems = "{&name}" # Note that '&' is illegal after a '{'.
#
global exceptionless
prevexceptionless = exceptionless
#
exceptionless = False
self.assertRaises(ValueError, Ovotemplate, tems)
#
exceptionless = True
tem = Ovotemplate(tems)
res = tem.render({})
self.assertTrue("Template error" in res)
exceptionless = prevexceptionless
def test_splitting(self):
self.assertEquals(splitfirst(""), ("", ""))
self.assertEquals(splitfirst("?hi"), ("?hi", ""))
self.assertEquals(splitfirst("?hi there"), ("?hi", "there"))
self.assertEquals(splitfirst("hi"), ("", "hi"))
self.assertEquals(splitfirst("hi there"), ("", "hi there"))
def test_render(self):
"""Test a number of progressively complex render cases."""
goodcases = (
# Empty template.
("", {}, ""),
# Just a letter.
("a", {}, "a"),
# Longer string.
("hi there", {}, "hi there"),
# Simple substitution.
("{=status}", {"status": "STATUS"}, "STATUS"),
("{=status}", {"status": 67.2334}, "67.2334"),
("{=status}", {"status": None}, ""),
("{=status}", {"status": False}, "False"),
("BEFORE{=status}", {"status": "STATUS"}, "BEFORESTATUS"),
("{=status}AFTER", {"status": "STATUS"}, "STATUSAFTER"),
# Two substitutions in different flavors.
("{=one}{=two}", {"one": "ONE", "two": "TWO"}, "ONETWO"),
("{=one}AND{=two}", {"one": "ONE", "two": "TWO"}, "ONEANDTWO"),
("{=one} {=two}", {"one": "ONE", "two": "TWO"}, "ONE TWO"),
("{=one} {=two}", {"one": "ONE", "two": "TWO"}, "ONE TWO"),
("{=one}, {=two}", {"one": "ONE", "two": "TWO"}, "ONE, TWO"),
("{=one} ({=two})", {"one": "ONE", "two": "TWO"}, "ONE (TWO)"),
# Substitution with text in between.
("well{=here}it{=goes}with{=some}test",
{"here": "HERE", "goes": "GOES", "some": "SOME"},
"wellHEREitGOESwithSOMEtest"),
('{?useimg hallo <img src="path/names/{=component}/with/{=component}.jpg">}',
{"useimg": True, "component": "filesystem"},
'hallo <img src="path/names/filesystem/with/filesystem.jpg">'),
# Simple repetitions.
("{#cls{=co}}",
{"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
"redgrbl"),
("{#cls <{=co}>}",
{"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
"<red><gr><bl>"),
("{#cls {=co}, }",
{"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
"red, gr, bl, "),
("{#cls {=co} x }",
{"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
"red x gr x bl x "),
("{#cls {=co} _}",
{"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
"red _gr _bl _"),
# Simple conditions.
("throw a {?condition big }party",
{"condition": True},
"throw a big party"),
("throw a {?condition big }tantrum",
{"condition": 42},
"throw a big tantrum"),
("throw a {?condition big }party",
{"condition": False},
"throw a party"),
("throw a {?condition big }tantrum",
{"condition": None},
"throw a tantrum"),
("A!{?condition B}!C!{!condition D}!E",
{"condition": True},
"A!B!C!!E"),
("A!{?condition B}!C!{!condition D}!E",
{"condition": False},
"A!!C!D!E"),
# Repeats.
("{#a{=b}{=c}}",
{"a": ({"b": 11, "c": 22},)},
"1122"),
("{#a {=b} {=c}}",
{"a": [{"b": 33, "c": 44}]},
"33 44"),
("{#a STA{=b}STO BEG{=c}END }",
{"a": ({"b": 55, "c": 66},)},
"STA55STO BEG66END "),
("{#a {=b} {=c}}",
{"a": ({"b": 7.70, "c": 88}, {"b": 99, "c": 1.234567})},
"7.7 8899 1.234567"),
("{#a {=b} {=c}}", {"a": ()}, ""),
# Repeat with variabele as last on the line.
("{#blop\n{=you}}",
dict(blop=(dict(you=123), dict(you=456))),
"123456"),
("{#blop\n{=you}\n}",
dict(blop=(dict(you=123), dict(you=456))),
"123\n456\n"),
# A join()-like separator.
("{#colors {=color}{/comma , }}",
dict(colors=(dict(color="red"), dict(color="green"),
dict(color="blue"))),
"red, green, blue"),
# More repeats.
("buy {=count} articles: {#articles {=nam} txt {=pri}, }", {
"count": 2,
"articles": ({"nam": "Ur", "pri": 1}, {"nam": "Mo", "pri": 2})
},
"buy 2 articles: Ur txt 1, Mo txt 2, "),
("sell {=count} stocks: {#articles {=nam} € {=pri}{/comma , }}",
{"count": 2, "articles": ({"nam": "APPL", "pri": 320}, {"nam": "GOOG", "pri": 120})},
"sell 2 stocks: APPL € 320, GOOG € 120"),
# Nested repeats.
("Contents: {#chapters Chapter {=name}. {#sections Section {=name}. }",
{"chapters": [
dict(name="Intro", sections=[dict(name="Foreword"), dict(name="Methodology")]),
dict(name="Middle", sections=[dict(name="Measuring"), dict(name="Calculation"), dict(name="Results")]),
dict(name="Epilogue", sections=[dict(name="Conclusion")])
]
}, "Contents: Chapter Intro. Section Foreword. Section Methodology. "
"Chapter Middle. Section Measuring. Section Calculation. Section Results. "
"Chapter Epilogue. Section Conclusion. "),
# Condition with repeat.
("Dear {=name}, {?market Please get the following groceries:\n"
"{#groceries \tItem: {=item}, {=count} pieces\n}}"
"{?deadline Please be back before {=time}!}",
{"name": "Joe",
"market": "True",
"count": 5,
"groceries": [dict(item="lemon", count=2), dict(item="cookies", count=4)],
"deadline": True,
"time": "17:30",
},
"Dear Joe, Please get the following groceries:\n\tItem: "
"lemon, 2 pieces\n\tItem: cookies, 4 pieces\nPlease be "
"back before 17:30!"),
)
for tems, temv, expected in goodcases:
tem = Ovotemplate(tems) # tem.pprint()
self.assertEquals(tem.render(temv), expected)
''' TODO: This still needs some work - sensible error reporting.
badcases = (
("{#a {=b} {=c}}", {}), # required variables missing
("=a}", dict(a=42)), # missing opening {
# ("{=a", dict(a=42)), # missing closing {
)
global exceptionless
exceptionless = False
for tems, temv in badcases:
self.assertRaises(Exception, Ovotemplate(tems).render(temv))
'''
def test_performance():
"""Ovotemplate and Jinja2 go head-to-head!
Result for nr=150 on my MacBook Air:
Ovotemplate: 467MB produced in 66.237 sec
Jinja2: 470MB produced in 156.205 sec
"""
import time
nr = 40 # Set to 2 to view the expanded templates as well.
books = []
d = {"books": books}
for booknr in range(nr):
chapters = []
book = dict(title="%d bottles of beer" % booknr, toc="This will be the table of contents.", chapters=chapters)
books.append(book)
for chapternr in range(nr):
sections = []
chapter = dict(title="%d. How to drink beer" % chapternr, intro="This will be an intro", sections=sections)
chapters.append(chapter)
for sectionnr in range(nr):
section = dict(title="%d. Procedure" % sectionnr, text="This will be an explanation of how to drink beer.")
sections.append(section)
tem = Ovotemplate("""
{#books
<h1>The Book Of {=title}</h1>
<p>{=toc}</p>
{#chapters
<h2>Chapter {=title}</h2>
<p>{=intro}</p>
{#sections
<h3>Section {=title}</h3>
<p>{=text}</p>
}
}
}
""")
start = time.time()
res = tem.render(d)
dur = time.time() - start
print "Ovotemplate: %dMB produced in %.3f sec:" % (len(res)/1024/1024, dur)
if nr < 3:
print res
#
import jinja2
from jinja2 import Template
tem = Template("""
{% for book in books %}
<h1>The Book Of {{book.title}}</h2>
<p>{{book.toc}}</p>
{% for chapter in book.chapters %}
<h2>Chapter {{chapter.title}}</h2>
<p>{{chapter.intro}}</p>
{% for section in chapter.sections %}
<h3>Section {{section.title}}</h3>
<p>{{section.text}}</p>
{% endfor %}
{% endfor %}
{% endfor %}
""")
start = time.time()
res = tem.render(dict(books=books))
dur = time.time() - start
print "Jinja2: %dMB produced in %.3f sec:" % (len(res)/1024/1024, dur)
if nr < 3:
print res
if __name__ == "__main__":
# For the usual unittests:
unittest.main()
# Uncomment for a benchmark comparison:
# test_performance()
Comments
Michiel • 25 Apr 2011
However, it is already useful.
Cees Joppe • 14 Aug 2011
Thanx for sharing your clever template Michiel!
Pavel • 26 Nov 2011
GREAT engine! One question: How can you enter a '{' or a '}' into the template for expansion in the out? Thanks!
Bogomil • 20 Mar 2013
I do the following change on rendering all children: output = [] output = [child.render(vars, last, level+1) for child in self] return "".join(output) Now the string operations are much, much faster
Michiel • 6 Sep 2014
Bogomil, I've tried that too, but I get only a marginal speedup (19.5sec instead of 20sec on a *very* large template). I guess Python's string concatenation is very optimized. I also profiles the code and most of the time is spent in the enumeration of child nodes, and after that, converting numbers to strings. Idea for optimisation: make it possible to have the template work with strings only, so that it can always assume the input are string values.
Michiel • 21 Dec 2010
This is not complete yet!