michielovertoom.com

Simple Python template engine • 5 Jan 2011

UPDATE 2012: A more recent version of the source code can be found at github.com/.../ovotemplate.py

UPDATE 2014: The most recent version of the source code is below. I recently revamped the template parsing code.

An idea for a simple templating engine

I want to do basic template expansion, like this:

Hello {=name}, thanks for your order.

And then feed a 'name' to it, in this way:

vars = { "name": "Joe"}

So that it expands to this:

Hello Joe, thanks for you order.

If "{=name}" occurs multiple times in the template, each instance should be substituted with 'Joe'; and not only the first one encountered.

Reinventing the wheel?

Why did I write this, you ask? Perfectly valid question, because there are already many templating engines out there. I even use a few of them ;-) But for some situations I want to have a template solution which doesn't use embedded Python statements, nor an XML-ish syntax, but does support repeating and conditional blocks. I didn't find an existing one, so I wrote it myself.

In my opinion templates should not contain statements: code belongs in the application. Intermingling code with literal page fragments makes me think of how things were done in the classic ASP days, and stirs up forgotten memories of horrible mashed-together PHP pages. It becomes an unmaintainable mess. Some people object: "So? Why not have code in the template? In the end you are the one looking at them anyway!" But that's not true. I regularly have my templates translated by translators who don't speak any programming language, but understand that something like {=product_id} will be filled in later and shouldn't be translated.

On with the feature list! ...

Conditionals

I also want to be able to mark certain parts of the template so they will only appear in the output when a variable is True:

Hello {=name}, thanks for your order. {?ccpayment The sum of {=grandtotal} will be deducted from your creditcard. }

...combined with this data...

vars = { "name": "Joe", "ccpayment": True, "grandtotal": "23.99", }

Result:

Hello Joe, thanks for your order. The sum of 23.99 will be deducted from your creditcard.

If ccpayment would be False, the entire line "The sum of..." would not be displayed.

Repeating groups

And it should do repeating groups, too! For example, to generate the following output:

Hello Joe, thanks for your order. The sum of 23.99 will be deducted from your creditcard. You bought: 2x 'Coders at work' by Peter Seibel = 15.10 1x 'Hackers and painters' by Paul Graham = 3.12 1x 'More Joel on Software' by Joel Spolsky = 7.62

I'd like to use this template:

Hello {=name}, thanks for your order. {?ccpayment The sum of {=grandtotal} will be deducted from your creditcard. } {#purchases {=count}x '{=title}' by {=author} = {=total} }

And I'd have to feed it with this data:

vars = { "name": "Joe", "ccpayment": True, "purchases": ( {"count": 2, "title": "Coders at work", "author": "Peter Seibel", "total": 15.10}, {"count": 1, "title": "Hackers and painters", "author": "Paul Graham", "total": 3.12}, {"count": 1, "title": "More Joel on Software", "author": "Joel Spolsky", "total": 7.62}, ) "grandtotal": "23.99", }

Nesting

Nesting would be a great feature. For example, to make a list with sublist like the one below, a nested repeat can be used:

1. Introduction 1.1. Features 1.2. FAQ 1.3. Getting started 2. Usage 2.1. Daily usage 2.2. Exceptional cases

The template would look like this:

{#chapters {=nr}. {=title} {#subchapters {=subnr}. {=subtitle} } }

and the data structure like this:

vars = { "chapters": ( {"nr": "1", "title": "Introduction", "subchapters": ( {"subnr": "1.1", "subtitle": "Features"}, {"subnr": "1.2", "subtitle": "FAQ"}, {"subnr": "1.3", "subtitle": "Getting started"}, ) }, {"nr": "2", "title": "Usage", "subchapters": {"subnr": "2.1", "subtitle": "Daily usage"}, {"subnr": "2.2", "subtitle": "Exceptional cases"}, }, ) }

Source code

# -*- coding: utf8 -*-

"""
Simple templating class, software by Michiel Overtoom, motoom@xs4all.nl.
"""

import os
import pprint
import re
import unittest
import codecs
import functools

whitechars = re.compile("\s")

verbose = False
exceptionless = True  # False: throw exceptions when something is wrong with the template or rendering it; True: insert an error in the output text instead.


def indent(level):
    return "| " + "    " * level


class Container(list):
    "Generic container."

    def __init__(self, name=""):
        self.name = name

    def __repr__(self):
        tag = "%s %s" % (self.__class__.__name__, self.name)
        return "%s: %s" % (tag.strip(), super(Container, self).__repr__())

    def render(self, vars, last=False, level=0):
        if verbose:
            print "%sContainer.render(vars=%s,last=%s) type(vars)=%s, self.name=%s" % (indent(level), vars, last, type(vars), self.name)
        output = ""
        for child in self:
            if verbose:
                print "%sContainer.render child %s" % (indent(level), child)
            value = child.render(vars, last, level + 1)
            if value:
                output += value
        return output


class Lit(Container):
    "Container for literal content."

    def __init__(self, contents=""):
        super(Lit, self).__init__()
        self.append(contents)

    def __repr__(self):
        return super(Lit, self).__repr__()

    def render(self, vars, last=None, level=None):
        if verbose:
            print "%sLit.render(vars=%s,last=%s) type(vars)=%s, self=%s" % (indent(level), vars, last, type(vars), self)
        return self[0]


class Sep(Container):
    "Container for separator. Same as Lit, but doesn't result in output in the last iteration of a Rep."

    def __init__(self, name):
        super(Sep, self).__init__(name)

    def __repr__(self):
        return super(Sep, self).__repr__()

    def render(self, vars, last, level):
        if verbose:
            print "%sSep.render(vars=%s) type(vars)=%s, self=%s" % (indent(level), vars, type(vars), self)
        if last:
            if verbose:
                print "%sSep.render last is True, empty string returned" % indent(level)
            return ""
        output = ""
        for child in self:
            if verbose:
                print "%sSep.render child %s" % (indent(level), child)
            value = child.render(vars, last, level + 1)
            if value:
                output += value
        return output


class Sub(Container):
    "Container for a variable substitution."

    def __init__(self, name):
        super(Sub, self).__init__(name)

    def __repr__(self):
        return super(Sub, self).__repr__()

    def render(self, vars, last, level):
        if verbose:
            print "%sSub.render(vars=%s) type(vars)=%s, self.name=%s" % (indent(level), vars, type(vars), self.name)
        try:
            value = vars[self.name]
        except KeyError:
            return '<span class="ovotemplate_error" style="background-color: red; color: white;">Template error in Sub: unknown variable "%s"</span>' % self.name
        if isinstance(value, (int, float, long)):
            value = str(value)
        return value


class Cond(Container):
    "Container for conditional content."

    def __init__(self, name, inverting=False):
        super(Cond, self).__init__(name)
        self.inverting = inverting

    def __repr__(self):
        return super(Cond, self).__repr__()

    def render(self, vars, last, level):
        if verbose:
            print "%sCond.render(vars=%s) type(vars)=%s, self.name=%s, self.inverting=%s" % (indent(level), vars, type(vars), self.name, self.inverting)
        ok = vars.get(self.name)  # Assume missing template variable is False.
        if self.inverting:
            ok = not ok
        if not ok:
            if verbose:
                print "%sCond.render cond is False, empty string returned" % indent(level)
            return ""
        output = ""
        for child in self:
            if verbose:
                print "%sCond.render child %s" % (indent(level), child)
            value = child.render(vars, last, level + 1)
            if value:
                output += value
        return output


class Rep(Container):
    "Container for repeating content."

    def __init__(self, name):
        super(Rep, self).__init__(name)
        self.verbose = False

    def __repr__(self):
        return super(Rep, self).__repr__()

    def render(self, vars, last, level):
        if verbose or self.verbose:
            print "%sRep.render(vars=%s) type(vars)=%s, self.name=%s" % (indent(level), vars, type(vars), self.name)
        output = ""
        # TODO: Dit kan nuttige debug info opleveren: if not self.name in vars: raise NameNotFound("A required variable name '%s' was not present in '%r'" % (self.name, vars))
        try:
            subvars = vars[self.name]  # A KeyError here means that a required variable wasn't present.
        except KeyError:
            return '<span class="ovotemplate_error" style="background-color: red; color: white;">Template error in Rep: unknown variable "%s"</span>' % self.name
        for nr, subvar in enumerate(subvars):
            if verbose or self.verbose:
                print "%sRep.render subvar=%s, type(subvar)=%s" % (indent(level), subvar, type(subvar))
            for child in self:
                last = nr == len(subvars) - 1
                if verbose or self.verbose:
                    print "%sRep.render child %s, last=%s" % (indent(level), child, last)
                value = child.render(subvar, last, level + 1)
                if value or self.verbose:
                    output += value
        return output


def splitfirst(s):
    "Split a string into a first special word, and the rest."
    if not s:
        return "", ""
    if s[0] in createinfo:
        parts = whitechars.split(s, 1)
        if len(parts) < 2:
            return s, ""
        else:
            return tuple(parts)
    else:
        return "", s


def feed(seq):
    for item in seq:
        yield item


def lexer(it):
    """Split input into tokens. A token is either an open curly brace, a closing curly brace, or a string without curly braces."""
    tokens = []
    token = ""
    for c in it:
        if c == "{":
            if token:
                tokens.append(token)
                token = ""
            tokens.append(c)
        elif c == "}":
            if token:
                tokens.append(token)
                token = ""
            tokens.append(c)
        else:
            token += c
    if token:
        tokens.append(token)
    return tokens


def parse(it, node, nesting=0):
    """Build a (recursive) nested list from the tokens."""
    for token in it:
        if token == "{":
            subnode = []
            node.append(subnode)
            parse(it, subnode, nesting + 1)
        elif token == "}":
            if nesting == 0:
                raise Exception("Unbalanced }")
            return
        else:
            node.append(token)


CHOPNAME = 1
CHOPITEM = 2

createinfo = {
    "?": (Cond, CHOPNAME),
    "!": (functools.partial(Cond, inverting=True), CHOPNAME),
    "#": (Rep, CHOPNAME),
    "=": (Sub, CHOPITEM),
    "/": (Sep, CHOPNAME),
    }


def compile(node, into, level=0):
    if verbose:
        print "%s compile: " % indent(level), node
    for pos, item in enumerate(node):
        if isinstance(item, list):
            if verbose:
                print "%s #%d list: %r" % (indent(level), pos, item)
            head = item[0]
            if not head[0] in createinfo:
                if exceptionless:
                    return Lit("<span style=\"background-color: red; color: white;\">Template error: '{' without a following valid metachar</span>")
                else:
                    raise ValueError("'{' without a following valid metachar")
            first, rest = splitfirst(head)
            operator, name = first[0], first[1:]
            if verbose:
                print "%s operator %s, name %s, rest %r" % (indent(level), operator, name, rest)
            # Create correct container
            factoryfunc, options = createinfo[operator]
            ob = factoryfunc(name)
            if options == CHOPNAME:
                item[0] = rest
            elif options == CHOPITEM:
                item = item[1:]
            into.append(compile(item, ob, level + 1))
        else:
            if verbose:
                print "%s #%d item: %s" % (indent(level), pos, item)
            into.append(Lit(item))
    return into


def process(sourcetext):
    if verbose:
        print "\n\n\nCompile phase"
    tokens = lexer(feed(sourcetext))
    root = []
    parse(feed(tokens), root)
    result = compile(root, Container())
    if verbose:
        print "Compile result:", result
    return result


class Ovotemplate(object):
    """Simple templating class."""

    def __init__(self, s=None, name=None):
        """Initialize a template, optionally from a template string."""
        if s:
            self.root = process(s)
        elif s is not None:
            self.root = Container()
            self.root.append(Lit(""))
        else:
            self.root = None
        self.name = name

    def fromfile(self, fn):
        """Load a template from a file.
        Allows: tem = Ovotemplate().fromfile("hello.tpl")
        The template file should contain UTF-8 encoded unicode text
        """
        self.root = process(codecs.open(fn, "r", "utf8").read())
        self.name = fn.replace(" ", "_")
        return self

    def pprint(self):
        """Pretty-print the template structure."""
        pprint.pprint(self.root)

    def render(self, vars):
        """Renders the template to a string, using the supplied variables."""
        if verbose:
            print "\nRender phase"
        if not self.root:
            raise Exception("You should either pass a template as a string in the constructor, or use 'fromfile' to read the template from file")
        result = self.root.render(vars)
        if verbose:
            print "Render result:", result
        return result


class Test(unittest.TestCase):
    """Unittest for Ovotemplate."""

    def test_naming(self):
        """Test the naming; every template instance can have a name (usually the filename where it was loaded from).
        This name is used in error reporting."""
        tems = "{=name}"
        tem = Ovotemplate(tems, "nametest")
        self.assertEquals(tem.name, "nametest")

    def test_badmetachar(self):
        tems = "{&name}" # Note that '&' is illegal after a '{'.
        #
        global exceptionless
        prevexceptionless = exceptionless
        #
        exceptionless = False
        self.assertRaises(ValueError, Ovotemplate, tems)
        #
        exceptionless = True
        tem = Ovotemplate(tems)
        res = tem.render({})
        self.assertTrue("Template error" in res)
        exceptionless = prevexceptionless

    def test_splitting(self):
        self.assertEquals(splitfirst(""), ("", ""))
        self.assertEquals(splitfirst("?hi"), ("?hi", ""))
        self.assertEquals(splitfirst("?hi there"), ("?hi", "there"))
        self.assertEquals(splitfirst("hi"), ("", "hi"))
        self.assertEquals(splitfirst("hi there"), ("", "hi there"))

    def test_render(self):
        """Test a number of progressively complex render cases."""
        goodcases = (
            # Empty template.
            ("", {}, ""),
            # Just a letter.
            ("a", {}, "a"),
            # Longer string.
            ("hi there", {}, "hi there"),
            # Simple substitution.
            ("{=status}", {"status": "STATUS"}, "STATUS"),
            ("{=status}", {"status": 67.2334}, "67.2334"),
            ("{=status}", {"status": None}, ""),
            ("{=status}", {"status": False}, "False"),
            ("BEFORE{=status}", {"status": "STATUS"}, "BEFORESTATUS"),
            ("{=status}AFTER", {"status": "STATUS"}, "STATUSAFTER"),
            # Two substitutions in different flavors.
            ("{=one}{=two}", {"one": "ONE", "two": "TWO"}, "ONETWO"),
            ("{=one}AND{=two}", {"one": "ONE", "two": "TWO"}, "ONEANDTWO"),
            ("{=one} {=two}", {"one": "ONE", "two": "TWO"}, "ONE TWO"),
            ("{=one}   {=two}", {"one": "ONE", "two": "TWO"}, "ONE   TWO"),
            ("{=one}, {=two}", {"one": "ONE", "two": "TWO"}, "ONE, TWO"),
            ("{=one} ({=two})", {"one": "ONE", "two": "TWO"}, "ONE (TWO)"),
            # Substitution with text in between.
            ("well{=here}it{=goes}with{=some}test",
                {"here": "HERE", "goes": "GOES", "some": "SOME"},
                "wellHEREitGOESwithSOMEtest"),

            ('{?useimg hallo <img src="path/names/{=component}/with/{=component}.jpg">}',
              {"useimg": True, "component": "filesystem"},
              'hallo <img src="path/names/filesystem/with/filesystem.jpg">'),

            # Simple repetitions.
            ("{#cls{=co}}",
                {"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
                "redgrbl"),
            ("{#cls <{=co}>}",
                {"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
                "<red><gr><bl>"),
            ("{#cls {=co}, }",
                {"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
                "red, gr, bl, "),
            ("{#cls {=co} x }",
                {"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
                "red x gr x bl x "),
            ("{#cls {=co} _}",
                {"cls": ({"co": "red"}, {"co": "gr"}, {"co": "bl"})},
                "red _gr _bl _"),
            # Simple conditions.
            ("throw a {?condition big }party",
                {"condition": True},
                "throw a big party"),
            ("throw a {?condition big }tantrum",
                {"condition": 42},
                "throw a big tantrum"),
            ("throw a {?condition big }party",
                {"condition": False},
                "throw a party"),
            ("throw a {?condition big }tantrum",
                {"condition": None},
                "throw a tantrum"),
            ("A!{?condition B}!C!{!condition D}!E",
                {"condition": True},
                "A!B!C!!E"),
            ("A!{?condition B}!C!{!condition D}!E",
                {"condition": False},
                "A!!C!D!E"),
            # Repeats.
            ("{#a{=b}{=c}}",
                {"a": ({"b": 11, "c": 22},)},
                "1122"),
            ("{#a {=b} {=c}}",
                {"a": [{"b": 33, "c": 44}]},
                "33 44"),
            ("{#a STA{=b}STO  BEG{=c}END }",
                {"a": ({"b": 55, "c": 66},)},
                "STA55STO  BEG66END "),
            ("{#a {=b} {=c}}",
                {"a": ({"b": 7.70, "c": 88}, {"b": 99, "c": 1.234567})},
                "7.7 8899 1.234567"),
            ("{#a {=b} {=c}}", {"a": ()}, ""),
            # Repeat with variabele as last on the line.
            ("{#blop\n{=you}}",
                dict(blop=(dict(you=123), dict(you=456))),
                "123456"),
            ("{#blop\n{=you}\n}",
                dict(blop=(dict(you=123), dict(you=456))),
                "123\n456\n"),
            # A join()-like separator.
            ("{#colors {=color}{/comma , }}",
                dict(colors=(dict(color="red"), dict(color="green"),
                     dict(color="blue"))),
                "red, green, blue"),
            # More repeats.
            ("buy {=count} articles: {#articles {=nam} txt {=pri}, }", {
                "count": 2,
                "articles": ({"nam": "Ur", "pri": 1}, {"nam": "Mo", "pri": 2})
                },
                "buy 2 articles: Ur txt 1, Mo txt 2, "),

            ("sell {=count} stocks: {#articles {=nam} &euro; {=pri}{/comma , }}",
                {"count": 2, "articles": ({"nam": "APPL", "pri": 320}, {"nam": "GOOG", "pri": 120})},
                "sell 2 stocks: APPL &euro; 320, GOOG &euro; 120"),
            # Nested repeats.
            ("Contents: {#chapters Chapter {=name}. {#sections Section {=name}. }",
                {"chapters": [
                    dict(name="Intro", sections=[dict(name="Foreword"), dict(name="Methodology")]),
                    dict(name="Middle", sections=[dict(name="Measuring"), dict(name="Calculation"), dict(name="Results")]),
                    dict(name="Epilogue", sections=[dict(name="Conclusion")])
                    ]
                 }, "Contents: Chapter Intro. Section Foreword. Section Methodology. "
                    "Chapter Middle. Section Measuring. Section Calculation. Section Results. "
                    "Chapter Epilogue. Section Conclusion. "),
            # Condition with repeat.
            ("Dear {=name}, {?market Please get the following groceries:\n"
                "{#groceries \tItem: {=item}, {=count} pieces\n}}"
                "{?deadline Please be back before {=time}!}",
                    {"name": "Joe",
                     "market": "True",
                     "count": 5,
                     "groceries": [dict(item="lemon", count=2), dict(item="cookies", count=4)],
                     "deadline": True,
                     "time": "17:30",
                      },
                 "Dear Joe, Please get the following groceries:\n\tItem: "
                 "lemon, 2 pieces\n\tItem: cookies, 4 pieces\nPlease be "
                 "back before 17:30!"),
            )

        for tems, temv, expected in goodcases:
            tem = Ovotemplate(tems)  # tem.pprint()
            self.assertEquals(tem.render(temv), expected)

        ''' TODO: This still needs some work - sensible error reporting.
        badcases = (
            ("{#a {=b} {=c}}", {}), # required variables missing
            ("=a}", dict(a=42)), # missing opening {
            # ("{=a", dict(a=42)), # missing closing {
            )
        global exceptionless
        exceptionless = False
        for tems, temv in badcases:
            self.assertRaises(Exception, Ovotemplate(tems).render(temv))
        '''


def test_performance():
    """Ovotemplate and Jinja2 go head-to-head!
    Result for nr=150 on my MacBook Air:
        Ovotemplate: 467MB produced in 66.237 sec
        Jinja2: 470MB produced in 156.205 sec
    """
    import time
    nr = 40 # Set to 2 to view the expanded templates as well.
    books = []
    d = {"books": books}
    for booknr in range(nr):
        chapters = []
        book = dict(title="%d bottles of beer" % booknr, toc="This will be the table of contents.", chapters=chapters)
        books.append(book)
        for chapternr in range(nr):
            sections = []
            chapter = dict(title="%d. How to drink beer" % chapternr, intro="This will be an intro", sections=sections)
            chapters.append(chapter)
            for sectionnr in range(nr):
                section = dict(title="%d. Procedure" % sectionnr, text="This will be an explanation of how to drink beer.")
                sections.append(section)
    tem = Ovotemplate("""
        {#books
            <h1>The Book Of {=title}</h1>
            <p>{=toc}</p>
            {#chapters
                <h2>Chapter {=title}</h2>
                <p>{=intro}</p>
                {#sections
                    <h3>Section {=title}</h3>
                    <p>{=text}</p>
                }
            }
        }
        """)
    start = time.time()
    res = tem.render(d)
    dur = time.time() - start
    print "Ovotemplate: %dMB produced in %.3f sec:" % (len(res)/1024/1024, dur)
    if nr < 3:
        print res
    #
    import jinja2
    from jinja2 import Template
    tem = Template("""
        {% for book in books %}
            <h1>The Book Of {{book.title}}</h2>
            <p>{{book.toc}}</p>
            {% for chapter in book.chapters %}
                <h2>Chapter {{chapter.title}}</h2>
                <p>{{chapter.intro}}</p>
                {% for section in chapter.sections %}
                    <h3>Section {{section.title}}</h3>
                    <p>{{section.text}}</p>
                {% endfor %}
            {% endfor %}
        {% endfor %}
        """)
    start = time.time()
    res = tem.render(dict(books=books))
    dur = time.time() - start
    print "Jinja2: %dMB produced in %.3f sec:" % (len(res)/1024/1024, dur)
    if nr < 3:
        print res

if __name__ == "__main__":
    # For the usual unittests:
    unittest.main()

    # Uncomment for a benchmark comparison:
    # test_performance()

Comments

Michiel • 21 Dec 2010

This is not complete yet!

Michiel • 25 Apr 2011

However, it is already useful.

Cees Joppe • 14 Aug 2011

Thanx for sharing your clever template Michiel!

Pavel • 26 Nov 2011

GREAT engine! One question: How can you enter a '{' or a '}' into the template for expansion in the out? Thanks!

Bogomil • 20 Mar

I do the following change on rendering all children: output = [] output = [child.render(vars, last, level+1) for child in self] return "".join(output) Now the string operations are much, much faster

Michiel • 6 Sep

Bogomil, I've tried that too, but I get only a marginal speedup (19.5sec instead of 20sec on a *very* large template). I guess Python's string concatenation is very optimized. I also profiles the code and most of the time is spent in the enumeration of child nodes, and after that, converting numbers to strings. Idea for optimisation: make it possible to have the template work with strings only, so that it can always assume the input are string values.

Leave a comment

name (required)



content last edited on September 6, 2014, 18:19 - rendered in 81.74 msec