tl;dr

This is the tale of why and how I re-wrote an ageing Java-basedcustomer & order management system in Python and how the samefeature set was implemented using only one 10th of the code base.

If it ain't broke, don't fix it

During Christmas in the year of our Lord 2007, I wrote a simplecustomer and order management system using what I found to be cooltechnology at the time:Java 6,iBatis (persistence framework),Tomcat 6 (application server),MySQL (database),Debian Linux (operating system) and (deep breath)JSP (templating),

After 1-2 weeks of coding, I had something that was usable and, aftersome initial bug fixing after going live in January, it ran happilywithout any maintenance or supervision of any kind for 7 years. Theonly time it wasn't available, was when there was a power outage inthe server room.

The customer was happy and the story could have ended here. Butsomething made the system to be completely re-written after 7 years ofblissful existence. What was it?

Change. The customer wanted new features. Which meant picking up thedevelopment and, lo and behold, I discovered how outdated the systemhad gotten when looking at it again through the goggles of 2014. Thearchitecture was pure RPC over HTTP, JSPs was such an old technologyit's wasn't even fun thinking about it,MySQL hadgone from being a hip and cool Open Source database to being acrippled step child ofOracle, but worst of all:some of the technology, most notably iBATIS, wasn't supportedanymore. Not supported meant no security updates, outdateddocumentation, a shrinking user base and last but not least: the sadfeeling of missing out on all the fun stuff happening in the world.

The obvious choice

There was a fork of iBATIS available calledMyBatis but using it meant making changesto both my code and build setup, so I started looking somewhereelse. Having usedJava as my main programminglanguage at work for more than 10 years, the natural choice seemed tobe the now nicely standardised and usableJPA forpersistence with the rest of the JEE7 stack for the other buildingblocks:Java 7,JAX-RS (REST framework) andJSF,Rich Faces orWicket (templating).

After a month or two of hacking away at this evenings and weekends, Ihad something that worked and was Java wise nicely layered,object oriented, de-coupled, unit testable and so on. It was just notfun.

Andprogramming should be fun. Especially when working for free onopen source projects like this system. So I asked myself, what would Iprefer programming in? In which language and on what platform would Ihave the most fun programming in while at the same time being able toquickly add new features? Although I'm ahuge fan of BASH, I didadmit that it wasn't ideal for developing a rich web based ordermanagement system. My choice was then easy, it had to bePython.

I've always been very fond of Python ever since learning it inuniversity but I quickly moved away from it in favour ofBASH for writing command line programsas BASH is always available on all kinds of UNIX and Linux variants(exception being HP-UX). For larger applications, my work has alwaysbeen Java related, so Python has continued to sit quietly in mytoolbox, waiting for the task at hand being right for it. And now itfinally was.

What a joy

And what a wonderful relief it was to start programming in Python!Just like coming home. I like programming in Python so much becauseits programming model maps so well with the way I think. There's somany thing that justfeels right. Natural.

Another nice welcome was to find how excellent supportEmacs has for Python. There'smany Python plugins to choose from.I settled foranaconda-mode, whichtogether withpyflake andother great plugins I already use, likeflymake,auto-complete-modeandprojectile, give meeverything I could ever wish for when coding: auto completion, on thefly syntax checking, code navigation, interactive shell forprototyping and documentation lookup.

And there was more to be rejoice about. Lots. LikeFlask. It's a lightweight web frameworkwhich is just wonderful to work with. It's blissfully free of bloat orhalf thought out ideas, incomplete documentation or patchy librariesthat haunt so many other frameworks. You know, the frameworks whichlet's you easily do mundane tasks, but don't scale up to worldapplications. Your applications. I cannot recommend you enough to tryout Flask. It's strikes an impressive balance between simplicity andfeature richness.

Another affable acquaintance was that ofJinja. It's templating done right. Againstriking a good balance between being easy to use and having all thefeatures you need to cover all your project needs. As an example ofhow well behaved Jinja is, take a look at this error message after Idid something illegal in Jinja:

{%blockhead-title%}

And Jinja2 told me in this super useful way:TemplateSyntaxError:Block names in Jinja have to be valid Python identifiers and may notcontain hyphens, use an underscore instead.

Man, I wish other API and framwork authors would take note.

Another amazing piece of the puzzle was theWerkzeug web server which Flaskbundles. It's a lightweight server, making Flask a perfect microservice platform. If you thinkDropWizardfor Java is nice and easy, try out Flask. It's a ten times easier touse, configure and start. This is all you need to do:

  # apt-get install python-pip  # pip install flask

Then, add a REST endpoint, instantiate and run Flask:

#! /usr/bin/env pythonfromflaskimportFlaskapp=Flask(__name__)@app.route("/search")defget_search():returnrender_template("search.html")if__name__=='__main__':app.run(debug=True)

With this in place, you can start your micro service in pure UNIXfashion with:

  $ ./atelier.py

And the server was up! That's it.Nothing more. Now, go back to yourMaven based, Java andDropWizard project:

1. AddDropWizard dependency/ies to your POM
2. Create you DropWizard YAML configuration file
3. Create a Java application class to bootstrap it all.
4. Add whatever REST end points you want (the actual fun bit).
5. Run Maven to compile your code and package it all up withmvn package.
6. Start your application, e.g. with:java -jar target/application-0.1-SNAPSHOT.jar server path/to/app.yml

And this is something of the easiest you get in the java world! Now,tell me you're not already feeling homesick and want to go back tousing Flask. I for sure am!

Keep it really simple, stupid

Whenever I'm discussing with fellow Java developers, we all agree thatwe should make things as simple as possible. However, the moment weget back to our IDEs after the coffee break, we continue creating oursuper intricate, object oriented, multi layered solutions with anindefinite number of indirections.

This time, getting a clean slate from a new language and softwarestack, I set out at the other end: what's the minimum number ofindirections, layers and objects I can get away with while keeping thecode reasonably loosely coupled, easy to read, maintain and extend?

One decision I made, was to work directly on JSON structures insteadof transforming HTML forms to domain objects and then translatingthese into database tables and rows again. Since the MySQL Pythondriver has a cursor which delivers a Python dictionary, which ispractically an immutable JSON structure, I was almost home free. Andwhen discovering that the HTML form data from the web client also camein JSON wrapping, I was good to go.

"But what about type safety?" I hear you cry. Well, Idid a fewissues with this, but surprisingly few. Much of this is because Pythonis really good at doing "what you mean". It tries to be smart andunderstanding and most of the time it gets it right. If you have adate time field in the database and you instert a string with'2015-04-11', it'll happily convert it to a date time object. And soon. So far, I've spent about two hours on type (conversion)issues. Not bad, and definitely no more than I would have had if I'denforced entity objects with types. Because that's the funny thing:even with Java, JPA, Hibernate,Joda,Jadira and all that jazz, you stillventure into type problems. There's no getting away from thatyouneed to be in command of your application, your stack, no matter thetechnology you use.

So I ditched the domain objects and just used JSON, very closelyresembling the database tables. Another thing I did, was to just havetwo layers on the Python side: the data layer and the RESTlayer. Nothing in between. I actually wanted to add a middle layer,but I stalled at the last minute because I saw justhow easy it wasto understand and debug the code when having just two layers.

My architecture thus became:

view:  <Jinja HTML templates with Python objects>model: <Python code with Flask REST routing>data:  <Python code with MySQL connectivity>

Less than 1000 lines of Python code. It's easy to understand (yes, Iknow I wrote it, but still), it's easy to debug and it's easy toextend.

If the system is to grow, however, I do see some challenges with thecurrent structure. I would need to introduce some delegation in thetwo Python layers as the files shouldn't grow significantly more now(main Flask file is ~400 lines, main data file ~500 lines). Of course,I've crated own Python modules for the Jinja filters, data handling,SQL generation and configuration file parsing. Still crammingeverything in ~900 lines of codes.

To be clear: My point is not that complexity is a bad thing in itself,it' just that very often in the Java world (and I'm sure in other subcultures as well) we tend to introduce great complexity up front,something which we seldom need.Be a great master of complexity, butbe equally afraid of introducing it.

So how did you do...?

Unicode support all through the stack

Database: Set the character set on the database connection:

importMySQLdbasmdbcon=mdb.connect(self.db_host,self.db_user,self.db_password,self.db)con.set_character_set('utf8')

Date, currency formats and the like: Set the encoding togetherwith the desired locale using the standard locale module, just as youwould do on the UNIX command line:

fromlocaleimportsetlocalesetlocale(locale.LC_ALL,"nb_NO.utf8")

Make sure the HTML render characters using UTF-8: Add this to theHTML head. Typically this will be in a Jinja HTML template that alltemplates inherits:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>

Create a web service accepting multiple HTTP methods

Here, I have a REST service accepting two HTTP methods and takes aparameter from the URI template defined, passing it to the method,picking up the values from the HTML form, saves it to the DB andperforms a redirect to another method performing a GET of the nowupdated customer.

@app.route("/customer/<id>",methods=["POST", "PUT"])defupdate_customer(id):db.update_customer(request.form)returnredirect(url_for("get_customer",id=id,updated=True))

Database transactions

Making several database queries run in the same transactions androllback is so easy and elegantly done in Python, I thought it wasn'tthere! As often before,Python just works the way you want it to,taking care of you and cleaning up the mess without whining.

Here, I delete all related order items before deleting the orderitself. All in the same transaction and using prepared statements toavoid SQL injection:

def delete_order(self, id):    con = self.get_db_connection()    with con:        cur = con.cursor()        cur.execute("delete from order_item where order_id = %s", (id))        cur.execute("delete from customer_order where id = %s", (id))

It's so neat and tidy, you'd be forgiven to think you'd missedsomething.

Input validation

HTML5 has matured so much now that Idelegated most of the client input validation to it (<inputtype="email"/> is great!).

I also let the browser's HTML5 capabilities provide the data pickers.At the time of writing (2015-04-11), this means that only users ofGoogle Chrome andOpera get the date pickers, the rest must type inthe dates using good old ASCII. In both cases, HTML5 takes care of thevalidation. Wonderful!

Consistent, professional layout of web pages

I've usedTwitter's Bootstrap. Once I gotmy head around its grid system and form layout, it was nice to havesome pre-defined, well tested layout system that works on resolutionsand devices without too much hassle.

Was it all bad, the old stuff?

No, not at all. When re-writing all of the Java and JSPs in Python andJinja, I did keep the database as it was. It's not just out ofconvenience, I really liked the database modelling and kept it the wayit was. It's proof, if you needed one, that it's good to keep awayfrom mixing application code with the database, like PL/SQL.

The operating system is also something I kept. Debian is rock solidand the most wonderful Linux distribution as far as I'm concerned.

As I mentioned in the introduction, MySQL has become somewhat of astep child in Oracle's helm. I don't know what to make of it. Whatdoes Oraclewant with MySQL? Hence, I always pick one of the MySQLforks or patch sets if you will. For many years now, my favourite hasbeenPercona. Apart from being a differentdistribution, it's basically MySQL, so all the SQL scripts and datacould be kept without modification.

And of course, having implemented this system once before was ahugebenefit. Domain knowledge is crucial for developing good software andall the user feedback from the old system helped me creating a goodnew version.

Conclusion

Together, Python, Flask, Jinja and Emacs made this such a smoothexperience, bringing back the fun in programming while at the sametime implementing all the features of the old system in one tenth ofthe code: ~900 lines of Python versus ~9000 lines of Java code,excluding templating, HTML, JS and CSS.

I publish this in the hope that more people will get inspired to tryout a new stack when diving into their next big project as well astaking a step back and reconsidering all the industry standard, bestpractise and modelling principles when tackling "enterprise"challenges . Do you really need all the levels of abstractions? Allthe objects? All the type safety? All the frameworks? What does itreally give you? How many bugs do you actually avoid by adding allthese layers of complexity and indirections? And also the other wayaround: how many bugs do you think this complexity have introducedover the last 5 years in your system? I'm just asking 😊

Useful articles

• Flask:Flask quickstart;Handling a POST request in Flask.
• ParsingJSON and YAML config files;Parsing config files
• Templating:Jinja;Iterating a dictionary
• Unicode
• Python & MySQL:Excellent article on using MySQL in Python; Some notes onPython, MySQL and UTF-8