Sneaking Ruby Through Google App Engine (and Other Strictly Python Places)

May 5th 11:11
by why

Despite all the clicking that has been done on a certain star.gif, I really don’t think Ruby is coming to Google’s App Engine. Google doesn’t have much of a toe in the Ruby scene and App Engine is clearly entrenched in Python technologies (such as Pypy and WSGI). Well, cripes.

But, for my purposes, I am really enticed by App Engine, because it gives beginners a supremely cinchy way to make a web app.

So, maybe let’s just convert Ruby 1.9 bytecode to Python 2.5 bytecode and decompile. In other words: put in a .rb and get out a .pyc. Yes, could work?


A Drop of Puts

Let’s start with converting the most trivial Ruby app to an App Engine script. A translation of the intro code in the App Engine docs.

puts "Content-Type: text/plain"
puts
puts "Hello, world!"

If you’ve got Ruby 1.9 installed, you can output the bytecode for this script by looping through the arrays that come out of VM::InstructionSequence. Here’s the code from my rb-compile script:

#!/usr/bin/env ruby
iseq = VM::InstructionSequence.compile(IO.read(ARGV[0]))
iseq.to_a.last.each do |inst|
  p inst
end

When I run ./rb-compile hello.rb, the bytecode dump is:

1
[:putnil]
[:putstring, "Content-Type: text/plain"]
[:send, :puts, 1, nil, 8, nil]
[:pop]
2
[:putnil]
[:putstring, ""]
[:send, :puts, 1, nil, 8, nil]
[:pop]
3
[:putnil]
[:putstring, "Hello, world!"]
[:send, :puts, 1, nil, 8, nil]
[:leave]

The numbers are line numbers. I’m going to skip those for now, but Python bytecode has a place for those as well. (And, in fact, you can read about all of that in Ned Batchelder’s The Structure of .pyc Files.)

We’re just going to morph these messages into Python method calls. But one problem with this bytecode: the receiver of puts is nil. You see all those :putnil lines? Since there’s no receiver, Ruby sees these as Kernel methods.

Instead, when pythongineering this code, let’s send all these methods to a Kernel class, where we can store the Ruby built-ins. The resulting py-bytecode for a single method call will look like this:

36 LOAD_NAME                0 (Kernel)
39 LOAD_ATTR                1 (puts)
42 LOAD_CONST               4 ('Hello, world!')
45 CALL_FUNCTION            1

This is really the largest challenge when converting rbc to pyc. The ordering of things. In Ruby, the message name is part of the :send instruction. However, in Python, the method name is popped on to the stack, right after the receiver.

But, hey, that’s all the fun of it, right?

And, so, we store in Kernel.py:

def puts(*args):
  for x in args: print x
  if not args: print

Getting Unholy

Okay, to watch the above take place, you’ll need to download Unholy, the Ruby-to-PYC converter I’m working on. The project is here. HEED THIS: It needs Ruby 1.9 and Python 2.5.

$ git clone git://github.com/why/unholy.git
$ cd unholy
$ vim hello.rb
$ bin/unholy hello.rb
$ PYTHONPATH=python python hello.rb.pyc

Okay, great, we’ve got some Ruby compiling to Python bytecode! (The PYTHONPATH var is there to get our Kernel.py loading.)

Unfortunately, you can’t just upload the .pyc files to Google’s machines. How about we generate some Python source code for this?

So, I’m using a patched version of decompyle, which still has some issues, but let’s get it installed.

$ cd unholy/decompyle
$ python setup.py build
$ sudo python setup.py install
$ decompyle hello.rb.pyc > hello.py

And, there you are, the hello.py should amount to:

# emacs-mode: -*- python-*-
import Kernel
Kernel.puts('Content-Type: text/plain')
Kernel.puts()
return Kernel.puts('Hello, world!')

# local variables:
# tab-width: 4

Yeah So Puts is Useless

Now, I admit, I’m not that far into this idea. And, you know, I can’t see this thing holding my attention for another four hours. But I am close to running this script. The main problem is inheriting from webapp.RequestHandler, which Ruby sees as a chained Kernel method call.

The code I’m working off of looks like this so far:

import "wsgiref.handlers"
from "google.appengine.ext", "webapp" => :WebApp

class MainPage < WebApp::RequestHandler
  def get
    self.response.headers['Content-Type'] = 'text/plain'
    self.response.out.write('Hello, webapp World!')
  end
end


def main
  application = WebApp::WSGIApplication.new([tuple('/', MainPage)])
  wsgiref.handlers.CGIHandler().run(application)
end

if __FILE__ == $0
  main
end

So, here’s what I’d do if I were you and I (playing you) wanted this to be perfectly class act. I would call up the JRuby guys and get them to write a compiler backend for targetting Python bytecode. I’ll bet they could crank this out in two licks of an anteater’s tongue.

What amazes me is how close Ruby 1.9 bytecode and Python 2.5 bytecode are. Some things translate almost directly. It is completely obvious that Koichi took his cues from Python. Storing argcount, nlocals, stacksize first. Marshalling bytecodes. Storing classes and methods as nested bytecode fragments.

And, really, if that’s true (and I vouch that it is truly, truly true,) then how are Python and Ruby still on separate runtimes? All of these bogus scaling wars and indented code battles are a huge waste of time. Do we still have to be better than each other in 2008? No way, Rufus said, “Be excellent to each other.” Very slight (albeit legit) syntax choices, people.

Now, imagine a hypothetical potion:

$ potion hello.py
HELLO FROM PYTHON
$ potion hello.rb
KONNICHIWA FROM RUBY

Neither of us stands a chance against Javascript. Why persist with this pitiful feud?

Now begin the comments …

33 comments

Kumar McMillan

said on May 5th 15:01

This sounds like fun. Using JRuby would be less fun ;) BTW , if you want to know Python opcodes necessary for implementing ruby-style anonymous blocks, Fuzzyman made an experiment that seemed to work: http://www.voidspace.org.uk/python/articles/code_blocks.shtml

mister t

said on May 5th 15:49

you are sick!

Dr Nic

said on May 5th 16:39

And then we could use all the Python libraries, and we could work at places only Python is loved like Google, and on our CVs we could put “Expert Python” under “Expert Ruby”.

Dag

said on May 5th 17:12

You, sir, are both a scholar and a gentleman.

L

said on May 5th 17:23

Don’t you mean parrot?

Charles

said on May 5th 17:34

This is beautiful, horrible, and awesome.

I wonder how long it’ll take for parrot to become potion…

Tijn

said on May 5th 17:40

Now, imagine requiring/importing libraries from that other language. It would be like using swig without actually ‘using’ swig :-D. This would make me happy!

Moreover, afaik it is totally possible—even trivial—when Ruby can be compiled to py-bytecode (…garbage collection is different, do we think libs depend on a certain gc-algorithm?)

Funny

said on May 5th 18:24

Already seven comments and nobody asks if Rails can be translated with unholy ;-)

James Block

said on May 5th 18:43

Speaking as a Python guy, I agree with you that the Python and Ruby camps fight way more than they ought to. For whatever reason, I’ve never liked Ruby’s syntax (Maybe it’s that I learned Python first? Who knows), but I’ve always respected that it’s a damn fine language and people who choose Ruby are still making a Good Choice. I don’t like Rails much (or Django, for that matter), but I always try to remember that Ruby != Rails.

Potion sounds absolutely awesome. The best part of a combined runtime isn’t really the code savings; it’s the potential for a good bytecode optimizer. With the best and brightest people from both the Python and Ruby communities working on an optimized runtime, the sky’s the limit for what could be done.

Avi Bryant

said on May 5th 19:42

Can unholy run rails? Kthxbye.

_why

said on May 5th 19:55

L and Charles: My concern with Parrot is that it (like .Net and Java) is a people pleaser, promising everything to everyone, eager to serve every language equally. Well, except they each have their true intended targets: Perl 6 for Parrot, C# for .Net and Java for the JVM .

To be fair, general purpose VMs do work. They do perform computations! JRuby has really pulled off some amazing things. I have no power of prediction, but Java will probably never be the ultimate platform for Ruby. As long as Java is outside the kernel, it seems the most desirable platform (for purposes of speed, linkage and lightness of distro) will be native. And you can’t just go with a bare interpreter, so I think the strategy will continue to be a VM, tailored to the language.

Which is probably a good argument for Ruby sticking with its own VM, despite its proximity to Python’s. Fortunately, arguments have no place in this friendly land of peace.

Funny and Avi Bryant: Instead of asking “can unholy run Rails?” try asking “can unholy run 4 plus 7?” And my answer to that is: no, no, no, no, it probably can’t!!

evan

said on May 5th 20:08

Be sure to use “3 + 4” as your first program. Smalltalk linage must be upheld.

4+3

said on May 6th 00:02

3+4

Red Pie

said on May 6th 05:36

Ruby —> Python —> Effort City —> Bizarre Software State

markus

said on May 6th 06:50

_why for president!
_why for president!

topfunky

said on May 6th 12:04

I’m looking forward to running Campyng on Google App Engine.

sam

said on May 6th 13:30

It seems to me that you’re doing part of the work that PyPy is doing (interpreting the bytecode then compiling down to the target). They’re working with Python bytecode → other things, but it’d certainly be interesting to compare and contrast your work with theirs.

IdeaHamster

said on May 6th 13:53

My oh my oh my…what to do? Should we add to the list?

  • Matz Ruby Interpreter
  • Rubinius
  • JRuby
  • IronRuby
  • MacRuby
  • Potion (?)

I love all of the innovation, but the state of things makes me worry that we need a specification lest we loose sight of that thing we like to call “Ruby”

failrate

said on May 6th 14:08

Ruby on the server is nice, but how about Ruby on ECMAScript also?!
I saw this floating around the other day, and ’tis most awesome: http://hotruby.accelart.jp/

mvalente

said on May 6th 15:41

>Neither of us stands a chance against Javascript.
>

Thats right. I'm working on it : "CoffeePie":http://coffeepie.dyndns.ws/ -- MV

Brian

said on May 6th 21:45

The Goog App Engine is a GREAT way to get up and running with WebApps – fast and fun. It was tough to put down Ruby and pick up Python for just this purpose, but this looks like a promising reunion. Peeking behind the runtime curtain into the bytecode is educational too. It seems like Ruby and Python have just grown closer over the years. Rufus has it: “Be excellent to each other.”

gekoudi

said on May 7th 01:12

hello

gekoudi

said on May 7th 01:13

hello world

Andrew

said on May 7th 02:11

Excellent work! I had to do a sudo apt-get install python2.5-dev on ubuntu to get decompyle working

testing

said on May 7th 18:56

this comment box.

sorry.

Elia Schito

said on May 8th 03:14

Wow!!!
If somebody will end up with a Python to Ruby translator i will happily build a webapp like blahblahfish.com to see ruby code through Python and vice-versa ..I’ll write it either in Ruby (using Django) or in Python (with Rails). (See This Comment Through Japanese)

pedro mg

said on May 8th 10:00

topfunky’s +1
Camping @ AppEngine, would be sweet

Obama

said on May 9th 00:15

We want change…we want change…we want change…we want change, and…

Yes we can!

MenTaLguY

said on May 9th 11:18

Compiling to the Python VM is … well, actually it’s quite an accomplishment. But I think the main thing for practical use is going to be figuring out how to deliver, package, and use the Ruby core classes and libraries in the context of a Python runtime environment.

Tombone

said on May 9th 14:28

Elia’s link to his mangled comment didn’t work. Here it is:
click

Tzury

said on May 10th 04:05

highly appriciateed

yang

said on May 11th 02:09

Hi, I’m actually highly interested in just the patched decompyle itself, which could be an immensely valuable standalone tool. I tried it out on some python-2.5 .pyo files, but it didn’t get very far. Do you have any plans to do further work on this? If so, do you have an issue tracker of some sort where I can report with fuller details of the failure? Thanks!

_why

said on May 11th 23:16

yang: The patches to decompyle were done by the depyc project. Take up any issues with them.

Comments are closed for this entry.