Sneaking Ruby Through Google App Engine (and Other Strictly Python Places)
Despite all the clicking that has been done on a certain star.gif, I really don’t think Ruby is coming to Google’s App Engine. Google doesn’t have much of a toe in the Ruby scene and App Engine is clearly entrenched in Python technologies (such as Pypy and WSGI). Well, cripes.
But, for my purposes, I am really enticed by App Engine, because it gives beginners a supremely cinchy way to make a web app.
So, maybe let’s just convert Ruby 1.9 bytecode to Python 2.5 bytecode and decompile. In other words: put in a .rb and get out a .pyc. Yes, could work?
A Drop of Puts
Let’s start with converting the most trivial Ruby app to an App Engine script. A translation of the intro code in the App Engine docs.
puts "Content-Type: text/plain"
puts
puts "Hello, world!"
If you’ve got Ruby 1.9 installed, you can output the bytecode for this script by looping through the arrays that come out of VM::InstructionSequence
. Here’s the code from my rb-compile
script:
#!/usr/bin/env ruby
iseq = VM::InstructionSequence.compile(IO.read(ARGV[0]))
iseq.to_a.last.each do |inst|
p inst
end
When I run ./rb-compile hello.rb
, the bytecode dump is:
1
[:putnil]
[:putstring, "Content-Type: text/plain"]
[:send, :puts, 1, nil, 8, nil]
[:pop]
2
[:putnil]
[:putstring, ""]
[:send, :puts, 1, nil, 8, nil]
[:pop]
3
[:putnil]
[:putstring, "Hello, world!"]
[:send, :puts, 1, nil, 8, nil]
[:leave]
The numbers are line numbers. I’m going to skip those for now, but Python bytecode has a place for those as well. (And, in fact, you can read about all of that in Ned Batchelder’s The Structure of .pyc Files.)
We’re just going to morph these messages into Python method calls. But one problem with this bytecode: the receiver of puts
is nil
. You see all those :putnil
lines? Since there’s no receiver, Ruby sees these as Kernel methods.
Instead, when pythongineering this code, let’s send all these methods to a Kernel
class, where we can store the Ruby built-ins. The resulting py-bytecode for a single method call will look like this:
36 LOAD_NAME 0 (Kernel)
39 LOAD_ATTR 1 (puts)
42 LOAD_CONST 4 ('Hello, world!')
45 CALL_FUNCTION 1
This is really the largest challenge when converting rbc to pyc. The ordering of things. In Ruby, the message name is part of the :send
instruction. However, in Python, the method name is popped on to the stack, right after the receiver.
But, hey, that’s all the fun of it, right?
And, so, we store in Kernel.py:
def puts(*args):
for x in args: print x
if not args: print
Getting Unholy
Okay, to watch the above take place, you’ll need to download Unholy, the Ruby-to-PYC converter I’m working on. The project is here. HEED THIS: It needs Ruby 1.9 and Python 2.5.
$ git clone git://github.com/why/unholy.git
$ cd unholy
$ vim hello.rb
$ bin/unholy hello.rb
$ PYTHONPATH=python python hello.rb.pyc
Okay, great, we’ve got some Ruby compiling to Python bytecode! (The PYTHONPATH var is there to get our Kernel.py loading.)
Unfortunately, you can’t just upload the .pyc files to Google’s machines. How about we generate some Python source code for this?
So, I’m using a patched version of decompyle, which still has some issues, but let’s get it installed.
$ cd unholy/decompyle
$ python setup.py build
$ sudo python setup.py install
$ decompyle hello.rb.pyc > hello.py
And, there you are, the hello.py
should amount to:
# emacs-mode: -*- python-*-
import Kernel
Kernel.puts('Content-Type: text/plain')
Kernel.puts()
return Kernel.puts('Hello, world!')
# local variables:
# tab-width: 4
Yeah So Puts is Useless
Now, I admit, I’m not that far into this idea. And, you know, I can’t see this thing holding my attention for another four hours. But I am close to running this script. The main problem is inheriting from webapp.RequestHandler
, which Ruby sees as a chained Kernel method call.
The code I’m working off of looks like this so far:
import "wsgiref.handlers"
from "google.appengine.ext", "webapp" => :WebApp
class MainPage < WebApp::RequestHandler
def get
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('Hello, webapp World!')
end
end
def main
application = WebApp::WSGIApplication.new([tuple('/', MainPage)])
wsgiref.handlers.CGIHandler().run(application)
end
if __FILE__ == $0
main
end
So, here’s what I’d do if I were you and I (playing you) wanted this to be perfectly class act. I would call up the JRuby guys and get them to write a compiler backend for targetting Python bytecode. I’ll bet they could crank this out in two licks of an anteater’s tongue.
What amazes me is how close Ruby 1.9 bytecode and Python 2.5 bytecode are. Some things translate almost directly. It is completely obvious that Koichi took his cues from Python. Storing argcount, nlocals, stacksize first. Marshalling bytecodes. Storing classes and methods as nested bytecode fragments.
And, really, if that’s true (and I vouch that it is truly, truly true,) then how are Python and Ruby still on separate runtimes? All of these bogus scaling wars and indented code battles are a huge waste of time. Do we still have to be better than each other in 2008? No way, Rufus said, “Be excellent to each other.” Very slight (albeit legit) syntax choices, people.
Now, imagine a hypothetical potion
:
$ potion hello.py
HELLO FROM PYTHON
$ potion hello.rb
KONNICHIWA FROM RUBY
Neither of us stands a chance against Javascript. Why persist with this pitiful feud?
Now begin the comments …
Kumar McMillan
This sounds like fun. Using JRuby would be less fun ;) BTW , if you want to know Python opcodes necessary for implementing ruby-style anonymous blocks, Fuzzyman made an experiment that seemed to work: http://www.voidspace.org.uk/python/articles/code_blocks.shtml
mister t
you are sick!
Dr Nic
And then we could use all the Python libraries, and we could work at places only Python is loved like Google, and on our CVs we could put “Expert Python” under “Expert Ruby”.
Dag
You, sir, are both a scholar and a gentleman.
L
Don’t you mean parrot?
Charles
This is beautiful, horrible, and awesome.
I wonder how long it’ll take for parrot to become potion…
Tijn
Now, imagine requiring/importing libraries from that other language. It would be like using swig without actually ‘using’ swig :-D. This would make me happy!
Moreover, afaik it is totally possible—even trivial—when Ruby can be compiled to py-bytecode (…garbage collection is different, do we think libs depend on a certain gc-algorithm?)
Funny
Already seven comments and nobody asks if Rails can be translated with unholy ;-)
James Block
Speaking as a Python guy, I agree with you that the Python and Ruby camps fight way more than they ought to. For whatever reason, I’ve never liked Ruby’s syntax (Maybe it’s that I learned Python first? Who knows), but I’ve always respected that it’s a damn fine language and people who choose Ruby are still making a Good Choice. I don’t like Rails much (or Django, for that matter), but I always try to remember that Ruby != Rails.
Potion sounds absolutely awesome. The best part of a combined runtime isn’t really the code savings; it’s the potential for a good bytecode optimizer. With the best and brightest people from both the Python and Ruby communities working on an optimized runtime, the sky’s the limit for what could be done.
Avi Bryant
Can unholy run rails? Kthxbye.
_why
L and Charles: My concern with Parrot is that it (like .Net and Java) is a people pleaser, promising everything to everyone, eager to serve every language equally. Well, except they each have their true intended targets: Perl 6 for Parrot, C# for .Net and Java for the JVM .
To be fair, general purpose VMs do work. They do perform computations! JRuby has really pulled off some amazing things. I have no power of prediction, but Java will probably never be the ultimate platform for Ruby. As long as Java is outside the kernel, it seems the most desirable platform (for purposes of speed, linkage and lightness of distro) will be native. And you can’t just go with a bare interpreter, so I think the strategy will continue to be a VM, tailored to the language.
Which is probably a good argument for Ruby sticking with its own VM, despite its proximity to Python’s. Fortunately, arguments have no place in this friendly land of peace.
Funny and Avi Bryant: Instead of asking “can unholy run Rails?” try asking “can unholy run 4 plus 7?” And my answer to that is: no, no, no, no, it probably can’t!!
evan
Be sure to use “3 + 4” as your first program. Smalltalk linage must be upheld.
4+3
3+4
Red Pie
Ruby —> Python —> Effort City —> Bizarre Software State
markus
_why for president!
_why for president!
topfunky
I’m looking forward to running Campyng on Google App Engine.
sam
It seems to me that you’re doing part of the work that PyPy is doing (interpreting the bytecode then compiling down to the target). They’re working with Python bytecode → other things, but it’d certainly be interesting to compare and contrast your work with theirs.
IdeaHamster
My oh my oh my…what to do? Should we add to the list?
I love all of the innovation, but the state of things makes me worry that we need a specification lest we loose sight of that thing we like to call “Ruby”
failrate
Ruby on the server is nice, but how about Ruby on ECMAScript also?!
I saw this floating around the other day, and ’tis most awesome: http://hotruby.accelart.jp/
mvalente
>Neither of us stands a chance against Javascript.
Thats right. I'm working on it : "CoffeePie":http://coffeepie.dyndns.ws/ -- MV>
Brian
The Goog App Engine is a GREAT way to get up and running with WebApps – fast and fun. It was tough to put down Ruby and pick up Python for just this purpose, but this looks like a promising reunion. Peeking behind the runtime curtain into the bytecode is educational too. It seems like Ruby and Python have just grown closer over the years. Rufus has it: “Be excellent to each other.”
gekoudi
hello
gekoudi
hello world
Andrew
Excellent work! I had to do a sudo apt-get install python2.5-dev on ubuntu to get decompyle working
testing
this comment box.
sorry.
Elia Schito
Wow!!!
If somebody will end up with a Python to Ruby translator i will happily build a webapp like blahblahfish.com to see ruby code through Python and vice-versa ..I’ll write it either in Ruby (using Django) or in Python (with Rails). (See This Comment Through Japanese)
pedro mg
topfunky’s +1
Camping @ AppEngine, would be sweet
Obama
We want change…we want change…we want change…we want change, and…
Yes we can!
MenTaLguY
Compiling to the Python VM is … well, actually it’s quite an accomplishment. But I think the main thing for practical use is going to be figuring out how to deliver, package, and use the Ruby core classes and libraries in the context of a Python runtime environment.
Tombone
Elia’s link to his mangled comment didn’t work. Here it is:
click
Tzury
highly appriciateed
yang
Hi, I’m actually highly interested in just the patched decompyle itself, which could be an immensely valuable standalone tool. I tried it out on some python-2.5 .pyo files, but it didn’t get very far. Do you have any plans to do further work on this? If so, do you have an issue tracker of some sort where I can report with fuller details of the failure? Thanks!
_why
yang: The patches to decompyle were done by the depyc project. Take up any issues with them.
Comments are closed for this entry.