It was a “JVM performance week” in JRuby-land. It all started when some recent code changes is JRuby caused an interesting performance degradation with JDK6 on multi-core CPUs. See the original post to the [jvm-l] list and extended coverage of what happened on Charlie’s blog. So, while following the discussion and participating, I learned a few new things and got some interesting additional info and links that I’d like to share:

1. Let’s start with basics. John Rose posted a good starting guide for those who’d like to micro-benchmark things on JVM: So You Want to Write a Micro-Benchmark.
2. Performance techniques used in the Hotspot JVM wiki entry with the list of optimizations JVM does, sweet and to the point.
3. Fundamental work by Ulrich Drepper: What Every Programmer Should Know About Memory, a MUST read, but painful
4. Deep dive into assembly code from Java blog post by Kohsuke Kawaguchi, on how to print out the assembly code that JIT is producing.
5. Examining generated code with OpenJDK 7, exciting stuff!
6. Polymorphism Performance Mysteries Explained, the Issue #158 of The Java Specialists’ Newsletter.
7. How fast is Java Volatile? or Atomic? or Synchronized?, with nice charts and comparisons.
8. Escape analysis can help optimize synchronization by Brian Goetz.

Enjoy!

The Rubyspecs, an extensive set of conformance tests for Ruby platform, is a great tool to enforce the compatibility of JRuby (and Rubinius, and IronRuby). But the specs are currently being mostly run on Linux/BSD/MacOS (against Ruby, JRuby, Rubinius), and are rarely (if ever) executed in Windows environment. Until now, that is.

I tried to run them on Windows some time ago and the mspec, the testing framework that powers Rubyspecs, had some issues with Windows paths. Since then, the issues have been resolved and the Rubyspecs are functional on Windows.

I’ve run them against JRuby, found 4-5 bugs right away (crashes in calls that are not supported on Windows even in Matz Ruby), fixed those, and was able to run the specs to the completion! There are couple of hang-ups here and there, and lots of failures due to fact that rubyspecs were never used on Windows and don’t expect Windows-specific behavior.

I guess, that’s the next big thing, to go over all the specs and adjust them to take Windows specifics into account when needed.

To my surprise, Git is becoming one of the most popular source control systems in Ruby community. New blog entries on how to use Git are popping up all over the place and the amount of excitement is just very unexpected. Who would have thought that source control tools might be so exciting! Well, here are my recipes on how to use Git with Ruby/JRuby subversion repositories.

Personally, I started using Git about 6 months ago, since Rubinius is under Git, and I really wanted to get access to “rubyspecs” which are part of Rubinius repository. At first, I really disliked Git just because the command line is rather complicated, and it takes some time to get used to it and to get familiar with basic Git concepts. After a while, I noticed that I started using Git more and more, and slowly but surely the full power of the tool just started to amaze me. Switches between branches are LESS THAN A SECOND, instant access to the entire history, powerful branch merging, git grep, and the list could go on and on.

So I started to think about switching to Git as my main SCM tool, even on projects that are under Subversion. Again, the git svn turned out to be very capable and complete tool, allowing bidirectional access to subversion repository from Git. Basically, git svn allows to clone the subversion repository with ENTIRE HISTORY into Git repository, and the changes to that Git repository can be pushed back to main Subversion one. On top of that, one can use Stacked Git to maintain a series of patches. This is very useful in case when there is no write access to the main repository. So the patches are handled by Stacked Git and are being sent out to the maintainers via email, or attached to the bug reports. And that’s how I worked on JRuby until I got the commit rights:

1. stg pull svn - get the latest sources from the main (svn) repository, this command will first pop all your currently applied patches, get the latest sources, and re-apply the patches on top of them.
2. stg edit - create new patch, provide description
3. hack, hack, hack
4. stg refresh - record the changes into the patch
5. stg export -p - export the patches into standalone files
6. send the patch to the maintainers

It’s sooo much easier to maintain the patches that way rather than just having the diff files flying around. First, the patches are always against the latest sources, and never outdated (well, at least when somebody changes something in the main repository that conflicts with the patch, you’ll see the issue immediately during the step #1 above, and can correct it right away). Second, it’s trivial to push/pop patches, reorganize them, test with and without them, easy to see which ones applied and which ones are not, etc.

For those who are serious on working on some open source project which is currently under subversion, I recommend to investigate git-svn and stgit in more detail.

Now, here are a couple of examples on how to create a Git repository out of public subversion one.

Let’s start with Ruby itself:

git svn clone -Ttrunk -ttags -bbranches http://svn.ruby-lang.org/repos/ruby ruby.git

Warning: This would take about 12 hours to complete! For those who don’t want to wait that long, here’s the repository I’ve already created: ruby.git.tgz (md5sum: 576b5667affe040fd33478ea074c13b8). Think about it, these 60mb contain entire Ruby history! This is LESS than size of subversion’s working copy! I measured the time to switch between Ruby 1.8 and Ruby 1.9 branches, that’s 500.000 lines changed in 2000 files, and it takes on my Ubuntu Linux about  1.5 SECONDS. git-grep of entire 1.9 branch is 0.2 seconds. git-log of entire 1.9 branch is 0.3 seconds, from the very last revision to the revision #1, is 0.3 seconds. What else can I say?

Now, the JRuby repository:

git-svn clone -Ttrunk/jruby -ttags -bbranches http://svn.codehaus.org/jruby jruby.git

If you have commit rights to JRuby repository, just change HTTP to HTTPS. The process would take about 2.5 hours (and I saw reports that it can be interrupted and then resumed too).

Once you have the local Git repository, you’ll probably need to do the following (when your are in the root of the repository) :

   1: git gc –aggressive –prune

   2: (echo; git-svn show-ignore) >> .git/info/exclude

This would compact the repository further, and will set the proper excludes.

This is just a very basic outline of the process, so if you’re interested, do check out the official documentation and feel free to ask questions.

I’ve learned my lesson the hard way. I’ve been running this blog on the older version of Wordpress, and some day it got hacked, and hidden spam links were inserted into the blog entries. I must say that was brilliant, nobody could see the spam links, but the search engines were indexing them. Luckily, the problem was visible at least in the RSS aggregator. So I started digging trying to figure out what’s going on and to my horror it turned out that these kinds of hacks are very common, especially against the outdated versions of Wordpress:

1. Detailed Post-Mortem of a Website Hack Through WordPress
2. Support » Weird and Dangerous
3. Justaddwater.dk hacked
4. Another Day, Another WordPress Hack

I checked with Wordpress folks on the #wordpress IRC channel, and was advised to purge the compromised install and redo it from scratch, and I did…

And the main lesson here is to pay attention to new Wordpress releases and upgrade when new security update is out. This entry sums it up nicely. On top of that, I decided to use the BARE MINIMUM of external plugins to minimize the risk.

Just in time, a new version of Wordpress has just been released, v2.5, so I took the opportunity to upgrade, since quite some changes were specifically improving the security situation. Also, I decided to change the theme for the blog. Using the default one was getting pretty old, too damn many blogs use it!

And so far, I like the new version, the admin interface is much cleaner and useable. But I’m not really sure that some “usability” improvements are useful, things like one-click plugin updates. After the security breach, I’m paranoid on this issue, and I’d like to minimize what can be done via web interface. Yes, doing manual updates via command line is not as fun, but I’d prefer it to stay that way.

P.S. If you’d like to make sure your blog is not hacked, just take a look into HTML code for your latest blog entry and make sure there are no hundreds of links at the end of it.

For the past 5 months I’ve been involved in JRuby project, since it’s a perfect match for my interests: great object oriented scripting language on top of powerful Java platform. And so far, I’m enjoying every second of my time with JRuby! So I thought that I might just post this entry and describe how to contribute to JRuby effectively and comfortably, based on experience and what worked for me.

First major rule: Don’t be shy and submit the bug reports. It’s straightforward, just go to the Jira page for JRuby, get the account and follow the “Create a new issue in project JRuby” wizard. Sure, some bugs would have better chances to be accepted and fixed than others. Clear, precise description on what’s going on, with specified environment (which OS, which JRuby version) is always a plus. Short, reproducible test case attached to the bug would be great! Having a patch that fixes the bug is just perfect. History shows that bugs with test cases and with patches are being resolved much faster, which is not a surprise.

Second rule: Talk to core developers in case when you have some problems/issues with JRuby, or even if you just have some questions. And the best way to do that is to hang out on IRC channel #JRuby (more info here). This is probably the most useful way to be involved, since the core developers and many active JRuby users are always there, 24 hours a day. That heavy use of IRC was kind of unexpected for me, I was assuming that most of communication happens over email and thought that IRC is an outdated way to communicate. And I was very wrong. Now, I can’t even imagine how some other open source projects get away without using IRC. Seriously, IRC is the way. But there are also mailing lists if you insist, they are just being used less actively.

Third rule: Get the sources and start hacking. The sources are under Subversion, but many been using git-svn successfully as well. To build JRuby is very easy, just invoke ant, and that’s it. Everything that’s needed for the build is already in the repository, and there is no need to download any 3rd party dependencies. All you really need is the JDK and Ant installed. Once you have the fix, there are couple of ways to test it, to run unit tests via “ant test” and to run “rubyspecs” via “ant spec”. Also, most of the changes should be accompanied by the new unit tests that verify/validate the new behavior.

Probably the easiest way to find stuff to work on is to follow “rubyspecs” and try to make JRuby pass those spec tests that currently fail. The rubyspecs is a set of conformance/compatibility tests currently maintained as part of Rubinius project (to be extracted into stand alone project sooner or later). I’ll probably post more about rubyspecs in the future since for me this is an incredibly important piece of overall Ruby compatibility story. Currently, you can easily see which rubyspec tests are failing with JRuby and hence are excluded, just invoke “ant spec-show-excludes”.

Another great way to help out not only JRuby but other Ruby implementations is to start writing new rubyspecs. The Rubinius folks are very open about that and after few good submissions would probably offer the commit rights.

Exactly one year ago, ME Framework has been released to the public under GPLv2 license. A LOT happened during this first year in the open source. And here are the most notable events and milestones:

• Major upgrade version ME Framework 1.2 has been released. (See what’s new).
• Minor bug-fix version ME Framework 1.1.2 has been released. (See what’s new).
• License improvement: We added “Classpath Exception” to the GPLv2.
• Completely new Developer’s Guide for 1.2 has been created.
• 4 sample test suites have been created, to simplify the first steps for the new users.
• JavaOne 2007 session devoted to ME Framework.
• Created a prototype for ME Framework NetBeans Development Module. More info about the module here and here.
• 1460 commits to the source repository.
• 641 messages posted to the ME Framework discussion forum.
• 12 engineers and 2 doc writers were contributing (most are part-time). THANKS!

And most importantly, it was fun! OK, now on to the second year, with hope that it will be even crazier!

We (the ME Framework team) are getting closer and closer to the release date of ME Framework 1.2, and I thought that it would be useful to provide a high-level list of most important changes, improvementns and new features that we implemented in this upcoming release.

• New Test Export feature:
  • Export of Tests with the sources (CLDC and MIDP modes)
  • JAD files are now generated for exported tests (MIDP mode)
  • Produces Ant builds to easily rebuild the tests
  • Produces Netbeans projects (allows to easily modify, build, run and debug the exported test right from the Netbeans).
  • Distributed tests can now be exported too.
  • OTA provisioning server for exported tests (ISSUE #6)

• New “Debug Agent” for MIDP, that provides tracing and debugging information on the device screen, i.e., the name of the currently executing test. This agent also alows to manually cancel long-running tests.

• New “Debug Agent” in MIDP test export mode, alows to run the exported tests and to see and traverse the test results right on the device. (ISSUE #106).

• Created test execution over generic/pluggable communication channel (in CDC mode only). This allows to execute tests with help of a new Agent that uses the pluggable CommToJavaTest service to communicate with the Javatest harness via variety of different protocols.
  • Standard Interview is updated as well
  • Added Serial line based communication
  • Added Socket-based communication
  • Also, HTTP- and UDP-based communications present as well

• Major DTF (Distributed Test Framework) update.
  • Removed code duplication
  • Added support for CDC mode
  • See “DTF Update” in the Changelog for more details

• Test Suite hierarchy updated: Introduced a service based TestSuite classes which delegate starting/stopping services to a service manager and corresponding service classes. Service classes are created for passive agent, Http, Https, CommService, messaging and test bundling.

• Various performance improvements for the test execution in CLDC/MIDP mode (faster JAR file creation, faster signing, I/O buffering).

• New DTF (distributed test framework) interview (ISSUE #72)

• ME Framework’s standard Interview reorg:
  • Better Debugging options (ISSUE #2)
  • Better interview’s visualization
  • Fixed some interview usability and inconsistency issues (ISSUE #47)

• Better identification of Framework JAR files implemented. Invoke java -jar j2mefw_jt.jar to see a list of all JAR files, with checksumms and the name of the version and the build. (ISSUE #97).

• Better error messages in various places (ISSUE #73, ISSUE #84, ISSUE #78, ISSUE #170, ISSUE #168, ISSUE #159).

• Moved common MIDP interactive and OTA classes to the framework (ISSUE 6407297).

• Corrected Framework’s handling of the testsuite.jtt entities:
  • interview
  • finder
  • declaratively enabling DTF

• Three Samples added
  • SimplestTestSuite (bare minimum to execute automated tests)
  • SimpleTestSuite (features automated and distrubuted tests in MIDP and CDC mode)
  • AdvancedTestSuite
  • The samples reside in the “samples” directory, and can be built as part of ME Framework build via “ant samples” command.

• ME Framework JAR files can now be placed in any location, not “lib”, as was previously required.

• SignatureTest Interview added.

• Added common interview questions for CDC-based test suites.

• Improvements in the TestBundler (CLDC and MIDP modes):
  • Improvements in handling of testClasses.lst (ISSUE #5064774)
  • The main test class is now being automatically bundled, there is no need to specify it in the testClasses.lst

• Better out-of-box experience and better customization with ME Framework:
  • Better default values selected without the need to manually specify them (correct Script, depending on the test execution mode) ISSUE #174
  • BasicTckInterview is now much easier to customize and extend if needed
  • ISSUE #156: “Remote” test description filed should not be mandatory for distributed tests
  • ISSUE #157: Passive agent should be an optional component of the distributed test suite.

• IPv6-related bugfixes (ISSUE #92, ISSUE #96, ISSUE #108)

• Preverifier Interview added

• Transition to JDK 5.

• As usual, there were a lot of various bug fixes as well.

During the work on ME Framework 1.2, we’ve produced three “development” releases with snapshots of our current work. Here are the announcements with detailed change logs:

• ME Framework 1.2 dev release 1
• ME Framework 1.2 dev release 2
• ME Framework 1.2 dev release 3
  

For some reason, code review tools topic is my “soft spot”, and I keep watching the area hoping to find the ideal solution. Until now, the closest thing to my ideal was Mondrian code review tool by Python author, Guido van Rossum. The excellent Mondrian video presentation is here, and I highly recommend it for everyone who interested in code review tools. Too bad, the Mondrian is tightly tied to Google infrastructure and is not open source.

But today I saw the announcements (here and here) by VmWare engineers who just published a new code review tool, Review Board, written also in Python, but unlike the Mondrian, under open source license, and immediately available! There is also an online demo site, to play with the new tool. The Review Board supports Perforce and Subversion (for now), and looks great.

I’ll play a little bit more with the Review Board, and report back.

Here’s a list of cqME, JT harness and ME Framework bloggers that I’m aware of:

• Mikhail Gorshenev
• Alexey Popov
• Alexander Glasman
• Vladimir Sizikov (yeah, that’s me)

Mikhail has just started blogging. Misha, welcome to the club.

With the recent fall of prices on Intel’s Core Duo processors, it’s that time of the year when I upgrade my hardware. I’ve been reading literally everything that I can find on the net about the recent hardware: reviews, blogs, articles, magazines, you name it.

And here’s the line-up:

• Intel Core 2 Duo 6600
• Scythe Infinity SCINF-1000 CPU Cooler
• Coolermaster Stacker STC-T01 Case
• Asus P5B-Deluxe Motherboard
• 4 GB of DDR2-800 memory
• Any recent video card with Dual-DVI outputs
• Westen Digital “Raptor” Hard Drive (10000 rpm)

Processor choice was easy. You can’t go wrong with the Core 2 Duo CPUs, they are much faster and more energy efficient then AMD offerings. The 6600 was the cheapest CPU with 4mb of cache, and I really wanted to get as much cache as possible - my (unfounded) theory is that big cache should be nice for various build/compiles that I do all the day. Note: Intel recently released lower-end CPU models with 4Mb cache too, if you want to be cheap.

Core 2 Duo is extremely overclockable, from 266Mhz front bus up to 400Mhz! So, the great cooler is in order. And the Scythe Infinity seems to be in the sweet spot - it cools great, especially with 2×12cm low-speed coolers and it’s very quiet. The only downside - the cooler is huge!

Well, with huge Cooler one needs a huge case! Personally, I selected Coolermaster Stacker STC-T01, it looks nice, it has lots of space, and I can use quiet power supply and 12cm low-speed fans in it.

So, the plan is to have highly-overclocked PC (up to 400Mhz front side bus), and for this I need a very solid motherboard. Based on all the reviews, the Asus P5B-Deluxe is the king! This board, it seems, is de-facto standard for seasoned overclockers. And, personally, I always loved Asus boards.

Next, the memory. Obviously, 4GB is the bare minimum now. Java IDEs can be quite a memory hogs, and running various OSes in emulators very quickly eats up all the memory. A warning: in WinXp and WinVista (32-bit), only about 3GB can be used, which is funny - we reached yet another memory barrier. Hopefully, there will be no “memory expanders” and “emulators” like the last time with 640kb in DOS. Seriously, the faster everybody switches to 64 bit the better. But based on various reviews, it seems that 64bit versions of Microsoft OSes are not the best option right now - too much quirks and incompatibilities. I’ll go with Linux/Solaris dual-boot, thank you.

The next thing - a hard drive. I’m still debating this. The RAID support in the latest motherboards (and in Intel 865 chipset, in particular) is good, and probably it’s the time to start seriously using it. As for the “Raptor” hard drives, I’ve been using them in the past, and they work great. Obviously, it’s not that easy to notice ANY performance improvements between 10000 rpm drives and the rest of 7200rpm drives, but it gives a nice warm feeling…

Video card. Since I don’t plan to play games much, there is no point to buy expensive video card. Any decent card would be fine, as long as it has two DVI outputs. Dual monitors with DVI out is the _only_ way to go for software developers. I’ve been using this setup for years and can say that switching from one monitor to two monitors was the most successful upgrade for me. I can see more, I can have more windows, editors, browsers, consoles, etc. Seriously, if you have just one monitor, do yourself a favor and buy a second one, the rest of hardware upgrades can wait!