IBM (still) delivers more performance at lower cost – response to the Oracle’s latest (misleading) performance claims


UPDATE – June 4, 2014: Added new results for Oracle and IBM to the JOPS/core table. However I have not done the cost comparison for those results as it takes time to get hardware quote from Oracle to calculate their cost.

There is a saying in Russia: “If you want to have your glass intact, do not throw stones into other people windows” (or its English version). I am sure other cultures have similar idioms (which I would love to see in the comments section below). At Oracle OpenWorld 2013 conference Oracle had thrown more than a handful of “stones” into IBM proverbial “windows” and I would like to clarify few performance related things in this post.

Oracle posted this press-release claiming WebLogic performance superiority over WebSphere (you did not you expect otherwise, did you?). In this press-release Oracle claims to have the highest virtualized performance. In sessions during the OpenWorld conference Oracle claimed to have higher performance per socket compared to IBM Power7+ chips. Elisabeth Stahl described her point of view quite nicely in her blog posts “Guns and Butter at OpenWorld” and “Born to run benchmarks“.

I agree with Elisabeth – people do not buy sockets or cores, they buy performance. Nor do people get free sockets or cores. All of these things cost money. Oracle comes up with all kinds of metrics to measure their software. Why did they stick with performance per socket? Why not pick performance per cubic foot, or per meter of copper wire? Are WebLogic Suite or Oracle SOA Suite or other enterprise products that Oracle sells priced per socket? No, they are not. Only Standard editions of handful of Oracle products have socket based pricing and even that with restrictions – many advanced features (clustering being one example) and not all add-on options are available on Standard Edition products. So why does Oracle likes the meaningless per socket performance so much?

Another flaw in Oracle performance claims is the comparison of old IBM results to latest Oracle results. Would it be fair to compare latest Galaxy S4 to iPhone 4s? Oracle is doing exactly this with their selective benchmark comparisons. They pick what they like and dismiss what does not fit their goal. Why not compare historical SPECjEnterprise2010 results mapped over time? I looked at my previous posts that I wrote in the past 2 years responding to Oracle press releases and copied all those results into these two simple tables. The SPECjEnterprise2010 spec has more results than these shown below, but I do not have unlimited time in my day to calculate costs for every hardware spec published by IBM and Oracle, so I had to use what I had collected so far, just added Oracle September 2013 result and arranged it into simple table format:

specj_jops_core

 

DollarPerJOPS

As you can see from tables above, IBM has mostly dominated SPECjEnterprise2010 in the past 2 years in performance per core and cost per JOPS. In 2010 IBM published 10 results and Oracle published only 1 – I did not bother to do cost calculation on those results and will leave it as an exercise to the reader. Oracle tends to publish big numbers by throwing large hardware into the test, but how many customers are actually running at that level? Majority of the market is not doing 50,000+ transactions per second. Majority of the market is interested in low to medium level scale and more important – cost per transaction when you count hardware and software together. My second table above is exactly this – comparing the cost of transaction per core considering hardware and software costs. After all – enterprise software is licensed by core and this is true for enterprise versions of WebSphere, WebLogic, JBoss, tc Server and others. (FYI – IBM does have other licensing options)

Taking it all together and combining two best IBM and two best Oracle results on x86, Power and SPARC respectively, here is what we get:

  • IBM WebSphere delivers 80% more performance at almost half the cost on Power7+ compared to Oracle WebLogic on SPARC T5-8
  • IBM WebSphere is 17% faster per core on Intel Sandy Bridge at less than half the cost compared to Oracle WebLogic

SPECJGraph

Please note that costs above are calculated using WAS ND and WLS Enterprise.

Perhaps this is why customers, such as PT Bank ANZ Indonesia (and others) are migrating from WebLogic to WebSphere?

video

You can read my earlier posts comparing IBM and Oracle SPECjEnterprise2010 results here: http://whywebsphere.com/?s=SPECj

Obligatory legal stuff:

******************* Notes:

SPEC and SPECjEnterprise2010 are registered trademarks of the Standard Performance Evaluation Corporation. Results from http://www.spec.org:

Oracle WebLogic 12c on SPARC T5-8, 36,571.36 SPECjEnterprise2010 EjOPS

Oracle WebLogic 11g on SUN SPARC T5-8 57,422.17 SPECjEnterprise2010 EjOPS

Oracle WebLogic 11g on SUN Fire X4170M3 8,310.19 SPECjEnterprise2010 EjOPS

Oracle WebLogic 11g on SUN Blade Server X6270 M2 5,427.42 SPECjEnterprise2010 EjOPS

Oracle WebLogic 11g on SUN SPARC T4-4 40,104.86 SPECjEnterprise2010 EjOPS

Oracle WebLogic 11g on Dell x86 11,946.60 SPECjEnterprise2010 EjOPS

IBM WebSphere 8.5 on Power730+ 13,161.07 SPECjEnterprise2010 EjOPS

IBM WebSphere 8.5 on Power780+ 10,902.30 SPECjEnterprise2010 EjOPS

IBM WebSphere 8.5 on System x3650 M4 Intel Sandy Bridge 9,696.43 EjOPS

IBM WebSphere 8.5 on IBM HS 22 Blade 6,295.46 SPECjEnterprise2010 EjOPS

IBM WebSphere 8.5 on IBM HS 22 Blade 3,694.35 SPECjEnterprise2010 EjOPS

IBM WebSphere 8.5 on IBM HS 22 Blade 2,341.12 SPECjEnterprise2010 EjOPS



Categories: Oracle

Tags: , , , , , , ,

5 replies

  1. If you agree with Elisabeth – “people do not buy sockets or cores, they buy performance”, then why is it that you are trying to compare core performance especially when the performance are at completely different levels?

    The Oracle’s SPARC T5-8 results with 8 x CPUs delivering 36,571.36 SPECjEnterprise2010 EjOPS vs the Power780+ with the same 8 x CPUs delivering 3x fewer SPECjEnterprise2010 EjOPS @ 10,902.30?

    You are assuming that you can do a performance/core calculation and then infer a $/JOPS, but you’re not calculating the fact that you’ll need 3 x Power 780+ systems, with 24 x Power7+ CPUs to match the EJOPs performance throughput of the single SPARC T5-8 system?

    And those 3 x 8 CPU Power780+ systems will take up 24 RU of space compared to just 8 RU for the SPARC T5-8. Did you calculate the additional power consumption and space required?

    What is the performance/core of Power 780 delivering ~36,000 EJOPs? Oops sorry, its not possible or IBM hasn’t published a result with all 128-cores? Why? Does IBM want to show only best performance/core using only 4-core highest GHZ Power7+?

    Problem is, you’ve traded throughput for perf/core and once you need to go beyond 11K EJOPs, you’ll see significant drop in perf/core either requiring more systems or dropping to slower Power7+’s.

    Note that theres only a 19% perf/core difference between Oracles SPARC T5-8 and IBM’s Power 780+, yet SPARC T5-8 delivers 3x more throughput, and certainly less HW costs.

    SPARC T5-8 = 36,571.36 SPECjEnterprise2010 EjOPS / 128-cores = 285.71 EJOPs/core
    Power 780+ = 10,902.30 SPECjEnterprise2010 EjOPS/ 32-cores = 340.7 EJOPs/core

    As some wise man said: “give me enough data and, statistically, I will prove anything”

    • Phil, thanks for your comments. Let me answer your questions as well as clarify and correct a couple of your mistakes.

      Q >>> If you agree with Elisabeth – “people do not buy sockets or cores, they buy performance”, then why is it that you are trying to compare core performance especially when the performance are at completely different levels?

      A: While I agree that overall Total JOPS of these results are at different levels, however those two summary tables in my post do compare a broad range of results from IBM and Oracle. In some cases IBM has higher total # of JOPS in other cases Oracle does.

      Q >>> The Oracle’s SPARC T5-8 results with 8 x CPUs delivering 36,571.36 SPECjEnterprise2010 EjOPS vs the Power780+ with the same 8 x CPUs delivering 3x fewer SPECjEnterprise2010 EjOPS @ 10,902.30?

      A: I do not think we need to selectively pick exact same number of CPUs for comparison – the reason is that those CPUs are not created equal. Power7+ is not the same CPU as SPARC T5. However I do agree that it is quite useful to pick exact same number of CPUs if those were identical, which is exactly the case for the Oracle WebLogic 11g on SUN Fire X4170M3 8,310.19 SPECjEnterprise2010 EjOPS vs. IBM WebSphere 8.5 on System x3650 M4 Intel Sandy Bridge 9,696.43 EjOPS. These are on the same 16 cores of Intel Xeon E5-2690 CPUs and as you can see WAS runs 14.3% faster and costs 2.2 times less when calculated $/JOPS.

      There are more recent results with Power7+ machines that you should be using for comparison – and there, again IBM has almost twice performance per core. See details in my comparison tables in the post. Please do not use old IBM results to compare to new Oracle results.

      Q >>> You are assuming that you can do a performance/core calculation and then infer a $/JOPS, but you’re not calculating the fact that you’ll need 3 x Power 780+ systems, with 24 x Power7+ CPUs to match the EJOPs performance throughput of the single SPARC T5-8 system? And those 3 x 8 CPU Power780+ systems will take up 24 RU of space compared to just 8 RU for the SPARC T5-8. Did you calculate the additional power consumption and space required?

      A: Partially valid point here. One can not always extrapolate scale of the workload, however you do not have to take 3 Power7+ machines to triple the performance (that is possible, but may not be optimal in some cases – although it wont change the $/JOPS much). You could upgrade Power7+ machine. I agree that IBM has not submitted results for the high end Power7+ machines, however I am pretty sure $/JOPS and definitely Total JOPS results will be similarly compelling to existing benchmarks we have published to date.

      Q >>> What is the performance/core of Power 780 delivering ~36,000 EJOPs? Oops sorry, its not possible or IBM hasn’t published a result with all 128-cores? Why? Does IBM want to show only best performance/core using only 4-core highest GHZ Power7+?

      A: :-)) Just FYI – the Power795+ can scale up to 256 cores. I can only imagine the Total JOPS for that kind of system. However for that many cores it will be hard to find the appropriate machine capable of handling Database IO on such a scale. See the problem is that application servers are stateless for the most part, while databases keep the state and require disk IO, which is harder to scale. This becomes a question of IO (database) scalability before it is a question of app server scalability.

      Q >>> Problem is, you’ve traded throughput for perf/core and once you need to go beyond 11K EJOPs, you’ll see significant drop in perf/core either requiring more systems or dropping to slower Power7+’s.

      A: What makes you think that going beyond 11K JOPS will force IBM to drop to slower machines? Latest result proves this not to be the case. In April 2013 IBM published result of 13,161.07 JOPS on Power7+ and you can see that in the table above. Higher end Power7+ machines will produce more transactions total, it is just the matter of putting that configuration together.

      See, IBM and Oracle have two choices here:
      1) Play performance games and keep putting together bigger and bigger configurations with gazillions of JOPS (I am told these performance benchmarks take several man-months to produce)
      or
      2) Spend engineering budget improving the product.

      Which of the above two do you think is better for IBM and Oracle customers (hint – the correct answer is #2).

      Q >>> Note that theres only a 19% perf/core difference between Oracles SPARC T5-8 and IBM’s Power 780+, yet SPARC T5-8 delivers 3x more throughput, and certainly less HW costs.

      A: If you look at the tables comparing IBM and Oracle results, you can see that Power7+ delivers 80% more JOPS per core compared to SPART T5 – at almost half the $ per JOPS.

      Q >>> As some wise man said: “give me enough data and, statistically, I will prove anything”

      This is exactly why I published all of the cost and performance data for so many results over the past 2 years – I want readers of my blog to make their own conclusions, instead of filtering the data and selectively sharing only those bits and pieces that suit me and my employer – IBM.

      As Abraham Lincoln once said: “I am a firm believer in the people. If given the truth, they can be depended upon to meet any national crisis. The great point is to bring them the real facts.”

Trackbacks

  1. Oracle announces GlassFish roadmap and discontinues commercial support | Why WebSphere? Blog
  2. Developer point of view on the App Server debate | Why WebSphere? Blog
  3. WebSphere MQ 7.5 and Apache ActiveMQ 5.9 performance comparison (part 1) | Why WebSphere? Blog

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 685 other followers

%d bloggers like this: