For background about R, Ashlee Vance wrote:

“…R is a popular programming language used by a growing number of data analysts inside corporations and academia. It is becoming their lingua franca partly because data mining has entered a golden age, whether being used to set ad prices, find new drugs more quickly or fine-tune financial models. Companies as diverse as Google, Pfizer, Merck, Bank of America, the InterContinental Hotels Group and Shell use it.”

Zack previously suggested that R could become  an alternative to SAS or IBM/SPSS’s offerings in the business intelligence space. However, it seems that both SAS and SPSS have recognized the opportunity presented by R.

For instance, Jon Peck of SPSS wrote:

“Starting with Version 16, SPSS offers a free plug-in that lets users run R code within SPSS having full access to the active SPSS Statistics data, and writing its output to the SPSS Statistics Viewer. With Version 17, we began creating dialog box interfaces and SPSS-style syntax for R packages we thought would be interesting to SPSS users…We see the SPSS-R connection as a way for users to take advantage of the large number of R packages without the pain part of R.”

As Ashlee points out, R is being used by academics, university students and enterprises. If ignored, R could very well have become a threat to SAS and IBM/SPSS franchises.

IBM has a history of utilizing open source for competitive advantage. Instinctively, I thought SPSS decided to support R after being acquired by IBM. I’m encouraged to learn that SPSS made the decision to support R well before the IBM acquisition. It’s also great that SAS has followed suit. I suspect that SPSS and SAS made their individual decisions based on three factors. First, they likely both realized that based on the penetration of SAS and SPSS in the statistical community, neither were going away anytime soon. Second, adding R support enabled both vendors to take advantage of the community of users building extensions and new statistical methods for R. Finally, both vendors likely realized that customers have different skills and analysis needs, and as such, R would be used in conjunction with SAS and SPSS’s programming languages for statistical analysis. In short, both vendors had more to gain by adding R support than by attempting to fight an customer-driven trend.  It’s great to see vendors responding to the opportunities posed by open source projects instead of solely focusing on the risks.  As expected, commercial software vendors are quickly adopting their stance on open source as an enabler for growth.

Comments

Low startup costs

One of the first things we had to do after we defined our business architecture, was to create a demo that our CEO could take with him on the road. My engineer, who had never used Amazon AWS before, created an account using the company credit card, and quickly built a working model of how our system would work. He built both a web site and WAP interface that allowed a consumer to select online coupons and built a prototype of a store system that redeemed the coupon by calling our redemption engine in the cloud. The process took about 2 weeks and our IT costs for the month (excluding labor) was $86. Our CEO could now go on the road and show potential customers and investors just how our system worked (or would work in the future). The wow factor of walking into a client site, having them pull up their cell phone or their browser and perform an end-to-end shopping experience that utilized the cloud live was huge. And we did it for under $100!!! In fact, the only hardware we have purchased to date is our netbooks.

Optimal R&D environment

In my post called Building Better Software in the Cloud I discussed how we were able to fire up many different configurations to prove out our architecture. In my previous jobs, we would normally either be told what hardware our target application would run on or we would have to take a best guess very early on because of the long procurement cycles that often took several months. Then when the hardware came it we hoped like hell that our projections were accurate. In the world of cloud computing, we were able to experiment with many different CPU and memory configurations and simply only paid for the time we spent using those virtual machines. After performing numerous tests with numerous configurations, we knew exactly what “hardware” we needed to get the job done. There is tremendous value in proving out your design before you buy your target architecture. For one, if the project ever got canned, you are not stuck with hardware. Second, you no longer need to guess what you need and risk buying too little or too much computing resources. The “Try before you buy” model is a key differentiator in my book!

Ability to perform huge performance tests

One of the most impressive experiences I had in my first year in the cloud was the massive performance test we performed with our testing as a service partner, SOASTA. As documented in a SOASTA press release, we were able to prove that our architecture could easily process one million concurrent transactions in 1/10th of a second! To perform this test, SOASTA fired up over 700 virtual servers to run their software and we fired up over 50. After the test we both quickly tore these servers down to “shut the meter off”. A test like this is not even feasible in the on-premise world! This test was executed in one day. That means both companies fired up a combined 800+ servers, ran the test, and decommissioned the servers all in a day’s work. We only paid for the time we used the servers. You simply just cannot do that on premise. What this means is that in the cloud, you have the ability to stress test your system and discover and fix your bottlenecks when running worst case scenarios before you go into production.

Speed to market

This part is every system administrator and DBA’s nightmare. We built an entire platform without having a single DBA or system administrator on staff. We invested some time upfront to create standard AMIs (Amazon Machine Images) that contained all of the appropriate security controls and configurations for the OS (Linux), the database (MySQL) and the app server (Apache). Once the standard AMIs were built, deploying more was as simple as a few clicks of a mouse and a 2-3 minute wait. We no longer had to go through long, painful procurement cycles. We no longer had to argue with the system administers or DBAs about the priority of setting up our server or our database. We no longer had to bother anybody about taking snapshots of production data for use in our test environment. I could go on and on but I think you get the point. Things that used to take weeks or months in our previous jobs now take minutes. This means we spend more time actually building business functionality and less time managing the infrastructure. Which leads to the next point.

Focus on the business needs

The beauty of building our architecture on Amazon is that they make it real easy to deploy and manage compute resources. We are able to build in auto-scaling capabilities, we can easily leverage S3 to replicate our data across multiple virtual data-centers, and quickly deploy a simple backup strategy consisting of snapshots and data replication (no tape!). What that means is we can easily build a self monitoring, self-healing system that requires minimal human resources to deploy and monitor, so our people can focus on building features for our customers. Amazon has recently released RDS which abstracts a lot of the core database administration functions for MySQL. We have been testing it recently and found some huge time savings with it. We are also testing the queuing service (SQS) which can save us an incredible amount of development. With SQS, we can get guaranteed delivery of all of our messages/transactions without having to build out the complex logic required to perform that function. Once again, Amazon provides the plumbing required to run systems and we focus on our business functionality.

Build secure systems

I have said it before and I’ll say it again, I can build highly secure systems on Amazon’s cloud platform. Security is something you build not something you buy. Many people underestimate the security and regulatory controls that are in place on the Amazon platform. I described it in detail in this post. As a startup, there is no way I would be able to build a solution even remotely close to being as secure with an on-premise solution than what I can build in the cloud. The reason is simple. I would never have enough cash to hire enough security professionals, build enough data centers, and develop all the proper controls. Even if I went the hosted route, I could never afford the costs of deploying across multiple data centers because of the lack of the “pay-as-you-go” model with hosted solutions.

Build high performance systems

When we first started this journey, our biggest concern was performance. We knew we could build things cheaper and faster but we did not know if the cloud would provide the level of performance our systems required. As I mentioned previously, we are able to process over a million transactions concurrently with sub second response time. It took a ton of sophisticated engineering to get there, but we proved it could be done. What is even better is that if we need to handle more transactions, we can just fire up more AMIs. We can scale this almost infinitely.

Amazon owns this space

Last but not least, Amazon is by far the clear leader in this space. Not everybody who has built systems in the cloud have enjoyed as much success as we have. Some companies like Rackspace have had severe outages. Others like Coghead have gone out of business. Then there are others who have lost customer data. The list goes on. Amazon has not had any of those issues. In addition, they are the thought leaders in this space and are providing many highly valuable services such as S3, SQS, RDS, and others that further abstract the infrastructure layer so that their customers (like me) can focus on their business. I was also fortunate enough to attend Amazon’s AWS Partner training and got some insights into future features and functionality (under NDA). I can tell you that building apps and services on AWS is going to continue to get easier and more cost effective as time goes on. Amazon has a great strategy of releasing services quickly with just enough functionality. Then they follow it up with many subsequent releases that increase the robustness of those services.

Summary

I had studied cloud computing for over a year before I had the opportunity to work on it. I was fully aware of the pros and cons and went into this journey with high expectations. A year later I can tell you that what we are able to do in the cloud far exceeded my high expectations. We have built a high speed transaction processing engine that is highly secure, meets all of our required regulatory demands, absolutely screams from a performance perspective, and allows us to get stuff done in days what used to take months. I actually feel sorry for all of the colleagues I left behind in the corporate world, well, most of them anyways!

The most amazing thing to me is that the cloud is still in its infancy which means that 2-3 years from now the capabilities will far exceed what we have today. I can see a day where an entire web site architecture can be a service that is provided by Amazon. Today I have to plan out a web server layer, a db server layer, a caching layer, etc. Tomorrow I might just access an AMI that represents that entire architecture.

So these are my lessons learned after one year. I don’t really have any “what not to do lessons” because we did our homework, we started with architecture first, we built in the appropriate level of security, and we didn’t have unrealistic expectations coming in. Oh, I do have one. Don’t listen to people who say that the cloud is not ready for prime time!

Comments

All four of these perspectives are extremely important, but the missing perspective is the one from architects who are actually building enterprise solutions in the cloud.  Rewind  about four or five years and this is very familiar to the early SOA days.

To make matters worse, I am not convinced that many of the vendors are clear on what cloud computing is.  Every day I read blogs and tweets where people are arguing what the definitions mean, what the pros and cons are, and if the cloud is even secure enough to consider for anything mission critical.  There is a heck of a lot of talking going on that’s for sure.  But how many of those talking the talk are walking the walk?  Which brings me to the question, “If you haven’t got your hands dirty, why should I listen to you?”

I think the issue is that we are so early in the hype cycle that many enterprises are taking the wait and see approach.  Others are using the cloud for simple projects that don’t require architectural changes.  Also, many early adopters are startups and small businesses who don’t typically spend a great deal of time focusing on architecture anyways.

I am calling out to my fellow cloud practitioners to start sharing your lessons learned evaluating cloud computing or implementing systems or services in the cloud.  Right now there is too much vendor influence.  Most of the information the analysts are discussing is being sourced from vendor conferences, blogs, or PR calls.  Without hearing the voice of reality from practitioners, we could be setting ourselves for failure like we have seen with big SOA initiatives over the past few years.  I have been sharing as much information about our hybrid cloud solution as I can without giving away company secrets (it’s a fine line to walk).  But I am not finding a lot of other blogs sharing design concepts or lessons learned.  If you know of some please let me know.

So I am begging the vendors and analysts who are out on the speaking circuit promoting cloud computing, “Please give us some real life customer case studies and/or lessons learned”.  By now we should all know what the acronyms are, what the risks are, and what the benefits are.  Now we need to know how.  And it is never a good idea to learn the how from a vendor.  We need to learn the how from people in the trenches.  We need to discuss enterprise cloud initiatives, not server consolidation stories, adhoc offload processing stories, or user solution cloud stories.  We need examples of companies leveraging the cloud to do something mission critical, something that drives mass change within the organization, or something that delivers incredible business value.

So to the vendors, analysts and others…. Bring us some customer stories that practitioners can learn from.  Tell me something of value.  Let’s get away from theory and talk about reality.  Let’s talk about how we can address the concerns about security and compliance (yes, they are doable in the cloud).  Let’s not relive the cycle of hype, hope, and failure that we did with SOA and many other buzzwords from days gone past.  Let’s stop talking the talk and let’s start walking the walk!

Comments

• Many of the vendors do not have a clear vision and are rushing to the cloud
• Microsoft’s Azure is so not ready for prime time
• There are not enough practitioners out there to tell us the truth about the cloud
• There is an amazing amount of FUD going around
• People are making blanket statements that are just not true
• David Linthicum is one of the few people on the cloud computing speaking circuit who is making sense

Now allow me to explain…

Vendor Confusion & Azure

Amazon, Google, Salesforce, and a host of early adopter pure players get it.  But the mega vendors joining the party late (IBM, Microsoft, HP) do not.  I remember being at a handful of SOA conferences last year when I mentioned the cloud and the IBM folks in the meeting almost lost their drinks through their nose.  Now they are on board but they really don’t have a well thought out vision and are almost interchanging the terms data centers and clouds.  Their message is all over the place and they are not thought leaders in the space.  Microsoft says a lot of the right things but when you ask questions it is really vapor ware.  It is very proprietary and the only reason why it has so much traction is because of the brand name Microsoft.  Their presentation was laughable, and the Q&A that followed exposed many weaknesses that were hidden behind the pretty power point.  Until they show me something real, it is the “Blue Screen of Death” in the cloud for me.

Each vendor has its own definition of the cloud.  Many are rebranding what they already have as cloud enabled.  It is not all the vendors fault though.  This is typical in the early stages of maturity once a technology starts up the hype cycle.

Where are the practitioners?

There are way too many vendors talking and not enough practitioners.  That is not a good thing.  If all of our education is coming from vendors we will be in a bad place.  This is SOA all over again from 3-4 years ago.  When I do see a practitioner, most of them have not done anything significant yet.  If I have to listen to the NY Times success story one more time I will puke.  I keep hearing the same case studies over and over which tells me that we are talking about the cloud a heck of a lot more than we are doing the cloud.  Most of the good case studies are from startups which is an obvious sweet spot for the cloud.  I haven’t heard many case studies where an established enterprise built something significant in the cloud.

FUD, FUD, and more FUD and “Blanket Statements”

As expected with anything that requires change, there is a ton of fear, uncertainty, and doubt out there.  Here are a few quotes I heard from practitioners (of all people)

Don’t do anything mission critical in the cloud

Don’t put any sensitive data in the cloud

Those are blanket statements that are simply untrue.  There are numerous startups that are 100% cloud (including my company).  There are also numerous financial transactions occurring in the clouds today as well.  What we should be saying is mission critical services and sensitive data require a greater effort to architect the proper level of security, compliance, and reliability than if we put them behind the fire wall.  That is not a negative thing.  That just means we can’t implement crappy stuff and hide behind the firewall like we do today and hope our firewall hides our design flaws. Now we actually have to build services like we were supposed to in the first place.  I say this is a good thing (if you have the talent to do it properly).

David Linthicum

Dave’s advice is the best that I have come across out there.  It is all about architecture and the cloud is just another tool in the toolbox.  You can see Dave’s presentation here>

There are some smart folks out there who understand that doing SOA right makes doing the cloud easier. They also understand that it really all boils down to architecture, patterns, and aligning with business drivers. Doing the cloud for the sake of doing the cloud is poor judgement. For a startup like mine, the cloud is a no-brainer. I have no legacy systems, I have no data center, and I have no money ;-} Perfect use case for the cloud. For well established enterprises, there needs to be real business drivers. As Dave recommends, the cloud is typically not a good place for legacy apps. Do new apps in the cloud.

Summary

To sum it up, there is a lot of hope, a lot of hype, a lot of potential for success, and a lot of potential for failure.  The current state of Cloud Computing reminds me of the early hype cycle for SOA.  If we don’t learn from our previous mistakes, we might have a lot of garbage in the clouds.  The vendors are rushing to the cloud party.  Eventually they will have a consistent message but right now it is very “cloudy”, pun intended!

Comments

Gigaom this morning posts the idea that cloud computing is going to have a negative influence on the IT Job market, as much as the computer had on the typewriter market. The idea is sound; the question is how do people learn enough about cloud computing to remain competitive in the market place? There are few viable training programs out there, no degrees, and very few certificates. While Google is taking this a step further by offering education in the clouds, my own experience with Amazon Web Services has shown that even just getting started in running virtual servers in the cloud can be painful at best, or bring you to a dead stand still at worst.

There is no way to get around this, if you are in IT and you stop learning, then you are doomed to be replaced in the near future.

Today’s laid-off systems administrators, however, are not likely landing these newly formed IT 2.0 jobs. They have been too busy applying duct tape and Band-Aids to existing infrastructures to stay on top of the cutting edge. Nearly a year and a half ago, already, I heard a FedEx Corporate Services IT executive bemoan how ill-equipped his team was to deal with the division’s increasingly fabric-like infrastructure. He was neither the first nor the last to express that sentiment. Source: Gigaom

The next big thing to learn is cloud computing. Business and IT budgets are already being framed around the idea of outsourcing many applications and functions to cloud systems. The problem is going to be finding education, but that does not mean you cannot purchase time on Amazon’s Web Service (AWS) and go play around with what you want to work on, and what you think can and cannot be done. There is a large body of very good information on how to connect and how to use these services. Although at times the information might be confusing if not in conflict with each other, there are always opportunities for people to learn how cloud computing works and what you can do with it.

The good part is if you get started now, when the Business office starts picking up and wants to use these services, you already will have hands on practical experience in cloud computing based on your own learning, and as education/training catches up with where business is going, this might just end up being something you can do that could be quite lucrative for you.

Comments