Re: No to SQL? Anti-database movement gains steam

Matthew Zito wrote,on my timestamp of 4/07/2009 3:15 AM:
Sorry for the long winded reply but I think you make very good points and I want to address them all.
I'd be very careful making these kinds of statements. In my experience, the folks working at companies like Google, Facebook, MySpace, Ning, LiveJournal, etc. are easily as bright and experienced as the folks who work in tech at banks, pharmaceuticals, etc.
"very careful" doesn't even make it into scope. And before any veiled mentions of my "career" are brought up by anyone: I have had a career, don't need another one. That's why I can speak with true independence, instead of "toeing lines". People seem to like that I do so, given they are willing to pay for it.
Now: the simple fact here is that folks from Google, Facebook, Myspace, Ning etcetc, and what they do as far as IT goes, are absolutely and totally irrelevant to the VAST majority of enterprise business.
For starters most of them don't have prior baggage: they can afford to do something totally new with no concerns whatsoever about existing data/code. 
Very few enterprise businesses are in that class, if any.
Then, there is a fundamental difference between the business model of those web-only companies and finance, retail, investment, banking, services, etcetc. Ie: the vast majority of enterprise IT users in this world.
Google et all are a drop in the ocean in the IT market and what they do does NOT define the general market, not by a long shot. 

Which is exactly the point Sunil made and I agreed to in my reply.
They've simply made a different determination - that the cost of using a relational database in a scale-up or scale-out configuration is greater than the cost of using one of these non-traditional data stores.
Nevertheless, I'd like to see factual proof that non-traditional data stores can indeed provide that scalability whereas traditional ones can't. 

That proof better be a litle more than just "because it works at such and such".
Such is no proof whatsoever that:

1- it indeed is the *only* solution for that such and such.
2- it does apply to ALL others.

Which is what the demented fringe of Web2.0 is trying to convince the world of.
In good old web 1.0 lunatic fring style: history after all is cyclic.
Many of these companies have data needs that scale exponentially with increases in revenue. Consequently, scale is of extreme importance, as is performance. 


Where is it not so?  Everyone has to scale and with good performance.
Some more, others less. But it's been a constant of IT for the last four decades. Not a new thing. 


> I know an online advertising startup that has been up
and running for less than a year and already has close to 50TB of data it's analyzing, and they expect that to grow around 1TB/day through 2009, and as high as 10TB/day through 2010. They do indeed collapse data periodically, etc., but still - if they were going to buy that kind of horsepower with Oracle, how much would they spend?
I don't know. But we did not spend anywhere near as much as many others, we churn through 0.5TB per day, and it has trebled in one year.
Our business is good old commercial property management. Something that is traditionally "low volume" 

Yet our Oracle DW db seems to manage quite well with the above, thank you.
Of course:  we collapse data periodically as well. And aggressively so.
That is IMHO part and parcel of ANY seriously and correctly designed storage facility.Has nothing to do with the storage software being used and all to do with good analysis and design.
Something which I think is missing from a lot of these "let's make an agile web site" brigade. Most of them don't even know how to write SQL and that is one of the simplest languages anyone could hope for.
Of course: what you don't know, is expensive to use. Always was, always will be. Doesn't mean it can't be made cheap: just hire the right people, that's all. And no: the right people are NOT expensive, not anymore than the right people for web 2.0 technology.
Instead, they use CouchDB and another non-SQL-esque database whose name I can't remember. Since they wrote their apps from scratch, there was equal cost to develop against those as there would be against Oracle/MySQL/SQL Server.
True. But how many sites are there in general IT that can afford the cost of developing and maintaining their own apps from scratch as well as implementing an entirely new data store technology, incompatible with their existing one?
I lost track of how many years ago I saw the last one, outside of the lunatic web fringe.
The vast majority nowadays is running some form of third party app or code that does most of what they need and refuse point blank to spend one cent with inhouse development of replacements.
It might surprise a lot of the web 2.0 folks, but the biggest cost in IT nowadays is inhouse development. Much more so than anything Oracle might charge. 

Let me cite one small example of how costs can blow out with the web 2.0 stuff.
We have had a Lotus Notes CRM system for many years. Developed inhouse, way back when that was affordable. That was running on an old Lotus version, very limited, buggy and unsupported.
The company underwent an exercise on selection of a "modern, scalable, cost-efective, yaddayadda" CRM system.
Two years later and after a considerable amount of money was spent picking the "right package", the whole thing was cancelled: too expensive, completely incompatible with existing framework, requiring tremendously expensive data and process re-mapping.
So we got back to IBM, upgraded Lotus to a version from this millenium, moved the whole kit and kaboodle into it. 

Timeframe? 6 months.  Not bad, compared to 2 years of "evaluation".
Fast and stable, now. It interfaces to existing Oracle, SQL Server and JDE data stores, the three main technologies we use for enterprise data processing, with ZERO additional effort/cost.
Total cost of the upgrade? 1/3 LESS than it cost us to EVALUATE a web 2.0-based solution! 

Cost of re-training staff and users?  Nill!
Performance and scalability? It now copes faster with 20 times more data than the original version did, 10 years ago. 

Cost of integration into existing infra-structure?  Nill!
Try something like this with the new fangled non-traditional data stores and their necessarily custom apps and check how much it'll cost. So much for the "cost-effective web 2.0 cloud" nonsense.
Bottom line for sensible enterprise infra-structure: before going off on unproven flights of fancy, make darn sure existing technology has reached the end of its cost effective life cycle.
We all would like to be at the "bleeding edge" of technology. Fact is: it's bleeding, because it costs a LOT. And enterprise IT doesn't like to spend moolah on unsafe practices.
Of course, the article is overblown and hyperbolic, because that makes for a much better story.
Exactly. That seems to be a constant with the web 2.0 brigade. It doesn't help their cause one single bit: everyone still remembers the web 1.0 tech wreck, where the same was rampant. 


> But the reality is that while SQL-based,
ACID-compliant databases are not going anywhere, there are other data storage models out there that may fit better, depending on your application.
Fact is: "not going anywhere" is tremendously cost-effective and efficient if perfectly capable of coping with general purpose requirements.
Storage models that purport to be "better" need to first define exactly how general purpose they can be.
Any fool can create a custom designed system, with custom designed code, and end up with a fast result. Heck: I know quite a few folks who could write a lot of apps in Assembler and make them lighting fast. Still true today. 
Would anyone in the enterprise universe pay them to do so?  No way!

Are web 2.0 and these non-traditional data stores easily maintainable?
No: it is custom code, any changes will involve costly recoding. Calling it "refactoring" instead of "recoding" doesn't make it any less costly. Change in requirements is a constant in modern IT. Ergo: these technologies are inappropriate and costly.
True IT professionalism and responsibility picks a general purpose data store and app technology and makes it perform within the requirements, for a much reduced overall cost and with easy and cheap maintenance.
That is what the IT enterprise market is all about. Ferraris are great for show, but what is really cost effective for day to day use is a station wagon. The rest is hype. 



So why can't we have both?
Of course we can have - and need! - both. Ferraris do exist and serve a purpose. What we can hardly afford is yet another round of demented "new black" where the whole of IT is told to ditch tried and proven cost effective technology for something that can only fit, at the very and costly best, a niche.
Which is what those articles are clearly promoting and why they need to be exposed for the fraud they are. 

--
Cheers
Nuno Souto
in sunny Sydney, Australia
dbvision@xxxxxxxxxxxx
--
http://www.freelists.org/webpage/oracle-l

Other related posts: