IT Briefcase Exclusive Interview: The Data Transformation with Martin Fowler and Pramod J. Sadalage, ThoughtWorksOctober 19, 2012 No Comments
As Big Data, Cloud Computing, and Data Analytics have played an ever increasing role in modern IT, a new era of databases has emerged.
In the below interview, Martin Fowler and Pramod Sadalage outline new trends in database management and emphasize the value of NoSQL databases within today’s organizations.
- Q. There is no question that the way in which people view data today is changing. In your opinion, what roles do Big Data, Cloud Computing, and Data Analytics play in this?
Pramod: People are seeing companies make use of large amounts of data such as logs, user activity on websites, event streams, etc. Traditionally, this type of data has been thought of as not valuable or too large to put to use. However, the rise of cloud providers who can help companies deploy large number of machines on demand and eliminate the need to keep hardware forever are helping people to deal with considerable data volumes. NoSQL databases are helping people to store this bulky and messy data and process it using clusters of machines on the cloud. The ability to process such large amounts of data drives the capability to produce analytics at a much cheaper price point and in a much more compressed timeframe. We think NoSQL and cloud are driving a desire to tackle the big data problems and generate analytics never seen before.
- Q. In what way do you see the rise of NoSQL databases as effecting relational database dominance?
Martin: Relational databases have been dominant for so long that you need gray hair to remember a time when they weren’t the automatic choice for enterprise data needs. NoSQL databases are shaking this dominance in two ways. The initial drive came from the fact that many NoSQL databases (namely the aggregate-oriented ones) work well on large clusters. But more people we talk to are working with NoSQL because they provide a simpler way to interact with data in many situations.
Pramod: We see NoSQL databases giving a choice to the IT industry – this choice is really helpful in designing systems that meet the specific needs of the application. We think this choice – which is called Polyglot Persistence – is really a blessing and the IT industry should embrace these technologies and understand how to use them. The traditional database professional should be open to learning some of these technologies and be able to choose the right database technology for the application or enterprise requirements.
- Q. ThoughtWorks is known as a pioneer in the realm of agile software development. What do you feel is the key to doing agile well and creating a positive user experience?
Pramod: Agile is a mindset change and needs to be thought about not only in terms of management but also in terms of the engineering practices.
Martin: There is an important overlap here with Big Data. Often analytics efforts get into too much trouble because they try to address too much at once. Our lessons from agile teach us to push for a more focused approach, concentrating on identifying important questions to answer and then figuring out how to get at the data to answer these questions with visualizations that help convey understanding of what the data means. An iterative approach is vital here to find an effective route to take.
- Q. How does your recent book NoSQL Distilled work to provide people with a better understanding of what is happening with data today?
Martin: A year ago we found that there was no easy way for a software professional to get acquainted with the way of thinking you need to evaluate the usefulness of NoSQL databases. There were lots of individual tools out there, but no coherent story.
Pramod: The “NoSQL Distilled” book is our attempt to make people aware of the monumental shifts in data technologies that are happening today. These shifts are going to make people think about how to implement the requirements of applications to deal with lots of data. This includes understanding how to deal with aggregates, what level of aggregates to use and whether to tune the system towards more availability or greater consistency. Finally understanding how to use multiple database technologies in the same application or enterprise to meet disparate needs and move towards Polyglot Persistence.
Pramod J. Sadalage, Principal Consultant at ThoughtWorks, enjoys the rare role of bridging the divide between database professionals and application developers. He regularly consults with clients who have particularly challenging data needs requiring new technologies and techniques. He developed pioneering techniques that allowed relational databases to be designed in an evolutionary manner based on version-controlled schema migrations. With Scott Ambler, he coauthored Refactoring Databases (Addison-Wesley, 2006).
Martin Fowler, Chief Scientist at ThoughtWorks, focuses on better ways to design software systems and improve developer productivity. His books include Patterns of Enterprise Application Architecture; UML Distilled, Third Edition; Domain-Specific Languages (with Rebecca Parsons); and Refactoring: Improving the Design of Existing Code (with Kent Beck, John Brant, and William Opdyke). All are published by Addison-Wesley.DATA and ANALYTICS , Fresh Ink