Shaping Tomorrow's Data Platforms: Insights from Databricks CEO Ali Ghodsi

In the ever-evolving data management landscape, staying ahead of the curve is crucial for success. That's where we come in. Our newsletter Datamanagement.AI is your go-to source for the latest news, exclusive insights, and expert analysis on everything related to data management.

Whether you're interested in cutting-edge technologies, best practices, or the latest industry trends, Datamanagement.AI covers it all. Join us as we explore the dynamic world of data, from AI integration and data security to innovative storage solutions and beyond. Stay informed and empowered with the knowledge you need to navigate the complexities of today's data-driven world.

Shaping Tomorrow's Data Platforms: Insights from Databricks CEO Ali Ghodsi

Data management is experiencing significant changes as companies face cost pressures and a need for tool consolidation. The trend towards platformization is making data operations more efficient and cost-effective, driven by the adoption of open data formats, artificial intelligence (AI) integration, and improved governance solutions. This shift is geared towards more sustainable and flexible data strategies, promising future innovation and operational efficiency.

Ali Ghodsi, CEO and co-founder of Databricks, recently discussed these trends with theCUBE at the Supercloud 7: Get Ready for the Next Data Platform event. Ghodsi emphasized that all of Databricks' data is stored in open-source formats, with 92% in Delta Lake, predicting that this trend will continue to grow.

During the interview, Ghodsi explained that the use of open data formats like Delta Lake and Apache Iceberg allows companies to avoid being tied to proprietary systems, offering flexibility in choosing tools. He noted that CIOs and business leaders are under pressure to reduce costs, driving this shift towards open formats.

A key innovation from Databricks is the Unity Catalog, which provides a comprehensive governance solution for data, AI models, dashboards, and notebooks. This tool not only manages security and access but also helps with cost control, sharing, and collaboration. The integration of AI into data governance is a critical aspect, ensuring responsible AI model development and deployment.

Looking forward, Ghodsi is optimistic about the role of open source in the data platform industry. Databricks' acquisition of Tabular, a company focused on open-source data formats, highlights its commitment to open standards and interoperability. The upcoming Delta UniForm project aims to unify Delta Lake and Apache Iceberg, providing seamless compatibility, which has been well-received by customers.

Ghodsi also highlighted the strategic importance of AI and generative AI (gen AI) for Databricks. The company's focus on data intelligence helps organizations derive actionable insights from their data, positioning it as a leader in the AI space. Ghodsi emphasized the need for robust governance and security in AI and data management, which are critical concerns for their customers.

30% of Indian Firms Face Data Management Challenges, Lenovo Report Reveals

Lenovo has released the second edition of its ‘Smarter Data Management Playbook’ report, featuring insights from over 550 CIOs and IT decision-makers across the Asia Pacific region, including more than 100 from India. Commissioned by Lenovo, the IDC study examines crucial aspects of AI data management for 2024, such as data security, AI model integrity, data architecture and management, and data-driven enterprise IT strategies.

Sumir Bhatia, President of Lenovo ISG, AP, stated, "Data innovation is vital for today's CIOs to unlock new business opportunities and drive digital success. The playbook reveals that 29% of Asia Pacific enterprises identify data management complexity as their primary challenge. This indicates a significant shift towards hybrid cloud environments designed for AI. Lenovo's open, reliable, and secure infrastructure enables organizations to modernize their data strategies, ensuring agility and a competitive edge."

The IDC survey highlights that hybrid cloud is the preferred infrastructure for enterprise AI needs, shaped by each enterprise's unique data architecture and strategy. In India, 33% of organizations are repurposing existing hybrid architectures for AI, while 22% are utilizing cloud vendors to meet AI infrastructure demands.

Amit Luthra, MD of Lenovo ISG in India, commented, "As emphasized in the Data Management Playbook, 2024 will be a crucial year for organizations navigating the evolving data management landscape. We see a clear shift towards prioritizing data security and optimizing data for AI-driven insights and business outcomes. Lenovo ISG is committed to providing agile, hybrid cloud solutions that streamline data management by up to 85%, enhancing operational efficiency. This commitment underscores our dedication to delivering tangible business value in an increasingly competitive market."

Data Where You Want It, How You Want It

In the Asia Pacific region, 29% of enterprises report that data management complexity is a major challenge in their data initiatives, with India showing one of the highest concerns at 30%. This underscores the need for robust data management solutions to streamline processes and drive efficiency.

Consequently, the top three priorities for Indian CIOs are data backup, analytical data stores, and data security. For the Asia Pacific region, data security is the top priority, followed closely by data preparation for AI projects and data architecture.

David Mooney, VP of Storage Sales – WW at Lenovo ISG, added, “With 30% of Asia Pacific organizations prioritizing data security for 2024, Lenovo ISG reaffirms its commitment to providing fast, simple, and secure data management solutions. Our ThinkSystem DG and DM Storage solutions, with built-in ransomware protection, safeguard critical business data against evolving threats. As we aim to be the world's leading end-to-end infrastructure solutions provider, we deliver innovations that help customers manage and protect their data, enabling AI integration and business transformation."

Top Data Quality Concerns with AI/GenAI

In 2024, the top three data quality concerns for Indian CIOs using GenAI are data manipulation, lack of data quality assurances, and AI hallucination (misleading information generated by AI models). This contrasts with the broader Asia Pacific region, where the top concerns are AI training on false information and lack of data tags and assurances.

The key limiting factors for AI/GenAI adoption in Indian organizations are high infrastructure costs, lack of a clear business case, and concerns over data security and control. For the Asia Pacific region, the primary factor is data security and control, followed by exposure to brand or regulatory risks and concerns about data accuracy and its potential for unethical use.

Repurposing existing hybrid architectures for AI is the top approach for Indian companies to address AI infrastructure requirements, with 33% of CIOs favoring this method.

David Mooney emphasized, "In today's digital economy, data is the new currency, unlocking valuable insights that drive business growth and competitive edge. IDC's research highlights that the top data quality challenge in GenAI is training AI on false or undetected incorrect information. To address this, we provide Smarter Data Management and Storage solutions for diverse workloads, from AI and analytics to data-intensive enterprise tasks, ensuring our customers have a robust and reliable data foundation."

SAP and Hilti Partner to Innovate Cloud-Based Product Lifecycle Management

SAP) has introduced new cloud capabilities for its SAP Product Lifecycle Management (SAP PLM) solutions. This development, in collaboration with Hilti AG—a global leader in construction products and services—aims to enhance efficiency and cost savings by integrating business data with engineering and research and development functions.

The latest SAP PLM features offer seamless connectivity with SAP's cloud ERP and supply chain management solutions, enabling businesses to ensure timely and cost-effective product delivery by providing comprehensive visibility and control over the entire value chain. By effectively linking product data with business processes, companies can better manage product information, streamline workflows, and enhance cross-functional collaboration, thereby improving their ability to meet market demands. Additionally, partnerships with companies like DSC Software AG and Cideon enable integration with third-party product data management and CAD tools, reducing implementation times.

SAP PLM's centralized data management repository offers a single source of truth, simplifying data organization, reducing errors, and speeding up time to market by promoting effective collaboration among stakeholders.

"Integration is crucial for companies modernizing their manufacturing strategies and operational processes," stated Dominik Metzger, head of engineering for SAP Digital Supply Chain. "Our collaboration with Hilti showcases their confidence in our PLM solutions and allows us to innovate the future of cloud-native SAP PLM offerings."

Johann Stadler, head of large IT projects at Hilti, emphasized the benefits of adopting cloud-native SAP PLM solutions, noting that it not only enhances product development processes for customer-specific products but also improves construction efficiency. Hilti, a company operating in over 120 countries, is advancing its digital transformation by implementing SAP PLM to achieve new levels of efficiency and transparency in its product development processes, offering specifically engineered and on-demand prefabricated products to its customers.

As we wrap up this edition, we hope you’ve gained valuable insights and are inspired to continue exploring the world of data management. Whether you're optimizing your current systems, planning for future innovations, or simply staying informed, our goal is to support you every step of the way. Keep an eye out for our next issue, where we’ll bring you more exclusive news, expert opinions, and actionable strategies. Until then, stay curious, stay connected, and keep pushing the boundaries of what's possible with data. Happy data managing!

Thank you for being a part of the Datamanagement.AI community