Accelerating Analytics With Intel Optane Persistent Memory

by Przemek Tomczak

​The onslaught of data continues. Predictions vary but some estimate the world will be generating 175 zettabytes annually by 2025, of which 90% would be from IoT alone. Drawing context and insights quickly is ever more important, and ever more challenging.

​What is driving this demand is that organizations are deploying high-resolution sensors to improve the efficiency and effectiveness of their operations and products through analytics and automation.

​Organizations are challenged with supporting analytics on higher data volumes cost-effectively. To address these challenges organizations had two options:

• ​Use more and larger infrastructure – more servers with a lot more DRAM to cache more data
• Use expensive high-performance storage and network infrastructure

​Now we have a new option for consideration by performance-demanding customers – Intel® Optane™ persistent memory (PMem). We partnered with Intel to test their persistent memory with our Streaming Analytics platform running mission-critical industrial workloads.

​Key takeaways

Use of Intel Optane persistent memory as a block-storage device with KX Streaming Analytics delivers 4× to 12× improved analytics performance over high performance NVMe storage – similar performance to DRAM for query workloads. For key data-processing workloads, we found DRAM requirements were significantly reduced. ​This lets organizations support more demanding analytic workloads on more data with less infrastructure.

​Summary results

Analytics

• ​Performed within 10% of DRAM for queries involving table joins
• Performed 4× to 12× faster than 24 NVMe storage devices in RAID configuration
• DRAM performed 3× to 10× faster when performing single- threaded calculations and aggregations on data

​Data processing and I/O operations

​Processed 1.6× more data per second than NVMe-only storage where data was read from PMem and written to NVMe storage

• 2× to 10× faster reading data from files in parallel
• Seed of reading data similar to page cache (DRAM)
• Single-threaded file-write performance within 10% in both configurations
• Multithreaded file-write performance 42% slower

Infrastructure resources

• Required 37% less RAM to complete key I/O-intensive data processing
• Required no page cache for querying or retrieving data stored in PMem

Business benefits

• Collect and process more data with higher velocity sensors and assets
• Accelerate analytics and queries on recent data by 4× to 12×
• Reduce cost of infrastructure running with less servers and DRAM to support data processing and analytic workloads
• Align infrastructure more closely to the value of data by establishing a storage tier between DRAM and NVMe or SSD backed performance block storage

​Organizations should consider Intel Optane persistent memory where there is a need to accelerate analytic performance beyond what is available with NVMe or SSD storage.

​Please click on this link to code.kx.com for more details 

​

About Streaming Analytics

KX Streaming Analytics has been at the forefront of helping organizations ingest, process, analyze, and deliver actionable insights on massive amounts of real-time and historical data. KX Streaming Analytics is an integrated platform that runs in the cloud and/or on-premises to enable organizations to reduce costs and improve productivity through anomaly detection, analytics, predictions, corrective action and automation based on both real-time and historical data. It provides full life-cycle data ingestion, processing, analytics, and data management enabling mission-critical operations, 24×7, with no downtime and no data loss – all of your data, in any environment, instantly.

​

About Intel Optane persistent memory

Intel Optane persistent memory (PMem) redefines traditional architectures, offering a large and persistent memory tier at an affordable cost. With breakthrough performance levels in memory-intensive workloads, virtual machine density, and fast storage capacity, Intel Optane persistent memory accelerates IT transformation to support the demands of the data era.

​PMem is an affordable alternative to expensive DRAM. It can provide huge capacity and accommodate demanding workloads and emerging tools such as in-memory databases.

​PMem is propelling major infrastructure consolidation. The increase in memory size from Intel® Optane™ media is an opportunity to consolidate workloads. Concentrating them on fewer nodes reduces deployments and improves use of processors previously underutilized because of memory constraints.

​PMem is the world’s smallest and fastest storage device sitting on a memory bus. Architects and developers are using it to innovate with restart and replication, and to break system bottlenecks.