As they travelled to their first investigation, Dr Watson remarked to Sherlock Holmes that he didn’t seem to be giving the case his immediate attention:

“No data yet,” Holmes replied. “It is a capital mistake to theorize before you have all the evidence. It biases the judgment.”

That was in 1887. Imagine what Holmes could achieve today with data available from sources like criminal databases, social media and CCTV, advanced analytics, and visualization tools for deriving actionable insights and making informed decisions. Consider, for example, the benefits for predictive policing in identifying potential crime hotspot, high-risk periods, and allowing for pre-emptive measures. And it’s not just crime prediction and prevention where data can help. Among others are:

• Enhanced Investigative Capabilities: The integration and analysis of various data sources provides comprehensive insights, aiding investigations. Detectives can uncover hidden connections, track Subjects of Interest, and solve cases more efficiently.
• Real-Time Monitoring and Response: This enables operational units to benefit from real-time data analysis, enhancing response times and situational awareness.
• Resource Allocation and Planning: Applying predictive analytics to assist in strategic planning and resource allocation.
• Community Policing and Engagement: Using social media and public data analysis aids in community policing efforts.
• Legal Compliance and Ethical Standards: Ensuring compliance with legal and ethical standards, maintaining public trust, and safeguarding privacy.

All are premised on efficient data management and analytics across the vast and varied data types in policing operations. But those requirements bring equally big and varying challenges. Some of the difficulties include:

• Data Volumes: Handling and analysing massive volumes of data, both historical and real-time.
• Integration and Connectivity: Supporting multiple data formats, communication protocols, and interfaces.
• Data Quality and Accuracy: Ensuring that data is accurate, up-to-date, and reliable.
• Data Security and Privacy: Enforcing appropriate levels of access and protection from outside threats.
• Ease of use: For widespread and appropriate adoption.
• Flexibility: To expand functionality and adapt to evolving operational and technology environments.

KX has the proven technology to meet these needs. It provides data management and analytics solutions in industries ranging from finance and manufacturing to telecommunications, automotive, and pharmaceutical to empower their digital transformation with faster, more informed, data-driven insights. Moreover, businesses have realized it using standard hardware with outcomes, including achieving 100x performance at a tenth of the cost. Sample applications include:

• Anomaly detection for fraud detection, cybersecurity, and trading.
• Pre- and post-trade analytics in Capital Markets.
• Model training, back-testing and calibration.
• Predictive maintenance and operational equipment efficiency in Manufacturing, Telecommunications, and IoT workflows.
• Predictive healthcare for medical and healthcare practitioners.

Imagine what Holmes could achieve today with data available from sources like criminal databases, social media,  CCTV, advanced analytics, and visualization tools.

At a non-functional level, the technology offers the high-performance, high availability, resilience, and security needs demanded of mission-critical applications in highly regulated environments. Those combined capabilities align closely with the needs of law enforcement agencies across a range of areas.

Across communication channels KX enables network analysis to map out social networks and relationships, geospatial analysis for tracking movements, sentiment analysis using NLP, real-time monitoring and alerts, machine learning for predictive analysis, integration with other data sources, encrypted and anonymized data analysis, legal and ethical compliance, and customizable dashboards for law enforcement.

For intelligence data, the capabilities are similarly impactful. It becomes possible to integrate and analyze data from multiple intelligence sources, providing real-time intelligence analysis, pattern recognition and anomaly detection, network analysis, predictive policing through machine learning, sentiment analysis, geospatial analysis, custom alerts and notifications, compliance with legal standards, and user-friendly dashboards and reporting tools.

Functionality that enables these insights include:

• Real-Time Analysis: Monitor processes and analyze data from multiple sources in real time for timely threat identification and response.
• Pattern Recognition and Anomaly Detection: Identify patterns, trends, and anomalies in behaviors and communications for predicting potential criminal activities and hotspots.
• Machine Learning for Predictive Analysis: Predict future activities or communication patterns based on historical data to aid in preventative strategies.
• Custom Alerts: Notify Law enforcement officers based on specific criteria, such as unusual activities, potential threats, or important events detected across data sources.
• Network and Geospatial Analysis: Analyze social networks and track the flow of information for relationships between individuals of interest.
• Resource Allocation and Planning: Use predictive analytics to help strategic planning and resource allocation, ensuring that police resources are used efficiently and effectively.
• Integration with Third-Party Applications: Enable best-of-breed integration across applications. For example, enhance biometric matching accuracy by processing data from facial recognition technology and cross-referencing it with existing databases.
• User-Friendly Dashboards and Reporting Tools: Use intuitive interfaces and dashboards, enable law enforcement personnel to easily access, visualize, and interpret intelligence data, even without advanced technical skills.

Collectively, these tools represent a transformative advancement in law enforcement technology, offering unparalleled capabilities in data analytics and predictive policing, contributing to more effective law enforcement, better resource allocation, and, ultimately, improved crime prevention and public safety measures.

If only Sherlock had those tools, he might have been more productive in that handsome ride with Watson!

Please contact us or demo kdb to learn more about how KX technology enables law enforcement with data.

by Steve Wilcockson

When the artist most synonymous with Ragtime, Scott Joplin, learnt the piano, serious musical circles felt Ragtime was too “Tin Pan Alley.” Joplin would blend styles and formalize compositions that would be re-played with the same fidelity as when he wrote them, for example, The Entertainer.

So, what do Scott Joplin and Ragtime have to do with Generative AI tech and KDB.AI Server Edition?

First, Generative AI must be carefully teased, like the notes of Scott Joplin’s The Entertainer. “Playing the right notes in the wrong order,” is akin to a GenAI-inspired hallucination that could lead to an adverse outcome: a flash crash, or a mistimed medical or political intervention.

Second, the “Ragtime” genre of yesteryear plays nicely with the RAG Time advantage of KDB.AI Server. RAG represents the Retrieval Augmented Generation orchestrated Generative AI workflow, while KDB.AI provides a natural time-based advantage.

A Vector Database Built for RAG and Time

RAG is tech’s favored approach for incorporating enterprise data into Generative AI workflows. Enterprise data stores of vector embeddings, brought to life through fast semantic search in KDB.AI, accompany Foundation and Large Language Models orchestrated through development tools like LangChain.

Organizations that implement RAG can deploy data-sets and provide guardrails around prompts and responses mitigating against LLM “hallucinations.” Like how Scott Joplin intended his compositions to be played – not over-improvised – enterprises require their AI infrastructures, particularly LLM prompt responses, to provide similar fidelity.

Below, “Figure 1” shows KDB.AI in a simple Naïve RAG scenario. RAG has two primary steps:

• “Retrieval” where, based on the query, additional related information is extracted from internal sources
• “Generation” where that additional data is passed and augmented along with the original question to the LLM for generating a response

Fig 1. Simple naïve RAG architecture

With RAG, the vector embeddings saved in the KDB.AI vector database, digital encodings of text, videos, pdfs, or other data, provide information your LLM didn’t have access to when it was trained initially. For example:

• Private, internal information such as contacts, internal policies or personnel records
• Corrections or updates to previously published information or official records
• Price changes, news updates or corporate action events that have occurred in the interim

With this vector knowledge store, you can explore semantic search in conjunction with your LLM environment, orchestrating with LangChain or your own data pipeline tool.

What about time? KDB.AI is built for time, incorporating temporal information into prompt responses through filtering, time windowing, and search over time.

Pre-trained models like GPT, which stands for Generative Pre-trained Transformer, are static models tied to the time at which they were trained. Handling and responding to data and analytics over time, including real-time, underpins the core technology on which KDB.AI is based. This makes the KDB.AI vector database, working alongside the pre-trained static foundation models, well suited to temporal use cases at all levels of granularity.

Why KDB.AI Server?

KX recently released KDB.AI Server, a highly-performant, scalable, vector database for real-time contextual AI that runs on-premises, hybrid, or in the cloud with a single container deployment. In short, KDB.AI Server takes the freely available developer capabilities from KDB.AI Cloud Edition, and helps CDAOs, CTOs, and their business-facing teams build fast and run their data anywhere, in secure, enterprise, or customer-managed deployments. Vector database functionality is ported to the enterprise architecture of choice. You can:

• Create an index of vectors (Flat, IVF, IVFPQ, or HNSW)
• Append vectors to an Index
• Perform fast vector similarity search with optional metadata filtering. Available distance metrics include:
• dotProduct (similarity or alignment between two vectors)
• L2/Euclidian (measures straight-line distance between two points in Euclidean space)
• Cosine (assesses similarity in terms of the direction of the vectors)
• Persist an index to disk
• Load an index from disk
• Integrate with foundation and large language models through developer tooling such as LangChain and LlamaIndex, and with the KDB.AI ChatGPT Retrieval Plug-in

Note the deployment benefits of KDB.AI Server. KDB.AI’s efficiency in filtering and searching vector datasets, deploying unlimited numbers of time windows, and no performance hits, means less need for GPUs and other expensive hardware. You can deploy to your existing hardware, on-premises and/or on cloud, in conjunction with your enterprise data resources and production environments.

RAG Time Examples

Let’s explore some time-related prompts to understand the use cases KDB.AI can be applied to. Fig 2 shows a prompt response asking, “What products did Apple release in 2020?”

Fig 2. What products did Apple release in 2020?

The answer is returned in the bottom left: “iphone SE and an updated 13 inch Macbook Pro14.” If we’re focussed on market activities, we can now ask “What was the response of the share price during the period?”

Fig 3. What was the impact to Apple’s share price?

Our queries can extend further. KDB.AI can recognize patterns and provide charts, highlight key source release information from original documents, quantify impacts, and compare with similar trends from prior years or different releases.

Were we a retailer with similar interests in Apple launch metrics, holding our own and industry-aggregated retail data-sets, instead of assessing the impact on share price, we could probe the impact on sales. We may, for example, compare our own sales figures, including the most recent ones, alongside the aggregate market knowledge “known” by the LLM. We might then benchmark our company’s sales to others, identify and compare to similar launches back in time, or to launches from Apple’s rivals.

Let’s now consider a more elaborate financial workflow example. Consider an investment strategy that trades BitCoin based on streaming sentiment, for example around regulation. A vector database with an LLM in a RAG architecture provides an excellent alternative to off-the-shelf black-box sentiment feed providers, of whom there are many. We can join social media feeds with price data and assess how regulatory announcements impact price, e.g. BTC.

Step 1:

• Identify news about a specific corporate entity and a government regulator from the news feed
• Use the LLM with an embeddings strategy to create vector embeddings stored in KDB.AI Server Edition
• Find time windows where a specific corporate entity (BTC, Bitcoin, bitcoin.org) is mentioned in the context of a regulator (SEC, FINRA, Fed, etc.)

Step 2:

• Determine if sentiment is positive or negative using natural language syntax
• Identify price movements in KDB.AI tick history during the time windows from step 1
• Use the selected LLM, e.g., Anthropic Claude or GPT and a RAG architecture to identify sentiment using vector embeddings with KDB.AI

A sample RAG architecture is shown in Figure 4, below.

Figure 4: A Market-Driven Investment Model Process: Regulation Sentiment & BTC

With KDB.AI and a RAG architecture, we can quickly build and deploy a sentiment and price infrastructure, leverage information known by the LLM, guardrail, and make valuable with our own price and trade data to inform the investment strategy.

Banishing the RAG Time Blues

With your chosen LLM and RAG architecture, KDB.AI can leverage your knowledge base and provide insightful responses with speed, efficiency, and security, leveraging time to facilitate decision-making. And like the Ragtime of old, guardrail and provide structure to ensure your AI works as intended, like how Scott Joplin wanted pianists to reproduce his pieces with fidelity.

When running time-based queries, KDB.AI Server helps deploy AI routines with enterprise data-sets within your own secure firewalls – like our exemplified proprietary trading or retail data-sets alongside stores of unstructured data, e.g. analyst reports.

In our two earlier examples, we highlighted windowing and temporal analysis. Whatever your industry, be assured there is a time-based use case for you, for example:

• Telecommunications: Optimize capacity and coverage, deliver control at the edge, and maintain critical services with self-organizing networks
• Manufacturing:  Quickly spot real-time predictive maintenance anomalies for smarter and more efficient plants
• Energy and Utilities: Analyze billions of data points instantly for grid and resource optimization and improved customer experience
• Aerospace and Defense: Analysis of operational data for correlation of intelligence, improving command decision making
• Healthcare: Model disease and interventions in virtual clinical trials to predict patient responses and required treatment dosages

Get started with KDB.AI

Alternatively, please visit our dedicated Learning Hub to learn more about vector databases.

RELATED RESOURCES

Implementing RAG with KDB.AI and LangChain

3 Steps To Make Your Generative AI Transformative Use Cases Sticky

KDB.AI – The Smarter Database for AI

By Steve Wilcockson

Our whitepaper, Cultivate Sustainable Computing and Environmentally Responsible Analytics in an AI Age, had me thinking, how many of us strive for sustainability, only to fall short? I recycle, shop locally, live in a small house, and advocate for environmental causes. Yet my family drives two petrol-guzzling cars, we regularly fly short-haul, and our drafty one-time Tudor merchant house is anything but energy-efficient.

My passive environmentalism helped draw me to KX. Yes, I love the analytics, Python, and AI, but it was my previous experience in enterprise middleware, which helped me realize computing should be much greener.

Everywhere I looked in middleware, I saw computational waste. I don’t mean technical debt. I mean inefficient compute, inefficient memory utilization, and over-complicated stacks, probably held together by spiderwebs of APIs and Kubernetes processes. This comes from the programming languages used in the core tooling, and cultures that reward performance over efficiency. Let’s explore.

The Language Bloat Problem

When in middleware, I worked in the Java ecosystem. It could have been Python, C++, JavaScript. But I worked with Java, and it so happens that many big data tools are built in the so-called Java Virtual Machine, such as Kafka, Cassandra, Spark, Solr, ElasticSearch, Solr, Hadoop, ZooKeeper, and Flink. Python and JavaScript, also relevant to data analytics, use similar virtual machines or runtime environments.

Runtimes are great for development. They encapsulate general-purpose functionality and facilitate “write once, run anywhere” development. Trouble is, runtimes tend to “price” for all use cases and the hardest workloads – memory utilization, compute. It’s a bit like driving ten cars rather than one to the grocery store, with all ten prepped to drive thousands of miles.

In Java’s case, glitches and inconsistent performance mean users must assign memory and compute accordingly. This affects its performance and efficiency. Python, with its own runtimes, has struggled with the so-called Global Interpreter Lock (GIL for short), which has impacted its ability to optimally thread processes and thus run efficient compute.

Such factors have been compared in the often-cited Energy Efficiency Across Programming Languages Survey where Java seemingly performs better than Python. The survey is somewhat academic and should not be read too literally, but it highlights the green credentials of programming languages that underpin enterprise and data stacks.

The Massive Data Stack Problem

Who hasn’t seen a highly elaborate architecture presented at a conference or in an engineering blog? Has the presenter or author sought to “wow” you with their stupendous ability to navigate multiple tools? Sometimes, those “complexity wows” get institutionalized, like with the legendary Hadoop stack with its plethora of confusingly named components, like Hive, Yarn, Pig, HBASE, etc.  When a complex stack meets key SLA targets, normally latency, throughput, and carrying capacity, the green efficiency of the bloated stack hasn’t always been scrutinized. “Job done” matters more than “how well it was done.”

In AI, Matt Turck’s oft-cited AI landscape illustrates something similar and is exacerbated further by compute-intensive LLMs and GenAI workflow injections. So troublesome is the GenAI stack compute load on the environment that the topic is discussed frequently in the Telegraph, New York Times, and Economist. For example, ‘did you know a 50-question interaction with an AI chatbot can consume around 500mls of water?’

In sum, the modern stack has been complacent about languages and the proliferating nature of applications, which GenAI compute exacerbates. Unchecked by proper measurement, data centers – which power the everyday needs of you and me – can consume the electricity consumption and carbon footprints of large nations!

Squish the Stack

KX is built differently to service the heaviest analytics workloads, from the biggest data sets and the fastest streams to the most intensive AI application. Not a single byte of code or memory is wasted. With KX you can:

• Run vector processing in the simplest, tersest way possible. No bloat.
• Combine data management and analytics.
• Perform single vector operations to reduce compute; no multiple loops or batched relational queries.
• Deploy the smallest footprint – a few hundred bytes.

For Environmental Social Governance (ESG) analysis, KX ingests, aggregates, and analyzes disparate and inconsistent data. For industrial applications, it ingests data from sensors, databases, and shopfloor metrics, providing observability, which, in turn, provides insights to reduce waste.

Lower Environmental Overheads? Lower Bills Too?

Environmental efficiencies should equate to cost efficiencies and performance improvements.

For cloud scale – measured economically by FinOps – cost optimizations come to the forefront. FinOps, CloudOps and AIOps practitioners should complement common Performance SLAs with Cost per Query and Queries per Watt indicators, like in the following chart.

In an ideal world, Queries per Watt (sustainable computing), Cost per Query (FinOps) and SLA efficacy (throughput over a given time) should correlate. When they do, performance, cost and sustainability harmony ensue. When high performance comes at the cost of low numbers of Queries per Watt and/or high Cost per Query measures, bloat ensues, exacerbated at cloud and GenAI scale.

(Above) The Virtuous Trinity of Cloud Scale Analytics: Optimal Scaling in the Presence of Lowest Costs

Download our whitepaper, ‘Cultivating Sustainable Computing and Environmentally Responsible Analytics in an AI Age’ if you’re interested in ESG issues and want practical solutions to your business needs.

I also recommend following and engaging with the Green Software Foundation.

RELATED RESOURCES

Forrester TEI Study – The ROI of Real-time Data Analytics

Tackling Data Challenges with KX and Capco: A Fresh Perspective

KX and [ui!] Deliver Advanced Data Management and Time Series Analytics for Smart City Solution Provider ENE.HUB – KX

The Gartner® Assess the Value and Cost of Generative AI With New Investment Criteria report provides a decision framework for assessing and realizing value from enterprise generative AI initiatives. Organizations need to architect transformative use cases that deliver competitive advantages amidst industry disruption from new GenAI products and business models. I found the report’s accompanying background strategic planning assumptions striking. According to Gartner:

• By 2025, growth in 90% of enterprise deployments of GenAI will slow as costs exceed value, resulting in accelerated adoption of smaller custom models.
• By 2026, more than 80% of independent software vendors (ISVs) will have embedded generative AI capabilities in their enterprise applications, up from less than 1% today.
• By 2028, more than 50% of enterprises that have built their own large models from scratch will abandon their efforts due to costs, complexity and technical debt in their deployments.

For the remainder of this blog, I’ll give my perspective on the techniques and technologies that can help you reach those right transformative use cases and highlight the role of the vector database.

Leveling the Playing Field Through Generative AI

First, some context about the democratic opportunity for all. Generative AI is a great disruptive leveler. All firms and sectors can stand on the shoulders of giants to be successful. Past leaders in AI exploitation may not be the future leaders. They who dare, win.

From support and marketing chatbots to automated customer emails, most organizations have AI projects under way. Perceived industry leaders in big tech, high tech, and capital markets have long used structured data and “discriminative AI” to observe, predict, and find anomalies. They have serviced use cases such as detecting fraudulent transactions or trades, predicting equipment failures of key assets (defense, space, manufacturing, connected vehicles), personalizing social media recommendations, or dispensing robo-advice.

Some sectors may appear to be less out there, e.g. consumer, health, and insurance, but if you take this view don’t be complacent. Many actuaries claim they were the original data scientists and predictive modelers, and they have a point. Reinsurance stacks are every bit as sophisticated as the smartest hedge funds, servicing more challenging pricing and risk regimes. I know, I’ve worked with both. Medical devices have long been a focus of AI, while predictive healthcare has already affected my health, in a good way.

Whatever your starting point, I offer three tips for successfully implementing differentiating and transformative use cases soonest, with the least technical debt:

1. Prepare data well – It’s not enough to have data; it needs to be in the right format, organized, and engineered quickly. Make sure your tooling can expedite essential data engineering operations efficiently, for example, join disparate data-sets and perform filtering and aggregation.

2. Select the Large Language Model (LLM) that’s right for you. There’s no right answer. “Open source” (e.g. LLaMA, Falcon) versus proprietary (e.g. GPT, Bard) gets debated online. We at KX strive to give you the option of working with different LLMs, as we do other LLM-neutral developer tooling such as LangChain. AWS’s Eduardo Ordax’s guidance on LLMOps and what he calls FMOps (Foundational Model Ops) is also helpful:

1. Resource appropriately for providers, fine-tuners, and consumers. Lifecycle demands differ.
2. Adapt a foundational model to a specific context. Consider different aspects of prompting, open source vs. proprietary models, and latency, cost, and precision.
3. Evaluate and monitor fine-tuned models differently. With LLMs, consider different fine-tuning techniques, RLHF, and cover all aspects of bias, toxicity, IP, and privacy.
4. Know computation requirements of your models. ChatGPT uses an estimated 500ml of water for every five to 50 prompts. Falcon 180B was trained on a staggering 4096 GPUs over 7M GPU hours employing huge computational resources – computation requirements matter. OK you’re unlikely to train Falcon, but if you use it or anything else, know what you’re consuming.

3. Determine your optimal “taker, shaper, or maker” profile

• #taker - uses publicly available models, e.g. general-purpose customer service chatbot with prompt engineering and text chat only via interface or API, with little or no customization.
• #shaper - integrates internal data and systems for customisation e.g. get data from HCM, CRM or ERP into LLM with fine-tuning of company data.
• #maker - builds large proprietary models. To my mind, only specialists will adopt this trajectory. As previously mentioned, according to Gartner, “By 2028 more than 50% of enterprises that have built their own large models from scratch will abandon their efforts due to costs, complexity and technical debt in their deployments.”

The Role of Vector Databases

Until now, I’ve not referenced a vector database. A vector database is a shaper technology that I believe offers the greatest opportunity to implement your golden use cases. Think of a vector database as an auxiliary to an LLM. A vector database helps you quickly find the most similar embeddings to a given embedding with customization. For example, if you’ve embedded the text of a search query and want to find the top 10 documents from your organization’s legal archive most similar to the query in order to assess the new case.

Vector databases should therefore help you incorporate contextual understanding with your own data into the search. In short, use vector embeddings of your own datasets to augment queries to your favorite LLM with supporting validated information. Then, query and search for contextualized differentiating and transformational use cases relevant to your organization, reducing risks of hallucinations.

Not all vector databases are the same. Some are more flexible with datatypes allowing combinations of generative and discriminative AI. KDB.AI is one such database. Make sure your vector database is efficient. Think of the environmental and financial costs of your search. Ensure your database matches your business needs. If your organization requires searches with to-the-minute data sets, choose one which can work with edge workflows. Choose wisely.

No need for most organizations to “make,” but stand on the shoulders of giants to “shape” for optimal impact and YOUR golden use cases.
Download the Gartner ‘Assess the Value and Cost of Generative AI With New Investment Criteria’ report.

Alternatively, visit KDB.AI to learn more about vector databases.

Gartner, Assess the Value and Cost of Generative AI With New Investment Criteria, Rita Sallam, James Plath, Adnan Zijadic, Pri Rathnayake, Graham Waller, Matt Cain, Andrew Frank, Bern Elliot, 13 July 2023. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

This KX article is by Steve Wilcockson

RELATED RESOURCES:

Exploring the Gartner® Report: Applying AI – Techniques and Infrastructure

Why Do You Need Vector Processing to Win in the AI Era?

3 Real Capital Markets Examples that Unlock GenAI Value

By Steve Wilcockson

kdb+ is well known as a high-performance, in-memory database optimised for real time analytics on timeseries data. It is designed to handle large volumes of data and complex queries efficiently and cost effectively, using multithreading capabilities to optimise memory and compute resources. Those capabilities for parallel processing similarly enable horizontal scaling for extreme workloads and ultra-high-performance use cases. A number of its parallelization features are outlined below:

1. Parallel Query Execution: kdb+ has built-in functionality to automatically parallelize queries over multiple CPU cores or threads. This can lead to significant performance improvements for complex queries. Read this paper for more information on multi-threaded primitives in kdb.
2. Vectorized Operations: kdb+ is known for its vectorized operations, where operations are applied to entire arrays of data rather than individual elements. This enables parallelism at the instruction level, as operations can be applied to multiple data points simultaneously.
3. Data Partitioning: kdb+ allows you to partition your data across multiple nodes or servers. This data partitioning can be used to distribute the workload across multiple processors or machines, enabling parallel processing of queries.
4. Parallel I/O: kdb+ optimizes its input and output operations for parallelism. This means that when reading or writing data, kdb+ can take advantage of parallel disk or network operations to maximize throughput.
5. Interprocess Communication: kdb+ provides efficient interprocess communication (IPC) mechanisms, such as messaging and shared memory, which enable parallel processing across multiple kdb+ instances or nodes.

It is important to note that while kdb+ provides these capabilities for achieving parallelism, effective parallelization often requires careful design of data structures, queries and overall system architecture. Properly partitioning and distributing data, as well as optimizing query logic, are essential for achieving the best performance in a parallel processing environment. This whitepaper discuss these other points in more detail.

For further information on kdb please visit code.staging.kx.com 

By Steve Wilcockson

A new kdb Insights extension for Visual Studio Code (VS Code) from KX enables programmers to develop and maintain kdb Insights data analytics applications with the popular VS Code Integrated Development Environment. By enabling users to edit q files, connect to kdb processes and run queries, the plugin enables rapid development and implementation of high-performance data analytics applications powered by kdb.

VS Code is a lightweight, powerful source code editor made by Microsoft with the Electron Framework. It is designed to be open and supports multiple programming languages like JavaScript, TypeScript and Node.js as well as C++, C#, Java, Python, PHP, Go, .NET. The kdb VS Code plug-in extends that list to include q. It provides interactive code editing capabilities, including syntax highlighting, linting and code prediction for writing and visualizing the results of q queries.

The screenshot below illustrates some of its code navigation functionality.

The plugin facilitates application development convenience in an extremely popular IDE used widely by developers, data scientists and data engineers. It can be used with kdb+, including its Personal Edition, kdb Insights, including its Personal Edition for writing, running and exploring query results in q. Both are extended by a version for kdb Insights Enterprise that uses a shared kdb process to add data sources, create data pipelines and interact with API endpoints.

Please visit the Visual Studio marketplace for more detailed information on the functionality of the kdb VS Code plug-in, an installation guide and a Q&A.

By Steve Wilcockson

If you’re reading this, you likely want to add speed and performance to your Python code and notebooks while also reducing time from research to production.

In our whitepaper Accelerate Your Python Development Journey With PyKX, we’ve compiled a series of use cases, technical examples, and some “Getting Started” guides to introduce PyKX and how it provides interoperability between Python and kdb.

Having spent 25 years with quants, data scientists, and, more recently, data engineers, architects, and CTOs, I’ve seen Python emerge from a fringe functional language into a functional language powerhouse, driving data science and AI forward. I witnessed its rise in the late 2000s, and now work for an organization that embraces its culture and helps Python achieve new things in production big data analytics.

Python from the Other Side

I joined KX in May 2022. Prior to this I’d worked with two technologies strongly impacted by the rising tide of Python.

For many years, I worked with MATLAB, which, between 2010 and 2015, was the language of data science before Python (and arguably R) took over.

MATLAB is the language that Geoffrey Hinton used to explore neural nets (I had the pleasure of speaking with a university secretary when he needed an academic license). And Andrew Ng based his first online data science course around it. In fact, numerous other greats of the industry built their neural nets models and developed machine learning and statistical functionality with MATLAB.

I then worked with Java, the most popular language of the enterprise and in computer science education, until Python took over, in popularity terms at least.

I specifically note November 2020 when the influential Tiobe Index placed Python above Java for the first time, a trajectory that has not (yet) reverted. When I speak with practitioners, I increasingly hear phrases like “We no longer teach Java in our Computer Science course; we’re using Python primarily,” “Java is for us a legacy, albeit a useful one, but the new stuff is primarily Python,” and “Docker and Kubernetes changed everything – Java is no longer required, and I can more easily take languages like Python into cloud and microservices environments.”

It was tough to take because I was loyal to the languages I served – MATLAB and then Java – but I really loved Python’s focus on open source and community. On one hand, Python was simple. I give you some code, and you run it in your Python IDE or notebook. You didn’t have to “buy” a license for something.

But there was much more besides. If the community wanted something, it built it. Indeed, the reason I believe Python took off was because a chap called Wes Mckinney developed Pandas in 2008.

Pandas is essentially a data management and analysis application that wraps up low-level mathematical optimizations and operations of key Python package NumPy into simple data transformations, joins, aggregations, and analyses.

In short, Pandas made Python usable for everyday numerical applications, which happened to include big statistical calculations. Pretty soon, if not overnight, those who were from a world where “My professors taught me MATLAB during my master’s degree” were now in a situation where they could say: “So I studied 76,948 Python code repositories to teach myself Python.”

Python: The Language of Data Science

That key bit of functionality Wes McKinney developed gave coders a great technical reason, beyond “it’s free,” to build statistical, time-series, and machine-learning applications in Python. It changed the game.

What happened next? Well, Pandas invigorated the development and application of scikit-learn, the key machine-learning package in the Python ecosystem. In the meantime, Big Tech built and open-sourced deep learning code bases, for example, Facebook with PyTorch, Google with Tensorflow, Berkeley with Caffe.

For those familiar with OpenAI and LLaMa, PyTorch is the training giant on which LLM giants stand.

Thus, Python is now the language of Generative AI implementation, which is only natural as it’s been the language of data science, machine, and deep learning for 10 to 15 years.

But there’s a massive catch in the story I just told.

Python is the front-end, the thing people see and interact with. It’s a hub for data sets, running and sharing understandable code, often as Notebooks. The glue that binds the user experience together – allowing rapid, simple, reproducible prototyping and data science.

However, those deep learning libraries were mostly implemented in C++ or C. The predominant big and fast data platforms – those that stuck the course like Spark, Kafka, and Snowflake – were implemented, if not in C++, then in Java. The ecosystem that gives Python wings is commonly not Pythonic, but Python beautifully front-ends it.

Python: The Culture of Data Science

But Python is more than just a programming front-end, for standalone notebooks or extensive code bases powered by, and integrating with, a multitude of other technologies. It’s a vibrant culture, a global community, a Zen, a way of life.

Why KX?

So why PyKX? From a practitioner level, PyKX is a data and analytics technology that interoperates Python with KX production data management and analytics to give Python wings.

It’s supremely powerful, providing orders of magnitude greater efficiency and performance than Python and Pandas for key tasks, meaning it’s possible to take the best bits of both for something production-worthy sooner.

Pandas and KX are built from the same foundational mathematical and computational constructs.  The two technologies naturally align.

Incidentally, going back to my MATLAB days, while Python became the time series, matrix algebra, and vector processing tool of choice for research, in the sector where I operated (financial services), KX rapidly grew to be the same in production. I was squeezed on both sides! And I loved it! And the two environments work together.

From a cultural perspective too, KX is an organization that gets open source. I work with committed Pythonistas, which I love. They live the open-source mentality and participate in the Python communities, often alongside KX users who are open-source aficionados.

And that culture matters.

Read the whitepaper to learn how the technologies of Python and KX help accelerate performance, improve efficiency, and significantly reduce time to production.

To learn more about KX technologies, vector databases and time series analytics, sign up to your local Vectorize World Tour Community Meetup.

By Steve Wilcockson

AI has had many false dawns, but we are far removed from early days of “fuzzy logic.”

With the recent explosion of generative AI, AI is clearly evolving from a sideline research curiosity into an exciting core business imperative. The Gartner “Applying AI — A Framework for the Enterprise”, illustrated below, outlines how enterprises should approach their AI initiatives:

In outlining the fundamental techniques and practices for implementing AI within the enterprise, the report states, “IT leaders struggle to implement AI within AI applications, wasting time and money on AI projects that are never put in production.”

Organizations want simplicity and flexible tools for more tasks and use cases, from research to production. At KX, we help to “squish your stack” and get more data science applications into production sooner. Here’s how:

AI Simulation

Decades into the lifespans of digital transformation, there is little “low hanging fruit” left. Efficiencies are harder to find, anomalies harder to spot, and decisions involve more people, from Subject Matter Experts to IT Teams, CTOs, and FinOps teams.

Simulation and synthetic data have always been important.  From scenario stress testing to digital-twin modelling, results are inferred from simulations over swathes of historical and synthetically generated data – KX does this 100x faster at a 10^th of the cost of other infrastructures. Common simulation case studies include:

• Monte Carlo simulations for pricing and risk analysis
• Wind Tunnel analytics in FI
• Stress testing across more scenarios, particularly amongst high volume, outliers and burst conditions

 Data Science

Your investment in AI infrastructure is wasted if data scientists cannot explore data and models easily and flexibly. With KX, tooling is driven from SQL and Python, data, and modelling languages of choice.

• Researchers, developers, and analysts use their existing skills and code bases, from model research to reporting, and from cloud data pipelines to real-time model deployments
• With PyKX, get 100x power of kdb to Python developers and notebooks
• With Pro-code interfaces, users can collaborate and augment functionality from different programming languages empowered by the unsurpassed performance of kdb

AI Engineering

As AI goes mainstream, enterprise features and guardrails are needed to ensure stability, robustness, and security. For governance, cataloguing and security, kdb integrates with tools of choice to share data and AI insights throughout the enterprise safely, securely and with speed.

• Industrialize application development by supporting the reuse of existing libraries and skills
• Adopt open standards and cloud protocols to simplify integration of governance protocols, facilitate access and authentication, ensure security, safety, resilience, and high availability
• Abstract infrastructure concerns so that developers can focus on getting projects to production sooner, providing golden insights and ensuring business value

Compute Infrastructure

AI consumes very Big Data. That, unfortunately, means huge storage and compute requirements.  With KX, storage and compute efficiencies mean cost savings and lower environmental overheads.

• KX operates directly on data, eliminating the need for latency-inducing transfer and costly data duplication
• It’s column design provides optimized storage and better compression levels
• With a low KX footprint, CPU use is optimized meaning less need for power-hungry GPUs

Generative AI

With AI no longer simply discriminative or code-queried, but also generative and prompt-queried, KDB.AI has been built to apply what’s a given for GenAI workflows with greater efficiency, and augment with direct raw vector processing power. Sample use cases range from more accurate fraud detection and insightful risk management to automated financial analysis and improved regulatory compliance – all harnessing real time information, and particularly temporal data, for improved decision making.

• With proven vector-based representation and processing, KDB.AI provides more efficient vector database vector embeddings storage and search for all types of LLMs
• Additionally, storing and computing over raw data for large data models, including, but not limited to, time-series data models
• Prompt integration for better prompt engineering and more flexible intuitive querying

Gartner, Applying AI — Techniques and Infrastructure, Chirag Dekate, Bern Elliot, 25 April 2023. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.