Kong recently introduced a collection of six open source AI plugins for its Kong Gateway 3.6, which offer integration with multiple LLMs and significantly improve the accessibility of AI technology for developers and platform teams.
According to Kong, the goal of this release is to increase developer productivity by simplifying the process of integrating LLM into various products. It provides a secure, compliant and controlled environment for AI requirements within organizations.
“Today marks a significant milestone on our journey to democratize artificial intelligence for developers and businesses around the world. By open-sourcing this suite of innovative AI capabilities, including no-code AI plugins, we’re removing barriers to AI adoption and enabling developers to effortlessly leverage more LLMs and deliver AI-powered applications faster. At the same time, we provide governance and visibility for all AI traffic generated by the organization,” said Marco Palladino, CTO and co-founder of Kong Inc.
Kong’s introduction of AI capabilities into its Kong Gateway via the “ai-proxy” plugin enables the incorporation of multiple LLMs, such as those offered by OpenAI, Azure AI and others, providing a single interface that allows developers to switch between models without changing their application code. It also simplifies the management of AI credentials by centralizing them within the Kong Gateway. This approach improves security by abstracting credential management from applications, allowing for easier rotation and update of credentials, according to Kong.
The “ai-proxy” plugin extends its functionality into the realm of analytics by enabling the collection of detailed AI Level 7 metrics, including tracking the number of request and response tokens and usage data across different LLM providers and models. With support for integration with analytics platforms such as Datadog and New Relic, along with Kong Gateway’s native logging plugins, developers gain valuable insights into the usage patterns of their AI integrations.
Additionally, the introduction of code-free AI integration plugins, mainly “ai-request-transformer” and “ai-response-transformer”, allows users to inject AI capabilities directly into API requests and responses, enabling real-time API-like functions response translation for broader internationalization efforts without the need for coding.
Kong also introduced plugins that specialize in creating, decorating and managing AI queries. The “ai-prompt-template” plugin enables centralized management of prompt templates, enabling quick updates and compliance with approved templates without the need to modify application code. The “ai-prompt-decorator” and “ai-prompt-guard” plugins provide mechanisms to ensure that AI prompts comply with organizational policies and are protected from unauthorized or sensitive content.