...
Jira Legacy server System Jira serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key PLFM-8485 Jira Legacy server System Jira serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key PLFM-8484 Jira Legacy server System Jira serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key DESIGN-1463 Jira Legacy server System Jira serverId ba6fb084-9827-3160-8067-8ac7470f78b2 key IBCDPE-1014
Introduction
To support searching across the content of Synapse the backend employs Amazon Cloudsearch as its index and query engine. Cloudsearch is a fully managed solution that supports both unstructured text and structured data, synapse today makes use of both types of data for its search capabilities and additionally makes use of row level filtering to exclude results that users might not have read access to.
...
The power of LLMs comes from their ability to adapt and take context into consideration, this allows to increase the performance for domain specific knowledge. There are a few techniques available today to provide the models up-to-date information that they can they adopt to more precisely answer questions:
...
Solution | Integration Complexity | Fully Managed | Comment |
|---|---|---|---|
Open AIOpenAI | Very High | No | Provide access to separate APIs that would allow us to build a RAG system, but it’s not a managed solution and would require a consistent amount of effort and maintenance to build. Custom GPTs would allow to build an application that integrates external knowledge for example from a search API. This might be worth investigating for “chatting with a project/folder”. |
Azure AI Search | High | No | See https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview. Similar to the Open AI solution, Azure provides all the tools needed (plus the search index management) to build a RAG solution but it would nevertheless be a big effort. |
AWS Bedrock | Medium | Yes | Provides a fully managed solution that is integrated with AWS, provide access to several open source models and does most of the heavy lifting. The knowledge base component is what enables the RAG system managing the storage and retrieval of external data. The default vector database is an AWS OpenSearch index. |
AWS Kendra | High | No | It’s a managed semantic search index (starts from $2k/month for 100K documents) that can potentially be used as part of a RAG system. Kendra can potentially be integrated into Bedrock as the backing document index. |
Vertex AI Search | High | Yes | Provides a (presumably) fully managed solution to deploy a RAG system using their own models. |
...
From the various options above our recommendation is to build a prototype based on AWS Bedrock, creating a knowledge base with at least public content (one idea might be to initially search only through projects, and in the context of a project/folder further customize the search/chat experience to the files/wiki/discussions contained in the project) and release it under a feature flag for the internal users to try out with a feedback collection mechanism (e.g. question/answer plus satisfaction score).
The choice is driven by the fact that:
We are already on AWS, we know how to use it and we do not have to learn another cloud platform/api
There is no easy way to test this out today quickly and compare across solutions. Ideally with a first prototype we could build a static knowledge base that can be reused, including the indexed content and the questions/answers plus feedback.
Bedrock seems to be the most comprehensive and easier to integrate solution out there, despite that it might not have the best models availablewith several tools that include model evaluation (https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation.html ), prompt management (https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-management.html ), content filtering (See https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html ) etc.
Bedrock provide access to several different foundation models that can be tried (See https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html )
Bedrock has native support for metadata filtering (https://aws.amazon.com/about-aws/whats-new/2024/03/knowledge-bases-amazon-bedrock-metadata-filtering/) that could be used for access control in a similar way we do with CloudSearch today.
We could potentially reuse OpenSearch for both normal search and the backing vector database
Pricing is a bit of a complex topic as in all the above solutions it follows a pay-for-what-you-use model that depends on the user input, number of tokens, type of model used and amount of data. Given my recommendation I can estimate the cost of bedrock to be around $1k-$2k/month depending on usage (the default vector database is AWS OpenSearch, which should cost between $400 and $800/month).
Note that Thomas Yu already setup a similar experiment with public wikis, see /wiki/spaces/DPE/pages/3492282370