Did you take the Langchain RAG 8 chapter tutorial? It is really a great one and their Jupyter notebooks are a great resource. Maybe you can share what you’re not clear on and we can help answer or guide you.

I teach ai and can pass this along to my students. This semester they have been working on rapid prototyping apps using those exact libraries in python. Send me your email and I’ll share it TM with them. Specifically I had them work with spaCy, PyMuPDF (fitz) and they built a complete RAG solution using openAI and unstructured.io

Thank you Brandon for the heads up. With your permission - I am a member of the community and for those who would like to get together as a community, I will open a zoom call at noon EST in case you want to have a informal chat: https://fdu.zoom.us/j/9447104575 I can record the call and share it in this thread afterwards

If I understood your concern - I believe he starts showing a 1-shot (1 question 1 answer) and then moves on to explaining how he realized he needed to ingest the entire thread of back and forth to create complete question/answer pairs. Your approach might be similar - you would look for topic based groups of q&a gathering the customer journey documents (from emails, support chats, CRM, etc) and create a topic question:answer pair using prompts that extract such pairs from the source materials you provide. Hope that makes sense?

a few other thoughts & suggestions for you, Sam: 1. Vector Database and Chunking Strategy: - Hierarchical Chunking: Consider a hierarchical approach to chunking your data. Start by chunking individual messages but also create embeddings for sequences of messages (like conversational turns). This way, you can capture both fine-grained and broader context. I've had success with unstructured.io for ingest, partitioning and chunking. - Metadata Enrichment: When creating embeddings, include rich metadata like property IDs, guest journey states, timestamps, and platform identifiers. This will help you filter and retrieve relevant information more effectively. - Separate but Connected Vector Spaces: You might benefit from using separate vector spaces for property-specific information and global policies but ensure they're connected through metadata. This allows you to search within a specific context while still accessing overarching policies when needed. 2. Temporal Relevance: - Time-Decayed Embeddings: Implement a temporal decay mechanism where the relevance of an embedding decreases over time unless specified otherwise. For instance, WiFi passwords might have a higher decay rate compared to static info like nearby parks. - Query Classification: Develop a system to classify queries based on their temporal sensitivity. This can help in deciding whether to prioritize recent data or include older conversations. 3. Context-Aware Retrieval: - State-Aware Retrieval: Incorporate the guest's journey state into your retrieval logic. For example, if a guest is in the "BEFORE_BOOKING" state, the system knows not to share sensitive info like lockbox codes. - Property Relationships: Use the metadata to understand property hierarchies and similarities. When a guest asks about an issue with Property X, the system can also retrieve information from similar properties in the same building. - Balancing Specific and Global Info: When retrieving data, pull both property-specific details and global policies, then let your response generation logic determine which information is most relevant to include.