From Data to Insights: Building a Bluesky Bot powered by AI

A common challenge developers face when working with data streams is collecting and analyzing this data as fast as possible to uncover meaningful insights. It’s a complex problem that requires the right combination of real-time data technologies and AI for instant, intelligent decision-making.

In this talk, I’ll show you how I tackled this by building a Bluesky bot that turns raw data into actionable insights using GenAI. We’ll dive into the process of collecting data, transforming it into streams, and using Redis 8 to power real-time analysis. Along the way, I’ll explore how probabilistic data structures, like Count-Min Sketch and Bloom Filters, help optimize performance and enable scalable analytics without compromising accuracy.

I’ll also demonstrate how Redis 8 supports vector similarity search, making it possible to compare and classify data efficiently—an essential step for enhancing AI-driven insights. You’ll see how this can be applied to find patterns, group similar content, and make smarter recommendations.

Finally, I’ll bring it all together by showing how Redis and GenAI work hand in hand to extract patterns and generate insights, with practical examples implemented in Java.

Whether you’re curious about GenAI, interested in data-driven analytics, or simply love experimenting with creative tech solutions, this session will inspire you with practical techniques and real-world applications.

Transcript

1. Building a Bluesky Bot powered by AI Raphael De Lio
2. Who is in Bluesky?
3. Decentralized Network
4. Bluesky’s Jetstream
5. How many posts are created in Bluesky per minute?
6. Approximately 2200
7. Approximately 95 million per month
8. How can we efficiently analyze this data?
9. What are we building today? • Listen to Bluesky’s Jetstream • Filter in posts related to AI • Enrich the data from these posts with sub topics • Keep track of frequency of topics • Allow Bluesky users to interact with our bot and ask questions
10. How are we gonna do that? • Redis Streams • Semantic Classification • Semantic Routing • Semantic Caching • Probabilistic Data Structures
11. Challenge #1: Storing the messages
12. Why? • Messages are ephemeral • We want to process these messages with multiple services.
13. Redis Streams
14. • In-memory first (Fast) • Easy setup • Support for Consumer Groups • Perfect fit for realtime pipelines
16. Application 1 Application 2 Application 3
17. Application 2 Application 3 Application 1 Consumer 1 Consumer 2
18. Demo time!
19. Challenge #2: Filtering the messages
20. Classifying Messages about Artificial Intelligence Why?
21. All Bluesky Posts Challenge #1 Filtered Bluesky Posts Filtering Service Challenge #2
22. Filtering Service
23. Approach #1: Using an LLM
24. Filtering Service Is this post about AI? Filtered Bluesky Posts Store the post in Redis as a Hash or a JSON
26. Approach #2: Using Vector Database (like Redis)
27. Generate 500 Bluesky Posts about AI Turn these samples into Vectors Store the vectors in a Vector Database
28. Filtering Service Turn the post into a vector Compare the vector of the post with the vector of the samples in the Vector Database Write the post to the filtered-posts stream Store the post in Redis as a Hash or a JSON
29. Demo time!
30. Challenge #3: Enriching the data
31. Topic Extraction
32. Increase the granularity of what’s being talked about Why? Keep track of how popular these topics are
33. Filtering Service Filtered Bluesky Posts Challenge #2 Challenge #3 Topic Extractor Service
34. Topic Extractor Service
35. Topic Extractor Service What topics can be implied from this post? Update post in Redis (Hash or JSON) Increment topics counters in the TopK
36. Demo time!
37. Demo time!
38. Challenge #4: Analyze the data & take action
39. Understand what functions to call
40. I have two functions: • Listing trending topics • Summarizing posts
41. Approach #1: Using a LLM
42. Bot These are the possible functions: […] What function should I call based on the post? Bot Invokes the specific function Generates final response with information from the function Bot Reads post that mentions the bot Posts response
44. Approach #2: Using a Vector Database (like Redis) (Semantic Routing)
45. Generate references for a certain route Turn these references into Vectors alongside their route Store the vectors in a Vector Database
46. Bot Turn the post into a vector Compare the vector of the post with the vector of the routes in the Vector Database Invoke returned route (function) if similarity is high enough Bot Generates final response with information from the function Bot Post response
47. Demo time!
48. Challenge #5: Repeated Questions
49. Users may ask the same question phrased differently
50. Semantic Cache
51. Demo time!
52. Challenge #6: Deduplication
53. Messages are delivered At Least Once
54. Approach #1: Using a SET
55. Consumer Stream Process Message Check if ID exists in SET Consumer Stores ID in SET
57. Approach #2: Using a Bloom Filter
58. Consumer Stream Process Message Check if ID exists in BLOOM FILTER Consumer Add ID to the Bloom Filter
59. Demo time!
60. Recap • How to use Redis Streams for consuming realtime data • How to use Semantic Classification for filtering data • How to use LLMs to extract data • How to use Semantic Routing for calling functions • How to use Semantic Caching for saving money & time • How to efficiently analyze huge data streams with TopK and Bloom Filter
61. RAPHAEL DE LIO DEVELOPER ADVOCATE *