How to Design a Price Drop Tracker System

TL;DR

To design a price drop tracker system, users can subscribe to price alerts for products on Amazon. The system scrapes Amazon for price updates, stores price data, and notifies users via email when prices drop below their set threshold. The architecture involves scrapers, databases for storing price data and alerts, and a notification service to handle user alerts efficiently.

Transcript

all right so we're going to design a price tracker like camel camel camel uh it specifically looks like it's focused on just amazon.com is what it tracks prices through so it kind of is going to scrape it this is taken from somebody's real uh system design interview at meta uh I think they only do uh system design interviews... Read More

Key Insights

• A price drop tracker system allows users to subscribe to alerts for price changes on Amazon products.
• The system uses web scrapers to monitor and update prices from Amazon at regular intervals.
• Two main databases are used: one for storing current prices and another for user price alerts.
• Users set a price threshold, and the system notifies them via email when the price drops below this threshold.
• Change Data Capture (CDC) with database triggers is used to detect price changes and trigger notifications.
• The system design must handle partial failures, such as failed scrapes or notification sends, with retries or dead letter queues.
• Adaptive scraping intervals can be implemented to adjust the frequency of price checks based on recent changes.
• Notification services can be either rolled out manually or by using existing solutions like AWS SNS for email alerts.

Questions & Answers

Q: How does a price drop tracker notify users?

A price drop tracker notifies users by sending email alerts when a product's price falls below their specified threshold. The system uses web scrapers to monitor price changes on platforms like Amazon. When a price update is detected, Change Data Capture (CDC) triggers the notification process, ensuring users receive timely alerts about price drops.

Q: What is the role of scrapers in a price drop tracker?

Scrapers in a price drop tracker are responsible for regularly checking and updating product prices from online platforms like Amazon. They run at defined intervals to ensure the price data is current. The scrapers update the price database, which is then used to trigger notifications if a price drop meets a user's alert criteria.

Q: How does the system handle partial failures in the scraping process?

The system handles partial failures in the scraping process by implementing retries and using dead letter queues. If a scraper fails to update the price data, the system retries the operation. Persistent failures, such as products being delisted, are managed by logging these cases in a dead letter queue for further investigation, ensuring reliability and robustness.

Q: What databases are used in the price drop tracker system?

The price drop tracker system uses two main databases: one for storing the current prices of products and another for managing user subscriptions and price alert thresholds. The databases are designed to handle high volumes of read and write operations, with considerations for partitioning keys and indexing strategies to optimize performance and scalability.

Q: How are price change notifications deduplicated?

Price change notifications are deduplicated by using a unique identifier (UUID) for each notification event. The system stores these identifiers in a deduplication database. Before sending a notification, it checks if the UUID already exists, preventing duplicate alerts. This ensures users receive only one notification per price drop event, avoiding unnecessary repetition.

Q: What is Change Data Capture (CDC) and its role in the system?

Change Data Capture (CDC) is a technique used to track changes in database records in real-time. In the price drop tracker system, CDC is used to detect when a product's price changes in the database. This triggers the notification process, ensuring users are promptly alerted to price drops. CDC helps maintain an efficient and responsive notification system.

Q: How can adaptive scraping intervals benefit the system?

Adaptive scraping intervals benefit the system by optimizing the frequency of price checks based on recent changes. If a significant price change is detected, the system can increase the scraping frequency to capture further updates quickly. Conversely, if prices remain stable, the interval can be reduced, conserving resources and improving system efficiency.

Q: How does the notification service ensure reliable delivery?

The notification service ensures reliable delivery by using a combination of message queues and database checks. Before sending a notification, the system checks a deduplication database to avoid duplicates. It then writes a record indicating the notification attempt. This approach, combined with retries and dead letter queues for failures, ensures robust and reliable alert delivery.

Summary & Key Takeaways

• The system design for a price drop tracker involves subscribing users to price alerts and notifying them when prices drop. Scrapers regularly check Amazon for price updates, storing this data in databases. When a price falls below a user's set threshold, they are notified via email. The architecture includes scrapers, data stores, and a notification service.

• Change Data Capture (CDC) is used to track price changes and trigger notifications. The system must handle partial failures, ensuring reliability through retries or dead letter queues. Adaptive scraping intervals can optimize performance by adjusting the frequency of checks based on recent price changes.

• A notification service can be implemented manually or using services like AWS SNS. The design should ensure efficient handling of user alerts and manage potential issues like duplicate notifications through deduplication strategies, ensuring users are not overwhelmed with alerts.