Back to Market
Crawler API
Provides a modular web crawler and a REST API for managing and storing webpage data, integrated with an MCP server.
1
This project serves as a comprehensive learning resource, featuring a robust web crawler designed for modularity and efficient task queuing. It's complemented by a powerful REST API, enabling full CRUD operations for webpage data, and an MCP server, facilitating seamless integration with external LLMs and AI agents. With automatic Swagger documentation, TypeORM/PostgreSQL integration, and comprehensive unit test coverage, it offers a solid foundation for developing intelligent data collection and management systems.
Database Management
API Development
Web Scraping & Data Collection
