Introduction

This project is a web scraping and crawling application built using the Selenium package in Python. The application allows users to search for products and retrieves their names and links, which are then displayed in a table. The backend is built using FastAPI and PostgreSQL, while the frontend is developed using Next.js.

Features

• Web scraping and crawling using Selenium
• Product search functionality
• Retrieval of product names and links
• Storage of search results in a PostgreSQL database
• Display of search results in a table using Next.js

Technologies Used

• Python
• Selenium
• FastAPI
• PostgreSQL
• Next.js

Installation

Clone the repository for the Backend: https://github.com/Elanchezhian2712/FastAPI_and_Nextjs_WebScraping_WebCrawling.git

Create a virtual environment and activate it:

python3 -m venv env
source env/bin/activate

Install the required dependencies:

pip install -r requirements.txt

Run the FastAPI server:

uvicorn main:app --reload