Build a Website Scraper with Astra DB + Python Examples | DataStax

Learn how to scrape websites with Astra DB, Python, Selenium, Requests HTML, Celery, & FastAPI.180 minutes • AdvancedUpdated October 19, 2021180 minutes, Advanced, Start BuildingLearn how to scrape websites with Astra DB, Python, Selenium, Requests HTML, Celery, & FastAPI by following along with CodingEntrepreneurs' video, located here.Quick StartSignup for DataStax Astra, or login to your already existing account.Create an Astra DB Database if you don't already have one.Create an Astra DB Keyspace called sag_python_scraper in your database.Generate an Application Token with the role of Database Administrator for the Organization that your Astra DB is in.Get your secure connect bundle from the connect page of your database and save it to your project folder. Rename it to bundle.zipSetup your system: Below is a preflight checklist to ensure you system is fully setup to work with this course. All guides and setup can be found in the setup directory of this repo.Install Selenium & Chromedriver - setup guideInstall Redis - setup guideCreate a virtual environment & install dependenciesSetup an account with DataStaxCreate your first AstraDB and get API credentialsUse cassandra-driver to verify your connection to AstraDBHow this worksFollow along in this video tutorial: https://youtu.be/NyDT3KkscSk.This series is broken up into 4 parts:Scraping How to scrape and parse data from nearly any website with Selenium & Requests HTML.Data models how to store and validate data with cassandra-driver, pydantic, and AstraDB.Worker & Scheduling how to schedule periodic tasks (ie scraping) integrated with Redis & AstraDBPresentation How to combine the above steps in as robust web application serviceHere's what each tool is used for:Python 3.9 download - programming the logic.AstraDB sign up - highly perfomant and scalable database service by DataStax. AstraDB is a Cassandra NoSQL Database. Cassandra is used by Netflix, Discord, Apple, and many others to handle astonding amounts of data.Selenium docs - an automated web browsing experience that allows:Run all web-browser actions through codeLoads JavaScript heavy websitesCan perform standard user interaction like clicks, form submits, logins, etc.Requests HTML docs - we're going to use this to parse an HTML document extracted from SeleniumCelery docs - Celery providers worker processes that will allow us to schedule when we need to scrape websites. We'll be using redis as our task queue.FastAPI docs - as a web application framework to Display and monitor web scraping results from anywhere