Scaling up a Django + React Web Scraping Website

$30-250 USD

Avslutat

Publicerad

över tre år sedan

$30-250 USD

Betalning vid leverans

I am working on a platform that runs a Django/Postgres backend with a React frontend. The platform scrapes data from multiple sources with Scrapy, cross-relates them, and gives some basic analysis of them via the front-end and HighCharts. I am running into some issues with scaling the platform and some strange issues getting the entire system to work together. For example, running Scrapy & Twisted on a worker server has been problematic at times and I am having issues using Python multi-processing in tandem with the Django ORM (objects appear to be created randomly).

Project ID: 26870645

Om projektet

6 anbud

Distansprojekt

Senaste aktivitet fyra år sedan

Ute efter att tjäna lite pengar?

E-postadress

Fördelar med att lägga anbud hos Freelancer

Ange budget och tidsram

Få betalt för ditt arbete

Beskriv ditt förslag

Det är gratis att registrera sig och att lägga anbud på uppdrag

6 frilansar lägger i genomsnitt anbud på $161 USD för detta uppdrag

@kkc264043kkc

Can help in scaling the scraper with scrapy. want to know more about it. These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomjs, python. Have done testing and automation with selenium also. Know to deal with database like mongo, mysql, Elasticsearch. Also know to handle proxy and captcha while scrapping. Can scrape data from various websites . have done scraping of millions of products from Amazon eBay Walmart. have done scraping of social media websites like Facebook Instagram LinkedIn

$253 USD Om 2 dagar