Amazon web scraper, web scraping into Magento MySQL database
$250-750 AUD
Avslutat
Publicerad ungefär tio år sedan
$250-750 AUD
Betalning vid leverans
We would like to scrape Amazon products with photos and import them into our Magento website by providing a URL to scrape from.
The solution has to be as simple as possible.
We have a dedicated server with full access. It is an Apache/CentOS/MySQL/WHM/PleskPanel environment.
For example, we would like to scrape all 9000+ products from the URL:
http://www.amazon.com/s/ref=sr_hi_3?rh=n%3A172282%2Cn%3A!493964%2Cn%3A541966%2Cn%3A1232597011&hidden-keywords=-B0089RPUHO+-B00DG0NUEE&ie=UTF8&qid=1385551805
Another sample URL:
http://www.amazon.com/s/ref=sr_nr_p_89_2?rh=i%3Aaps%2Ck%3Aspigen+case%2Cp_89%3ASpigen|SGP&keywords=spigen+case&ie=UTF8&qid=1391908765&rnid=2528832011
Rules:
1) Scraped products need to match our product columns (we will provide you with a CSV file so that you know how to match the columns)
2) Your scraper will need to interact with our MySQL database and archive products that are no longer available (the scraped data needs to go directly into our database, we are not interested in creating XML or CSV files for import)
3) We must have the ability to scrape by brand, category, price range, Amazon store etc. Basically any URL should work.
4) We need to scrape all product info as directed, for example: SKU, title, availability, price, colour, description, short description and so on
5) You need to be able to employ solutions to defeat current anti-scraping methods: proxies, random scraper pauses etc
6) The scraper needs to have an UPDATE function where we can scrape a URL and only update certain attributes (ie. price, availability) - that way we don't have to scrape all items in full again (and this will save a lot of time on our end)
7) You must create a Magento ADMIN menu for the scraper (we are not interested in running scripts via SSH etc)
So in a nutshell, we enter a URL in our Magento ADMIN and the scraper starts importing all products from that URL into our MySQL database.
What we have:
1) Magento online store, running CE 1.8.1
2) MySQL database
3) CentOS dedicated server will full root access
Please ask any questions before bidding. If you bid a certain amount, do not try to revise it later on.
Good luck!
Hi
I am a developer and I can develop a script to scrap the data. I have kept the cost a little on higher side and will stick to it.
Message me if you are interested in working with me and we shall discuss the project.
You can check my latest feedback and ratings at https://www.freelancer.com/u/madhavdutta.
I have made a bid only for script. If you want the website to be integrated as well, it will have additional cost.
Also, it is illegal to scrap data from Amazon and if you are caught, I will not be a part of it. I will be signing a legal declaration for it.
Hi,
I was a Mangento developer for about 2 years and i've made over 50 scrapers (including an Amazon scraper). Be advised the api has certain limitations, the scraper i made a while back doesn't.
My bid includes 2 months of free support.
Warm regards,
George
Sounds like a fun project!
Using php I plan to send out scraping bots through randomized vectors(to escape detection) to scrape the data you need. As the data is found it will be inserted into your database...this sounds like a rather simple project.