I have a program written in VB.NET that scrapes data from a public site. The program submits a case number to the site, the site looks up the case, and then my program stores the html returned by the site into an Access database.
My problem is that after looking up a few hundred cases, the site realizes my program is a bot, and it stops serving the pages to me.
My proposed solution is to use a proxy server ([login to view URL]) to use different IP addresses when scraping the site. Also, I think I should randomize the order of cases that I process. For example instead of scraping the data for ID 101V, then 102V, then 103V, 104V, 105V,106V, 107V, 108V, 109V, 110V, I think I should scrape the data in a random order like: 108V, 102V, 105V,110V, 101V, 107V, 109Vv, 104V, 103V, 106V.
I think this is an easy project. I just don’t have time to do it.
I can pay you $100 to do this project. And I will pay for the cost of proxy server.
Please let me know when you could have this done by. If your proposal looks ok, I will send you the VB.NET code and the Access database.
Thanks.
Hi there.
I know which skill are necessary for you and I am expert in this fields.
You can confirm about it via my previous client's reviews.
Hope to contact you.
Hi there,
I am Rashid, I am a full stack developer with 4+ years of experience in both front-end and back-end development. I have read your
use proxy server for VB.NET data scraping program description very carefully and would like to have a detailed chat about this project as to resolve some queries I have
regarding this project that needs to be cleared to get things started. We always look for a long term relationship. Moreover, we are a
team with expertise in Data Mining and VB.NET.
Looking forward from your response,
Thanks
Rashid Amjad.
✔ Expert Here || Available Now.!
Hello.! Will C. I’ve already understood the project because it requires the expertise in VB.NET and Data Mining. I've expertise in 400+ Technologies & awarded multiple times.
You will be amazed by having a look at my "Completed projects", "Success Rate" and "Verified Portfolio". I also have Passed the verification by Freelancer.
Our Expertise including:-
✔ 25+ programming languages.
✔ 37+ Frameworks.
✔ AI, DL and ML.
✔ AR/VR Technologies.
✔ Networking..
✔ VFX and CGI.
✔ Graphic Designing/Video Editing
✔ Complete Digital Security.
✔ Blockchain.
✔ CMS Systems.
✔ Game Development.
✔ Research & Writing etc..
I'm the only Freelancer who is selling "Premium Domains", Source Code of Enterprise "Games", "Apps" & "Software". I could also provide Premium "UI Elements" with 40%-90% Discounts.
We are a small team of best buddies who love to help clients to solve Complex Problems. We 100% Guarantee about results, As you can see we've a strong background of Successful Projects.
Feel comfortable while contacting, I am waiting for your response.
Thanks and regards,
Hamza Ali k.
HI
What would you expect from the best candidate?
I have scraper thousand of cases from different counties of different states.
What approach do you use for scraping? With Chrome? Within VB.net? If different way?
What volume do you want to scrape?
Depending on the volume the proxy might not be the solution for you.
What other issues do you experience while scraping? Any error with data?
Have you checked the data scraped?
How about awarding me this job and start working on it right away?
Bests,
George