Stage 1
Scraping is to be performed on the nine sites listed below. It is assumed that this number will increase, so scraper and database need to be easily capable of adding other sites in the future.
Create a MySQL database with individual tables for the freelance sites that appear below. In the first instance determine the fields required to capture all the projects and all employers scrape data into separate tables.
Guru
Elance
GetACoder
Get A Freelancer
Rent A Coder
Go Freelance
oDesk
ScriptLance
Project4Hire
Stage 2
Design MySQL Schema with translation tables.
As an example categories in Elance are not the same as the other scraped sites
Company
HrlyRate (number only)
Biz
Location
Summary Skills
Employment
AboutUs
PaymentTerms
ServiceDescription
Jobid
Website (Elance)
JobName
MinedTime (not in UNIX epoch format but 2003-01-06 00:00:00*)
Category (becomes CategoryID with link to translator table of generic categories for all site scrapes)
Description
Details
ProposalsReceived
Budget
ClientLocation
PostedTime*
EndTime*
Originator
Stage 3
Agree feeds & database schema
Provide automated update to content every 24 hrs