Ditutup

Multi-Threaded PHP Search Engine Scraper

Task:

Create a multi-threaded PHP script that will scrape data from search engines and record the scraped data into a sql database. Also create a Proxy scraper / tester.

Conditions:

Forum / Board url scraper

1. The script will run on a dedicated Linux server.

2. The Script will include a web browser type gui for total user control.

3. The Script will include updated stats as the scraper is progressing.

4. The Script will upload a compressed file of search term files and decompress and loop thru said search term files. The search term files will contain search terms 1 term per line.

5. The Script will have the ability to search Google, Google Blogsearch, Yahoo, MSN & Boardreader with the ability to add engines in the future.

6. The Script will have the ability to use proxies (see PROXIES below)

7. The script will include a gui to enable user to chose proxy settings for each search engine. Proxy settings will be proxy timeout time in seconds & whether or not to use proxies.

8. The script needs to allow user to choose the ips to broadcast to the search engine in the event no proxies are used.

9. The script needs to allow the user to set the thread timeout for each search to avoid proxy banning by the search engines.

10. The script needs to include error handling for captchas. If a Captcha image is encountered the script needs to choose a new proxy and try again with the same keyword.

11. Running stats for total urls found

12. Running stats for keyword list

Example keyword 238 of 2,000

13. Running stats for keyword file

Example: file 57 of 400

14. Running stats of running time and time to completion

15. Pause Resume & Stop Functions

Proxy Scraper

1. The proxy scraper will scrape urls from google's cache based on urls input via a text file.

2. Visit the google cache links and retrieve both the google "text only" cache for these links obtained by adding "&strip1" to the url and the actual link to the site with the proxies.

3. Provide the choice of downloading the google cache proxy links the regular proxy links or both.

4. Provide the option to use proxies when downloading the google cache proxy links and use proper proxy error handling as outlined above.

Proxy Tester

1. Query an environment script to check for anonymity

2. Options to set connect timeout when testing proxies.

3. Option to set time to retest proxies.

4. Create seperate local files for each search engine that encounters

a captcha (see above).

Example:

proxy [url removed, login to view] encounters a captcha while attempting to

scrape from google that proxy should then be placed in the google's

bad proxy text file as to not be used again with google. There would

be a seperate file for all engines.

5. The ability to set a time to reset the files in step 4.

6. Reset the files in step 4 whenever the program is closed (not paused).

STANDARDS:

To be coded entirely in PHP

C/C++ should be used where PHP is not possible/plausible

The Anatomy of a search:

1. Queries engine 1 for dog.

2. Collect applicable url on landing page.

3. Press the next button (if available) and collect all urls.

4. Repeat step 3 until not available.

5. Remove duplicates from applicable urls

a) To do this it would query the good url db and the timeout url db

6. Connect to all urls collected (without the use of proxies after removing duplicates and again scrape all applicable urls.

7. Place timeout urls in the timeout db and good urls in the good url db. (the timeout for connecting to sites is set by user in the settings).

8. Have to ability to retest timeout urls as a separate process.

In addition it should collect the following information and place into the DB.

1. Google PR of the actual full URL (not just the domain name).

2. Age of the domain name.

3. Number of back links that URL has going to it

4. Meta-Tag keyword

a) create an entry to the the db for every meta tag keyword associate with the url.

Example: Keyword = dog Meta-Tags = dogs, dog toys, dog food, dog treats", the scraper would make 4 'entries' into the sql database, and record that URL 4 times. So that when I query the database for 'dog food', or for 'dog treats', that same URL would be returned, along with any other URL that also had dog food or dog treats in its meta tag keywords. If a URL has no meta tag keywords, then obviously there is nothing to record for that. In this instance, if a URL has no meta tags, I still want all available data saved into the database

5. Date URL was scraped

All 5 variables must be searchable in the browser.

EXAMPLE:

Search for URLS with a Google PR of 3 or higher, with domains that are 1 year old or older, with 1,000 back links or more, with keyword 'dog toys' in meta tag keywords, scraped within the past 90 days.

The interfaces should have the ability to choose any or none of these terms and the ability to export the URLs returned. It should also have the ability to get urls from the database based on a count, all or by keyword.

Duplicates DB

The duplicates db should contain all applicable urls scraped to include the timeout urls. while the scraped url db itself will keep the entire url the Duplicates db should only keep the information

between http:// and the first occurrence of a forward slash "/".

Example:

Scraped URL= [url removed, login to view]

Saved to duplicates db = [url removed, login to view]

Save to Scraped url db = [url removed, login to view]

CONCLUSION:

If C/C++ is also used, I will receive the source code for it as well. You will not be able to sell this script in any format, nor parts of it, to any other party. I am not sure what this will cost, so I am keeping the price open, but I do have a budget. I want this scraper to be started on as soon as possible, and completed in a timely manner. If you have any questions, comments, or suggestions, please feel free to PM me. Thank you.

Kemahiran: PHP

Lihat lebih lanjut: data base multithreaded search engine, php search engine scraper, scraper search engine php, php multi thread search, multi threaded php script, scraper multi engine search php, adsense scraper, php google blogsearch url scraper, search engine scraper script, search engine scraper php, search engine scraper, you proxy google, yahoo search engine, what to include in a resume, what should be on a resume, what should a resume include, what program do i use to open a php file, what is in a good resume, what are the parts of a resume, web testing resume, web search remove, web search engines list, web search engine proxy, web proxy free, user testing sites

Tentang Majikan:
( 32 ulasan ) Doral, Costa Rica

ID Projek: #279277

19 pekerja bebas membida secara purata $1082 untuk pekerjaan ini

cssaglobal1

Hello, Please see the PMB. Regards, Bhupendra

$1400 USD dalam 45 hari
(49 Ulasan)
7.9
zeke

Dear Sir! I am professional developer, have a lot of experience with UNIX/Linux Network Programming. This script could be written entirely in C programming language, you do not have php installed on your linux server Lagi

$1000 USD dalam 10 hari
(141 Ulasan)
6.7
SoftSandila

ready to start. thanx

$899 USD dalam 20 hari
(43 Ulasan)
6.2
pgcoding

pl check pmb.

$1200 USD dalam 30 hari
(26 Ulasan)
6.2
Shattenjagger

Let's do it!

$1100 USD dalam 5 hari
(27 Ulasan)
5.6
rolanro

Have a nice day, Your project requires a dedicated person how work before whit this kind of project, I develop a blogger generator and I know how to work whit elite proxies, regards.

$1000 USD dalam 10 hari
(24 Ulasan)
4.9
semosoftware

We have gone through the information provided by you and we understand your requirement very well. We have excellent expertise in PHP5 , MYSQL, JOOMLA, HTML, CSS, AJAX, ASP.NET, VB.NET. We have worked on variety of Lagi

$1400 USD dalam 45 hari
(3 Ulasan)
4.8
ridhisolutions

Hi we have gone through your existing site. I can do according to your requirment . I will show a professional design with finctionality We are providing services for Web Templates, Logo & Banner Designs, and Web Devel Lagi

$1100 USD dalam 35 hari
(1 Ulasan)
3.6
CDNProjects

Please check PM.

$1200 USD dalam 15 hari
(2 Ulasan)
3.5
sinfotechindia

Sir Look at your PMB

$1305 USD dalam 40 hari
(0 Ulasan)
0.0
jagtapinfotech

Dear Sir/Madam, We are INDIA based leading ISO 9001:2000 Certified Web Development Company, located at Baroda, Gujarat. Our development teams who are able to provide skills and expertise in almost every area of web de Lagi

$1300 USD dalam 15 hari
(0 Ulasan)
0.0
amoltaskar

Best web designing, hosting & software development available here

$750 USD dalam 15 hari
(0 Ulasan)
0.0
namtv42

I can handle this Project. Contact me now.

$1100 USD dalam 20 hari
(0 Ulasan)
0.0
raidarmax

Hi, We are an outsourcing firm with a panel of talented designers and coders. We have specialized in web development and internet marketing since 2003 and we have proven work delivery policies. Please see PMB.

$1000 USD dalam 10 hari
(0 Ulasan)
0.0
Samarnarendra

I am a software engineer in reputed us mnc, i able to deliver u qualityful product on time. So u should provide this bid to me because i am fullfilling all ur requirements... Thanks

$1000 USD dalam 21 hari
(0 Ulasan)
0.0
solutionwizards

gifdear sir, we are very exprence in that tipe of [url removed, login to view] of our team member has a minimum of three years of experience in their related [url removed, login to view] vision is to create a well-rounded team to deliver quality produc Lagi

$900 USD dalam 14 hari
(0 Ulasan)
0.0
moneypro200x

Please check your pm

$1000 USD dalam 30 hari
(0 Ulasan)
0.0
raawmedia

Hello. We look forward to working with you. Thank you.

$1100 USD dalam 10 hari
(0 Ulasan)
0.0
john2496

This project sounds like a lot of fun! I've developed many apps similar to this in the past and already have most of the code developed! I have the highest coding & design standards and over 8 years working with php! T Lagi

$800 USD dalam 5 hari
(0 Ulasan)
0.0