Compare 1.4 million sites against 1 million authoritative sites

I have a list of about 1.4 million URLs (Sites A) that need to be checked against another 1 millions sites (Sites B). Sites B are the authoritative sites that are from Alexa’s top 1 million sites.

I would like to have a program or script made that each of Sites A (1.4 million URLs) checked if it is listed in Sites B in a reasonable time (say, in less than 1 hour, but faster the better). The program can be made with: C, Ruby, Perl, or Python etc.

When the URL of one site in Sites A is listed in Sites B, I will need a flag be put up, and it must be saved as a text file along with the URL, one site per line. When the site in Sites A is not listed in Sites B, it will have a flag put down and it also be saved as a text file along with the URL.

Kemahiran: Algoritma, Pengaturcaraan C, Perl, Python, Kejuruteraan Perisian

Lihat lagi: top ruby, sites like alexa, hour time frame, python, perl, compare betting api sites, translate faster better german, make ajax sites faster, java code online exam automation hour time , hour time autoit, adding hour time net, hour time, hour time counter website

Tentang Majikan:
( 20 ulasan ) Hiratsuka-shi, Japan

ID Projek: #10107416

Dianugerahkan kepada:


Hi. I would like to work on your project. Im ready to create code without milestone and show you demo result to confirm my skills. Also before urls will be processed they need to be normalized.

$30 USD dalam 2 hari
(7 Ulasan)

4 pekerja bebas membida secara purata $40 untuk pekerjaan ini


Hello Sir... I have a very good experience in C, Java, Ruby and Python,. I can do this within the mentioned constraints well (time, output and size). Please contact me for more details when possible. I look forward Lagi

$50 USD dalam sehari
(53 Ulasan)

Hi Sir/Madam, I'm expert in Python programming and I can help You with this script. Can You send me examples of those 2 files (I suppose they are too big to be send whole). Best regards, Fejs.

$30 USD dalam 2 hari
(75 Ulasan)

Hi! My name is Tomasz Kustra, and I am from Poland. I am interested in this project. I can do such thing for you in PERL. I am experienced programmer, you can see ma resume on VWorker (now part of FREELANCER. Lagi

$50 USD dalam 3 hari
(44 Ulasan)