I have 10+ years industry experience in Web crawling, Web Scraping, Data extraction, Python, AWS, Databricks, Data Analysis, Data engineering, Excel, MySQL (RDBMS), Mongo DB (No Sql) database, snowflake, redshift etc.
Good experience in building data pipeline and data streaming and provide solution architect.
I have also experience in Pyspark, Big data, AWS - s3, ec2, EMR, Athena, Glue etc.
I have built crawler for approx. 400+ website like Amazon, ebay, flipkart, walmart, bestbuy, shopee, snapdeal, and most of OEM website etc.
I have done advance level of crawling using “API” and “POST” method also.
I am able to scrape any difficult site. Used Scrapy, Beautiful Soup, Requests, Lxml, Selenium, Splash libraries for crawling data.
Apart form this I have basic knowledge of java, aws ec2 and s3, sql server, oracle, xml, json, data science - ML etc.