
Completed
Posted
Paid on delivery
I need a small sales data pipeline (proof of concept) that runs directly on Windows 10/11. NO Docker. Delivery in 4 days, $100-$300 fixed, 2 milestones. DATA FLOW C++ simulator -> MQTT (Mosquitto) -> C# .NET collector -> SQLite -> Python loader -> Hadoop HDFS -> sales reports WHAT TO BUILD 1) C++ sales simulator - Simulates 10 stores and 50 products, sends each sale as JSON to MQTT (QoS 1) - Each sale: message_id, store_id, product_id, quantity, price, timestamp (UTC) - Settings: number of messages and speed. Stops after N messages (default 50,000) - Sends a few invalid messages on purpose (for testing) 2) C# .NET 8 collector - Subscribes to MQTT (e.g. MQTTnet) with a persistent session, so no data is lost when it restarts - Checks every message: good rows -> "sales" table, bad rows -> "rejected" table - No duplicates (unique message_id) 3) SQLite database - Tables: sales, rejected, load_log 4) Python loader -> Hadoop HDFS - Copies new SQLite rows to HDFS as CSV, in folders by date: /sales/dt=YYYY-MM-DD/ (UTC) - Safe to run twice: no duplicates, no missing rows - Lock, so two loaders cannot run at the same time 5) Hadoop sales reports - Hadoop Streaming jobs with Python mapper/reducer. Output 5 CSV files: 1. Total sales per day 2. Sales per store per day 3. Top 10 products by revenue 4. Sales by hour (UTC) 5. Average basket value per store 6) Verify script - One command that checks: sent = SQLite = HDFS = report totals. Prints OK or FAIL. TECHNICAL REQUIREMENTS - NO Docker, NO containers of any kind - On Windows: Mosquitto (Windows installer), .NET 8, Python 3.11+, C++ with CMake (+ vcpkg) - Hadoop 3.3.x single node in WSL2 Ubuntu (Hadoop does not run well on Windows directly; WSL2 is OK) - All ports bound to [login to view URL] only (MQTT 1883, Hadoop web UI 9870) - All times in UTC - All settings in one config file. No hard-coded paths or passwords. - Setup scripts (PowerShell + bash) and a clear step-by-step README - Unit tests for the collector and the loader MILESTONES Milestone 1 (40%) - Day 2: C++ simulator + Mosquitto + C# collector + SQLite working. 50,000 messages sent = 50,000 rows stored (sales + rejected), 0 duplicates. Milestone 2 (60%) - Day 4: Hadoop setup, Python loader, 5 reports, verify script, final README, short screen recording of a full run. ACCEPTANCE - I will run everything on my own Windows machine before I release payment - Fresh install works by following only the README, without Docker - Collector stopped for 60 seconds, then started again: 0 messages lost - Loader run twice: 0 duplicates in HDFS - Verify script prints OK, and the 5 report CSVs are correct DELIVERY - One ZIP per milestone: full source code, build and setup scripts, README, sample config - Full ownership of the code goes to me HOW TO BID - Start your bid with the word PIPELINE so I know you read this - Tell me your experience with MQTT and Hadoop (links to similar work if possible) - Confirm you can deliver in 4 days without Docker - Your fixed price
Project ID: 40774620
95 proposals
Remote project
Active 6 days ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

PIPELINE "WINDOWS SALES PIPELINE WITH VERIFIED DATA COUNTS" - build the full flow from simulated sales through MQTT, SQLite, HDFS, and five reports, without Docker or containers. I'd keep message IDs unique, validate incoming JSON before storing it, and track HDFS exports with a transactional load log so reruns can resume safely without duplicating rows. The verification script should compare counts and totals across stages, including deliberately rejected messages. Hadoop will run in WSL2 Ubuntu, with local services bound to [login to view URL] as requested. Can you confirm that your Windows machine already supports WSL2 and has enough memory and disk space for a single-node Hadoop setup? Looking forward to work with you.
$250 USD in 4 days
0.0
0.0
95 freelancers are bidding on average $167 USD for this job

As a Certified Python Developer with over 5 years of experience in Full Stack Web Technologies and Software Development, I can assure you I have the skills needed to complete your sales data pipeline project with utmost efficiency. My vast experience in Python, C++, C#, SQLite, Hadoop, and MQTT makes me well-suited for the tasks required. I have successfully worked on similar projects in the past that required sophisticated data management pipelines and accurate reporting. Finally, allowing you full ownership of the code upon completion is standard practice that I endorse wholeheartedly. With my dedication and proven track record of timely results, you can trust that your project will be delivered to the highest standards within your specified timeframe and budget. Let's build this advanced sales data pipeline together!
$140 USD in 7 days
8.0
8.0

Hi There, Hope you are well. I am well-versed in creating a Windows-based sales data pipeline without Docker. Proficient in C++, C#, Python, and Hadoop. Experienced in MQTT and Hadoop with successful project deliveries. Rest assured, I can complete this proof of concept within the specified timeline. Let's discuss further details. Thanks in advance for the opportunity to working with you. Best Regards, Serazum
$55 USD in 1 day
7.6
7.6

PIPELINE Hi, I am a software engineer with over 16 years of experience. I have extensive experience with MQTT messaging and Hadoop data processing, and I can deliver this proof of concept in 4 days without Docker or containers. My fixed price is $250. I will focus on reliable message capture, clear handling of rejected records and safe incremental loads into HDFS. I will test the 50,000-message run, the collector's 60-second outage and repeated loader runs, with verification that accounts for rejected messages correctly. I will deliver the simulator, Mosquitto, .NET collector and SQLite database by Day 2 (40%), followed by the WSL2 Hadoop setup, loader, five reports and verification script by Day 4 (60%). I will include collector and loader unit tests, one shared configuration, localhost-only services, setup scripts, a clear README and a full-run recording. Each milestone will include a source ZIP, with full code ownership transferred to you. For average basket value, should each valid sale message count as one basket, since the specified fields do not include a basket ID? Similar projects cannot be shared publicly, but I can show examples privately. Please contact me to discuss details.
$250 USD in 4 days
7.6
7.6

Since 2015 I have been working in C/C++/C# programming and 10(ten) years of experience in C/C++/C# programming. Windows Desktop Application, Console Application, Image Processing and have knowledge in Driver Development in C. Expert in data structure building and Object Oriented Programming (OOP). Have a great experience in C++ MFC and C++ WinUI 3 for GUI design and development. Also expert in C/C++ GPU CUDA programming. If you want a good delivery of the project, then send me a message, please.
$150 USD in 4 days
7.4
7.4

PIPELINE — Hello, I’m confident I can build this Windows-only pipeline with clear validation at every stage. Is Hadoop 3.3.x already available in WSL2, or should setup be included from scratch? Can we use SQLite checkpoints and HDFS partition files to make retries safe without duplicates? I’ll deliver the C++/MQTT simulator, .NET 8 collector, and SQLite tests by Day 2, then complete the Python loader, five Hadoop Streaming reports, and end-to-end verification by Day 4. I’ll include PowerShell/bash setup scripts, sample config, and README. Fixed price: €250. Best, Dev S.
$300 USD in 3 days
6.6
6.6

PIPELINE With a decade of experience as a full-stack developer, I have an in-depth understanding and hands-on experience with all the technologies you've mentioned, including MQTT and Hadoop. Aside from being well-versed in .NET and C programming needed for the collector, my strong command of Python will be handy for data loading and generating sales reports. The intricate processes involved in your project don't falter me, but rather entice me. I relish challenges and fine-tuning pipelines to ensure that data flows seamlessly from start to finish. To guarantee the absence of duplications and data loss, I will create reliable checks at every stage. All the unit tests required for the collector and loader will be extensively sorted to verify synchronization among SQLite, HDFS, and reports. What distinguishes me is not simply my technical skills but my unwavering commitment to producing exemplary results within tight timeframes. As such, your project's timeline poses no concern; I'm fully confident that I can deliver everything you need within the stipulated 4-day timeframe without resorting to containers like Docker. Throughout the project lifecycle, you'll find that I'm proactive in providing timely updates including setup scripts, comprehensive READMEs, sample configuration files, and a clear screen recording of a full run for Milestone 2 delivery.
$300 USD in 7 days
6.2
6.2

PIPELINE =====$ Hi, Aleksandar here from Serbia $===== Your pipeline needs to prove that every sale survives the full journey from MQTT to HDFS and reporting, especially across collector restarts and repeated loader runs. I noticed you require "0 messages lost" and "0 duplicates in HDFS", so I'd focus on persistent MQTT sessions, SQLite deduplication, and checkpointed, idempotent loading with reconciliation checks. I'd build the C++ simulator, C# collector, SQLite storage, Python loader, Hadoop Streaming reports, and Windows setup scripts. I've built similar systems. Should the 50,000-message acceptance test include invalid messages in the total, or should all 50,000 be valid sales? Let's connect over a chat so we can discuss this in more detail.
$150 USD in 1 day
6.0
6.0

PIPELINE I can deliver this in 4 days exactly as specified, with no Docker or containers: native Mosquitto/C++/.NET/Python on Windows 10/11 and Hadoop 3.3.x inside WSL2 Ubuntu. I’ve worked with message-driven ingestion and Python/SQL data pipelines where validation, duplicate prevention, restart recovery, and idempotent loading are critical. For this POC I’d use MQTT QoS1 with a persistent client session, SQLite unique `message_id`, and a loader that writes each batch to a temporary HDFS path, verifies it, then atomically finalizes it before updating `load_log`. The 60-second collector outage test is important, so Mosquitto persistence/session configuration will be part of the setup rather than relying only on collector reconnect logic. I’d also keep every path/rate/port/batch setting in one config and bind MQTT/Hadoop interfaces to localhost as requested. Day 2: simulator + MQTT + .NET collector + SQLite and 50k reconciliation. Day 4: HDFS loader, 5 Hadoop Streaming reports, verify command, tests, README and recording. I confirm delivery in 4 days without Docker.
$30 USD in 7 days
5.6
5.6

PIPELINE Hi Redzinam, I will deliver your Windows POC pipeline featuring the C++ simulator, C# collector, SQLite, Python loader, WSL2 Hadoop HDFS, and 5 reports. I confirm complete 4-day delivery for $200 with zero Docker. I can start immediately.
$140 USD in 3 days
5.3
5.3

Hello, As the founder of A4Logic and a seasoned professional with over 15+ years of experience, I can confidently say that I am the perfect fit for your Sales Data Pipeline project. My expertise spans across multiple technologies including C++, C#, .NET, SQLite, Python, and Hadoop - which aligns perfectly with the tech stack you require. Being an expert in Windows OS and having hands-on experience with Mosquitto, MQTTnet, WSL2 Ubuntu, and Hadoop's operation on Windows through WSL2, I can elegantly navigate through the intricacies of your project without relying on any Docker or container-based dependencies. To be more specific about my skillset; I have profound knowledge of C++ and have developed multiple simulators with faster message processing while still maintaining great quality of MQTT messaging like in your project. Furthermore, my expertise in C# Development and extensive experience working with various database technologies like SQLite instills in me the ability to build a robust collector setup complying with your needs to the core. I am well-versed in Python too, which is required for dealing with HDFS operations and creating sales reports as per your specifications. To ensure a smooth development lifecycle and eliminate any chances of redundancy or quality compromise, I will subject my work to strict unit testing. Additionally, having worked on similar projects in the past (you may check my GitHub link), I am well aware Thanks!
$180 USD in 3 days
5.4
5.4

==== Hi - Truong here ==== "WINDOWS SALES DATA PIPELINE WITH MQTT AND HADOOP" - you need 50,000 messages processed reliably, with zero duplicates and a repeatable setup without Docker. I'll build the C++ simulator, Mosquitto QoS 1 pipeline, .NET 8 collector with SQLite validation, and Python HDFS loader. Unique message IDs and checkpoint tracking will prevent duplicate records and repeated exports. Hadoop 3.3.x will run in WSL2, with five sales reports, automated verification, setup scripts, unit tests, and README documentation. My fixed bid is $230 for 4 days, split into 40% and 60% milestones. Will your test machine have WSL2 Ubuntu installed, or should the setup scripts handle that too? Looking forward to work with you.
$150 USD in 10 days
5.2
5.2

PIPELINE With extensive experience in both MQTT and Hadoop, I am confident that I can deliver an exceptional solution for your sales data pipeline. At Infinity Sols, we have successfully implemented numerous complex systems involving MQTT for real-time message handling and stream processing as well as Hadoop-based data processing for large scale analytics. One notable project was developing a similar solution for a retail client that processed millions of sales transactions daily. Being well-versed in C++ and C# for high-performance computing and SQLite for efficient database management, I can expertly develop the C++ sales simulator, the C# .NET collector, and the SQLite database components of your project. Additionally, my strong command over Python will enable me to build an error-free loader and Hadoop streaming jobs for generating the necessary sales reports. Our price estimate for this project is $750. We are committed to delivering high-quality work in 4 days without using Docker. You can rely on our ownership mentality; at the end of the project you will have complete access to all source code, build and setup scripts, along with a clear README file to ensure smooth operation going forward. Let's make your data pipeline flow seamlessly. Hire us now!
$250 USD in 5 days
5.6
5.6

PIPELINE — Hi, I can build this Windows-native sales pipeline using C++, MQTT, .NET 8, SQLite, Python, and Hadoop on WSL2—without Docker. I’ll prioritize QoS 1 delivery, duplicate-safe ingestion, restart recovery, idempotent HDFS loading, five validated reports, and end-to-end verification. I can target your 4-day, two-milestone schedule for $250 fixed, with setup scripts, unit tests, documentation, and milestone ZIPs. I’ll confirm the Hadoop/MQTT environment early to reduce integration risks. Best regards, Shakila Naz
$250 USD in 3 days
5.3
5.3

Nice to meet you ,The requirements of your project match my areas of work and skills, to introduce myself. My name is Anthony Muñoz and i am the lead engineer for DS Pro IT agency. I have worked for over 10 years as a Full-Stack and software development engineer and have successfully done multiple jobs. It will be a pleasure to work together to make your project. Feel free to discuss about the project with me, greetings.
$170 USD in 7 days
4.9
4.9

Hi, Fernando here, from Mexico. Building your fortify Cash App ++ security with React Native and Node.js is doable; what matters most is an app that passes store review and works on the phones your users own. I'm a senior mobile app developer with 12+ years of experience working with Node.js and Mobile App Development. With React Native, the real challenge is device differences and store review. A common issue: ignoring the app store rules until submission, then losing time to rejections. My plan: first, map the screens, user flows and store requirements. Then, build the core flows in React Native, testing on real devices. Finally, test on older phones, then prepare the store listings and submit. I've done mobile app work in Node.js, so I check early for store rejections and older Android phones. A few questions to better understand the scope: 1. Do you have Apple and Google developer accounts? Company verification can be slow, so I would start it now. 2. One cross-platform codebase for iOS and Android? I'd recommend it, every fix lands on both stores at once. 3. Any users in California? CCPA needs a clear privacy notice and a data deletion path, simple to add now. I can start right away. Happy to go over the details here in chat and suggest the best approach. Looking forward to hearing from you, Fernando
$100 USD in 1 day
4.6
4.6

A lightweight pipeline reading sales data into SQLite, running pure Python/C# on Windows with no Docker, is a clean fit. I would pull from your source via MQTT or a scheduled pull, stage it, then load into SQLite with a simple C# or Python runner, no containers anywhere. Budget and days here are starting points from the post. Once we confirm the actual data source and volume, I will firm both up. Can start today, working PoC in 3 to 4 days. Want me to send a quick scope doc?
$150 USD in 6 days
4.5
4.5

Hi there, With over 4.9 stars and a strong track record of delivering on time, I'm M Mobasher, your go-to developer for the "Sales Data Pipeline Development" project. Having deep experience in .NET and C++, this project strikes an exciting chord with me as it incorporates both of my core strengths, Python and SQLite as well. With reference to your specific technical requirements, I assure you a Docker-free pipeline designed around efficient MQTT utilization using MQTTnet and Mosquitto. My proficiency extends to working with data processing tools such as SQLite, Python's Hadoop engine, and crafting reports derived from raw data using Hadoop Streaming jobs. My previous engagements include similar works surrounding MQTT and Hadoop tasks leveraging Kafka and Spark for accelerated data behandling in respect to WSL2 Ubuntu-based platforms, so Windows-to-Ubuntu OS transfer won't be an issue. While I understand that Docker usage is excluded in this endeavour, I believe we can surpass that limitation with my proven record of flexibility and adaptability. Best regards, Mobasher Reza
$140 USD in 3 days
4.5
4.5

Hello, PIPELINE: This proof of concept requires a dependable Windows-first data flow from C++ sales generation through MQTT, .NET, SQLite, HDFS, and verified Hadoop reports, without Docker or containers. I can deliver the requested two milestones within four days. I will implement QoS 1 MQTT messaging, persistent .NET 8 collection, validation and deduplication, SQLite logging, and configurable UTC-based processing. The Python loader will use locking and idempotent load tracking, while Hadoop Streaming jobs will generate the five required CSV reports. PowerShell, bash, setup scripts, unit tests, localhost-only bindings, and a clear README will be included. Similar work includes MQTT device and event pipelines, reliable message ingestion, SQLite persistence, Python ETL workflows, and Hadoop-compatible batch reporting. I will structure the code so the simulator, collector, loader, reports, and verification command can be tested independently and run from a fresh Windows and WSL2 setup. Milestone 1 will be ready by day 2, with the complete pipeline, reports, verification script, documentation, and recording delivered by day 4. The final ZIPs will include source code, configuration, tests, and setup instructions. Best regards, Bohdan
$250 USD in 4 days
4.3
4.3

PIPELINE Hello, The main challenge is ensuring reliable delivery across MQTT, SQLite, and Hadoop without losing or duplicating sales records, especially during collector restarts and repeated loader executions. I'd implement the pipeline using C++, Mosquitto, C# .NET 8, SQLite, Python, and Hadoop 3.3.x through WSL2, entirely without Docker. 1) Simulator, MQTT and Collector I'll build the configurable C++ simulator for 10 stores and 50 products, publishing JSON messages with QoS 1. The C# collector will use persistent MQTT sessions, broker persistence, message validation, and unique IDs to prevent duplicates. Valid transactions will enter the sales table, while invalid messages will be recorded separately. 2) Hadoop and Reporting I'll implement an idempotent Python loader with locking, date-partitioned HDFS output, and checkpoint tracking. Hadoop Streaming will generate all five requested CSV reports using Python mappers and reducers. 3) Verification and Delivery I'll provide automated verification, collector and loader unit tests, PowerShell/bash setup scripts, and a reproducible README. I'll test 50,000 messages, collector restart recovery, repeated loading, and reconciliation between SQLite, HDFS, and reports. I can work within your four-day schedule, subject to environment readiness. One clarification: Should each sale message represent one basket, or can you provide a basket ID for accurate average basket calculations? Best regards, nicolas
$140 USD in 7 days
4.0
4.0

Hi, I notice the data pipeline needs to handle 50,000 messages without duplicates, which is a challenge. I'd approach this by setting up a persistent MQTT session in the C# collector to ensure no data is lost. My focus will be on getting the C++ simulator and C# collector working together seamlessly. I've seen issues with data consistency when loading data into HDFS, so I'll make sure the Python loader is designed to handle duplicates and missing rows safely. The client can expect a fully functional sales data pipeline with accurate reports and a verify script that checks for data integrity, all delivered without using Docker and with a clear step-by-step README.
$140 USD in 5 days
4.2
4.2

Kumasi, Ghana
Payment method verified
Member since Dec 23, 2010
$30-250 USD
$30-250 USD
$10-30 USD
$30-250 USD
$250-750 USD
$10-30 USD
$10-30 USD
£10-20 GBP
$250-750 USD
$30-250 USD
€30-250 EUR
$30-250 CAD
₹12500-37500 INR
₹750-1250 INR / hour
$250-750 USD
€8-30 EUR
$250-750 USD
₹12500-37500 INR
₹12500-37500 INR
$250-750 USD
₹12500-37500 INR
$250-750 USD
$15-25 USD / hour
₹600-1500 INR
₹12500-37500 INR