Implementing Spark in Airflow

$30-250 CAD

Avslutat

Publicerad

elva månader sedan

$30-250 CAD

Betalning vid leverans

I am looking for someone who is familiar with both Spark and Airflow. The main goal of implementing Spark in Airflow for my project is to improve scheduling and automation. Tools: Airflow, Docker, Spark. Task: Using Airflow dags, build a pipeline based on distributed computation offered by Spark, but not Pyspark, and keep a log of the pipeline execution and Dockerize it. 1. Download the ETF and stock datasets from the primary dataset available at [login to view URL] 2. Set up a data structure to retain all data from ETFs and stocks in the following columns. Symbol: string Security Name: string Date: string (YYYY-MM-DD) Open: float High: float Low: float Close: float Adj Close: float Volume: int Note: Do not change Adj Close to Adj_Close 3.1. Convert the resulting dataset into a structured format (Parquet). 3.2. Calculate the moving average of the trading volume (Volume) of 30 days per each stock and ETF, and retain it in a newly added column vol_moving_avg. 3.3. Similarly, calculate the rolling median and retain it in a newly added column adj_close_rolling_med. 3.4. Retain the resulting dataset into the same format (Parquet). 3.5 write a unit testing.

Python

Apache Spark

Docker

Project ID: 36777701

Om projektet

14 anbud

Distansprojekt

Senaste aktivitet tio månader sedan

Ute efter att tjäna lite pengar?

E-postadress

Fördelar med att lägga anbud hos Freelancer

Ange budget och tidsram

Få betalt för ditt arbete

Beskriv ditt förslag

Det är gratis att registrera sig och att lägga anbud på uppdrag

14 frilansar lägger i genomsnitt anbud på $173 CAD för detta uppdrag

@anayapallavi

I am familiar with spark and airflow and i can do this project. Please initiate a message for further discussion.

$180 CAD Om 5 dagar

5,0

(12 omdömen)

5,2

@omnamasshivay

hello i am very familiar with both Spark and Airflow. so i can do that job so please contact me on chat for further discussion

$300 CAD Om 7 dagar

5,0

(9 omdömen)

4,8

@rashidamjad

Hi there, How are you? I can work with your project requirements mentioned. I can definitely pull it off once provided with detailed file of your requirements. Some points are needed to be discussed before starting so lets talk. I have good experience in Implementing Spark in Airflow including Apache Spark, Docker and Python. It would be great to have you on board and start it as quickly as possible. So it would be my pleasure to build long term relationship with you. All my skills are related to this particular project. Lets have a detailed chat about this project. Waiting from your end now. Thanks. Rashid Amjad.

$250 CAD Om 8 dagar

5,0

(5 omdömen)

4,5

@ashrafsbai6655

Hi there, I checked your requirements and guarantee you that i have relevant experience in Spark and Airflow it's gonna be done within the next few hours . Let's contact via chat so that I can start work immediately

$200 CAD Om 1 dag

5,0

(13 omdömen)

4,1

@Demenntor

Dear Client, Being a 'Top Rated Freelancer' with extensive experience in both Spark and Airflow, I am thrilled to have the opportunity to bid on your project. I am confident in my ability to deliver exceptional results for your project's scheduling and automation needs. I have a proven track record of delivering high-quality projects within deadlines, and I am confident that I can exceed your expectations. My expertise in Spark and Airflow, combined with my commitment to excellence, make me an ideal candidate for this project. I look forward to the opportunity to work with you and discuss any further details to ensure the success of your project. Regards, Lalit

$210 CAD Om 7 dagar

5,0

(16 omdömen)

3,7

@ITMed

Dear client, I am thrilled about the opportunity to contribute to this exciting project. Over the past three years, I have honed my expertise in Apache Spark as well as Apache Airflow. I am confident that my skills and dedication can alleviate the burden of this project and work harmoniously with you to ensure successful outcomes. If you are seeking a motivated and reliable professional to collaborate with, someone who will not only shoulder the responsibilities but also prioritize the project's success, I would love to discuss further. Let's have a chat ! Ps. Contact me to show you my projects where I used Airflow Or spark, or both combined Ps2. I'm able to finish your task within few hours, contact me to start working right away. Best regards, Mohammed.

$190 CAD Om 2 dagar

5,0

(6 omdömen)

3,3

@PowerRapto

Hi, Hemsworth H. I've gone through the job posting. It seems that you're looking for a software engineer who have rich experiences on Apache Spark, Python and Docker. I've worked with another client in this space, I think you might find it very interesting to have chat. Regards, Peter

$155 CAD Om 3 dagar

0,0

(0 omdömen)

0,0

@phamanh8072

Tôi muốn nhận công việc này để kiếm thêm thu nhập và giúp bạn giải quyết các vấn đề của dự án. Vơi kinh nghiệm nhiều năm trong lĩnh vực liên quan đến dự án của bạn tôi có thể giúp bạn giải quyết các vấn đề của dự án

$140 CAD Om 7 dagar

0,0

(0 omdömen)

0,0

@minhhong1191995

Tôi đa từng build nhiều dự án với Spark và tôi đã tham gia các dự án liên quan BigData. Hy vọng toi cùng bạn sẽ giải quyết được các vấn đề mà bạn gặp phải

$140 CAD Om 7 dagar

0,0

(0 omdömen)

0,0

@techline113

I understand that you are looking for someone who is familiar with both Spark and Airflow. As a Full Stack Developer with 15 years of experience in building full stack applications, I am confident that I am the best fit for this project. I have extensive knowledge in Python which can help us to implement Spark in Airflow for improved scheduling and automation. My experience includes working with Airflow dags, setting up a data structure to retain all data from ETFs and stocks in the following columns (Symbol, Security Name, Date, Open, High, Low, Close), converting the resulting dataset into a structured format (Parquet), calculating the moving average of the trading volume (Volume) of 30 days per each stock and ETF and retaining it in a newly added column vol_moving_avg), as well as calculating the rolling median and retaining it in a newly added column adj_close_rolling_med). Additionally, I have written unit testing for our project which will ensure quality control during development.

$140 CAD Om 7 dagar