We are representing an interesting Data Engineering position with a reputable hedge fund firm in Midtown Manhattan / Greenwich, CT.
- Build and maintain robust data pipelines that ingest TBs of data.
- Build new Airflow DAGs that manage different operators or individual tasks.
- Create logical and physical data models for big data to be stored in cloud data warehouses.
- Optimize Spark and SQL queries.
- Create new SQL tables for reports and dashboards.
- Work with data warehouses: Snowflake, AWS Redshift, and other cloud data warehouses.
- Work with data analysts on implementing dashboards and performing analysis.
- Use Databricks Spark and data visualization tools to analyze business problems.
- Generate reports and dashboards for business insights.
- Write analytical queries to extract insights for large datasets (billions of rows, multi-TB in size).
- 3+ years of professional working experience
- A degree or advanced degree in Computer Science, Engineering, Physics, Mathematics, Statistics, or Machine Learning, with a record of academic success.
- Excellent development experience in Python, Apache Spark, and SQL.
- Extensive experience with large-scale data processing solutions.
- Extensive working knowledge of Apache Spark, Databricks and pandas.
- Extensive working experience with AWS ecosystem.
- Excellent computer science fundamentals and problem-solving skills.
- Strong ability to work in Linux environment.
- Experience in the fields of data warehousing, business intelligence and big data related technologies.
- Experience in creating logical and physical data models.
- Experience with one or more cloud MPP data warehouses including: Snowflake, AWS Redshift, Azure Data Warehouse (Synapse), or Google BigQuery.
Note: Qualified candidates will be contacted within 2 business days of application. If an applicant does not meet the above criteria, we will keep your resume on file for future opportunities and may contact you for further discussion.