Loading...
Загрузка
Назад

Senior Remote Big Data Engineer (AWS, Spark, ETL)

Заработная плата: от 5 000 рублей
Опыт работы: нет опыта
Занятость: полная занятость
Откликнуться

Описание вакансии

ClearScale (headquartered in San Francisco, California, USA) - AWS Premier Consulting Partner has been offering a full range of professional cloud computing services for over 10 years, including architecture design, DevOps automation, refactoring and cloud-native applications development, integration, migration, solving all sorts of security issues (from just a security check to preventing cyber-attacks) and 24/7 technical support using the best advanced technologies.

The list of our customers is diverse: from government companies (ClearScale is an official cloud partner of the State of California) and educational institutions (University of California, San Francisco) to well-known global brands (IBM, Samsung, GoPro, HP, Conde Nast, Carl Zeiss, etc.) The number of satisfied customers has been well over 850, some of which can be found on the company's website in the Case Studies section.

  • We were the third company to have gained a new AWS competence: Applied AI and Machine Learning Operations (MLOps). Less than 15 partners have this competency!
  • We have confirmed status in the Database Freedom program (total less than 20 companies in)
  • Proven expertise as a Managed Services Provider (total less than 16 companies in). This means that ClearScale can perform a full cycle consultancy and service: from audits, system or software development to the 24/7. You can read more about us on the company page - Managed Services.

Since the very foundation, we work 100% remotely from various cities and countries. We work on a high trust basis within the company, therefore we do not monitor work being done via tracking screen captures, webcams or keyboard typing, as many other companies do. The professional reputation of our employees is of the highest value.

Job Overview

ClearScale is looking for an AWS Data Engineer to participate in building cost-efficient scalable data lakes for the wide variety of customers, from the small startups to the large enterprises. Usually, our projects fit into one of the categories (but not limited to them):

  • Collect data from the IoT edge locations, store it in the Data Lake, orchestrate processes to ETL that data, slice it into the various data marts. Then put those data marts into the machine learning or BI pipelines
  • Optimize pre-existing Spark/Elasticsearch/Solr clusters, find the bottlenecks in the code and/or architecture, plan and execute improvements
  • Build a data deliver pipeline to ingest high volume of the real-time streams, detect anomalies, slice into the window analytics, put those results in the Elastic search system for the further dashboard consumption

Responsibilities

  • Analyze, scope and estimate tasks, identify technology stack and tools
  • Design and implement optimal architecture and migration plan
  • Develop new and re-architecture solution modules, re-design and re-factor program code
  • Specify the infrastructure and assist DevOps engineers with provisioning
  • Examine performance and advise necessary infrastructure changes
  • Communicate with client on project-related issues
  • Collaborate with in-house and external development and analytical team
  • Write ETL/ELT jobs

  • Design Data Lake House pipelines

Requirements

ClearScale expects successful candidate to have most of the following qualifications and skills (not necessary all have to be presented):

  • Hands-on experience designing efficient architectures for high-load enterprise-scale applications or ‘big data’ pipelines
  • Practical experience in implementing of micro-services architectures (using Java, Python or Scala stack)
  • Hands-on experience with message queuing, stream processing and highly scalable ‘big data’ stores
  • Advanced knowledge and experience working with SQL and noSQL databases
  • Proven experience in re-design and re-architecting of the large complex business applications
  • Strong self-management and self-organizational skills
  • Successful candidates should have experience with any of the following software/tools (not all required at the same time):

    • Java/Scala/Python - strong knowledge
    • Big data tools: Kafka, Spark, Hadoop (HDFS3, YARN2,Tez, Hive, HBase)

    • Stream-processing systems: Kinesis Streaming, Spark-Streaming, Kafka Streams, Kinesis Analytics
    • NoSQL DBs: Kassandra, DynamoDB, Mongo
    • Desired:
      • AWS cloud services: EMR, RDS, MSK, Redshift, DocumentDB, Lambda
      • Graph databases development and optimization (Neo4j, Neptune, Tinan)

      • Message queue systems: ActiveMQ, RabbitMQ, AWS SQS
      • Federated identity services (SSO): Okta, AWS Cognito
  • We are looking for a candidate with 3+ years of experience in Java/Data, Cloud or Software Engineer role, who has attained a degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field

We offer

# 1 Fair wage

  • High hourly rate in USD
  • Full-time, 40 hours per week contract. It’s your main and permanent place of work, not a freelance job
  • Payments every 2 weeks
  • Annual Rate Review: received client's appreciation, obtained an AWS certificate, saved the world?The reward is based on facts, not on a personal attitude
  • Reference program: many engineers joined the team by the recommendation from their friends and former colleagues and we have a reward for this.

# 2 Independence

  • from the office location or office conditions. The only requirement - fast and reliable internet. At the same time, our engineers do not work evenings or weekends: we're focused at your local timezone
  • from any annoyance or big team buzz - we are the community of professional engineers, valuing each others' time and effort
  • from corruption and bureaucracy - we operate in an honest and competitive environment and we are one of the AWS's top 10 key partners.

# 3 Professional Development

  • Work with innovative Silicon Valley companies and traditional American companies at the cutting edge of digital transformation
  • We work with the newest technologies in AWS cloud and open-source tools like Jira, Confluence, Lucidchart, Slack etc
  • Constant practice in written and spoken English
  • The team willing to share its experience
  • Paid AWS certifications: we provide training material, paid time off and examination itself
  • Horizontal and vertical career growth - We keep growing and people keep growing with us
  • A nice bonus if you accept our offer early
Откликнуться
Время карьеры
Найти работу мечты
Время карьеры
Зарегистрируйся сейчас и получай информацию об актуальных вакансиях, стажировках, событиях
Регистрация