・Applicants must have permission to work in Japan
・Applicants must be able to speak fluent ~ native level Japanese (Equivalent to JLPT N1)
☑English language required ☑Japanese language required ☑Remote ☑Flex time ☑Hybrid (On-site x Remote)
ソフトウェアエンジニア (Data Platform)
- Our Group utilizes data for business decisions and other processes in a wide variety of areas, including marketing, machine learning, and R&D. As part of our Data Platform Team, you will design, develop and operate the data infrastructure/data pipelines which support data utilization for the entire Group. You will also be responsible for work on the Group's data utilization as a whole.
- The Data Platform Team is an engineering organization which builds the systems necessary to a wide range of domains that utilize data. This work includes development of an ecosystem enabling data collection, Data Lake, DWH, use of collected data and other processes required by data utilization. The following describes the responsibilities of a software engineer on the Data Platform Team, as well as the environment and organization in which they work:
- Collecting data from microservices
o Developing batch-based data pipelines
Developing SDK for logging within microservices
o Developing streaming-based data pipelines
- Data Lake, DWH
o Constructing Data Lakes for storing collected data
Managing partial permissions and resources on DWH
- Data utilization
o Developing tools to support use of data in Data Lakes and on DWH
Developing a platform to provide stream data processing function
o Improving performance, aiming for low latency/high throughput according to application and middleware development, operation, and requirements
Identifying and solving technical issues with the system caused by engineering
o Developing tools to automate operations and/or lower costs
- Tools Used/Dev Environment
o Programming languages: Python, Java, Scala
o Infrastructure: Google Cloud Platform (Compute Engine, Kubernetes Engine, Dataflow, Pub/Sub, Cloud Storage, BigQuery, Spanner, Cloud SQL, Cloud Logging etc.)
o Distributed processing: Apache Flink / Apache Spark
o Workflow: Apache Airflow
o BI tools: Looker / Data Studio
o Configuration management tools: Terraform
o Monitoring: Stackdriver Monitoring / Datadog / PagerDuty
o CI/CD: Circle CI / Spinnaker
o Knowledge Tool: Crowi / Confluence
o Other: GitHub / Slack
o Development Method: Agile (Scrum, Kanban, etc.)
- Dev Organizations
o As part of the Data & Machine Learning Team, you will work on development alongside members with a broad range of expertise from the following organizations.
o Data & Machine Learning Platform
Machine Learning (Anti-Fraud)
o Data Management
o Machine Learning (Credit)
- Opportunity to build large-scale data infrastructure to support the massive amounts of data generated by our group, including an app with its over 20M monthly users.
- Work on development of an ecosystem to support data-driven business expansion by addressing the data utilization needs of many domains including analytics/decision-making, marketing, machine learning, and R&D.
- Take on the entire data platform creation process, from design to development to operation, to revise existing data infrastructure and develop new features which meet new data utilization needs.
- A degree in computer science or a related field, or else five or more years of practical experience in software development
- Experience designing, developing, and operating large-scale services and/or distributed systems
- At least two of the following:
o Experience developing in the cloud with AWS, GCP, etc.
o Experience developing systems utilizing container technologies such as Kubernetes
o Development experience using message queues like Cloud Pub/Sub and Apache Kafka
o Experience in data processing development using distributed processing frameworks like Apache Flink and Apache Spark
o Experience with ETL system using workflow engines like Airflow and Digdag
o Experience designing application logs for large-scale services
- Experience working at a financial institution, Fintech company, or EC company
- Ability to identify the cause of technical system issues (e.g. drop in performance) and to resolve them
- Knowledge of network protocols such as TCP/IP, HTTP, gRPC, etc.
- Experience developing and operating software using two or more of the following: Go, Java, Scala, or Python
- Developed data collection systems using Treasure Data-created OSS (such as Fluentd, Embulk)
- Experience in development using DWH like BigQuery, Redshift, or Snowflake
- Experience in development using RDBMS, including MySQL and RDS
- Experience in development or cluster operation using the Hadoop ecosystem
- Experience developing software based on a microservice architecture
- Experience publishing and contributing to OSS
- Capable of going bold and possessing a strong sense of responsibility in regards to their work
- Possesses excellent communications skills which can push organizations and projects forward
- Able to collect information independently and identify the essence of issues to solve problems through engineering
- Thinks independently, acts independently, and leads operations with All for One mentality for the sake of the team and while holding on to professional mindset
Technical Assessment Criteria
- Documentation skills
- Ability to code (efficiency, readability, maintainability)
- Design aptitude
- English: Basic (CEFR - A2) optional
- Japanese: Independent (CEFR - B2)
・ Complete health and social insurance
・ Incentive program
・ Support systems, including those that benefit employee family members
・ Language learning support
・ Translation/interpretation support
・ Two days off per week (as well as national holidays, New Year's break, etc.)
・ Paid leave, congratulatory and bereavement leave, relax days, sick leave
Please click "apply" if you are interested in the job.
We will review your profile and contact you within five business days should we find that you satisfy the requirements of the hiring company.
Further details about the company and position will be notified at a later date.
Seniority Level: Mid-Senior level
Job Function: Information Technology