The New York City Department of Transportation’s (NYC DOT) provides safe, efficient, and environmentally responsible movement of pedestrians, goods, and vehicular traffic on the streets, highways, bridges, and waterways of the City's transportation network. NYC DOT rehabilitates and maintains the City's infrastructure, including bridges, tunnels, streets, sidewalks, and highways.

The NYC DOT IT & Telecom division is an award-winning team leading business transformation through technology innovation to enhance the agency’s ability in fulfilling its mission. IT & Telecom acts as a strategic partner with all business units to promote technology initiatives by delivering quality service, and secure solutions that provide new and improved capabilities for the Agency. We work hard to embrace diversity and inclusion and encourage everyone in the division to bring their authentic selves to work every day.

We offer our full-time employee’s competitive salaries, excellent benefit options and perks that include:

- Pension and Retirement Plans- Upon retirement, qualified members of the City’s generous pension program receive a guaranteed salary and health benefits for the rest of their lives. There are also 401K and 457 plans available to supplement your retirement needs.

- Health Coverage- We offer health coverage to meet your needs and your family.

- Dental and Vision Coverage- We offer a wide variety of excellent civil service title-based union dental and vision coverage to meet the needs of you and your family.

- Paid Time Off- We offer paid vacation and sick time off which accrues over time based on your years of service.

- Professional Development - We offer numerous training programs, leadership development opportunities, and career coaching services, while encouraging employees to attend approved off and on-site trainings and seminars.

- Additional Perks - Our employees are eligible for discounts on top theme parks, hotels, shows, events, movies and more.

The IT & Telecom division is seeking a Senior Data Engineer to be a member of the Enterprise Data Science and Engineering Unit (EDSE). EDSE provides the Agency with custom architected data storage and processing solutions that can handle very large datasets, or “Big Data”. In addition to managing Big Data operations, we work alongside Agency analysts and data scientists to help the Agency’s leadership and planners make data-driven decisions for critical projects and strategic planning initiatives. The candidate will work with the Director to support our Big Data analytics platform and develop new data systems and services.

Major Responsibilities

- Data platform and Linux administration/maintenance.
- ETL maintenance and development.
- Platform and query performance tuning.
- Ensure proper classification and handling of sensitive data.
- Diagnose and resolve end-user issues regarding server performance, database access issues, ETL and query performance.
- Writing performant, reusable, and maintainable code.
- Perform exploratory data analyses as needed.
- Develop data dashboards and reports for end-users.
- Assist in the ongoing deployment of server hardware, software, and applications; provide technical support, troubleshooting, diagnosis and problem resolution and maintenance.
- Assist in planning and migration of on-premise big data environment to public cloud.

1. A baccalaureate degree from an accredited college in computer science, engineering or a related field and four years of satisfactory full-time experience related to datacenter engineering and operations, cloud engineering and operations, complex IT infrastructure engineering; or,

2. A baccalaureate degree from an accredited college and eight years of satisfactory full-time experience related to datacenter engineering and operations, cloud engineering and operations, complex IT infrastructure engineering; or,

3. Education and/or experience which is equivalent to "1" or "2" above.

- Professional experience working with hundreds of billions of records in a distributed compute environment. - Minimum of (3) years’ professional experience in Hortonworks Data Platform (HDP), Hortonworks Data Flow (HDF), and/or Cloudera Data Platform (CDP)/Cloudera Data Flow (CDF). - 3-5 years of experience with Kafka, HBase, Hive, Nifi, Spark. - 3-5 years of data lake and data warehouse design. - Experience working with geospatial data at scale. - Fluency in Python, Scala, Java, and SQL. - Public cloud experience (AWS/GCP/Azure), certification a plus. - Proven professional experience migrating on-premise big data infrastructure to cloud based solutions. - Excellent organizational and interpersonal skills. - Ability to effectively prioritize and execute tasks in a high-pressure environment. - Ability to understand and gather client requirements as well as provide a solution that meets or exceeds client expectations.

As a prospective employee of the City of New York, you may be eligible for federal loan forgiveness programs and state repayment assistance programs. For more information, please visit the U.S. Department of Education’s website at https://studentaid.gov/pslf/

New York City Residency is not required for this position

The City of New York is an inclusive equal opportunity employer committed to recruiting and retaining a diverse workforce and providing a work environment that is free from discrimination and harassment based upon any legally protected status or protected characteristic, including but not limited to an individual's sex, race, color, ethnicity, national origin, age, religion, disability, sexual orientation, veteran status, gender identity, or pregnancy.